The manager API on gateway at LDAS_MIT
Legend:
Normal status or debugging message
Notable condition which may be a non-fatal error
Error condition not fatal to job
Error condition fatal to job
Notable condition which is not an error
Currently undefined
Condition requires email notification of the responsible administrator of this API
Condition requires phone notification of the responsible administrator of this API Link: API Status Page for LDAS_MIT
911941052 STARTUP archiveLog /ldas_outgoing/logs/LDASmanager.log.html (file4) closed (archived as /ldas_outgoing/logs/archive/managerAPI/LDASmanager.911941051)
911941052 STARTUP execOverload overloaded exec
911941052 STARTUP openListenSock port 10002 (emergency) opened on gateway as sock7
911941052 STARTUP ::exec exec /usr/sbin/ifconfig -a
911941052 STARTUP mgr::perApiRunningStats read perAPIstats into ::mgr::apirunstats array
911941052 STARTUP getPid getting pids '18253 20358 ' from lockfile /ldas_outgoing/diskcacheAPI/.diskcache.20358.lock
911941053 STARTUP mgr::bootstrapAPI diskcache pid is ''
911941053 STARTUP mgr::shutdownAPI obtained diskcache pid 20358 from lock file
911941053 STARTUP mgr::shutdownAPI deleted lock file: /ldas_outgoing/diskcacheAPI/.diskcache.20358.lock
911941053 STARTUP mgr::postMortem stdout/stderr for previous run of diskcache API: http://18.120.0.97/ldas_outgoing/logs/archive/diskcacheAPI/diskcache.log.911941053
911941053 STARTUP exec exec uname -n
911941054 STARTUP mgr::bootstrapAPI bootstrapping diskcache API on dataserver
911941055 STARTUP mgr::bootstrapAPI Startup command for diskcacheAPI: 'alias cd=cd && cd /ldas_outgoing/diskcacheAPI && /bin/env RUNDIR=/ldas_outgoing HOST=dataserver PATH=/ldas/bin:/ldas/bin:/ldcg/bin:/bin:/usr/bin LD_LIBRARY_PATH=/ldas/lib/diskcacheAPI:/ldcg/lib DB2INSTANCE=ldasdb GLOBUS_GSI_CRED_DEBUG_LEVEL=0 GLOBUS_GSI_PROXY_DEBUG_LEVEL=0 GLOBUS_GSI_GSS_ASSIST_DEBUG_LEVEL=0 GLOBUS_ERROR_OUTPUT=1 GLOBUS_ERROR_VERBOSE=1 LD_PRELOAD= GLOBUS_GSI_OPENSSL_ERROR_DEBUG_LEVEL=0 GLOBUS_GSI_SYSCONFIG_DEBUG_LEVEL=0 GLOBUS_GSSAPI_DEBUG_LEVEL=0 /ldas/bin/diskcacheAPI >& /ldas_outgoing/diskcacheAPI/diskcache.log &'
911941065 STARTUP getPid getting pids '27773 ' from lockfile /ldas_outgoing/diskcacheAPI/.diskcache.27773.lock
911941065 IDLE emergency_callback executed: eval {archiveIndex /ldas_outgoing/logs/archive/diskcacheAPI/LDASdiskcache.911522478}
911941065 IDLE emergency_callback executed: mgr::pushFTPandHTTPinfo diskcache 1.10.1
911941065 IDLE emergency_callback executed: set ::DISKCACHE_HASHFILE_NAME_ASCII frame_cache_dump
911941065 IDLE emergency_callback executed: set ::DISKCACHE_HASHFILE_NAME_BINARY .frame.cache
911941066 STARTUP mgr::bootstrapSuccessEmail done. diskcache API is now running as pid:
911941066 STARTUP getPid getting pids '20462 ' from lockfile /ldas_outgoing/frameAPI/.frame.20462.lock
911941067 STARTUP mgr::bootstrapAPI frame pid is ''
911941067 STARTUP mgr::shutdownAPI obtained frame pid 20462 from lock file
911941067 STARTUP mgr::shutdownAPI deleted lock file: /ldas_outgoing/frameAPI/.frame.20462.lock
911941067 STARTUP mgr::postMortem stdout/stderr for previous run of frame API: http://18.120.0.97/ldas_outgoing/logs/archive/frameAPI/frame.log.911941067
911941067 STARTUP exec exec uname -n
911941069 STARTUP mgr::bootstrapAPI bootstrapping frame API on dataserver
911941069 STARTUP mgr::bootstrapAPI Startup command for frameAPI: 'alias cd=cd && cd /ldas_outgoing/frameAPI && /bin/env RUNDIR=/ldas_outgoing HOST=dataserver PATH=/ldas/bin:/ldas/bin:/ldcg/bin:/bin:/usr/bin LD_LIBRARY_PATH=/ldas/lib/frameAPI:/ldcg/lib DB2INSTANCE=ldasdb GLOBUS_GSI_CRED_DEBUG_LEVEL=0 GLOBUS_GSI_PROXY_DEBUG_LEVEL=0 GLOBUS_GSI_GSS_ASSIST_DEBUG_LEVEL=0 GLOBUS_ERROR_OUTPUT=1 GLOBUS_ERROR_VERBOSE=1 LD_PRELOAD= GLOBUS_GSI_OPENSSL_ERROR_DEBUG_LEVEL=0 GLOBUS_GSI_SYSCONFIG_DEBUG_LEVEL=0 GLOBUS_GSSAPI_DEBUG_LEVEL=0 /ldas/bin/frameAPI >& /ldas_outgoing/frameAPI/frame.log &'
911941109 STARTUP getPid getting pids '27877 ' from lockfile /ldas_outgoing/frameAPI/.frame.27877.lock
911941109 IDLE emergency_callback executed: eval {archiveIndex /ldas_outgoing/logs/archive/frameAPI/LDASframe.910379006}
911941110 IDLE emergency_callback executed: mgr::pushFTPandHTTPinfo frame 1.10.1
911941110 STARTUP mgr::bootstrapSuccessEmail done. frame API is now running as pid:
911941110 STARTUP getPid getting pids '15118 ' from lockfile /ldas_outgoing/metadataAPI/.metadata.15118.lock
911941111 STARTUP mgr::bootstrapAPI metadata pid is ''
911941111 STARTUP mgr::shutdownAPI obtained metadata pid 15118 from lock file
911941111 STARTUP mgr::shutdownAPI deleted lock file: /ldas_outgoing/metadataAPI/.metadata.15118.lock
911941111 STARTUP mgr::postMortem stdout/stderr for previous run of metadata API: http://18.120.0.97/ldas_outgoing/logs/archive/metadataAPI/metadata.log.911941111
911941111 STARTUP exec exec uname -n
911941111 STARTUP mgr::bootstrapAPI bootstrapping metadata API on metaserver
911941111 STARTUP mgr::bootstrapAPI Startup command for metadataAPI: 'alias cd=cd && cd /ldas_outgoing/metadataAPI && /bin/env RUNDIR=/ldas_outgoing HOST=metaserver PATH=/ldas/bin:/ldas/bin:/ldcg/bin:/bin:/usr/bin LD_LIBRARY_PATH=/ldas/lib/metadataAPI:/ldcg/lib DB2INSTANCE=ldasdb GLOBUS_GSI_CRED_DEBUG_LEVEL=0 GLOBUS_GSI_PROXY_DEBUG_LEVEL=0 GLOBUS_GSI_GSS_ASSIST_DEBUG_LEVEL=0 GLOBUS_ERROR_OUTPUT=1 GLOBUS_ERROR_VERBOSE=1 LD_PRELOAD= GLOBUS_GSI_OPENSSL_ERROR_DEBUG_LEVEL=0 GLOBUS_GSI_SYSCONFIG_DEBUG_LEVEL=0 GLOBUS_GSSAPI_DEBUG_LEVEL=0 /bin/env LD_PRELOAD=/ldas/lib/libpreload.so /ldas/bin/metadataAPI >& /ldas_outgoing/metadataAPI/metadata.log &'
911941116 STARTUP getPid getting pids '26093 ' from lockfile /ldas_outgoing/metadataAPI/.metadata.26093.lock
911941116 IDLE emergency_callback executed: eval {archiveIndex /ldas_outgoing/logs/archive/metadataAPI/LDASmetadata.910379011}
911941116 IDLE emergency_callback executed: mgr::pushFTPandHTTPinfo metadata 1.10.1
911941117 STARTUP getPid pids 26093 obtained from ps for /ldas/bin/metadata
911941117 STARTUP mgr::bootstrapSuccessEmail done. metadata API is now running as pid: 26093
911941117 STARTUP getPid getting pids '15218 ' from lockfile /ldas_outgoing/ligolwAPI/.ligolw.15218.lock
911941117 STARTUP mgr::bootstrapAPI ligolw pid is ''
911941117 STARTUP mgr::shutdownAPI obtained ligolw pid 15218 from lock file
911941117 STARTUP mgr::shutdownAPI deleted lock file: /ldas_outgoing/ligolwAPI/.ligolw.15218.lock
911941117 STARTUP mgr::postMortem stdout/stderr for previous run of ligolw API: http://18.120.0.97/ldas_outgoing/logs/archive/ligolwAPI/ligolw.log.911941117
911941117 STARTUP exec exec uname -n
911941118 STARTUP mgr::bootstrapAPI bootstrapping ligolw API on metaserver
911941118 STARTUP mgr::bootstrapAPI Startup command for ligolwAPI: 'alias cd=cd && cd /ldas_outgoing/ligolwAPI && /bin/env RUNDIR=/ldas_outgoing HOST=metaserver PATH=/ldas/bin:/ldas/bin:/ldcg/bin:/bin:/usr/bin LD_LIBRARY_PATH=/ldas/lib/ligolwAPI:/ldcg/lib DB2INSTANCE=ldasdb GLOBUS_GSI_CRED_DEBUG_LEVEL=0 GLOBUS_GSI_PROXY_DEBUG_LEVEL=0 GLOBUS_GSI_GSS_ASSIST_DEBUG_LEVEL=0 GLOBUS_ERROR_OUTPUT=1 GLOBUS_ERROR_VERBOSE=1 LD_PRELOAD= GLOBUS_GSI_OPENSSL_ERROR_DEBUG_LEVEL=0 GLOBUS_GSI_SYSCONFIG_DEBUG_LEVEL=0 GLOBUS_GSSAPI_DEBUG_LEVEL=0 /ldas/bin/ligolwAPI >& /ldas_outgoing/ligolwAPI/ligolw.log &'
911941123 STARTUP getPid getting pids '26191 ' from lockfile /ldas_outgoing/ligolwAPI/.ligolw.26191.lock
911941123 IDLE emergency_callback executed: eval {archiveIndex /ldas_outgoing/logs/archive/ligolwAPI/LDASligolw.910379024}
911941123 IDLE emergency_callback executed: mgr::pushFTPandHTTPinfo ligolw 1.10.1
911941123 STARTUP getPid pids 26191 obtained from ps for /ldas/bin/ligolw
911941123 STARTUP mgr::bootstrapSuccessEmail done. ligolw API is now running as pid: 26191
911941123 STARTUP getPid getting pids '3809 ' from lockfile /ldas_outgoing/datacondAPI/.datacond.3809.lock
911941124 STARTUP mgr::bootstrapAPI datacond pid is ''
911941124 STARTUP mgr::shutdownAPI obtained datacond pid 3809 from lock file
911941124 STARTUP mgr::shutdownAPI deleted lock file: /ldas_outgoing/datacondAPI/.datacond.3809.lock
911941124 STARTUP mgr::postMortem stdout/stderr for previous run of datacond API: http://18.120.0.97/ldas_outgoing/logs/archive/datacondAPI/datacond.log.911941124
911941124 STARTUP exec exec uname -n
911941124 STARTUP mgr::bootstrapAPI bootstrapping datacond API on datacon
911941124 STARTUP mgr::bootstrapAPI Startup command for datacondAPI: 'alias cd=cd && cd /ldas_outgoing/datacondAPI && /bin/env RUNDIR=/ldas_outgoing HOST=datacon PATH=/ldas/bin:/ldas/bin:/ldcg/bin:/bin:/usr/bin LD_LIBRARY_PATH=/ldas/lib/datacondAPI:/ldcg/lib DB2INSTANCE=ldasdb GLOBUS_GSI_CRED_DEBUG_LEVEL=0 GLOBUS_GSI_PROXY_DEBUG_LEVEL=0 GLOBUS_GSI_GSS_ASSIST_DEBUG_LEVEL=0 GLOBUS_ERROR_OUTPUT=1 GLOBUS_ERROR_VERBOSE=1 LD_PRELOAD= GLOBUS_GSI_OPENSSL_ERROR_DEBUG_LEVEL=0 GLOBUS_GSI_SYSCONFIG_DEBUG_LEVEL=0 GLOBUS_GSSAPI_DEBUG_LEVEL=0 /ldas/bin/datacondAPI >& /ldas_outgoing/datacondAPI/datacond.log &'
911941129 STARTUP getPid getting pids '18270 ' from lockfile /ldas_outgoing/datacondAPI/.datacond.18270.lock
911941130 IDLE emergency_callback executed: eval {archiveIndex /ldas_outgoing/logs/archive/datacondAPI/LDASdatacond.910379051}
911941130 IDLE emergency_callback bad key recieved: fe01d67a002dfa0f3ac084298142eccd
911941130 IDLE emergency_callback executed: mgr::pushFTPandHTTPinfo datacond 1.10.1
911941130 STARTUP getPid pids 18270 obtained from ps for /ldas/bin/datacond
911941130 STARTUP mgr::bootstrapSuccessEmail done. datacond API is now running as pid: 18270
911941130 STARTUP getPid getting pids '15319 ' from lockfile /ldas_outgoing/eventmonAPI/.eventmon.15319.lock
911941130 STARTUP mgr::bootstrapAPI eventmon pid is ''
911941130 STARTUP mgr::shutdownAPI obtained eventmon pid 15319 from lock file
911941130 STARTUP mgr::shutdownAPI deleted lock file: /ldas_outgoing/eventmonAPI/.eventmon.15319.lock
911941130 STARTUP mgr::postMortem stdout/stderr for previous run of eventmon API: http://18.120.0.97/ldas_outgoing/logs/archive/eventmonAPI/eventmon.log.911941130
911941130 STARTUP exec exec uname -n
911941131 STARTUP mgr::bootstrapAPI bootstrapping eventmon API on metaserver
911941131 STARTUP mgr::bootstrapAPI Startup command for eventmonAPI: 'alias cd=cd && cd /ldas_outgoing/eventmonAPI && /bin/env RUNDIR=/ldas_outgoing HOST=metaserver PATH=/ldas/bin:/ldas/bin:/ldcg/bin:/bin:/usr/bin LD_LIBRARY_PATH=/ldas/lib/eventmonAPI:/ldcg/lib DB2INSTANCE=ldasdb GLOBUS_GSI_CRED_DEBUG_LEVEL=0 GLOBUS_GSI_PROXY_DEBUG_LEVEL=0 GLOBUS_GSI_GSS_ASSIST_DEBUG_LEVEL=0 GLOBUS_ERROR_OUTPUT=1 GLOBUS_ERROR_VERBOSE=1 LD_PRELOAD= GLOBUS_GSI_OPENSSL_ERROR_DEBUG_LEVEL=0 GLOBUS_GSI_SYSCONFIG_DEBUG_LEVEL=0 GLOBUS_GSSAPI_DEBUG_LEVEL=0 /ldas/bin/eventmonAPI >& /ldas_outgoing/eventmonAPI/eventmon.log &'
911941136 STARTUP getPid getting pids '26291 ' from lockfile /ldas_outgoing/eventmonAPI/.eventmon.26291.lock
911941136 IDLE emergency_callback executed: eval {archiveIndex /ldas_outgoing/logs/archive/eventmonAPI/LDASeventmon.910379055}
911941136 IDLE emergency_callback executed: mgr::pushFTPandHTTPinfo eventmon 1.10.1
911941137 STARTUP getPid pids 26291 obtained from ps for /ldas/bin/eventmon
911941137 STARTUP mgr::bootstrapSuccessEmail done. eventmon API is now running as pid: 26291
911941137 STARTUP getPid getting pids '3273 ' from lockfile /ldas_outgoing/cntlmonAPI/.cntlmon.3273.lock
911941137 STARTUP exec exec /bin/ps -fu ldas | grep cntlmon
911941137 STARTUP getPid child process exited abnormally
911941137 STARTUP mgr::bootstrapAPI cntlmon pid is ''
911941137 STARTUP mgr::shutdownAPI obtained cntlmon pid 3273 from lock file
911941137 STARTUP mgr::shutdownAPI deleted lock file: /ldas_outgoing/cntlmonAPI/.cntlmon.3273.lock
911941137 STARTUP mgr::postMortem stdout/stderr for previous run of cntlmon API: http://18.120.0.97/ldas_outgoing/logs/archive/cntlmonAPI/cntlmon.log.911941137
911941137 STARTUP exec exec uname -n
911941138 STARTUP mgr::bootstrapAPI bootstrapping cntlmon API on gateway
911941138 STARTUP ::exec exec /bin/env GLOBUS_GSI_CRED_DEBUG_LEVEL=0 GLOBUS_GSI_PROXY_DEBUG_LEVEL=0 GLOBUS_GSI_GSS_ASSIST_DEBUG_LEVEL=0 GLOBUS_ERROR_OUTPUT=1 GLOBUS_ERROR_VERBOSE=1 LD_PRELOAD= GLOBUS_GSI_OPENSSL_ERROR_DEBUG_LEVEL=0 GLOBUS_GSI_SYSCONFIG_DEBUG_LEVEL=0 GLOBUS_GSSAPI_DEBUG_LEVEL=0 RUNDIR=/ldas_outgoing LD_LIBRARY_PATH=/ldas/lib/cntlmonAPI:/ldcg/lib DB2INSTANCE=ldasdb cntlmonAPI >& /ldas_outgoing/cntlmonAPI/cntlmon.log &
911941138 STARTUP mgr::bootstrapAPI Startup command for cntlmonAPI: '/bin/env GLOBUS_GSI_CRED_DEBUG_LEVEL=0 GLOBUS_GSI_PROXY_DEBUG_LEVEL=0 GLOBUS_GSI_GSS_ASSIST_DEBUG_LEVEL=0 GLOBUS_ERROR_OUTPUT=1 GLOBUS_ERROR_VERBOSE=1 LD_PRELOAD= GLOBUS_GSI_OPENSSL_ERROR_DEBUG_LEVEL=0 GLOBUS_GSI_SYSCONFIG_DEBUG_LEVEL=0 GLOBUS_GSSAPI_DEBUG_LEVEL=0 RUNDIR=/ldas_outgoing LD_LIBRARY_PATH=/ldas/lib/cntlmonAPI:/ldcg/lib DB2INSTANCE=ldasdb cntlmonAPI >& /ldas_outgoing/cntlmonAPI/cntlmon.log &'
911941143 STARTUP getPid getting pids '15733 ' from lockfile /ldas_outgoing/cntlmonAPI/.cntlmon.15733.lock
911941143 STARTUP exec exec /bin/ps -fu ldas | grep cntlmon
911941143 STARTUP mgr::bootstrapSuccessEmail done. cntlmon API is now running as pid:
911941143 STARTUP bgLoop Looping process cntlmonping started
911941143 STARTUP bgLoop Looping process datacondping started
911941143 STARTUP bgLoop Looping process diskcacheping started
911941143 STARTUP bgLoop Looping process eventmonping started
911941143 STARTUP bgLoop Looping process frameping started
911941144 STARTUP bgLoop Looping process ligolwping started
911941144 STARTUP bgLoop Looping process metadataping started
911941144 STARTUP leakLogger inital size of manager API: 31688 kB
911941144 STARTUP bgLoop Looping process etchosts started
911941144 STARTUP bgLoop Looping process memflag started
911941144 STARTUP bgLoop Looping process statpagefile started
911941144 STARTUP bgLoop Looping process statpage started
911941145 STARTUP bgLoop Looping process hidedirs started
911941145 STARTUP bgLoop Looping process leapsecs started
911941145 STARTUP bgLoop Looping process bslock started
911941145 STARTUP bgLoop Looping process blockage started
911941146 STARTUP bgLoop Looping process createassistant started
911941146 STARTUP bgLoop Looping process writestatus started
911941146 STARTUP bgLoop Looping process expireusers started
911941146 STARTUP bgLoop Looping process diskusagelimit started
911941146 STARTUP exec exec df -m /export/ldcg_server/ldas_outgoing/managerAPI
911941146 STARTUP bgLoop Looping process logrotate started
911941146 STARTUP bgLoop Looping process livejobpush started
911941146 IDLE exec exec /ldcg/bin/lsof -b -p 15540
911941146 IDLE openListenSock port 10001 (operator) opened on gateway as sock5
911941146 IDLE setResourceLimit vmemoryuse=unlimited; datasize=unlimited; core=unlimited; getRLimit on maxproc error:'Invalid or unsupported resource name: maxproc'; descriptors=65536; getRLimit on memorylocked error:'Invalid or unsupported resource name: memorylocked'; filesize=unlimited; cputime=unlimited
911941146 IDLE mgrGlobusChannel::init cmd gt_xio_socket -server mgrGlobusChannel::accept -gridmap_file -myaddr gateway 10031
911941147 IDLE mgrGlobusChannel::init Globus channel gtxiosock0 opened for gsi service at 10031
911941147 IDLE mgrGlobusChannel::init cmd gt_xio_socket -server mgrGlobusChannel::accept -gridmap_file -myaddr gateway 10030
911941147 IDLE mgrGlobusChannel::init Globus channel gtxiosock1 opened for gsi service at 10030
911941147 IDLE emergency_callback executed: eval {archiveIndex /ldas_outgoing/logs/archive/cntlmonAPI/LDAScntlmon.911272024}
911941148 IDLE emergency_callback executed: mgr::pushFTPandHTTPinfo cntlmon 1.10.1
911941185 IDLE mgr::addUser new user added to users.queue: '-name evang -password ***** -email egoetz@umich.edu -fullname {Evan Goetz} -phone 734-764-5146 -expires 2009-12-4 {}'
911941185 IDLE mgr::addUser new user added to ::QUEUE(USERS): '-name evang -password ***** -email egoetz@umich.edu -fullname {Evan Goetz} -phone 734-764-5146 -expires 2009-12-4 {}'
911941185 IDLE emergency_callback user info updated successfully
915772874 IDLE gridFtpToplevel Subject: LDAS-MIT managerAPI gridftp config error!; Body: ldas-mit managerAPI gridFtpToplevel: grid ftp directory: '/usr1/grid/ldas' not found. are you sure it is visible from the host gateway ?