make printable	new:	see also:
	- v1.1: new optional config keys: CONDOR_AB (if DRS_TYPE=CPL) DAILY_CASCADES (if no autoDaily processing) Ignore in standard systems! See special.	The tool visualizes the condor processing queue on the MUC blades. Current muc02 view ... See also cascadeMonitor
	- v1.2: improved display for CONDOR_AB=NO	databases none dfos tools condor_status, condor_q output mucMonitor_mucnn.html, exported to http://qcweb.hq.eso.org/ALL/ upload/download upload: mucMonitor_mucnn.html (see output); down: config.mucMonitor_mucnn (with wget)
topics: description | interface: scheduled jobs | condor output | details | queue | cascades | operations | configuration | special

mucMonitor

Description

tool monitors condor execution

This tool visualizes the condor processing situation on a MUC blade. It is particularly useful on the muc01...muc05 systems with their multi-user setup.

The mucMonitor gives an overview of the current condor activity (nodes executing condor jobs) and the pending queue. It further links to the cascadeMonitor of the operational users (called 'registered' users). It helps to understand the processing activity on the MUC blade.

The top panel has update information. The cadence of the tool is quite high since the condor processing might be quite dynamic (timescale of changes less than a minute). The browser refresh is hard-coded to 10sec. The 'ago' bracket (calculated by the browser) turns red as soon as the tool is older than one minute, this to indicate that the tool is not up-to-date (either due to the fact that it is currently not called by any of the operational users, or there is a problem with condor.

The horizontal navigation has the following links:

muc02

cascade:

ALL

giraffe

uves

xshooter

queue

other

link to mucMonitor

overview per muc

links for the operational users

list of current queue jobs

link to other mucMonitors

The links 'muc02' and 'queue' relate to the mucMonitor, the others to the cascadeMonitor. These display an overview ('ALL') and details for the operational users (these are configured; there could be more accounts on a muc server but they do not get listed here unless configured):

Operational users: giraffe uves xshooter

vultur_exec_cascade jobs. Next it lists the currently executing condor queues:

Scheduled vultur_exec_cascade jobs:

UID	STIME	JOB
uves	14:40	/home/uves/jobs/execAB_CALIB_2013-01-13
xshooter	14:38	/home/xshooter/jobs/execAB_CALIB_2013-01-13
giraffe	14:42	/home/giraffe/jobs/execAB_CALIB_2013-01-13

UID=user ID; STIME=start time; JOB=job file under $DFO_JOB_DIR being currently executed.

Main table. Then it displays the 'condor_q' output:

Condor nodes on muc02         slot4         slot8         slot12

busy | idle | reserved/not available

'busy' indicates that condor currently executes a job on that core; 'idle' means no current condor job; 'reserved/not available' means the core is not configured for condor jobs, or is currently not available. Note that this overview indicates only the condor situation. Some cores are reserved for interactive jobs, so they might be busy e.g. with certifyProducts or trendPlotter and would not be indicated on this monitor. Likewise, an idle condor core might actually be running a non-condor job.

The monitor includes the Ganglia load report (the last-hour overview of the load on that muc server).

If a node is condor-active, the job is displayed (e.g. processAB), along with the AB name and the user ID, in the Details panel:

Details: This is the output of 'condor_status':

The load is the last-minute average.

Queue. The 'queue' tab lists all jobs in the condor queue that are currently waiting (i.e. not dependency-locked but not yet executing due to the muc blade being core-limited).

Cascade monitor. The tabs between the first and the last tab are filled by links to the account-specific cascade monitors. They provide a visualization of the current status of the latest condor processing cascade (e.g. CALIB_2013-01-13 for xshooter). The cascade monitor is created by the tool cascadeMonitor, see more details there.

Output

• mucMonitor_<hostname>.html (the main page, behind the first link) and mucMonitor2_<hostname>.html (the page behind the 'queue' link) in $DFO_MON_DIR, also exported to http://qcweb/~qc/ALL

How to install

• standard installation with dfosInstall

How to use

Type mucMonitor -h for a quick help, mucMonitor -v for the version number. Type

mucMonitor

on the command line to create or refresh http://qcweb/~qc/ALL/mucMonitor_<hostname>.html.

Operations

While calling on the command line is possible anytime, the tool should be running operationally in an infinite loop on one account per muc blade only. Remember it is a visualization of 'condor_q', and condor is an installation on a machine, not on an account. For instance, there is one condor installation on muc02, and no matter if giraffe, uves or xshooter call 'condor_q', they will all see the same result. This is also true for 'mucMonitor'. Calling it individually from each account would not break anything but cost unnecessary performance. On the other hand it doesn't matter which account is executing mucMonitor. For that reason, the output indicates the current execution node of mucMonitor.

Self-organized mode. Therefore the tool should run operationally in a self-organized way: If the browser has discovered the tool is not refreshed anymore (how?), you should start the tool on your own account and thereby "take over". Killing or starting a session on another account is not possible (unless in emergency you log in via qc_shift). Of course, if your own session is hanging, you can/should fix it right at the source.

The proper way of calling the infinite loop is

watch -n 10 mucMonitor

Do this in a dedicated window. It's a good idea to do it in a separate xterm so that you easily recognize it.If needed, you (but nobody else) can always terminate the session with Ctrl+C.

On the AB monitor, you also have a button to start the mucMonitor in the infinite loop. The AB monitor has a browser-independent way of finding whether a mucMonitor session exists on the muc blade: it greps in the 'ps' output for the string mucMonitor. If nothing is found, it displays the standard red cross as alert.

The output will be linked to the dfoMonitor (this is TBD). The output is linked to the public monitor page as "MUC monitor".

Outdated.

The browser calculates the difference between the current time and the timestamp in the mucMonitor (same way as e.g. in the trendPlotter or calChecker pages). If the page is older than 1 minute (!), it turns red and tries to catch your attention. Thereby you can normally expect the output to be near-real time.

Timeouts. Sometimes the command 'condor_q' is hanging. The tool has a timeout mechanism: if after 6 seconds there is no condor_q response, it exits and tries again after the 'watch' time. The configuration file download has also a timeout mechanism.

Configuration file

Because of the self-organized way of operations, the tool needs a central configuration being identical for all participating accounts. This is why there is a master configuration file http://www.eso.org/~qc/dfos/tools/config.mucMonitor_<hostname> that is downloaded automatically each time the tool executes. It overwrites the local version under $DFO_CONFIG_DIR/config.mucMonitor_<hostname>.local. The download is done via wget and has a timeout protection (10 sec). On timeout, the local version is read instead. It is pointless to edit the local version (it gets overwritten upon the next execution).

The central configuration file should not be edited unless after coordination.

It contains the topology of the executing muc blade, and labels marking the role of the nodes ('condor_execution' or 'free'):

Special cases

There are two special cases:

a) the PHOENIX processing on muc08/muc09/muc10 (having no autoDaily process): there is no monitoring of the condor cascade (cascadeMonitor). The corresponding standard links are disabled there via the optional config key DAILY_CASCADES (by default YES, here set to NO).

b) the MUSE processing on muc09/muc10, using DRS_TYPE=CPL (because of internal parallelisation); then there is no need to monitor the condor pattern. This is controlled by the optional key CONDOR_AB, set to the default YES.