New antenna control computers for the Compact Array

History

The ATOMS (Australia Telescope Observatory Management System) project was proposed in 1995 as an upgrade path for the observatory control/management software and hardware. A specific aim of the project was to replace the aging PDP-11 based antenna control computers (ACCs) with Intel PC-based hardware to provide a continued maintenance path. The old ACCs were proving increasingly more difficult and expensive to maintain.

Initial efforts went into a software development framework based on the ATOMS web server. In keeping with the times, most software was to use an object-oriented design. The platform independent and object-oriented java language was chosen as a base for all high level software. This decision was based mainly on its perceived status as a clean object-oriented language with good prospects for future support, without the baggage of old features carried by languages like C++. The lower level C and C++ code running the new ACCs was developed for the pSOS operating system, mainly because the Epping electronics group was already using this product. We later came to regret this decision, but it is not clear a superior alternative existed at the time.

In the early days the project was carried by David (aka George) Loone. (While at Narrabri, George was forced to take on his middle name due to an excess of Daves confusing all conversations.) His efforts were instrumental in getting the project underway and we are still rediscovering useful software in the far corners of the ATOMS library. He was responsible for the design of the ATOMS software environment, major parts of the infrastructure software (e.g. message logging) and the java interface to the new ACC's servo control system. The servo system is the software that controls the motion of the antenna as it slews to and tracks sources. Wim Brouw implemented the pSOS servo code during 1998/1999 and initial tests were done towards the end of that time using antenna 6 and the Mopra antenna.

Early in 2000 David Loone announced his imminent departure to the private sector and I was approached to take over the project to complete the ACC transition. Foolishly I accepted and began work on an interface to the current VMS software, to enable a transitional phase where we could drive the new ACCs from the existing software. During a few feverish months before David Loone's departure, two Daves (McConnell and Loone) created a design for a major missing component: Cycle. This component takes care of all `cycling' interfaces in the antenna (i.e. devices that need to know about the integration cycle) and precisely timed monitoring of things like total power and sampler statistics. It was then up to Scott Cunningham, who had been working on the pSOS dataset interface software until then, to implement this component.

From David Loone's departure until the arrival of a new Dave, Dave Brodrick, we were down to two active developers and progress slowed. With the arrival of Dave Brodrick, we gained a developer who was not afraid of hardware and low-level pSOS software (unlike me) and also spoke java. Things started speeding up again and Scott's pSOS knowledge was rapidly transferred to Dave. This turned out to be very timely since Scott was soon to move on as well, leaving Dave and me to hold the fort. By this stage the software was far enough advanced that all major components had been tested on the antenna, and, we naively thought, only minor problems remained. Meanwhile Dave McConnell found some spare time to design and help with work on the initial version of the new monitoring system which was used to display data from the new mm-receivers on the web.

But the bugs continued to bite. Once we were able to drive the antennas and control the various devices we thought we were close to the final switchover, especially after we obtained our first fringes with the new system on 14 March 2002. However, pSOS had some surprises in store for us in the form of intermittent faults. There were several network related problems, some of which were resolved with a vendor supplied patch, but others plague us to this day. We also ran into problems with the floating-point libraries, forcing us to write our own square-root routine and other math functions. David managed to find paths around the various problems, and eventually the ACCs were booting stand-alone from FlashMemory (i.e. a memory card used in digital cameras). After a few more trial runs with friendly observers we switched over for what will hopefully turn out to be the last time in early May 2003.

Current status and plans

The new ACCs have now been running the array for several weeks. Some observers have been lucky to encounter no problems, while others have had more trouble. The main failure mode of the new ACCs is related to the creation of new network connections, so failures tend to occur at program change-over, especially remote observing start and finish. Recovery is generally quick, via remote reset. We expect to iron out most problems over the coming weeks, but may be stuck with the network problem for longer, since it may originate in proprietary pSOS code. In the longer term we may wish to migrate away from pSOS and use Linux instead.

At present the new ACCs do not provide much extra functionality to the observer, but this will change with time. The main thing observers will notice is the DriveMon window, providing fast graphical monitoring of the antenna tracking errors (and other drive information). It has already uncovered a number of previously unknown tracking problems and drive oscillations and will be a valuable tool in improving the millimetre pointing performance. The monitoring system is the next big change in the pipeline. We currently have an industrial experience student, Le Cuong Nguyen, working with us for six months and making good progress. The new system will move away from the CAMON displays and provide more flexible graphical monitoring tools and watchdogs to keep an eye on array performance and system health. It will include more sources of monitoring and be a lot more flexible, e.g. monitoring intervals are adjustable per item from 0.1 second to minutes or hours, and trends can be graphically explored by viewing monitor point values against time. We aim to provide on-line access to an archive of monitor data going back at least one year.

With the new software comes increased flexibility we will be able to control the subreflectors to improve system performance at high frequency (pending hardware fixes) and accommodate requests for unusual observing modes. We could, e.g. use continuously scanning mosaics instead of individual pointings or scan the dishes in a circle around your source to reduce pointing errors.

If you are thinking about novel ways to use the Compact Array antennas or would just like some specific information about the new system, feel free to contact David (David.Brodrick@csiro.au) or myself (Mark.Wieringa@csiro.au).

Mark Wieringa
(Mark.Wieringa@csiro.au)