ABSTRACT

The ARC 3.5-m telescope began operations in 1994. Shortly thereafter a program was undertaken by the observatory management and staff to improve durability, reliability, maintainability and improvements to the telescope and facility to reach the planned scientific potential of the telescope. This program is built around the minimal staffing level for the observatory.

A maintenance plan was developed with the objectives of reducing down time and providing data consistency. Preventive maintenance was addressed with respect to preventing system failures and performance degradation. An online reporting system was established for staff and observers to report telescope and instrument problems.

Two types of improvement plans were devised. The first was for ongoing improvements that could be handled with existing observatory resources. These improvements consisted of new or redesign of current systems and the support of visiting instruments. Collaborative visiting instruments are brought in to enhance the observatory cache. These instruments come in for 2 to 3 night observing runs and greatly increase the science benefit of the telescope by using the latest advances in scientific instruments. Finally an enhancement plan was established that provided for additional funding and technical support to design and install new systems for the telescope and instrument upgrades.

Keywords: maintenance, visiting instruments, improvements, 3.5-meter telescope, Astrophysical Research Consortium, Apache Point Observatory

1.0 Introduction

There are two main objectives for having a telescope maintenance program. The first is to reduce lost observing time. Downtime for equipment failures can consume valuable on sky time. Secondly a well maintained telescope system will enhance data consistency. By maintaining telescope performance qualities we attempt to provide data consistency to all the instruments. Maintenance is subdivided into Routine and Repairs. The key for routine repairs is to make them easy and requiring minimal time so that technicians can perform them and so that they will be performed. Routine maintenance items that require; more than an hour to complete, special tools, and more than one person would often get overlooked for other pressing jobs. Successful repairs were those that were accomplished quickly with minimal loss of on sky time. This requires preplanning in the way of spares, training, and documentation.

1.1 Routine

Generally, persons consider routine maintenance as Preventive Maintenance (PM). However, Preventive Maintenance is usually thought of those items that are done to prevent mechanical failures such oiling and changing filters. Routine operations are much more than that. While routine maintenance must include the typical PM items it must also include performance-oriented items and actively looking for problems so that they can be caught early.

a Further author information - Deputy Site Manager, APO, Email mark/apo.nmsu.edu; www. apo.nmsu.edu,, voice: (505) 437-6822 , fax (505) 434-5555

An annual PM checklist was initially compiled by looking at manufacturers’ data sheets on recommended service intervals and making adjustments based on actual use, but never less than the manufacturer’s recommended intervals. Additionally, systems were looked at with respect to use or criticality. If heavy use could be predicted, maintenance was scheduled immediately following this time. Critical items were serviced 30 days prior to their use. Finally, items were sorted based on weather conditions. Items that must be accomplished outside were scheduled for the warmer, drier months.

The result was twelve PM sheets that spread various daytime maintenance tasks throughout the year. Training was provided to the technicians and manufacturers' data sheets were copied and attached for easy reference. The first of the month was set aside for the PM items and usually these are accomplished in 1 to 1 1/2 days. If higher priority items came about at that time there was still the rest of the month to complete the PM. Once completed the monthly sheet would be signed off and returned to the engineer. Since these sheets often included safety items, a year of completed sheets are sent off-site for storage in the event of a catastrophe at the facility.

1.1.2 Performance Maintenance

The telescope optics are maintained with a regular cycle of CO2 snow cleaning ¹. Table 1-1 shows measured optical data of the 3.5m mirrors after routine CO2 cleanings. The data illustrates the effects of cleaning and dust accumulation. In September of 1998 we were still experiencing some wind blown dust then again in late January levels picked up again. As the scatter increased the reflectance continued to drop. The highest drop was in the tertiary which is continuously exposed, next, the primary, which is covered during the daytime and in high dust events, and finally the secondary, which while continually exposed, is downward facing. The data also shows little dust accumulation during November, December and most of January when there is little wind blown dust in the air. Initially the mirrors were CO2 snow cleaned monthly. This has been increased to bi-monthly, after major dust events ^*, or when scattered light is observed to have increased during visual inspections. In 1996 and 1997 tests were carried out to quantify the effects of C02 cleaning and compare that against laser cleaning. The test was carried out between the observatory, University of Washington, and STI Optronics.³ The results showed that CO2 cleaning on average removes approximately 35% of the accumulated dust while the Laser cleaning removed approximately 60% of the accumulated dust. While the increase in cleaning efficiency with the laser is significant, it was not deemed cost effect at the time.

Some observations about CO2 Snow cleaning.

• Use of CO2 above 70% Rh. will likely cause condensation on the mirror surface,
  
• The TMA Microscan requires contact with the mirror surface, consequently it is not used until after the mirror has been CO2 cleaned. Use of it prior to cleaning may force dirt particles into the aluminum coating or cause scratches,
  
• The TMA Microscan must be at the same temperature as the measured surface. Errors as large as 8% have been seen with a 20 Deg. C differential,
  
• A dielectric mirror is used as a calibration reference surface and used to correct reflectance data. Without this calibration mirror errors as large as 5 % are common. This mirror is kept covered and in a controlled environment.

As referenced to earlier, particle counters are used in addition to standard meteorological instruments to monitor the environmental conditions. The site has two Met 1 particle counters. One located outside the dome and used for determining opening conditions. The second located inside the dome near the primary mirror that is used to determine closing conditions, filter efficiency, and major dust events for mirror cleaning.

Mirror Reflectance 

Mirror Scatter 50,180 BSDF

The dome is equipped with a 7000 CFM fan that pressurizes the inside of the dome to .1" of water through high efficiency filters. The fan operates whenever the dome is closed and the humidity of the incoming air is less than 80% Rh. Additionally the fan is used when open if the inside dust counts approach the closing limits. While each particle counter measures particles greater than .3 and 1 microns, the larger particles determine the operating limits. These systems require periodic servicing including calibration and filter changes. All of this data is placed onto a easily accessible web page and archived.

Recently, contact washing of the mirrors has taken place with some success. The procedures used were a variation of the Canadian France Hawaii Telescope (CFHT) washing procedures as detailed in reference 2. The significant variation was in the water temperature used in the washing and rinse applications. Due to the structure and type of glass used in the Hextek and Spun-cast honeycomb mirrors used on the 3.5-m the use of hot water was deemed too risky in that a large temperature gradient could be generated across the glass structure causing undue stresses to buildup. Consequently the water temperature is stabilized to the mirror temperature before application. This has likely caused some reduction in cleaning efficiency. Table 1-2 shows before and after data for several mirrors all using the same general procedure and measured with the TMA Microscan.

Drive system components also require performance-oriented maintenance. Each day as part of the pre-opening checklist the drive surfaces are checked then cleaned as necessary. This cleaning becomes a daily event during moth season from early June through October. The drives are wiped clean with WD-40 except for every 4th time during when alcohol is used to reduce any oily buildup. Encoder wheels must also be cleaned regularly. Failure to keep the drive surfaces clean results in severely degraded pointing and tracking errors. Approximately quarterly pointing models, servo parameters and the position maps are checked and often need adjustment. It is believed that the changes are required in part due to thermal expansion of the pier and the telescope support structure.

Future plans for continued quantitative performance monitoring involve a monthly engineering observing night to measure ;

• Blind tracking performance,
  
• Random pointing accuracy,
  
• Nightly routine image quality measurements,
  
• Standard star throughput measurements,
  
• Guided tracking performance,
  
• Telescope efficiency using shutter-open criteria.
  

2.1 Introduction

System malfunctions are inevitable even with the most thorough designs, best quality components, and a comprehensive maintenance program. There are a number of things that are done to make these repairs have the least impact on the telescope operations. They start with staffing, carry through to spares inventory and end only when the system is back in operation, preferably better than it was before.

2.1.1 Staffing

One of the most important aspects of getting systems back on line is the people that not only repair the equipment but also the persons that report problems. For repair systems to be efficient, problems must be first noticed, then conveyed to the repair-person completely and in a language they understand with all the required information. Since many of the problems are discovered during nighttime operations by non-engineering personnel and repaired during the daytime by the technical staff this can become a problem. It is important to hire personnel that will not fall into endless turf battles and will work together to solve the problem regardless of whose area it falls into. One quality to look for is persons with education and experience in multiple disciplines. Prior work experience especially in industry is helpful in developing these interdisci plinary skills. Sometimes the formal education level isn’t the most valuable asset.

Another quality that helps in reporting and correcting problems is a person’s powers of observation. Countless times problems were identified and repaired before failure because a sound or characteristic was noticed and brought forward for discussion at its discovery. Listening to systems often tell you well in advance of impending failure. Creating a staffing plan that does not put sole responsibility on a person or group promotes broader knowledge and experience encourages persons to follow through on a problem until corrected. Table 1.3 shows the APO 3.5-m onsite technical support staff. The instrument builders and scientist off site provide instrument support with the on-site staff providing local assistance.

• 1 FTE Mechanical Engineer with experience in vacuum systems and machining,
  
• 1/2 FTE Mechanical Engineer with electrical and instrumentation experience,
  
• 1 FTE Electronic Technician with experience in camera and control systems,
  
• 1/2 FTE Computer administrator with experience in software and telecommunications,
  
• 1/2 FTE Maintenance Technician with experience in plumbing and HVAC.
  

Hiring the right staff is useless if they are not retained. Retention is based on many factors but they are all taken into account. The list below is in no particular order. The value of each is dependent upon the individual employee.

• Pay and Benefits,
  
• Working hours, facilities, relationship with other employees,
  
• Professional development,
  
• Living conditions away from work, (hobbies, recreation, family, social life),
  
• Support and recognition from management,
  
• Sufficient budget to support their needs (facilities, state of the art equipment).
  

2.1.2 Problem Reporting

A web-based database was created that can be accessed by anyone, (i.e. visitors, astronomers, operators, engineers) to report a problem. The database sends an email to a central point of contact that reviews the information for completeness and content. Problems are then forwarded to a person assigned to address the issue. Once addressed a reply is sent to the point of contact who updates the database and decides whether the problem report should be closed or moved into another suspense category. The database is reviewed monthly to check on open problems. The person who files the problem report is sent an email once the problem is closed or other permanent action is decided upon.

Other reports that are used in recording either potential problems or actions are a nightlog and a day log. The nightlog is used for many purposes including time accountability, focus information etc. but it will contain comments about items that may at the time not warrant a problem report. If a problem report is issued a note is also mentioned in the night log. The day log is used to inform the nighttime staff of events that occurred during the day that correct deficiencies or might cause problems during operations (i.e., accessed an area and might pull a cable loose)

2.1.3 Documentation

Complete system documentation is essential in getting repairs done quickly. This has been one of the hardest aspects to control in getting new equipment. It seems that in-order to get the system delivered so much time and money has been spent that there is nothing left to create or organize the documentation. It is essential that this final step not be overlooked. Table 2.1 is a list of documentation desired for a given system. The form of the documentation will be dependent on the preferences of the staff. Electronic files are often the most useful in the long run since it supports changes easily and can be turned into hardcopy with a minimal amount of time. However one must be careful that only authorizes changes be incorporated. We have elected a combination of both electronic files and paper copies based on the information type.

It is not possible to address each item of documentation in this limited space however I would like to mention one often overlooked item, design philosophy. Systems that are built off site by temporary staff such as graduate students can be difficult to repair years later if the builder's design philosophy is not known. We generally deal in state of the art systems that are complex and quite valuable. For an outside technical person to address the system after years in service it is most helpful for that person to know the rationale behind the design. Unfortunately this documentation requires the builder to write a narrative of the design, something not necessarily required to build the system as would be a schematic for instance. This is time consuming and to the builder redundant. Consequently, it is often left out.

1. Theory of Operation: Detail description of how the instrument functions and purpose of individual component assemblies. Assemblies should be broken down into; Optics, Detector, Power Supply, Control/Status, Digital Signal Processing, Analog Signal Processing, I/O, Computer Software/ Firmware, and Control and Display Software,

2. Operating Procedures: Detail procedures for startup, operation and shutdown. Includes details on all input commands and responses. Procedures should include any special handling instructions, startup and power down sequences. Procedures should be written at the telescope operator level,

3. Design Criteria and Specifications: Description of the criteria governing the design and function. Includes detailed specifications that would be of interest to the Astronomer. Show calculations and test data involving performance characteristics. Include electrical loading, CG, and weight,

4. Electrical Schematics: Detail schematics of all wiring and circuit cards including pin functions. Includes detail part list,

5. Mechanical Drawings: Complete assembly and detail drawings of parts as manufactured. Parts list of purchased parts,

6. Troubleshooting and Test Equipment: Assembly/Disassembly instructions of component assemblies. Voltage and waveforms for testpoints, adjustable bias and signal levels and other crucial signals. Procedures for identifying the fault to a specific assembly. Procedures for setting; gains, bias, etc. List any special test equipment required,

7. Periodic Maintenance: List any periodic maintenance items that may be required,

8. Control Software/Firmware: Detailed description of operation. Include copy of source code. Signal/Math Flow-chart,

9. Control and Display Software: Detailed description of operation. Signal processing/math flow chart. Hardware and software configuration requirements and I/O interface.

2.1.4 Spares Inventory

Another often overlooked area essential to quick and efficient repairs is a suitable spares inventory. The driving factor in what and how much to spare must be determined on a system-by-system basis at the time of design and assembly. One of the most important factors that must be known by the designer is the value of telescope time and the impact of the system loss on the telescope. If the telescope is non-functional as a result of the failure as in an axis drive motor then the delivery time versus telescope time is a direct relationship. If the system failure effects only 1 of 3 instruments then the telescope is still operational but at only a 66% value. Designers tend to be overly optimistic about their designs and if left without a guideline will likely supply only minimal spares.

• If it completely shuts down operations then 100% spares for items under $2,000/ea up to 20% of project cost or lead times greater than 3 days,
  
• If it is an isolated system that would allow partial science operation (i.e. Nasmyth 2 port only) then 100% spares on items under $500/ea up to 10% of project cost or lead times greater than 4 weeks,
  
• If a part is common for more than 1 system then do not duplicate if spared at 100% with at least 1 spare per two systems,
  
• When approaching the maximum dollar amount, look at high failure items or easily damaged items first,
  
• Some expensive items that are critical to a critical system (shuts down all operations) may have to be spared anyway based on: cost, lead time, long term availability, MTBF, serviceability, etc. The designer must weigh these decisions but the system must have an estimated life of at least ten years with less than 1-day downtime,
  
• Spares must be delivered with the system to the site.
  

2.1.5 Redesign

Finally, when systems require repair it is important to consider improvements to the system during the repair process. Often time constraints will not allow a redesign immediately and if that is the case the data that is necessary must be collected before placing the system back into operation. A redesign is not necessarily a major effort. These efforts can be anything from adjusting bearing pre-loads to a replacement control system. In the past four years we have made significant changes to problems in operating systems that were discovered during a repair process. These include: axis drive box bearing and disk alignments, rotator bearing pre-loads, collimation procedure, secondary and tertiary mirror support bearings and lubrication. In all of these cases the engineer overseeing a repair noticed a problem that upon further investigation revealed flaws in the initial design or assembly. In some cases these systems had undergone numerous repairs but the process was simply to follow the standard procedure and not address underlying issues, such a s appropriate lubrication. In particular we had numerous parts that were in operation for years that when re-researched were found to not meet the environmental conditions such as humidity and temperature they were being used in. This caused higher than expected vibration and friction loads and caused the material to fail and corrode at a substantially quicker rate.

2.2 Results

Table 2.3 illustrates the success in part through implementation of these intensive maintenance efforts with respect to improvement in lost time. The lost time due to equipment failures has dropped 6.8% of total telescope time which goes directly back to science time. The amount of time needed for engineering has dropped back to the original level of 12% although still slightly higher than the planned 10% for routine operations as extra time is still being taken to implement the Enhancement Plan projects.

Year	Engineering Use (% of total)	Equipment Loss (% of total)
1999*	12.8	2.6
1998	13.3	4.8
1997	37.4	4.7
1996	20.3	6.4
1995	12.1	9.4
*1st half of the year

With regard to data consistency, one Princeton University program headed by Dr. Edwin Turner has reported their DIS (Dual Imaging Spectrograph) gravitational lens monitoring program achieved g and r band *relative* photometry (comparing variable QSO images to presumed constant stars) at slightly better than1% rms. accuracy in a half dozen or so different fields through the period of Dec 1994 through June 1998.

3.0 Introduction

For a telescope to stay competitive it must stay current. To stay competitive we took on three stages of improvements to the telescope that was designed to;

Improve reliability and efficiency that translates directly to cost savings,

Improve performance that provides more and better science,

Use of visiting instruments that increase the amount and type of science data being generated.

Some of these projects are accomplished with little additional funding or manpower, others require a substantial increase over the annual budget. What needs to be done and how it is accomplished becomes a substantial job for the management team based on inputs from users. The larger the dollar amount required, and the longer implementation takes, the more effort required by management to acquire funding. Thus there are different levels of management for each improvement category.

3.1 Ongoing

Ongoing improvements are managed by a single engineer and the group is comprised of permanent support staff primarily on site and over-sight is done by observatory management. Projects are generally funded from the annual budget and use existing staff. Duration is usually no more than 30 man-days of effort and the priority is low to moderate. Table 3.1 is a list of some of the previous improvement projects. As these same staff members are responsible for the day-to-day operation of the facility, long- term and high-priority projects are not well suited to be worked by this group. Completion schedules often become lost due to more immediate issues that arise and must take precedence. Which projects go into this category is based on input from users, the problem reporting system, and the group. These projects often have the objective of improved reliability and efficiency.

3.2 Enhancement Plan

Major projects that will require additional resources in both staff and funding are planned in three-year cycles. Inputs come primarily from the users and the Director annually, based on three-year predictions, acquires funding. The group consists of staff, and additional institutional technical and astronomical personnel. Management is by the Telescope Scientist with oversight by the Director. Projects in this category generally require; more than $5,000 in funding, more than three months to complete, are high priority, and can not be accomplished due to time constraints or expertise by on-site staff. Recent projects accomplished through this plan were; Nasmyth 2 guider upgrade, tertiary mirror rotation, and replacement secondary mirror figuring. Table 3.2 is a list of the first three enhancement plan projects.

Task

Task

Task leader

Priority

Creation

Suspense

%

Notes

#

Date

Date

C/l

1

Mirror cover counterweights

J Davis

low

3/5/1998

7/1/2000

5%

2

Relocate controllers

J Davis

low

3/5/1998

8/1/1998

100%

3

Replace vent tubes

J Davis

mod>high

3/5/1997

6/15/1997

100%

4

2dry EPROM mech. limit

J Davis

high

3/5/1997

7/1/1997

100%

5

ln2 fill

J Bri.nk

low

3/5/1997

7/1/2000

90%

grim remaining

6

enclosure snow

M Klaene

moderate

3/5/1997

open

60

ongoing

7

Hartmann mask

J Brink

moderate

3/5/1997

3/1/1999

100%

8

r naught Tele

C Stubbs

moderate

3/5/1997

7/1/1997

100%

11

Pri mirror temp

C Stubbs

mod>high

3/19/1998

6/1/1999

100

Highest Priority

New Guider Installation,

Primary Mirror Support System,

Temperature measurement system,

Collimation Procedures,

New Secondary Procurement.

Lower Priority

Motorized rotation and positioning of tertiary mirror,

Set up parameter monitoring system,

Baffling,

DIS improvements, including new detectors,

New observing software and user interfaces,

Defeat 20 Hz problem,

Mirror cover and lamp automation,

Image quality aspects: r_o telescope, closed-loop focus system, fast guiding.

 

------------------------------------------------------ ---------------------------------------------------------------------- ---------------------------

3.3 Visiting Instruments

Visiting instruments have had a significant presence at APO. Several instruments have been integrated onto the 3.5-m telescope with success. How much success varies with the amount of preplanning that is accomplished. These instruments often come on site for a single 2- or 3- night observing run. In order to make the most of this limited time the instrument, facilities, telescope, and staff must be ready upon arrival. All visiting instruments have a Primary Investigator (PI) assigned to them from within the consortium. Usually it is this PI that starts the planning process to bring the instrument to the observatory. The PI also becomes the primary point of contact between staff and the instrument team.

For the observing run to be a success several steps must be completed. Initially a staff member provides the instrument team a list of questions that starts the dialogue. The questions address items such as required utilities, shipping, weight and balance, physical dimensions, instrument block data, as well as policies on heat rejection, mounting and dismounting, vacuum pump operation, need for lights and access. Once the questions are returned completed, the staff address the impact to the observatory and resources that will be required to integrate this particular instrument. Generally the observatory limits resource commitment to 2 man-days. After reviewing the impact and the instrument design the observatory approves the run and it can then be scheduled. Proper instrument scheduling can play a significant part in success. Often the needed time on the sky is only 1 to 2 half-nights.

Approximately 30 to 60 days prior to the observing runs the instrument team is required to visit the observatory to discuss final details and see the operation. Upon arrival, the instrument is placed in the lab where it is assembled and checked out. This usually takes 1 day. Once it is operational the instrument is moved to the observing level the day of or prior to observing. After the instrument is on the observing level, access to it and run time is limited so not to interfere with other observing. Currently the 3.5-m is scheduled in 1/2 night blocks. Since visiting instruments often require additional staffing and time to get ready for observing, scheduling observing the first 1/2 night is critical especially for the first night. When practical full nights are desired so to avoid instrument changes and potential impacts to other programs. In addition weekdays are preferred so that the maximum day staff is available in case of problems. When possible an engineering observing night is scheduled the night prior to the visiting instrument such that some engineering time can be used for integration. The first time on the sky is used for focusing and instrument block data. For first-time instruments this may take three hours to generate. If the instrument is new to the telescope or in early stages of development, a day off the telescope after the first night may be scheduled to allow time to recover from significant problems. By adding additional days to the schedule and adding days between the schedule it also allows for flexibility in the event of poor weather.

The following visiting instruments have either observed or in planning stages for observing on the 3.5m

• Sodium Cell Jupiter imager
  
• Occultation imager
  
• Goddard Space Flight Center (GSFC) InSb imager
  
• GSFC Fabry-Perot
  
• GSFC Near IR Acousto Optic Tunable Filter imager
  
• Livermore Labs Fourier Transform imager
  
• GSFC 12-micron imager
  

4. Acknowledgments

Table 2.1 was in part provided by Jim Fowler, Apache Point Observatory

Table 2.2 was provided by Karen Loomis , Apache Point Observatory

Table 3.2 was provided by Christopher Stubbs, University of Washington

Thanks to Bruce Gillespie for his review, encouragement, and comments.

5. References

1. B. Mcgrath, D. Nahrstedt "A Cleaning Process for the CFHT Primary Mirror", Astronomical Society of the Pacific

108:620-623, 1996 July

2. B. Mcgrath, D. Nahrstedt, "Cleaning the CFHT Primary Mirror", Large Mirror Workshop CFHT April 1995

3. W. Kmura, L. Sanborn, APO Mirror Cleaning Experiment, Under contract to University of Washington, 1997 August