ARC 3.5-m Instrument history
Past problems and technical issues which our users should be aware of when reducing archival data. This page is in construction. More dates will be added for each instrument as time allows. If you have specific questions regarding your data and the status of the instrument in the past contact APO for more information.
AGILE
- AGILE Users Guide Appendix B. Instrument History
- 2014-07-22: Checked out after summer shutdown
- 2014-07-05: Warm for summer shutdown
- 2013-08-05: The instrument will remain powered down unless requested. We attempt to make every effort to power up and check out the instrument 3 days prior to a scheduled observing run. All other requests for the instrument should be made at least 48 hours in advance.
- 2012-01-24: APO still does not recommend taking dome flats with AGILE. More information can be found here.
- 2013-06-27: A new instrument block was obtained this week. As a result of an important update to the camera firmware while it was at Princeton Instruments the orientation in images has now changed. Images prior to April 5, 2013 will have North up and East to the left. Now as a result of the NVRAM update the sky in images will appear North up and East to the right by default.
- 2013-06-13: Agile was reassembled on the telescope. Continuing testing this week and next.
- 2013-06-03: Camera is back from Princeton Instruments. Performing bench testing in the computer room at this time.
- 2013-04-10: External trigger circuit for the camera appears to have failed. Sending the camera back to Princeton Instruments for repair.
- 2013-04-05: Camera is failing when commanded to take a sequence of triggered exposures.
- 2013-01-18: The instrument appears to be functional again. Waiting on Anjum to perform timing testing.
- 2013-01-17: The instrument control computer (ICC) is back up but only for slow readout rates (>10.8s). Hopefully this is just a library or driver issue. Fritz is working on the problem.
- 2013-01-07: The RAID (HDD) controller card in the instrument control computer (ICC) failed this weekend. A spare is on order. The instrument is unusuable. We will not be able to discover the full extent of the damage to the system until this card arrives.
- 2012-01-24: APO still does not recommend taking dome flats with AGILE. More information can be found here.
ARCES
- 2015-08-01: There was an instance where the ICC crashed apparently because of a stuck cal mirror in the guider box. We have had issues with the dirvers for these motors before. The motor control software is quite antiquaited. It is possible there are no timeouts in the software or the timeout was never triggered due to a loss of signal. Therefore the higherlevel software led to a unstable condition and crashed. Recovery was not difficult. The motor was assisted into the proper position, and the ICC needed a restart. Will look for a repeat of the trouble and consider action at that time.
- 2015-01-26: There was one instance in the past month where the science detector produced an image of all zeros. A reboot of the ICC corrected this. Please report repeat of the problem to the Observing specialist or the Telescope Engineer. So far we do not anticipate this is a sign of inital failure of a component or subsystem.
- 2014-09-22: Continued monitoring of the interoder light ratios indicate good results from the warm-up and cooldown to reduce scattered light from contamination in the light path
- 2014-09-11: CCD is again cold and interorder light ratios in both red and blue (18 and 45 respectively) look good. Will continue to monitor ovver the next week.
- 2014-09-08: Warming the CCD camera given there is a short break in the science schedule.
- 2014-09-03: 00:13 Interorder light ratio: 8.2
- 2014-09-02: Interorder light update
| date | UT | Interorder ratio | Chip Temp (C) | Notes |
|---|
| 2014-08-19 | 11:57 | 6.7 | -89 | |
| 2014-08-20 | 00:33 | 7.0 | -89 | |
| 2014-08-20 | 01:30 | | | raised set point further to -85 |
| 2014-08-20 | 22:02 | 7.4 | -85 | |
| 2014-08-20 | 23:00 | | | lowered set point to -110 |
| 2014-08-21 | 04:00 | 7.0 | -110 | |
| 2014-08-21 | 12:39 | 7.1 | -110 | |
| 2014-08-22 | 00:00 | 6.9 | -110 | |
| 2014-08-23 | 19:28 | 6.9 | -98 | |
| 2014-08-24 | 23:40 | 7.1 | -98 | |
| 2014-08-20 | | | | heater raised to maximum current, chip temp floating |
| 2014-08-26 | 11:45 | 7.2 | -84 | |
| 2014-08-27 | 00:00 | 7.2 | -82 | |
| 2014-08-28 | 00:17 | 7.3 | -86 | |
| 2014-08-30 | | 7.5 | -82 | |
| 2014-08-31 | | 7.6 | -82 | |
| 2014-09-01 | | 8.0 | -81 | |
| 2014-09-02 | 00:30 | 8.2 | -83 | |
- 2014-08-22: Increased the chip heater voltage to maximum. The CCD will only warm to about -82C at this voltage. Hopefully running at this value will slowly reduce the contamination. We will be monitoring the interorder light values daily as a way of checking. Tests show that even at this level the dark current is negligible in 20 minutes and should not be a concern for users.
- 2014-08-20: Russet reports that the CCD is showing signs of contamination. Attempting to remove it by raising the heater temperatures.
| date | UT | Interorder ratio |
|---|
| 2014-08-15 | 00:54 | 15.3 |
| 2014-08-15 | 22:14 | 9.2 |
| 2014-08-16 | 21:46 | 7.4 |
| 2014-08-17 | 22:10 | 7.2 |
| 2014-08-19 | 00:47 | 7.0 |
| 2014-08-19 | 07:08 | 7.1 |
| 2014-08-19 | 11:57 | 6.7 |
| 2014-08-20 | 00:33 | 7.0 |
- 2014-08-14: Instrument is cold and available for science. Completed ion pump swap earlier in the week and vacuum pumpdown. The Observing Specialists did a full checkout in the evening.
- 2014-08-11: Vacuum servicing this week. Instrument is unavailable
- 2014-07-16: Big tank has been pumped down to a vacuum greater than 100 milliTorr
- 2013-07-17: Jon Holtzmann (holtz at nmsu.edu) reports that he finally has a good spectral line solution for the new ARCES grating. Please contact him for further details.
- The new ARCES grating was successfully installed during 2012 summer shutdown. Do not use calibrations older than August 1, 2012. Ask Jon Holtzmann (NMSU) for the new iraf wavelength and dispersion solution files.
- ARCES Users Guide Appendix B. Instrument History
- 2013-03-20: External driver circuit is working. Back in service.
- 2013-03-18: After multiple reports of shutter problems with the echelle, we believe we have narrowed the trouble down to an intermittent shutter driver circuit for the instrument. This circuit is not repairable. Ed Leon intends to build a custom driver circuit to replace this portion of the instrument controls. The new driver circuit will be built and tested today and tomorrow. It is our hope to have the instrument ready for science Tuesday evening.
- 2013-02-05: The echelle is again cold with a good vacuum. It still is unclear at this time the true cause of the vacuum failure and warmup. Some additional characterization testing still needs to be done tonight but it should be ready for use tomorrow night.
- 2013-01-28: The instrument warmed over the weekend. It is unclear at this time if it was a failure of the autofill system or a loss of vacuum. We are investigating and will pumping on it this week.
- 2012-11-29: Installed shorter cables. Saw improved performance in echelle cal mirror performance but hope to get further improvement from reduceing cable length further. Another cable is on order.
- 2012-11-09: Replacement cables arrived but want to schedule the next round of repairs until there is a break in the echelle schedule.
- 2012-11-05: We are trying a number of possible solutions for the calibration mirror. Software and various hardware solutions are proving difficult due to the age of the equipment and the lack of spares. Our next attempt will be to replace some of the more key cables in the system. Those are on order.
- 2012-09-20: We believe the problem with varing QTH lamps is related to a problem with the calibration mirror. We have changed the stepper controlling this mirror, inspected the driver board and have aligned calibration light path. We will continue to work on this problem. Replacing the stepper driver board will be next during next weeks engineering time.
- 2012-09-05: Reports of varying levels in the CuS (blue) Flats with no change in the unfiltered Qth Flats caused me to suspect the bulb is starting to fail. Changed out the bulb in the QTH lamp today.
- 2012-08-10: The new MR152 grating was installed and aligned over summer shutdown. Please be aware not to use old cals or wavelength solutions since the wavelengths in each order have shifted. Contact the Observing Specialists or the Telescope Engineer for more details.
- 2012-06-22: Replaced Red and Blue slitviewer filters with new ND4 and ND5 filters. User guide updated with more information on the ND filters.
- 2012-03-04: Echelle slitviewer masks fixed
DIS
- 2015-10-05: Sticky bit issue resolved. The instrument with both cameras is available
- 2015-10-05: Cooled the red camera again on Friday. Cool down biases looked great. However once I started putting light on the detector a sticky bit (2E14) appeared. We will have to investigate further this week. Simple resets of the electronics did not fix or resolve it. Otherwise, we were able to focus the camera by moving the actuators all the way to the forward most end of their range and moving the red collimator lens near the forward end of its range.
- 2015-09-21: Red camera is on the vacuum pump and baking out water vapor. This will take a few more days with the water vapor content that is still in the dewar.
- 2015-09-16: Installed the spare red CCD42-20. Initial images taken warm look like the chip is alive but thermal signal is too great to tell much. Currently pumping on the dewar to remove as much water vapor as possible before cooling. We will make every effort to make the instrument available for blue science requests in the mean time.
- 2015-09-11: Preamp PCD boards check out. In discussions with E2V wether the chip can be RMA-ed. The instrument was made available for Blue science only.
- 2015-09-08: Removed DIS red ccd from dewar, preparing (waiting on parts) to test the pre-amplifier board. Attempting to RMA the red CCD with E2V. Waiting for RMA number. Working to make the instrument available for blue camera only.
- 2015-09-03: DIS is currently unavailable until further notice. The red CCD (or preamp) was damaged during the most recent vacuum servicing (to remove contamination from the field lens). APO staff are troubleshooting further.
- 2015-08-24: Warming the DIS cameras to clean contamination off the field lenses. It will be unavailable for science all week.
- 2015-08-01: Potential shutter problem with short exposures on the slitviewer and bright stars. Vertical bleeding of stars seen in short (less than 2 second) readouts. Does not appear to be a problem with guiding.
- 2015-06-04: A new version of TUI (2.4.0) is required to display the new slitviewer. DIS Users should install this new version of TUI. Most users will find binning the slitviewer camera 1x1 is fine for guiding and display however in the event of poor seeing or slow image downloads consider switching to 2x2 mode. Users will also notice a much faster readout time and increased sensitivity for the new camera.
- 2015-06-01: New slitviewer camera installed. Commissioning this next 2 nights. User manual to be updated as well this week.
- 2014-10-02: REd camera contamination significantly improved. We still do not have a quantitative measurement but visually using a pinhole mask it is nearly gone.
- 2014-09-30: Red camera is cold again. Investigating if contamination is still present. Instrument is usable at this time.
- 2014-09-26: Warmed Red camera to remove reported contamination. Instrument is not usable until this is resolved.
- 2014-07-15: Red and Blue cameras are cold. Instrument is ready for science
- 2014-07-07: Red and Blue cameras warm for vacuum servicing
- 2014-01-09: Grating 1 blue bay repaired. The problem was a bad/failed home switch. Some dis-assembly was required of that bay. After reassembly, the instrument robotics were rehomed and tested.
- 2014-01-03: DIS high res bay home switch has failed. Parts on order. The observing specialists will need to make grating swaps (sometimes in the middle of the night) until the problem is fixed. The Blue high resolution grating bay motor is not available until then.
- 2013-08-08: DIS Red camera is running 10 degrees warmer than prior to shutdown and servicing. Cause is under investigation. Currently there is no evidence that users will need to take extra darks with their calibrations.
- DIS Users Guide Appendix B. Instrument History
- 2013-08-08: DIS Red camera was showing a light leak. The bolts that mount the camera to the instrument are not creating a good seal. Taped over the interface.
- 2013-08-05: DIS Red and Blue cameras were serviced during shutdown. The Red camera received a new field flattener lens, a new aluminum nitride disk and the tensioning springs holding the field lens was removed. This last item was installed during the summer of 2012 to protect the lens from thermal cycling which we believed may have stressed the glass. This was certainly the cause for the dark current because the CCD was no longer in good thermal contact with the coldplate (in a sense floating). The blue camera was opened to clean the field lens which showed signs of water spots. Some investigation of the cause of the contamination was discussed by several engineers. Both cameras had their ion pumps replaced as well.
- 2013-06-26: DIS red is once again cold. There was marked improvement in the contamination/ scattered light as seen in the large wings in the profiles of spectral lines. Nothing was done with the blue camera at this time. We will address it during the summer shutdown unless it gets much worse. An improvement in the red cameras dark current was also seen however still is not fixed. There is a small tilt in the red camera spectra(20 pixels across the width of the chip) that will be addressed during shutdown. This is just an alignment issue with reinstallation.
- 2013-06-18: Warming of the red camera. I will be placing it on the vacuum pump this week in hopes we can reduce the magnitude of the contamination / scattering of the spectral wings. Other issues of the cause of this contamination, replacing the field corrector lens, vacuum issues will be addressed (or at least investigated further) later this summer during shutdown when we can open up the camera itself.
- 2013-06-03: Verified that the red camera has contamination on the field lens. To remove we will need to bring the instrument down for a minimum of 4 days to warm and bake out the water vapor. Any vacuum leaks will be addressed later in the summer during shutdown.
- 2013-01-10: Binning of the chips do not work for odd number of rows and odd number of columns. Odd number of one or the other or even number of both work. This is a lower priority problem. Recently it was also discovered that the TAI time in the headers is wrong due to the hardwiring of the UTC-TAI offset on the ICC. UTC time is correct in the headers. We will address these problems when we find the manpower. Users should be aware they exist though.
- 2013-01-04: Lowered heater setpoint to 160K.
- 2012-08-28: Yesterday we added an Aluminum Nitrade electrical isolation disk between the cold head and cold strap block inside the dewar. The Red camera was pumped on for 24 hours and cooled this morning. Initial tests show that the biases look normal again and that dark current is low (< 0.02 DN/sec).
- 2012-08-24: The Red camera is experiencing excessive noise as in readout, possibly due to extra dark current. It was taken offline today and warmed in preparation for a more thorough investigation next week.
- 2012-08-10: Over summer shutdown both DIS cameras and cryotigers were fully serviced. The Blue camera with a new cryogenic cold head. Both camera showed water vapor spots on their field lenses. They were removed and cleaned. There was minor damage done to the Red camera field lens when it was reinstalled.
- 2012-05-07: DIS Blue contamination shown to be gone.
- 2012-05-04: DIS Blue retruned to service late in the day. Unable to determine yet is contamination is still present.
- 2012-04-30: DIS removed from service today to deal with contamination on the blue camera. Expect it to be out of service until Thursday of this week
- 2012-04-24: DIS blue is tracking ambient temperatures and is running about 7 - 10 degrees warm (-87C). Extended wings on the spectrum suggest a scattering source on optics or the ccd. Will monitor and address if it continues to worsen. Otherwise, servicing this summer is planned.
- 2012-03-28: NEW B400 grating installed and aligned today.
- 2012-03-26: DIS Blue camera ion pump replaced. It is warm and still needs more pumping this week. It is not in service until further notice. DIS B400 Grating has been removed for swap with new grating.
- 2012-02-17: DIS Blue camera temperature is back to normal.
- 2012-02-16: DIS Blue camera vacuum has gone soft and is experiencing temperature regulation issues today. We are putting it on the vacuum pump. THis will take DIS out of service until the vacuum issue is resolved. If all goes well it will still b e available for this evening.
- 2012-02-10: DIS Users Guide updated today with new recommended exposure times for flat fielding using the truss lamps.
- 2012-02-09: New recommended exposure times for flat field lamps can be found here.The DIS User Guide will be updated shortly.
GIFS
- 2015-10-05: Hold time is down to about 13 hours. Carol and another Goddard person is here this week. As soon as they leave I'll put the camera back on a vacuum pump for a few days.
- 2015-08-03: Pump and cooled this week for upcoming scheduled usage
- 2015-03-10: The instrument is not on the schedule for the next quarter, leaving warm. It will not be available for use without 1 week advanced notice.
- 2014-09-30: Goddard team added additional thermal isolation to the cold block. The LN2 hold time nearly tripled to 26 hours. Camera is once again cold and with a good vacuum.
- 2014-09-15: Placed on vacuum pump in preparations for Sept 29th usage
- 2014-07-03: Warm for summer shutdown
- 2014-05-27: Recommend TUI 2.3.0b2 or later versions to control the instrument.
- 2014-05-19: The instrument is now fully configurable in TUI 2.3.0b1 or later.
- 2013-09-04: There is now a simple exposure widget for the instrument. Configuration commands must still be entered in the log window
NICFPS
- 2014-07-28: Ready for science, darks look normal.
- 2014-07-25: Nicfps is cooling. Should be ready by mid-weekend for science
- 2014-06-27: Replaced ion pump. Pumped instrument to 2e-3Torr. Left warm until later in July near the end of the shutdown when will return to service. The instrument is not usable at this time.
- 2014-06-23: Started to warm the instrument for a vacuum serving (ion pump change out) of the instrument.
- 2013-01-17: The instrument is currently not usable for Fowler Sampling modes greater than 8. NFS 8 and below still work fine.
- 2012-01-24: NICFPS still experiences intermittent failures since the fall of 2010. Please coordinate with the observing specialist for monitoring of failures and recovery modes as you take data. More information can be found here.
- NICFPS Users Guide Appendix B. Instrument History
- 2013-06-04: Ion pump was swapped and instrument on vaccum pump for the past 4 days. Cooled down today. Will do a functional test tomorrow after Leech controller has been reinstalled.
- 2013-05-31: Warming the instrument for vacuum servicing. The ion pump failed last night
- 2013-01-17: The instrument is currently not usable for Fowler Sampling modes > 8. NFS 8 and below still work fine.
- 2012-10-30: A report of a glowing obstruction as seen in the K-band was tracked down to a loose wire harness. That harness was re-secured
- 2012-01-24: NICFPS still experiences intermittent failures since the fall of 2010. Please coordinate with the observing specialist for monitoring of failures and recovery modes as you take data. More information can be found here.
- 2012-08-10: The NICFPS ICS Hard drive failed bringing the instrument back up. Fritz was able to install a new hard drive and reinstall the most current version of the software. A back-up was done at the same time.
SPICAM
- 2015-05-18: Evaluating the ion pump controller for the past 6 weeks has shown no dropouts of data or repeats of symptoms where the controller has shut itself down. Will continue to monitor but its unclear if their is a problem at all.
- 2015-03-30: Working with the manufacturer of the ion pump and controller to determine why the SPCe ion pump controller is going off line periodically. This will not affect data collection of the instrument.
- 2015-03-18: The instrument vacuum was soft as evidenced by the ion pump controller and independent MKS vacuum guage. Pumped on the dewar for 6 hours during the day today. Ion pum is again regulating the vacuum.
- 2014-06-18: Still no evidence that discharges of the ion pump result in a glow on the ccd. We believe the problem has been resolved.
- 2014-06-02: There is no evidence of light on the detector from ion pump discharge from small to moderate spikes in vacuum pressure, however we have yet to catch a large spike (> 4.5e-5 torr)
- 2014-05-09: SPIcam is cold again and available for science. I am still investigating the ion pump spikes, which still exist. During the servicinga 90 degree elbow and a screen were placed between the ion pump and dewar in hopes this will dispate the glow discharge from the ion pump when it spikes.
- 2013-05-02: SPICAM is still unavailable
- 2013-04-21: SPICAM is being warmed for vacuum and ion pump servicing. Hopefully we can address the high dark current spikes that occur on the order of once per day. This is a major servicing session and is planned to take the instrument out of use for science for several weeks.
- 2014-03-26: Ion pump controller returned to intermediate level. Instrument is again usable. Test did not disclose a change in behavior.
- 2014-03-20: Ion pump controller moved to inside instrument enclosure. Because cable lengths are short the instrument is not available until the night of March 27th. This test will eliminate cable length and age as a possible cause of the current spikes.
- 2014-01-12: There is still an intermittent dark current problem, we have yet to find a solution. The instrument is usable, but extra darks may be needed.
- SPICAM Users Guide Appendix B. Instrument History
- 2013-11-21: Instrument is cold and operational.
- 2013-11-12: Replaced ion pump with different style of pump from Gamma Vacuum
- 2013-11-10: Instrument is being warmed to replace ion pump
- 2013-09-30: Instrument is cold
- 2013-09-24: Instrument is being warmed to replace ion pump
- 2013-09-04: Russet has shown the dark current source originally appeared to be correlated with ion pump current but last night showed this is not always the case. Source of the noise is still under investigation
- 2013-08-17: There is an unknown source of dark current. It appears intermittent. See PR/1453 for details.
- 2012-09-28: Spicam is back with its new ion pump and controller. There is an electrical noise problem seen in biases and darks. Looks like a grounding problem from the ion pump through the coldhead and cold strap. Will address asap.
- 2013-01-25: Checked for contamination. None found. Back in service.
- 2013-01-24: Instrument cryotiger lost refrigerant last night and warmed. Pumped and recharged.
- 2013-01-04: Lowered heater setpoint to 160K.
- 2012-11-28: New instrument block. Instrument appears to be behaving fine. Lowered heater setpoint to 165K.
- 2012-11-21: Pressure transducers on the PCC was swapped. I relabled the connectors and corrected this problem. For monitoring purposes only but there will be a discontinuity in the logs. This does not affect instrument use.
- 2012-11-19: SPIcam is back together and ready for tonights observations. The instrument has a new window and the CCD is better electtically isolated from the vacuum vessel.
- 2012-11-08: UPDATE: We expect the new window to be shipped to APO from the coater on or before November 15. We also will be looking for a shipment of aluminum nitride electrical isolation disks to arrive on or before this date as well to help with the noise issue earlier described.
- 2012-10-06: A chip in the dewar window appears to be the source of the vacuum leak. The instrument will be down until further notice until a replace window can be made and AR coated.
- 2012-10-04: Appears to have a vacuum leak. System warmed for leak check and to replace vacuum fittings to ion pump. Instrument unavailable until these issues fixed
- 2012-10-01: Parts on order to address noise issue.
- 2012-09-21: The ion pump for SPIcam failed this evening. The instrument is warming for a replacement. It will not be available for use until this repair is done
- 2012-08-28: The cryotiger was serviced over shutdown. Temperatures continue to be running consistently 10 - 15 degrees warmer than expected. APO has a call in with Brooks Polycold for input.
- 2012-05-17: Returned to service today. Still needs an instrument block due to the amount of disassembly that was performed.
- 2012-04-26: Removed from service for maintenance
- 2012-04-24: Servicing scheduled starting Thursday April 26. The instrument will be unavailable after that date until further notice.
- 2012-04-09: SPicam's cold head appears to be partially restricted, limiting its cooling capacity. It is running 20 degrees above nominal. We therefore strongly recommend that users take darks to accompany their data. The following page gives more information of the problem. Plans are in progress to replace the coldhead the end of April and beginning of May.
- 2012-04-09: SPicam's cold head appears to be partially restricted, limiting its cooling capacity. It is running 20 degrees above nominal. We therefore strongly recommend that users take darks to accompany their data. The following page gives more information of the problem. Plans are in progress to replace the coldhead the end of April and beginning of May.
- 2012-04-03: Instrument returned to service.
- 2012-03-30: Instrument removed from service while we service the Cryotiger, lines and coldhead
- 2012-03-28: Camera partially warmed again. Still usable but darks are recommended. Will address the cryogenics system in the next few days
- 2012-03-06: Camera again cold. Contamination appears to be gone.
- 2012-03-01: Warmed and placed on vacuum pump to remove contamination. Not available until next week when we can take off pump and cool.
- 2012-02-27: Contamination on the CCD has reappeared. We were able to get the frost to partially sublimate off the chip by raising the chip heater a few degrees. Both visual and flat fields show evidence of contamination. About 400 counts above background. We are attempting to reduce the amount further however we will need several days to warm and pump on the instrument. For an example of what we are seeing look here.
- 2012-02-09: Spicam User Guide updated with new recommended exposure times for flat fielding using the truss lamps.
- 2012-01-24: SPIcam exhibits a low level pattern noise as seen in biases. This noise is very intermittent and has existed for the life of this instrument. Cause has never been determined. Examples of the SPIcam pickup noise as seen in biases can be seen here.
TRIPLESPEC
2015-10-05: Triplespec is back to misbehaving. Last night the observing specilist too several thoughsand exposures and there were perhas a hundred sequential images that were good, all the rest were bad. Attempting to do a reset this morning.
- 2015-09-18: The instrument still appears to be behaving. No failures seen in afternoon checkout images since mid August.
- 2015-08-24: The instrument has worked well the past few weeks. Nothing has changed with the instrument configuration that would have corrected the problem of corrupted images. We fully expect them to return at any time. We are planning a thermal cycle the beginning of September in hopes that it helps.
- 2015-06-15: Still in shared risk mode. Except for a few dozen bad images that were monitored over the weekend it appeared to perform well contrary to the 2 weeks where no good images could be obtained.
- 2015-06-08: The science camera is still producing corrupted images after repeated resets and hopes it will self heal. Check with the observing specialist before expecting to use it. Also plan your time with a back up instrument or program.
- 2015-06-01: Shared Risk Usage, currently we can not get it to take good images. Will continue to monitor.
- 2015-05-11: At this point the instrument appears to be more stable than unstable. Consider its use as shared risk. Plan a back-up instrument should you choose to use it. The observing specialist can try to recover from the corrupted image problem by power cycling and resetting the controller however this is not a sure fix. Sometimes it may take the instrument sitting idle for minutes, hours or days before it produces good images again. The problem is not well correlated with mounting / dismounting from the telescope, time of day, instrument rotation, etc. Nearly every piece of electronics outside the dewar has been tested or swapped with spares.
- 2015-05-07: No progress on determining the cause of the failures. It does not correlate with time of day, instrument rotation, rolling across the floor or jostling. We are running out of ideas.
- 2015-05-01: The controller is back from Leach. The controller and cards have been blessed by Bob Leach as being fine. The instrument is still exhibiting intermittent corrupted images. Continuing to test.
- 2015-04-13: The detector controller is still off at Leach for evaluation
- 2015-03-23: Another series of failures over this past weekend did not result in self correction by the controller or instrument. Taking the controller off the instrument and bringing it down to the bench to get to the individual boards to clean contacts, reseat and clean jumpers and their pins as well did not correct the problem. It was decided to ship the entire controller back to Astronomical Research Cameras (Bob Leach) for evaluation. The instrument will be unavailable and down until its return.
- 2015-03-18: Further troubleshooting of corrupted images problem. Got the instrument to work with the spare PCI card. We discovered however the reset switch on this card is bad. To reset the card, a full power-cycle of the computer is needed. After getting the instrument to take images again we saw only corrupted images. Tried a number of additional things with Matt's oversight. Will continue tomorrow. Unless the instrument self corrects itself again during checkout and starts taking good images again, the instrument is not usablefor science and is unavailable.
- 2015-03-16: The instrument has not corrected itself. Images taken during the weekend and this morning still show bad images.
- 2015-03-13: The obs-spec reported an occurange of the image corruption issue again during afternoon checkout.
- 2015-02-20: Another occurance of the image corruption error. The intermittant problem corrected itself again (within 24 hours) before further action of troubleshooting could take place.
- 2015-02-02: Power restored to the instrument. Images look fine.
- 2014-11-15: This problem is intermittant. It corrected itself by this morning. We will continue to watch each day if the problem returns but intermittant problems such as these are hard to troubleshoot.
- 2014-11-14: Image correction issue returned. Will investigate on Monday.
- 2014-11-05: Problem seems to have corrected itself. Will continue to monitor.
- 2014-11-04: Strange image corruption problem. Investigating. Instrument can not be used until corrected. An example can be found here.
- TRIPLESPEC Users Guide Instrument information
- 2014-07-24: Instrument is cold. Ready for science.
- 2014-07-16: Instrument is now warm. Started pumping.
- 2014-07-06: started warming for vacuum servicing.
- 2012-11-08: A reminder to users. Since the slitwheel is locked in place at the 1.1 arcsec slit, true instrument darks are not possible at this time.
- 2013-03-15: The ICC was found powered down yesterday. Recovery was a major undertaking requiring power cycles of both tspec and tcam leach controllers and reloading the dsp code. Most likely a gap in the documentation which will get updated.
- 2012-11-01: New instrument cart is now in full use. It was causing interference with the lip on the floor below the NA2 rotator so we cut a section of that kickplate lip away.
- 2012-10-20: Temperature logging fixed but not reported to APO however in the meantime critical alerts were off. They are back on now.
- 2012-09-21: Found the instrument was not properly logging temperatures. Power cycled the aux box for Matt Nelson. It is back up
- 2012-08-10: A reminder to users. Since the slitwheel is locked in place, True instrument darks are not possible at this time.
- 2012-02-19: The slitwheel mechanism has been powered off and disabled at this time. We are working with the UVa Instrument team to develop a repair plan. It currently is locked in the 1.1 arcsecond slit position.
- 2012-02-10: Exposure times for flat fielding using the truss lamps have not changed with the new lamps for Triplespec.
- 2012-02-10: Triplespec slit wheel has been experiencing problems where it freezes position or is sluggish to respond to commands. We are afraid that it will move to a non slit position and render the instrument unusable. As a result the engineers at UVa and APO staff are recommending the slit not be moved from the default 1.1 arcsecond slit, at least until we have a better handle on what is going on.
- 2012-02-07: Tripespec was exhibiting a low level intermittent pattern noise that can be seen in FS=8. It is on the order of +/- 20 DN noise. It has also been seen at lower levels (factor of 4) in FS=1 going back to April, 2010. This noise has been identified as getting worse than a few DN peak to peak as recently as this past December. For more information look here. APO is actively working on resolving this problem. We believe we may have made improvements today but it may take afew days to a week of monitoring to tell. Contact us if you see this in data after this date.
Misc
