Äîêóìåíò âçÿò èç êýøà ïîèñêîâîé ìàøèíû. Àäðåñ îðèãèíàëüíîãî äîêóìåíòà : http://www.eso.org/~qc/docu/detmon/detmon.pdf Äàòà èçìåíåíèÿ: Tue Sep 22 11:16:02 2009 Äàòà èíäåêñèðîâàíèÿ: Sat Oct 17 00:07:35 2009 Êîäèðîâêà:

Detector Monitoring as part of VLT Science and Data Flow Operations
Wolfgang Hummela , Lander de Bilbaob,d,e , Andrea Mo diglianib , Lars Lundinb , Paola Amicoc , Pascal Ballesterb , Gaspare LoCurtoc , Leonardo Vanzic,f European Southern Observatory, Data Management and Operations Division, Karl-Schwarzschild-Str. 2, D-85748 Garching, Germany b Europ ean Southern Observatory, Software Development Division, Karl-Schwarzschild-Str. 2, D-85748 Garching, Germany c Europ ean Southern Observatory, Paranal Science Op erations Department, Alonso de Cordova 3107, Vitacura, Casilla 19001, Santiago 19, Chile d Fundaci´n Espanola para la Ciencia y la Tecnolog´a, o ~ i Rosario Pino 14-16, E-28020 Madrid, Spain Instituto de F´sica de Cantabria - Centro Superior de Investigaciones Cient´ficas, i i Avenida de los Castros s/n, E-39005 Santander, Spain f Potificia Universidad Catolica de Chile, Department of Electrical Engineering Avenida Vicuna Mackenna 4860, Santiago Chile
ABSTRACT
The ESO Paranal observatory is operating a heterogeneous set of science detectors. The maintenance and quality control of science detectors is an important routine task to retain the technical and science performance of the instrumentation. In 2006 a detector monitoring working group was built devoted with the following tasks: inventory of the currently existing detector calibration plans and monitored quality characteristics, completion and homogenization of the detector calibrations plans, design and implementation of cross-instrument applicable templates and data reduction pipeline recipes and monitoring tools. The instrument calibration plans include monthly and daily scheduled detector calibrations. The monthly calibrations are to measure linearity, contamination and gain including the inter-pixel capacitance correction factor. A reference recipe has been defined to be applicable to all operational VLT instruments and has been tested on archive calibration frames for optical, near- and mid-infrared science detectors. The daily calibrations measure BIAS or DARK level and read-out noise in different ways. This has until now prevented cross detector comparison of performance values. The upgrade of the daily detector calibration plan consists of the homogenization of the measurement method in the existing pipeline recipes. Keywords: detector monitoring, calibrations, quality control, gain, linearity
a

e

1. INTRODUCTION
The Paranal Observatory in Chile is hosting the ESO VLT and the VLTI. Observatory operations includes science observations and calibrations, which are scheduled according to an instrument specific calibration plan. The purpose of the calibration plan is to provide sufficient calibrations to reduce science data and to remove
Further author information: (Send corresp ondence to W.H.) W.H.: E-mail: whummel@eso.org L.d.B.: E-mail: lbilbao@eso.org A.M.: E-mail: amodigli@eso.org P.A.: E-mail: pamico@eso.org


instrumental signatures. A second purpose of the calibration plan is to derive quantities describing the quality of relevant instrument components and to monitor these to carry out preventive maintenance. One of the most important and sensitive components of the VLT facility are the science detectors of which a versatile set is operating for different instruments of the VLT. In 2006 the detector monitoring group was created to review the instrument specific detector related calibration plans and to coordinate and implement an improved cross-instrument detector calibration strategy. The mile stones can be summarized as: · Review operational calibration plans of detector specific calibrations. · Design a cross-instrument detector monitoring plan including the definition of priorities of the characteristics to be monitored and their observing frequency. · Design and implement cross-detector pipeline recipes to monitor in a common way the performance of all VLT science detectors. In this article we describe the design and implementation of the pipeline recipes and the underlying algorithms. The results of the first two goals have been described in Ref. 1. The calibrations on which the detector monitoring pro ject has focused its scope are: · a set of flat and bias/dark frames acquired with monthly cadence to measure the detector linearity, their gain (corrected for interpixel capacitance autocorrelation), the fixed pattern noise, the number of nonlinear pixels used to generate a map of non-linear pixels. For the optical detectors the contamination is also measured. · a set of daily calibrations to measure the bias/dark level and the read-out noise.

2. THE DETECTOR MONITORING PLAN AND TEMPLATE DESIGN
VLT observations are carried out through the execution of observing and calibration blocks which consist of one or several templates. The review of the instrument calibration plans has shown that to monitor the read-out noise of each science detector a set of bias or dark calibrations is acquired daily and that there is no need to upgrade or modify the current daily calibration templates. The calibration templates generate in most cases five bias frames, for optical detectors, or three dark frames, in the near infrared domain. We found instead that in case of the monthly scheduled calibrations, the templates strategy is, when available, still incomplete and heterogeneous. Most of the existing templates need to be migrated smoothly. In case of the linearity templates for UVES, GIRAFFE and SINFONI, that provide a sequence of consecutive flat pairs and dark or bias frames, the requirements of the cross-instrument pipeline pro ject are matched and no change is required. For the infrared detectors of ISAAC, NACO and CIRIRES, and for the optical detectors of FORS1, FORS2 and VIMOS the existing linearity templates generate a sequence of only single flats per integration time, which is found not to be appropriate to fulfill the requirements set up by the monthly detector monitoring strategy. Thus a structural upgrade of the corresponding template has started. For reasons of data integrity a new template involving distinct classification keys to allow a smooth and controllable transition of VLT operations from the old to the new template has been generated. The detector monitoring pro ject scope is limited to the science detectors only. Technical detectors used e.g. for guide probes and wavefront sensing are not covered. The science detectors of the VLT fall in two main groups: optical CCDs and near and mid infrared arrays. Although the underlying technology and terminology is different, optical CCDs and IR arrays share the same detector characteristics: dark current, non-linearity, saturation, gain.


3. CPL AND PIPELINE RECIPES AND QUALITY CHARACTERISTICS
The ESO Common Pipeline Library (CPL, www.eso.org/cpl) has been developed by ESO in order to unify the pipeline production effort and to minimize code duplication. The ESO pipelines are executed at the observatory for quick-look assessment of the data, and at Garching to check the quality of the observations and to reduce the observation data according to a predetermined calibration cascade. The pipeline recipes are also made public to the user community, to allow a personalized processing of data from the instruments. CPL is used for the development of all ESO pipelines, both for internal developments and for developments by the consortia. It is an ISO-C library, designed to work with FITS data, and provides a collection of image processing ob jects and standard algorithms. The ESO Common Pipeline Library is built on four ma jor components, reflecting the algorithmic level of the software contained therein. These components are: · The cpl core library provides the fundamental CPL data types (such as cpl image, cpl table, cpl vector, cpl matrix etc.), the operations defined on these data types, and elementary utility functions. · The cpl ui (UI for User Interface) provides services defining the standard interface for recipes and provides more complex data reduction related utilities and services. In particular, it contains the necessary types to handle the plugin interface, the parameters used to handle command-line options and the (set of ) frames used to store the input list of files. · The cpl drs (DRS for Data Reduction System) uses the CPLCORE data types and functions to implement higher level data processing algorithms. CPLDRS is itself organized in sub-modules corresponding to different stages of data reduction: detector calibration, geometrical calibration, aperture definition, and photometry. · Finally, the most recent addition to CPL, the cpl dfs module (DFS for Data Flow System) provides DFS related utilities for the creation of DFS compliant data products, or quality control logs. The routines of the detector monitor pro ject will be implemented as part of the detector correction module in the cpl drs layer (Fig. 1). All pipelines will be able to invoke the same function, e.g. the detector linearity algorithm. Each pipeline will call these functions from the instrument specific recipes, thus providing the flexibility required to cope with instrument-specific features, keywords, or data structures. As a first step the algorithms have been implemented in a dedicated pipeline and, after testing and verification of the individual algorithms, have been deployed into the instrument specific pipeline recipes. The operational science detectors of the VLT produce frames in different formats. Three kinds of interdetector differences were found and are controlled by configurable pipeline recipe options (data reduction): · Some detector heads host more than one detector. The control software writes each detector image in a dedicated extension of the single fits frame (e.g. UVES, CRIRES). Other instruments like VIMOS and FORS produce for each of the CCDs a separate frame. · An option to confine the analysis to a defined detector's area has been implemented to support proper data reduction of some of the operational detectors which show vignetting within the optical path (Fig. 2) of the instrument (e.g. FORS1, NACO). · The operational templates of some detectors, in particular those of optical CCDs, generate flat frames with very high flux levels slightly exceeding the saturation limit without persistence effects. For the pipeline recipe it is important that the count range used for the analysis is always the same and is independent of the count range provided by the template. For this reason a configurable recipe parameter defines the upper counts threshold level. · The interpixel capacitance correction has been found to be very sensitive to lamp flux variations between consecutive flats, even for low amplitude count variations below 1 percent. For this reason a simple but efficient scaling option for the two flats to be subtracted from each other has been added.


Figure 1. Constructing pip eline recip es with CPL. The detector monitoring functions will b e part of the cpl drs modules. Table 1. Pip eline products and quality control parameters derived by the detector monitoring pip eline recip es.

QC parameter bias/dark level read-out out noise non-linearity lamp flux count range structural noise gain correction factor number of bad pixel map number of hot pixel and map number of cold pixel and map number of deviant pixel and map

units ADU ADU ADU/sec ADU per cent e-/ADU

schedule daily daily monthly monthly monthly monthly monthly monthly monthly daily daily daily

comments per read-out mode (and per DIT for arrays) per read-out mode (and per DIT for arrays) polynomial coefficients, and reference value minimum and maximum count level derived from the lamp flat series photon transfer curve for interpixel capacitance IR version optional in the IR version optional in the IR version optional in the IR version

3.1 NON-LINEARITY
The non-linearity algorithm as a first step simply subtracts CCDs this is a master bias, in case of IR arrays it is a master frames embedded in the same template. The median counts The polynomial is applied to each detector pixel, for the IR from each flat an offset frame. In case of optical dark. The master products are generated from raw and the exposure time are fit using a polynomial. arrays, and to the median value of all counts over


Figure 2. The detector recip e has to handle to several instrument sp ecific data issues: Upp er Left: Imaging flat field frame of the UVES red arm mosaic CCD. Both parts of the mosaic CCDs are stored in two separate extensions of a single FITS frame. The right CCD shows contamination effects in the corners. Upp er Right: SINFONI sp ectroscopic flat showing a strong p eriodic illumination pattern introduced by the image slicer. Preferable imaging flats cannot b e generated by the instrument. Lower: FORS1 lower CCD imaging flat; partially saturated. To avoid extended vignetted regions the data reduction needs to b e confined to a user-defined detector area. The template covers count ranges up to detector saturation, which must b e caught by the recip e via a user-defined signal upp er threshold option.


all pixels of each detector frame, for the optical detectors. The inverse relation, that is used to apply the nonlinearity correction e.g. in science frames, is obtained with a Newton-Raphson iteration. The derived quality control parameters include the polynomial coefficient and the relative correction at a configurable reference count level. The IR version of the recipe extracts also a map of all pixels deviating more than a configurable threshold value from the mean non-linearity relation. Although the procedure comes in two flavors (for optical CCD and IR arrays), the underlying algorithms are identical. This founding principle characterizes also all other recipes implementation.

3.2 GAIN
In a detector element each registered photon induces one electron, which thanks to an amplifier generates a number of counts. The gain [e - /ADU ] describes the conversion factor between photons and counts. The recipe implements the photon transfer curve method (PTC), which relates the statistical noise with the counts. The inverse of the slope of the linear fit of the PTC gives the gain (Fig. 5). The gain and its rms are monitored for quality control. The offset of the fit is nominally related to the flux-independent statistical noise (the read-out noise). However, as to determine the linearity and detector gain, it is necessary to scan a large intensity range, the read-out noise value determined with the PTC method is not very accurate as is obtained by extrapolation. The read-out noise is determined from dark and bias calibrations.

3.3 INTERPIXEL CAPACITANCE CORRELATION
The illumination of a single pixel, having neighboring pixels, on a multi-pixel detector is represented by a point spread function of characteristic size wider than the pixel size (Fig. 4). This effect can be explained by assuming the existence of a small but finite interpixel capacitance correlation. This smear-out effect results in effective lower noise and overestimates the measured gain value.2 The corresponding correction can be optionally determined by the implemented autocorrelation algorithm. The recipe computes the correction factor and the corrected gain value which are monitored as quality control parameters. On request the corresponding autocorrelation image, see Fig. 4, can be generated for further investigation.

3.4 CONTAMINATION
Optical detectors may suffer from contamination, which usually appears as a large scale, low-efficiency pattern, slowly increasing with time. The effect appears in normalized imaging flats (Fig. 2). The detector contamination is estimated by the flux ratio between target and reference detector areas. Up to four different target areas have been defined. Although being a very simply defined quality parameter, its monitoring allows for scheduled detector interventions.

3.5 READ-OUT NOISE
On optical CCDs, the read-out noise is derived from bias frames; on IR arrays it is derived from dark frames. The following three methods have been implemented: · RANDOM: The read-out noise is the rms of a random sample of detector areas. · HISTO: the recipe builds an histogram of the pixel flux levels, and fit this with a Gaussian. The read-out noise is the Gaussian standard deviation parameter. · REGION The read-out noise is a robust rms computed on a user defined detector region. · PREOVERSCAN: the read-out noise is the spatial noise measured over the pre-scan and over-scan region. The spatial noise of the pre and over scan regions will be used as reference values. If verified to be reliable, the method can be applied to science frames for an independent read-out noise estimate. The methods can be applied to a single raw frame or to a difference of two raw frames, the latter application being mandatory for dark frames with dominating structural noise.


Figure 3. Upp er Left: Central part of a SINFONI flat difference frame of two consecutively acquired sp ectroscopic flat frames, showing a residual structural noise pattern due to slightly different signal levels. Upp er Right: Scaled difference flat to comp ensate for slightly different signal levels. The residual structural noise is smaller than the statistical noise. Different levels of statistical noise from slice to slice are still apparent. The black circle is a bad pixel region. Lower Left: Autocorrelation function derived from the contaminated difference flat frame (Upp er Left). Lower Right: Autocorrelation function derived from the scaled difference flat frame. The background imp ortant for the normalization is well defined and the satellite p eak induced by slitlet pattern at a distance of 64 pixel has vanished. The profile is given in Fig. 4.


Figure 4. Upp er Left: Autocorrelation profile due to the interpixel capacitance effect of the ISAAC Hawaii detector. The four closest neighb or pixels show values of ab out 6e-4, resulting in a gain correction factor of 1.00576. Upp er Right: Corresp onding profile for the GIRAFFE CCD. The four closes pixels have correlation values of 1e-4, resulting in a correction of the gain by a factor of 1.0009. Lower Left: SINFONI detector. The scaling option has b een applied (Fig. 3). The profile shows a second low amplitude comp onent. The correction factor is 1.09. Lower Right: The CRIRES #1 detector, showing some p eriodic correlation noise pattern in the background due to imp erfect flat field reproducibility.


3.6 BIAS, DARK LEVEL
For the bias or dark level the average of the raw dark median is implemented.

4. TESTING THE RECIPES ON DIFFERENT OPERATIONAL DETECTORS
Three instruments SINFONI, UVES, and GIRAFFE have operational templates that acquire flat pairs since the start of operations. SINFONI and UVES are pipeline supported since the start operations, while GIRAFFE analyzes the monthly stacks via quality control scripts. The tests include the recipe functionality, with the different parameter values, and the verification of the physical results, by comparison with QC parameters obtained in the old recipes or by successive manual data reduction. The new implementation of the autocorrelation was straightforward and the application to optical CCD frames convincing (Fig. 4). The application of the autocorrelation analysis to the SINFONI spectroscopic flats became a real challenge. For the autocorrelation analysis, two flats of the same signal level are subtracted from each other to get rid of structural noise and leaving a frame with only statistical noise. In the case of SINFONI which can only produce spectroscopic flats of 32 individual pseudo slits, the structural noise does not completely vanish (Fig. 3), due to persistence effects generating different signal levels for consecutively acquired flats of the same exposure time. Short term lamp flux variations can give the same results. To compensate for this effect before subtraction, the recipe simply scale one of the flats to the signal level of the other: (FLAT1 - DARK1 ) M2 - (FLAT2 - DARK2 ) M1

where M1 and M2 are the median counts of both flats respectively. Fig. 3 shows that the structural noise is considerably reduced. Residual structural noise after flat field scaling, if correlated, contributes to the autocorrelation profile. In any case a further investigation from case to case is required, as this indicates short-term instabilities of the optical path or of the detector. For ISAAC, NACO, FORS1, FORS2, for which the existing linearity template does only generate single flats per signal level, the frames of two close in time submitted templates have been interleaved in order to simulate a frame stack generated by the planned new template. This is to check the whole functionality and progress the template. A further side effect is that we can measure detector characteristics, which were measured either once in the laboratory or only during the commissioning of the instrument before the start of operations or at a detector intervention. As an example, we found a non-linearity in the photon transfer curve in one of the read-out modes of the NACO detector (Fig. 5), an effect known from optical CCDs.3 This finding triggered the inclusion of an upper signal threshold option. The upper signal threshold option is also used to reject frames from the analysis with signal level close to or exceeding the saturation level. The large area vignetting apparent for the mosaic CCD FORSes and for NACO led to the definition of parameters to allow to confine the analysis to a given detector area (see Fig. 2). The analysis of the ISAAC Hawaii detector, based on non-consecutive flat pairs with considerable changes of the structural noise due to imperfect stability of the camera allowed to better understand in the sensitivity of the autocorrelation algorithm. For the two instruments VIMOS and CRIRES, no linearity template exists and test frame stacks have been composed by imaging flats obtained with different signal levels and filters in order to at least test the functionality including the fits frame extension handling in the IR array version. The recipes are currently being tested for templates of the IR science detectors of AMBER and the mid IR detectors of MIDI, VISIR.


photon transfer curve: NACO 500 400 300 200 100 0 0 5000 10000 signal / ADU 15000 measured f(x) = x / 35.5 + b1 f(x) = x / 27.6 + b2 10000 8000 6000 4000 2000 0

photon transfer curve: ISAAC Hawaii measured f(x) = x / 5.0 + b

variance

variance

0

15000 30000 45000 signal / ADU

60000

photon transfer curve: GIRAFFE 40000 35000 30000 25000 variance 20000 15000 10000 5000 0 -5000 0 30000 60000 signal / ADU 90000 variance 40000 30000 20000 10000 0 measured f(x) = x / 2.71 + b 70000 60000 50000

photon transfer curve: UVES-RED CCD CCD f(x) = f(x) = 20 44 x / 1.57 + b1 x / 1.78 + b2

0

25000 50000 75000 100000 signal / ADU

Figure 5. Examples of photon transfer curves of four op erational science detectors. The gain is the inverse of the slop e. Upp er Left: Non-linear PTC of the NACO detector in Double RdRstRd HighSensitivity mode. The frames of two single flat stacks have b een interleaved to test the new recip e with flat pairs. For the second fit only values with signal < 8000 ADU are taken into account, demonstrating the flexibility of the signal upp er threshold option. Upp er Right: PTS of the ISAAC Hawaii array. Lower Left: PTC of GIRAFFE CCD. Lower Right PTCs of b oth UVES RED mosaic CCDs.


5. TEMPLATE AND RECIPE COMMISSIONING PLAN
The VLT/I is one of the very few observatories that archives a calibrations and science frames. This requires a careful design of complex VLT dataflow. For e.g. a new value of the FITs header data interface control board to classify the raw frames generated issues have been negotiated within the involved departments. ll and processes a considerable fraction of all all relevant keys and formats used within the keyword DPR.TYPE was requested to the ESO by the new template. Many data flow related

The commissioning of the detector monitoring recipes is currently taking place in different phases. These different phases are required because nearly all instruments are affected, and the impact on operations should be reduced to the minimum as far as possible: · Due to the modular structure of CPL4 it is more flexible to deploy the algorithms into the individual instrument pipeline recipes instead of providing a general detector related pipeline package. In the final stage each instrument pipeline will consists of the monthly and daily detector recipe, as an interface to newly added libraries. · During this phase the new templates are implemented at the observatory and will generate the requested frame stacks. The new templates will produce frames while the old template are still operational for a limited period. · The deployed recipes are currently being refined for SINFONI, UVES and GIRAFFE, while those of other optical and IR instruments will be iterated as soon as the new frame stacks are available. This task will mainly be performed by the instrument QC scientists. · After the certification of the deployment, the final decision when to change operations is left to the individual instrument operation teams. The pro ject completion is expected for the end of 2008.

ACKNOWLEDGMENTS
The software support for this pro ject is part of the Spanish in-kind contribution to ESO.

REFERENCES
[1 ] P. Amico, P. Ballester, W. Hummel, G. LoCurto, L. Lundin, A. Modigliani, P. Sinclaire, L. Vanzi, "The detector-monitoring pro ject", ESO Instrument Calibration Workshop, ESO Astrophysics Symposia, 2008, in press [2 ] G. Finger, J. Belletic, R. Dorn, M. Meyer, L. Mehrgarn, A.F.M. Moorwood, J. Stegmeier, "Conversion gain and interpixel capacitance of CMOS hybrid focal plane arrays", in Scientific Detectors for Astronomy, p477, 2005. [3 ] M. Downing; D. Baade; P. Sinclaire; S. Deiries; F. Christen "CCD riddle: a) signal vs time: linear; b) signal vs variance: non-linear", in "High Energy, Optical, and Infrared Detectors for Astronomy II", Proc. SPIE 6276, (2006) [4 ] P. Ballester, K. Banse, S. Castro, R. Hanuschik, et al. "Data reduction pipelines for the Very Large telescope", Proc. SPIE 6270, 1 (2006)