Документ взят из кэша поисковой машины. Адрес оригинального документа : http://www.vista.ac.uk/Files/docs/vdfs/VIS-SPE-IOA-20000-0010_v1.6_DRLD.pdf Дата изменения: Wed Apr 20 17:04:18 2011 Дата индексирования: Mon Oct 1 19:39:35 2012 Кодировка: Поисковые слова: п п п п п п п п

Data Flow System
Document Title: Document Number: Issue: Date: VISTA Data Reduction Library Design VIS-SPE-IOA-20000-0010 1.6 2006-12-20

Document Prepared by: Document Approved by: Document Reviewed by: Document Released by:

Jim Lewis Peter Bunclark Simon Hodgkin (CASU) Mike Irwin (CASU Manager) Will Sutherland (Project Scientist) Malcolm Stewart (VDFS Manager)

Signature and Date:

Signature and Date:

Signature and Date:

Signature and Date:


VISTA Data Flow System

Data Reduction Doc: Library Design Issue:
Date: Page:

VIS-SPE-IOA-20000-0010 1.6pre6 2006-12-12 2 of 145

Change Record
Issue 1.0 1.1 1.2 1.3 1.4 1.5 1.6 Date 2004-12-17 2005-05-03 2005-08-12 2005-12-25 2006-06-15 2006-09-28 2006-12-20 Sections Affected All All 4 All All 0,5,6,7 2,6,8 Reason/Initiation/Documents/Remarks New Document post-FDR revision post DICB comments Consistency with DRL v0.1, JRL, MJI, PSB updates Many changes to all sections to help bring the document into line with DRL 0.3 Explanations of dummy products expanded. Expanded list of fatal and non-fatal errors for recipes. Many other smaller changes Added vircam_destripe + description Added parameters to jitter_microstep_process. Many other smaller changes, including touchups to table 10-1 update QC & DRS dictionaries Update vircam_standard_process, modify vircam_jitter_microstep_process, and modify entries in table 10-1. Modified vircam_illum entry in chapter 6. Modified vircam_mesostep_analyse and vircam_defringe Spell check and added QC STRIPERMS

Notification List
The following people should be notified by email that a new issue of this document is available. IoA RAL QMC Will Sutherland Gavin Dalton Jim Emerson


VISTA Data Flow System Contents

Data Reduction Doc: Library Design Issue:
Date: Page:

VIS-SPE-IOA-20000-0010 1.6pre6 2006-12-12 3 of 145

Change Record ............................................................................................................... 2 Notification List ............................................................................................................. 2 Figures............................................................................................................................ 6 Tables ............................................................................................................................. 6 1 Introduction ............................................................................................................ 7 1.1 Scope .............................................................................................................. 7 1.2 Applicable Documents ................................................................................... 7 1.3 Reference Documents .................................................................................... 7 1.4 Abbreviations and Acronyms ........................................................................ 8 1.5 Glossary ......................................................................................................... 8 2 Mathematical Description .................................................................................... 10 2.1 Reset Correction........................................................................................... 10 2.2 Non-Linearity ............................................................................................... 10 2.3 Gain Correction ............................................................................................ 13 2.4 Measurement of Read Noise and Gain ........................................................ 13 2.5 Dark-correction, flat-fielding and sky-correction ........................................ 14 2.6 Stripe Removal............................................................................................. 15 2.7 Fringe Removal ............................................................................................ 15 2.8 Image persistence ......................................................................................... 16 2.9 Crosstalk ...................................................................................................... 16 2.10 Astrometric Calibration ............................................................................... 17 2.11 World Coordinate System ............................................................................ 18 2.12 Effect of Scale Change on Photometry ........................................................ 18 2.13 Confidence Maps ......................................................................................... 18 2.14 Catalogue generation ................................................................................... 20 2.15 Photometric Zeropoint ................................................................................. 22 2.16 Illumination Correction ................................................................................ 23 3 Functional Description ......................................................................................... 25 3.1 Recipes ......................................................................................................... 26 4 Instrument Data Description ................................................................................ 37 5 DRL Data Structures ............................................................................................ 43 5.1 Introduction to Data Products ...................................................................... 43 5.2 Channel Table .............................................................................................. 45 5.3 Bad Pixel Mask ............................................................................................ 46 5.4 Confidence Maps ......................................................................................... 46 5.5 Dark Current Image ..................................................................................... 47 5.6 Crosstalk Matrix........................................................................................... 47 5.7 Illumination Correction Table...................................................................... 47 5.8 Difference/Ratio Images .............................................................................. 48 5.9 Difference/Ratio Image Statistics Tables..................................................... 48 5.10 Persistence Mask Table................................................................................ 48 5.11 Standards Table ............................................................................................ 49 5.12 Object Catalogues ........................................................................................ 49 5.13 Matched Standards Table ............................................................................. 51 5.14 Readnoise/Gain File ..................................................................................... 51


VISTA Data Flow System
6

Data Reduction Doc: Library Design Issue:
Date: Page:

VIS-SPE-IOA-20000-0010 1.6pre6 2006-12-12 4 of 145

5.15 Photometric Calibration Table ..................................................................... 52 DRL Functions ..................................................................................................... 54 6.1 vircam_crosstalk .......................................................................................... 55 6.2 vircam_darkcor ............................................................................................ 56 6.3 vircam_defringe ........................................................................................... 57 6.4 vircam_destripe ............................................................................................ 59 6.5 vircam_flatcor .............................................................................................. 60 6.6 vircam_genlincur ......................................................................................... 61 6.7 vircam_getstds ............................................................................................. 62 6.8 vircam_illum ................................................................................................ 64 6.9 vircam_imcombine ...................................................................................... 66 6.10 vircam_imcore ............................................................................................. 67 6.11 vircam_imdither ........................................................................................... 69 6.12 vircam_interleave ......................................................................................... 71 6.13 vircam_lincor ............................................................................................... 73 6.14 vircam_matchstds ........................................................................................ 74 6.15 vircam_matchxy ........................................................................................... 76 6.16 vircam_mkconf ............................................................................................ 77 6.17 vircam_persist .............................................................................................. 78 6.18 vircam_photcal ............................................................................................. 80 6.19 vircam_platesol ............................................................................................ 82 7 Data Reduction CPL Plugins ............................................................................... 85 7.1 vircam_reset_combine ................................................................................. 85 7.2 vircam_dark_combine.................................................................................. 87 7.3 vircam_dark_current .................................................................................... 89 7.4 vircam_dome_flat_combine ........................................................................ 90 7.5 vircam_detector_noise ................................................................................. 92 7.6 vircam_linearity_analyse ............................................................................. 93 7.7 vircam_twilight_flat_combine ..................................................................... 94 7.8 vircam_mesostep_analyse............................................................................ 97 7.9 vircam_persistence_analyse ......................................................................... 98 7.10 vircam_crosstalk_analyse ............................................................................ 99 7.11 vircam_jitter_microstep_process ............................................................... 100 7.12 vircam_standard_process ........................................................................... 103 8 Validation tests................................................................................................... 107 8.1 vircam_darkcor .......................................................................................... 107 8.2 vircam_destripe .......................................................................................... 108 8.3 vircam_flatcor ............................................................................................ 108 8.4 vircam_getstds ........................................................................................... 108 8.5 vircam_imcombine .................................................................................... 108 8.6 vircam_imcore ........................................................................................... 109 8.7 vircam_imdither ......................................................................................... 110 8.8 vircam_interleave ....................................................................................... 110 8.9 vircam_lincor ............................................................................................. 110 8.10 vircam_matchstds ...................................................................................... 111 8.11 vircam_matchxy ......................................................................................... 111 8.12 vircam_mkconf .......................................................................................... 111 8.13 vircam_platesol .......................................................................................... 111


VISTA Data Flow System
9 10 11 12

Data Reduction Doc: Library Design Issue:
Date: Page:

VIS-SPE-IOA-20000-0010 1.6pre6 2006-12-12 5 of 145 112 116 117 128 135

8.14 Validation Test Data .................................................................................. Development Plan .............................................................................................. Appendix: QC1 Parameters ............................................................................... Appendix: DRS Dictionary ................................................................................ Appendix: Raw FITS Header.............................................................................


VISTA Data Flow System Figures

Data Reduction Doc: Library Design Issue:
Date: Page:

VIS-SPE-IOA-20000-0010 1.6pre6 2006-12-12 6 of 145

Figure 3-1 Relationship between recipes, calibration data and data products. ............ 25 Figure 3-2 Bootstrapping table relating Calibration Observations, Recipes and Calibration Products............................................................................................. 26 Figure 3-3 vircam_reset_combine ............................................................................... 27 Figure 3-4 vircam_dark_combine ................................................................................ 27 Figure 3-5 vircam_badpix_mask ................................................................................. 28 Figure 3-6 vircam_dome_flat_combine ....................................................................... 28 Figure 3-7 vircam_detector_noise ............................................................................... 29 Figure 3-8 vircam_linearity_analyse ........................................................................... 29 Figure 3-9 vircam_twilight_combine .......................................................................... 30 Figure 3-10 vircam_illumination_analyse ................................................................... 31 Figure 3-11 vircam_mesotep_analyse ......................................................................... 32 Figure 3-12 vircam_persistence_analyse ..................................................................... 33 Figure 3-13 vircam_crosstalk_analyse ........................................................................ 34 Figure 3-14 vircam_sky_flat_combine ........................................................................ 35 Figure 3-15 vircam_jitter_microstep_process ............................................................. 36 Figure 4-1 A VIRCAM engineering readout shown displayed in the ESO-VLT RealTime Display Tool ............................................................................................... 37 Figure 4-2 Synthetic VISTA data shown organised by GASGANO ........................... 39

Tables
Table Table Table Table Table Table Table 4-1 Data Processing Table ................................................................................. 42 5-1 DO and PRO categories for data products .................................................. 45 8-1 Description of test data files ..................................................................... 113 8-2 Files to be used to test each vircam function ............................................. 114 8-3 Files to use in testing each vircam plugin ................................................. 115 9-1 Development Schedule .............................................................................. 116 10-1 The origin of QC Parameters ................................................................... 127


VISTA Data Flow System 1

Data Reduction Doc: Library Design Issue:
Date: Page:

VIS-SPE-IOA-20000-0010 1.6pre6 2006-12-12 7 of 145

Introduction

This document forms part of the package of documents for the design of the Data Flow System for VISTA, the Visible and Infra-Red Survey Telescope for Astronomy.

1.1 Scope
This document describes the VISTA Infra-Red Camera Data Reduction Library Design for the output from the 16 Raytheon VIRGO IR detectors in the Infra RedCamera for VISTA (VIRCAM). The baseline requirements for calibration are included in the VISTA Infra-Red Camera Data Flow System User Requirements [AD2], and the Calibration Plan is described in [AD3].

1.2 Applicable Documents
[AD1] Data Flow for the VLT/VLTI Instruments Deliverables Specification, VLTSPE-ESO-19000-1618, issue 2.0, 2004-05-22. [AD2] VISTA Infra Red Camera DFS Impact, VIS-SPE-IOA-20000-00001, issue 1.3, 2005-12-25. [AD3] VISTA Infra Red Camera DFS Calibration Plan, VIS-SPE-IOA-20000-00002, issue 1.3, 2005-12-25. [AD4] VISTA Infra Red Camera DFS Data Reduction Library Specification, VISSPE-IOA-20000-00003, issue 1.0, 2005-02-08. [AD5] Data Interface Control Document, GEN-SPE-ESO-19940-0794, issue 3, 2005-02-01. [AD6] Common Pipeline Library User Manual, VLT-MAN-ESO-19500-2720, issue 2.0.1, 2005-04-14 [AD7] Common Pipeline Library Reference Manual, VLT-MAN-ESO-19500-2721, issue 2.0, 2005-04-08

1.3 Reference Documents
[RD 1] VISTA IR Camera Software Functional Specification, VIS-DES-ATC-0608100001, issue 2.0, 2003-11-12. [RD 2] IR Camera Observation Software Design Description, VIS-DES-ATC-060840001, issue 3.2 2005-02-24. [RD 3] VISTA Science Requirements Document, VIS-SPE-VSC-00000-0001, issue 2.0, 2000-10-26 [RD 4] Overview of VISTA IR Camera Data Interface Dictionaries, VIS-SPE-IOA20000-0004, 0.1, 2003-11-13 [RD 5] Definition of the Flexible Image Transport System (FITS), NOST 100-2.0 [RD 6] The FITS image extension, Ponz et al, Astron. Astrophys. Suppl. Ser. 105, 5355, 1994 [RD 7] Representations of world coordinates in FITS, Griesen, & Calabretta, A&A, 395, 1061.2002 [RD 8] Representations of celestial coordinates in FITS, Calabretta & Griesen, A&A, 395, 1077, 2002


VISTA Data Flow System

Data Reduction Doc: Library Design Issue:
Date: Page:

VIS-SPE-IOA-20000-0010 1.6pre6 2006-12-12 8 of 145

[RD 9] Detectors and Data Analysis Techniques for Wide Field Optical Imaging, Irwin M.J., 1996, Instrumentation for Large Telescopes, VII Canary Islands Winter School, eds. J.M. RodrМguez Espinosa, A. Herrero, F. SАnchez, p35. Also available from http://www.ast.cam.ac.uk/~mike/processing.ps.gz [RD 10] INT WFS pipeline processing, Irwin M and Lewis J, New Astronomy Reviews, 45, Issue 1-2, p105, 2001. [RD 11] VISTA Data Flow System: pipeline processing for WFCAM and VISTA, Irwin et al, Ground-based telescopes, ed. Oschmann, proc SPIE, 5493, p411, 2004 [RD 12] Automatic analysis of crowded fields, Irwin M. 1985 MNRAS 214,575 [RD 13] Understanding Robust and Exploratory Data Analysis, Hoaglin, Mosteller & Tukey 1983, Wiley. [RD 14] Analysis of astronomical images using moments, Stobie R, British Interplanetary Journal (Image Processing), 33, p323, 1980

1.4 Abbreviations and Acronyms
2MASS ADU CDS DFS DIT FITS FWHM HOWFS LUT MAD MEF NDR RHS RRR VDFS VIRCAM VISTA WCS WFCAM 2 Micron All Sky Survey Analogue to Digital Unit Correlated Double Sampling Data Flow System Digital Integration Flexible Image Transport System Full Width at Half Maximum High-Order Wavefront Sensor Look Up Table Median Absolute Deviation from median Multi-Extension FITS Non-Destructive Read Right Hand Side Reset-Read-Read mode VISTA Data Flow System VISTA Infra Red Camera Visible and Infrared Survey Telescope for Astronomy World Coordinate System Wide Field Camera (on UKIRT)

1.5 Glossary
CDS Correlated-Double Sampling; before the charge of each pixel is transferred to the output node of the detector, the output node is reset to a reference value. The pixel charge is then transferred to the output node. The final value of the charge assigned to this pixel is the difference between the reference value and the transferred charge. An integer array, normalised to a median of 100%, which is associated with an image. Combined with an estimate of the

Confidence Map


VISTA Data Flow System

Data Reduction Doc: Library Design Issue:
Date: Page:

VIS-SPE-IOA-20000-0010 1.6pre6 2006-12-12 9 of 145

DAS DIT Exposure Integration Jitter (pattern)

Mesostep Microstep (pattern)

OB Object Pawprint

Preset Robust Estimate Tile

sky background variance of the image, it assigns a relative weight to each pixel in the image and automatically factors in an exposure map. Bad pixels are assigned a value of 0. It is especially important in image filtering, mosaicing and stacking. Data Acquisition System Digital Integrations mean that separate readouts are summed digitally. The stored product of many individual integrations that have been co-added in the DAS. The sum of the integration times is the exposure time. A simple snapshot, within the DAS, of a specified elapsed time. This elapsed time is known as the integration time. A pattern of exposures at positions each shifted by a small movement (<30 arcsec) from the reference position. Unlike a microstep the non-integral part of the shifts is any fractional number of pixels. Each position of a jitter pattern can contain a microstep pattern. A sequence of exposures designed to completely sample across the face of the detectors in medium-sized steps, in order to monitor residual systematics in the photometry. A pattern of exposures at positions each shifted by a very small movement (<3 arcsec) from the reference position. Unlike a jitter the non-integral part of the shifts are exact fractions of a pixel, which allows the pixels in the series to be interlaced in an effort to increase resolution. A microstep pattern can be contained within each position of a jitter pattern. Observation Block In the context of image analysis, an astronomical object. 16 non-contiguous images of the sky produced by VIRCAM with its 16 non-contiguous chips (see Fig 2-2 of [AD3]). The name is from the similarity to the prints made by the padded paw of an animal (the terminology was more appropriate to 4-chip cameras). A telescope slew to a new position requiring a reconfiguration of various telescope systems. A statistical estimator that is resilient to small perturbations on the assumed shape of the underlying distribution. A filled area of sky fully sampled (filling in the gaps in a pawprint) by combining multiple pawprints. Because of the detector spacing the minimum number of pointed observations (with fixed offsets) required for reasonably uniform coverage is 6, which would expose each piece of sky, away from the edges of the tile, to at least 2 camera pixels. The pipeline does not combine pawprints into tiles.


VISTA Data Flow System 2

Data Reduction Doc: Library Design Issue:
Date: Page:

VIS-SPE-IOA-20000-0010 1.6pre6 2006-12-12 10 of 145

Mathematical Description

In this section we include a mathematic description of some of the methods we will use to calibrate and correct data from VIRCAM. The main technical challenges in processing VISTA data stem from the fact that: IR detectors are currently inherently more unstable than their optical counterparts; the sky emission, roughly 100 times brighter than most objects of interest, varies in a complex spatial and temporal manner; and the large data volume that arises from NIR mosaic cameras. To minimise the subsequent data volume several basic pre-processing steps will be carried out in the VISTA data-acquisition system, including reset-correction and co-addition of successive DITs from the same exposure. The first stage of the VDFS pipeline will be to apply a linearity correction as outlined in section 2.2. Subsequent processing steps including: dark and reset-anomaly correction; flat-fielding and inter-channel gain correction; and sky artefact removal (e.g. fringe patterns), are designed to remove the instrumental and residual sky signatures from the images. The algorithms used in the VIRCAM pipeline are the result of 25 years development in the analysis of digital images. An excellent and detailed review of the mathematical techniques involved in wide-field image analysis is given in [RD 9]. In particular, the robust estimator is detailed and an in-depth description of image detection and parameterization, as used in section 2.14, is given. Several of the effects included in this section may not even exist in VIRCAM data; it is prudent however to make arrangements for dealing with such issues if early experience with the data shows the effects to be present. We outline in the following sections the salient points of the mathematical operations to be performed, for further detail see [AD2], [AD3] and [AD4].

2.1 Reset Correction
As with most electronic detectors infrared detectors are given a pedestal bias level by the driving electronics. As such the first step in any reduction of such data is to remove that bias. For VIRCAM this will be done in the DAS. This removes the need for explicit bias removal in the pipeline.

2.2 Non-Linearity
The Calibration Plan [AD3] lays out the necessity and the methodology for calibrating and correcting for the expected non-linearity in the response of the detector system to incident radiation. 2.2.1 Correcting for non-linearity In default CDS reset-read-read (RRR) mode, downstream of the data acquisition system (DAS) the output that we see is
I = I 2 - I 1 = f ( I 2 ) - f ( I 1 )
(2-1)


VISTA Data Flow System

Data Reduction Doc: Library Design Issue:
Date: Page:

VIS-SPE-IOA-20000-0010 1.6pre6 2006-12-12 11 of 145

where I1 and I 2 denote the non-linear first (i.e. the reset-frame) and second readouts respectively and I1 and I 2 the desired linear quantities. The non-linear function f(I) maps the distortion of the desired linear counts to the non-linear system I . If we define the inverse transform g( I ) that maps measured counts I to linearized counts I as the inverse operator g() = f -1() then I = g ( I ) and I 1 = g ( I 1 ) I 2 = g ( I 2 )
(2-2)

If I1 and I 2 were directly available this is a one-to-one mapping and can be done efficiently and accurately using Look Up Tables (LUT). This is the conventional way of implementing the correction prior to other image manipulation operations. However, if I1 and I 2 are not separately available and all we have to work from is the difference I then a simple LUT transformation is not possible.

For example, taking the simplest case where the illumination level across the detector has not changed during the course of the RRR and no on-board co-addition is happening then, in principle given only I and knowledge of the timing of the RRR operations we can deduce I1 and I 2 by using the effective integration time for each to estimate their scaling to the measured difference I such that,
I 1 = kI and I 2 = (1 + k )I
(2-3)

Unfortunately, the ratio k will not be constant for the non-linear quantities I1 and I 2 forcing us to adopt a scheme along the following lines.

Given I and defining the non-linear operator f () as a polynomial with coefficients am (typically up to quartic) we have
I =


m

am ( I

m 2

- I1 ) =

m


m

a m [(1 + k ) m I m - k m I m ]

(2-4)

The quantity we want I is buried in the non-linearity of the RHS and we have to solve an equation like this for every pixel. This is possible, and relatively simple to program using something like a Gauss-Seidel iterative scheme, but is more inefficient than a direct mapping. If we wanted to use a completely general LUT approach we would require a 2D LUT for all possible values of I1 and I 2 i.e. 65k в 65k in size, or 4.3 в 2 Gbytes. Most likely we would need a different correction for each "channel" making a total of 256 в 8.6 Gbytes = 2.2 Tbytes of LUT for the VIRCAM! Of course if the range of values of k is limited via exposure time quantisation this decreases the size of the total number


VISTA Data Flow System

Data Reduction Doc: Library Design Issue:
Date: Page:

VIS-SPE-IOA-20000-0010 1.6pre6 2006-12-12 12 of 145

of LUTs required considerably for the constant illumination case, but would be an ugly and possibly impractical solution. Practical considerations (e.g. data volume), suggest two alternative solutions for nonlinearity correction: either correct the individual frames directly in the DAS by measuring and downloading the appropriate LUTs, or polynomial coefficients, to the DAS; or use a non-linear inversion on the reset-corrected frames as outlined here. This methodology is not generally applicable, e.g. to multi-NDR/gradient fitting readouts, but is directly applicable to co-added (or co-averaged) frames of the same exposure times, assuming constant illumination over the series. For reset-corrected data, the non-linear inversion is competitive with complex operations on LUTs and much simpler to implement. It also has the added advantage of removing all aspects of the non-linearity correction from the DAS. The main disadvantages are the method is restricted to CDS RRR mode, and if the illumination level is rapidly varying (e.g. twilight) the effective scale factors ki may be hard to compute accurately - although for all realistic practical situations the knock-on effect is likely to be negligible.
2.2.2 Measuring non-linearity If all that is available are reset-corrected data from say a time series of dome flats, it is still feasible to directly compute the non-linearity coefficients.

Given a series of measurements {i} of I i and using the previous notation and polynomial model
I i =


m

am ( I

m 2

- I1 ) =

m


m

a m I i [(1 + k i ) m - k i ]

m

m

(2-5)

where k i are the exposure ratios under the constant illumination. In general I i = st i where ti is the exposure time of the ith reset-corrected frame in the series and s is a fixed (for the series) unknown scale factor. The k i are computable from a knowledge of the exposure times and the reset-read overhead, ti and I i are measured quantities leaving the polynomial coefficients am and the scaling s to be determined. Thus the model is defined by

I i =


m

am ( I

m 2

- I1 ) -

m


m

a m s m t i [(1 + k i ) m - k i ]

m

m

(2-6)

and can be readily solved by standard linear least-squares methods using the following sleight-of-hand. Since the scaling s and hence the polynomial so