Event Filtering

Contents of This Section

• Event Filtering
• Examples
• Header Parameters
• Math Functions
• Region Filtering
• Advanced Filtering

Event Filtering

We have developed a very promising technique for filtering column data such as that which is contained in FITS Binary tables and binary event files. Events can be selected based on a flexible set of user-specified criteria that allows column values to be compared with numeric values, header parameters, functions, or other column values.

To do this, we use a technique in which a filter specification is converted into a tiny C program, which is compiled and linked automatically so that events can be fed to this program and filter results returned to the calling program. The power of the technique lies in these considerations:

• The generated filter program is very small, containing less than 200 lines of code, so that it compiles and links in 2 seconds or less on most machines.
• The filter specification itself becomes a hard-wired part of this program, so that filter checking is performed as part of compiled code, not in the usual interpreted mode. This use of a compiled filter results in a speed-up factor of 4-5 over previous techniques, even after the program compilation time is added.
• All C syntax and C operators become valid parts of the filter syntax, making available a much wider range of filter possibilities than previously.
• It is easy to replace the standard filter program with a user-specified filter program for more sophisticated applications.

Filtering generally is performed on columns of event records in a FITS binary table or a raw event file. The following description applies to both types of data, unless otherwise noted. For illustrative purposes, we will consider data containing the following event structure:

• double TIME
• int X
• int Y
• short PI
• short PHA
• int DX
• int DY

Examples

Events can be filtered on these columns using IRAF/QPOE range syntax or any valid C syntax. The following examples illustrate the possibilities:

pha=10

select events whose pha value is exactly 10

pha=10:50

select events whose pha value is in the range of 10 to 50

pha=10:50,100

select events whose pha value is in the range of 10 to 50 or is equal to 100

pha>=10 && pha<=50

select events whose pha value is in the range of 10 to 50

pha < 10 || pha>50

select events whose pha value is less than 10 or greater than 50

pi=1,2&&pha>3

select events whose pha value is 1 or 2 and whose pi value is 3

pi=1,2 || pha>3

select events whose pha value is 1 or 2 or whose pi value is 3

pha==pi+1

select events whose pha value is 1 less than the pi value

pha==pi+1&&time>50000.0

select events whose pha value is 1 less than the pi value and whose time value is greater than 50000

(pi+pha)>20

select events in which the sum of the pi and pha values is greater than 20

pi%2==1

select events in which the pi value is odd

Header Parameters

The filter syntax supports comparison between a column value and a header parameter value of a FITS binary tables (event files have no such header). The header parameters can be taken from the binary table header or the primary header. For example, assuming there is a header value MEAN_PHA in one of these headers, you can select photons having exactly this value using:

• pha==MEAN_PHA

Math Functions

It also is possible to specify C math functions as part of the filter syntax. When the filter parser recognizes a function call, it automatically includes the math.h and links in the C math library. Thus, it is possible to filter events by crazy expressions such as:

• (pi+pha)>(2+log(pi)-pha)
• min(pi,pha)*14>x
• max(pi,pha)==(pi+1)

Region Filtering

SAOtng supports an extended version of IRAF/PROS spatial region filtering. We have implemented region filtering for the following region types:

• annulus(x,y,ri,ro) or annulus(X,Y,x,y,ri,ro)
• circle(x,y,r) or circle(X,Y,x,y,r)
• ellipse(x,y,a,b) or ellipse(X,Y,x,y,a,b)
• pie(x,y,r1,r2) or pie(X,Y,x,y,r1,r2)
• point(x,y) or point(X,Y,x,y)
• box(x,y,w,h) or box(X,Y,x,y,w,h)

Each of these region specifications comes in two forms: the first is the same as the standard IRAF/PROS region specification. It is applied to the columns that are being used to bin the data, i.e., to the columns pointed to by the binkey= switch or the FITS_BINKEY or EVENTS_BINKEY environment variables. The second form has two additional argument that specify which columns to use as the "X" and "Y" columns. For example, if the EVENT table above is binned on X and Y, you can specify a filter for all events within a circular region of the detector coordinates DX=400, DY=350 using:

• circle(DX,DY,400,350,20)

• circle(400,350,20)&&pie(400,350,0,45)

Note that we do not yet support rotation angles in ellipse or box regions, nor do we support polygon regions. These will be added in future versions. Or do it yourself ...

Advanced Filtering

The default filter program that is generated by a filter expression can be changed by users for more sophisticated applications. For example, you might want to run a special subroutine within the filter expression (e.g. a phase() subroutine that act on the time column).

To support such advanced functionality, the filter scheme recognizes two environment variables, FILTER_BODY and FILTER_HEADER, that determine the contents of the compiled filter program. You can set these variables to name C code files that will be used at the beginning or end of the generated C program. The default FILTER_HEADER and FILTER_BODY are listed here. You should refer to that code for the discussion that follows.

When a filter is being compiled, the FILTER_HEADER first is copied to the temporary C program. The default filter header currently only defines the min and max functions. We deliberately do not include C header files in order to minimize compilation time, but more sophisticated applications might need to do so.

Then, as the filter specification gets compiled, filter lines are automatically entered into the C program for each of columns being used in the filtered. Three types of entries are made:

• general information is added to the program about the size of the event record, the filter to be used, etc.
• directives are added to extract needed data columns value from the event stream.
• directives are added to allocate data storage for each column

#define FILTER (pi==pha)
#define ESIZE 20
#define GET_pi LoadColumn(eptr+6,2,1,convert,&pi)
#define GET_pha LoadColumn(eptr+4,2,1,convert,&pha)
#define GET GET_pi;GET_pha;
short pi;
short pha;

Once this filter-specific information is written to the temporary C file. the FILTER_BODY is copied to the temporary C program. The essence of this code is contained in the following lines:

  while( read(0, &get, 4) >0 ){
    ebuf = (char *)malloc(get);
    for(etop=ebuf; get>0; etop += got, get -= got){
      if( (got=read(0, etop, get)) <=0 )
	break;
    }
    for(rptr=ebuf, eptr=ebuf; eptr < etop; rptr++, eptr += ESIZE){
      GET;
      *rptr = FILTER;
    }
    got = ((etop - ebuf)/ESIZE);
    write(1, &got, 4);
    write(1, ebuf, got);
    free(ebuf);
  }

When executed, the first read() call retrieves a 4-byte integer from SAOtng that tells how many bytes of actual data are being sent to the program. The first for loop then reads all of these bytes from SAOtng into the event buffer ebuf. (This must be done in a loop because of limits on the amount of data sent through a pipe at one time). The second for loop runs over each event in the data stream: the GET macro expands to extract the relevant columns for each event from the byte stream and the FILTER macro then executes to determine if the event passes the filter. As described earlier, both of these macros were generated by the input filter expression. The result of the FILTER directive is a 1 or 0, depending on whether the event passes or fails the filter. A pass/fail value is stored in a char array for each event, and the whole array of pass/fail values is written back to SAOtng at completion. SAOtng then looks at each value in this char array to determine if the associated event passes or fails the filter.

The interaction between SAOtng and the compile program is relatively simple: SAOtng writes events to the filter program, which program in turn checks these events against the compiled filter and returns a simple flag that indicates pass/fail. Users can change the filter body to add routines, etc. so long as they ensure that the basic I/O structure of the program (reading n events and returning a char array of n flags) is maintained. (But note that new function names should begin with a letter or underscore, not a number.) Once a new filter body is written, its use is specified by setting the FILTER_BODY environment variable the pathname of the new file. Additional compile switches and libraries can be added by placing these in the FILTER_EXTRA environment variable. Finally, a specific compiler can be specified by means of the FILTER_CC variable. (The default is to use gcc or cc, if these can be found). When a filter is used, the new C module will be compiled and events will be passes to it for filtering.

Last Updated March 24, 1998