This guide is intended to provide a comprehensive overview of the processing and analysis of Chandra data obtained with either of the Chandra X-Ray Observatory's transmission grating instruments (HETG or LETG). It will provide a description of the tools, data, and processes in the contexts of pipeline processing, data-quality assessment, re-processing, and analysis. It will cover some of the special instrument modes, such as ACIS CC-mode, or blocked zero-order cases. It will also give examples of common problems and their solutions.

This guide is not a detailed reference manual. To that end, the help-files are available for each tool and provide an explicit list of all input and output files and control parameters. Manuals for other components of the system, set-by-step threads, and useful scripts are also available. These will be referenced here.

Intended Audience

This guide is intended to be a ``living'' document, and will be updated as frequently as required. It will hence be available on-line only, and provide links to detailed information. The most recent version of this document will be available at http://space.mit.edu/CXC/analysis/AGfCHRS.html.

If there is information you would like to find here, please contact me.

Models can also be found in the form of ``off-the-shelf'' responses, particularly as a Response Matrix File (RMF) and Auxiliary Response File (ARF) (see proposal planning files; grating RMFs are also in the CALDB).

Chandra Dictionary

If you have CIAO installed, here are some example help commands which summarize grating processing tools:

• ahelp hetg
• ahelp letg
• ahelp tg | grep '(tools)'
• ahelp grating
• ahelp gratings

The figure shows a schematic of the process, starting at ``Level 1'' (focal-plane instrument-specific and aspect transformations have been done).

The three panels correspond roughly represent the levels of processing. The pipeline does most of the first two (omitted are: destreak, mkgarf, asphist, and mkgrmf (mkgrmf is ciao new for 2.2)); the bottom panel is user analysis.

In the diagram, ovals represent processes and are labeled with tool names. The rounded-corner rectangles represent calibration data, and the square-cornered rectangles represent data-products. Not all inputs, outputs or control parameters are shown (see the detailed help files).

Observers receive a binned spectrum from the pipeline as a standard product. They may not need to use the event file, but may take the ancillary products (such as aspect solution, bad-pixel list), make responses, and proceed with spectral analysis. However, improvements in calibration, discovery of bugs in the system, or improved analysis techniques may require re-processing of the events. Or healthy scientific skepticism may generate interest in details of the event processing and files in order to verify the products and better understand the quality, limitations, and potential of the data. So these steps and data will be described, but the first look may skip past the events and go right into spectral analysis using the binned spectrum.

[At the time of this writing (October, 2001), the pipelines do not compute grating effective areas (ARFs). We are evaluating whether we can add this step to the pipelines.]

Event Processing

The spatial region is used to classify the ``Level 1'' events geometrically according to the part of the spectrum in which they fall. Using the aspect solution, the instantaneous transformation is done using the event's three-dimensional chip coordinate, projection of zero-order centroid from sky to chip, location of the grating node, grating facet mean characteristics, and camera location along the translation direction (SIM_Z). If the detector is ACIS, the energy resolution is used to sort orders, according to a spatially-dependent detector response calibration file. The attributes of each photon thus determined are appended to the event-list as new columns. These columns are (TG stands for ``Transmission Grating''):

TG_SRCID

Source identifier index. Up to ten sources can be resolved simultaneously. The pipeline only detects the brightest source.

TG_PART

The spatial part of the spectrum: 0 - zero-order; 1 - HEG; 2 - MEG; 3 - LEG; 99 - unresolved.

TG_R

Diffraction coordinate. This is a signed, real value, and is the photon's diffraction angle in degrees from the zero order centroid, in the direction parallel to the dispersion. The focal-length used is the Rowland spacing - the separation of the grating node and detector on an axial ray.

TG_D

The cross-dispersion angle, in degrees, also using the Rowland spacing as the focal-length. Note that that the grating plate scale is a slightly different from that of imaging-mode. Grating mode will maintain square pixels in tg_r,tg_d angular coordinates, but will not be the same as sky X,Y angles, by the factor of about 8637/10062 = 0.858

TG_MLAM

The order times wavelength, in Angstroms.

TG_M

If the detector has sufficient energy resolution (e.g., is ACIS), then the order can be resolved using the photon ENERGY coordinate (linearized, scaled PHA). This is a signed integer diffraction order. Unresolved photons are assigned to order 99.

TG_LAM

If the photon is resolved, then this is the wavelength, in Angstroms. TG_LAM is non-negative; unresolved photons are assigned wavelengths of 0.0.

TG_SMAP

This is a bitmap used to record whether a photon's source is ambiguous. If there are multiple sources in the field, then HETG orders will cross. It is possible that the CCD ENERGY will not resolve the photon uniquely. If not, a bit is set for each possible source. Note that only one diffraction coordinate is stored. It is up to user-modeling to either apply source knowledge to use these photons in the spectrum.

By default, tg_create_mask generates region sizes which are wide enough in the cross-dispersion direction to contain both source and local background. When binning (see about PHA files below), sub-selections are made with smaller cross-dispersion widths.

Details of the ASCII region format are here.

The FITS format is much more general, and more generally supported. For example, regions in this format can be used as Data-Model filters. An example columns and contents of a region FITS file are here.

The ASCII regions can be converted to FITS (and optionally appended to another file) with dmrega2fits.

Zero-order Region and Source Detection

Aspect Dither, Aspect Offsets, and Coordinate Transformations

In addition to controlled motion of the optical axis, there is uncontrolled (but measured) thermal flexure of the optical bench, resulting in science instrument module (SIM) motion. The six aspect parameters are stored in the aspect solution (asol) file, and in an aspect offsets (aoff) file. The latter differs in units, storing the optical axis coordinates as a difference from the mean value in detector pixels.

Grating event coordinates are computed by using the aspect solution to project the zero-order sky centroid onto the detector at the time of each event, then solving for the diffraction coordinates using the chip coordinates (in 3 dimensions) of the detected photon, the zero-order location, and grating geometry.

Source Table

The CCD gain is the calibration quantity which relates the detected signal (``PHA'') to nominal (or blurred) energy (ENERGY column in the event file). The gain depends upon the CCD, the CCD quadrant, and upon x,y location within the quadrant, and to some extent on energy. The gain also depends on epoch, mainly through operating temperature.

The blurring of the input photon energy to CCD detected ENERGY is stored in the Response Matrix Function, whose width, like the gain, also depends upon CCD and event location within the CCD.

Grating event order sorting is done by taking the ratio of the diffraction order*wavelength (uniquely determined from the diffraction angle) to the CCD ``wavelength''. If the value is within the CCD resolution of an integer value, then that integer value is assigned as the order.

The expected boundaries in CCD energy vs energy are pre-computed for each CCD, and for each position on each CCD. The tables are called ``osip'' files, for ``Order Sorting and Integrated Probability''. They are maps vs chip position of the CCD main peak's energy width vs energy. The widths are approximately 3*sigma of the Gaussian fit to the main peak of the CCD response. Since the CCD resolution changes substantially with CHIPY, the width of the 3*sigma region is asymmetric in plus and minus orders. We have tuned the widths slightly from 3*sigma to accomodate gain correction inaccuracies and order crowding. The software truncates any overlap at the halfway point (i.e., order 2 is always from ratios between 1.5 and 2.5).

It is possible to bypass the OSIP tables and to specify order-sorting limits which are constant with wavelength (see the order-sorting figures below), via the osort_hi and osort_lo parameters of tg_resolve_events.

Prior to definition of the OSIP tables, a position-independent order-sorting table was used. This had widths dependent on CCD_ID, since the CCD energy resolution can change abruptly between chips (e.g., Front Illuminated to Back Illuminated). This table is somewhat broader than the OSIP, and also allows a user-selectable ``fudge-factor'' on the width. This can be somewhat more forgiving for troublesome data (unstable gain, CC-mode). However, the effective area is not calibrated for arbitrary ``fudge-factors''; this should not matter increased widths, but may for smaller regions which truncate the PHA distribtions. The old-style file is termed ``IRMF'', for Integrated Response Matrix File (but is NOT a response matrix). The fudge factors are the energ_lo_adj and energ_hi_adj parameters to tg_resolve_events.

LETG/HRC-S and Overlapping Orders

Bad Pixels (standard):

These are hot or dead detector pixels, for which valid events cannot be determined. Bad pixel lists come as a calibration database file of permanently known bad pixels, and as an obervation-specific file of transient bad pixels (as for a temporarily corrupted CCD bias map). These are merged and applied during standard processing.

Grade (ACIS only; standard):

relates to geometry of PHA signals an event ``island'' (3x3 or 5x5 neighborhood). Valid grades are 0, 2, 3, 4, and 6. A different grade-set would require a different QE calibration. No difference in grade-filters are used for grating data.

Energy (ACIS only; non-standard):

Background photons (or piled zero-order) can have non-physical energies. The size of the event file can be reduced by keeping only events with ENERGY<10000, for example.

Status (standard):

Event files have a STATUS column, which is a bitmap. Each bit pertains to a different instrumental characteristic which can generate a bad event. If status is non-zero, the event gets filtered.

Streak (ACIS-only; non-standard):

CCD S4 (chip_id=8) has a spurious background signal which looks like horizontal (parallel to CHIPX) streaks in an image. These can create significant artifacts in a grating spectrum. The destreak program (two varieties: stand-alone C, or CIAO version) can be used to look for the correlations along a row in the same frame, and remove them or flag them. This should be the last filter applied.

``Bow-Tie'' (HRC-S only; standard):

LETG/HRC-S spectra have significant instrumental background. The width of the spectrum increases with wavelength, due to Rowland geometry astigmatism. The bow-tie is a spatial region filter applied before binning to follow the astigmatic width and reduce the area binned in the cross-dispersion region. The bow-tie also defines a background binning region which maintains a constant background to source region size.

PI-wavelength region (LETG/HRC-S only; non-standard):

HRC-S does not have enough spatial resolution to sort orders, but there is a non-uniform PI distribution which depends upon energy. Calibration analysis has shown that some non-source events can be discriminated via their PI values. There is a small loss of source photons, and there are several region filters with different rejection criteria. (One of these may become part of pipeline processing).

A picture of the bow-tie and PI region geometries are shown in the figures:

Afterglow (ACIS-only; standard):

Cosmic rays and other energetic particles can leave a large amount of charge in a pixel. The initial charge is rejected by it's grade (geometric distribution), but the decaying charge can masquerade as photon events. Acis_detect_afterglow is run routinely by the pipelines to detect these glowing pixels statistically, and a status bit is set for the ones found. This is not foolproof, however, since bright emission lines can have a high probability of being flagged; but then, they also have a high probability of having ``pileup''. (For the brightest lines in the brightest sources, e.g., Capella, the effect is about 3%.)

``Type I'' files have sequential channels stored in sequential rows, and the corresponding counts in another column. ``Type II'' is a transpose of this: the COUNTS and CHANNEL columns are array columns. For a single-spectrum file, there would be only one row.

``Type II'' is the default for Chandra grating spectra, since each observation is comprised of at least two orders (LETGS). For HETGS, we bin from -3 to +3 for two gratings by default, and thus have 12 counts histograms per observation. Instead of creating 12 files, we use one Type II format.

The CXC program used to bin Chandra spectra is tgextract.

Extensions to the Standard Formats

SPEC_NUM

is a column which gives a serial index to the row.

TG_M

is the diffraction order.

TG_PART

is an index to the type of grating (1 =>HEG; 2=>MEG; 3=>LEG).

TG_SRCID

is a source identifier, and is equal to the source number in the source table.

X, Y

are columns giving the sky centroid of the zero-order. This is used when computing the effective area, since the chip-boundaries in the spectrum depend upon where the source is placed on the detector.

BIN_LO, BIN_HI

These are by default wavelength coordinate grids. The standard format does not store the energy coordinate, since it is deferred to the response matrix. Since grating spectra are high-resolution (i.e., the response matrix is nearly diagnonal), the energy coordinate is well defined. We use wavelength, since that is a linear coordinate for transmission grating spectra. These are also array columns in Type II files. For back-compatibility, we store them in increasing energy order (but linearly in wavelength).

Other coordinates are possible through custom use of tgextract.

BACKGROUND_UP, BACKGROUND_DOWN

These are background counts arrays, binned in regions adjacent to the source region. (Default spectrum widths are given in the tgextract help file.) Background events undergo the same event-resolution process as source events. In fact, tg_resolve_events does not distinguish between background or source; that is only done when binning. There are two arrays because the geometry is not necessarily symmetric, especially for HETGS near the zero-order, or if there are confusing sources in the field.

It is up to the user to decide how (and whether) to combine and apply these background arrays. Background rejection is high with ACIS because of the order-sorting. Two non-background components of these pseudo-background arrays are the faint wings of the cross-dispersion line spread function (LSF), and charge aliasing during the ACIS frame shift. Since the charge still collects during the 41 msec frame shift, the charge which would have been in a few pixels is spread among them all. The fraction of charge spread is equal to the frame-shift-time divided by the frame time, wwhich for the default timed exposure, is 0.041/3.200 = 0.0128.

BACKSCAL, BACKSCUP, BACKSCDN:

These are the divisors by which to scale the background counts arrays to represent the expected background counts in each of the source, BACKGROUND_UP, and BACKGROUND_DOWN regions (I.e., the value should be greater than 1.0; default values are near 5.0, so that the combined background regions have ten times the width of the source region). If the values are constant with row (spectrum order and grating), then they are keywords in the header. If not, then they become columns in the SPECTRUM table.

Currently, the ratio of the background region width to the source region width is constant with wavelength, even for the ``bow-tie'' region. If this restriction is lifted, then the quantites are no longer scalar and will become array columns in the Type II PHA file, or regular columns in a Type I PHA file.

REGION

The binning region(s) are attached as an extension in the PHA file, as a block named ``REGION''. A region extension has the columns:

ColNo  Name      Unit        	      Type         Comment
   1   SPEC_NUM              	      Int2       Spectrum number
   2   ROWID                 	      String[64] Source or a background region.
   3   SHAPE                 	      String[16] Shape of region
   4   TG_LAM    angstrom    	      Real4      Dispersion coordinate vector for SHAPE
   5   TG_D      degrees     	      Real4      Cross-dispersion coordinate vector for SHAPE
   6   R[2]      (angstrom, degrees)  Real4(2)   Radius vector for SHAPE
   7   ROTANG    degrees     	      Real4      Rotation angle for SHAPE
   8   COMPONENT             	      Int2       Component number to which SHAPE belongs.
   9   INCLUDE               	      Int2       Inclusion (1;default) or exclusion (0)
  10   TG_SRCID              	      Int2       Source identification number
  11   TG_M                  	      Int2       Diffraction order

If there are both _UP and _DOWN background arrays, then there are three rows for each order. The SHAPE column is BOX for HETG, and could be either BOX for LETG or POLYGON for LETGS with the ``bow-tie'' filter. The box centers are given by the TG_LAM, TG_D columns, and the R column gives the full-width of the box.

Coarser gridding, if desired for lower signal data, may be obtained with parameters to tgextract, or by applying ``grouping'' during a fit.

If the detector is ACIS, then orders -3 to +3 (excluding 0) are binned into the standard PHA file.

If the detector is HRC, then the orders are called -1 or +1; it must be understood that these are the sum of overlapping orders, to be deconvolved (if necessary) through modeling.

ACIS, Blocked Zero Order

If ACIS is in CC-mode and zero order is blocked, there is no frame-shift streak. In this case, one may be able to use an initial guess, then refine the position by bisecting the detector silicon edge features in the spectrum, or by bisecting hyperbolas in ENERGY vs X,Y plots.

Neither of these work-arounds has been implemented automatically. These modes require intervention to create a valid region file (usually done with tgdetect and tg_create_mask), after which processing can proceed as usual with tg_resolve_events.

Pileup/Zero Order Centroid Error

If the centroid is erroneous, then the wavelength scale will be offset in an antisymmetric fashion in each grating. Offsets can be different in HEG and MEG, depending on the direction of the zero-order centroid error.

This error can be handled by either manually editing the zero-order centroid (in the src1a.fits file or in the region file), or by averaging the wavelengths of plus and minus order feature measurements.

Multiple Sources

Tg_create_mask will create masks for up to 10 sources. Tg_resolve_events will apply this mask and attempt to resolve orders and sources in spatially confused regions by the CCD pulse-height, which for some source configurations can result in unambiguous identification. The resulting event list has columns for the source ID, and a column which has bits set (a source map) to indicate other all possible sources, if the event is not resolved. Quantitative use of these ambiguous events is left to the user.

Extended Sources

Some help is available with current tools. For example, tg_create_mask can be run for one grating arm (HEG or MEG), which may be desired to omit collision of HEG and MEG near zero order. Then, the mask widths can be manually edited to make them very wide (or the width_factor_arms used to expand the region). Thus, tg_resolve_events will order-sort photons within that region, and they can be binned with tgextract for customized cross-dispersion regions.

Note, however, that the wavelengths are determined for a zero order point. Interpretation of wavelengths is ambiguous. Also, mkgarf computes the ARF for a point source. We currently have no provision for extended source grating ARFs.

The ARF is observation dependent, since it depends upon the zero-order position and dither pattern, which determine the mean position of wavelengths on the detector (whose QE(E) depends upon position) and the history of position with time. The RMF is weakly dependent upon the observation, particularly for HRC-S, whose 3-plate geometry deviates significantly from the ideal focal surface.

The grating ARFs can be made with the program, mkgarf. Since each detector element is independent and can have its own live-time, mkgarf works on one chip at a time. Some useful Grating Spectroscopy Scripts package multiple runs and the final merging.

Until CIAO 2.2, grating RMFs were on-the-shelf (in the calibration database). With CIAO 2.2, custom RMFs can be made with mkgrmf. The customization is primarily in the choice of grids and spectral regions, but this important to facilitate analyses which use non-default wavelength gridding. Mkgrmf also incorporates effects due to off-axis angle and cross-dispersion width on the LSF.

The rigorous definition of the responses can be found in Davis (2001).

Summary of Response Generating Software and Data

mkgarf

Make a grating ARF for a specified chip, order, and source position given an aspect histogram.

mkgrmf

(New tool for CIAO 2.2) Make a grating RMF given a grid specification. The grating RMF is the redistribution from input energy to output channel in the dispersion direction. It is nearly diagonal for any single order. It depends upon the cross-dispersion region. Default regions are wide enough to contain 98% of the signal. Narrower regions may be desired in crowded regions, or in attempt to improve the resolution by omitting a scattering halo. Mkgrmf applies this cross-dispersion factor to the resulting matrix.

dmarfadd

Add the per-chip ARF pieces into one ARF, applying weighting by EXPOSURE.

EXPOSURE:

the sum of the Good Time Intervals (GTI) times the live-time-factor (or 1-DTCOR, the Dead Time CORrection factor). DTCOR for ACIS comes from the header, and for HRC, from a table. For ACIS, there is one GTI table per CCD.

asphist:

program to compute the aspect histogram for a given chip. The aspect histogram is the duration of the pointing, weighted by the GTI and DTCOR, for each aspect offset bin in the dither pattern. It is a table of duration vs. x-offset and y-offset. The sum of the duration column is the EXPOSURE.

asp_apply_sim: [superceded by asphist in CIAO 2.2]

A program to add the SIM (science instrument module) offsets to the optical axis offsets before making an aspect histogram. This reduces the dimensionality from 6 to 3 without losing any information crucial to the response. (But it does lose information crucial for coordinate transformations, so don't use the product for event-processing.) Also adds some range-keywords to the header, which are used by asphist.

asp_calc_offsets: [superceded by asphist in CIAO 2.2]

program to convert the aspect solution into into offsets.

CALDB:

the calibration database, which is the repository of all instrument calibration files (such as quantum efficiency, mirror area, and geometry). Many parameters can simply say ``CALDB'', and the proper file will be looked up by date of observation, for time-dependent calibration quantities.

ARDLib:

Analysis Reference Data Library: an interface to the CALDB and other reference data. The ARDlib interface isolates the mission dependence from the generic object (such as a quantum efficiency vs energy). It can provide a many-to-one construction of CALDB files to one analysis object (such as by multiplying filter and detector efficiencies). It has a large parameter file, ardlib.par. See Davis (1999) for a description of ARDLib.

OSIP:

the Order Sorting and Integrated Probability table is used by tg_resolve_events for order-sorting, but it also contains the Integrated Probability: the fraction of the CCD response enclosed within the order-sorting energy limits. This is analogous to a PSF-fraction used in imaging analysis, and is a necessary factor in computing the ARF with mkgarf.

Spectral Analysis

GUIDE is the initial implementation of S-lang based high-resolution spectroscopic functions in the integrated CIAO software. CIAO has a larger suite of generalized fitting routines (Sherpa) and a FITS output model format (Model Descriptor List, or MDL file). CIAO 2.2 has a fully integrated S-lang interpreter, and will allow direct import of ISIS modules to extend or replace GUIDE. While ISIS can be imported, connections have not yet been made between underlying libraries (that is a major CIAO 3 effort).

Out-House

RMF-less analysis:

With high-resolution data, it is not always necessary to use the RMF (line-profile). If only the integrated flux is of interest, and there are no instrumental features within a spectral feature, then it suffices to divide the counts by the ARF to get a ``smeared'' flux per bin. (Both ISIS and Sherpa/GUIDE support RMF-less, or more accurately, diagonal RMF analysis.)

Order summing?

Summing of orders is definitely useful for inspection, visualization and presentation. It is not clear that it is better than joint-fitting for analysis, since each order contains slightly different information (or even systematic errors). ISIS and Sherpa/GUIDE support multiple-order fitting with a single model. If orders are combined, care must be taken to combine ARFs in a meaningful way.

The script, add_grating_orders adds plus and minus orders (same absolute value), and divides by the ARF.

Orders with different absolute values (i.e., 1,2,3) may be combined by either running tgextract once for each order with the proper gridding parameters (remember: tgextract scales by order), or by binning with dmcopy, e.g. (in sh syntax):

       fevt=Evt/acisf01451_000N003_evt2.fits
       flt_1="tg_srcid=1,tg_part=1,2,tg_m=-3,-2,-1,1,2,3"
       flt_2="tg_d=-0.000663889:0.000663889,tg_lam=1:25.005"
       fbin="bin tg_lam=1:25.00:0.01"
       fout=iip_lc_pha.fits
       dmcopy ${fevt}"[${flt_1},${flt_2}][${fbin}]"  ${fout} clob+   

(However, the result will not be in standard CXC PHA Type II grating spectral format. That is left as an exercise for the reader. And there may not be a totally-CIAO solution (yet).)

If orders are summed, then summed ARFs are also needed, on the proper grid. The grid can be selected with a mkgarf parameter. The summing can be done with dmtcalc and dmpaste.

Multi-order Response:

For LETG/HRC-S observations, a multi-order response may be required to assess higher order counts, especially for hard continuum sources. The multi-order response is either the sum over m of ARF(m)*RMF(m), or the support of multiple RMF,ARF pairs for a single PHA file and model. Neither are adequately supported yet, but both are in progress (as of 2001.10.02). The former implementation has restrictive gridding issues: to add the responses, all have to be on the same grid, and the grid has to be as fine as the highest resolution you wish to obtain. The latter allows each to be optimally gridded, but entails multiple model evaluations for each grid.

Parametric or responsive fitting:

Emission lines can either be fit by using parametric models, such as a sum of Gaussian components, or by using the LSF contained in an RMF, and folding the model through the response. Parametric fits can be done either on counts or fluxed counts. RMF-based fitting removes line width and shape parameters, and also automatically applies the energy dependence of the resolution.

Spectral-Timing Analysis

This is not explicitly supported in CIAO or ISIS, but it is possible to construct data products for spectral-timing work. Dmcopy can be used to bin into a wavelength-time image. Lightcurve can be used with dm-filters to form light curves in spectral regions, and an exposure record. Mean ARFs can be used if bins are comparable to the dither period (~1000 s), or if features of interest are not near chip gaps.