Page translated by Claude — switch to Italian to read the original article.

Image calibration is a crucial step during the pre-processing of astronomical images; this short article takes a closer look at the calibration of cameras that suffer from the phenomenon known as bias drift, thanks to calibration using the overscan area.

Overscan calibration is a calibration procedure that is little known among amateurs: although it is essential for professional data processing, it is generally of marginal importance for aesthetic astrophotography.

In some cases, however, it may become mandatory in order to solve certain calibration problems arising from peculiar characteristics of the sensor, such as the bias drift described below.

Flat fielding problems are among the most annoying and puzzling issues in astrophotography, and often the solution to these problems is sought in the wrong place. For example, one of the most frequent questions from beginners is: “I have a problem with my flat fields: at how many ADU should I shoot my flats to make them work?”.

The answer to this question is simple: “You are looking for the solution in the wrong place; if your camera is linear it does not matter: the ADU level is not an important parameter for the effectiveness of flats”.

In most cases, when flats seem not to work well, the problem does not lie in the flats themselves: it is just a symptom of a calibration problem hidden elsewhere.

Typical problems can be flexure of the equipment, which changes the configuration of the optical train, improper repositioning of the filter that makes it impossible to remove dust spots or, in the worst case, a non-linearity of the sensor that makes calibration practically impossible.

In this article I will describe bias drift, a little-known phenomenon that can cause significant calibration problems with apparently random behavior. The only solution to this problem is to perform calibration using the overscan area.

Example of vignetting overcorrection

A typical calibration problem: overcorrection of vignetting by the flat field. This problem can come from various sources, but it is usually the symptom of a problem in the calibration procedure.

The theory of calibration is widely discussed in various books and articles (those who have PixInsight should refer to the ImageCalibration documentation).
In short, the basic calibration equation is:

Where I_real is the calibrated light frame, I_Detected is the uncalibrated one, MD_u is the uncalibrated master dark for the light frame, MF_c is the calibrated master flat and s₀ is the flat scale factor used to normalize the final image.

An uncalibrated dark frame is made up of two additive components: the BIAS B and the dark current D_c

The calibration of flat frames consists of removing the additive components coming from the dark current and the bias by subtracting a suitable dark frame FD_f (called flat dark) with the same exposure time as the flat field.

Therefore, taking into account equation [2] and equation [3], equation [1] becomes:

Where LD_c and FD_c are the contribution of the dark current in the light frame and in the flat field respectively, and B is the bias frame.

In astrophotography practice, the Bias frame, the Dark frame (for both lights and flats) and the calibrated flat frame are usually replaced by master frames, obtained from the integration of numerous single frames (mainly to reduce noise and remove artifacts due to cosmic rays and charged particles).

The previous equations assume that the bias is the same for all the frames involved in the calibration process; in other words, the bias must not change over time: without this prerequisite the whole calibration workflow fails.

Introduction to bias drift

As explained above, during the calibration phase it is essential to remove, from both the light frame and the flat frame, all the additive component coming from the bias and the dark current. In particular, in the flat only the multiplicative component must remain, which is used to remove from the light frame defects such as vignetting, dust spots and variations in quantum efficiency across the surface of the sensor.

When bias drift is present, the bias can be considered as the sum of two components. The first component depends only on the position on the sensor surface B_p: this part is constant over time (at a given temperature) and is the same for all the frames in the calibration sequence. The second is an “offset” component that describes the time-dependent component O_t: it is, in practice, an offset that changes the average value of the bias over time.

Since O_t varies unpredictably from frame to frame, it becomes impossible to completely remove the additive part; in fact the flat frame and its flat dark could have two different biases.

To fully understand this fundamental concept, consider the calibration of the flat frame:

The flat is

the flat dark is

so the calibrated flat is

Since the offset components O_tF and O_tD of the flat and dark bias are different because of the bias drift phenomenon, the pure multiplicative component cannot be isolated, making the flat ineffective and, above all, the behavior can become erratic, with flats that sometimes work well (when O_tF and O_tD happen to be equal) and other times do not work at all.

The Solution: Overscan calibration

To solve the bias drift problem, the offset component must be measured independently for each frame before performing any other calibration step: this zeroes out the offset level of each frame and allows the time-dependent additive component to be completely removed from the individual frames.

The operation can be carried out by suitably measuring a specific area of the sensor’s overscan region.

The camera’s overscan region is a part of the sensor outside the active pixel area. It is usually covered with an opaque coating and does not collect light. Besides the “dark” signal useful for calibration, the overscan area could also contain other information: however, these additional pixels do not contain useful data for image processing and must not be included in the analyzed area, which is why the overscan source region must be chosen carefully.

In the case described above, the overscan pixels are physical (they are actual portions of the sensor) and include bias, read noise and dark current.

Sometimes, however, part of the overscan pixels are generated synthetically and added to the image; in this case, they contain only the information about the bias offset level.

The size, structure and content of the overscan area vary from sensor to sensor; it is therefore impossible to provide a single set of parameters for calibration: the information must be found in the manufacturer’s documentation or, if not available, found by trial and error.

Figure 2, for example, shows the structure of the KAF 16200 CCD, whose data are easily available on the manufacturer’s website.

Figure 2 — KAF 16200 architecture

From this image we can identify the pixels to be used for overscan calibration: with this particular CCD, the overscan area is a physical part of the sensor, covered by a metallic coating.

Around the periphery of the device, in fact, there is a border of light-shielded pixels that create an unilluminated region. Within this dark region there are light-shielded pixels, including 36 initial dark pixels on each row. There are also 30 completely dark lines at the beginning and 23 completely dark lines at the end of each frame. Under normal circumstances, these pixels do not respond to light and can be used as a dark reference for overscan calibration.

Figure 3 — Example of overscan

This image shows a small portion of the upper left corner of an uncalibrated light: the overscan area is visible as a dark frame surrounding the active region.

Diagnosing a bias drift problem

As mentioned earlier, bias drift appears as a significant variation in the average value of the bias or the dark from image to image.

The simplest way to identify it is to acquire a series of consecutive biases after cooling the CCD camera, letting it stabilize for a few minutes and, finally, analyzing them with a statistical tool. If the bias is stable, the median value of the various frames will be practically constant; if instead the bias level were unstable, the median value would change significantly from one bias to another.

Figure 4 shows a graph obtained from a series of biases of an astronomical camera based on a KAF 16200 sensor.

Figure 4 — Bias drift

The graph shows the variation in the ADU level over a series of consecutive biases acquired with a KAF 16200-based camera set to -15°C. A significant variation in the bias level caused by the bias drift phenomenon is easily noticeable.

The same analysis can be performed with a series of dark frames.

Figure 5 — Dark frame instability

The graph shows the variation in the ADU level over a series of consecutive 300-second darks, acquired with a KAF 16200-based camera set to -15°C. A significant variation in the level caused by the bias drift phenomenon is easily noticeable.

In PixInsight, the simplest way to analyze a large number of frames is to use the BatchStatistics script, which allows several images to be analyzed, obtaining key statistical information and then saving it to a CSV file that can subsequently be analyzed with a spreadsheet application.

Figure 6 — BatchStatistics script interface

Typical workflow for Overscan calibration in PixInsight.

PixInsight is able to easily handle the entire calibration process through the ImageCalibration process using up to four different overscan areas.

The first piece of information needed before starting calibration is, obviously, the overscan configuration of your camera, defined by the following parameters:

Image Region

This set of four parameters defines the active image region of the whole camera; the parameters contain:

• Left pixel coordinate of the image region
• Top pixel coordinate of the image region
• Width of the image region in pixels
• Height of the image region in pixels

Source Region

This set of four parameters defines the source region of the current overscan area, in which the overscan value is measured.

• Left pixel coordinate of the overscan source region
• Top pixel coordinate of the overscan source region
• Width of the source region in pixels
• Height of the source region in pixels

Target Region

This set of four parameters defines the target region of the current overscan area; ImageCalibration subtracts the average value measured in the source region from every pixel in the target region. Typically, for a camera with a single sensor, these settings are the same as the image region.

• Left pixel coordinate of the overscan target region
• Top pixel coordinate of the overscan target region
• Width of the target region in pixels
• Height of the target region in pixels

As mentioned above, PixInsight’s overscan correction can handle up to four source and target regions; this is essential for multi-sensor CCD cameras (or multi-amplifier ones such as those based on KAF–50100) where each sensor has its own amplifier with, possibly, a slightly different bias level: in this case overscan correction is mandatory even if there is no bias drift.

The typical calibration workflow involves the following steps:

1. Uncalibrated master dark integration
2. If necessary, creation of the uncalibrated master bias
3. Flat frame calibration
4. Calibrated master flat integration
5. Light frame calibration
6. Master light integration

The entire overscan calibration workflow is fully supported in the WeightedBatchPreprocessing script from version 2.5.0: earlier versions of the script have only limited support and may give sub-optimal results.

Master dark and master bias integration

From the calibration point of view, dark frame, flat dark frame and bias frame are exactly the same: they contain only an additive signal with no light information. Note that it is recommended not to calibrate either the master dark frame or the master bias frame; this avoids data clipping without the need to use a PEDESTAL.

As explained in the ImageCalibration documentation, these master frames are simple averages of the single frames without any weighting or normalization: only pixel rejection is enabled to remove cosmic rays and other charged-particle traces from the single frames. This applies, in particular, to overscan calibration, which can be applied indifferently before or after the integration of the master frame. In fact, for example, for the master bias, thanks to the associative property of addition, we can write:

Similar equations can also be written for the master darks; therefore, calibrating the bias or dark frame for overscan before or after integration leads to exactly the same result. However, as mentioned above, overscan calibration without applying an appropriate PEDESTAL can cause heavy data clipping in the master bias and master dark frames; this is why I strongly recommend using uncalibrated master frames.

Figure 7 — Recommended setting for the integration of dark frames and bias frames

Figure 7 shows the recommended settings for performing the integration, using PixInsight’s ImageIntegration process, of the master darks and master biases; the main settings are:

• Combination: Average
• Normalization (integration): No normalization
• Weight: Don't care (all weights=1)
• Rejection algorithm: Winsorized sigma clipping (or Percentile clipping if there are less than 6 frames
• Normalization (rejection): No Normalization
• Rejection settings: Use the default values
• Signal and noise Evaluation: Unchecked (there are no stars to measure)

The final result of this integration step is the uncalibrated master dark frame or the uncalibrated master bias frame.

Flat frame calibration.

As explained in the Introduction to bias drift section, flat frames must be calibrated before the integration phase in order to remove the entire additive component coming from bias and dark current. Calibration is fundamental because, during integration, the flat frames must be normalized with a multiplicative factor, and a residual additive component would contaminate the pure multiplicative information needed in the master flat: this is why they must be calibrated individually before integration.

At this stage it is essential to know the overscan parameters to be used in the ImageCalibration process.

Figure 8 — Recommended setting for flat frame calibration

Figure 8 — Recommended setting for flat frame calibration

Figure 8 shows the default setting for performing flat calibration. In this example, the assumption is that the exposure time of the flat frames is short enough to consider the dark current negligible. Therefore, calibration will be carried out exclusively with the uncalibrated master bias. If instead the exposure time were long enough to create a significant dark current, then an uncalibrated master flat-dark would be needed (a master dark with the same exposure time as the flats).

Finally, as a last working assumption, suppose the master dark has a different exposure time from the flat frame. In that case, it can be optimized using a master bias to remove the time-independent additive component (see the light frame calibration further on for an example).

The main settings are:

• Enable CFA: Disabled or enabled depending on the type of file (from a monochrome or color camera).
• Signal evaluation: Disabled, there are no stars to measure.
• Overscan: Active with the overscan region data set.
• Master bias: with Calibrate active, this applies overscan calibration to the uncalibrated master bias.

for other relevant parameters, see the ImageCalibration documentation.

Master flat integration

After the calibration phase, the individual flat frames must be integrated to obtain the calibrated master flat. As mentioned earlier, during integration the flat frame should be normalized both in the pixel rejection phase and in the integration phase; normalization is mandatory for flats obtained with non-constant illumination (such as sky-flats), but it is also important for those obtained with a flat-box.

Normalization must not alter the pure multiplicative signal of the flat field, so it must be set to multiplicative for the integration phase and equalize fluxes for the pixel rejection phase.

Perfect pixel rejection is essential, particularly for flats taken on the sky, which may contain some stars in the field that must therefore be rejected. However, it is also necessary for flats acquired with a flat-box because the probability of recording the trace of a cosmic ray or a charged particle is not negligible, even though the exposure times are generally short.

Figure 9 — Recommended setting for master flat integration

Figure [9] shows the default setting for performing master flat integration; the main settings are:

• Combination: Average
• Normalization (integration): Multiplicative
• Weight: Don't care (all weights=1)
• Rejection algorithm: Winsorized sigma clipping (or Percentile clipping if there are less than 6 frames)
• Normalization (rejection): Equalize Fluxes
• Rejection settings: Use the default values
• Signal and noise Evaluation: Inactive (there are no stars to measure to evaluate the signal-to-noise ratio)

The final result of this integration phase is the uncalibrated master flat frame.

Light frame calibration

To calibrate the light frames, at least two masters are needed:

• An uncalibrated master dark, taken at the same temperature and exposure time as the light frame.
• A calibrated master flat

If the exposure time or temperature of the master dark differ from those of the light frame, the master dark must be optimized and another file is needed: an uncalibrated master bias.

The following example refers to this second scenario.

Figure 10 — Recommended settings for light frame calibration

The main settings:

• Enable CFA: Disabled or enabled depending on the type of frame (monochrome or One Shot Color)
• Signal evaluation: Enabled, the default settings are suitable for most cases.
• Overscan: Active with the overscan region data set.
• Master bias: With Calibrate active, this applies overscan calibration to the uncalibrated master bias (obtaining a calibrated master bias).
• Master dark: With Calibrate and Optimize active, this applies overscan calibration to the uncalibrated master dark, then subtracts the (previously calibrated) master bias and, finally, optimizes the dark current.
• Master flat: With Calibrate inactive. As seen in the previous paragraphs, the master flat has already been calibrated, therefore the master flat must absolutely not be re-calibrated at this stage. For CFA images (obtained from color cameras) I strongly recommend enabling the "Separate CFA flat scaling factors" flag (see the ImageCalibration documentation for more details on this point)

The light frames are now ready for further processing, such as “Cosmetic correction” or registration and final integration of the image

The final result

Below are some calibration examples: the comparison with and without overscan compensation shows how this technique can solve a significant problem in the use of flat fields.

The data sets are exactly the same; the only difference is in the calibration phases; only an automatic transfer function (automatic ScreenTransferFunction with the “nuclear” button) has been applied to the linear images.

Example 1: M27

M27: Image calibrated without overscan compensation

M27: Direct comparison between the image without compensation and with compensation (hover the mouse over the image to toggle between the images)

4.5.2 Example 2: SH2–280

Example 2: SH2–280

SH2–280: Image calibrated without overscan compensation

SH2–280: Direct comparison between the image without compensation and with compensation (hover the mouse over the image to toggle between the images)