.. C-Munipack - User's manual

   Copyright 2012 David Motl

   Permission is granted to copy, distribute and/or modify this document under the
   terms of the GNU Free Documentation License, Version 1.2 or any later version published
   by the Free Software Foundation; with no Invariant Sections, no Front-Cover Texts, and
   no Back-Cover Texts.

   $Id: photometry_files_format.rst,v 1.2 2015/07/12 07:44:57 dmotl Exp $

.. index:: 
   pair: photometry; files

.. _photometry-files-format:

Photometry files
================

The photometry files are produced by the photometry process. The file
consists of a table of detected objects and their properties. 

The photometry files are stored as binary streams in order to get 
optimum performace, because it does not require analyzing any kind of 
language and decoding the values.

The binary format of photometry files was introduced to latter versions of C-Munipack 1.2,
because it has turned out that the XML-based formats are too slow to parse.

Multi-byte data is stored in little-endian format (Intel). Real numbers are stored in the
*binary64* format of IEEE 754-1985.

.. rubric:: File header

The general structure of binary photometry file is given in table :ref:`1 <table:pht_bin_head>`.

.. _table:pht_bin_head:

Table 1

====== ====== ======== ===========
Offset Type   Length   Description
====== ====== ======== ===========
 0     char[] 28       Format identifier
28     long   4        Format revision number
32     long   4        Length of metadata in bytes
====== ====== ======== ===========

The format identifier must be the following text::

   C-Munipack photometry file\r\n

Current format revision number is **4**.

.. rubric:: Metadata

The metadata block immediately follows the file header. The parameters are stored on fixed offsets. 
The length of the header is 540 bytes. See the table :ref:`2 <table:pht_bin_meta>` for description of 
header fields.

.. _table:pht_bin_meta:

Table 2

====== ============ ======== ===========
Offset Type         Length   Description
====== ============ ======== ===========
0      long         4        unused    
4      long         4        Frame width in pixels    
8      long         4        Frame height in pixels    
12     binary64     8        Julian date of observation 
20     char[]       70       Optical filter, right-padded with spaces 
90     binary64     8        Exposure duration in seconds 
98     binary64     8        CCD temperature in Centigrades 
106    char[]       70       Reduction software 
176    short        2        Date of creation (year) 
178    byte         1        Date of creation (month) 
179    byte         1        Date of creation (day) 
180    byte         1        Date of creation (hour) 
181    byte         1        Date of creation (minute) 
182    byte         1        Date of creation (second) 
183    byte         1        unused  
184    binary64     8        Lowest good pixel value 
192    binary64     8        Highest good pixel value 
200    binary64     8        Gain in electrons per ADU 
208    binary64     8        Readout noise in ADU 
216    binary64     8        Expected FWHM of objects 
224    binary64     8        Mean FWHM of objects 
232    binary64     8        Standard error of FWHM of objects 
240    binary64     8        Detection threshold 
248    binary64     8        Low sharpness cutoff 
256    binary64     8        High sharpness cutoff 
264    binary64     8        Low roundness cutoff 
272    binary64     8        High roundness cutoff       
280    long         4        Matching status: 0=not matched, 1=matched 
284    long         4        Number of stars used in matching 
288    long         4        Number of polygon vertices 
292    long         4        Number of matched stars   
296    binary64     8        Clipping threshold 
304    binary64     8        Offset in X axis to the ref. frame in pixels  
312    binary64     8        Offset in Y axis to the ref. frame in pixels  
320    char[]       70       Object's designation, right-padded with spaces 
390    binary64     8        Object's right ascension
398    binary64     8        Object's declination
406    char[]       70       Observer's location designation, right-padded with zeros 
476    binary64     8        Observer's longitude
484    binary64     8        Observer's latitude
492    binary64[]   48       Transformation matrix (xx, xy, x0, yx, yy, y0)
====== ============ ======== ===========

Object's right ascension - the reader must interpret a value that is outside range of 0 and 24 as undefined value.
The writer is recommended to use DBL_MAX value to indicate an undefined value.

Object's declination - the reader must interpert a value that is outside range of -90 and 90 as undefined value.
The writer is recommended to use DBL_MAX value to indicate an undefined value.

Observer's longitude - the reader must interpret a value that is outside range of -360 and 360 as undefined value.
The writer is recommended to use DBL_MAX value to indicate an undefined value.

Observer's latitude - the reader must interpret a value that is outside range of -360 and 360 as undefined value.
The writer is recommended to use the DBL_MAX value to indicate an undefined value.

.. rubric:: Transformation matrix

The transformation matrix holds an affine transformation, such as a scale, rotation, 
shear, or a combination of those. The transformation of a point (x, y) is given by:

.. math::
   :label: trafo

    x_{new} = x_x * x + x_y * y + x_0 \\
    y_{new} = y_x * x + y_y * y + y_0

.. rubric:: WCS data
    
The WCS data block immediately follows the metadata block. The first four
bytes (long type) specify the length of the block in bytes, the data starts 
at offset 4. The WCS data are encoded as a FITS header. 
    
.. rubric:: Table of apertures

The table of apertures immediately follows the WCS data block. The first four
bytes (long type) specify a number of records in the table, the first record starts 
at offset 4, records have 12 bytes each. The description of fields in the table
of apertures is given in table :ref:`3 <table:pht_bin_aper>`.

.. _table:pht_bin_aper:

Table 3

====== ============ ======== ===========
Offset Type         Length   Description
====== ============ ======== ===========
0      long         4        Aperture identifier
4      binary64     8        Radius in pixels
====== ============ ======== ===========

.. rubric:: Table of objects

The table of objects immediately follows the table of apertures. The first four
bytes (long type) specify a number of records in the table, the first record starts 
at offset 4, records have 48 bytes each. The description of fields in the table
of objects is given in table :ref:`4 <table:pht_bin_object>`.

.. _table:pht_bin_object:

Table 4

====== ============ ======== ===========
Offset Type         Length   Description
====== ============ ======== ===========
0      long         4        Object identifier within a file (starts with 1 for valid entries)
4      long         4        Global identifier (assigned in matching), zero or negative value if the objects was not matched (*)
8      binary64     8        X-coordinate of position of the object 
16     binary64     8        Y-coordinate of position of the object
24     binary64     8        Local mean background level in ADU
32     binary64     8        Std. deviation of local background level
40     binary64     8        Full width at halt maximum (FWHM) in pixels
====== ============ ======== ===========

(*) It is recommended to assign valid object identifiers from the range of 1 to 2,147,483,647 (MAX_INT). 
The reader should interpret zero or negative value of object identifier as an invalid entry; the record should be silently ignored.
The reader should interpret zero or negative value of global identifier (a result of matching) as a non-matched object.
The writer is recommended to use the value (-1) to indicate invalid entries and non-matched objects.

.. rubric:: Table of measurements

The table of measurements immediately follows the table of objects. The number
of records is not given explicitly, but it is always number of objects multiplied
by number of apertures. Objects and apertures are stored in the same order as they
appear in the tables of objects and table of apertures, respectively. The first
records that specify measurement data for the first object and aperture
stars at offset 0, the second record specify measurement data for the first object
and second aperture, etc. Each record has 12 bytes. Unlike previous fields, 
magnitude and its error is stored in 24.8 format with fixed decimal point. The
value ``0x7FFFFFFF`` means undefined value. The description of fields in the 
table of measurements is given in table :ref:`5 <table:pht_bin_body>`.

A non-zero integer value at offset 8, introduced in revision 3, specify a reason why
the object was not measured. The table :ref:`6 <table:pht_bin_codes>` gives a list
of possible values and their meaning.


.. _table:pht_bin_body:

Table 5

====== ============ ======== ===========
Offset Type         Length   Description
====== ============ ======== ===========
0      long         4        Raw instrumental brightness in magnitudes, 8.24 fixed point format
4      long         4        Error estimation of brightness, 8.24 fixed point format
8      long         4        A value indicating the reason why the object was not measured
====== ============ ======== ===========

.. _table:pht_bin_codes:

Table 6

====== ============
Value  Description
====== ============
0      The object was measured successfully
1014   Computed magnitude is outside the range of -99 and 99 magnitudes
1600   The net intensity in the aperture is zero or negative (*)
1601   The aperture size has an invalid value
1602   The aperture is too close to the frame border    
1603   There are overexposed pixels in the aperture
1604   There are pixels of invalid value in the aperture
1605   There are not enough pixels with valid value in the sky annulus
====== ============

(*) The net intensity is defined as sum of pixel values in the aperture minus
mean background intensity multiplied by aperture area.
