\section{Introduction}

\CLIC is a software developped at IRAM to support the calibration of
the data from the NOEMA interferometer. It stands for Continuum and
Line Interferometric Calibration and is part of the \GILDAS
package. We refer to the \GILDAS documentation for further information
on the \GILDAS software.  \CLIC uses code from the so-called \GILDAS
kernel like \SIC for command interpretation or \texttt{GREG} for the
plots. It also uses routines from \ASTRO to do astronomical
computations. \CLIC uses a specific binary format for data files (see
section \ref{sec:file}) produced at the NOEMA observatory by an online
reduction software called \texttt{RDI}, that itself uses the \CLIC
libraries (for data writing, atmospheric calibration, pointing and
focus solving, etc.).

\CLIC is also adapted to the NOEMA observatory in terms of
commands. It provides the tools to do the required calibrations using
the NOEMA nomenclature (for example, the presence of a noise source at
NOEMA allows to align all spectral windows connected to a baseband in
amplitude and phase, so the passband calibration is then done per
baseband and applies to all the spectral windows that pertain to that
baseband).

As part of the ALMA project, \CLIC has been used to assess the
performances of the ALMA antennas through holographies. A branch of
\CLIC has then been ported to TELCAL, the online calibration software
of ALMA. This activity has led to the development of import
capabilities, first as TI-FITS, then as ASDM.  While these
capabilities are deprecated using IRAM HEAD version, branches of \CLIC
are still used to reduce holographies for ALMA as previously mentioned
and for the Yebes 40m telescope (using TI-FITS). IRAM has also
developped a package, called MRTHOLO, that process IMB-FITS from the
30m telescope and produces IPB file, for reduction of 30m holographies
within \CLIC.

The output files after calibration are uv tables, in the \GILDAS data
format (GDF). Export possibilities are provided throuhg UV-FITS (see
MAPPING documentation).

This document describe first how \CLIC files are organized, then
describes the commands. The details of NOEMA data calibration are left
to the documentation of the pipeline.


The appendix contains the online help of \CLIC.

\newpage
\section{Formalism}

\subsection{Nomenclature}
\label{ref:nomenclature}
For NOEMA, we follow the nomenclature defined in the document
"Frequency Band Naming Convention".

The receiver transforms and filters the input RF signals. Its output
consists of one or more intermediate frequency bands, that we call
IFs. These IFs are eventually then further processed by an IF
processor that does another round of transformation and filtering
bringing the signal to baseband for further use downstream. That input
to the correlator is what we call baseband in this document and in
\CLIC. The correlator then processes the baseband in one or more
spectral windows, that are a continuous set of adjacent channels
sharing a correlator mode (including channel spacing). A given data
set may hold spectral windows with different channel spacing though.

\subsection{Frequency and time}
\subsubsection{Calibration}
The goal of the calibration is to estimate and correct for the
instrumental gain in order to have an unbiased estimate of the true
visibilities. We use a simplified version of the radio-interferometry
measurement equation. The (complex) visibility $\widetilde{V}_{ijk}$
measured on the baseline from antenna $i$ to antenna $j$ at frequency
channel $k$ is related to the true object visibility
$V_{ij}$ by\\
\begin{equation}
  \widetilde{V}_{ijk} = g_i(t) g_j^*(t) b_{ijk}(t) V_{ij}(u_k(t),v_k(t)) + noise term
\label{eq:measurement}
\end{equation}
where $u_k(t)$ and $v_k(t)$ are the spatial frequencies corresponding
to baseline $ij$ at time $t$ and frequency $k$, and we assume the
object has a flat spectrum. Calibrating the data is computing the
complex ``calibration curves'' $g_i(t)$ and $b_{ijk}(t)$.

\subsubsection{Frequency dependance}

$b_{ijk}(t)$ is the bandpass of the detection system, and is usually
almost constant with time. It can be formally decomposed as a product
of RF bandpass, caused by receivers and cables and usually with weak
dependence on frequency, and IF bandpass, caused by the backend
(spectral and continuum correlators at Bure).

\subsubsection{Time dependance}
For $g_i(t)$, we must separate the calibration of amplitude and phases
since amplitude and phase errors have very different origins.  The
amplitude corrections are related to several effects: atmospheric
absorption, receiver gain, antenna gain (affected by pointing errors,
defocussing, surface status and systematic elevation effects), and
correlation losses due to phase noise. Phase errors may come from
delay errors, baseline errors, or a slow drift in atmospheric or
receiver phases.

\subsection{Noise}
\subsubsection{Temperatures}
\label{sec:noise-temp}
The power $P_a$ received by an antenna in a bandwidth $\delta\nu$ is
conveniently expressed in term of {\bf antenna temperature, $T_a$},
which is the temperature for which an equivalent resistor would give
the same power $P_a$, following the Nyquist noise formula:
\begin{equation}
P_a = k T_a \delta\nu \label{eq:pt}
\end{equation}
As the quantity $T_a$ is affected by the atmospheric absorption
$e^{-\tau_{atm}}$ and the antenna foward coupling factor $\eta_f$, we
define the quantity $T_a^*$ through:
\begin{equation}
T_a^* = \frac{(1+g_{im})}{\eta_f}e^{\tau_{atm}}T_a \label{eq:ta*}
\end{equation}
The $(1+g_{im})$, where $g_{im}$ is the image band rejection factor,
accounts for a single sideband signal.

{\bf Brightness temperature, $T_b$,} is the Rayleigh-Jeans temperature
$T_b$ of an equivalent blackbody which would give the same power per
unit area per unit frequency and per unit solid angle $I_{\nu}$ as the
celestial source:
\begin{equation}
T_b = \frac{c^2}{2 k\nu^2} I_{\nu}
\end{equation}  
For a resolved source, the antenna temperature is equal to the sky
brightness temperature. For an unresolved source, the coupling between
antenna temperature $T_a^*$ and source flux density $S_{\nu}$is:
\begin{equation}
T_a^*=\frac{\eta_a A}{2k}S_{\nu} =\frac{1}{J} S_{\nu}
\end{equation}
where $J = \frac{2k}{\eta_a A}$ is the antenna efficiency.


The {\bf noise temperature} is the sum of the various noise
contributions:
\begin{equation}
T_{noise} = T_{bg} + T_{sky} + T_{spill} + T_{rec} \label{eq:tnoise}
\end{equation}
where $T_{bg}$ is the cosmic background, $T_{sky}$ is the sky noise,
$T_{spill}$ is the ground noise pickup and $T_{rec}$ is the receiver
noise temperature. The forward efficiency $\eta_f$ is a property of
the antenna that indicates how much coupling there is in the forward
hemisphere with respect to the full $4\pi$ sphere. With an atmosphere
at a physical tempeture $T_{atm}$ and with an opacity $\tau_{atm}$ at
the observed frequency, the sky noise and ground pickup temperature
are expressed as:
\begin{eqnarray}
T_{sky}  &=& \eta_f (1-e^{-\tau_{atm}})T_{atm} \\
T_{spill}&=& (1-\eta_f)T_{ground}
\end{eqnarray}
$T_{sys}$ is the {\bf system temperature}, the noise equivalent
temperature $T_{noise}$ of the receiving chain refered to a perfect
antenna located outside the atmosphere and for a single sideband
signal:
\begin{equation}
T_{sys} = \frac{1+g_{im}}{\eta_f}e^{\tau_{atm}} T_{noise} \label{eq:tsys}
\end{equation}





\subsubsection{Radiometric equation}
The radiometric equation (eq. \ref{eq:noise}) gives the thermal noise
$\sigma$ as a function of observing bandwidth $\delta\nu$, integration
time $\delta t$, for an array with $n$ antennas, where $A$ is the
geometric surface of an antenna and $\eta_a$ is the aperture
efficiency. It also depends on the correlator efficiency $\eta_q$ and
the phase decorrelation factor $\eta_p$.

For a baseline, we have the following noise equation:
\begin{equation}
\sigma= \frac{1}{\eta_q\eta_p}\frac{\sqrt{2}k}{\eta_a A}\frac{T_{sys}}{\sqrt{\delta\nu\delta t}} \label{eq:sigma-base}
\end{equation}

The array point source sensitivity is given by:
\begin{equation}
\sigma= \frac{1}{\eta_q\eta_p}\frac{2k}{\eta_a A}\frac{T_{sys}}{\sqrt{n(n-1)\delta\nu\delta t}} \label{eq:noise}
\end{equation}

\subsubsection{Weights}
As usual, the weights $W_{ij}$ associated to a visibility $V_{ij}$ are
defined to be:
\footnote{The weights in the GILDAS format \texttt{.uvt} tables
are computed with a bandwidth in MHz and an antenna efficiency in Jy/K 
(rest of units are S.I.), so there is a factor $10^6$ to be 
applied to the weights to get 1/Jy$^2$ unit for the weights (or $10^{-3}$
to the sensitivity to get Jy).}

\begin{equation}
W_{ij} = \frac{1}{\sigma_{ij}^2}
\end{equation}


\subsection{Amplitude and phase}
The visibilities are complex numbers, whose real and imaginary parts
are measured in the correlator. However, since amplitude and phase
distortions have different physical origin, and that amplitude and
phase errors have quite distincts consequences in the image plane, one
often decomposes the visibilities in amplitude and phase following:
\begin{equation}
  V_{ij}(t) = a_{ij} (t) e^{i\varphi_{ij}(t)}
\end{equation}

\subsubsection{Closure relationships}
It follows that if the gains are antenna based like in equation
\ref{eq:measurement}, we have the following closure relationships:
\begin{eqnarray}
\widetilde{\varphi}_{ij} + \widetilde{\varphi}_{jk} - \widetilde{\varphi}_{ik} 
&=& \varphi_{ij} + \varphi_{jk} - \varphi_{ik} \\ 
\frac{\widetilde{a}_{ij}\widetilde{a}_{kl}}
{\widetilde{a}_{ik}\widetilde{a}_{jl}} &=& \frac{a_{ij}a_{kl}}{a_{ik}a_{jl}}
\end{eqnarray}

\subsubsection{Statistics}
\label{sec:statistics}
It is worth noting that while the noise in the real and imaginary
parts of the visibilities follows gaussian statistics, with a null
expected value, the amplitude and phase do not, especially at low
signal-to-noise. The amplitude in particular is by definition
positively biased, i.e. has a non-null expected value even in the
absence of signal.

\begin{figure}
\begin{center}
\includegraphics[width=0.8\textwidth]{visibility-statistics.pdf}
\caption{Probability distribution for the amplitude ({\bf Top}) and real part ({\bf Middle}) of the visibilities for various signal-to-noise ratio (0,1,2,3,5,10). Adapted from Wrobel and Walker (1999). {\bf Bottom:} Expected value for the amplitude as a function of signal strength (in unit of $\sigma$) for the amplitude (plain curve) and real part (dashed curve). \label{fig:amp-stat}}
\end{center}
\end{figure}
Figure \ref{fig:amp-stat} shows the probability distribution for the
amplitude and real part of the visibilities for various
signal-to-noise ratios and compares the expected values for both
quantities. Figure \ref{fig:phase-stat} shows the probability
distribution for the phase. It follows that at low signal-to-noise a
detection is more easily seen on the phase than on the amplitude.

\begin{figure}
\begin{center}
\includegraphics[width=0.8\textwidth]{phase-statistics.pdf}
\caption{Probability distribution for the phase of the visibilites for various
signal-to-noise ratio (0,1,2,3,5). Adapted from Wrobel and Walker (1999). 
\label{fig:phase-stat}}
\end{center}
\end{figure}



\subsubsection{Averaging}
\label{sec:averaging}
The averaging of phase fluctutations will produce an amplitude
decorrelation.  For example, a phase jitter with gaussian noise $\phi$
will introduce a factor $e^{-\phi^2/2}$ of decorrelation if the real
and imaginary parts are averaged (this is the classical definition of
complex averages). \CLIC offers the possibility to do averaging on the
real and imaginary part or to average amplitude and phase (see
\texttt{SET AVERAGING} for more details). In the latter case, the
amplitude bias may lead to surprising results in the case of low
signal-to-noise. For instance, the amplitude of the low spectral
resolution windows may not be the same as the high spectral resolution
spectral windows. Note also that if the averaging implies a
decorrelation, then baseline based factors are introduced and the
closure relationship does not hold anymore for amplitude.


\subsection{Baseline and antennas}
In equation \ref{eq:measurement}, the instrumental gain is implicitely
described as a product of two antenna gains:
\begin{equation}
G_{ij}(t) = g_i(t) {\cdot} g_j^*(t) 
\end{equation}
This is usually satisfied, unless averaging in time or frequency
introduce a decorrelation. For a point source at the phase center, the
true visibilities are constant in the uv plane with value ($S$,0),
where $S$ is the source flux density, so we have a set of equations:
\begin{equation}
\widetilde{V_{ij}} = g_{i} {\cdot} g_j^*S
\end{equation} 
or equivalently:
\begin{eqnarray}
\widetilde{\varphi}_{ij} = \phi_j - \phi_i \\
\widetilde{a}_{ij}  = a_{i}  {\cdot} a_j \cdot S
\end{eqnarray}


\newpage
\section{Files}
\label{sec:file}
\subsection{Principle}
The \CLIC data format was derived from the \CLASS data format, and
while there are some specificities unique to \CLASS or \CLIC, the data
format itself is generic enough that in 2013, the low-level routines
from both were factorized in a new \CLASSIC library. The readers
interested in more details are invited to refer to the \CLASSIC
documentation. In essence, a file is composed of a file header and one
or more extension(s). The file header provides information about the
file system used to encode the file, so both little endian and big
endian are supported. \CLASSIC files are fortran direct access,
unformated binary files that are accessed record by record. The record
length has an impact on the I/O depending on the file system and is
stored in the file header. This should be considered for any
discussion on performances.

The extensions contain an extension index, and observations. The
extension index contains all the observation titles of the
observations in the extension.  The observation title is composed of
32 4-bytes word (see table \ref{tab:index}), allowing a quick search
for observations matching any combination of these words without
having to read all of the file. Observations are independant and can
be read individually, which allows only the desired subset of data to
be read. With large datasets of hundreds of gigabytes, it is also
important to use memory as efficiently as possible so \CLIC uses its
own cache. The size of the memory that \CLIC has access to is
controled by the \texttt{SPACE\_CLIC} logical name.  It can be set
with the command \texttt{SIC LOGICAL SPACE\_CLIC} value, with value in
MB, unless the unit is specified, e.g. 64GB or a fraction of the RAM
size is specified, e.g. 50\%.  The ideal situation is when the value
accessible to \CLIC is larger than the size of the raw data file
(which is another reason to have as compact as possible the
files). This is typically the case on the data reduction machines
available at IRAM. This reduces the I/O to reading the file once and
writing a few calibrations, before producing large uv tables.

\subsection{Index}
The index containing the title of all the observations in an input
file is read when the file is opened. Starting with the use of
\CLASSIC, and to allow files with a larger number of observations or
records, the version 2 of files uses 8-bytes integers wherever
needed. In addition, with version 2, observations do not have to start
at the beginning of a record, so no space is left blank in the
file. Table \ref{tab:index} describes the content of an observation
title.
 
\begin{table}[htb]
\begin{center}
\begin{tabular}{|l|l|l|l|l|l|}
\hline
Word & Starting word & Length & type  & unit & comment \\
\hline
bloc    & 1  & 2 & INTEGER*8 & [records]  & record number of entry start \\
word    & 3  & 1 & INTEGER*4 & [4-bytes] & offset in words in record \\
\hline
ver     & 4  & 1 & INTEGER*4 &         & version of the observation \\
num     & 5  & 2 & INTEGER*8 &         & observation number \\
sourc   & 7  & 3 & INTEGER*4(3) &  & source name \\
line    & 10 & 3 & INTEGER*4(3) &  & tuning name \\
teles   & 13 & 3 & INTEGER*4(3) &  & interferometer configuration \\
dobs    & 16 & 1 & INTEGER*4    & [days] & classic date of observation\\
dred    & 17 & 1 & INTEGER*4    & [days] & classic date of reduction \\
off1    & 18 & 1 & REAL*4       & [radian] & lambda offset \\
off2    & 19 & 1 & REAL*4       & [radian] & beta offset \\
typec   & 20 & 1 & INTEGER*4    & & type of coordinates \\
kind    & 21 & 1 & INTEGER*4    & & data kind \\
qual    & 22 & 1 & INTEGER*4    & & data quality \\
scan    & 23 & 1 & INTEGER*4    & & scan number \\
proc    & 24 & 1 & INTEGER*4    & & procedure type \\
itype   & 25 & 1 & INTEGER*4    & & observation type \\
houra   & 26 & 1 & REAL*4       & [radian] & jour angle \\
project & 27 & 2 & INTEGER*4(2) & & project name \\
bpc     & 29 & 1 & INTEGER*4    & & baseline bandpass cal status \\
ic      & 30 & 1 & INTEGER*4 & & instrumental cal status \\
recei   & 31 & 1 & INTEGER*4 & & receiver band number \\
ut      & 32 & 1 & REAL*4    & [s] & UT time \\
\hline
\end{tabular}
\caption{Version 2 of observation title. \label{tab:index}}
\end{center}
\end{table}

\subsection{Observations}
The observations themselves contain the data but also all the metadata
associated with the observations. The observation is hence composed of
an observation header and data.

\subsubsection{Observation Header}
In the observation header, a number of sections can be stored. Table
\ref{tab:sections} displays the list of sections known to \CLIC and
whether they are present in hpb and IPB files.

\begin{table}[htb]
\begin{center}
\begin{tabular}{|l|r|c|c|}
\hline
Name                   & Code & IPB & hpb  \\
\hline
General                & -2  & x & x \\
Position               & -3  & x & x \\
Interferometer         & -21 & x & x \\
RF Frequency setup     & -22 & x & x \\
Continuum setup        & -23 & x & x \\
Line setup             & -24 & x & x \\
Scanning parameters    & -25 & x & x \\
Atmospheric parameters & -26 & x & x \\
Antenna Passband Cal   & -27 &   & x \\
Baseline Passband Cal & -28 &   & (x) \\
Instrumental Cal.      & -29 &   & (x) \\
Data descriptor        & -30 & x & x \\
Antenna instrumental Cal & -31 & & x \\
Atmospheric monitoring & -32 & x & x \\
Data modifiers         & -33 &   & x \\
Raw data file          & -34 &   & x \\
WVR parameters         & -35 & x & x \\
ALMA                   & -36 &   &\\
Instrument monitoring  & -37 & x & x \\
\hline
\end{tabular}
\caption{\CLIC sections, code and presence in hpb and IPB files. 
\label{tab:sections}}
\end{center}
\end{table}

\subsubsection{Data: dumps}
\label{sec:data-dumps}
At NOEMA, the correlator outputs a spectrum every second. Storing the
spectra every second would result in an huge file. In practice one can
average the spectrum to the limit that it does not produce
unacceptable beam smearing. Depending on the desired field of view,
observing frequency and interferometer configuration, the acceptable
averaging time ranges from a few seconds to a few minutes. Two kind of
averages are stored in the data. First, every second, a spectral
average is produced for each spectral window, resulting in a data
point every second. These are refered to as the continuum dumps. Then,
for each integration time, two spectra are produced for each spectral
window: one is the average of all the spectra obtained within the
integration window, the second is the same, but the spectra are
corrected for an atmospheric phase before integration. The correction
is nowadays coming from the water vapor radiometers, but for some
time, a correction using the 1mm receiver total power was also
used. These constitutes the line data.

%\begin{table}[htb]
%\begin{center}
%\begin{tabular}{|l|l|l|}
%\hline
%Dump &  \\
%\hline
%\end{tabular}
%\caption{\CLIC sections. \label{tab:dumps}}
%\end{center}
%\end{table}

An observation is hence usually composed of $n+2$ dumps, where $n$ is
the integration time in seconds. The first $n$ dumps contain only
continuuum data, the last two dumps contain both continuum and line
data, for the uncorrected and phase-corrected spectra respectively.

\subsubsection{The content of a dump}

A dump contains a data header (called DAPS, Data Associated
ParametersS, in \CLASS) and the data themselves. The data header is
used to store metadata that vary on the second basis (in opposition to
metadata stored in the observation header), such as the uv
coordinates, wvr counts, cable phase, etc.  It is important to note
that the data flags are stored in the data header.  The continuum data
are then stored along the axis $(spectral window, sideband,
baseline)$. Optionally (see above), line data are then stored along
the axis $(channel,sideband,baseline)$, with spectral windows all
stitched together in the $channel$ axis.

\subsubsection{Scans}
As can be seen, \CLIC understands data dumps and observations. The
concept of a scan is a collection of observations (possibly only one)
that share an observing intent and have the same scan number. For
example, a \texttt{CALIBRATE COLD} scan consists of three
observations: one on the cold load, one on the hot load and one on the
sky.

\subsection{IPB file}
At NOEMA, \texttt{RDI} produces IPB (which stands for
Interf\'erom\`etre du Plateau de Bure) files, one for each track. A
track consists of observations of one scientific subproject and last a
few hours (typically two to ten hours). The IPB file produced at NOEMA
contain sections (-2,-3,-21,-22,-23,-24,-25,-26,-30,-32,-35,-37) and
of course the raw data themselves.  Following the online atmospheric
calibration, data are on a temperature scale in $T_a^*$ (i.e. the
coupling factor considered in the atmospheric calibration is the
forward efficiency). Contrary to single dish, one does not need to
worry about the temperature scale as after amplitude calibration, the
visibilities are in Jy. These files are stored in the NOEMA archive
and can be accessed from IRAM computers using the \texttt{getproj}
command.

\subsection{hpb file}
However, the calibration is not done directly on the IPB files, which,
in principle, are never updated offline. Instead, \CLIC normally works
on header files, or hpb files. Before one starts the calibration, the
observation headers from the IPB file are copied into the hpb file,
and calibration sections are added in the header for later use. The
default is to add only the antenna calibration sections, because the
instrumental calibrations are usually done per antenna and the
baseline sections are much bigger in size. However, one can also
create an hpb file with baseline sections if needed (see also table
\ref{tab:sections}).

Besides the calibration sections, designed to store the
amplitude/phase gain solutions as a function of time or frequency, two
sections are mandatory in the hpb file: the data file section and the
data modifier section.  The data file section contains the name of the
IPB file from which the data were copied and the index number of that
observation in the IPB file. The data modifier section contain
parameters which modify the data after they are read from the IPB
file. This allows to correct the data for e.g. new baselines
measurements (see section \ref{sec:modify} for more details) without
modifying the IPB file. This section also contains a flags structure
similar to the one in the IPB file. \CLIC considers the logical
.or. of the two flags structure when an hpb file is opened.

After the calibration is complete, one just needs to save the hpb file
(and the uv table for analysis) and the IPB file can be deleted. If
there is a new need to redo part of the calibration or produce new uv
tables one just has to download again the IPB file. Note that any
access to the data requires the IPB file to be present on the file
system in any of the \texttt{IPB\_DATA1:} to \texttt{IPB\_DATA9}
logical names containing directories.

\subsection{Backward compatibility}
NOEMA, previously Plateau de Bure Interferometer (PdBI), has seen
continous improvements since its start, with more than five
generations of receivers and correlators along with the development of
atmospheric phase correction techniques. Given the nature of the \CLIC
data file, this was associated with many, sometimes small, changes of
the data format. \CLIC is designed so that its latest version can read
and process any version of the data format and map it in memory to the
latest data format.  Data of any date can hence be processed with the
latest \CLIC version. Note that writing the data is always done with
the latest data format (except if one updates an observation,
i.e. modify an existing .hpb file). In practice, the data format used
the section length (stored in the file) to see whether there were
extensions to a section.  With the introduction of \CLASSIC, a version
number is stored with the observation.

\newpage
\section{Commands}
In this section, I quickly introduce the commands, in a ordered way to
illustrate the \CLIC philosophy. For more complete information, the
reader is directed to the commands \HELP. The details of the data
calibration strategy are refered to the pipeline documentation.

\subsection{File}
\CLIC handles primarily two files:
\begin{itemize}
\item an input file
\item an output file
\end{itemize}
In case the input file is an hpb file, a third file, the raw data file
is also handled by \CLIC, to be used when data are to be read.

\subsubsection{FILE}
Command \texttt{FILE IN$|$OUT$|$BOTH} selects the file to be opened
for input/output. These commands also read the input or output file
index, and copy a subset of it in memory so that subsequent index
searches can proceed even more efficiently.  The logic is the same as
\CLASS. If the argument \texttt{BOTH} is used, the same file is opened
for input and output. Internally, \CLIC deals with two write mode for
files: \texttt{NEW} and \texttt{UPDATE}. The logic is that if the
input file and output file are different, the output file is in
\texttt{NEW} mode, whereas if the same file is opened for input and
output, the output file is in \texttt{UPDATE} mode.

The files opened for input and output are displayed with \texttt{SHOW
  FILE}.

\subsubsection{COPY}
Command \texttt{COPY} will copy the observations in the current index
(see section \ref{sec:index}) to the output file. Either the
observations header only or the full observation is copied according
to command options. In the same vein, the baseline and antenna
calibration sections can be included or not, as the bandpass
calibration section(s). This choice is to be made at the time of
\texttt{COPY}, since space has to be reserved for such sections which
are updated later in the calibration process.

\subsubsection{NEW\_DATA}
Command \texttt{NEW\_DATA} waits until new data is written to the
input file, then builds a new current index with all the new
data. Mainly useful at the observatory.

\newpage
\subsection{Index}
\label{sec:index}
In this section, I introduce the commands used to build and manipulate
a current index. \CLIC has a notion of three indexes: an input file
index, an output file index and a current index. Practically all \CLIC
commands work on the current index. A notable exception is
\texttt{FIND} that works on the input index for obvious reasons.

\subsubsection{FIND}
Command \texttt{FIND} will search the input file according to some
selection criteria, either specified as command options or set with
the \texttt{SET} command. The current index will be ordered according
to the sorting criteria, set with \texttt{SET SORT} command. Some
options allows the input file to be updated before the search or to
append the results to a previous current index.  \texttt{FIND} will
update the \SIC variables \texttt{FOUND} that gives the number of
observations found and \texttt{CX\_NUM[:]}, a list containing the
observation numbers of the current index. These can be used in
procedures, as is the case of the pipeline scripts.


\subsubsection{LIST}
Command \texttt{LIST} will list the content of the current
index. \texttt{LIST IN} or \texttt{LIST OUT} list the content of the
input or output index. Various options are available to list some
specific information or with some formatting.  Some options create
\SIC variables: \texttt{/SOURCE, /PROJECT, /TELESCOPE} and
\texttt{/OFFSET}. These variables contain the number of respective
items and their values (see the \texttt{HELP}).

\subsubsection{IGNORE}
Command \texttt{IGNORE} can be used to declare a list of observations
to be ignored in \texttt{FIND} commands until a new \texttt{FILE IN}
command has been typed.

\subsubsection{DROP}
Command \texttt{DROP} can be used to remove an observation from the
current index.

\newpage
\subsection{Observation}
In this section, I introduce the commands to access the observation.
\CLIC knows only one observation at a time, the current
observation. The other observations already read are kept in memory so
subsequent calls do not have to read again the corresponding
records. The limit to that scheme is the amount of memory. In case,
not enough memory is available to get an new observation, \CLIC will
remove the oldest observations in the cache until sufficient room is
found.

\subsubsection{GET}
The main command, if one wants to read an observation is
\texttt{GET}. One can either get a specific observation in the current
header by providing its observation number (number in the left column
of a \texttt{LIST} output), or go sequentially through the current
index with \texttt{GET NEXT}.  This will read the data and update the
current observation.

\subsubsection{HEADER}
Command \texttt{HEADER} displays in the terminal the content of the
observation header in a short format. Useful options are
\texttt{/PLOT} which will produce a plot of the spectral setup similar
to what is done in \ASTRO (see Fig. \ref{fig:header-plot}) and /LIST
that will list the spectral windows.

\begin{figure}[h]
\begin{center}
\includegraphics[width=1\textwidth]{header-plot.pdf}
\caption{Spectral setup plot produced by \texttt{HEADER \/PLOT} command. 
\label{fig:header-plot}}
\end{center}
\end{figure}

\subsubsection{VARIABLE}
Command \texttt{VARIABLE} creates various \SIC variables mapping the
observation and data headers (see corresponding \HELP for more
details). This allows use in procedures and also to modify some value
in the header or data file with the \texttt{MODIFY} command.  Be very
cautious in doing so, because some parameters have to be modified
consistently.

\newpage
\subsection{Plotting}
This section describes one of the basic functionnality of \CLIC, its
plotting capability, which is a 2 dimensional plot (abcissa and
ordinate).
\subsubsection{Plot selection}
Plotting in \CLIC is mainly controled by two criteria:
\begin{itemize}
\item \texttt{SET X} and \texttt{SET Y}, this allows you to choose whose
  quantity to plot. Several X or Y may be set.
\item data selection. The selected data is the intersection of the
  data meeting 3 selection criteria: 
  \begin{itemize}
  \item antenna or baseline, 
  \item sideband
  \item any of the baseband, IF, spectral window or polarisation criteria.
  \end{itemize}
\end{itemize} 
The plot can be either a time or spectral plot, i.e. X data have to be
only spectral-like or time-like quantities.  The user can provide
limits with \texttt{/LIMIT} option. Code \texttt{"*"} and \texttt{"="}
may be used.  The former means that the limits are automatically
adjusted to the data separately in each box, \texttt{"="} means that
common limits to all boxes are computed.

\subsubsection{Data selection}
\CLIC will draw one box for each x,y,antenna (or baseline), sideband,
baseband or IF or spectral window or polarisation. Each criteria can
be used to define a group or many groups of a value. For example
\texttt{SET BBAND 1 TO 2} will create one baseband group containg
basebands 1 and 2. \texttt{SET BBAND 1 2 TO 4 5 AND 6 AND 8} will
create 3 basebands groups containing baseband 1, basebands 2, 3, and
4, and basebands 5, 6 and 8.  A box will be created for each antenna
or baseline, sideband and each of the 3 groups.

\begin{figure}[h]
\begin{center}
\includegraphics[width=1\textwidth]{example-plot.pdf}
\caption{Example of \CLIC plot. The red rectangles are not part of the original
plot and were added to ease identification. 
\label{fig:example-plot}}
\end{center}
\end{figure}

\texttt{SET ANTENNA} or \texttt{SET BASELINE} may be used to specify
whether to plot per antenna or per baseline and can be used to plot
only a subset of what is in the file.  The corresponding antenna or
baseline is shown in the left box header of the box (red rectangle
number 1 in Fig. \ref{fig:example-plot})


\texttt{SET SIDEBAND} is used to control which sidebands to plot. The
corresponding sideband is shown in the right header of the box (red
rectangle number 3 in Fig.  \ref{fig:example-plot}).


\texttt{SET BBAND, IF, SPW, POL} are used to control how the data are
selected.  Selecting according to any of these criteria will also
select the corresponding control parameter. For example \texttt{SET BB
  1} will automatically select the corresponding IF, spectral windows
and polarisation. The corresponding value is shown in the central
header of the box (red rectangle number 2 in Fig.
\ref{fig:example-plot}).

Both metadata and data can be plotted. The metadata may have a
physical origin at a specific point in the signal transport or the
hardware, hence have a natural dependency on baseband, IF, or spectral
window, so will be selected according to that criteria, and averaged
if the group has more than one value.

For data, there are both "continuum" and "line" data in the IPB files
(see section \ref{sec:data-dumps}). Depending on how the spectral
windows are selected, continuum (\texttt{SET SPW C1}) or line
(\texttt{SET SPW L1}) will be used. In case the line data are used,
the selection of the phase-corrected spectral dump or uncorrected
spectral dump (both are stored in the IPB file) is done according to
the \texttt{SET PHASE ATM$|$NOATM} selection. The actual use of
phase-corrected data is also conditionned by the validity of the phase
correction, which is normally checked on the nearest calibrator. For
the phase-corrected to be selected on a baseline requires a valid
correction for both antennas.


\subsubsection{Averaging}
The data are averaged in time according to the \texttt{SET AVERAGING}
command. It is possible to average by scan or time. In addition, this
command also controls if the data are averaged as complex numbers
(\texttt{VECTOR}) or if amplitude and phase are averaged separately
(\texttt{SCALAR}). Be cautious about the amplitude bias especially in
spectral plots where the much lower bandwidth of channels can bring
the signal to noise to low values even on calibrators (see section
\ref{sec:statistics} and \ref{sec:averaging}).

In addition to and after the time averaging, the data in the plot may
be averaged in bins of the x coordinates with the command \texttt{SET
  BINNING}. It is useful to average data in spectral mode. Any
\texttt{SET X} command resets the binning.

\subsubsection{Calibration}
Calibration may be applied on the fly before plotting. This is
controlled by the following commands (see respective help for more
information):
\begin{itemize}
\item \texttt{SET PHASE RELATIVE$|$ABSOLUTE}
\item \texttt{SET AMPLITUDE RELATIVE$|$ABSOLUTE}: see also Table \ref{tab:amp}
\item \texttt{SET RF ON$|$OFF}
\end{itemize}
The control parameters for calibration are displayed in the left
header of the plot (red rectangle number 4 in
Fig. \ref{fig:example-plot}).  Each calibration solution can be
antenna or baseline based (provided a solution was computed and stored
beforehand, see section \ref{sec:solving} and \ref{sec:storing}).

\begin{table}
\begin{center}
\begin{tabular}{|c|c|c|c|}
\hline
{\em Unit $\backslash$ Mode}    &{\tt ABSOLUTE} &{\tt RELATIVE} &{\tt SCALED}\\
\hline
{\tt KELVIN} & 1 & $A_{ij}(t)$   & $1/S$\\
 {\tt JANSKY} & $\sqrt{J_i*J_j}$ & $A_{ij}(t)/  \sqrt{J_i*J_j}$  &$1/S$\\
\hline
\end{tabular}
\caption{Scaling of amplitude depending on unit/mode. $J_i$ is the antenna 
efficiency in (Jy/K), $A_{ij}$ is the amplitude calibration factor (in K/Jy)
and $S$ the source flux (in Jy)
\label{tab:amp}}
\end{center}
\end{table}

\subsubsection{PLOT}
The plot itself is done with command \texttt{PLOT}. Many options are
available and the reader is invited to refer to the \texttt{HELP} for
more details. One handy option is the \texttt{/BOX nx ny} that tells
\CLIC to use nx times ny boxes. One can provide code "*" for one of
the arguments so that only the number of rows or columns is set.

Another interesting feature of plot is that the content of the plot
buffer is available in \SIC variables: \texttt{X\_DATA, Y\_DATA} and
\texttt{W\_DATA} containing respectively the X values, the Y values
and the weights associated to each data point. These values can be
used in procedures to do your own computations.

Two options further enrich the possibilities: 
\begin{itemize}
\item \texttt{PLOT /NODRAW} will only update the plot buffers but will
  not produce a plot
\item \texttt{PLOT /SAME} will not read data but use the plot buffers
  to do a plot
\end{itemize}
Hence one can modify the content of the plot buffers before plotting
them.

Another useful option is \texttt{/NOFRAME}, that allow to not reset
the plot before doing it, which in effect allows to overplot some
another set of data on top of a previous plot (with another color if
wanted).

\subsubsection{CURSOR}
Command \texttt{CURSOR} calls an interactive cursor, mainly aimed at
pinpointing data points and measuring their coordinates. It can also
be used for editing although this is not the recommanded way of
editing data. See corresponding \HELP.

\subsubsection{BLANK}
Command \texttt{BLANK} can be used to blank some channels in a given
spectral window.  Corresponding data will not be displayed
(\texttt{Y\_DATA} value equal to the blanking value, \texttt{W\_DATA}
equal to 0).

\subsubsection{Flags}



\newpage
\subsection{Solving}
\label{sec:solving}
In this section, I describe or shortly introduce the SOLVE engines
that are available in \CLIC. I start with the command used in any
pipeline reduction, then introduce the commands used for real-time
observations or observatory measurements.

\subsubsection{SOLVE RF}
\label{sec:solve-rf}
Command \texttt{SOLVE RF} will solve for the spectral variation of the
gain.  As for the temporal variations, one can either solve for the
baseline gain $B,_{ijk}$, or the antenna gains, $b_{ik}$, with
$B_{ijk} = b_{ik}b_{jk}^*$ according to the mode selected with
\texttt{SET RF ANTENNA$|$BASELINE}.

To get sufficient signal to noise in each channel, specific
observations of a bright source are normally carried out in each
project.  There are basically two modes for solving RF bandpass. One
can either fit a Chebishev polynomial to the data (low signal to noise
ratio) or use the data (or a smoothed version of the data) as the RF
solution. In the latter case, the actual solution is linearly
interpolated between data points. The mode used is controlled by the
command \texttt{SET RF FREQUENCY$|$SPECTRUM.}

RF bandpass calibration is done baseband per baseband (see section
\ref{ref:nomenclature}). It is hence necessary to select a proper
baseband with \texttt{SET BBAND} command before solving for the
RF. The solution is computed in an intermediate frequency space adapted to
each generation of signal transport, selected with \texttt{SET X
  I\_F}.

\subsubsection{SOLVE FLUX}
Command \texttt{SOLVE FLUX} is used to derive the flux of the phase
calibrators from a known reference (remember quasar fluxes are
variable in time). At NOEMA we have two main flux references: MWC349
and LkH$\alpha$101. Anyhow, the command is generic, as long as the
flux of one of the sources is provided.

Visibilities are expressed on an antenna basis and an antenna
efficiency is computed for each antenna by dividing the flux of the
reference by the observed antenna temperature. Then a selection of the
best antennas (i.e. the ones with the best efficiencies) are used to
compute the fluxes of the other sources in the current index using the
antenna efficiencies.

This of course will only work if the actual antenna efficiencies are
constant in the considered time range (either being good the whole
time or, which is more difficult to ensure, being equally bad on all
sources).

\subsubsection{SOLVE AMP PHASE}
Commands \texttt{SOLVE AMPLITUDE} and \texttt{SOLVE PHASE} are used to
derive the temporal variation of the gain through its amplitude and
phase separately. One can either solve for the baseline gains
$G_{ij}(t)$ or the antenna gains $g_{i}(t)$ (with $G_{ij}(t)=
g_i(t)g_j^*(t)$) according to the mode selected with \texttt{SET
  AMPLITUDE ANTENNA$|$BASELINE}.

In antenna mode, the averaged phase and amplitude closures and their
standard deviation are computed, that allows to check for
baseline-based errors, mostly in the form of amplitude
decorrelation. The solutions are found by fitting directly the
baseline-based quantities with the appropriate antenna-based
combination (product of amplitudes, difference of phases).

Amplitude is mainly affected by variations of the antenna gain
(pointing errors, defocussing, thermal and/or gravitational
deformations), receiver gain and variation in the atmospheric
absorption not accounted for in the atmospheric calibration (see
sec. \ref{sec:ATM}) in addition to decorrelation due to phase
noise. For amplitude calibration, one should use \texttt{SET AMPLITUDE
  SCALED} prior to the \texttt{SOLVE} command so that the amplitude,
which is on a temperature scale, is divided by the source flux. If the
flux calibration is correct, the different sources should have the
same values in K/Jy, which is actually the inverse of the antenna
efficiency.

\begin{figure}[h]
\begin{center}
\includegraphics[width=1\textwidth]{solve-amp.pdf}
\caption{Example of \texttt{SOLVE AMPLITUDE /PLOT}. 
\label{fig:solve-amp}}
\end{center}
\end{figure}

Phase is affected by delay errors, in the form of baselines or time
errors, atmospheric delays or drift in the receiver delay or phase
(local oscillators).  For the phase in particular, the atmospheric
fluctuations introduce changes on a timescale that is usually smaller
than the calibration period used for NOEMA. In order not to alias this
fast varying component in the calibration, which would increase the
final phase noise, it is important to keep in mind that the solution
does not need to pass through the data points.

\begin{figure}[h]
\begin{center}
\includegraphics[width=1\textwidth]{solve-phase.pdf}
\caption{Example of SOLVE PHASE /PLOT. 
\label{fig:solve-phase}}
\end{center}
\end{figure}


Two mathematical functions can be fitted to the data: cubic splines
(the knots spacing is controlled by the \texttt{SET STEP} command),
which the default, or polynomials through the \texttt{/POLYNOM} degree
option. It is possible to introduce breaks in the fuction or its
derivatives with the \texttt{/BREAK} option.

Amplitude and phase calibration is done per IF (see section
\ref{ref:nomenclature}) by the combination of \texttt{SET
  POLARISATION} and \texttt{SET BAND } commands. In case of a
polarized calibrator, the measured amplitude is:
\begin{equation}
\widetilde{A_{ij}}= a_i a_j \left(I+ Q\cos(2\chi)+U\sin(2\chi)\right)
\end{equation}
where $\chi$ is the parallactic angle and $I,Q,U,V$ are the Stokes
parameters.  The (implicit) assumption of constant flux density for
the calibrator breaks down if the linear polarisation fraction
$P=\frac{\sqrt{Q^2+U^2}}{I}$ is more a few percents. In that case, one
should use the average of both polarisation to perform the amplitude
calibration (\texttt{SET POLARISATION BOTH}).

\subsubsection{SOLVE DELAY}
Command \texttt{SOLVE DELAY} is used mainly at the observatory. After
plotting the phase of a calibrator as a function of IF1 frequency, one
fits a residual delay $\Delta \tau$ to the data, following:
\begin{equation}
\phi = \pm 2\pi\Delta \tau \nu_{IF1} 
\end{equation}
The $\pm$ sign depend on the sign of the baseband conversion (LSB or
USB). It is recommended to use an antenna-like plot to get better
signal to noise.

\subsubsection{SOLVE GAIN}
Command \texttt{SOLVE GAIN} is used mainly at the observatory. With
Walsh $\frac{\pi}{2}$ switching on the LO1 and demodulation in the
backend, the sidebands which are superposed after the LO1 conversion
can be separated. This allows the receiver sideband rejection,
$g_{im}$, to be measured by computing the ratio of the image sideband
over the signal sideband amplitudes. This is done on the uncalibrated
data by dividing each sideband by its $T_{sys}$ and correcting for the
possible difference in atmospheric opacity between the sidebands.

This command also derives the mean baseband phases that are fed back
to the system so that further online averages do not suffer
decorrelation.

\subsubsection{SOLVE POINT}
Command \texttt{SOLVE POINT} is used to reduce pointing scans by
fitting a gaussian to the antenna-based visibilities. For that reason,
it only works if three antennas or more are present. Else, one antenna
has to be fixed, and the baseline visibility is fitted.

\subsubsection{SOLVE TOTAL}
Command \texttt{SOLVE TOTAL} and \texttt{SOLVE POINTING /TOTAL} are
used to reduce pointing scans by fitting a gaussian to the total
power. This requires a strong source.

\subsubsection{SOLVE FIVE}
Command \texttt{SOLVE FIVE} is used to reduce a special pointing
pattern where five points in the shape of a cross are observed and a
2D gaussian fitting is done.

\subsubsection{SOLVE FOCUS}
Command \texttt{SOLVE FOCUS} will reduce focusing scans by fitting a
gaussian to the offset along the focus in the Z-axis (along the
optical axis).

\subsubsection{SOLVE SKYDIP}
Skydips are special scans where the antenna is moved in elevation at a
fixed azimuth. For each elevation (regularly spaced in airmass), total
power from the sky, $P_{sky}$, and the hot load, $P_{hot}$, are
recorded.  Since one knows the effective temperature of the hot load,
$T_{hot}$, one can derive either the receiver temperature, $T_{rec}$,
or the forward efficiency ,$\eta_f$, via:
\begin{equation}
(T_{hot}+T_{rec})\times \frac{P_{sky}}{P_{hot}} - T_{rec} = \eta_f \times 
Airmass \times \tau_{zenith} + (1-\eta_f)\times T_{ground}
\end{equation}

\subsubsection{SOLVE BASE ; RESIDUAL BASE}
The geometrical delay is computed through:
\begin{equation}
\tau_g = \frac{\bf{b_{\cdot}s}}{c}
\end{equation}
where ${\bf b} = (X_{ij},Y_{ij},Z_{ij})$ and ${\bf s}=(\cos h \cos
\delta, -\sin h\cos\delta,\sin\delta)$ are the baseline and source
vectors respectively in a referential .

The resulting phase is:
\begin{eqnarray}
\phi_{ijk} &=& 2\pi \tau_g \nu_{RF}  \nonumber \\ 
          &=& \frac{2\pi}{\lambda}{\bf b_{\cdot}s} \nonumber \\ 
          &=& \frac{2\pi}{\lambda_k}\left( X_{ij} \cos h\cos \delta -
Y_{ij}\sin h\cos\delta + Z_{ij}\sin\delta\right)
\end{eqnarray}
Using sources with known position distributed in hour angle and
declination, a least square fit to the observed phases gives is used
to derive the antenna position errors.

An additional term can be fitted corresponding to the offset between
the azimuth and elevation axis of each antenna. It has a $\cos(El)$
dependance (in that case, data must be plotted as a function of
elevation also in addition to the hour angle and declination of the
"normal" baseline measurements).

Command \texttt{RESIDUAL} baseline will plot the residual phases
resulting from the last \texttt{SOLVE BASELINE} command.

\subsubsection{SOLVE HOLOGRAPHY}
Holography is powerful way of caracterizing an antenna. It builds on
the fact that there exists a Fourier transform relationship between
the far-field antenna beam (including its amplitude and
\underline{phase}) and the (complex) aperture field distribution. A
modified Fourier relationship can be derived in the near-field case
(far field is $> 2d^2/\lambda$, or $~130$km for a 15-m antenna at a
wavelength of 3.4mm). The interested readers will find in Baars et
al. 2007 a lot more of details. \CLIC can handle both near-field
(Fresnel) and far-field (Fraunhofer) measurements.

In essence, a source is scanned with one or more antennas while one or
more antennas are pointing at the source. The data are gridded and
Fast Fourier Transformed to an aperture plane. The aperture plane is
then analyzed in terms of amplitude and phase (left and right plots in
Fig. \ref{fig:solve-holo})

\begin{figure}[h]
\begin{center}
\includegraphics[width=1\textwidth]{holo-example.pdf}
\caption{Example of SOLVE HOLOGRAPHY. 
\label{fig:solve-holo}}
\end{center}
\end{figure}

The amplitude map is normalized and transformed in decibels (dB), so
that it peaks at 0dB. In radio-astronomy, the aperture is often
tapered to reduce the beam sidelobes. \CLIC does a fit of this
illumination with a 2-d gaussian (2-d parabola in the dB maps) to
measure the actual taper and center of illumation. The values are
given in the first line of the header.

The phase map is more interesting in that it gives the deviations
w.r.t. a perfect surface and can be used to derive corrections to be
applied to the individual panels of the parabola. The advantage is
also that it can be measured, and panel corrections derived, at any
antenna elevation.

A least squares fit to the phase map of the following terms is
performed, where $\left(\xi,\eta\right)$ are aperture coordinates,
$r=\sqrt{\xi^2+\eta^2}$ is the radius coordinate from the center of
the aperture, $f$ is the focal length of the antenna:
\begin{eqnarray}
\phi_1 (\xi,\eta) &=& a \\
\phi_2 (\xi,\eta) &=& b\xi \\
\phi_3 (\xi,\eta) &=& c\eta \\
\phi_4 (\xi,\eta) &=& \delta z \left(1-\frac{1-\frac{r^2}{4f^2}}{\sqrt{\frac{r^2}{f^2}+(1- \frac{r^2}{4f^2})^2}}\right) \\
\phi_5 (\xi,\eta) &=& \delta x\frac{\xi}{f}\left(1-\frac{1}{\sqrt{\frac{r^2}{f^2}
+\left(1-\frac{r^2}{4f^2}\right)^2}}\right) \\
\phi_6 (\xi,\eta) &=& \delta y\frac{\eta}{f}\left(1-\frac{1}{\sqrt{\frac{r^2}{f^2}
+\left(1-\frac{r^2}{4f^2}\right)^2}}\right) \\
\phi_7 (\xi,\eta) &=& a_{+}\frac{\xi^2-\eta^2}{2f^2} \\
\phi_8 (\xi,\eta) &=& a_{\times}\frac{2\xi \eta}{2f^2} \\
\end{eqnarray}
$a$ is a constant phase offset, $b$ and $c$ account for pointing
offset that would be constant during the observations, $\delta x$,
$\delta y$ and $\delta z$ correspond to focus offsets in the x
(horizontal), y (vertical) or z (along optical axis)
directions. $a_{+}$ and $a_{\times}$ are the antenna astigmatism along
the horizontal or vertical axis, $a_{\times}$ along directions rotated
by 45$^\circ$. The corresponding astigmatism angle $\psi_a$ is:
\begin{equation}
\psi_a = \frac{\arctan{(a_{\times},a_{+})}}{2}
\end{equation}

In addition, holographies provide an estimate of the actual antenna
surface, one can compute the aperture and illumination efficiencies
$\eta_A$ and $\eta_I$:

\begin{eqnarray}
\eta_A &=& \frac{\lvert \int_A A(\xi,\eta)\exp(i \phi (\xi,\eta))d\xi d\eta
\rvert^2}{\pi r^2\int_A A(\xi,\eta)^2d\xi d\eta} \\ \nonumber \\
\eta_I &=& \frac{\lvert \int_A A(\xi,\eta)d\xi d\eta
\rvert^2}{\pi r^2\int_A A(\xi,\eta)^2d\xi d\eta} 
\end{eqnarray}

and the antenna efficiency:
\begin{equation}
J=\frac{2k}{\eta_A \pi r^2}
\end{equation}

One can also fit panel displacements. According to the panel mode
selected, translation only, tilt in both radial and tangential
directions, torsion mode and a fit of the inner fifth support point
can be fitted.

Finally, one can produce either a phase map or a surface error
map. The latter is linked to the former through:

\begin{equation}
\delta p = \delta \phi \times \frac{\lambda}{4\pi}\times\sqrt{1+\frac{r^2}
{4f^2}}
\end{equation}
The term to the right corresponds to the projection factor between the
incident or reflected ray and the normal to the surface.

\newpage
\subsection{Storing}
In this section, I describe the command \texttt{STORE} which is used
to store various parameters in the header file. These commands are
relatively straightforward to use, but there are a few
technicalities. Remember that all \texttt{STORE} commands are working
on the current index, so be sure it includes the target observations
before storing calibrations.

\label{sec:storing}
\subsubsection{Correction}
As explained in section \ref{sec:data-dumps}, there are usually two
spectra per observation: one which is the simple average of all the
dumps within the observations, one for which each individual spectra
is corrected by a phase $\phi_j-\phi_i$, where $\phi_i$ is the
correction for atmospheric phase derived from the Water Vapor
Radiometer (WVR) for antenna $i$, prior to the averaging. For older
data (before 2006), the 1mm total power data have also been used to
monitor atmospheric phases (see \texttt{HELP MONITOR}).

\texttt{STORE CORRECTION} can be used to store in the observation
header whether the atmospheric phase correction is valid or
not. Besides the \texttt{GOOD} or \texttt{BAD} keywords, keyword
\texttt{AUTO} is the one that is recommended to be used. It will
compare the amplitude of the phase corrected or uncorrected data (with
a bias toward uncorrected data) on calibrators, and then automatically
decide which should be used for the science scans according to the
closest in time phase calibrator correction validity.

\subsubsection{Bandpass} 
As explained in section \ref{sec:solve-rf}, there are two ways to
perform a bandpass calibration, a polynomial fit to the calibrator's
data or a direct use of the calibrator's data. Accordingly, there are
two ways to store an RF bandpass solution in a hpb file.

The historic way is to store the polynomial coefficients in a
dedicated section of the hpb file. Because of the way files are
handled in \CLIC (see section \ref{sec:file}), this is only possible
if room for the appropriate section (-27 or -28, see Table
\ref{tab:sections}) was reserved at the time of the hpb creation.

When the data or a smoothed version of it is used, one has to use
another solution, which is to store the full bandpass solution
(interpolated for all data points, including the high spectral
resolution spectral windows if there are any). This is done through
option \texttt{STORE RF /SCAN}. This solution can also be used to
store a bandpass solution obtained through polynomial fitting. The
advantage of that solution is that it can be useful to create an hpb
file without the bandpass calibration section(s), which is much
smaller in size.

\subsubsection{Flux and spectral index}
Command \texttt{STORE FLUX} will store the fluxes determined by the
last \texttt{SOLVE FLUX} command in the file, which is a prerequisite
of the amplitude calibration.  Spectral index is for now stored with
command \texttt{MODIFY SPIDX} so it has to work on an source per
source index iteratively.

\subsubsection{Amplitude and phase}
This is a straightforward operation that will copy the cubic spline or
polynomial coefficients from each IF from memory to the relevant
calibration section (-29 or -31, see Table \ref{tab:sections}) of the
hpb file (provided it was initialized at hpb creation).  One can
override this default by forcing it to use a given sideband or
polarisation (or average of sidebands or polarisations) to calibrate
both polars/sidebands.

\subsubsection{Quality}
\texttt{STORE QUALITY} can be used to store a quality in an
observation. In practice, this functionnality is seldomly used except
for a \texttt{STORE QUALITY 9}, which will prevent further
\texttt{FIND} command to find this observation.

\subsubsection{Flag}
\texttt{STORE FLAG} is the command to be used if one wants to flag
data. Many keywords are available and flags can be stored for antennas
or baselines. Two basic kinds of flag exist: the spectral flags, which
will flag individual spectral windows and the non-spectral flags, that
flag all of the data.

Flags are available at the dump level in the IPB, as they are recorded
on the second basis by the online system. In the hpb file, they are
stored in the data modified section, and only one value per
observation is kept. In the case when one uses an hpb file, the actual
flag will be a logical \texttt{.or.} of the IPB and hpb flags.

An option exists to store the flags only in memory if they are to be
reset after some operation, as it is much more efficient.  Command
\texttt{MASK} allows to mask a given flag, while command \texttt{MARK}
can be used to set a given flag (for all observations, contrary to the
\texttt{STORE FLAG /MEMORY} option that works on the current index).


\newpage
\subsection{Modify}
\label{sec:modify}
The \texttt{MODIFY} command is used to modify the data or metadata.
The behaviour is not exactly the same for IPB and hpb files. For IPB
files, the data are modified. For hpb files, data modifiers are stored
in the relevant section of the header and applied on the fly when the
data are read from the IPB file.


\subsubsection{Geometry}
The geometric delay is computed in the following way:
\begin{eqnarray}
\tau_g &=& \frac{{\bf b_{\cdot} s}}{c} \nonumber \\
       &=&\frac{1}{c}\left(X_{ij}\cos h\cos\delta - Y_{ij}\sin h\cos \delta 
+Z_{ij}\sin \delta\right) \label{eq:taug}
\end{eqnarray}
This geometric delay is computed in real-time and applied in the
correlator and on the LO for fringe tracking. However, it may be the
case (and is quite usual) that after the observation, a better
baseline model was determined.  Another more unusual case would be new
coordinates were determined for an observed source or that the time
reference used at the observatory was offset w.r.t. UT1 time. All
these can be corrected after the observations and the phases
recomputed with commands \texttt{MODIFY ANTENNA, MODIFY BASELINE,
  MODIFY POSITION} or \texttt{MODIFY TIME}. In all cases, equation
\ref{eq:taug} is used with the updated parameters provided by the
user.

\subsubsection{Phases}
There are another set of commands that modify the phases but do not
take their origin in the computation of the geometrical delay. A good
example is the \texttt{MODIFY DELAY} command that will apply a new
(constant) delay to the current index. Some more specific commands,
for example \texttt{MODIFY CABLE}, will correct faulty cable phase
correction.

\subsubsection{Frequency and velocity scales}
\label{sec:freq-scale}
NOEMA operates with Doppler tracking. When the user provides a setup,
with a given reference rest frequency, $F_{rest}^{ref}$, and a source
velocity, $v$, within some referential (usually LSR referential) on
one side and an IF frequency, $F_{IF}^{ref}$, at which the rest
frequency should appear on the other side. The Doppler tracking
consists in modifying the LO1 frequency so that the two frequencies
stay aligned to within one tenth of the smallest channel spacing of
the spectral setup being observed.

The rest, $F_{rest}$, and topocentric frequencies, $F_{topo}$, are
related through:
\begin{equation}
F_{topo}  = F_{rest} \left(1+\delta-\frac{v}{c} \right) \label{eq:freq-doppler}
\end{equation}
where $\delta$ is the Doppler factor between the chosen referential
and the topocentric referential, and $v$ is the source systemic
velocity within this referential.

Since $F_{IF} = \pm(F_{topo}-F_{LO1})$ (+ for an USB conversion, - for
a LSB conversion), we have:
\begin{equation}
F_{IF} = \pm \left(F_{rest} \left(1+\delta-\frac{v}{c}\right) - F_{LO1}\right)
\end{equation}
so by setting
\begin{equation}
F_{LO1} = F_{rest}^{ref}\left(1+\delta-\frac{v}{c}\right)\mp F_{IF}^{ref}
\end{equation}
we ensure the Doppler tracking as defined above.

\begin{table}
\begin{center}
\begin{tabular}{|l|l|l|l|l|}
\hline
Parameter                & Freq & Frame & Notation & \SIC variable \\
\hline
Reference rest frequency & RF   & Rest  & $F_{rest}^{ref}$ & FREQUENCY  \\
Sideband                 &      &       &                  & ISB        \\
Reference IF frequency   & IF   & Topo  & $F_{IF}^{ref}$   & FIF1       \\
LO1 frequency            &      & Topo  & $F_{LO1}$        & FLO1       \\
Source referential       &      &       &                  & TYPE\_VEL  \\
Source velocity          &      & Ref.  & $v$              & VELOCITY   \\
Doppler factor           &      &       & $\delta$         & DOPPLER    \\
\hline
\end{tabular}
\caption{Parameters for receiver tuning. For the source referential TYPE\_VEL, 1 means LSR, 2 heliocentric, 3 geocentric, 4 no doppler tracking, 5 unknown.
\label{tab:line}}
\end{center}
\end{table}



The frequency value of channel $k$ from spectral window $j$ are
defined as:
\begin{eqnarray}
F_{IF}(j,k)   &=& \left(k-i_{IF}(j)\right)*\delta F_{topo}+F_{IF}^0(j) \\
F_{rest}(j,k) &=& \left(k-i_{rest}(j)\right)*\delta F_{rest}+F_{rest}^0(j)
\end{eqnarray}

where $\delta F_{topo}$ is the channel spacing of the backend in the
topocentric frame and $\delta F_{topo} = \delta
F_{rest}\left(1+\delta- \frac{v}{c}\right)$

\begin{table}
\begin{center}
\begin{tabular}{|l|l|l|l|l|}
\hline
Parameter             & Freq & Frame & Notation & \SIC variable      \\
\hline
Number of channels    &      &       & $n_{ch}$     & N\_CHANNELS    \\
Channel number of (2) &      &       & $i_{IF}$     & REF\_NUM       \\
Center frequency      &  IF  & Topo  & $F_{IF}^0$   & LFCEN          \\
Reference channel     &      &       & $i_{ref}$    &  REF\_CHANNEL  \\ 
Rest frequency        & RF   & Rest  & $F_{rest}^0$ & F\_OFFSET      \\
Frequency spacing     &      & Topo  & $\delta F_{rest}$ & LFRES     \\
Frequency spacing     &      & Rest  & $\delta F_{topo}$ & F\_RESOLUTION  \\
\hline
\end{tabular}
\caption{Parameters for spectral windows and corresponding \SIC variables
\label{tab:spw}}
\end{center}
\end{table}

\subsubsection{Custom modifications}
It is possible to modify any header or data header data using generic
command \texttt{MODIFY HEADER, MODIFY DATA} and \texttt{MODIFY RECORD}
that allow us to modify parameters of the observation header, spectral
dumps or temporal dumps respectively.

Caution is to be exercised when using these commands as changes must
be done coherently.


\newpage
\subsection{Atmosphere}
Various commands relates to the atmosphere: the "atmospheric"
calibration or chopper-wheel method, that is used to put the
visibilities on a $T_a^*$ scale (see \ref{sec:noise-temp}), the water
vapor radiometer (WVR) calibration that will derive atmospheric phase
correction, and the total power phase correction that was used for the
PdBI before the use of WVRs.

\subsubsection{Atmosphere}
\label{sec:ATM}
\paragraph{Measurements}
Assume we have a system that measures the total power $P$ received at
the output terminal and that input power is measured in temperature
scale $T$ (see eq.  \ref{eq:pt}) with an unknown gain ($K_{p}$)
i.e. we have:
\begin{equation}
P = K_{p} T
\end{equation}
Similarly, let's assume that the autocorrelations are on a power scale
with a gain $K_{c}$, different from $K_{p}$, but the same for auto-
and cross- correlations (this point is checked with measurement on a
noise source, which is fully correlated for different antennas, so
that autocorrelations match the crosscorrelation amplitudes). Then we
have:
\begin{equation}
\widetilde{A_{ij}} = K_p T_{ij}
\end{equation} 
where $T_{ij}$ is the source power at the input port.

In using the chopper wheel method, we place loads with known effective
temperature in front of the receiver. For NOEMA, one uses a hot load
made of absorber which is at the temperature of the receiver cabin and
a cold load located inside the cryostat on the 15K stage ; regular
observations are made on the sky:
\begin{eqnarray}
P_{hot}  &=& K_p \left(T_{hot}+T_{rec}\right) \label{eq:phot} \\
P_{cold} &=& K_p \left(T_{cold}+T_{rec}\right)  \label{eq:pcold} \\
P_{sky}  &=& K_p \left(T_{sky}+T_{spill}+T_{rec} + T_a \right) \\
C_{ii}   &=& K_c \left(T_{sky}+T_{spill}+T_{rec} + T_a \right) \\
|C_{ij}|  &=& K_c T_a
\end{eqnarray}
The difference between the cross-correlation amplitudes $|C_{ij}|$ and
the autocorrelations $C_{ii}$ is that the "noise"
(i.e. $T_{sky}+T_{spill}+T_{rec}$) is uncorrelated between different
antennas and vanishes from the cross-products.

\paragraph{Receiver temperature} By combining eq. \ref{eq:phot} and
\ref{eq:pcold}, we can derive the receiver temperature $T_{rec}$:

\begin{equation}
T_{rec} = \frac{P_{cold}T_{hot}-P_{hot}{T_{cold}}}{P_{hot}-P_{cold}}
\end{equation}
or equivalently with the autocorrelations.

\paragraph{System temperature}
We start by deriving the sky temperature with:
\begin{equation}
T_{sky} = \left(T_{hot}+T_{rec}\right)\frac{P_{sky}}{P_{hot}}-T_{rec}-T_{spill}
\end{equation}
Estimating $T_{spill}$ requires knowledge of the forward efficiency
$\eta_f$.  It is also used to compute the sky emission $T_{emi}$:
\begin{equation}
T_{emi} = \frac{T_{sky}}{\eta_f}
\end{equation}
The water vapor content variable of an atmospheric model predicting
sky brightness ($T_{emi,s}^{atm}$ and $T_{emi,i}^{atm}$) and opacity
($\tau_{atm,s}$ and $\tau_{atm,i}$) in the signal and image sideband
is then varied in order to minimize the following quantity:
\begin{equation}
T_{emi} - \frac{T_{emi,s}^{atm}+g_{im}T_{emi,i}^{atm}}{1+g_{im}}
\end{equation}
The system temperatures for the signal $T_{sys,s}$ and image
$T_{sys,i}$ sidebands are then computed as follows:
\begin{eqnarray}
T_{sys,s} = \frac{e^{\tau_s}\left(T_{hot}-T_{sky}-T_{spill}\right)(1+g_{im})}{\eta_f}\frac{P_{sky}}{P_{hot}-P_{sky}} \\
T_{sys,i} = \frac{e^{\tau_i}\left(T_{hot}-T_{sky}-T_{spill}\right)(1+\frac{1}{g_{im}})}{\eta_f}\frac{P_{sky}}{P_{hot}-P_{sky}} 
\end{eqnarray}
where:
\begin{equation}
T_{sky}+T_{spill} = (1-\eta_f) \left(\frac{T_{emi,s}^{atm}+g_{im}T_{emi,i}^{atm}}{1+g_{im}} \right)+\eta_f T_{ground}
\end{equation}

\paragraph{Scaling of data}
The visibilities $C_{ijk}$ out of the correlator are normalized in the
following way to get observed visibilities on a temperature scale
$T_{ijk}$:
\begin{equation}
T_{ijk} = \frac{C_{ijk}}{\sqrt{C_{iik}C_{jjk}}} \sqrt{T_{sys}(i)T_{sys}(j)}
\end{equation}
In the left side of the right term, cross-correlations are normalized
to the total power measured in autocorrelations (and the dependency on
$K_c$ disappears), so that a rate of correlation is measured. The
right side corresponds to a conversion to $T_a^*$ scale by
multiplication by the baseline effective system temperature.

\paragraph{Command ATMOSPHERE}
This atmospheric calibration is normally applied online by the RDI
program.  However in some cases, one may want to update the
atmospheric calibration because of an online problem at the time of
the observations (i.e. a bug, a wrong value for an input
parameter). Command \texttt{ATMOSPHERE} allows us to do this and will
process all of the scans in the current index.

\subsubsection{WVR} 
The water vapor radiometer operates around the water line at 22
GHz. By monitoring the amount of water vapor along the line of sight
of each antenna, one can derive a corresponding atmospheric phase at
the frequency of interest and correct the individual dumps from the
correlator by this phase before averaging, hence mitigating
atmospheric decorrelation (see also section \ref{sec:data-dumps}).

This requires calibration of the data coming from the radiometer. A
number of modes are available to derive the receiver gain, $T_{cal}$,
and noise, $T_{rec}$, either from tabulated laboratory measurements or
observations of loads of known brightness. The classical way of doing
this for NOEMA is to use $T_{rec}$ derived from skydip measurements
which are carried out regularly. The use of a hot load then allows us
to derive the gain $T_{cal}$. With $T_{rec}$ and $T_{cal}$, the sky
antenna temperature, $T_i$, for the different channels can be
computed.

The first generation of WVR used at NOEMA had 3 channels (centered at
frequencies $\nu_1$, $\nu_2$ and $\nu_3$), that were combined in a way
that removed any $\nu^2$ contribution (hence cloud emission):
\begin{equation}
T_{triple} = \frac{1-\left(\frac{\nu_1}{\nu_2}\right)^2}{1-\left(\frac{\nu_2}{\nu_3}\right)^2}\left(T_2-T_3\left(\frac{\nu_2}{\nu_3}\right)^2\right)-\left(T_1-T_2\left(\frac{\nu_1}{\nu_2}\right)^2\right)
\end{equation}
As for the astronomical receiver calibration, the water vapor content
of an atmospheric model is varied until a good match with the measured
triple temperature is found. The derivative of the optical pathlength
to the channel temperatures is then computed, in order that a scaling
factor can be applied to the radiometer counts, so that their
variation track atmospheric phases.

The offline version will do the same, but the spectral averaging was
done online. So it is useful only to check calibration or do use an
updated calibration for continuum records.


\subsubsection{Monitor}
The first atmospheric phase correction scheme was based on the total
power measured in the 1mm band (when dual-frequency observing was
available).  The calibration scans are processed similarly to the
regular astronomical calibration (see section \ref{sec:ATM}), then the
derivative of the path length over the emission temperature of the
atmosphere at the atmospheric monitoring frequency is computed.

It is possible to fit a straight line within a time $\delta t$ to
remove drift of non-atmospheric origin (such as receiver gain
variations), before computing the corrections.

As for the WVR, this correction was applied online and can only be
recomputed for the continuum units.

\newpage
\subsection{($u,v$) tables}
The final goal of the calibration is to produce a set of calibrated
visibilities that can be imaged or analysed in the ($u,v$) plane. In
\GILDAS, we write .uvt tables for that purpose.  The .uvt file is a
flavor of the more generic gildas data format (GDF).

\subsubsection{GDF v1}
The original GDF format contains visibilities consisting of:
\begin{itemize}
\item 7 data associated parameters, in a fixed order: \texttt{ u, v,
    w, date, time, first antenna, second antenna}
\item an array of complex visibility elements, that contain each a
  real part, an imaginary part and a weight
\end{itemize}
Note that in this original format, \texttt{u,v} and \texttt{w} are in
meters, the \texttt{date} is a \CLASS date, \texttt{time} is in
seconds.  The \texttt{w} column was often replaced by the scan number
\texttt{scan} for flagging and editing purpose.

\subsubsection{GDF v2}
The original GDF data format had a number of limitations, including 32
bits addressing, which led to the development of a revised version,
the so-called GDF v2.

GDF v2 addresses the previous limitations in size or number of
elements and provides provisions for new modes, including:
\begin{itemize}
\item extra columns in $(u,v)$ tables
\item handling of polarization data
\item an extra telescope section 
\end{itemize}

The interested reader will find in "Preparing \GILDAS for large
datasets II - \GILDAS Data Format Version 2" more details about GDF
v2.

\subsubsection{UVT versions}
The analysis of $(u,v)$ data needs to handle two frequency
referentials: the topocentric frame that corresponds to the observed
frequency and the source or rest frame. The former defines the angular
resolution of the telescope, or the $(u,v)$ coordinates that are
expressed wavelength units, the latter is used for the identification
of lines or in converting frequency shifts to velocity shifts. Both
are linked through equation \ref{eq:freq-doppler}.

Historically (for GDF v1), the frequency axis was in the topocentric
frame.  The rest frequency was given separately. The spectral
resolution did not distinguish between topocentric and source frame:
it was only approximate

With GDF v2, UV table contains a UV Tables version stored in the table
header.  Version 2.0 corresponds to the historical GDF v1
implementation (see above).  This version has both a lousy spectral
axis definition and a lousy angular resolution. For narrow bands, it
can still produce correct results, provided that the correct rest
frequency is accounted for (option \texttt{/FREQUENCY} of command
\texttt{TABLE}).

UV Table versions 2.1 and 2.2 provide a clear definition of the
spectral axis: it is the rest frequency, in the source frame (at the
source velocity in the selected system frame). The spectral resolution
is in that same source frame.  The doppler factor is provided to
convert the source frame frequency to topocentric frame for computing
the effective angular resolution from the \texttt{u,v} coordinates
measured as a length in meters.

The difference between versions 2.1 and 2.2 is that in 2.1 the mean
Doppler factor is provided where with version 2.2, a time dependent
Doppler factor is stored as an extra column. Note that if the
observatory section of the uv table is filled, the time dependent
Doppler factors can be recomputed.

\subsubsection{Command TABLE}
Command \texttt{TABLE} will produce a UV table. See corresponding help
for more information. Two options are very frequently used: the
\texttt{/FREQUENCY} option that is used to recompute the frequency and
velocity scales for a new rest frequency (see also section
\ref{sec:freq-scale}). The second useful option is \texttt{/RESAMPLE}
that is used to resample the data with less channels, another spectral
resolution, etc. One has to take care to not oversample the data.

UV tables can be read/edited as \SIC variables using the command
\texttt{DEFINE UVTABLE} (see associated help for more details).

UV tables can also be exported in UVFITS format using the 
\texttt{VECTOR$\backslash$FITS } command (see corresponding \HELP).
