\section{The Imager PIPELINE tools}
\label{sec:all}

With \ALMA{} or \NOEMA{}, you may end up with an observational data set
that contains several sources, each of them observed with
a number of spectral windows of different spectral resolutions,
covering many spectral lines. 

The bookkeeping of such data sets can be intricate. To help the
users to focus on the science, we have developped a pipeline
(namely a suite of scripts named \script{all-*.ima}) that automates the whole imaging process in this case.
This pipeline can be run through the \com{PIPELINE} command,
or through a control panel launched by the \comm{PIPELINE}{/WIDGET} command.

The pipeline control parameters are available in the \sicvar{ALL\%} global structure.

The pipeline is intended to handle all spectral sub-bands for a single receiver tuning of the interferometer, which
the data (possibly with several observing dates) for each sub-band and each source stored in a separate file 
(in UVFITS or GILDAS UV format). 

The principle is to gather all UV tables in a sub-directory (named
\texttt{./RAW/} by default), and to store the results of the
various processing steps (Self Calibration,  Spectral line
extraction, Imaging) in different sub-directories (respectively 
named \file{./SELF/, ./TABLES/, ./MAPS/} by default).

The behaviour of the Imaging step depends on whether a \com{CATALOG}
is used or not. 
\begin{itemize}
\item{Without} a spectral line catalog, the Imaging step will make
no attempt to identify spectral lines, and thus no attempt to image
each line separately.  Instead, each ``high spectral resolution'' UV table will be imaged 
completely.  An automatic estimate of the continuum level is made,
and is imaged separately from the continuum-subtracted data. At
the end of the process, the deconvolved ``continuum'' image
is added back to the spectral image.```Low spectral resolution'' data only produce
a continuum image.\\
This mode is appropriate for e.g. low spectral resolution data, 
very wide lines, and/or nearly confusion limited data with many spectral lines.
\item{With} a line catalog, a user-specified velocity range is imaged
separately around each line in the catalog that falls into the
observed frequency coverage.  Continuum and Line are separated, but
no attempt is made to add back the continuum data at end.
\end{itemize}
Naming conventions apply to identify which data set covers which spectral line. 


\subsection{Preparing the data}

For \NOEMA{}, the UV data can be created from the \file{.IPB, .hpb}
raw data files using the \clic{}  script \qom{@}\script{~all-tables}. The 
resulting UV tables (\file{.uvt} files) should be placed in your
working directory.

For \ALMA{},  UVFITS files should be created from the original Measurement
Set (\texttt{.ms} directory) using the \texttt{casagildas()} Python tool 
in CASA, and placed in your working directory. 

Once your working directory is loaded with the \file{.uvt} or 
\file{.uvfits} files, you can start using the \script{all-widget} script,
usually through the \comm{PIPELINE}{/WIDGET} command. 

It is recommended to separate data from different sources into different
directory structures, to simplify further processing.

\subsection{The \com{PIPELINE} Widget}

This section describes the widget that can be used to control
the image processing in a systematic way for an ensemble of 
UV tables.   The (selected) UV tables must come from
observations of the same source \textbf{and} correspond to a single
frequency setup, so that all spectral windows share the same
\textit{uv} coverage and observing dates and times.

\subsubsection{Getting started: \comm{PIPELINE}{/WIDGET}}

\qomm{PIPELINE}{/WIDGET} creates a Widget that is used to customize
the process and launch the various steps. It also creates
(through a call to \qom{@}\script{~all-create}) the \sicvar{ALL\%} structure
and its components.
\begin{figure}[!h]
  \centering
  \includegraphics[height=16cm]{all-widget.png}
  \caption{The Pipeline control widget. The two first lines
  of the widget cannot be modified directly. The \texttt{Telescope}
  information is derived by button \texttt{ORGANIZE} from the data
  set. The \texttt{Spectral Line Catalog} is defined through command \qom{CATALOG}.
\label{fig:allwidget}}
\end{figure}

\subsubsection{\texttt{ORGANIZE} step}
The \texttt{ORGANIZE} button will move the initial files
(\file{.uvfits} and \file{.uvt} files) into an appropriate sub-directory
structure. By default, \file{.uvfits} files will be in \file{./UVFITS/}
(name controlled by \sicvar{all\%uvfits})
sub-directory, while \file{.uvt} files will be in \file{./RAW/} sub-directory
(name controlled by \sicvar{all\%raw}).

If absent, the \file{.uvt} files will be created from the \file{.uvfits} ones,
using default parameters (i.e. assuming unpolarized emission).

\subsubsection{\texttt{FIND} step}

The \texttt{FIND} button will explore the directory containing
the initial UV tables (\file{./RAW/} by default) to identify
\textit{Wide Band} UV tables, i.e. those that cover enough 
bandwidth to provide enough sensitivity
for self-calibration on continuum flux.

This information will be used later to derive the self-calibration
solutions from some UV tables, and apply it to others, typically
narrow band spectral line data.

\subsubsection{\texttt{SELECT} step}

The \texttt{SELECT} button will restrict the subsequent work on
the files matching the \texttt{"File Filter"} field value. 

\subsubsection{\texttt{CHECK} step}
The \texttt{CHECK} button will use the list of \textit{Wide Band}
UV tables to check whether Self calibration may be needed and possible.
The method is currently approximate and will be improved in future
releases.
\subsubsection{\texttt{SELF} step}

The \texttt{SELF} button will use the list of \textit{Wide Band}
UV tables to compute a (phase and amplitude) Self calibration
solution, and apply it to all UV tables. It will identify which
UV tables correspond to a given Wide Band one, so that the proper
self calibration solution is applied.

The self calibrated UV tables are placed in another sub-directory
(\file{./SELF/} by default, controlled by \sicvar{all\%self}).

A prefix (controlled by \sicvar{all\%prefix\_self}) can be added
to the file names to remind the user that they have been self-calibrated.\\


\textbf{Caveats: CURRENT LIMITATIONS} \\
- No provision is made to use spectral line flux when the continuum flux
is too low.\\
- Thus, if the Self-calibration solution is not good, you should not apply
it. The \texttt{COLLECT} step may help at this stage.

\subsubsection{\texttt{SHOW} step}

This step will display the phase and amplitude correction that have been
computed in the Self Calibration step. Optionally, these plots can be
saved on Hardcopies (\file{.pdf} files for Pdf, \file{.eps} for EPS).

\subsubsection{\texttt{COLLECT} step}
Most NOEMA data include several wide bands (4 with PolyFix in its standard
mode), and an independent Self-calibration solution is computed for each
of them. The \texttt{COLLECT} button will merge all these solutions
into a single one, assuming the phase errors are due to pathlength changes. 

This improves the S/N of the self-calibration solution, and can
be beneficial when the S/N is the limiting factor. On the other hand,
if the data has very high S/N, this step may result in lower dynamic
range than the application of the separate self-calibration solutions.

\subsubsection{\texttt{APPLY} step}
The \texttt{APPLY} button will apply the Self-calibration solution
found in the previous steps (\texttt{SELF} and optionally \texttt{COLLECT})
to all UV tables. The selection of the solution to be applied is based
on matching frequencies between the UV table and the solutions.

If the \texttt{COLLECT} step has been performed, one may still
use the individual solutions by setting variable \sicvar{ALL\%COMBINE} to \texttt{NO}. 

\subsubsection{\texttt{TIME} step}

The \texttt{TIME} button will time average the self calibrated
UV tables to save space and speed up further processing.

The integration time can be specified, or left to 0. In this case,
the data will be time-averaged to an appropriate sampling time 
given the longest baselines and field of view.

\subsubsection{\texttt{IMAGING} step}

The \texttt{IMAGING} button will scan all spectral windows to
identify whether they can be used to produce a Continuum or
spectral line image. 

  Low resolution spectral windows, identified as those whose
resolution is coarser than \sicvar{ALL\%MINFRES} will be used to
produce continuum images, by filtering any detected spectral
signature in the data before.

  High resolution spectral windows, identified as those whose
resolution is better than \sicvar{ALL\%MINFRES}, data cubes will
be produced.

    If a Line Catalog is present, these high spectral resolution
windows will be scanned
for line identifications. For each spectral line in the current 
catalog(s) (defined by command \com{CATALOG}) 
that fall in a spectral window, a continuum-free UV table will be
created, covering the velocity range specified by the user (by
variable \sicvar{ALL\%RANGE}) around the line frequency. The naming
convention is the following:
\begin{verbatim}
  original-molecule-I-X
\end{verbatim}
where \\
- \texttt{original} is the name of the initial high resolution
spectral window\\
- \texttt{molecule} is the name of the spectral line in the catalog\\
- \texttt{I} is a sequence number, incremented each time a new
line is found from the same original UV table.\\
- \texttt{X} is a character code, equal to \texttt{D} if the line
is suspected to be detected, and to \texttt{U} if not. When several
lines are too close together, the \texttt{D} status may be incorrectly
affected, but this is just a naming convention, not a strong result.

  In addition an
\begin{verbatim}
  original-C
\end{verbatim}
file that contains (presumably) line-free emission only is created for each
original UV table. 

  The imaging results are stored in a specific sub-directory 
(\texttt{./MAPS/} by default, controlled by \sicvar{all\%maps}). Only 
the Clean Component Table (\file{.cct}) and Clean image 
(\file{.lmv-clean}) files are written.

  When no Line catalog is present, only 3 images are produced for each UV tables:
starting froml the \texttt{original}  name
\begin{verbatim}
  original-C       original-U    and   original-A
\end{verbatim}
The \ldots\texttt{-C} contains (an estimate of) line-free emission, averaged
over the bandwidth of the UV data. The \ldots\texttt{-U} is the continuum
subtracted image, while in \ldots\texttt{-A} the pseudo-continuum image
obtained in \ldots\texttt{-C} has been added back. The \ldots\texttt{-A}
thus contains all the signal, and can be used to provide a better estimate
of the continuum level using e.g. the \com{MAP\_CONTINUUM} command.

  Spectral resampling can be performed by appropriate setting of \sicvar{ALL\%RANGE}.
If non zero, \sicvar{ALL\%RANGE[3]} indicates the desired spectral resolution.
Furthermore
In the CATALOG case, \sicvar{ALL\%RANGE[1:2]}, if non zero, specify the Min and Max velocities
imaged around each spectral line.
  

\subsubsection{\texttt{TABLES} step}

The \texttt{TABLES} button performs the same scanning and
identification process as the \texttt{IMAGING} button, but stores
the resulting UV Tables (instead of the .cct and .lmv-clean files for
the \texttt{IMAGING} button) in a sub-directory, named \file{./TABLES/}
by default, controlled by \sicvar{all\%tables}.

  This step is optional, and only needed if the user intends to 
perform some UV data analysis (like UV plane fitting, line stacking,
continuum spectral index determination, direct modeling, etc...).

  This step makes no real sense if no \qom{CATALOG} is provided by
the user. Currently no spectral resampling is performed, only extraction
of the velocity range specified by \sicvar{ALL\%RANGE[1:2]}

\subsection{The non interactive \com{PIPELINE}}


The whole series of actions available through the Pipeline Widget can be
performed in series through the \qom{PIPELINE} command.

Command \qom{PIPELINE *} will perform all the actions, without
any user interaction. The \texttt{PIPELINE} button in the Pipeline widget will do the same
(and also the \texttt{GO} button, as usual in {\mbox{{\color{red} \sc Gildas}}} widgets).

A specific step can be performed by command \qomm{PIPELINE}{StepName}.
The available step sequence is 
\texttt{ORGANIZE, FIND, SELECT, COMPUTE, COLLECT, APPLY, SHOW, TIME, TABLES,
IMAGE, SHOW} and ultimately \texttt{SAVE} to remember the parameters used
in the processing. \qomm{PIPELINE}{LAST} will repeat the last step (presumably
after you changed a control variable !...) while \qomm{PIPELINE}{NEXT} will
execute the next one.

\subsection{Mosaics and other limitations.}

Well, the pipeline does not currently handle mosaics very well. In fact, it does
work provided that the data fit in memory, but there is no provision to add short
spacings which are often needed in such a case.

Also, the \qom{PIPELINE} is intended for simultaneous observations with a single
tuning and single source.  Although multiple sources can be handled by using the appropriate
file filter, it is recommended to process them in separate sub-directories.

Non simulaneous observations can lead to puzzling inconsistencies, as the self-calibration
from one date might be applied to the  wrong time or setup, flagging that data!

\section{Really Huge Problems}
\label{sec:huge}

\imager{} assumes everything fits into memory. This is in general
quite fine for \NOEMA{}  data. However, for \ALMA{} data, if you are 
working with a too small computer (such as my laptop, which is 
otherwise fine), you may be lacking physical memory (RAM, Random Access Memory), 
and \imager{} may become really inefficient by using Virtual Memory instead.

To avoid time losses, \imager{} prevents reading UV data whose size
exceeds the available RAM, and warns the user if it exceeds half
of the RAM.  To treat these cases, \imager{} provides instead
a number of tools that can work sequentially on the data set,
instead of loading it in memory all at once. 

\begin{enumerate}
\item Working by subsets:\\
  the \comm{READ}{/RANGE} command allows to select an ensemble of 
  channels from a UV data. If this ensemble is small enough, \imager{} 
  can work.  At the end the \comm{WRITE}{/APPEND} and 
  \comm{WRITE}{/REPLACE} command will allow to put these channels at 
  their proper places in a deconvolved data cube.
\item Working on UV data files:\\
  Some operations on UV data, like time averaging, separation of line 
  from continuum emission, or self-calibration, and of course, spectral 
  resampling, are best done using all valid channels to avoid loosing 
  sensitivity. 
  To allow \imager{} to do them even for large data
  files, most UV-related commands have a \qom{/FILE} option which
  instructs the command to work from the corresponding data file,
  instead of the UV buffers. This includes \comm{UV\_PREVIEW}{/FILE}, 
  \comm{UV\_BASELINE}{/FILE}, \comm{UV\_FILTER}{/FILE} and especially 
  the \comm{UV\_SPLIT}{/FILE} commands. Time averaging can be performed 
  by \comm{UV\_TIME}{/FILE}, and prior sorting by time order can be
  done by \comm{UV\_SORT}{/FILE}. Spectral range extraction is possible by 
  \comm{UV\_EXTRACT}{/FILE}, spectral resampling by \comm{UV\_COMPRESS}{/FILE},
  \comm{UV\_RESAMPLE}{/FILE},  and \comm{UV\_HANNING}{/FILE}.
\end{enumerate}
By using the above commands, all operations can be done in a
quasi sequential way, avoiding to load in memory whole data sets.

Two commands have no equivalent using the \imager{} buffers, and work
only on files. Their \qom{/FILE} option is used to provide an
homogeneous syntax, but must be present for the command line
to be valid.
\begin{enumerate}
\item the \comm{UV\_MERGE}{/FILE}\\
that allows to merge together an arbitrary number of UV tables,
in spectral line (with spectral resampling) or continuum (with 
flux scaling according to a spectral index) modes.
\item the \comm{UV\_SPLIT}{/FILE}\\
that combines the capabilities of \com{UV\_BASELINE} and 
\com{UV\_FILTER} in a single command, since both operations require 
the same parameters and provide complementary informations.
\end{enumerate}

The \comm{UV\_SORT}{/FILE} command also has a different behaviour than
its memory only version \com{UV\_SORT}. While the latter creates
a transposed version of the UV table for internal use (it \textbf{cannot}
be saved), the former keeps the normal
organisation with the visibility axis first. 

An example of use of such facilities is the \qom{@}\script{~image-mosaic} script
that splits the \qom{uv\_map} step into chunks that fit in the
computer memory available to the user. 
