\documentclass{article}

\usepackage{graphicx}
\usepackage{fancyhdr}

\makeindex{}

\makeatletter
\textheight 625pt
\textwidth 460pt
\oddsidemargin 0pt
\evensidemargin 0pt
\marginparwidth 50pt
\topmargin 0pt
\brokenpenalty=10000

% Makes fancy headings and footers.
\pagestyle{fancy}

\fancyhf{} % Clears all fields.

\lhead{\scshape Bardeau et al., 2015}
\rhead{\scshape \nouppercase{\rightmark}}   %
\fancyfoot[C]{\bfseries \thepage{}}

\renewcommand{\headrulewidth}{0pt}    %
\renewcommand{\footrulewidth}{0pt}    %

\renewcommand{\sectionmark}[1]{%
  \markright{\thesection. \MakeLowercase{#1}}}
\renewcommand{\subsectionmark}[1]{}

% \fancypagestyle{firststyle}
% {
%    \fancyhf{}
%    \fancyhead[C]{Original version at \tt http://iram-institute.org/medias/uploads/class-herschel-fits.pdf}
%    \fancyfoot[C]{\bfseries \thepage{}}
% }


\newcommand{\ie} {{\em i.e.}}
\newcommand{\eg} {{\em e.g.}}
\newcommand{\cf} {cf.}

\newcommand{\gildas}    {\mbox {\bf GILDAS}}
\newcommand{\sic}       {\mbox {\bf SIC}}
\newcommand{\greg}      {\mbox {\bf GREG}}
\newcommand{\class}     {\mbox {\bf CLASS}}
\newcommand{\classic}   {\mbox {\bf CLASSIC}}

\newcommand{\aas}  {{\sc Associated Arrays}}
\newcommand{\aaa}  {{\sc Associated Array}}
\newcommand{\raa}  {{\sc Reserved Associated Array}}
\newcommand{\raas} {{\sc Reserved Associated Arrays}}
\newcommand{\ry}   {{\tt RY}}
\newcommand{\rx}   {{\tt RX}}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\begin{document}

% \thispagestyle{firststyle}

\begin{center}
    \includegraphics[width=5cm]{gildas-logo}\\[2\bigskipamount]
  \huge{}
  IRAM Memo 2015-4\\
  Introducing \aas\ in \class\\[3\bigskipamount]
  \large{}
  S. Bardeau$^1$, J. Pety$^{1,2}$\\[0.5cm]
  1. IRAM (Grenoble)\\
  2. Observatoire de Paris\\[\bigskipamount]
  \normalsize{}
  November, 26$^{th}$ 2019\\
  Version 1.3
\end{center}

\begin{abstract}
  The concept of \aas{} is being added in the \class\ data format. It
  is an additional section that will store a set of data arrays whose
  spectral axis matches the brightness spectral axis, \ie, one-to-one
  channel correspondance. At start, this will be used in two
  contexts. For Herschel/HIFI, it will store a logical array of
  blanked channels and a logical array to define the channels where
  the lines appear (see IRAM Memo 2015-3). At some point, it will be
  used at the IRAM-30m to store the results of the calibration scans,
  \ie, the sky counts in the {\tt RY} array and the hot/cold counts,
  and the receiver, system, and calibration temperatures in a set of \aas.\\

  This document introduces the detailed specifications of the concept
  of an \aaa{}, its implementation/integration in \class{}, and it
  gives an example of use, namely how to baseline a spectra using a
  {\tt LINE} associated array to define the baseline windows. The
  specifications of the \aas\ is foreseen to evolve following feedback
  from its use at IRAM and elsewhere.\\

  Keywords: \class\ Data Format

  Related documents: \class\ documentation, Importing Herschel-FITS
  into \class
\end{abstract}

\newpage{}
\tableofcontents{}

\newpage{}

\section{Nomenclature reminder}

It is reminded that:
\begin{enumerate}
\item A \class\ observation is used to store and retrieve a flux array
  for analysis into \class. This flux array is called \ry\ hereafter.
\item In a \class\ observation, there are 3 ways to define the X axis
associated to \ry:
\begin{enumerate}
\item the spectroscopic section for regularly sampled (in frequency
  and velocity) spectroscopic data,
\item the drift section for regularly sampled (in time and angular
  position) continuum drift data,
\item the so-called "X coordinate section" for irregularly sampled
  spectroscopic or continuum drift data.
\end{enumerate}
One of these 3 sections is used to define the \rx\ array describing
the X axis associated to \ry. In the cases (a) and (b), \rx\ in memory
is built so that it has as many values as \ry. In the case (c), the
\class\ Data Format ensures that \rx\ has as many values as \ry. For
simplicity hereafter, we call \rx\ the X axis coming from one of these
sections, and {\tt Nchan} its number of values.
\end{enumerate}

\section{Specifications}

The concept of \aas\ is introduced in \class. They are specified as
follows:

\begin{enumerate}

\item The \aas\ section is added to store 1 or more arrays of data
  directly \emph{related} to \ry.

\item The \aas\ must be of size {\tt Nchan} (1st dimension). A second
  dimension can be used. Their X axis associated to the 1st dimension
  is \textbf{exactly} described by \rx. In other words, there is a
  one-to-one channel relationship between the flux array \ry\ and any
  of the \aas. This also means that any modification on the
  spectroscopic section has a similar and consistent impact on \ry\
  and on the \aas. This includes resizing the axis ({\tt EXTRACT}) and
  shift/stretch of the axis ({\tt MODIFY VELO|FREQ}).

\item The type and kind of data stored as an \aaa\ is flexible, in the
  limit of what the Gildas internal engines are used to (\ie\ numeric
  data types). Expected types are integer or real (floating point)
  types. See details in Table~\ref{tab:typekind}.

\item An \aaa\ is identified by its name (12 characters maximum). It
  must be unique in one observation. The name may contain only
  characters valid for Sic variable names (typically the alphanumeric
  character set), so that the arrays can also be used as Sic
  variables. The \aaa\ may also provide a unit string (12 characters
  maximum), left blank if not used. And it must provide a \emph{bad}
  (blanking) value, valid in the supported range of values
  corresponding to the data format.

\item There is no predefined/limited list of supported \aas. Some may
  be built and saved by the \class\ internal engines with a precise
  definition (so-called \raas, see section~\ref{txt:reserved}), others
  may be created and saved by the user on his own choice. Examples
  could be:
  \begin{itemize}
  \item a weight array
  \item a mask array
  \item wavelets
  \item fitted model and residuals
  \item etc.
  \end{itemize}
  One typical counter example which does not satisfy the rule in item
  \#2 is:
  \begin{itemize}
  \item the Fourier transform: its X axis (inverse frequency) is not
    described by the spectroscopic section, {\tt EXTRACT}ing the same
    subset of \ry\ and FFT would be incorrect (the FT of a subset is
    not the subset of the FT).
  \end{itemize}

\item Resampling and its other flavors ({\tt RESAMPLE}, {\tt AVERAGE},
  ...) is a particular problem since \class\ has to know how the
  values can be \emph{mixed} (\eg\ what is the meaning of resampling a
  flag array?). There is no straightforward answer. Basic ideas are:
  \begin{itemize}
  \item the usual {\tt RY} resampling engines can be opened for all
    floating point arrays, and can be extended to integers (floating
    point averaged then rounded back).
  \item a nearest neighbour algorithm could also be offered (may be
    better suited for integers like boolean arrays).
  \item for \raas, the \class\ programmer can offer custom resampling
    engines, suited exactly for a given array,
  \item one can think in \emph{hooks} so that the users can describe
    how the resampling works. Implementation is unclear at this stage.
  \item if \class\ does not have any solution for resampling, the
    ultimate solution is to remove the \aaa. Leaving it non-resampled
    would violate the rules in item \#2.
  \end{itemize}
  As of today, \class\ uses the first proposition as a generic
  solution.
\end{enumerate}

\begin{table}[htb]
\centering
\caption{Type and kind of data supported for \aas. Note that 4-bits and
  2-bits integers are provided for storage on disk with low space
  comsuption. In particular, 2-bits integers provide 4 possible values: they
  are well suited for storing a boolean (0 and 1) and a bad value (e.g. -1).
  Note also that in practice, integers smaller than 32 bits are always
  loaded as INTEGER*4 in memory, so that Sic variables can be mapped on them
  and the \gildas\ tools can be used.}
\vspace{1ex}
\begin{tabular}{ccclc}
\hline
\hline
Internal code & Equivalent Fortran  & Nbits  & Range of values         & Type and kind \\
              & type and kind       & (disk) &                         &  in memory    \\
\hline
fmt\_r8        & REAL*8             &   64   &                        & REAL*8     \\
fmt\_r4        & REAL*4             &   32   &                        & REAL*4     \\
fmt\_i8        & INTEGER*8          &   64   & $-2^{63}$ to $+2^{63}-1$ & INTEGER*8  \\
fmt\_i4        & INTEGER*4          &   32   & $-2^{31}$ to $+2^{31}-1$ & INTEGER*4  \\
fmt\_i2        & INTEGER*2          &   16   & $-2^{15}$ to $+2^{15}-1$ & INTEGER*4  \\
fmt\_by        & INTEGER*1          &    8   & $-2^{7}$  to $+2^{7}-1$  & INTEGER*4  \\
fmt\_b4        &    N/A             &    4   & $-2^{3}$  to $+2^{3}-1$  & INTEGER*4  \\
fmt\_b2        &    N/A             &    2   & $-2^{1}$  to $+2^{1}-1$  & INTEGER*4  \\
\hline
\end{tabular}
\label{tab:typekind}
\end{table}


\section{Implementation details}

The \aas\ support was officialy introduced in the oct15 version of
\class, together with the Herschel-HIFI FITS reader. They are
available in a new section (identifier code -19) in the \class\
observation header. The size of this section is variable, as there can
be several arrays in the section, and the size of each array is the
same as {\tt RY}. The section provides the number of arrays stored,
then for each array its description and its data. See
Table~\ref{tab:section} for
details.\\

\begin{table}[htb]
\centering
\caption{Description of the section \aas\ in the \class\ observation
  header. Each array is described by its format, 2nd dimension, name,
  and unit. The \emph{bad} (blanking) value is stored in the same
  format as the data, then the data array itself follows. Order
  matters here.}
\vspace{1ex}
\begin{tabular}{clll}
                    & Name          & Format (disk) & Description  \\
\hline
Once in the section & {\tt N}       & INTEGER*4     & Number of arrays \\
Repeated per array  & {\tt NAME}    & CHARACTER*12  & Array name (identifier) \\
         ''         & {\tt UNIT}    & CHARACTER*12  & Unit (blank if not relevant) \\
         ''         & {\tt DIM2}    & INTEGER*4     & Second dimension (0 for 1D array) \\
         ''         & {\tt FMT}     & INTEGER*4     & Data format (internal code) \\
         ''         & {\tt BAD}     & FMT           & Blanking value \\
         ''         & \emph{DATA}   & FMT           & The data array (Nchan values, times {\tt DIM2})
\end{tabular}
\label{tab:section}
\end{table}

A basic example using 2 arrays is shown in Appendix~\ref{app:section}.

\section{\raas}
\label{txt:reserved}

The \aas\ are a generic concept offered to store any kind of numeric
data with minimal restrictions. However, \class\ will be the first one
to offer such arrays. This will be done by reserving some array
names. The reserved arrays have a strict definition and, in some
cases, \class\ knows how to use them directly.

\subsection{Pre-reserved array names}

The names {\tt X, Y, C, V, F, I, T} and {\tt A} are pre-reserved
because they refer to the \rx\ or \ry\ arrays in the different
possible units in \class\ (command {\tt SET UNIT}). The use of these
names is restricted in order to avoid confusion. In the future they
could also be used as real arrays saved on disk.

\subsection{Reserved arrays}

As of today, \class\ reserves with a special meaning the arrays named
{\tt BLANKED} and {\tt LINE}. The name {\tt W} is also reserved for
use as a channel-based weight array. See Table~\ref{tab:reserved} for
their specifications.

\begin{table}[htb]
\centering
\caption{Specifications of the \raas.}
\vspace{1ex}
\begin{tabular}{lccccl}
Name          & Format (disk) & Dim2 & Unit &  Bad  & Description \\
\hline
{\tt W}       & fmt\_r4       &  0   &      & -1000 & Weight array \\
{\tt BLANKED} & fmt\_b2       &  0   &      &   -1  & Flag for bad channels \\
{\tt LINE}    & fmt\_b2       &  0   &      &   -1  & Flag for spectral line
\end{tabular}
\label{tab:reserved}
\end{table}

\subsubsection{W} 

The {\tt W} \aaa\ is a single-precision floating point array providing
a channel-based weight array of the spectrum. It can be used as input
and output of the {\tt AVERAGE} command (and its other flavors) thanks
to the switch {\tt SET WEIGHT ASSOC}. The user can define the weight
array of his choice for the input spectra thanks to a sequence like:
\begin{verbatim}
  FILE IN ...
  FIND
  FILE OUT ... SINGLE
  FOR I 1 TO FOUND
    GET NEXT
    ! Fill MYARRAY as desired here
    ASSOCIATE W MYARRAY
    WRITE
  NEXT I
\end{verbatim}
and then reuse the output file as input for {\tt SET WEIGHT ASSOC} +
{\tt AVERAGE}.

\subsubsection{BLANKED} 

The {\tt BLANKED} \aaa\ is an integer array of 0 or 1 indicating if
the \ry\ data should be flagged out (1) or not (0). The purpose is
similar to the bad values in \ry, except that the \emph{actual}
(non-bad) values are still available by default, and it is the choice
of the user to apply the {\tt BLANKED} flags or not\footnote{This
  array was actually introduced in the context of Herschel-HIFI data
  filler to \class.}. The user can apply these flags to {\tt RY} \eg\
with the following command:
\begin{verbatim}
  LAS> LET RY R%HEAD%SPE%BAD /WHERE R%ASSOC%BLANKED%DATA.EQ.1
\end{verbatim}

\subsubsection{LINE}

The {\tt LINE} \aaa\ is an integer array of 0 or 1 indicating if the
\ry\ channels are in (1) or out (0) a spectral line, i.e. in signal or
noise windows. This array is intended to be used by the command {\tt
  BASE} in order to exclude the given channels. Here is the specific
integration in \class:
\begin{itemize}
\item if {\tt SET WINDOW} is invoked with the option {\tt
    /ASSOCIATED}, the command {\tt BASE} will ignore the channels
  where the {\tt LINE} array is 1. The usual windows (defined by their
  min-max range) are not used in this context. As a result, the
  element {\tt R\%HEAD\%CAL\%NWIND} is set to -1 (code for ``\aaa\
  {\tt LINE} was used'') in the baseline section.
\item if {\tt SET WINDOW} is invoked with the argument {\tt AUTO}
  (\ie\ take windows from data), and if the element {\tt
    R\%HEAD\%CAL\%NWIND} is -1, the {\tt LINE} \aaa\ will be used by
  the command {\tt BASE}.
\item {\tt SHOW WINDOW} will display a specific message if the current
  {\tt SET WINDOW} tuning refers to the \aaa\ {\tt LINE}.
\item {\tt DRAW WINDOW} will plot the \aaa\ {\tt LINE} if the current
  {\tt SET WINDOW} tuning refers to the \aaa\ {\tt LINE}.
\end{itemize}

\section{Generic integration in \class}

We describe here what are the generic tools supporting the \aas\ in
\class\ and how the end-user can interact with them.

\begin{itemize}
\item The command {\tt ASSOCIATE} allows the user to attach or detach
  an \aaa\ to/from the current R buffer. Some other commands may also
  attach \aas\ implicitly, \eg\ the Herschel-HIFI FITS reader.
\item The command {\tt DUMP} can be used for a quick overview of the
  \aas\ attached to the {\tt R} buffer, if any.
\item The \aas\ are also instantiated in the Sic structure {\tt
    R\%ASSOC\%} (read-only) for direct use by the end-user. For each
  array a sub-structure of the same name is provided, with the form:
\begin{verbatim}
R%ASSOC%NAME%DIM2 =        0      ! Integer GLOBAL RO
R%ASSOC%NAME%UNIT =               ! Character*12 GLOBAL RO
R%ASSOC%NAME%BAD =      -1        ! Integer GLOBAL RO
R%ASSOC%NAME%DATA is an integer Array of dimensions  8257
\end{verbatim}
where {\tt NAME} is to be replaced by the array name.
\item The arrays can be plotted thanks to the commands {\tt PLOT} and
  {\tt SPECTRUM}, passing their identifying name as first
  argument, \eg\ {\tt PLOT LINE} or {\tt SPECTRUM W}.
\item The command {\tt EXTRACT} (spectrum and index mode) supports
  extracting a subset of \ry\ and the same consistent subset of all
  \aas.
\item The commands {\tt SMOOTH}, {\tt RESAMPLE}, and {\tt FOLD} apply
  the same exact engines on the \aas\ as the ones which are used for
  RY. This is straightforward for floating point \aas\ (reasonable
  solution for most physical quantities). For an integer \aaa, the
  values are smoothed (resp. resampled) as floating point values and
  then rounded back to nearest integer (reasonable solution for
  integer physical quantities, e.g. counts). This solution is generic
  but may not be satisfying in all cases (e.g. flag arrays). No custom
  solution is possible yet.
\item {\tt AVERAGE} and its other flavors ({\tt ACCUMULATE}, {\tt
    STITCH}, {\tt RMS}) support averaging the \aas\ of the current
  index, as long as all the observations have the same number of \aas,
  with the same name and in the same order. They are also averaged
  using a floating point engine as a generic solution (see above). All
  arrays are averaged except {\tt W}, as it makes no sense to average
  the weight arrays (the resulting weight array is actually saved if
  {\tt SET WEIGHT} is {\tt ASSOC}.
\item Finally, some commands may interpret a specific \aaa\ when one
  is found. See the related \aaa\ description for details.
\end{itemize}

\section{Evolution}

This document describes the first version of the \aas. However, the
\class\ team evaluates the feasibility of a new version of the arrays
in order 1) to provide a more efficient access to the arrays, and 2)
to ensure a better integration of the arrays in \class.

% \section{Fortran API}
% TBD.

\appendix
\newpage
\section{\aas\ section example}
\label{app:section}

\begin{verbatim}
N                [I*4]  = 2  ! 2 arrays in the section

### Array 1 ###
  # Description
  FMT(1)         [I*4]  = fmt_b2  ! i.e. 2-bits integer
  DIM2(1)        [I*4]  = 0       ! i.e. array is 1D of size Nchan
  NAME(1)        [C*12] = "LINE"
  UNIT(1)        [C*12] = ""
  BAD(1)         [B*2]  = -1      ! 2-bits value
  # Data
  DATA1(1)       [B*2]            ! 2-bits-value
   ...
  DATA1(Nchan)   [B*2]            ! 2-bits value

### Array 2 ###
  # Description
  FMT(2)         [I*4]  = fmt_r4
  DIM2(2)        [I*4]  = 2          ! i.e. array is 2D of size Nchan x 2
  NAME(2)        [C*12] = "WAVELET"  ! All wavelet orders in a 2D array
  UNIT(2)        [C*12] = ""
  BAD(2)         [R*4]  = -1000.0
  # Data (column-major)
  DATA2(1,1)     [R*4]
   ...
  DATA2(Nchan,1) [R*4]
  DATA2(1,2)     [R*4]
   ...
  DATA2(Nchan,2) [R*4]
\end{verbatim}

\end{document}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
