\documentclass{article}

\usepackage{graphicx}
\usepackage{fancyhdr}
\usepackage{multirow}
\usepackage{rotating}

\makeindex{}

\makeatletter
\textheight 625pt
\textwidth 460pt
\oddsidemargin 0pt
\evensidemargin 0pt
\marginparwidth 50pt
\topmargin 0pt
\brokenpenalty=10000

\newcommand{\ie} {{\em i.e.}}
\newcommand{\eg} {{\em e.g.}}
\newcommand{\cf} {cf.}

\newcommand{\gildas}    {\mbox {\bf GILDAS}}
\newcommand{\sic}       {\mbox {\bf SIC}}
\newcommand{\greg}      {\mbox {\bf GREG}}
\newcommand{\otfcal}    {\mbox {\bf OTFCAL}}
\newcommand{\mira}      {\mbox {\bf MIRA}}
\newcommand{\class}     {\mbox {\bf CLASS}}
\newcommand{\oldclass}  {\mbox {\bf CLASS77}}
\newcommand{\newclass}  {\mbox {\bf CLASS90}}

\newcommand{\analyse}{\texttt{ANALYSE}}
\newcommand{\fit}{\texttt{FIT}}
\newcommand{\fort}{\texttt{F90}}
\newcommand{\plait}{\texttt{PLAIT}}
\newcommand{\ave}{\texttt{AVERAGE}}
\newcommand{\accu}{\texttt{ACCUMULATE}}

\newcommand{\emm}[1]{\ensuremath{#1}}   % Ensures math mode
\newcommand{\emr}[1]{\emm{\mathrm{#1}}} % Uses math roman fonts

\def\degr{\hbox{$^\circ$}}
\def\arcmin{\hbox{$^\prime$}}
\def\arcsec{\hbox{$^{\prime\prime}$}}

% Makes fancy headings and footers.
\pagestyle{fancy}

\fancyhf{} % Clears all fields.

\lhead{\scshape Averaging spectra with \class}
\rhead{\scshape \nouppercase{\rightmark}}   %
\fancyfoot[C]{\bfseries \thepage{}}

\renewcommand{\headrulewidth}{0pt}    %
\renewcommand{\footrulewidth}{0pt}    %

\renewcommand{\sectionmark}[1]{%
  \markright{\thesection. \MakeLowercase{#1}}}
\renewcommand{\subsectionmark}[1]{}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\begin{document}

\title{IRAM Memo 2009-4\\[3\bigskipamount]
  Averaging spectra with \class}%
\author{S. Bardeau$^1$, J. Pety$^{1,2}$ \\[0.5cm]
  1. IRAM (Grenoble)\\
  2. LERMA, Observatoire de Paris}
\date{April, 11th 2011\\
      Version 1.1}

\maketitle

\begin{abstract}
  \newclass\ (hereafter \class) provides a set of commands capable to
  average two or more spectra. They provide many averaging modes,
  presented hereafter in this document. The different modes often
  imply to perform internally and silently some important
  computations, namely resampling and weighted average. Combining
  these operations at the same time may imply some non-trivial
  effects described here.

  On October 2008, it appeared that some particular combinations of
  the tunable modes were not behaving as expected (either in
  \newclass\ and in \oldclass). A complete cleaning and factorization
  of the algorithm was performed, associated to exhaustive tests of
  all combinations. A test suite was also provided to check the output
  of each commands and modes.

  Following this maintenance of the code, and the use of these
  capabilities to concatenate the new EMIR\footnote{see {\tt
  http://www.iram.es/IRAMES/mainWiki/EmirforAstronomers} } spectra, it
  was decided to write this document in order to keep a trace of all
  the methods applied. This was also the occasion to investigate
  deeply in the code and to examine the effects which can occur during
  all the possible processings.\\

  Keywords: \ave, \accu, resampling, weighting, concatenation

  Related documents: IRAM memo 2005-1: \emph{\class\ evolution: I. Improved OTF support}
\end{abstract}

%%%%%%%%%%
\newpage{}
\tableofcontents{}

%%%%%%%%%%
\newpage{}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{\class\ commands}

Here is a summary of the commands and their online helps which can be
used in this context.

\subsection{Main commands}
Two main commands are able to add spectra in \class: \accu\ and
\ave. \accu\ works on the two $R$ and $T$ \class\ buffers, whereas
\ave\ can combine all the spectra in the current index.

\subsubsection{ACCUMULATE}
\begin{verbatim}        LAS\ACCUMULATE

    ACCUMULATE  is  used  to  integrate step by step an ensemble of spectra.
    The T spectrum is added to the R spectrum with the  current  weights.  R
    thus  contains  the current sum, and T the last sum. As command AVERAGE,
    ACCUMULATE checks for the positional coincidence and  calibration  homo-
    geneity according to SET MATCH and SET CALIBRATION respectively. Aligne-
    ment of spectra is checked according to SET ALIGN  command.  Accumulated
    spectra may have different spectral resolution. In this latter case, the
    final spectrum has the coarsest resolution of the two input spectra, and
    each  channel will be the weighted average of the average values of each
    spectrum in the resulting channel. It is however preferable  to  accumu-
    late spectra which have the same resolution.

    For  EQUAL  weights, ACCUMULATE computes the sum of the two spectra, al-
    lowing addition of an ensemble of spectra after division  by  the  total
    number of spectra.

    For  other  weightings, ACCUMULATE computes the average of the two spec-
    tra.

    ACCUMULATE also works on Continuum drift. The alignement may be  Channel
    or Position in this case.
\end{verbatim}
Please consider that \accu\ behaves differently if using an {\tt
EQUAL} weight or another one: this provides to the user a way to
perform its own average by first adding the spectra and then dividing
them at its will.

\subsubsection{AVERAGE}
\begin{verbatim}        LAS\AVERAGE [/NOMATCH]

    Average all the spectra of the current index using the current weighting
    function (see SET WEIGHT) and alignment mode (see SET ALIGN). The  homo-
    geneity  of  calibration  is  checked  according to SET CALIBRATION. Bad
    channels are handled according to SET BAD.  A sum may be interrupted  by
    <^C>, but the results is then undefined.

    The  position  coincidence of consecutive spectra will not be checked if
    option /NOMATCH is present. Else, this check relies on the  current  SET
    [NO]MATCH status. See SET MATCH to build an index with a given position-
    ing tolerance.
\end{verbatim}

\subsection{Tuning of the addition}

The behavior of the two \accu\ and \ave\ commands can be tuned with a
set of associated parameters. The commands to change them are
presented hereafter.

\subsubsection{Alignment}
\begin{verbatim}        LAS\SET ALIGN Type Range

    Defines the way the spectra to be added are aligned (Type) and  combined
    (Range). Possible combinations are:
    Type: C[hannel]      for spectra and drift
          V[elocity]     for spectra only
          F[requency]    for spectra only
          P[osition]     for drift only
    Range: I[ntersect] or C[omposite]
    Default is SET TYPE C I. Note: This means that continuum drifts in oppo-
    site directions are not properly added: ALIGN Position must be specified
    for that.
\end{verbatim}

\subsubsection{Weighting type}
\begin{verbatim}        LAS\SET WEIGHT type

    Weighting to be used for summations: T[ime],  E[qual]  or  S[igma]  (for
    1/sigma**2).  The  default  is TIME. Sigma is not recommended unless you
    just made a baseline fit before. Equal weight behaves differently in AV-
    ERAGE and ACCUMULATE (which produces the sum).
\end{verbatim}

\subsubsection{Bad channels}
\begin{verbatim}        LAS\SET BAD Check

    Specify  the way ACCUMULATE and AVERAGE behaves in the case of bad chan-
    nels. Check may be OR or AND. Default is OR, i.e. the output channel  is
    bad if any of the input channels is bad.
\end{verbatim}

\subsubsection{Position matching}
\begin{verbatim}        LAS\SET MATCH Tol

    Turn on the checking of position-matching in ACCUMULATE and AVERAGE. Tol
    is  the  position tolerance in current units. The default is SET MATCH 2
    (arcsec). The tolerance is also used in FIND, MAP and a few  other  com-
    mands.
\end{verbatim}
\begin{verbatim}        LAS\SET NOMATCH

    Turn off checking of position-matching in ACCUMULATE and  AVERAGE.  This
    does not change the tolerance for the other functions like FIND and MAP.
\end{verbatim}

\subsubsection{Calibration matching}
\begin{verbatim}        LAS\SET CALIBRATION [Beam_Tol [Gain_Tol]] or OFF

    Specify the tolerance on the Beam efficiency (Beam_Tol) and  gain  image
    ratio (Gain_Tol), on turn or calibration checking. A value of 0 means no
    check. The default values are 0.02 and 0 respectively.

    These values are used by commands ACCUMULATE and AVERAGE to verify  that
    the calibration is consistant. They are also used by command WRITE which
    writes the corresponding information in the output file only if Beam_Tol
    is  non  zero. Note that because the beam efficiency is less than 1, you
    could use Beam_Tol=1 to suppress  the  calibration  checking  but  still
    write the information.
\end{verbatim}

%%%%%%
\section{Status of the \class\ releases}

\subsection{Bug fixes}

\subsubsection{{\tt Dec08}}
Following a bug reported by a \class\ user, it appeared that the
outputs from \ave\ and \accu\ in some exotic modes were incorrect. As
of the {\tt nov08} release of \gildas, only the following
configurations were fully \underline{correct}. These include the
default mode:
\begin{itemize}
\item {\tt ALIGN CHANNEL INTERSECT}, {\tt WEIGHT TIME}, {\tt BAD OR} (default)
\item {\tt ALIGN CHANNEL INTERSECT}, {\tt WEIGHT SIGMA|TIME}, any {\tt BAD}
\item {\tt ALIGN CHAN|VELO|FREQ INTERSECT}, {\tt WEIGHT SIGMA|TIME}, {\tt BAD OR}
\end{itemize}
Other configurations (in particular the composite mode) may have
produced correct output, but with no waranty since it was depending on
the input spectra and the level of complexity of the combination. This
bug was affecting the default \class\ (\ie\ \newclass), but \oldclass\
had a comparable level of bug for these commands.\\

This bug was fixed in the {\tt dec08} release of \newclass, by fully
cleaning and factorizing the code. \oldclass\ is not fixed according
to the policy concerning it: this package is obsolescent and is frozen
in order to keep a comparison point with \newclass.

\subsubsection{{\tt Jul09}}
The \accu\ and \ave\ commands perform many implicit features, and
perform silently non-trivial computations. When writing the current
document, the exhaustive check of all the combinations led to detect
two remaining bugs in the internal resampling loop.\\

These first configurations were incorrect:
\begin{itemize}
\item {\tt ALIGN VELO|FREQ COMPOSITE|INTERSECT}, {\tt WEIGHT EQUAL},
      any {\tt BAD}, on spectra with different resolutions
\end{itemize}
In this case a spectrum with a better resolution was overweighted
during the resampling process by a factor
$reso(resampled)/reso(input)$. This is not what user expects with an
{\tt EQUAL} weighting.\\

These other configurations were also incorrect:
\begin{itemize}
\item {\tt ALIGN CHAN|VELO|FREQ COMPOSITE}, any {\tt WEIGHT},
      {\tt BAD OR}
\end{itemize}
With this bug, \emph{no data available in one spectrum} could be
synonymous of \emph{bad data in this spectrum}. As a consequence a
piece of the output spectrum was wrongly flagged as bad with the
{\tt BAD OR} contamination rule. This behaviour was different from
the other modes where \emph{no data available in one spectrum}
implies to \emph{use data available in the other spectra}.\\

These two bugs were fixed in the {\tt jul09} release of \newclass.

\subsection{Test suites}
Consecutive to these bugs detection, it appeared that some attempts to
fix bugs in the old versions of \oldclass\ and \newclass\ had broken
other parts of the averaging code. Again, this is because of all the
different modes to take into account.\\

A set of procedures are now available as both demos and test
suites. They are in particular aimed to checked the influence of any
futur code modification:
\begin{itemize}
\item {\tt LAS90> @ gag\_demo:demo-accu [C|V|F|*] [I|C|*] [E|T|S|*] [A|O|*]}
\item {\tt LAS90> @ gag\_demo:demo-aver [C|V|F|*] [I|C|*] [E|T|S|*] [A|O|*]}
\end{itemize}
These procedures invoke respectively the \accu\ and \ave\ commands,
with Alignment, Range, Weight and Bad arguments. A star '*' will test
all the possible values for the considered argument. Default is to
check all the values and all the combinations.\\

These demos run on a set a spectra and try to combine them in
different ways.  The input spectra include: different intensities,
blanked channels, different abscissae definitions (ranges, reference
channels, resolutions), different parameters which influence the
weighting. On return the demos try to do their best to compute
themselves an average spectrum with can be compared to the output from
\accu\ or \ave. In particular, they invoke the {\tt RESAMPLE} command
of \class\ to perform their own resampling.

%%%%%%%%%%%%%%%%%%%
\section{Algorithm}

The \ave\ algorithm is based on a single run through the whole
index. The spectra are averaged two by two, each output sum becoming
an input spectrum when adding the next observation of the index.

This incremental method requires to associate a weight to the
output/input sum, in concordance with the number and weights of the
spectra already summed. Furthermore, this weight must be an array (one
weight per channel), because the command allows to combine different
spectra with different range of frequencies (\eg\ {\tt COMPOSITE}
mode), \ie\ not the same number of spectra may have contributed to
each channel in the output sum.

The \accu\ algorithm uses the same code but adds only two input
spectra ($R$ and $T$).\\

The algorithm described here apply only on spectra. The average of
continuum drifts rely on specific routines which are not considered
here.

% Setting the weights
\subsection{Setting the weights}
\label{txt:weights}

The Fortran type {\tt observation} in \class\ provides a weight array
({\tt obs\%dataw}) for each observation. It has the same dimension of
the abscissa and data arrays. One must take care that this array is
not part of the \class\ data format, \ie\ it is not saved in the
observation when it is written in the output file. As a consequence,
before adding any new spectrum, \ave\ and \accu\ will fill this array
according to the {\tt SET WEIGHT} method:
\begin{itemize}
\item {\tt EQUAL}: the weight array is filled with 1.0 values,
\item {\tt TIME}: the weight is set to
      $t \times f_{\emr{res}} / T_{\emr{sys}}^2$, where $t$ is the
      integration time in seconds, $f_{\emr{res}}$ the frequency
      resolution in Hertz, and $T_{\emr{sys}}$ the system temperature
      in Kelvin. If the integration time is null, an error is raised.
\item {\tt SIGMA}: the weight is set to $1/{\sigma}^2$, where $\sigma$
      is the rms noise in Kelvin. If it is null or not set
      \footnote{{\tt SIGMA} is usually set by the command {\tt BASE}.}
      in the observation header, an error is raised.
\end{itemize}
This value is unique for a single observation, and all the channels
have the same weight. Any previous values set in {\tt obs\%dataw} are
ignored and overwritten. On the other hand, the weight array of the
ongoing sum is preserved. This ensures its correct ponderation in
front of the input spectrum.\\

This method implies that all the weight arrays of the spectra of the
index are recomputed from scratch before addition. If a sum (returned
by a previous call to \ave\ or \accu) is used again as input, the
memory of all spectra it comes from (their number and their different
weights along all the channels) will be lost. One should take care
that \textbf{{\tt AVERAG}'ing the whole index leads into different
results than {\tt AVERAG}'ing it by part}! This is similar to first
calculate two weighted means but then calculate a new mean from these
without taking into account their weights.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Definition of the output spectrum}
\label{txt:params}

Let's denote $R$ and $T$ two spectra that one wants to add, and $S$
the output sum. With the \accu\ command, $R$ and $T$ are precisely the
corresponding buffers exposed to the user in \class{}. With the \ave\
command, $R$ is the average spectrum returned by the $(N-1)^{th}$
loop, and $T$ the $N^{th}$ spectrum in index, with its weight array
correctly set. We also call $r_R$ (resp. $r_T$) the resolution of $R$
(resp. $T$). This resolution is either in ``number of
channels''\footnote{ in this case the resolution is set to \emph{1
channel per channel}}, frequency, or velocity per channel depending on
the alignment mode ({\tt SET ALIGN CHANNEL|FREQUENCY|VELOCITY}
resp.).\\

From these, the output resolution is taken as the highest of the two
(absolute) resolutions in input:
\begin{equation}
  r_S = \emr{max}(|r_R|,|r_T|)
  \label{eq:resolution}
\end{equation}
This ensures that we never oversample a spectrum which has a low
resolution.\\

The range of abscissa in output spectrum depends on the combination
mode:
\begin{itemize}
  \item {\tt COMPOSITE} mode:
        \begin{eqnarray}
          x_{\emr{min},S} & = & \emr{min}( x_{\emr{min},R} , x_{\emr{min},T} ) \\
          x_{\emr{max},S} & = & \emr{max}( x_{\emr{max},R} , x_{\emr{max},T} )
        \end{eqnarray}
  \item {\tt INTERSECT} mode:
        \begin{eqnarray}
          x_{\emr{min},S} & = & \emr{max}( x_{\emr{min},R} , x_{\emr{min},T} ) \\
          x_{\emr{max},S} & = & \emr{min}( x_{\emr{max},R} , x_{\emr{max},T} )
        \end{eqnarray}
\end{itemize}

Given the output resolution and range, the number of channels $N$ can
be deduced:
\begin{equation}
  N_S = \emr{int}( \frac{x_{\emr{max},S}-x_{\emr{min},S}}{r_S} + 1.5),
\end{equation}
where the 1.5 value adds 1 or 2 extra-channels to avoid the erosion of
the spectra at the boundaries.\\

For the oncoming call of the addition routine, we also define the
abscissa at the channel $0$ used as reference:
\begin{equation}
   x_{\emr{val},S} = x_{\emr{min},S} - 1. \times r_S,
   \label{eq:reference}
\end{equation}
$x_{\emr{min},S}$ being the abscissa value of the first channel of the
output spectrum. Again, $x$ unit is either in ``channels'', velocity
or frequency depending on the {\tt SET ALIGN} mode.\\

This definition of the output spectrum is done only once with the
\accu\ command. On the other hand with the \ave\ command, the above
parameters are revised (if required) each time a new spectrum of the
index is added. For example, the output range
$[x_{\emr{min},S},x_{\emr{max},S}]$ and number of channels $N_S$ may
enlarge ({\tt SET ALIGN * COMPOSITE}) or reduce ({\tt SET ALIGN *
INTERSECT}). Also, the resolution is progressively reduced to the
coarsest of all the resolutions in the index
(eq.~\ref{eq:resolution}).

%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Aligned spectra}
In the simplest case, the spectra have all the same resolution,
reference channel and reference value: they are aligned and the
channel mode ({\tt SET ALIGN CHANNEL}) can be used. The intensity
(temperature) $T(i)$ and weight $w(i)$ of the output sum $S$ are
derived from a simple weighted mean of the input spectra:
\begin{center}
\fbox{
\begin{minipage}[t]{12cm}
\textsf{Averaged channel intensity and weight (aligned spectra):}
\begin{eqnarray}
  \label{eqn:align1}
  T_S(i) & = & \frac{w_R(i) \times T_R(i) + w_T(i) \times T_T(i) }
                    {w_R(i)               + w_T(i)} \\
  \label{eqn:align2}
  w_S(i) & = & w_R(i) + w_T(i)
\end{eqnarray}
\end{minipage}
}
\end{center}
On return, $S$ weight array can vary across the abscissae axis, \eg\
in {\tt COMPOSITE} mode the overlapping channels typically have a
double weight compared to the side channels.

%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Non-aligned spectra}
\label{txt:resampling}
When aligning the spectra by velocity or frequency ({\tt SET ALIGN
VELOCITY|FREQUENCY}), the input channels may not be aligned with the
output ones (because of different resolutions or different abscissae
at reference channels). A resampling must be performed in these cases.
Fig.~\ref{fig:align1} shows a generic example where the channels have
different size and are not aligned.

\begin{figure}[tbp]
  \centering
  \includegraphics[width=0.9\textwidth]{class-average-resampling}
  \caption{How the input spectra $R$ and $T$ are resampled into the
           intermediate spectra $R'$ and $T'$ resp. to match the
           desired output spectrum $S$ (general case). $R$ (resp. $T$)
           channels $j_{\emr{min}}$ to $j_{\emr{max}}$
           (resp. $k_{\emr{min}}$ to $k_{\emr{max}}$) have a partial
           or total contribution to $S(i)$.}
  \label{fig:align1}
\end{figure}

\subsubsection{Resampling}
Let's consider the input spectrum $R$. We assume that the intensity
$T_R$ at channel $i$ is a random variable which follows the normal
distribution $N(\mu_R,{\sigma_R}^2)$. Its probability density function
is:
\begin{equation}
  \emr{pdf}_R(x) = \frac{1}{\sigma_R \sqrt{2 \pi}} \,
                   \emr{exp} \left( - \frac{(x-\mu_R)^2}{2 {\sigma_R}^2} \right)
\end{equation}
where $\mu_R$ is the mean and $\sigma_R$ the standard deviation. The
variance of such a distribution is:
\begin{equation}
  \emr{var}(T_R(i)) = {\sigma_R}(i)^2
\end{equation}
Choosing a weight $w_R(i)$ of the following form:
\begin{equation}
  \label{eqn:var1}
  w_R(i) =  \frac{1}{{\sigma_R}(i)^2}
\end{equation}
ensure that the weighted mean is the maximum likelihood estimator of
the mean, under the assumption of independent (\ie\ uncorrelated at
first order) and normally distributed channel intensities with the
same mean.\\

Let's call $R'$ the resampled version of $R$ such as it is aligned
with the desired output spectrum $S$. The intensity of $R'$ at channel
$i$ can be written (see fig.~\ref{fig:align1}):
\begin{center}
\fbox{
\begin{minipage}[t]{12cm}
\textsf{Resampled channel intensity (all weights):}
\begin{equation}
  \label{eqn:resample1}
  T_{R'}(i) = \frac{
              \sum_{j=j_{\emr{min}}}^{j_{\emr{max}}} f_R(j) \times w_R(j) \times T_R(j)
                }
             {\sum_{j=j_{\emr{min}}}^{j_{\emr{max}}} f_R(j) \times w_R(j)}
\end{equation}
\end{minipage}
}
\end{center}
where:
\begin{itemize}
\item $j_{\emr{min}}$ and $j_{\emr{max}}$ are the channel range of
      input spectrum $R$ which are overlapped at least partially by
      the output channel number $i$,
\item $f(j)$ is the used fraction of the different input channels. It
      quantifies their contribution in the selected range:\\
      \begin{tabular}{ll}
        $ f(j) = 0 $      & if the input channel is out of the range, \\
        $ f(j) = 1 $      & if the output channel fully overlaps the input channel, \\
        $ 0 < f(j) < 1 $  & otherwise (proportionally to the overlap).
      \end{tabular}
\end{itemize}
We can also define the normalization factor $\beta$ and the weight
$\alpha(j)$:
\begin{eqnarray}
  \beta       = \sum_{j=j_{\emr{min}}}^{j_{\emr{max}}} f_R(j) \times w_R(j) \\
  \alpha_R(j) = \frac{f_R(j) \times w_R(j)}{\beta}
\end{eqnarray}
The normalization factor $\beta$ ensures that the total integrated
intensity is preserved during the resampling process. With these,
$T_{R'}(i)$ can be written:
\begin{equation}
  T_{R'}(i) =  \sum_{j=j_{\emr{min}}}^{j_{\emr{max}}} \alpha_R(j) \times T_R(j)
\end{equation}

\subsubsection{Channel weight of a resampled spectrum}
With the assumption of independent channel intensities, the weight
$w_{R'}(i)$ of this resampled channel can be derived from its
variance:
\begin{eqnarray}
  \emr{var}(T_{R'}(i)) & = & \emr{var} \left(
                \sum_{j=j_{\emr{min}}}^{j_{\emr{max}}} \alpha_R(j) \times T_R(j)
                                       \right) \\
  \label{eqn:var2}
                       & = & \sum_{ j=j_{\emr{min}}}^{j_{\emr{max}}}
                         \alpha_R(j)^2 \times \emr{var} \left( T_R(j) \right) \\
  \label{eqn:var3}
                       & = & \frac{1}{\beta^2}
                       \sum_{j=j_{\emr{min}}}^{j_{\emr{max}}} f_R(j)^2 \, w_R(j)
\end{eqnarray}
where eq.~\ref{eqn:var2} is derived from the general variance
property:
\begin{equation}
 \emr{var} (aX+bY) = a^2 \emr{var}(X) + b^2 \emr{var}(Y)
\end{equation}
where $a$ and $b$ are numerical constants and when $X$ and $Y$ are
independent random variables.

Finally from eqs.~\ref{eqn:var1} and \ref{eqn:var3}, the resampled
channel weight is\footnote{This is verified by numerical
simulations on spectra with known (simulated) rms noise.}:
\begin{center}
\fbox{
\begin{minipage}[t]{12cm}
\textsf{Resampled channel weight (weights {\tt TIME} and {SIGMA}):}
\begin{equation}
  \label{eqn:resample2}
   w_{R'}(i) = \frac{ \left( \sum_{j=j_{\emr{min}}}^{j_{\emr{max}}} f_R(j) w_R(j) \right) ^2}
                    {\sum_{j=j_{\emr{min}}}^{j_{\emr{max}}} f_R(j)^2 \, w_R(j)}
\end{equation}
\end{minipage}
}
\end{center}

\begin{figure}[tbp]
  \centering
  \includegraphics[width=0.7\textwidth]{class-average-shifted}
  \caption{Resampling of $R$ spectrum into ${R'}$, when the resolution is
           preserved but the reference channel is shifted by $0.0$ (1),
           $0.25$ (2) and $0.5$ (3) channel.}
  \label{fig:resampling}
\end{figure}

\vspace{2ex}
This relation has a non-intuitive effect on the resampled spectrum:
its weight is, in the general case, different from the original
one. Let's assume that the spectrum $R$ is resampled onto a spectrum
$R'$ with the same resolution but with a shifted value $x_{\emr{val}}$
at reference channel (fig.~\ref{fig:resampling}). In such a case, the
channel weight $w_{R'}$ we can deduce, and its associated $\sigma_{R'}$,
are:
\begin{center}
\begin{tabular}{llll}
1. & $x_{\emr{val}} \rightarrow x_{\emr{val}}+0.00$: &
     $w_{R'} = w_R$,                                 &
     $\sigma_{R'} = \sigma_R$                        \\
2. & $x_{\emr{val}} \rightarrow x_{\emr{val}}+0.25$: &
     $w_{R'} = 1.6 \times w_R$,                      &
     $\sigma_{R'} = \sigma_R / \sqrt{1.6}$           \\
3. & $x_{\emr{val}} \rightarrow x_{\emr{val}}+0.50$: &
     $w_{R'} = 2 \times w_R$,                        &
     $\sigma_{R'} = \sigma_R / \sqrt{2}$
\end{tabular}
\end{center}
assuming all the $R$ channels have the same weight (\ie\ same
$\sigma_R(j)$). The extra factor $w_{R'}/w_{R}$ affects either the
{\tt SIGMA} and the {\tt TIME} weightings. These examples show that,
depending on the desired resampling, the weight of the recomputed
spectrum may be different. In the two latter cases, a correlation has
been introduced between contiguous channels. From the physical point
of view, this can be explained from the fact that one resampled channel
contains more information than one original channel, \eg\ in case
number 3 it has an equal contribution of the two original ones: the
noise is reduced by a factor $\sqrt{2}$.

This should be kept in mind when averaging \eg\ two $R$ and $T$
spectra with same resolution but shifted X-axis. If a {\tt SIGMA}
weighting is invoked, the average won't be found at the mean distance
of the two spectra \textbf{even if their $\sigma$ are equal!}

One should also take care that eq.~\ref{eqn:resample2} assumes
\textbf{uncorrelated} input channels. Resampling a spectrum which
was already resampled (\eg\ $R' \rightarrow R''$) introduces
a correlation between more contiguous channels. In particular
this equation should not be used to compute the associated
weights.\\

The weight at eq.~\ref{eqn:resample2} apply to the {\tt TIME} and
{SIGMA} weighting, where the computations above have a physical
meaning and one can expect consistant values for integration time,
channel width and $\sigma$. For the {\tt EQUAL} weighting, user
expects the two input spectra to have the same weight whatever their
abscissa axis definition: this means that a re-{\tt EQUAL}ization of
the channels must come \emph{after} the resampling. The \emph{ad hoc}
weighting is in this case:
\begin{center}
\fbox{
\begin{minipage}[t]{12cm}
\textsf{Resampled channel weight (weight {\tt EQUAL}):}
\begin{equation}
  \label{eqn:resample_eq}
   w_{R'}(i) = \frac{ \sum_{j=j_{\emr{min}}}^{j_{\emr{max}}} f_R(j) w_R(j) }
                    { \sum_{j=j_{\emr{min}}}^{j_{\emr{max}}} f_R(j) }
\end{equation}
\end{minipage}
}
\end{center}
where $w_R(j)$ is either $1.0$ (see section~\ref{txt:weights}) for a
new input spectrum, or any (possibly not constant) value for the
reentrant sum. This preserves a correct ponderation of the reentrant
sum in front of a new input spectrum.

\subsubsection{Output spectrum}
The output intensity $T_S$ and weight $w_S$ at channel $i$ are set to:
\begin{center}
\fbox{
\begin{minipage}[t]{12cm}
\textsf{Averaged channel intensity and weight (non-aligned spectra):}
\begin{eqnarray}
  \label{eqn:value}
  T_S(i) & = & \frac{w_{R'}(i) \times T_{R'}(i) + w_{T'}(i) \times T_{T'}(i) }
                    {w_{R'}(i)                  + w_{T'}(i)} \\
  \label{eqn:weight}
  w_S(i) & = & w_{R'}(i) + w_{T'}(i)
\end{eqnarray}
\end{minipage}
}
\end{center}
where the $R'$ (resp. $T'$) index refers to the resampled version of
input spectrum $R$ (resp. $T$) computed from eqs.~\ref{eqn:resample1}
and \ref{eqn:resample2}/\ref{eqn:resample_eq}. The normalization in
eq.~\ref{eqn:value} ensures that the integrated intensity is also
preserved over the whole spectrum when averaging these two
spectra. These equations are the same as
eqs.~\ref{eqn:align1}/\ref{eqn:align2} (for aligned spectra) except
that they operate on their resampled versions.

% Data not available / bad channels
\subsection{Data not available / bad channels}
Equations \ref{eqn:value} and \ref{eqn:weight} describe the general
case when all the input channels which contribute to the output sum
are available. Nevertheless there may be some situations were data is
missing. In such cases, the general rule is to ignore the channels
with no data (\ie\ zero-weight them in
eqs.~\ref{eqn:align1}/\ref{eqn:value}), and to use data in the other
spectra when it is available:

\begin{itemize}
\item In the {\tt COMPOSITE} mode, if no data is available in an input
      spectrum (\ie\ out of its boundaries, in a frequency range covered
      by other input spectra), its corresponding resampled channels are
      zero-weighted.
\item In the {\tt COMPOSITE} mode, if no data is available in
      \textbf{all} the input spectra (\eg\ there is a gap covered
      by none of these), the corresponding channels in the output
      spectrum are flagged as bad.
\item In all modes, any bad channel in an input spectrum is
      zero-weighted. Additionally, if it contributes to the output
      channel ($f \ne 0$) but bad channels should propagate
      ({\tt SET BAD OR}), it contaminates the current output channel
      which is also set as bad.
\end{itemize}

%%%%%%%%%%%%%%%%%%%%%
\subsection{Controls}
There are several controls which ensures that user adds consistant
spectra. The first ones raise errors which stop the averaging process:
\begin{itemize}
\item The position compatibility is checked according to the position
      matching rule ({\tt SET [NO]MATCH}, or option {\tt /NOMATCH} for
      command \ave). Default is to match the positions with a
      2\arcsec\ tolerance.
\item The calibration compatibility is checked according to the beam
      efficiency and gain image ratio tolerances set by command {\tt
      SET CALIBRATION}. Default is to check the beam efficiencies with
      a 0.02 tolerance, but not the gain image ratios.
\item In the intersection mode ({\tt SET ALIGN * INTERSECT}), an error
      is raised if the input abscissae ranges do not intersect.
\end{itemize}
There is also a serie of conditions under which user is warned about
possible inconstent mix of spectra:
\begin{itemize}
\item for channel alignment of the input spectra ({\tt SET ALIGN
      CHANNEL}), a difference in the reference channels, in the
      frequency resolutions, or in the sky frequencies raises a
      warning,
\item for frequency alignment of the input spectra ({\tt SET ALIGN
      FREQUENCY}), a difference in the velocity offsets raises a
      warning (this indicates a different matching between the
      frequency and velocity axes),
\item for velocity alignment of the input spectra ({\tt SET ALIGN
      VELOCITY}), a difference in the rest frequencies raises a
      warning (for the same reason).
\end{itemize}

% Output parameters
\subsection{Output parameters}
\label{txt:output}
The main sections of the header contain some fundamental parameters
which are unique for all the spectrum. This is a problem for the
output spectrum from \accu\ or \ave\ which may contain different part
coming from a various number of spectra (e.g. in {\tt COMPOSITE}
mode). Actually, only the {\tt INTERSECT} mode with {\tt SET BAD OR}
ensures that the same number of input spectra have been used to
compute the non-bad channels of the output spectrum.\\

Nevertheless, typical values have to be given to these
parameters. First of all, when adding a new spectrum $obs$ with the
previous sum $S_{\emr{in}}$, the header of the output sum
$S_{\emr{out}}$ is completely copied from $obs$ header.  Then specific
values of the different sections of the header are adjusted.

\subsubsection{{\tt GENERAL} section}
The {\tt GENERAL} section is then revised:
\begin{itemize}
\item the integration time $t_{S_{\emr{out}}}$ is redefined with the
      sum of the two input integration times:
      \begin{equation}
        \label{eq:time}
        t_{S_{\emr{out}}} = t_{S_{\emr{in}}} + t_{\emr{obs}}
      \end{equation}
\item the system temperature $T_{\emr{sys},S_{\emr{out}}}$ is derived
      from the sum of $t \times |r| / T_{\emr{sys}}^2$ (where $r$ is
      here the frequency resolution):
      \begin{equation}
         \frac {t_{S_{\emr{out}}} \times |r_{S_{\emr{out}}}|}
               {T_{\emr{sys},S_{\emr{out}}}^2}
         =
         \frac {t_{S_{\emr{in}}}  \times |r_{S_{\emr{in}}}|}
               {T_{\emr{sys},S_{\emr{in}}}^2} +
         \frac {t_{\emr{obs}}     \times |r_{\emr{obs}}|}
               {T_{\emr{sys},\emr{obs}}^2}
      \end{equation}
\item the output observation number, version number, and scan number
      are arbitrarily reset to the $S_{\emr{in}}$ ones, \ie\ coming
      from the first observation in the index.
\end{itemize}

\subsubsection{{\tt SPECTROSCOPIC} section}

The {\tt SPECTROSCOPIC} section is revised as follows:
\begin{itemize}
\item from the first copy of all the input header of $obs$ into
      $S_{\emr{out}}$, the spectroscopic section inherits the $obs$
      values, in particular the rest and image frequencies.
\item the number of channels is set to the appropriate value
      (according to the {\tt COMPOSITE} or {\tt INTERSECT} mode)
      computed at section~\ref{txt:params}.
\end{itemize}
Then, depending on the alignment rule, another set of values can be
set:
\begin{itemize}
\item in case of channel alignment ({\tt SET ALIGN CHANNEL}), no more
      values of this section are redefined.

\item in case of frequency alignment ({\tt SET ALIGN FREQUENCY}), the
      frequency and associated velocity parameters are set:
      \begin{itemize}
      \item the output frequency resolution
            $f_{\emr{res},S_{\emr{out}}}$ is set to the value computed
            at Eq.~\ref{eq:resolution},
      \item the output frequency offset $f_{\emr{off},S_{\emr{out}}}$
            is set to $f_{\emr{off},S_{\emr{in}}}$,
      \item the reference channel associated to $f_{\emr{off}}$ is set
            to $(f_{\emr{off},S_{\emr{out}}} -
            x_{\emr{val},S_{\emr{out}}}) /
            f_{\emr{res},S_{\emr{out}}}$ where $x_{\emr{val}}$ is the
            frequency value at channel $0$\footnote{
            $f(i_{\emr{chan}}) = f_{\emr{res}} \times
             (i_{\emr{chan}}-r_{\emr{chan}}) + f_{\emr{off}}$
            which, at $i_{\emr{chan}} = 0$, can be written
            $x_{\emr{val}}     = f_{\emr{res}} \times
             (0       -r_{\emr{chan}}) + f_{\emr{off}}$
            },
            computed at Eq.~\ref{eq:reference},
      \item then the velocity resolution $v_{\emr{res},S_{\emr{out}}}$
            is set to $ -c \times f_{\emr{res},S_{\emr{out}}} /
            f_{\emr{rest},S_{\emr{out}}} $ where $c$ is the speed of
            light and $f_{\emr{rest}}$ the rest frequency,
      \item and the velocity offset $v_{\emr{off},S_{\emr{out}}}$ is
            set to $v_{\emr{off},S_{\emr{in}}}$
      \end{itemize}

\item in case of velocity alignment {\tt SET ALIGN VELOCITY}, the
      velocity and associated frequency parameters are set to:
      \begin{itemize}
      \item the output velocity resolution
            $v_{\emr{res},S_{\emr{out}}}$ is set to the value computed
            at Eq.~\ref{eq:resolution},
      \item the output velocity offset $v_{\emr{off},S_{\emr{out}}}$
            \footnote{even if $v_{\emr{off}}$ is always null in \class, the
            algorithm is as generic as possible.}
            is set to $v_{\emr{off},S_{\emr{in}}}$,
      \item the reference channel is set to
            $(v_{\emr{off},S_{\emr{out}}}-x_{\emr{val},S_{\emr{out}}}) / v_{\emr{res},S_{\emr{out}}}$
            where $x_{\emr{val}}$ is the velocity value at channel
            $0$, computed at Eq.~\ref{eq:reference},
      \item then the frequency resolution $f_{\emr{res},S_{\emr{out}}}$ is set to
            $ - f_{\emr{rest},S_{\emr{out}}} \times v_{\emr{res},S_{\emr{out}}} / c $,
      \item and the frequency offset $f_{\emr{off},S_{\emr{out}}}$ is
            set to $f_{\emr{off},S_{\emr{in}}}$
      \end{itemize}
\end{itemize}

\subsubsection{{\tt BASE} section}
In the {\tt BASE} section, the $\sigma$ value is reset to $0.0$
because, in most of the cases, it makes non-sense to provide a single
value for channels which may be the sum of different number of inputs.

\subsubsection{{\tt HISTORY} section}
This section is updated such as it contains the history of the scan
numbers of the input observations. This is available as a list of
sequences of consecutive scan numbers.

%%% Concatenation
\subsection{Concatenation}
The \ave\ command can be used to concatenate spectra of different
frequency ranges. This can be done by enabling the {\tt COMPOSITE}
mode ({\tt SET ALIGN VELO|FREQ COMPOSITE}).\\

All the rules presented in the different sections above apply without
exception:
\begin{itemize}
\item the output abscissa is computed as exposed in
      section~\ref{txt:params}. In particular, the output resolution
      is set to the coarsest resolution of all the spectra in the
      index (eq.~\ref{eq:resolution}), and the output range is the
      minimal which can contain all the input abscissae ranges,
\item the resampling (and averaging in overlapping parts) rules
      presented at section \ref{txt:resampling} apply,
\item the returned header is also set according to
      section~\ref{txt:output}.  In particular, the output integration
      time is the sum of all the times of input spectra
      (eq.~\ref{eq:time})\footnote{User is free is redefine at its
      convenience the header sections by using \eg\ {\tt SET VAR
      SPECTRO WRITE}}.
\end{itemize}

%%%%%%%%%%%%%%%%%%%%
\section{Discussion}
As raised in section~\ref{txt:weights}, the \class\ data format does
not provide a weight array, even if one is available in the Fortran
type {\tt observation} which is used to load in memory the
observations read from file. The {\tt observation} type is used only
when processing the data \eg\ in the current context of \ave\ and
\accu.

For consistency reasons over all the commands in \class\ which can
make use of the weight array, its content is assumed to be inexistant
or unset when entering the command: the weight array is recomputed
each time it is needed. This implies a unique value for all the
channels, \eg\ the $\sigma$ value computed by {\tt BASE}.\\

The discussions in this document raise the idea to save the weight
array in the \class\ data format. Indeed, we have seen that the single
value which is recomputed from scratch can not reflect the real weight
of each channel, \eg\ when the spectrum is the \ave\ of several others
in {\tt COMPOSITE} mode. But this has two major caveats:
\begin{itemize}
\item if it is introduced in the \class\ data format, the weight
      should properly be taken into account everywhere in \class, in
      each command which makes use of it, including \ave, \accu, and
      {\tt BASE}.
\item this implies an increased weight on disk for each spectrum: the
      amount of data written (data array + weight array) will
      typically double. This point has not been quantified in
      particular their relative weight compared to the observation
      header.

      For small amount of data (\eg\ with long integration times, line
      surveys,...), this would not be a problem. But when dealing with
      large quantity of data \eg\ multibeam OTF observations, the
      increase would become dramatic. We may need in this case to have
      the choice to save or not the weight arrays.
\end{itemize}

For the time being, saving (always or not) the weight array in the
\class\ data format remains an open question.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Conclusion}
In this document we fully described the current status of the \accu\
and \ave\ commands of \class, applied on spectra. They are very often
used, and for different purposes, \eg\ averaging a set of spectra or
concatenating them. User must be warned about several traps that he
may encounter:
\begin{itemize}
\item averaging a set of spectra with successive calls to \ave\ may
      lead into different results than averaging the whole set with
      a single call to the command,
\item averaging non-aligned spectra must invoke at one point a
      resampling. User can do it himself or rely on the internal
      resampling loops of the commands. The resampling has
      non-trivial effect of the weights.
\end{itemize}

As a solution to the first problem, the question of saving a weight
array (one weight per channel) in the \class\ data format has been
raised. This would have consequences in particular in term of disk
storage.
\end{document}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
