\documentclass{article}

\usepackage{graphicx}
\usepackage{fancyhdr}
\usepackage{xcolor}

\makeindex{}

\makeatletter
\textheight 625pt
\textwidth 460pt
\oddsidemargin 0pt
\evensidemargin 0pt
\marginparwidth 50pt
\topmargin 0pt
\brokenpenalty=10000

% Makes fancy headings and footers.
\pagestyle{fancy}

\fancyhf{} % Clears all fields.

\lhead{\scshape Bardeau \& Pety, 2021}
\rhead{\scshape \nouppercase{\rightmark}}   %
\fancyfoot[C]{\bfseries \thepage{}}

\renewcommand{\headrulewidth}{0pt}    %
\renewcommand{\footrulewidth}{0pt}    %

\renewcommand{\sectionmark}[1]{%
  \markright{\thesection. \MakeLowercase{#1}}}
\renewcommand{\subsectionmark}[1]{}

% \fancypagestyle{firststyle}
% {
%    \fancyhf{}
%    \fancyhead[C]{Original version at \tt http://iram-institute.org/medias/uploads/class-herschel-fits.pdf}
%    \fancyfoot[C]{\bfseries \thepage{}}
% }

\newcommand{\jp}[1]{{\color{magenta} #1}}
\newcommand{\jpcomment}[1]{{\color{magenta} JP: #1}}

\newcommand{\ie} {{\em i.e.}}
\newcommand{\eg} {{\em e.g.}}
\newcommand{\cf} {cf.}

\newcommand{\gildas}    {\mbox {\bf GILDAS}}
\newcommand{\sic}       {\mbox {\bf SIC}}
\newcommand{\greg}      {\mbox {\bf GREG}}
\newcommand{\class}     {\mbox {\bf CLASS}}
\newcommand{\classic}   {\mbox {\bf CLASSIC}}
\newcommand{\mrtcal}    {\mbox {\bf MRTCAL}}

\newcommand{\emm}[1]{\ensuremath{#1}}   % Ensures math mode
\newcommand{\emr}[1]{\emm{\mathrm{#1}}} % Uses math roman fonts

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\begin{document}

% \thispagestyle{firststyle}

\begin{center}
  \includegraphics[width=5cm]{gildas-logo}\\[2\bigskipamount]
  \huge{}
  IRAM Memo 2021-?\\
  Weighting scheme in \class\\[3\bigskipamount]
  \large{}
  S. Bardeau$^1$, J. Pety$^{1,2}$\\[0.5cm]
  1. IRAM (Grenoble)\\
  2. Observatoire de Paris\\[\bigskipamount]
  \normalsize{}
  April, 7$^{th}$ 2021\\
  Version 1.0
\end{center}

\begin{abstract}
  The weighting scheme of \class{} was slightly inconsistent when summing
  or averaging spectra observed in different observing modes (frequency
  switching, wobbler switching, and position switching). A first attempt at
  solving this problem that was implemented on Sep. 17th, 2019, led to
  incorrect system temperatures in the output spectrum when averaging or
  summing folded frequency switched spectra. This in turn brake the
  associativity of the average or sum commands: When averaging three folded
  frequency switched spectra S1, S2, and S3, the noise level of (S1+S2+S3)
  was slightly different from the noise level of ((S1+S2)+S3)).

  In this memo, we describe the problem and propose a solution that was
  implemented in the mar21 release of \gildas{}. We ask all our users that
  currently use a version of \gildas{} between oct19 and feb21 to update
  their version to the newest one.

  Keywords: \class\ Data Format

  Related documents: Averaging spectra with \class{} (memo 2009-4),
  \mrtcal{} user manual.

\end{abstract}

\newpage{}
\tableofcontents{}

\newpage{}

\section{Weighting modes}

In \class, the {\tt SET WEIGHT} command offers 3 weighting modes (see
IRAM memo 2009-4):

\begin{itemize}
\item {\tt EQUAL}: the weights are set to $w_{\emr{E}} = 1$,
\item {\tt TIME}: the weights are set to $w_{\emr{T}} = t \times
  f_{\emr{res}} / T_{\emr{sys}}^2$, where $t$ is the integration time
  in seconds, $f_{\emr{res}}$ the frequency resolution in megaHertz, and
  $T_{\emr{sys}}$ the system temperature in Kelvin.
\item {\tt SIGMA}: the weights are set to $w_{\emr{S}} =
  10^{-6}/{\sigma}^2$, where $\sigma$ is the rms noise in
  Kelvin. $\sigma$ is usually pre-computed by the {\tt BASELINE} on
  channel ranges with no signal. The $10^{-6}$ is introduced to match
  the $\emr{s.MHz}/\emr{K}^2$ unit of the {\tt TIME} weight.
\end{itemize}
These are relative weights between spectra to be combined in commands
like {\tt AVERAGE}, {\tt XY\_MAP}, etc. Absolute weights involve an
additional constant factor which is not described here. The weighting
modes can not be mixed: when combining spectra, all weights are {\tt
  EQUAL}, or all weights are {\tt TIME}, or all weights are {\tt
  SIGMA}. Note that the {\tt TIME} weight is a theoretical value based
on parameter values taken at observation time, while the {\tt SIGMA}
is a measured value based on the actual noise property of the
spectrum.

\section{Switching modes}

\class\ supports 3 major switching modes (see the \mrtcal{} user
documentation for a more complete description):
\begin{itemize}
\item position switching (abbreviated into psw),
\item wobbler switching (abbreviated into wsw),
\item frequency switching (abbreviated into fsw).
\end{itemize}
The frequency switching mode can be divided into 2 subcategories:
\begin{itemize}
\item unfolded frequency switching (abbreviated ufsw),
\item folded frequency switching (abbreviated ffsw).
\end{itemize}

Up to sep19 \gildas\ version, \class\ used the same weighting definitions
for all switching modes. In particular for the {\tt TIME} weights
\begin{equation}
  \label{eq:wsep19}
  w_{\emr{T,psw}} = w_{\emr{T,wsw}} = w_{\emr{T,ufsw}} = w_{\emr{T,ffsw}} = w_{\emr{T}},
\end{equation}
where $w_{\emr{T}}$ is defined above.

\section{Fixing folded frequency switching weight}

Looking closer to frequency switching, the folding operation folds a
spectrum on itself to shift and average the two phases into a single
line. This operation decreases the noise level by a factor $\sqrt{2}$
at the frequency of the line. Moreover, when doing this, \class\ keeps
constant key observing parameters like the observing time to remain
consistent with the actual observing run at the telescope. From
Eq.~\ref{eq:wsep19}, this means that an unfolded frequency switching
spectrum and its folded version wrongly had the same theoretical
noises (and, thus, the same {\tt TIME} weights), while they correctly
had different measured noise (and different {\tt SIGMA} weights).

This inconsistency between unfolded fsw and folded fsw spectra was in
general OK, as \class\ commands refuse to combine these two kinds of
spectra. However, it is possible to combine (average, sum, ...)
folded fsw spectra with psw and wsw. As the theoretical folded fsw
noise does not reflect the actual noise, the relative {\tt TIME}
weights were incorrect and the spectra from different switching modes
were not combined properly.

This problem was fixed on Sep. 17th, 2019, and the patch was first included
in the oct19 \gildas\ release. Starting from that date, folded frequency
switching spectra were given an additional factor 2 to their {\tt TIME}
weight:
\begin{equation}
  w_{\emr{T,ffsw}} = 2 \times w_{\emr{T}}
\end{equation}
This has no effect when mixing folded fsw spectra altogether (as all the
relative weights include the same extra factor), but this fixes the
combination of folded fsw with other allowed switching modes.

\section{Fixing the {\tt AVERAGE} output system temperature}

The commands {\tt AVERAGE}, {\tt ACCUMULATE}, and {\tt STITCH} combine
the spectra available in the current index according to their relative
weights, and produce a single output spectrum with a proper
description in terms of observing time, resolution, and system
temperature.

Assuming the simplest case of a set of $N$ spectra with identical
observing time, resolution, and system temperature, the output
spectrum has a resolution identical to the input one, and an observing
time which is the sum of all the inputs one. The output system
temperature is then derived by reverting the theoritical time weight
formula
\begin{equation}
  w_{\emr{out}} = \sum_N w_{\emr{in}}
\end{equation}
\begin{equation}
  T_{\emr{sys,out}} = w_T^{-1}(w_{\emr{out}},f_{\emr{res,out}},t_{\emr{out}})
\end{equation}
When averaging folded fsw with {\tt TIME} weighting, this gives:
\begin{equation}
  w_{\emr{T,out}} = 
  \sum_N w_{\emr{T,ffsw}} = N \times 2 \times t_{\emr{in}} \times f_{\emr{res}} / T_{\emr{sys,in}}^2 = 
  2 \times t_{\emr{out}} \times f_{\emr{res}} / T_{\emr{sys,in}}^2
\end{equation}
If we reverse the generic $w_{\emr{T}}$ formula, this gives an
unexpected change of system temperature:
\begin{equation}
  T_{\emr{sys,out}} = \frac{T_{\emr{sys,in}}}{\sqrt{2}}
\end{equation}

In other words, when the factor 2 was introduced in 2019, it was not
added symetrically in the $w_T^{-1}$ reverse operation done by {\tt
  AVERAGE}. The result was that when all the averaged spectra were
folded fsw, the output Tsys was underestimated by a factor
$\sqrt{2}$. As of 24-feb-2021 and release mar21, this is fixed now:
\begin{equation}
  w_{\emr{T,ffsw}}^{-1} = \frac{1}{2} \times w_{\emr{T}}^{-1}
\end{equation}

This reverse function is used when mixing folded fsw spectra
only. When mixing psw or wsw spectra only, the generic $w_{T}^{-1}$
function is used. Finally, when mixing folded fsw with psw or wsw, the
output Tsys is more complicated to evaluate, as there is no unique
weight formula to revert. The chosen solution is to use also the
generic $w_{\emr{T}}^{-1}$ function, leading to an approximate evaluation of
the output system temperature. In this latter case, we also introduce
a \emph{mix switching mode} (reflecting the combination of spectra
observed in different modes), and the associated section in the
spectrum header is emptied except for this code.

For example, when mixing a one psw or wsw spectrum with one fsw
spectrum of same system temperature, integration time, and frequency
resolution, this gives:
\begin{eqnarray}
w_{\emr{T,psw}} & = & t_{\emr{in}} \times f_{\emr{res}} / T_{\emr{sys,in}}^2 \\
w_{\emr{T,ffsw}} & = & 2 \times t_{\emr{in}} \times f_{\emr{res}} / T_{\emr{sys,in}}^2 \\
w_{\emr{T,mix}} & = & t_{\emr{out}} \times f_{\emr{res}} / T_{\emr{sys,mix}}^2
\end{eqnarray}
with $t_{\emr{out}} = 2 \times t_{\emr{in}}$ (sum of input integration
times). Since $w_{\emr{T,mix}} = w_{\emr{T,psw}}+w_{\emr{T,ffsw}}$,
this yields:
\begin{equation}
  T_{\emr{sys,mix}} = \sqrt{\frac{t_{\emr{out}} \times f_{\emr{res}}}{w_{\emr{T,psw}}+w_{\emr{T,ffsw}}}} =
  \sqrt{\frac{2 \times t_{\emr{in}}}{3 \times t_{\emr{in}} / T_{\emr{sys,in}}^2}} =
  T_{\emr{sys,in}} \times \sqrt{\frac{2}{3}}
\end{equation}
We see that the resulting system temperature is underestimated because
the reverse weight formula is not ideal in the mixed case.

\section{Consequences}

For all versions of \gildas{} between oct19 and feb21
\begin{itemize}
\item Summing, averaging, or stitching spectra whose observing mode is
  either position switching, wobbler switching, or \emph{unfolded}
  frequency switching gives the expected output spectrum.
\item Summing, averaging, or stitching \emph{folded} frequency switching
  spectra could lead to slightly incorrect noise estimations in some
  specific conditions.
  \begin{itemize}
  \item If the user is summing, averaging, or stitching all the
    spectra in a single command, the output spectrum intensities and
    their associated noise level are behaving as expected.
  \item Only when the user is summing, averaging, or stitching
    \emph{folded} frequency switching spectra in subsequent calls to
    the {\tt AVERAGE}, {\tt ACCUMULATE}, or {\tt STITCH} commands
    would lead to incorrect noise levels. The shape of the line is
    nevertheless preserved.
  \end{itemize}
\end{itemize}

\section{Test data}

We test here the {\tt AVERAGE} command behavior and the noise
properties on a frequency switch dataset. Here we use the 3 frequency
switch tracked spectra observed on 03-jun-2013, scans 100, 101, and
102, on the Vespa 4 backend part ({\tt 30ME2VLI-V04}).

\begin{table}[htb]
\begin{center}
  \caption{Status of the raw spectra w/t their theoretical and
    measured noise in all \class\ versions. $t$ is the integration
    time, $f_{\emr{res}}$ is the frequency resolution. The theoretical noise
    is computed from $T_{\emr{sys}}$, $t$, and $f_{\emr{res}}$ (in Hz): $\sigma =
    T_{\emr{sys}}/\sqrt{t \times f_{\emr{res}}}$. The measured noise is computed
    by the {\tt BASELINE} command, excluding the windows with signal.}
  \vspace{1ex}
\begin{tabular}{c|cccc|c|c}
\label{tab:raw}
Scan     & Tsys   &    t   &  $f_{\emr{res}}$  & Theoretical noise & Measured noise & Status \\
         &  (K)   &   (s)  &   (MHz)    &    (K)            &    (K)         &        \\
\hline
100      & 224.66 &  56.61 &   0.0195   & 0.214             & 0.219          &   OK   \\
101      & 224.53 &  56.61 &   0.0195   & 0.214             & 0.220          &   OK   \\
102      & 224.40 &  56.61 &   0.0195   & 0.213             & 0.217          &   OK   \\
\end{tabular}
\end{center}
\end{table}

\begin{table}[htb]
\begin{center}
  \caption{Same as Table~\ref{tab:raw}, for the averaged spectra
    produced by the command {\tt AVERAGE} in \class\ version feb21. We
    combine the 3 spectra either in a single call (100+101+102), or in
    2 calls with an intermediate average (100+101, then
    (100+101)+102). This \class\ version gives unexpected results (in
    red). 1) After a single average, the system temperature, and thus the
    theoretical noise, is underestimated by a factor $\sqrt{2}$, but the
    measured noise is correct as the weighted average used proper weights.
    2) when averaging in several calls, the error on the system temperature
    propagates also to the weighted average; we can show that the measured
    noise is divided by $\sqrt{14/5}$ instead of $\sqrt{3}$ ($3.5\%$
    overestimate) after the 2-average sequence. }
  \vspace{1ex}
\begin{tabular}{c|cccc|c|c}
Average & Tsys   &    t   &  $f_{\emr{res}}$  & Theoretical noise & Measured noise & Status \\
\hline
100+101+102   & \color{red}158.77 & 169.84 & 0.0195 & \color{red}0.087 & 0.127            & Unexpected$^1$ \\
100+101       & \color{red}158.81 & 113.23 & 0.0195 & \color{red}0.107 & 0.154            & Unexpected$^1$ \\
(100+101)+102 & \color{red}122.99 & 169.84 & 0.0195 & \color{red}0.068 & \color{red}0.131 & Unexpected$^2$
\end{tabular}
\end{center}
\end{table}

\begin{table}[htb]
\begin{center}
  \caption{Same as Table~\ref{tab:raw}, for the averaged spectra
    produced by the command {\tt AVERAGE} in \class\ version mar21.}
  \vspace{1ex}
\begin{tabular}{c|cccc|c|c}
Average     & Tsys   &    t   &  $f_{res}$  & Theoretical noise & Measured noise & Status \\
\hline
100+101+102    & 224.53 & 169.84 &   0.0195   & 0.123             & 0.127          &   OK   \\
100+101        & 224.59 & 113.23 &   0.0195   & 0.151             & 0.154          &   OK   \\
(100+101)+102  & 224.53 & 169.84 &   0.0195   & 0.123             & 0.127          &   OK
\end{tabular}
\end{center}
\end{table}

\end{document}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
