\documentclass{article}

\usepackage{graphicx}
\usepackage{fancyhdr}
\usepackage{multirow}
\usepackage{rotating}

\makeindex{}

\makeatletter
\textheight 625pt
\textwidth 460pt
\oddsidemargin 0pt
\evensidemargin 0pt
\marginparwidth 50pt
\topmargin 0pt
\brokenpenalty=10000

\newcommand{\ie} {{\em i.e.}}
\newcommand{\eg} {{\em e.g.}}
\newcommand{\cf} {cf.}

\newcommand{\gildas}    {\mbox {\bf GILDAS}}
\newcommand{\sic}       {\mbox {\bf SIC}}
\newcommand{\greg}      {\mbox {\bf GREG}}
\newcommand{\otfcal}    {\mbox {\bf OTFCAL}}
\newcommand{\mira}      {\mbox {\bf MIRA}}
\newcommand{\class}     {\mbox {\bf CLASS}}
\newcommand{\oldclass}  {\mbox {\bf CLASS77}}
\newcommand{\newclass}  {\mbox {\bf CLASS90}}

\newcommand{\analyse}{\texttt{ANALYSE}}
\newcommand{\fit}{\texttt{FIT}}
\newcommand{\fort}{\texttt{F90}}
\newcommand{\plait}{\texttt{PLAIT}}

% Makes fancy headings and footers.
\pagestyle{fancy}

\fancyhf{} % Clears all fields.

\lhead{\scshape Bardeau et al., 2009}
\rhead{\scshape \nouppercase{\rightmark}}   %
\fancyfoot[C]{\bfseries \thepage{}}

\renewcommand{\headrulewidth}{0pt}    %
\renewcommand{\footrulewidth}{0pt}    %

\renewcommand{\sectionmark}[1]{%
  \markright{\thesection. \MakeLowercase{#1}}}
\renewcommand{\subsectionmark}[1]{}

\fancypagestyle{firststyle}
{
   \fancyhf{}
   \fancyhead[C]{Original version at \tt http://iram-institute.org/medias/uploads/class-single-multiple.pdf}
   \fancyfoot[C]{\bfseries \thepage{}}
}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\begin{document}

\thispagestyle{firststyle}

\begin{center}
    \includegraphics[width=5cm]{gildas-logo}\\[2\bigskipamount]
  \huge{}
  IRAM Memo 2009-6\\
  Read-write optimization in \class\\[3\bigskipamount]
  \large{}
  S. Bardeau$^1$, J. Pety$^{1,2}$, S. Guilloteau$^3$\\[0.5cm]
  1. IRAM (Grenoble)\\
  2. Observatoire de Paris\\
  3. LAB, Observatoire de Bordeaux\\[\bigskipamount]
  \normalsize{}
  March, 13$^{th}$ 2009\\
  Version 1.0
\end{center}

\begin{abstract}
  \class{} is a \gildas{}\footnote{\texttt{http://www.iram.fr/GILDAS}}
  software for reduction and analysis of (sub)--millimeter
  spectroscopic data.  It is daily used to reduce spectra acquired
  with the IRAM 30m telescope. \class{} is currently used in many
  other facilities (\eg{} CSO, HHT, Effelsberg) and it is considered
  for use by Herschel/HIFI. \class{} history started in 1983. As a
  consequence, it was written in FORTRAN~77 and tailored to reduce
  pointed observations.

  When \class{} was re-written in Fortran~90, it was decided to split
  On-The-Fly spectra into one observation each\footnote{see IRAM memo
    2005-1: \emph{Improved OTF support} at {\tt
      http://iram-institute.org/medias/uploads/class-evol1.pdf}}. This
  eases spectra handling from the user side, but, of course,
  multiplies the number of observations in a single file (typically by
  a factor $\sim 1000$). The increasing number of observations
  (especially with the OTF acquisition schemes) and number of channels
  of the back-ends produces larger and larger files. \class{} has thus
  been revised with new file types and optimized with a faster
  algorithm in order to ensure a maximal data rate from the software
  side. We will expose here this new algorithm, and show that the
  major limitation for
  writing \class{} large files is currently on the hardware side.\\

  Keywords: On-The-Fly, \class{} file format, data rate, user and
  system times.

  Related documents: IRAM memo 2005-1: \emph{Improved OTF support}
\end{abstract}

%%%%%%%%%%
\newpage{}
\tableofcontents{}

%%%%%%%%%%
\newpage{}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Improved data format and algorithm}

\subsection{Historical status}
\label{txt:history}

In the original \class{} file format, any observation could have
different versions. When writing an observation $N$ to a \class{}
file, the algorithm had to reread the $N-1$ observations and all their
versions already written and check if a previous version of the same
observation was already present. If yes, the version number was
increased. This is typically a $N^2$ algorithm. On latest modern
computer, this started becoming a limiting factor (compared to the
OS/hardware incompressible time) when writing more than $10^5$ spectra
per file (see section~\ref{sec:benchmark}).

\subsection{Current version}
\label{txt:current}

On september, 29th 2008, an improvement was introduced on the
development branch of \class{} in its data format. It was officially
released in gildas dec08 version. There are now two types for the
\class{} files, both based on the standard \class{} data format. Each
type is associated to a new syntax of the command {\tt FILE OUT}:
\begin{itemize}
  \item {\tt FILE OUT Name MULTIPLE}: File of type MULTIPLE allows to
        have several version of the same spectrum (same
        ObservationNumber).
  \item {\tt FILE OUT Name SINGLE}: For file of type SINGLE, the
        ObservationNumber is a unique identifier of the spectrum: only
        one version exists.
\end{itemize}

The MULTIPLE type is the ``historical'' one for \class{}, but it has
been improved, though. Inserting a new observation or a new
observation version is done in a sorted list, which is a $N \times
\log{N}$ algorithm (behaving as $N$ when $N$ is large enough). The
counterpart is that the list of observations is sorted when opening an
output file with MULTIPLE type, but this is a minor cost compared to
the subsequent gain.\\

The SINGLE type has been introduced to simplify and speed up
On-The-Fly processing. When writing an observation, its number is
automatically set such as it is unique (or it must not exist if it is
provided). With this new feature, the algorithm has a $N$
dependency. This can be observed in the benchmarks performed
hereafter.

Another benefit of this new type is that merging two set of
observations in a SINGLE file ensures that all observations have a
different number. With the historical status, or with the MULTIPLE
type, two different spectra with the same number are merged under this
number but with different version.

\subsection{Cross-use of the different types}

\subsubsection{New types}
\begin{itemize}
  \item Re-opening an output file for appending new observations
        preserves its SINGLE or MULTIPLE type.
  \item Re-opening a MULTIPLE file for input+output is allowed, since
        several versions of the same observation is allowed.
  \item Re-opening a SINGLE file for input+output is not allowed:
        writing out a new version of an input spectrum is explicitely
        forbidden by the SINGLE type.
\end{itemize}


\subsubsection{Backward compatibility}

Opening for output a new file with the syntax {\tt FILE OUT NEW} is
now forbidden. This syntax is obsolete and produces an error.

Backward compatibility is ensured for reading and writing in old files
(created with an old version of \class{} with the syntax {\tt FILE OUT
NEW}): they are transparently opened with the MULTIPLE type.

\subsubsection{Transition}

During a transition time, users may encounter files with the new types
(\eg{} coming from the telescope) in their old versions of \class{}
(\eg{} sep08 or older). In such a case, MULTIPLE files are recognized,
as old type. On the contrary, SINGLE files are not recognized and
users must upgrade their \class{} version to a newer one.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Benchmark}
\label{sec:benchmark}

% % % % % % % % % %
\subsection{Method}

\begin{table}[htbp]
  \centering

\begin{tabular}{rr|*{12}{r}|}
\cline{3-14}
 & & \multicolumn{12}{|c|}{$N_{spectra}$} \\
\cline{3-14}
 & & 256 & 512 & 1024 & 2048 & 4096 & 8192 & 16384 & 32768 & 65536 & 131072 & 262144 & 524288 \\
\hline
\multicolumn{1}{|c|}{\multirow{14}{*}{\begin{sideways}$N_{channels}$\end{sideways}}} & \multicolumn{1}{|r|}{128} &    0.5 &    0.8 &    1.3 &    2.3 &    4.6 &    9.2 &   18.1 &   36.1 &   72.2 &  144.3 &  288.4 &  576.7 \\
\multicolumn{1}{|c|}{\multirow{14}{*}{}} & \multicolumn{1}{|r|}{256} &    0.7 &    1.0 &    1.8 &    3.3 &    6.6 &   13.2 &   26.1 &   52.1 &  104.2 &  208.4 &  416.5 &  833.0 \\
\multicolumn{1}{|c|}{\multirow{14}{*}{}} & \multicolumn{1}{|r|}{512} &    0.9 &    1.5 &    2.8 &    5.3 &   10.6 &   21.2 &   42.1 &   84.1 &  168.3 &  336.5 &  672.8 & {\bf    1.3} \\
\multicolumn{1}{|c|}{\multirow{14}{*}{}} & \multicolumn{1}{|r|}{1024} &    1.4 &    2.6 &    4.8 &    9.3 &   18.6 &   37.2 &   74.1 &  148.2 &  296.4 &  592.8 & {\bf    1.2} & {\bf    2.3} \\
\multicolumn{1}{|c|}{\multirow{14}{*}{}} & \multicolumn{1}{|r|}{2048} &    2.4 &    4.6 &    8.8 &   17.3 &   34.6 &   69.2 &  138.2 &  276.3 &  552.7 & {\bf    1.1} & {\bf    2.2} & {\bf    4.3} \\
\multicolumn{1}{|c|}{\multirow{14}{*}{}} & \multicolumn{1}{|r|}{4096} &    4.4 &    8.6 &   16.8 &   33.3 &   66.7 &  133.3 &  266.3 &  532.6 & {\bf    1.0} & {\bf    2.1} & {\bf    4.2} & {\bf    8.3} \\
\multicolumn{1}{|c|}{\multirow{14}{*}{}} & \multicolumn{1}{|r|}{8192} &    8.5 &   16.6 &   32.9 &   65.4 &  130.7 &  261.5 &  522.6 & {\bf    1.0} & {\bf    2.0} & {\bf    4.1} & {\bf    8.2} & {\bf   16.3} \\
\multicolumn{1}{|c|}{\multirow{14}{*}{}} & \multicolumn{1}{|r|}{16384} &   16.5 &   32.6 &   64.9 &  129.5 &  258.9 &  517.7 & {\bf    1.0} & {\bf    2.0} & {\bf    4.0} & {\bf    8.1} & {\bf   16.2} & {\bf   32.3} \\
\multicolumn{1}{|c|}{\multirow{14}{*}{}} & \multicolumn{1}{|r|}{32768} &   32.6 &   64.7 &  129.1 &  257.7 &  515.2 & {\bf    1.0} & {\bf    2.0} & {\bf    4.0} & {\bf    8.0} & {\bf   16.1} & {\bf   32.2} & {\bf   64.4} \\
\multicolumn{1}{|c|}{\multirow{14}{*}{}} & \multicolumn{1}{|r|}{65536} &   64.7 &  128.9 &  257.3 &  514.1 & {\bf    1.0} & {\bf    2.0} & {\bf    4.0} & {\bf    8.0} & {\bf   16.1} & {\bf   32.1} & {\bf   64.2} & - \\
\multicolumn{1}{|c|}{\multirow{14}{*}{}} & \multicolumn{1}{|r|}{131072} &  129.1 &  257.3 &  513.8 & {\bf    1.0} & {\bf    2.0} & {\bf    4.0} & {\bf    8.0} & {\bf   16.0} & {\bf   32.1} & {\bf   64.1} & - & - \\
\multicolumn{1}{|c|}{\multirow{14}{*}{}} & \multicolumn{1}{|r|}{262144} &  257.7 &  514.1 & {\bf    1.0} & {\bf    2.0} & {\bf    4.0} & {\bf    8.0} & {\bf   16.0} & {\bf   32.1} & {\bf   64.1} & - & - & - \\
\multicolumn{1}{|c|}{\multirow{14}{*}{}} & \multicolumn{1}{|r|}{524288} &  514.9 & {\bf    1.0} & {\bf    2.0} & {\bf    4.0} & {\bf    8.0} & {\bf   16.0} & {\bf   32.0} & {\bf   64.1} & - & - & - & - \\
\multicolumn{1}{|c|}{\multirow{14}{*}{}} & \multicolumn{1}{|r|}{1048576} & {\bf    1.0} & {\bf    2.0} & {\bf    4.0} & {\bf    8.0} & {\bf   16.0} & {\bf   32.0} & {\bf   64.1} & - & - & - & - & - \\
\hline
\end{tabular}

  \caption{\class{} file size for several combinations of number of
           spectra and number of channels per spectrum. Units are
           Megabytes (resp. Gigabytes) for regular font
           (resp. bold). Files are of type {\tt SINGLE}. The missing
           combinations were not performed because of the prohibitive
           file size.}
  \label{tab:size}
\end{table}

This benchmark measures the WRITE'ing capabilities of \class{}. A new
\class{} file is opened in the desired mode. A spectrum containing
$N_{channels}$ is generated thanks to the command {\tt
ANALYSE$\backslash$MODEL}, and is written $N_{spectra}$ times in the
file. The USER and SYSTEM times (described below) are measured at
typical stages. This is repeated for several values of $N_{channels}$
and for all output filetypes. Table~\ref{tab:size} shows the
combinations of $N_{channels}$ and $N_{spectra}$ which have been
benchmarked.\\

The tests were realized on an IRAM machine (64 bits, Fedora 6) with
\class{} compiled with the {\tt ifort} compiler (version 10), writing
on a local hard drive. For the old syntax {\tt FILE OUT NEW}, \class{}
version of the 22-sep-2008 was used. For the new syntax {\tt FILE OUT
SINGLE|MULTIPLE}, current version of \class{} was used.\\

 The different figures in this document aim to compare the
performances of \class{} when writing old and new file
types. Nevertheless, the SINGLE and MULTIPLE types provide the same
level of performance. The benchmark for the NEW, SINGLE, and MULTIPLE
types are thus presented only on Fig.~1. Later on, only the NEW and
SINGLE types are compared.

% % % % % % % % % % %
\subsection{File size}

One important parameter to keep in mind is the output file size which
becomes significant for several combinations of number of spectra and
number of channels per spectrum. Table~\ref{tab:size} shows the
observed file size for the benchmarked values of these
parameters. Typically, the file weights:

\begin{itemize}
  \item 1\hspace{2ex}  MByte\hspace{1ex} if $N_{spectra} \times N_{channels} \sim       130,000$
  \item 10\hspace{1ex} MBytes            if $N_{spectra} \times N_{channels} \sim     2,000,000$
  \item 100            MBytes            if $N_{spectra} \times N_{channels} \sim    25,000,000$
  \item 1\hspace{2ex}  GByte\hspace{1ex} if $N_{spectra} \times N_{channels} \sim   270,000,000$
  \item 10\hspace{1ex} GBytes            if $N_{spectra} \times N_{channels} \sim 2,700,000,000$
\end{itemize}

% % % % % % % % %
\subsection{Times}

\subsubsection{USER time}

The USER time is the time spent in \class{} and Gildas code in
general. \class{} developpers can act on this time by improving the
algorithms.\\

\begin{figure}[htbp]
  \centering
  \includegraphics[angle=270,width=0.9\textwidth]{class-single-multiple-time-nspec}
  \caption{SYSTEM and USER times for the three \class{} output files
           type: {\tt NEW} (old type), {\tt SINGLE} and {\tt MULTIPLE}
           (new type). Numbers indicate the number of channels per
           spectrum. Dashed lines are power law functions ($y \propto
           x^\alpha$) with their exponent.}
  \label{fig:all}
\end{figure}

Fig.~\ref{fig:all} displays the USER time as a function of the number
of spectra written, for several number of channels per spectrum and
for all the possible file types. We can observe that:
\begin{itemize}
  \item with the NEW (old) file type, for more than $\sim 10^4$
        spectra per file, the USER time was diverging. We can notice
        that the limit functions were not depending on the number of
        channels (lines converge), and we can observe the $N^2$
        dependency predicted at section \ref{txt:history}.
  \item the USER time was considerably improved thanks to the new file
        types. For high number of spectra, it is a power law which
        power was decreased from 2 to 1, as expected in section
        \ref{txt:current}.
\end{itemize}

\begin{figure}[htbp]
  \centering
  \includegraphics[angle=270,width=0.7\textwidth]{class-single-multiple-time-nchan}
  \caption{USER time for two \class{} output files type, but with time
           vs number of channels per spectrum. Left: keyword {\tt NEW}
           (old type), right: {\tt SINGLE} (new type). Numbers
           indicate the number of spectra written, and the dashed
           lines a linear increase of the time with the number of
           channels per spectrum.}
  \label{fig:all2}
\end{figure}

Fig.~\ref{fig:all2} shows the same USER time values, but the number of
channels per spectrum is now used as X-axis, for different number of
spectra written. We can confirm that:
\begin{itemize}
  \item the USER time is proportional to the number of channels, when
        this number is large enough,
  \item with the SINGLE and MULTIPLE file types, USER time is much
        lower than with the NEW file type.
\end{itemize}

\subsubsection{SYSTEM time}
The SYSTEM time is the time spent in system Input/Output. In this
context, it is roughly the time the Operating System spends to write
the file on buffer/disk.\\

From fig.~\ref{fig:all} and fig.~\ref{fig:all2} we can see that:
\begin{itemize}
  \item the SYSTEM time has not changed between old and new \class{}
        output file types. This was expected since the size of the
        output file has not significatively changed.
  \item the SYSTEM time is proportional to the number of spectra
        written.
  \item the SYSTEM time is proportional to the number of channel per
        spectrum, when this number is high (typically more 1024
        channels)
  \item on the other hand, there seems to be a threshold under which
        we cannot fall, e.g. writing spectra with 128 or 256 channels
        costs roughly the same SYSTEM time.
\end{itemize}
Finally, SYSTEM time is proportional to the amount of data written,
and thus the file size. This is particularly true when this amount is
high and hides over minor system actions.

\subsubsection{USER time vs SYSTEM time}

We aim to track here where are the limitations depending on the
context (number of channels per spectrum, number of spectrum per
file).\\

\begin{figure}[htbp]
  \centering
  \includegraphics[angle=270,width=0.8\textwidth]{class-single-multiple-2times-nspec}
  \caption{USER vs SYSTEM time for the NEW and SINGLE \class{} output
           file types.}
  \label{fig:system-user}
\end{figure}

Fig.~\ref{fig:system-user} shows that:
\begin{itemize}
  \item with the NEW file type, there was always a point where the
        USER time was equal to the SYSTEM time when writing more and
        more spectra (e.g. they were equal for 30,000 spectra of 4096
        channels).
  \item in the same context, USER time became a limiting factor a high
        number of spectra, e.g. it was $\sim1$ order of magnitude
        higher than the SYSTEM time when writing 524,288 spectra with
        4,096 channels.
  \item the above point is not true anymore with the current version
        of \class{}, and the USER time is allways lower than the
        SYSTEM time (typically a half with 4,096 channels per
        spectrum).
\end{itemize}

We can conclude that now (on the test machine), the SYSTEM time is the
limiting factor. The data rate is limited by both the hard disk driver
(software) and its writing speed (hardware). There is no easy way to
act on this time except from using a more performant hardware.

% % % % % % % % % % %
\subsection{Data rate}

\begin{figure}[htbp]
  \centering
  \includegraphics[angle=270,width=0.8\textwidth]{class-single-multiple-rate-nspec}
  \includegraphics[angle=270,width=0.8\textwidth]{class-single-multiple-rate-nchan}
  \caption{Top: data rate (number of spectra written in file per
           second) as a function of the number of spectra, for two
           \class{} output file types. Numbers indicate the number of
           channels per spectrum.
           Bottom: data rate as a function of the number of channels
           per spectrum. Numbers indicate the number of spectra
           written, and the dashed line a linear decrease of the rate
           with the number of channels.
           Left: {\tt NEW} (old file type), right: {\tt SINGLE} (new
           type).}
  \label{fig:rate}
\end{figure}

Finally, one of the main purpose of the new \class{} file types was to
increase the data rate to the output file. Fig.~\ref{fig:rate} shows
this rate for the old and new types. We can see (top-left) that, with
the old file type, this rate was decreasing when writing more and more
spectra, e.g. (with 16384 channels per spectrum) falling from 1300
spectra written per second (for the first thousands of spectra) to 300
when writing the $\sim$ 500,000ths. This is fixed with the new file
types: the data rate is constant over the number of spectra written
(top-right).\\

Again on the figure (bottom-right), with the new file types, we can
observe that the data rate is allways the same for a given number of
channels. For a low number of channels ($< 10,000$), we can observe
that the USER time has an effect on this rate. But for higher number
of channels, the rate is dominated by the SYSTEM time: it decreases
linearly with the amount of data written. Typical values are:
\begin{itemize}
  \item 10,000 spectra per second with 128 channels,
  \item  1,200 spectra per second with 16,384 channels,
  \item     20 spectra per second with 1,048,576 channels
\end{itemize}


%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Acknowledgments}

We wish to thank P.~Hily-Blant for useful discussion about the
improvement of the WRITE algorithm and its implementation.

\end{document}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
