\documentclass[11pt]{article}

\usepackage{graphicx}
% \usepackage{pdfpages}
\usepackage{fancyhdr}
\usepackage{listings}
\usepackage{xcolor}

% GILDAS specific definitions
\include{gildas-def}

\makeindex{}

\makeatletter
\textheight 625pt
\textwidth 460pt
\oddsidemargin 0pt
\evensidemargin 0pt
\marginparwidth 50pt
\topmargin 0pt
\brokenpenalty=10000

% Makes fancy headings and footers.
\pagestyle{fancy}

\fancyhf{} % Clears all fields.

\lhead{\scshape Bardeau, de Souza, Pety, 2020}
\rhead{\scshape \nouppercase{\rightmark}}   %
\fancyfoot[C]{\bfseries \thepage{}}

\def\uvfit{UV\textbackslash FIT}

\newcommand{\cube} {\texttt{CUBE}}

\renewcommand{\headrulewidth}{0pt}    %
\renewcommand{\footrulewidth}{0pt}    %

\renewcommand{\sectionmark}[1]{%
  \markright{\thesection. \MakeLowercase{#1}}}
\renewcommand{\subsectionmark}[1]{}

\lstset{language=[90]Fortran,
  basicstyle=\ttfamily,
  keywordstyle=\color{blue},
  commentstyle=\color{gray},
  frame=single,
  tabsize=2,
  morecomment=[l]{!\ }% Comment only with 
}

\fancypagestyle{firststyle}
{
   \fancyhf{}
   \fancyhead[C]{Original version at \tt http://iram-institute.org/medias/uploads/mapping-for-developers.pdf}
   \fancyfoot[C]{\bfseries \thepage{}}
}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\title{CUBE developers documentation}
\author{S.~Bardeau$^{1}$, V.~de~Souza~Magalhaes$^{1}$, J.~Pety$^{1,2}$\\
 \vspace{0.3cm}
  1. IRAM\\
  2. Observatoire de Paris
 }
\makeindex{}

\begin{document}

\maketitle

%\begin{rawhtml}
%  Note: this is the on-line version of the ``GREG 2012 UV table format''
%  document for Gildas users and programmers.
%  A <A HREF="../../pdf/greg-uvt-v2.pdf"> PDF version</A>
%  is also available.
%
%  Related information is available in:
%  <UL>
%  <LI> <A HREF="../greg-html/greg.html"> GREG:</A> Grenoble Graphics
%  <LI> <A HREF="../sic-html/sic.html"> SIC:</A> Simple Interpretor of Commands
%  </UL>
%\end{rawhtml}

\begin{abstract}
  Some abstract.
\end{abstract}

\newpage
\tableofcontents{}

\newpage

\section{User workflow}

User workflow is not yet finalized in \cube. This depends on the way
we deal with cubes in memory.

\section{IO}

\subsection{Nomenclature}

In the IO library, cubes are considered either as a collection of
\emph{channels} (a channel being a 2D subset along the V dimension),
or as a collection of \emph{pixels} (a pixel being a 1D subset along
the XY dimensions). Other actors might use different names for a
similar object: channels can be seen as 2D position-position images,
and pixels as 1D spectra (collection of scalar channels for a given XY
position).

\subsection{Data ordering and contiguous access}

Position-position-spectroscopy cubes can be stored in memory or disk
in several different ways. Only two major orders are offered as they
cover all the use-cases (see Fig~\ref{fig:cube-io-ordering}):
\begin{enumerate}
\item LMV: the spatial dimensions come first, channels or range of
  channels are contiguous subset of the cube;
\item VLM: the spectroscopic dimension comes first, pixels or ranges
  of pixels are contiguous subset of the cube.
\end{enumerate}
Each order is the transposition of the other one.

\begin{figure}[htb]
  \centering
  \includegraphics[width=0.45\textwidth]{cube-lmv}
  \includegraphics[width=0.45\textwidth]{cube-vlm}
  \caption{Left: LMV-ordered cube. Right: VLM-ordered cube. Both cubes
    are represented as a juxtaposition of LM (XY) images (one image
    per channel). In both figures, orientation is chosen so that the
    first (resp. last) element in memory or disk is displayed at the
    bottom-left-front (resp. top-right-back) corner. Contiguity goes
    from left to right, then bottom to top, then front to back. Red
    lines illustrate the contiguity path. The grey planes are
    contiguous 2D subsets.}
  \label{fig:cube-io-ordering}
\end{figure}

In order to support both memory and disk modes (see
subsection~\ref{txt:access-modes}), the IO library only offers access
to the data through subset objects, namely channels or pixels. For
best efficiency, accesses should be done\footnote{The IO library
  offers the possibility to read a non-contiguous object, e.g. a
  channel from a VLM cube, but this is possible only in memory mode,
  and this is less efficient than the contiguous access} with the
object which is contiguous for the given cube (i.e. channels for LMV
cubes, pixels for VLM cubes).

\subsection{Memory and disk modes}
\label{txt:access-modes}

\begin{figure}[htb]
  \centering
  \includegraphics[width=0.80\textwidth]{cube-io-blocks}
  \caption{How to IO library defines the current block to be
    processed. Left: in the memory mode, the whole cube is loaded in
    memory and is processed as a single block. Right: in the disk
    mode, a contiguous subset of the cube is loaded in memory; there
    will be several blocks to be processed to cover the full
    cube. Note that all the blocks may not have strictly the same size
    due to rounding issues in the general case.}
  \label{fig:cube-io-blocks}
\end{figure}

The IO library is thought to offer IO accesses to cubes of arbitrary
large sizes\footnote{More precisely: the whole 3D cube might not fit
  in memory, as long as at least a 2D subset along the third dimension
  does.}. Internally, the IO library can:
\begin{itemize}
\item store the whole cube in memory at once (\emph{memory
    mode}). This is suited for cubes smaller than a user-defined
  limit,
\item store contiguous subsets of the cube (\emph{disk
    mode}). \emph{Contiguous} being, as described in
  Fig~\ref{fig:cube-io-ordering}), a contiguous range of channels (LMV
  cubes) or a contiguous range of Y planes (VLM cubes). The size of
  the resulting block is ruled by the command {\tt
    SET$\backslash$BUFFERING /BLOCK}.
\end{itemize}
The figure~\ref{fig:cube-io-blocks} illustrates how the same cube can be
divided in blocks depending on the mode.

\subsection{Serial and parallel executions}
\label{txt:execution-modes}

\begin{figure}[htb]
  \centering
  \includegraphics[width=0.80\textwidth]{cube-io-tasks}
  \caption{How to IO library defines the tasks to be executed. Left:
    in case of serial execution, the current block is treated entirely
    by a single task (a single processor thread).  Right: in case of
    parallel execution, the current block is divided in contiguous
    subsets, each of them given for processing to a different parallel
    task.}
  \label{fig:cube-io-tasks}
\end{figure}

CUBE supports parallelization for both memory and disk modes. This is
achieved thanks to the blocks (as explained in
section~\ref{txt:access-modes}), as the IO library ensures a block is
entirely loaded in memory for processing. Once this block is
available, parallelization is performed with \emph{OpenMP tasks} (see
section~\ref{txt:parallelization}). Once the current block has been
processed, it is flushed and a new one (if any) is loaded in memory
and processed again by several tasks.

Note that the serial case is treated as a particular case of the
parallelization: a single \emph{pseudo-task} is defined to process the
current block by a single thread (Fig.~\ref{fig:cube-io-tasks}, left).

\subsection{Access to data by entries}
\label{txt:entries}

\begin{figure}[htb]
  \centering
  \includegraphics[width=0.80\textwidth]{cube-io-entries}
  \caption{The 3 data granularity offered to the external API.}
  \label{fig:cube-io-entries}
\end{figure}

The IO external API does not expose any difference between the disk
and memory modes, and between the serial and parallel execution. The
programmer just has to write the code which processes the data
associated to a task. This data can accessed with 3 granularity
levels:
\begin{itemize}
\item \emph{Image} access: the data is accessed (get-modify-put
  sequence) through a 2D contiguous object (channel/image); this
  access is suited for image-related processing like reprojection of
  LMV cubes.
\item \emph{Spectrum} access: the data is accessed through a 1D
  contiguous object (pixel/spectrum); this access is suited for
  spectrum-related processing like frequency resampling of VLM cubes.
\item \emph{Subcube} access: the data is accessed through a 3D
  contiguous object; this access is suited for processing where data
  order is not relevant (\eg\ data rescaling when converting its
  unit). The larger granularity minimizes the overheads of the 2
  previous accesses.
\end{itemize}
Given the granularity chosen by the programmer, a task can be
considered as collection of entries to be processed.

\section{Parallelization}
\label{txt:parallelization}

\subsection{Defining the tasks}

\begin{figure}[p]
  \centering
  \includegraphics[width=0.55\textwidth]{cube-parallelization-1}
  \caption{How the \emph{adm} library in CUBE fills the \emph{OpenMP
      task pool}. In short, a master thread divides the current block
    in smaller parts, and feeds them to OpenMP for \emph{later}
    processing by the parallel threads. In this example, the block is
    divided in 5 tasks.}
  \label{fig:cube-parallelization-1}
\end{figure}

The \emph{adm} library divides the current block (see
section~\ref{txt:access-modes}) in contiguous subsets, which size is
ruled by the command {\tt SET$\backslash$BUFFERING
  /PARALLELIZATION}\footnote{This size should be large enough to avoid
  domination of task overheads, but small enough to ensure many tasks
  can be defined for parallel execution.}. From these subsets, a
master thread fills (almost) instantaneously the \emph{OpenMP Task
  Pool} with all the tasks needed to process the current block. There
can be (much) more tasks than computing threads, this is part of the
OpenMP tasks design.

Note that, while the \emph{adm} library has the charge of dividing the
input and output blocks, reading them, flushing them, etc, it does not
have the responsibility to execute the tasks: it is the OpenMP
responsibility (see next section).

\clearpage

\subsection{Parallel execution of the tasks}

\begin{figure}[htb]
  \centering
  \includegraphics[width=0.7\textwidth]{cube-parallelization-2}
  \caption{How OpenMP distributes the tasks for execution by the
    threads. In this example, 3 threads are requested to execute 5
    tasks.}
  \label{fig:cube-parallelization-2}
\end{figure}

As soon as there is one or more tasks waiting for execution in the
task pool, the OpenMP engine starts to address them for execution by
parallel threads. The parallelization precisely comes at this level of
the code. CUBE has no control on the tasks execution. In particular,
there is no predictable order for executing them; which thread will
execute which task is unknown. Threads are working as long as there
remains tasks to be executed in the pool. Once all the tasks have been
distributed to the threads, the OpenMP engine waits for them all to
finish their processing. When this is done, hand comes back to the adm
library to process a new block by a new pool of tasks (see
section~\ref{txt:execution-modes}).

\subsection{Execution of one task}

\begin{figure}[htb]
  \centering
  \includegraphics[width=0.70\textwidth]{cube-parallelization-3}
  \caption{How each thread processes the data in the task. This relies
    on the entry kind, with internal loop in the task.}
  \label{fig:cube-parallelization-3}
\end{figure}

Within a given task executed by one thread, the processing runs
serially. For example, if the task provides N images, the task
internal loop will process the N images in order, one after the
other. Same rule applies for spectra within a task. Regarding
subcubes, they are meant to be treated as a whole entity within the
task.

\section{Conventions}

\subsection{Frequency vs Velocity}

In radio astronomy it is very common to use velocity in km/s as the
unit of spectral axes. However the sampling of the signal by a radio
telescope is done on frequency, and hence the native unit of the
spectral axes. All velocity axes are hence derived from an original
frequency axis based on a linear transformation that depends on a rest
frequency.

Due to this we have taken the decision to privilege frequency over
velocity when dealing with spectral axes in \cube{}. An example is the
derivation of the velocity axis in the header types. All the velocity
related quantities are computed from the frequency related ones and
not taken directly from the header of the file. Another example is the
spectral stitching of cubes, which is always done in frequency.

\subsection{Axis descriptions}

In GILDAS we use 4 values to describe a linear axis, be it a spatial
or a spectral axis. These values are:
\begin{itemize}
\item[$n$] The number of elements in the axis (long)
\item[$ref$] The reference pixel position (double)
\item[$val$] The value of the reference pixel in the axis unit (double)
\item[$inc$] The increment between two elements of the axis in the axis unit (double)
\end{itemize}

From these values the elements of an axis can be reconstructed
according to the following equation:
\begin{equation}
  ax[i] = (i-ref)*inc+val, \forall i \in \mathbf{N} < n
\end{equation}
Where $ax[i]$ is the $i$-eth element of the axis.

In the case of spectral axes, we use the convention that the reference
channel will be the same for both the velocity and frequency
axis. $val$ will be the Systemic velocity in the velocity axis and the
rest frequency at the systemic velocity in the case of the frequency
axis.

In the case of spatial axes, the convention is that these are always
offset axes, i.e. $val$ is always 0.0 and reflects the position of the
projection center.

In both cases $ref$ is a double as the reference pixel/channel does
not need to precisely coincide with an actual pixel/channel.

\section{Data Commands Structure}

Commands that work on the data in cubes follow a well defined
structure. This structure is defined by three main objects, command,
user and program.

\subsection{Command Object}

The command object is the first object encountered when executing a
command, this object is created at the moment a command is registered
and exists throughout the execution of cube. This object is called
into action when SIC finds out which command to execute. Currently the
subroutine register and command are not part of the command object due
to constraints in the interface with SIC. Currently this object
contains two important methods: parse, which fills the user object
from the user input; main, which creates the program object from the
user object and calls the program object for the execution of the data
manipulation. The command object type is always named commname_comm_t

\subsection{User Object}

The user object is created from the parsing. The members of this
object reflect {\emph ipsis literis} what the user has typed but in a
structured way. Usually the User object only has one method, this
method is called toprog. The toprog method makes the translation
between user inputs and program inputs and hence generates the program
object. The user object type is always named commname_user_t

\subsection{Program Object}
After the creation of the program object from the user object the
program object controls the flow of actions. The first method to be
invoked is usually the header method, this method constructs the
header of the eventual output cube(s). The following method to be
called is usually the data method. The data method may sometimes be
preceded by some pre-processing methods, or be succeeded by
post-processing methods. The data method is where the actual data
manipulation is split into different tasks which are sent to different
threads to be processed in parallel. Each task will then call
sub-methods of the data method to execute the actual data manipulation.
The program object type is always named commname_prog_t.


\end{document}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
