\documentclass[11pt]{article}

% GILDAS specific definitions
\include{gildas-def}

\title{Trouble Shooting Guide \\
  IRAM Plateau de Bure Interferometer}
\author{S.Guilloteau$^{1}$}
\date{Document probably older than you think}
\makeindex{}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\begin{document}

\maketitle{}

\begin{center}
Version 2.0
\end{center}

\begin{center}
  (1) Institut de Radio Astronomie Millim\'etrique\\
  300 Rue de la Piscine\\
  F-38406 Saint Martin d'H\`eres
\end{center}

This document gives some information about troubleshooting
at the IRAM Plateau de Bure interferometer.

\begin{rawhtml}
  Note: this is the on-line version of the "Trouble Shooting Guide"
  A <A HREF="../../pdf/pdbi-trouble.pdf"> PDF version</A>
  is also available.
\end{rawhtml}

Related information is available in:
\begin{latexonly}
  \begin{itemize}
  \item{IRAM Plateau de Bure Interferometer: Foreign Commands}
  \item{IRAM Plateau de Bure Interferometer: OBS Users Guide}
  \end{itemize}
\end{latexonly}

\begin{rawhtml}
  <UL>
  <LI> IRAM Plateau de Bure Interferometer:
  <A HREF="../obs-html/obs.html"> OBS Users Guide</A>
  </UL>
\end{rawhtml}

\newpage
\tableofcontents

\newpage
\section{Description}

The interferometer consists of:
\begin{itemize}
\item Five antennas, each controlled by a VME microprocessor (the {\em
    POINTING} micro).
\item One off-axis optical telescope per antenna, for pointing,
\item 2 (4-6 on the long term, 2 simultaneous IF) receivers per antenna,
  controlled by the {\em RECEIVER} micro.
\item The {\em PHASER} micro, which controls a 12-channel continuum
  detector, two HP-synthesizers for the LO frequencies, the phases and
  rates.
\item The {\em CLOCK} micro, which controls the time distribution and also
  handles the meteo station.
\item A 6-unit purpose spectral correlator. The digital part of each unit
  is divided in two VME crates, named the {\em MASTER} and the {\em
    SATELLITE}, which are linked by the ``{\tt GigaLink}''.
\item A HP-J200 workstation named BURE1 for real-time command and
  acquisition,
\item An HP-J200 workstation named BURE2 for data reduction, sharing NFS
  disks with the acquisition computer BURE1.
\item A dedicated Ethernet link between BURE1, the {\em PHASER} and the
  correlator micros.
\item A general Ethernet link between BURE1, the {\em CLOCK}, the antennas
  ({\em POINTING} and {\em RECEIVER}), BURE2 and all the other
  terminals/computers on the site.
\item Many cables : low and high quality for the IF transmission, twisted
  pairs for specific signals, etc...
\end{itemize}

The instrument may be used in several basic modes: {\tt OPTICAL} pointing,
{\tt INTERFEROMETER} observing, or {\tt TEST} mode.  In addition, within
the \obs\ program, there is a {\tt CONFIGURE} mode, which is used to
specify and modify the array configuration parameters.

Any part of the instrument may fail, in any operating mode. The purpose of
this guide is to pinpoint the most frequent failures, their effects and the
remedies. It is assumed the reader has good knowledge of the ``Users
Guide''.

\section{The Real Time system BURE1}

Before proceeding to the main topic, a detailed description of the software
is required.

\subsection{The Tasks and Programs}

The software is organised in two basic blocks:
\begin{itemize}
\item Real-time tasks that monitor the antenna and receivers:
\begin{itemize}
\item INTERP: this task dialog with the antenna microprocessors every
  second, and occasionaly send command to them.
\item FLAG1S: this task is a time checker. It is activated every second
  through a dialog with the Clock microprocessor. It also handles the meteo
  parameters.
\item ASTRJ : this task does the astronomical computations to prepare the
  input to INTERP. It is activated every time a new source is requested.
\end{itemize}
\item Tasks that control and perform the acquisition:
\begin{itemize}
\item CORREL: this task controls the correlator data acquisition.
\item CORREL\_SPECTRA: this task reads the data from the correlator micro
  when acquisition is finished.
\item \obs\ : High level observing program, with integrated observing
  procedures. Only one occurence of \obs\ can be running on BURE1.
\item RDI : interferometric data acquisition program.
\end{itemize}
\item Control and display tasks:
\begin{itemize}
\item XAFF : perform various display tasks, including computation of the
  running amplitudes and phases.
\item STSA : this program is used to display the antenna parameters related
  to the pointing.
\item STSH : this program is used to display antenna parameters related to
  ``survival'' (de-icing, etc...).
\end{itemize}
\end{itemize}

The \obs\ program is intended to provide complete normal operation of the
instrument. Accordingly, \obs\ assumes it has complete control over the
system.

\subsection{Commons}

All real-time tasks interact through global commons:
\begin{itemize}
\item{\tt GENERAL} is for general parameters,
\item{\tt ANTENNA} for antenna related parameters,
\item{\tt RECEIVER} for receiver related parameters,
\item{\tt ACQUIS} for continuum detector and correlator parameter and
  results,
\item{\tt OBSRED and ACTIVE} are shared between \obs\ and the acquisition
  program RDI.
\end{itemize}
A set of semaphores are used for synchronisation.
\begin{itemize}
\item S\_ANT ({\tt ANTENNA} common locked) (CLEARED) \\
  This semaphore is blocked (set) whenever a task uses the antenna common
  and released (cleared) when the operation if finished.  The next task
  requesting the common can grab it, and so on...
\item S\_GENERAL ({\tt GENERAL} common locked) (CLEARED)\\
  This semaphore is used to lock/unlock the general common.
\item S\_ACQUIS ({\tt ACQUIS}ition common locked) (CLEARED)\\
  This semaphore is used to lobck/unlock the acquisition common.
\end{itemize}
This scheme is used to avoid conflicting actions from several tasks.

\subsection{Synchronisation}

Flags are used for inter-process synchronisation, or to request some
specific actions.
\begin{itemize}
\item ASTRO (Astronomical Computation requested) (CLEARED) \\
  This flag is set by OBS when changing source. It is expected and cleared
  by ASTRJ when relevant computations have been done.
\item TCPIP (Dialog with antenna micros) (CLEARED)\\
  This flag is set by FLAG1S every second. It is expected and cleared by
  INTERP, which reads information from the antenna micros on such
  occasions.
\item TRACK Tracking (SET) \\
  This flag indicate the Antennas {\em and} subreflectors are in position
  and tracking. It is set and cleared by INTERP. It is expected by CORREL
  to start data acquisition.
\item UT  (UT time change) (CLEARED) \\
  This flag indicate the UT time has changed. It is set by FLAG1S and
  expected and cleared by CORREL.
\item COR (CORrelator acquisition required) (CLEARED)\\
  This flag is set by \obs\ when an acquisition is requested.  It is
  expected and cleared by CORREL.
\item SUP (Interferometer setup done) (CLEARED)\\
  This flag is set by task CORREL whenever the parameters of an acquisition
  are ready. It is expected and cleared by RDI.
\item DAT (Interferometer data ready) (CLEARED)\\
  This flag is set by task CORREL\_SPECTRA whenever new data is available
  from the correlators. It is expected and cleared by RDI.
\item SPECTRE (Correlator data ready) (CLEARED)\\
  This flag is set by CORREL when an acquisition finished. It is expected
  and cleared by CORREL\_SPECTRA which reads the data from the correlators
  and set flags DAT when done.
\item ALARM (Alarm) (CLEARED)\\
  This flag may be set by some program. It signals a problem...
\item STARTED (Acquisition started) (CLEARED) \\
  This flag is set by CORREL when the acquisition is running.  It is
  expected and cleared by RDI.
\item RED (RDI  ready) (SET) \\
  This flag is set by RDI when it is ready. It is expected (and later
  cleared) by \obs.
\item CONTROLC (\verb=^C= pressed) (CLEARED)\\
  Set when \verb=^C= is pressed in \obs.
\end{itemize}

\subsection{Configuration and Initialisation Files.}

The ``hardware'' configuration of the interferometer is listed in several
files:
\begin{itemize}
\item INTER\_BASE:CONFIG.DAT\\
  This file indicates some basic array parameters, specially the time
  corrections. It should never be changed by operators or astronomers.
\item INTER\_BASE:GENERAL.ANk\\
  These files contain antenna dependant parameters, such as pointing
  constants, focus and homology parameters. The parameters change only when
  the antenna is repaired or modified.
\item INTER\_BASE:Tij.ANk\\
  These files contain parameters for antenna k on station Tij (e.g. W00,
  not W0): IAZ, MVE, MVN, position errors dX,dY,dZ and delay offset. They
  have to be updated each time the antenna is set on station.
\item INTER\_BASE:START.OBS\\
  This procedure indicates on which stations the antennas are currently
  located, and to which correlator entries they are connected.  It also
  indicates which is the antenna connected to the single-dish backends.
\end{itemize}
A special language in \obs\ has been implemented to allow modifications of
the array configuration parameters: language SET$\backslash$, which is
available only in the CONFIGURE mode. These files should never be modified
by a text editor.

The acquisition programs RED and RDI write the data on files named
INTER\_DATA:Date.BUR and INTER\_DATA:Date.IPB respectively, where Date is
the current date in the form DD-mmm-YYYY.  Three log files are created,
Date.LOG-OBS by \obs\, Date.LOG-RED by RED and Date.LOG-RDI by RDI, in
directory INTER\_LOG.

\section{Diagnostic Tools}

\subsection{STSA}

STSA display the antenna coordinates and a summary of the interferometer
status.

\subsection{FLAG}

FLAG is a small tool to display the running status of the event flags.  It
is able to see flag changes, provided they are not too fast.

\subsection{WHY}

The WHY command is the major tool for trouble shooting. It performs several
consistency checks of the interferometer state, such as scanning through
the list of ``tasks'' (e.g.  INTERP) and ``programs'' (e.g. RDI) to see if
any of them is missing.

\subsection{DMP and READ\_ACQUIS}

DMP dumps parts of the general and/or antenna commons.  READ\_ACQUIS dumps
the acquisition common.

\section{Programs on the various microprocessor}

\subsection{The {\em POINTING} micro}

It may be necessary to log on the {\em POINTING} micro for checks in case
of anomalous behaviour.

The normal state can be check by typing {\tt procs -e} to check the running
processes. It should say:
\begin{verbatim}
ant51_s: procs -e
 Id PId Grp.Usr  Prior  MemSiz Sig S    CPU Time   Age Module & I/O
  2   0   0.0     128    4.75k   0 w        0.00 159:19 sysgo <>>>term
  3   0   0.0     128    2.00k   0 s    37:20.78 159:19 ifman
  4   0   0.0     128   20.25k   0 s     2:32.27 159:19 routed <>>>nil
  5   0   0.0     128    2.00k   0 a     2:17.03 159:19 sockman
  6   2   0.0     128    6.75k   0 w        0.00 159:19 shell <>>>term
  7   6   0.0     128    6.75k   0 w        0.13 159:19 shell <dd >>>term
  8   0   0.0     128   62.75k   0 e       29.00 159:19 nfsc <>>>term
  9   7   0.0     128   12.00k   0 e        0.01 159:19 ftpd <>>>nil
 10   7   0.0     128   12.00k   0 e        0.44 159:19 telnetd <>>>nil
 11   7   0.0     128   15.50k   0 s        0.03 159:19 tsmon <>>>term
 12  13   2.2     128    6.75k   0 e        0.16 75:15 shell <>>>pks00
 13   0   0.0     128   22.00k   0 e        0.25 75:15 telnetdc <pks00
 14   0   0.0     128   11.25k   0 e        1.53 159:19 rmshd <dd >pipe >>nil
 15   0   0.0     128    9.50k   0 e 10:37:06.41 159:19 ev_it1s <dd >>>nil
 16   0   0.0     128   11.25k   0 e       26.77 159:19 ev_itsync <dd >>>term
 17   0   0.0     128   11.25k   0 e        0.02 159:19 ev_itincr <dd >>>term
 18   0   0.0     128   11.25k   0 e  1:27:55.70 159:19 tcpip <dd >>>nil
 19   0   0.0     128   11.25k   0 s  1:25:13.52 159:19 subxmit <>>>nil
 20  21   0.0     128    6.75k   0 w        0.18  0:00 shell <>>>pks01
 21   0   0.0     128   22.00k   0 a        0.18  0:00 telnetdc <pks01
 22  20   0.0     128   18.25k   0 *        0.36  0:00 procs <>>>pks01
\end{verbatim}

Programs {\tt ev\_it1s, ev\_itsync, ev\_itincr} (the time keepers), {\tt
  tcpip} (which handles communications with BURE1), {\tt subxmit} (which
communicates with the {\em RECEIVER} micro program {\tt subrecv} to control
the subreflector), and {\tt rmshd} (which support the {\tt rmsh} commands
from BURE1) should all be active.

Program {\tt user} can display all the status bits of the antenna.  Program
{\tt UT} displays the running UT and LST times.  Program {\tt set\_time}
can be used to re-synchronize the micro and reset the UT time. LST will
become good again only after a command has been send by BURE1.  Program
{\tt incr} can be used to re-initialize the encoders/subreflector.  Program
{\tt dmp} dumps the antenna common.

Synchronisation problems may require to have a look at the ``history'' of
the synchro errors. They are logged in files named {\tt sync.ant11,
  sync.ant21, etc...}. These files should be small. To check their size,
use:
\begin{verbatim}
ant51_s: chd /dd/ANTENNA
ant51_s: dir -e sync.*
 Owner    Last modified  Attributes Sector Bytecount Name
-------   -------------  ---------- ------ --------- ----
  0.0     96/07/16 0314   ------wr    1B51       150 sync.ant11
  0.0     96/07/16 0230   ------wr    1B50       125 sync.ant31
  0.0     96/07/16 0230   ------wr    1BC0        50 sync.ant41
  0.0     96/07/16 0230   ------wr    1B52        50 sync.ant51
\end{verbatim}
The size (Bytecount) should be less than about 2000 bytes. Otherwise, too
many timing errors have appeared: check the TU01 time bus.  The
sync.ant$i1$ files contain the date and time of the error:
\begin{verbatim}
ant51_s: list sync.11
Tue Jul 16 09:31:42 1996
Tue Jul 16 09:31:43 1996
Tue Jul 16 10:15:27 1996
Tue Jul 16 10:15:35 1996
Tue Jul 16 10:15:41 1996
Tue Jul 16 10:15:47 1996
\end{verbatim}
If the file size keep increasing, the antenna are no longer properly
synchronized. Use command {\tt set\_time} to re-synchronize them and reset
the UT time. If the files get too large (Bytecount $> 10000$), delete them
using command
\begin{verbatim}
ant51_s: del sync.ant*
\end{verbatim}

\subsection{The {\em RECEIVER} micro}

This micro controls the receivers and the subreflector of the antenna.  The
following programs should be running:
\begin{verbatim}
ant12_o: procs -e
 Id PId Grp.Usr  Prior  MemSiz Sig S    CPU Time   Age Module & I/O
  2   0   0.0     128    4.75k   0 w        0.00 181:49 sysgo <>>>term
  3   0   0.0     128    2.00k   0 e  2:26:25.17 181:49 ifman
  4   0   0.0     128   20.25k   0 s     3:43.81 181:49 routed <>>>nil
  5   0   0.0     128    2.00k   0 s       49.74 181:49 sockman
  6   2   0.0     128    6.75k   0 w        0.00 181:49 shell <>>>term
  7   6   0.0     128    6.75k   0 w        0.16 181:49 shell <dd >>>term
  8   0   0.0     128   62.75k   0 e        3.35 181:49 nfsc <>>>term
  9   7   0.0     128   12.00k   0 e        0.01 181:49 ftpd <>>>nil
 10   7   0.0     128   12.00k   0 e        0.11 181:49 telnetd <>>>nil
 11   7   0.0     128   15.50k   0 w        0.04 181:48 tsmon <>>>term
 12  17   0.0     128    7.00k   0 e     2:23.16 12:59 rserverc <socket >>>nil
 13  17   0.0     128    7.00k   0 e        5.94  3:01 rserverc <socket >>>nil
 14   0   0.0     128    5.00k   0 s       43.21 181:48 sts <>>>nil
 15   0   0.0     128    5.00k   0 a       11.22 181:48 subloo <>>>nil
 16   0   0.0     128   11.25k   0 e    57:59.04 181:48 subrecv <>>>nil
 17   0   0.0     128   11.25k   0 e       12.72 181:48 rserver <socket >>>nil
 18   0   0.0     128   11.25k   0 e        0.01 181:48 rmshd <dd >pipe >>nil
 19  20   2.2     128   16.00k   0 a    26:19.50 14:33 user <>>>pks00
 20  21   2.2     128    6.75k   0 w        1.29 167:41 shell <>>>pks00
 21   0   0.0     128   22.00k   0 e    27:29.95 167:41 telnetdc <pks00
 22  24   2.2     128    6.75k   0 w        0.26  0:00 shell <>>>pks01
 23  11   0.0     128   14.75k   0 s        0.09 166:20 login <>>>term
 24   0   0.0     128   22.00k   0 e        0.19  0:00 telnetdc <pks01
 25  22   2.2     128   18.25k   0 *        0.54  0:00 procs <>>>pks01
\end{verbatim}
Programs {\tt sts} (receiver monitoring), {\tt subloo} and {\tt subrecv}
(subreflector control), {\tt rmshd} (support for the {\tt rmsh} commands
from BURE1) should all be active.

Program {\tt user} can display all the status of the receiver.

\subsection{The {\em CLOCK} micro}

The following programs should be running on that micro:
\begin{verbatim}
clock: procs -e
 Id PId Grp.Usr  Prior  MemSiz Sig S    CPU Time   Age Module & I/O
  2   0   0.0     128    7.75k   0 w        0.04 143:20 sysgo <>>>term
  6   0   0.0     128   18.75k   0 s     1:27.51 143:20 routed <>>>nil
  7   0   0.0     128    2.00k   0 a     9:39.09 143:20 ifman
  8   0   0.0     128    2.00k   0 s        0.01 143:20 sockman
 11   0   0.0     128  105.75k   0 e        0.24 143:20 nfsc <>>>term
 12   2   0.0     128    8.25k   0 w        0.00 143:20 shell <>>>term
 13  12   0.0     128    8.25k   0 w        0.12 143:20 shell <dd >>>term
 18  13   0.0     128   20.50k   0 e        0.01 143:20 ftpd <>>>nil
 19  13   0.0     128   20.50k   0 e        0.04 143:20 telnetd <>>>nil
 35   0   0.0     128   14.25k   0 e    43:48.04 143:20 ev_it1s1 <dd >>>nil
 39   0   0.0     128   10.25k   0 s        0.00 143:20 meteo <dd >>>nil
 41   0   0.0     128   15.00k   0 e        0.21 143:20 server <dd >>>nil
 43  13   0.0     128   23.75k   0 s        0.04 143:20 tsmon <dd >>>term
 45   0   0.0     128   30.75k   0 e        0.13  0:00 telnetdc <pks00
 46  45   0.0     128    8.25k   0 w        0.19  0:00 shell <>>>pks00
 53  46   0.0     128   26.75k   0 *        0.32  0:00 procs <>>>pks00
\end{verbatim}
Programs {\tt ev\_it1s1, server} (the time servers), and {\tt meteo} (which
reads the meteo station) should be running.

\subsection{the {\em PHASER} micro}

The following programs should be running on that micro:
\begin{verbatim}
$ procs -e
 Id PId Grp.Usr  Prior  MemSiz Sig S    CPU Time   Age Module & I/O
  2   0   0.0     128    9.50k   0 w        0.00 11:26 sysgo <>>>term
  3   2   0.0     128    6.75k   0 w        0.00 11:26 shell <>>>term
  4   0   0.0     128   20.25k   0 s       10.53 11:26 routed <>>>nil
  5   0   0.0     128    2.00k   0 e     1:25.73 11:26 ifman
  6   0   0.0     128    2.00k   0 s        3.25 11:26 sockman
  7   3   0.0     128    6.75k   0 w        0.10 11:26 shell <dd >>>term
  8   0   0.0     128   62.75k   0 e        0.28 11:26 nfsc <>>>term
  9   7   0.0     128   12.00k   0 e        0.00 11:26 ftpd <>>>nil
 10   7   0.0     128   12.00k   0 e        0.03 11:26 telnetd <>>>nil
 11  12   0.0     128    5.25k   0 s     8:47.09 11:26 rotator <dd >>>nil
 12   0   0.0     128   21.25k   0 e    10:37.16 11:26 server <dd >>>nil
 13  12   0.0     128    0.00k   0 -        0.00 11:26 <none>
 14  12   0.0     128   11.50k   0 s        0.03 11:26 hp_synt <dd >>>nil
 15  12   0.0     128    5.00k   0 s        0.00 11:26 noise <dd >>>nil
 16  12   0.0     128    5.00k   0 s        0.00 11:26 watchdog <dd >>>nil
 17   7   0.0     128   15.50k   0 s        0.01 11:26 tsmon <dd >>>term
 18  19   0.0     128    6.75k   0 w        0.46  0:00 shell <>>>pks00
 19   0   0.0     128   22.00k   0 a        0.19  0:00 telnetdc <pks00
 20  18   0.0     128   18.25k   0 *        0.30  0:00 procs <>>>pks00
\end{verbatim} %$
Programs {\tt rotator} (phase rotator control), {\tt hp\_synt} (frequency
synthesizer control), {\tt noise} (noise source commutation), {\tt
  watchdog} (radio alarm), and {\tt server} (total power output) should be
running.

\subsection{The {\em MASTER} and {\em SATELLITE} micros}

\begin{verbatim}
corl01: procs -e
 Id PId Grp.Usr  Prior  MemSiz Sig S    CPU Time   Age Module & I/O
  2   0   0.0     128    7.75k   0 w        0.01  1:37 sysgo <>>>term
  4  64   0.0     128   75.25k   0 s        0.13  0:04 calcul <>>>pks00
  5  64   0.0     128   28.50k   0 s        0.00  0:04 setIF <>>>pks00
  6   0   0.0     128   18.75k   0 s        0.02  1:37 routed <>>>nil
  7   0   0.0     128    2.00k   0 e        0.40  1:37 ifman
  8   0   0.0     128    2.00k   0 s        0.00  1:37 sockman
  9  64   0.0     128   34.25k   0 e        0.01  0:04 spectra <>>>pks00
 11   0   0.0     128  105.75k   0 e        0.19  1:37 nfsc <>>>term
 12   2   0.0     128    8.25k   0 w        0.00  1:37 shell <>>>term
 13  12   0.0     128    8.25k   0 w        0.02  1:37 shell <dd >>>term
 15  64   0.0     128   28.00k   0 s        0.00  0:04 tweak <>>>pks00
 17   0   0.0     128   30.75k   0 a        0.03  0:00 telnetdc <pks01
 18  13   0.0     128   20.50k   0 e        0.00  1:37 ftpd <>>>nil
 19  13   0.0     128   20.50k   0 e        0.01  1:37 telnetd <>>>nil
 20  17   0.0     128    8.25k   0 w        0.02  0:00 shell <>>>pks01
 27  13   0.0     128   23.75k   0 s        0.01  1:37 tsmon <dd >>>term
 28  20   0.0     128   26.75k   0 *        0.08  0:00 procs <>>>pks01
 29   0   0.0     128   30.75k   0 e        0.12  1:37 telnetdc <pks00
 30  29   0.0     128    8.25k   0 w        0.12  1:37 shell <>>>pks00
 64  30   0.0     128  211.50k   0 e        0.12  0:04 correl <>>>pks00
\end{verbatim}
Programs {\tt calcul}, {\tt setIF} (IF processor setting), {\tt spectra}
{\tt correl} and {\tt tweak} should be running on the {\em MASTER}.  The
{\em SATELLITE} is similar, except that program {\tt setIF} does not run.

\section{``Normal State'' Definitions}

This section defines a terminology used to refer to ``normal states'' of
the array afterwards.
\begin{itemize}
\item{Single-Dish Continuum OK}\\
  This means a continuum point source can be detected (in total power mode)
  with the one dish antenna. It implies that all the antenna control tasks
  work, that the pointing parameters are reasonably accurate for that
  antenna.
\item{Single-Dish OK}\\
  This means a (standard) line source can be detected in ONOFF with one
  antenna. In addition to the previous state, it also implies that the
  receiver is tuned to the right frequency.
\item{Multi-Dish continuum OK}\\
  This means a continuum point source can be detected (in total power mode)
  with the continuum dish detectors on all antennas.  It implies the
  antenna control taks work for all antennas.
\item{Multi-Dish OK}\\
  This means a (standard) line source can be detected on all antennas with
  the \obs\ ONOFF procedure. In addition to the previous state, it also
  implies that all receivers are tuned to the right frequency.
\end{itemize}

The last stage (which includes all the previous ones) is necessary for
interferometry to work, but it may not be sufficient.

\section{Standard Check List}

To avoid further potential problems, the following procedure should be
followed as closely as possible to get started.

\begin{itemize}
\item Start \obs , select the program source and the program frequency,
  then type LOAD and immediately after MICRO.  This is normally done by the
  recommended {\tt PR:Name.OBS} setup procedure, for project ``{\tt
    Name}''.
\item Tune the receivers.
\item In parallel, you may initialize the antennas and subreflectors if not
  done, open the central hub, etc... When antennas have been initialized,
  select a strong continuum source to check pointing, type LOAD.
\item As soon as receiver are tuned, use a CALIBRATE procedure (in INTER
  mode). All receiver temperatures should be reasonable.
\item Make an BANDPASS procedure to calibrate the IF passband
  characteristic.  Look at the result using CLIC. This checks the spectral
  correlator up to a certain point. If anything is unusual, notify the
  maintenance staff.
\item Select a strong line source (for that frequency, see catalog of
  strong line sources), and use an ONOFF procedure. Line should be detected
  in all antennas.
\item Select a strong continuum point source to determine delays on all
  antennas if unknown.
\item Make a POINT on that source. All antennas should be pointed.
\item Eventually send the phase calibrator used for your program source,
  and start interferometry. Phases on AFF should be constant.
\end{itemize}

\section{Trouble Shooting Receipes}

In many of these cases, the WHY command may be sufficient to pinpoint the
error. WHY is not allmighty, however, and a good understanding of the
common mistakes and failures help.

\subsection{Nothing Happens}

Scans won't start, or LST does not run, etc...

- A major real-time program probably died. Use command WHY, it will tell
you which. During working hours, try to call S.Guilloteau. Otherwise, use
command {\tt SET$\backslash$RESTART} in \obs to restart the missing
program, and notify S.Guilloteau. \\
- If no major program is missing, and LST does not run, check the {\tt
  CLOCK} micro: it may be necessary to reboot it.

\subsection{Antenna won't move}

- Emergency stop on, or antenna in local. Use STSA page to check. \\
- The antenna is not in the default ``telescope'' ({\tt anttel}). Use {\tt
  dmp} or WHY to check, and reload the configuration from \obs\ (using
command {\tt SET$\backslash$OBSERVE}.\\
- Use command WHY to check whether {\tt INTERP} is running. \\
- Door contact lost ({\tt C\_Po = 0}) in the antenna. Use program {\tt
  user} in the {\em POINTING} micro to check.

\subsection{Antenna does not reach source, but keeps trying}

Check source elevation (must be $>3$ and $<86$ degrees) and sun distance
(variable {\tt DSUN}, must be $>35$degrees).\\
Antenna encoders (Az and/or El) not initialized.\\

\subsection{Scan does not start}

- Antenna(s) not tracking. WHY will tell you\\
- Subreflector(s) not ready. Check init button on STSA display. WHY will
tell you.\\\
- RDI missing. Use command WHY to check.\\
- CORREL missing. Use command WHY to check.\\
- CORREL\_SPECTRA missing. Use command WHY to check.\\
- {\em PHASER} micro not properly running. Check and reboot it if needed.

\subsection{Crazy calibration results}

- Check that the central hub is opened (STSA)\\
- Check that the table moves ({\tt user} program on {\em RECEIVER} micro). \\
- Check the receiver LO power.

\subsection{No single-dish continuum data}

- Check that the central hub is opened (STSA), and that the table is in
position ({\em RECEIVER} {\tt user}).\\
- If both are correct, it means the pointing is bad. Check the antenna
time ({\tt UT} program on the {\em POINTING} micro).\\
- If bad, reset it using the {\tt set\_time} program on
the {\em POINTING} micro.\\
- Reload the pointing constants (by a {\tt SET$\backslash$OBSERVE}
command), and check the pointing. \\
- If still bad, reinitialize the antenna, and check the pointing.

\subsection{No single-dish spectral lines}

- Be sure you are observing a strong line source.  Assuming single-dish
continuum is working, this indicates a wrong harmonic number on receiver.

\subsection{Multi-dish continuum OK, but no fringes at all on XAFF}

- Could be the delay: check with CLIC whether fringes appear on the
spectral correlator. \\
- Check phase rotator ``flashing'' lights. \\
- Check correlator power supplies. \\
- If problem persists, reset the {\em PHASER} micro. \\
- If problem persists, reset the correlator {\em MASTER} and {\em
  SATELLITES} micros.

\subsection{Multi-dish continuum OK, but no fringes on one antenna on XAFF}
(This means at least two bad baselines, usually...)

- Phase lock not properly closed. Look on spectrum analyser\\
- Bad frequency (harmonic number) on that antenna. Make a single-dish
spectrum on a strong line source to check if possible. \\
- Retune receiver to correct harmonic number, after resetting LO2.

\subsection{Unusually high Tsys on one antenna}

- Check receiver physical temperature: it may need helium refilling.\\
- Check tuning and LO power.

\subsection{Lower intensity on some baselines on XAFF display}

- Bad pointing (check time), or bad focus on one antenna - bad LO power
adjustment or receiver tuning on one antenna.  - Strong atmospheric phase
fluctuations can also produce such an effect, if the baselines are much
longer than the other ones.

\subsection{Intensity jump on two baselines}

- Possible pointing problem. Check pointing. If jump due to pointing,
reinitialize antenna (only, not subreflector), check pointing again.\\
- Possible focus problem. Check focus.  If jump due to focus, reinitialize
subreflector separately, and check focus again.

Report pointing and focus jump if needed.

\subsection{Fringes suddenly disappear}

- Use command WHY to check whether all programs are running.\\
- Possible time problem. Check the Clock micro. \\
- Check the Phaser micro.\\

\subsection{Correlator subband not working on one antenna}

- Can be the LO3: check the ``lock'' indicator. \\
- Can also be the filters. Check several bandwidths to disentangle between
various filters.  Make a BANDPASS calibration, analyse it with CLIC: check
whether autocorrelation mode is OK. \\
- If autocorrelation not correct, sampler problem most likely.\\
- If OK, digital problem in the correlator board or {\em GigaLink}.
Try resetting the micro. \\

If problem persists, call maintenance. In all cases notify maintenance
team.

\subsection{Other funny problem}

Call Stephane Guilloteau, or Robert Lucas, or whoever may be ``expert''.
Notify Stephane Guilloteau in any of these cases.  Make a dump of the
commons (command {\tt DMP}).

\subsection{Other not funny problem}
As above, but we prefer the previous ones.

\end{document}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
