<!--  -->
<sect1 id="monitoring">
<title>Cluster Monitoring</title>

<indexterm><primary>monitoring &slony1;</primary></indexterm>

<para> As a prelude to the discussion, it is worth pointing out that
since the bulk of &slony1; functionality is implemented via running
database functions and SQL queries against tables within a &slony1;
schema, most of the things that one might want to monitor about
replication may be found by querying tables in the schema created for
the cluster in each database in the cluster. </para>

<para> Here are some of the tables that contain information likely to
be particularly interesting from a monitoring and diagnostic
perspective.</para>

<glosslist>
<glossentry><glossterm><envar>sl_status</envar></glossterm>

<glossdef><para>This view is the first, most obviously useful thing to
look at from a monitoring perspective.  It looks at the local node's
events, and checks to see how quickly they are being confirmed on
other nodes.</para>

<para> The view is primarily useful to run against an origin
(<quote>master</quote>) node, as it is only there where the events
generated are generally expected to require interesting work to be
done.  The events generated on non-origin nodes tend to
be <command>SYNC</command> events that require no replication work be
done, and that are nearly no-ops, as a
result. </para></glossdef></glossentry>

<glossentry><glossterm>&slconfirm;</glossterm>

<glossdef><para>Contains confirmations of replication events; this may be used to infer which events have, and <emphasis>have not</emphasis> been processed.</para></glossdef></glossentry>

<glossentry><glossterm>&slevent;</glossterm>
<glossdef><para>Contains information about the replication events processed on the local node.  </para></glossdef></glossentry>

<glossentry><glossterm>
&sllog1;
and
&sllog2;
</glossterm>

<glossdef><para>These tables contain replicable data.  On an origin node, this is the <quote>queue</quote> of data that has not necessarily been replicated everywhere.  By examining the table, you may examine the details of what data is replicable. </para></glossdef></glossentry>

<glossentry><glossterm>&slnode;</glossterm>
<glossdef><para>The list of nodes in the cluster.</para></glossdef></glossentry>

<glossentry><glossterm>&slpath;</glossterm>
<glossdef><para>This table holds connection information indicating how &lslon; processes are to connect to remote nodes, whether to access events, or to request replication data. </para></glossdef></glossentry>

<glossentry><glossterm>&sllisten;</glossterm>

<glossdef><para>This configuration table indicates how nodes listen
for events coming from other nodes.  Usually this is automatically
populated; generally you can detect configuration problems by this
table being <quote>underpopulated.</quote> </para></glossdef></glossentry>

<glossentry><glossterm>&slregistry;</glossterm>

<glossdef><para>A configuration table that may be used to store
miscellaneous runtime data.  Presently used only to manage switching
between the two log tables.  </para></glossdef></glossentry>

<glossentry><glossterm>&slseqlog;</glossterm>

<glossdef><para>Contains the <quote>last value</quote> of replicated
sequences.</para></glossdef></glossentry>

<glossentry><glossterm>&slset;</glossterm>

<glossdef><para>Contains definition information for replication sets,
which is the mechanism used to group together related replicable
tables and sequences.</para></glossdef></glossentry>

<glossentry><glossterm>&slsetsync;</glossterm>
<glossdef><para>Contains information about the state of synchronization of each replication set, including transaction snapshot data.</para></glossdef></glossentry>

<glossentry><glossterm>&slsubscribe;</glossterm>
<glossdef><para>Indicates what subscriptions are in effect for each replication set.</para></glossdef></glossentry>

<glossentry><glossterm>&sltable;</glossterm>
<glossdef><para>Contains the list of tables being replicated.</para></glossdef></glossentry>

</glosslist>

<sect2 id="testslonystate"> <title> test_slony_state</title>

<indexterm><primary>script test_slony_state to test replication state</primary></indexterm>

<para> This invaluable script does various sorts of analysis of the
state of a &slony1; cluster. Some administrators
recommend running these scripts frequently (hourly seems suitable) to
find problems as early as possible.  </para>

<para> You specify arguments including <option>database</option>,
<option>host</option>, <option>user</option>,
<option>cluster</option>, <option>password</option>, and
<option>port</option> to connect to any of the nodes on a cluster.
You also specify a <option>mailprog</option> command (which should be
a program equivalent to <productname>Unix</productname>
<application>mailx</application>) and a recipient of email. </para>

<para> You may alternatively specify database connection parameters
via the environment variables used by
<application>libpq</application>, <emphasis>e.g.</emphasis> - using
<envar>PGPORT</envar>, <envar>PGDATABASE</envar>,
<envar>PGUSER</envar>, <envar>PGSERVICE</envar>, and such.</para>

<para> The script then rummages through <xref linkend="table.sl-path">
to find all of the nodes in the cluster, and the DSNs to allow it to,
in turn, connect to each of them.</para>

<para> For each node, the script examines the state of things,
including such things as:

<itemizedlist>
<listitem><para> Checking <xref linkend="table.sl-listen"> for some
<quote>analytically determinable</quote> problems.  It lists paths
that are not covered.</para></listitem>

<listitem><para> Providing a summary of events by origin node</para>

<para> If a node hasn't submitted any events in a while, that likely
suggests a problem.</para></listitem>

<listitem><para> Summarizes the <quote>aging</quote> of table <xref
linkend="table.sl-confirm"> </para>

<para> If one or another of the nodes in the cluster hasn't reported
back recently, that tends to lead to cleanups of tables like &sllog1;,
&sllog2; and &slseqlog; not taking place.</para></listitem>

<listitem><para> Summarizes what transactions have been running for a
long time</para>

<para> This only works properly if the statistics collector is
configured to collect command strings, as controlled by the option
<option> stats_command_string = true </option> in <filename>
postgresql.conf </filename>.</para>

<para> If you have broken applications that hold connections open,
this will find them.</para>

<para> If you have broken applications that hold connections open,
that has several unsalutory effects as <link
linkend="longtxnsareevil"> described in the
FAQ</link>.</para></listitem>

</itemizedlist></para>

<para> The script does some diagnosis work based on parameters in the
script; if you don't like the values, pick your favorites!</para>

<note><para> Note that there are two versions, one using the
<quote>classic</quote> <filename>Pg.pm</filename> Perl module for
accessing &postgres; databases, and one, with <filename>dbi</filename>
in its name, that uses the newer Perl <function> DBI</function>
interface.  It is likely going to be easier to find packaging for
<function>DBI</function>. </para> </note>

</sect2>

<sect2> <title> &nagios; Replication Checks </title>

<indexterm><primary>&nagios; for monitoring replication</primary></indexterm>

<para> The script in the <filename>tools</filename> directory called
<command> psql_replication_check.pl </command> represents some of the
best answers arrived at in attempts to build replication tests to plug
into the <ulink url="http://www.nagios.org/"> &nagios; </ulink> system
monitoring tool.</para>

<para> A former script, <filename>
test_slony_replication.pl</filename>, took a <quote>clever</quote>
approach where a <quote>test script</quote> is periodically run, which
rummages through the &slony1; configuration to find origin and
subscribers, injects a change, and watches for its propagation through
the system.  It had two problems:</para>
<itemizedlist>

<listitem><para> Connectivity problems to the
<emphasis>single</emphasis> host where the test ran would make it look
as though replication was destroyed.  Overall, this monitoring
approach has been fragile to numerous error conditions.</para>
</listitem>

<listitem><para> &nagios; has no ability to benefit from the
<quote>cleverness</quote> of automatically exploring the set of nodes.
You need to set up a &nagios; monitoring rule for each and every node
being monitored.  </para> </listitem>
</itemizedlist>

<para> The new script, <command>psql_replication_check.pl</command>,
takes the minimalist approach of assuming that the system is an online
system that sees regular <quote>traffic,</quote> so that you can
define a view specifically for the replication test called
<envar>replication_status</envar> which is expected to see regular
updates.  The view simply looks for the youngest
<quote>transaction</quote> on the node, and lists its timestamp, age,
and some bit of application information that might seem useful to see.
</para>

<itemizedlist>

<listitem><para> In an inventory system, that might be the order
number for the most recently processed order. </para> </listitem>

<listitem><para> In a domain registry, that might be the name of the
most recently created domain.</para> </listitem>

</itemizedlist>

<para> An instance of the script will need to be run for each node
that is to be monitored; that is the way &nagios; works. </para>

</sect2>

<sect2 id="slonymrtg"> <title> Monitoring &slony1; using MRTG </title>

<indexterm><primary>MRTG for monitoring replication</primary></indexterm>

<para> One user reported on the &slony1; mailing list how to configure
<ulink url="http://people.ee.ethz.ch/~oetiker/webtools/mrtg/">
<application> mrtg - Multi Router Traffic Grapher </application>
</ulink> to monitor &slony1; replication.</para>

<para> ... Since I use <application>mrtg</application> to graph data
from multiple servers I use snmp (<application>net-snmp</application>
to be exact).  On database server, I added the following line to
<application>snmpd</application> configuration:</para>

<programlisting>
exec replicationLagTime  /cvs/scripts/snmpReplicationLagTime.sh 2
where <filename> /cvs/scripts/snmpReplicationLagTime.sh</filename> looks like this:
</programlisting>


<programlisting>
#!/bin/bash
/home/pgdba/work/bin/psql -U pgdba -h 127.0.0.1 -p 5800 -d _DBNAME_ -qAt -c
"select cast(extract(epoch from st_lag_time) as int8) FROM _irr.sl_status
WHERE st_received = $1"
</programlisting>

<para> Then, in mrtg configuration,  add this target:</para>
<programlisting>
Target[db_replication_lagtime]:extOutput.3&amp;extOutput.3:public at db::30:::
MaxBytes[db_replication_lagtime]: 400000000
Title[db_replication_lagtime]: db: replication lag time
PageTop[db_replication_lagtime]: &lt;H1&gt;db: replication lag time&lt;/H1&gt;
Options[db_replication_lagtime]: gauge,nopercent,growright
</programlisting>

<para> Alternatively, Ismail Yenigul points out how he managed to
monitor slony using <application>MRTG</application> without installing
<application>SNMPD</application>.</para>

<para> Here is the mrtg configuration</para>

<programlisting>
Target[db_replication_lagtime]:`/bin/snmpReplicationLagTime.sh 2`
MaxBytes[db_replication_lagtime]: 400000000
Title[db_replication_lagtime]: db: replication lag time
PageTop[db_replication_lagtime]: &lt;H1&gt;db: replication lag time&lt;/H1&gt;
Options[db_replication_lagtime]: gauge,nopercent,growright
</programlisting>

<para> and here is the modified version of the script</para>

<programlisting>
# cat /bin/snmpReplicationLagTime.sh
#!/bin/bash

output=`/usr/bin/psql -U slony -h 192.168.1.1 -d endersysecm -qAt -c
"select cast(extract(epoch from st_lag_time) as int8) FROM _mycluster.sl_status WHERE st_received = $1"`
echo $output
echo $output
echo 
echo
# end of script#
</programlisting>


<note><para> MRTG expects four lines from the script, and since there
are only two lines provided, the output must be padded to four
lines. </para> </note>

</sect2>

<sect2 id="bucardo-monitoring"> <title>Bucardo-related Monitoring</title>

<indexterm><primary>Bucardo monitoring</primary></indexterm>

<para> The <ulink url="http://bucardo.org/"> Bucardo </ulink>
replication system includes script,
<command>check_postgres.pl</command>, which can monitor a variety of
things about &postgres; status that includes monitoring &slony1;
replication status; see <ulink
url="http://bucardo.org/check_postgres/check_postgres.pl.html#slony_status">
check_postgres.pl - slony_status </ulink></para>
</sect2>

<sect2 id="search-logs"> <title> <command>search-logs.sh</command> </title>

<indexterm><primary> search &slony1; logs using search-logs.sh </primary></indexterm>

<para> This script is constructed to search for &slony1; log files at
a given path (<envar>LOGHOME</envar>), based both on the naming
conventions used by the <xref linkend="launchclusters"> and <xref
linkend="slonwatchdog"> systems used for launching &lslon; 
processes.</para>

<para> Errors, if found, are listed, by log file, and emailed to the
specified user (<ENVAR>LOGRECIPIENT</ENVAR>); if no email address is
specified, output goes to standard output. </para>

<para> <ENVAR>LOGTIMESTAMP</envar> allows overriding what hour to
evaluate (rather than the last hour). </para>

<para> An administrator might run this script once an hour to monitor
for replication problems. </para>
</sect2>

<sect2 id="wikigen"> <title> Building MediaWiki Cluster Summary </title>

<indexterm><primary> generating Wiki documentation of a cluster </primary></indexterm>

<para> The script <filename>mkmediawiki.pl </filename>, in
<filename>tools</filename>, may be used to generate a cluster summary
compatible with the popular <ulink url="http://www.mediawiki.org/">
MediaWiki </ulink> software.  Note that the
<option>--categories</option> permits the user to specify a set of
(comma-delimited) categories with which to associate the output.  If
you have a series of &slony1; clusters, passing in the option
<option>--categories=slony1</option> leads to the MediaWiki instance
generating a category page listing all &slony1; clusters so
categorized on the wiki.  </para>

<para> The gentle user might use the script as follows: </para>

<screen>
~/logtail.en>         mvs login -d mywiki.example.info -u "Chris Browne" -p `cat ~/.wikipass` -w wiki/index.php                     
Doing login with host: logtail and lang: en
~/logtail.en> perl $SLONYHOME/tools/mkmediawiki.pl --host localhost --database slonyregress1 --cluster slony_regress1 --categories=Slony-I  > Slony_replication.wiki
~/logtail.en> mvs commit -m "More sophisticated generated Slony-I cluster docs" Slony_replication.wiki
Doing commit Slony_replication.wiki with host: logtail and lang: en
</screen>

<para> Note that <command>mvs</command> is a client written in Perl;
on <ulink url="http://www.debian.org/"> Debian GNU/Linux</ulink>, the
relevant package is called
<application>libwww-mediawiki-client-perl</application>; other systems
may have a packaged version of this under some similar name. </para>

</sect2>

<sect2> <title> Analysis of a SYNC </title>

<indexterm><primary>analysis of a SYNC</primary></indexterm>

<para> The following is (as of 2.0) an extract from the &lslon; log for node
#2 in a run of  <quote>test1</quote> from the regression tests. </para>

<screen>
DEBUG2 remoteWorkerThread_1: SYNC 19 processing
INFO   about to monitor_subscriber_query - pulling big actionid list 134885072
INFO   remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 1
DEBUG2  ssy_action_list length: 0
DEBUG2 remoteWorkerThread_1: current local log_status is 0
DEBUG2 remoteWorkerThread_1_1: current remote log_status = 0
DEBUG1 remoteHelperThread_1_1: 0.028 seconds delay for first row
DEBUG1 remoteHelperThread_1_1: 0.978 seconds until close cursor
INFO   remoteHelperThread_1_1: inserts=144 updates=1084 deletes=0
INFO   remoteWorkerThread_1: sync_helper timing:  pqexec (s/count)- provider 0.063/6 - subscriber 0.000/6
INFO   remoteWorkerThread_1: sync_helper timing:  large tuples 0.315/288
DEBUG2 remoteWorkerThread_1: cleanup
INFO   remoteWorkerThread_1: SYNC 19 done in 1.272 seconds
INFO   remoteWorkerThread_1: SYNC 19 sync_event timing:  pqexec (s/count)- provider 0.001/1 - subscriber 0.004/1 - IUD 0.972/248
</screen>

<para> Here are some notes to interpret this output: </para>

<itemizedlist>
<listitem><para> Note the line that indicates <screen>inserts=144 updates=1084 deletes=0</screen> </para> 
<para> This indicates how many tuples were affected by this particular SYNC. </para></listitem>
<listitem><para> Note the line indicating <screen>0.028 seconds delay for first row</screen></para> 

<para> This indicates the time it took for the <screen>LOG
cursor</screen> to get to the point of processing the first row of
data.  Normally, this takes a long time if the SYNC is a large one,
and one requiring sorting of a sizable result set.</para></listitem>

<listitem><para> Note the line indicating <screen>0.978 seconds until
close cursor</screen></para> 

<para> This indicates how long processing took against the
provider.</para></listitem>

<listitem><para> sync_helper timing:  large tuples 0.315/288 </para> 

<para> This breaks off, as a separate item, the number of large tuples
(<emphasis>e.g.</emphasis> - where size exceeded the configuration
parameter <xref linkend="slon-config-max-rowsize">) and where the
tuples had to be processed individually. </para></listitem>

<listitem><para> <screen>SYNC 19 done in 1.272 seconds</screen></para> 

<para> This indicates that it took 1.272 seconds, in total, to process
this set of SYNCs. </para>
</listitem>

<listitem><para> <screen>SYNC 19 sync_event timing:  pqexec (s/count)- provider 0.001/1 - subscriber 0.004/0 - IUD 0.972/248</screen></para> 

<para> This records information about how many queries were issued
against providers and subscribers in function
<function>sync_event()</function>, and how long they took. </para>

<para> Note that 248 does not match against the numbers of inserts,
updates, and deletes, described earlier, as I/U/D requests are
clustered into groups of queries that are submitted via a single
<function>pqexec()</function> call on the
subscriber. </para></listitem>

<listitem><para> <screen>sync_helper timing:  pqexec (s/count)- provider 0.063/6 - subscriber 0.000/6</screen></para>

<para> This records information about how many queries were issued
against providers and subscribers in function
<function>sync_helper()</function>, and how long they took.
</para></listitem>

</itemizedlist>

</sect2>
</sect1>

<sect1 id="componentmonitoring"> <title>Component Monitoring</title>

<para> There are several ways available to see what &slony1; processes
are up to:</para>
<itemizedlist>
<listitem><para> <xref linkend="pgstatactivity">
<listitem><para> <xref linkend="slcomponents"> 
</itemizedlist>

<sect2 id="pgstatactivity"> <title> Looking at <envar>pg_stat_activity</envar> view </title>

<para> The standard &postgres; view <envar>pg_stat_activity</envar>
indicates what the various database connections are up to.</para> 

<para> On recent versions of &postgres;, this view includes an
attribute, <envar>application_name</envar>, which &slony1; components
populate based on the names of their respective threads.</para>

</sect2>
<sect2 id="slcomponents"> <title> Looking at &slcomponents; view </title>

<para> &slony1; has a table, &slcomponents;, introduced in &slony1;
2.1, which captures &slony1; activity for each node. </para>

<para><screen>
slonyregress1@localhost->  select * from _slony_regress1.sl_components  order by co_actor;
       co_actor       | co_pid | co_node | co_connection_pid |   co_activity    |      co_starttime      | co_event | co_eventtype
----------------------+--------+---------+-------------------+------------------+------------------------+----------+--------------
 local_cleanup        |  24586 |       0 |             24907 | thread main loop | 2011-02-24 17:02:55+00 |          | n/a
 local_listen         |  24896 |       1 |             24900 | thread main loop | 2011-02-24 17:03:07+00 |          | n/a
 local_monitor        |  24586 |       0 |             24909 | thread main loop | 2011-02-24 17:02:55+00 |          | n/a
 local_sync           |  24517 |       0 |             24906 | thread main loop | 2011-02-24 17:03:09+00 |          | n/a
 remote listener      |  24586 |       2 |             24910 | thread main loop | 2011-02-24 17:03:03+00 |          | n/a
 remoteWorkerThread_2 |  24586 |       2 |             24908 | thread main loop | 2011-02-24 17:02:55+00 |          | n/a
(6 rows)
</screen></para>

<para> This example indicates the various &slony1; threads that are
typically running as part of a &lslon; process:</para>
<glosslist>
<glossentry>
<glossterm>local_cleanup</glossterm>
<glossdef><para>This thread periodically wakes up to trim obsolete data and (optionally) vacuum &slony1; tables</para> 
</glossdef>
</glossentry>
<glossentry>
<glossterm>local_listen</glossterm>
<glossdef><para>This thread listens for events taking place on the local node, and changes the &lslon;'s configuration as needed.</para> </glossdef>
</glossentry>
<glossentry>
<glossterm>local_monitor</glossterm>
<glossdef><para>This thread is rather self-referential, here; it manages the queue of events to be published to the &slcomponents; table.</para> </glossdef>
</glossentry>
<glossentry>
<glossterm>local_sync</glossterm>
<glossdef><para>This thread generates SYNC events on the local database.  If the local database is the origin for a replication set, those SYNC events are used to propagate changes to other nodes in the cluster.</para> </glossdef>
</glossentry>
<glossentry>
<glossterm>remote_listener</glossterm>
<glossdef><para>This thread listens for events on a remote node database, and queues them into the remote worker thread for that node.</para> </glossdef>
</glossentry>
<glossentry>
<glossterm>remoteWorkerThread_2</glossterm>
<glossdef><para>This thread waits for events (from the remote listener thread), and takes action.  This is the thread that does most of the visible work of replication. </para> </glossdef>
</glossentry>
</glosslist>

</sect2>
<sect2><title>Notes On Interpreting Component Activity</title>

<itemizedlist>
<listitem><para> Many of these will typically report, as their activity,
<command>thread main loop</command>, which indicates that the thread
exists, and is simply executing its main loop, waiting to have work to
do.</para>

<para> Most threads will never indicate an event or event type, as
they do not process &slony1; events.</para></listitem>

<listitem><para> <envar>local_monitor</envar> thread never reports any
activity.</para>

<para> It would be a nice idea for this thread, which manages
&slcomponents;, to report in on its work.  Unfortunately, the fact of
adding in its own events would make it perpetually busy, as the action
of processing the queue would add a monitoring entry, in effect
becoming a repetitive recursive activity.</para>

<para> It does report in when it starts up, which means you may expect
that this entry indicates the time at which the &lslon; process
began. </para>
</listitem>

<listitem><para> Timestamps are based on the clock time of the &lslon;
process.</para>

<para> In order for the timestamps to be accurate, it is important to
use NTP or similar technology to keep servers' clocks synchronized, as
recommended in <xref linkend="requirements">.  If the host where a
&lslon; runs has its time significantly out of sync with the database
that it manages, queries against &slcomponents; may provide results
that will confuse the reader.</para>
</listitem>

<listitem><para> process.</para> </listitem>
</itemizedlist>
</sect2>
</sect1>

<!-- Keep this comment at the end of the file
Local variables:
mode: sgml
sgml-omittag:nil
sgml-shorttag:t
sgml-minimize-attributes:nil
sgml-always-quote-attributes:t
sgml-indent-step:1
sgml-indent-data:t
sgml-parent-document:"slony.sgml"
sgml-exposed-tags:nil
sgml-local-catalogs:"/usr/lib/sgml/catalog"
sgml-local-ecat-files:nil
End:
-->
