<!--  -->
<sect1 id="faq">
<title>Frequently Asked Questions</title>
<qandaset>
<indexterm><primary>Frequently Asked Questions about &slony1;</primary></indexterm>

<qandadiv id="faqcompiling"><title> &slony1; FAQ: Building and Installing &slony1; </title>

<qandaentry>

<question><para> I am using <productname> Frotznik Freenix
4.5</productname>, with its <acronym>FFPM</acronym> (Frotznik Freenix
Package Manager) package management system.  It comes with
<acronym>FFPM</acronym> packages for &postgres; 7.4.7, which are what
I am using for my databases, but they don't include &slony1; in the
packaging.  How do I add &slony1; to this?  </para>
</question>

<answer><para> <productname>Frotznik Freenix</productname> is new to
me, so it's a bit dangerous to give really hard-and-fast definitive
answers.  </para>

<para> The answers differ somewhat between the various combinations of
&postgres; and &slony1; versions; the newer versions generally
somewhat easier to cope with than are the older versions.  In general,
it is quite likely that you will need to compile &slony1; from
sources; depending on versioning of both &slony1; and &postgres;, you
<emphasis>may</emphasis> also need to compile &postgres; from scratch.
(Whether you need to <emphasis> use </emphasis> the &postgres; compile
is another matter; you probably don't...) </para>

</answer>

<answer><para> In these modern days, many people are using systems
like <application><ulink
url="http://www.puppetlabs.com/">Puppet</ulink></application> where
they expect to be able to get versions of &postgres; and &slony1;
packaged up using their systems' native packaging tools (such as
<application>RPM</application>, with <ulink
url="http://yum.postgresql.org/"> yum.postgresql.org</ulink>,
<application>dpkg</application> - with <ulink
url="http://wiki.postgresql.org/wiki/Apt"> PostgreSQL packages for
Debian and Ubuntu </ulink>).</para>

<para> It is possible that you will be able to find suitable
pre-compiled packages for &postgres; and &slony1; for your system, if
you are using a popular Linux distribution, and there is a single
version of &postgres; across your cluster. </para>

<para> If you are using a system distribution that has not attracted
such packaging, or if you are using &slony1; to upgrade between
&postgres; versions, and therefore require it compiled for multiple
versions of &postgres;, it is quite likely you will need to compile
&slony1; (and perhaps &postgres;) yourself.</para>

<para> You will likely find it useful to consult the appropriate
repository above (YUM, for RPM-based systems, and APT, for Debian
derivatives) to obtain configuration to help you build packages
yourself.</para> </answer>

<answer><para> In the <quote>old days</quote>, the build process was
more arduous than is the case with version 2.0 and above:

<itemizedlist>

<listitem><para> &slony1; version 1.0.5 and earlier required having a
fully configured copy of &postgres; sources available when you compile
&slony1;.</para>

<para> <emphasis>Hopefully</emphasis> you can make the configuration
this closely match against the configuration in use by the packaged
version of &postgres; by checking the configuration using the command
<command> pg_config --configure</command>. </para> </listitem>

<listitem> <para> &slony1; version 1.1 simplified this considerably;
it did not require the full copy of &postgres; sources, but could,
instead, refer to the various locations where &postgres; libraries,
binaries, configuration, and <command> #include </command> files are
located.  </para> </listitem>

<listitem><para> &postgres; 8.0 and higher have become much easier to
deal with in that a <quote>default</quote> installation includes all
of the <command> #include </command> files.  </para>

<para> If you are using an earlier version of &postgres;, you may find
it necessary to resort to a source installation if the packaged
version did not install the <quote>server
<command>#include</command></quote> files, which are installed by the
command <command> make install-all-headers </command>.</para>
</listitem>

</itemizedlist>

<para> In effect, the <quote>worst case</quote> scenario takes place
if you are using a version of &slony1; earlier than 1.1 with an
<quote>elderly</quote> version of &postgres;, in which case you can
expect to need to compile &postgres; from scratch in order to have
everything that the &slony1; compile needs even though you are using a
<quote>packaged</quote> version of &postgres;.</para>

<para> If you are running a recent &postgres; and a recent &slony1;,
then the codependencies can be fairly small, and you may not need
extra &postgres; sources.  These improvements should ease the
production of &slony1; packages so that you might soon even be able to
hope to avoid compiling &slony1;.</para>

</answer>

</qandaentry>

<qandaentry>
<question> <para> Problem building on Fedora/x86-64 </para>

<para> When trying to configure &slony1; on a Fedora x86-64 system,
where <application>yum</application> was used to install the package
<filename>postgresql-libs.x86_64</filename>, the following complaint
comes up:

<screen>
configure: error: Your version of libpq doesn't have PQunescapeBytea
 this means that your version of PostgreSQL is lower than 7.3
 and thus not supported by Slony-I.
</screen></para>

<para> This happened with &postgres; 8.2.5, which is certainly rather
newer than 7.3. </para>
</question>

<answer> <para> <application>configure</application> is looking for
that symbol by compiling a little program that calls for it, and
checking if the compile succeeds.  On the <command>gcc</command>
command line it uses <command>-lpq</command> to search for the
library. </para>

<para> Unfortunately, in this precise case that package is missing a
symlink, from <filename>/usr/lib64/libpq.so</filename> to
<filename>libpq.so.5.0</filename>; that is why it fails to link to
libpq.  The <emphasis>true</emphasis> problem is that the compiler
failed to find a library to link to, not that libpq lacked the
function call. </para>

<para> Eventually, this should be addressed by those that manage the
<filename>postgresql-libs.x86_64</filename> package. </para>
</answer>

<answer> <para> While the problem is reported with a particular
system, the complaint about <quote>version lower than 7.3</quote> has
been a common one, reported across various sorts of systems. </para>

<para> Note that this same symptom has been a common indicatior for
various similar classes of system configuration problems.  Bad
symlinks, bad permissions, bad behaviour on the part of your C
compiler, all may potentially lead to this same error message. </para>

<para> Thus, if you see this error, you need to look in the log file
that is generated, <filename>config.log</filename>.  Search down to
near the end, and see what the <emphasis>actual</emphasis> complaint
was.  That will be helpful in tracking down the true root cause of the
problem.</para>

</answer>

</qandaentry>

<qandaentry>

<question> <para> I found conflicting types for <envar>yyleng</envar>
between <filename>parser.c</filename> and <filename>scan.c</filename>.
In one case, it used type <type>int</type>, conflicting with
<type>yy_size_t</type>.  What shall I do?</para> </question>

<answer><para> This has been observed on <application>MacOS</application>,
where <application>flex</application> (which generates
<filename>scan.c</filename>) and <application>bison</application>
(which generates <filename>parser.c</filename>) diverged in their
handling of this variable. </para> 
<itemizedlist>
<listitem><para> You might might <quote>hack</quote> <filename>scan.c</filename> by hand to use the matching type. </para> </listitem>
<listitem><para> You might select different versions of <application>bison</application> or <application>flex</application> so as to get versions whose data types match. </para> </listitem>
<listitem><para> Note that you may have multiple versions of <application>bison</application> or <application>flex</application> around, and might need to modify <envar>PATH</envar> in order to select the appropriate one.</para></listitem>
</itemizedlist>
</answer>

</qandaentry>

</qandadiv>

<qandadiv id="faqhowto"> <title> &slony1; FAQ: How Do I? </title>

<qandaentry>

<question> <para> I need to dump a database
<emphasis>without</emphasis> getting &slony1; configuration
(<emphasis>e.g.</emphasis> - triggers, functions, and such). </para>
</question>

<answer> <para> Up to version 1.2, this is fairly nontrivial,
requiring careful choice of nodes, and some moderately heavy
<quote>procedure</quote>.  One methodology has been as follows:</para>

<itemizedlist>

<listitem><para> First, dump the schema from the node that has the
<quote>master</quote> role.  That is the only place, pre-2.0, where
you can readily dump the schema using
<application>pg_dump</application> and have a consistent schema.  You
may use the &slony1; tool <xref linkend="extractschema"> to do
this. </para> </listitem>

<listitem><para> Take the resulting schema, which will <emphasis>not</emphasis>
include the &slony1;-specific bits, and split it into two pieces:
</para>

<itemizedlist>

<listitem><para> Firstly, the portion comprising all of the creations
of tables in the schema. </para> </listitem>

<listitem><para> Secondly, the portion consisting of creations of
indices, constraints, and triggers. </para> </listitem>

</itemizedlist>

</listitem>

<listitem><para> Pull a data dump, using <command>pg_dump --data-only</command>, of some node of your choice.  It doesn't need to be for the <quote>master</quote> node.  This dump will include the contents of the &slony1;-specific tables; you can discard that, or ignore it.  Since the schema dump didn't contain table definitions for the &slony1; tables, they won't be loaded. </para> </listitem>

<listitem><para> Finally, load the three components in proper order: </para> 
<itemizedlist>
<listitem><para> Schema (tables) </para> </listitem>
<listitem><para> Data dump </para> </listitem>
<listitem><para> Remainder of the schema </para> </listitem>
</itemizedlist>
</listitem>

</itemizedlist>

</answer>

<answer> <para> In &slony1; 2.0, the answer becomes simpler: Just take
a <command>pg_dump --exclude-schema=_Cluster</command> against
<emphasis>any</emphasis> node.  In 2.0, the schemas are no longer
<quote>clobbered</quote> on subscribers, so a straight
<application>pg_dump</application> will do what you want.</para>

<para> There is a small issue that will remain; the log triggers that
are added to replicated tables will still be included in the dump,
although the functions that they refer to are in the excluded schema.
You should be prepared to find that those triggers will fail to load.
You might use <command>grep -v</command> on the dump to filter out the
triggers, should that prove to be a problem.  </para>
</answer>

</qandaentry>

<qandaentry id="howtorunslon">
<question> <para> How should I run &lslon; processes? </para>
</question>

<answer> <para> A &lslon; process is required for each database that
participates in replication.  There are a variety of ways of managing
these processes, none strictly <quote>the best.</quote> An
administrator may find one or another of these to be
preferable.
</para>

<itemizedlist>

<listitem><para> <link linkend="slonwatchdog"> Perl Watchdog</link>, a
script which monitors a node and restarts &lslon; for that node if it
sees no activity. </listitem>

<listitem><para> <link linkend="startslon">start_slon.sh</link>,
written with a view to managing the &lslon; configuration via the
<link linkend="runtime-config"> slon runtime configuraiton
</link>.</para></listitem>

<listitem><para> <link linkend=
"launchclusters">launch_clusters.sh</link>, which can start up
multiple &lslon; processes.  This does a one-time startup of missing
&lslon; processes, so should be run periodically using a service such
as cron. </para></listitem>

<listitem><para> <link linkend="upstartscript"> Upstart script </link>
may be used to start a &lslon; process when the system boots, suitable
for systems that use Upstart for managing system
services. </para></listitem> 

<listitem><para> <link linkend="bsd-ports-profile"> BSD Ports Profile
</link> provides Apache-style profiles, suitable for systems managing
&slony1; using FreeBSD Ports. </para></listitem>

<listitem><para> <link linkend="slononwindows"> Slon on Windows
</link> describes the use of &lslon; with the Windows
<quote>service</quote> concept.</para></listitem>

<listitem><para> Not provided, but a straightforward task, would be to
create an <quote>init</quote> script suitable for Unix systems that
use SysV <application>init</application>. </para>
</listitem>
</itemizedlist>

<para> A single answer will not fit all, as different operating
systems have different mechanisms for dealing with persistent
processes, and as administrators may wish to have one, or several
&lslon; processes running on a particular server. </para>
</answer>
</qandaentry>

<qandaentry id="cannotrenumbernodes">
<question> <para> I'd like to renumber the node numbers in my cluster.
How can I renumber nodes? </para> </question>

<answer> <para> The first answer is <quote>you can't do that</quote> -
&slony1; node numbers are quite <quote>immutable.</quote> Node numbers
are deeply woven into the fibres of the schema, by virtue of being
written into virtually every table in the system, but much more
importantly by virtue of being used as the basis for event
propagation.  The only time that it might be <quote>OK</quote> to
modify a node number is at some time where we know that it is not in
use, and we would need to do updates against each node in the cluster
in an organized fashion.</para>

<para> To do this in an automated fashion seems like a
<emphasis>huge</emphasis> challenge, as it changes the structure of
the very event propagation system that already needs to be working in
order for such a change to propagate.</para> </answer>

<answer> <para> If it is <emphasis>enormously necessary</emphasis> to
renumber nodes, this might be accomplished by dropping and re-adding
nodes to get rid of the node formerly using the node ID that needs to
be held by another node.</para> </answer>
</qandaentry>

</qandadiv>

<qandadiv id="faqimpossibilities"> <title> &slony1; FAQ: Impossible Things People Try </title>

<qandaentry>

<question><para> Can I use &slony1; to replicate changes back and
forth on my database between my two offices? </para> </question>

<answer><para> At one level, it is <emphasis>theoretically
possible</emphasis> to do something like that, if you design your
application so that each office has its own distinct set of tables,
and you then have some system for consolidating the data to give them
some common view.  However, this requires a great deal of design work
to create an application that performs this consolidation. </para>
</answer>

<answer><para> In practice, the term for that is <quote>multimaster
replication,</quote> and &slony1; does not support <quote>multimaster
replication.</quote> </para> </answer>

</qandaentry>

<qandaentry>

<question><para> I want to replicate all of the databases for a
shared-database system I am managing.  There are multiple databases,
being used by my customers.  </para> </question>

<answer><para> &slony1; cannot be directly used for this, as it
expects to capture changes from <emphasis>a database</emphasis>, not a
set of <emphasis>several databases</emphasis>.</para></answer>

<answer><para> For this purpose, something like &postgres; PITR (Point
In Time Recovery) is likely to be much more suitable.  &slony1;
requires a slon process (and multiple connections) for each
identifiable database, and if you have a &postgres; cluster hosting 50
or 100 databases, this will require hundreds of database connections.
Typically, in <quote>shared hosting</quote> situations, DML is being
managed by customers, who can change anything they like whenever
<emphasis>they</emphasis> want.  &slony1; does not work out well when
not used in a disciplined manner.  </para> </answer>
</qandaentry>

<qandaentry>
<question><para> I want to be able to make DDL changes, and have them replicated automatically. </para> </question>

<answer><para> &slony1; requires that <xref linkend="ddlchanges"> be planned for explicitly and carefully.  &slony1; captures changes using triggers, and &postgres; does not provide a way to use triggers to capture DDL changes.</para>

<note><para> There has been quite a bit of discussion, off and on,
about how &postgres; might capture DDL changes in a way that would
make triggers useful; nothing concrete has yet emerged despite several
years of discussion. </para> 

<para> The EVENT TRIGGERs added in &postgres; 9.3 are drawing steps
closer on this, but a usable solution has not yet emerged.  Perhaps in
9.4...</para>

</note> </answer>
</qandaentry>

<qandaentry>

<question><para> I want to split my cluster into disjoint partitions
that are not aware of one another.  &slony1; keeps generating <xref
linkend="listenpaths"> that link those partitions together. </para>
</question>

<answer><para> The notion that all nodes are aware of one another is
deeply imbedded in the design of &slony1;.  For instance, its handling
of cleanup of obsolete data depends on being aware of whether any of
the nodes are behind, and thus might still depend on older data.
</para> </answer>
</qandaentry>

<qandaentry>
<question><para> I want to change some of my node numbers.  How do I <quote>rename</quote> a node to have a different node number? </para> </question>

<answer><para> You don't.  The node number is used to coordinate inter-node communications, and changing the node ID number <quote>on the fly</quote> would make it essentially impossible to keep node configuration coordinated.   </para> </answer>
</qandaentry>

<qandaentry>
<question> <para> My application uses OID attributes; is it possible to replicate tables like this? </para>
</question>

<answer><para> It is worth noting that oids, as a regular table
attribute, have been deprecated since &postgres; version 8.1, back in
2005.  &slony1; has <emphasis>never</emphasis> collected oids to
replicate them, and, with that functionality being deprecated, the
developers do not intend to add this functionality. </para>

<para> &postgres; implemented oids as a way to link its internal
system tables together; to use them with application tables is
considered <emphasis>poor practice</emphasis>, and it is recommended
that you use sequences to populate your own ID column on application
tables.  </para> </answer>

<answer><para> Of course, nothing prevents you from creating a table
<emphasis>without</emphasis> oids, and then add in your own
application column called <envar>oid</envar>, preferably with type
information <command>SERIAL NOT NULL UNIQUE</command>, which
<emphasis>can</emphasis> be replicated, and which is likely to be
suitable as a candidate primary key for the table. </para> </answer>
</qandaentry>
</qandadiv>

<qandadiv id="faqconnections"> <title> &slony1; FAQ: Connection Issues </title>
<qandaentry>

<question><para>I looked for the <envar>_clustername</envar> namespace, and
it wasn't there.</para></question>

<answer><para> If the DSNs are wrong, then &lslon;
instances can't connect to the nodes.</para>

<para>This will generally lead to nodes remaining entirely untouched.</para>

<para>Recheck the connection configuration.  By the way, since <xref
linkend="slon"> links to libpq, you could have password information
stored in <filename> $HOME/.pgpass</filename>, partially filling in
right/wrong authentication information there.</para>
</answer>
</qandaentry>

<qandaentry>
<question><para> I'm trying to get a slave subscribed, and get the
following messages in the logs:

<screen>
DEBUG1 copy_set 1
DEBUG1 remoteWorkerThread_1: connected to provider DB
WARN	remoteWorkerThread_1: transactions earlier than XID 127314958 are still in progress
WARN	remoteWorkerThread_1: data copy for set 1 failed - sleep 60 seconds
</screen></para></question>

<answer> <para> There is evidently some reasonably old outstanding
transaction blocking &slony1; from processing the sync.  You might
want to take a look at pg_locks to see what's up:</para>

<screen>
sampledb=# select * from pg_locks where transaction is not null order by transaction;
 relation | database | transaction |  pid    |     mode      | granted 
----------+----------+-------------+---------+---------------+---------
          |          |   127314921 | 2605100 | ExclusiveLock | t
          |          |   127326504 | 5660904 | ExclusiveLock | t
(2 rows)
</screen>

<para>See?  127314921 is indeed older than 127314958, and it's still
running.  While the situation may be undesirable, there is no error in
the behaviour of &slony1;.</para>

<para> A long running accounting report, a runaway search query, a
<application>pg_dump</application>, all can open up transactions that
may run for substantial periods of time.  Until they complete, or are
interrupted, you will continue to see the message <quote> data copy
for set 1 failed - sleep 60 seconds </quote>.</para>

<para>By the way, if there is more than one database on the &postgres;
cluster, and activity is taking place on the OTHER database, that will
lead to there being <quote>transactions earlier than XID
whatever</quote> being found to be still in progress.  The fact that
it's a separate database on the cluster is irrelevant; &slony1; will
wait until those old transactions terminate.</para>
</answer>
</qandaentry>

<qandaentry>
<question><para>Same as the above.  What I forgot to mention, as well,
was that I was trying to add <emphasis>TWO</emphasis> subscribers,
concurrently.</para></question>

<answer><para> That doesn't work out happily: &slony1; can't work on
the <command>COPY</command> commands concurrently.  See
<filename>src/slon/remote_worker.c</filename>, function
<function>copy_set()</function></para>

<screen>
$ ps -aef | egrep '[2]605100'
postgres 2605100  205018	0 18:53:43  pts/3  3:13 postgres: postgres sampledb localhost COPY 
</screen>

<para>This happens to be a <command>COPY</command> transaction
involved in setting up the subscription for one of the nodes.  All is
well; the system is busy setting up the first subscriber; it won't
start on the second one until the first one has completed subscribing.
That represents one possible cause.</para>

<para>This has the (perhaps unfortunate) implication that you cannot
populate two slaves concurrently from a single provider.  You have to
subscribe one to the set, and only once it has completed setting up
the subscription (copying table contents and such) can the second
subscriber start setting up the subscription.</para></answer>
</qandaentry>

<qandaentry id="missingoids"> <question> <para> We got bitten by
something we didn't foresee when completely uninstalling a slony
replication cluster from the master and slave...</para>

<warning> <para><emphasis>MAKE SURE YOU STOP YOUR APPLICATION RUNNING
AGAINST YOUR MASTER DATABASE WHEN REMOVING THE WHOLE SLONY
CLUSTER</emphasis>, or at least re-cycle all your open connections
after the event!  </para></warning>

<para> The connections <quote>remember</quote> or refer to OIDs which
are removed by the uninstall node script. And you will get lots of
errors as a result...
</para>

</question>

<answer><para> There are two notable areas of
&postgres; that cache query plans and OIDs:</para>
<itemizedlist>
<listitem><para> Prepared statements</para></listitem>
<listitem><para> pl/pgSQL functions</para></listitem>
</itemizedlist>

<para> The problem isn't particularly a &slony1; one; it would occur
any time such significant changes are made to the database schema.  It
shouldn't be expected to lead to data loss, but you'll see a wide
range of OID-related errors.
</para></answer>

<answer><para> The problem occurs when you are using some sort of
<quote>connection pool</quote> that keeps recycling old connections.
If you restart the application after this, the new connections will
create <emphasis>new</emphasis> query plans, and the errors will go
away.  If your connection pool drops the connections, and creates new
ones, the new ones will have <emphasis>new</emphasis> query plans, and
the errors will go away. </para></answer>

<answer> <para> In our code we drop the connection on any error we
cannot map to an expected condition. This would eventually recycle all
connections on such unexpected problems after just one error per
connection.  Of course if the error surfaces as a constraint violation
which is a recognized condition, this won't help either, and if the
problem is persistent, the connections will keep recycling which will
drop the effect of the pooling, in the latter case the pooling code
could also announce an admin to take a look...  </para> </answer>

</qandaentry>

<qandaentry><question><para> I'm now the following notice in the
logs:</para>

<screen>NOTICE:  Slony-I: log switch to sl_log_2 still in progress - sl_log_1 not truncated</screen>

<para> Both <envar>sl_log_1</envar> and <envar>sl_log_2</envar> are
continuing to grow, and <envar>sl_log_1</envar> is never getting
truncated.  What's wrong? </para> </question>

<answer><para> Observe that it is indicated as a
<emphasis>NOTICE</emphasis>, not as an <emphasis>ERROR</emphasis>, so
this is an expected sort of behaviour.</para></answer>

<answer><para> This may be symptomatic of the same issue as above with
dropping replication: if there are still old connections lingering
that are using old query plans that reference the old stored
functions, resulting in the inserts to <envar>sl_log_1</envar> </para>

<para> Closing those connections and opening new ones will resolve the
issue. </para> </answer>
</qandaentry>

<qandaentry>
<question><para>I pointed a subscribing node to a different provider
and it stopped replicating</para></question>

<answer><para>
We noticed this happening when we wanted to re-initialize a node,
where we had configuration thus:

<itemizedlist>
<listitem><para> Node 1 - provider</para></listitem>
<listitem><para> Node 2 - subscriber to node 1 - the node we're reinitializing</para></listitem>
<listitem><para> Node 3 - subscriber to node 2 - node that should keep replicating</para></listitem>
</itemizedlist></para>

<para>The subscription for node 3 was changed to have node 1 as
provider, and we did <xref linkend="stmtdropset"> /<xref
linkend="stmtsubscribeset"> for node 2 to get it repopulating.</para>

<para>Unfortunately, replication suddenly stopped to node 3.</para>

<para>The problem was that there was not a suitable set of
<quote>listener paths</quote> in <xref linkend="table.sl-listen"> to
allow the events from node 1 to propagate to node 3.  The events were
going through node 2, and blocking behind the <xref
linkend="stmtsubscribeset"> event that node 2 was working on.</para>

<para>The following slonik script dropped out the listen paths where
node 3 had to go through node 2, and added in direct listens between
nodes 1 and 3.

<programlisting>
cluster name = oxrslive;
 node 1 admin conninfo='host=32.85.68.220 dbname=oxrslive user=postgres port=5432';
 node 2 admin conninfo='host=32.85.68.216 dbname=oxrslive user=postgres port=5432';
 node 3 admin conninfo='host=32.85.68.244 dbname=oxrslive user=postgres port=5432';
 node 4 admin conninfo='host=10.28.103.132 dbname=oxrslive user=postgres port=5432';
try {
  store listen (origin = 1, receiver = 3, provider = 1);
  store listen (origin = 3, receiver = 1, provider = 3);
  drop listen (origin = 1, receiver = 3, provider = 2);
  drop listen (origin = 3, receiver = 1, provider = 2);
}
</programlisting></para>

<para>Immediately after this script was run, <command>SYNC</command>
events started propagating again to node 3.

This points out two principles:
<itemizedlist>

<listitem><para> If you have multiple nodes, and cascaded subscribers,
you need to be quite careful in populating the <xref
linkend="stmtstorelisten"> entries, and in modifying them if the
structure of the replication <quote>tree</quote>
changes.</para></listitem>

<listitem><para> Version 1.1 provides better tools to help manage
this, and later versions of &slony1; <emphasis>hopefully</emphasis>
populate the listener table automatically <emphasis>and
correctly</emphasis> such that you should not encounter this
problem. </para>
</listitem>

</itemizedlist></para>

<para>The issues of <quote>listener paths</quote> are discussed
further at <xref linkend="listenpaths"> </para></answer>
</qandaentry>

<qandaentry id="multipleslonconnections">

<question><para> I was starting a &lslon;, and got the
following <quote>FATAL</quote> messages in its logs.  What's up??? </para>
<screen>
2006-03-29 16:01:34 UTC CONFIG main: slon version 1.2.0 starting up
2006-03-29 16:01:34 UTC DEBUG2 slon: watchdog process started
2006-03-29 16:01:34 UTC DEBUG2 slon: watchdog ready - pid = 28326
2006-03-29 16:01:34 UTC DEBUG2 slon: worker process created - pid = 28327
2006-03-29 16:01:34 UTC CONFIG main: local node id = 1
2006-03-29 16:01:34 UTC DEBUG2 main: main process started
2006-03-29 16:01:34 UTC CONFIG main: launching sched_start_mainloop
2006-03-29 16:01:34 UTC CONFIG main: loading current cluster configuration
2006-03-29 16:01:34 UTC CONFIG storeSet: set_id=1 set_origin=1 set_comment='test set'
2006-03-29 16:01:34 UTC DEBUG2 sched_wakeup_node(): no_id=1 (0 threads + worker signaled)
2006-03-29 16:01:34 UTC DEBUG2 main: last local event sequence = 7
2006-03-29 16:01:34 UTC CONFIG main: configuration complete - starting threads
2006-03-29 16:01:34 UTC DEBUG1 localListenThread: thread starts
2006-03-29 16:01:34 UTC FATAL  localListenThread: "select "_test1538".cleanupNodelock(); insert into "_test1538".sl_nodelock values (    1, 0, "pg_catalog".pg_backend_pid()); " - ERROR:  duplicate key violates unique constraint "sl_nodelock-pkey"

2006-03-29 16:01:34 UTC FATAL  Do you already have a slon running against this node?
2006-03-29 16:01:34 UTC FATAL  Or perhaps a residual idle backend connection from a dead slon?
</screen>

</question>

<answer><para> The table <envar>sl_nodelock</envar> is used as an
<quote>interlock</quote> to prevent two &lslon; processes from trying
to manage the same node at the same time.  The &lslon; tries inserting
a record into the table; it can only succeed if it is the only node
manager. </para></answer>

<answer><para> This error message is typically a sign that you have
started up a second &lslon; process for a given node.  The &lslon; asks
the obvious question: <quote>Do you already have a slon running
against this node?</quote> </para></answer>

<answer><para> Supposing you experience some sort of network outage,
the connection between &lslon; and database may fail, and the &lslon;
may figure this out long before the &postgres; instance it was
connected to does.  The result is that there will be some number of
idle connections left on the database server, which won't be closed
out until TCP/IP timeouts complete, which seems to normally take about
two hours.  For that two hour period, the &lslon; will try to connect,
over and over, and will get the above fatal message, over and
over. </para>

<para> An administrator may clean this out by logging onto the server
and issuing <command>kill -2</command> to any of the offending
connections.  Unfortunately, since the problem took place within the
networking layer, neither &postgres; nor &slony1; have a direct way of
detecting this. </para> 

<para> You can <emphasis>mostly</emphasis> avoid this by making sure
that &lslon; processes always run somewhere nearby the server that
each one manages.  If the &lslon; runs on the same server as the
database it manages, any <quote>networking failure</quote> that could
interrupt local connections would be likely to be serious enough to
threaten the entire server.  </para></answer>

<answer><para> It is likely that the <quote>several hours</quote> TCP
timeout is rather longer than it should be in these days where people
expect their servers to be pretty synchronously connected.
Configuring the &postgres; to use much shorter timeouts is likely a
wise idea.  The relevant <filename>postgresql.conf</filename>
parameters are <envar>tcp_keepalives_idle</envar>,
<envar>tcp_keepalives_interval</envar>, and
<envar>tcp_keepalives_count</envar>.  See <command>man 7 tcp</command>
for more details.  </para></answer>
</qandaentry>

<qandaentry>
<question><para> When can I shut down &lslon; processes?</para></question>

<answer><para> Generally, when things are running properly, it's no
big deal to shut down a &lslon; process.  Each one is
<quote>merely</quote> a &postgres; client, managing one node, which
spawns threads to manage receiving events from other nodes.  </para>

<para>The <quote>event listening</quote> threads are no big deal; they
are doing nothing fancier than periodically checking remote nodes to
see if they have work to be done on this node.  If you kill off the
&lslon; these threads will be closed, which should have little or no
impact on much of anything.  Events generated while the &lslon; is
down will be picked up when it is restarted.</para>

<para> The <quote>node managing</quote> thread is a bit more
interesting; most of the time, you can expect, on a subscriber, for
this thread to be processing <command>SYNC</command> events.  If you
shut off the &lslon; during an event, the transaction
will fail, and be rolled back, so that when the &lslon; restarts, it
will have to go back and reprocess the event.</para>

<para> The only situation where this will cause
<emphasis>particular</emphasis> <quote>heartburn</quote> is if the
event being processed was one which takes a long time to process, the
worst case being where <command>COPY_SET</command> is doing initial
processing of a large replication set. </para>

<para> The other thing that <emphasis>might</emphasis> cause trouble
is if the &lslon; runs fairly distant from nodes that it connects to;
you could discover that database connections are left <command>idle in
transaction</command>.  This would normally only occur if the network
connection is destroyed without either &lslon; or database being made
aware of it.  In that case, you may discover that
<quote>zombied</quote> connections are left around for as long as two
hours if you don't go in by hand and kill off the &postgres; backends.
See the <link linkend="multipleslonconnections">Multiple Slon
Connections</link> entry for more details.</para>

<para> There is one other case that could cause trouble; when the
&lslon; managing the origin node is not running, no
<command>SYNC</command> events run against that node.  If the &lslon;
stays down for an extended period of time, and something like <xref
linkend="gensync"> isn't running, you could be left with <emphasis>one
big <command>SYNC</command></emphasis> to process when it comes back
up.  But that should only be a concern if that &lslon; is down for an
extended period of time; shutting it down for a few seconds shouldn't
cause any great problem. </para> </answer>
</qandaentry>

<qandaentry>
<question><para> Are there risks to shutting down the slon?  How about
benefits?</para></question>

<answer><para> In short, if you don't have something like an 18 hour
<command>COPY_SET</command> under way, it's normally not at all a big
deal to take a &lslon; down for a little while, or perhaps even cycle
<emphasis>all</emphasis> the &lslon; processes. </para> </answer>
</qandaentry>

<qandaentry>
<question><para> I was trying to subscribe a set involving a multiple GB table, and it failed.</para> 
<screen>
Jul 31 22:52:53 dbms TICKER[70295]: [153-1] CONFIG remoteWorkerThread_3: copy table "public"."images"
Jul 31 22:52:53 dbms TICKER[70295]: [154-1] CONFIG remoteWorkerThread_3: Begin COPY of table "public"."images"
Jul 31 22:54:24 dbms TICKER[70295]: [155-1] ERROR  remoteWorkerThread_3: PGgetCopyData() server closed the connection unexpectedly
Jul 31 22:54:24 dbms TICKER[70295]: [155-2]     This probably means the server terminated abnormally
Jul 31 22:54:24 dbms TICKER[70295]: [155-3]     before or while processing the request.
Jul 31 22:54:24 dbms TICKER[70295]: [156-1] WARN   remoteWorkerThread_3: data copy for set 1 failed 1 times - sleep 15 seconds
</screen>
<para> Oh, by the way, I'm using SSL-based &postgres; conenctions. </para></question>
<answer><para> A further examination of &postgres; logs indicated errors of the form: </para> 
<screen>
Jul 31 23:00:00 tickerforum postgres[27093]: [9593-2] STATEMENT:  copy "public"."images"
("post_ordinal","ordinal","caption","image","login","file_type","thumb","thumb_width","thumb_height","hidden") to stdout;
Jul 31 23:00:00 tickerforum postgres[27093]: [9594-1] LOG:  SSL error: internal error
Jul 31 23:00:00 tickerforum postgres[27093]: [9594-2] STATEMENT:  copy "public"."images" ("post_ordinal","ordinal","caption","image","login","file_type","thumb","thumb_width","thumb_height","hidden") to stdout;
Jul 31 23:00:01 tickerforum postgres[27093]: [9595-1] LOG:  SSL error: internal error
</screen>

<para> This demonstrates a problem with &postgres; handling of
SSL connections, which is <quote>out of scope</quote> for &slony1;
proper (<emphasis>e.g.</emphasis> - there's no <quote>there</quote>
inside &slony1; for us to try to fix).  </para> 

<para> The resolution to the underlying problem will presumably be
handled upstream in the &postgres; project; the workaround is to, at
least for the initial <command>SUBSCRIBE SET</command> event, switch
to a non-SSL &postgres; connection.  </para> </answer>
</qandaentry>

</qandadiv>

<qandadiv id="faqconfiguration"> <title> &slony1; FAQ: Configuration Issues </title>

<qandaentry>
<question><para>ps finds passwords on command line</para>

<para> If I run a <command>ps</command> command, I, and everyone else,
can see passwords on the command line.</para></question>

<answer> <para>Take the passwords out of the Slony configuration, and
put them into <filename>$(HOME)/.pgpass.</filename></para>
</answer></qandaentry>

<qandaentry>
<question><para>Table indexes with FQ namespace names

<programlisting>
set add table (set id = 1, origin = 1, id = 27, 
               full qualified name = 'nspace.some_table', 
               key = 'key_on_whatever', 
               comment = 'Table some_table in namespace nspace with a candidate primary key');
</programlisting></para></question>

<answer><para> If you have <command> key =
'nspace.key_on_whatever'</command> the request will
<emphasis>FAIL</emphasis>.</para>
</answer></qandaentry>

<qandaentry>
<question> <para> Replication has fallen behind, and it appears that
the queries to draw data from &sllog1;/&sllog2; are taking a long time
to pull just a few
<command>SYNC</command>s. </para>
</question>

<answer> <para> Until version 1.1.1, there was only one index on
&sllog1;/&sllog2;, and if there were multiple replication sets, some
of the columns on the index would not provide meaningful selectivity.
If there is no index on column <function> log_xid</function>, consider
adding it.  See <filename>slony1_base.sql</filename> for an example of
how to create the index.
</para>
</answer>

<answer><para> That's not quite the end of the story; a critical query
had the potential to scan the entirety of these log tables even in
&slony1; 2.1.  You generally want to be on the latest version of
&slony1; as there have been efficiency improvements, and in this area,
pretty substantial improvements.</para></answer>
</qandaentry>

<qandaentry><question> <para> I need to rename a column that is in the
primary key for one of my replicated tables.  That seems pretty
dangerous, doesn't it?  I have to drop the table out of replication
and recreate it, right?</para>
</question>

<answer><para> Actually, this is a scenario which works out remarkably
cleanly.  &slony1; does indeed make intense use of the primary key
columns, but actually does so in a manner that allows this sort of
change to be made very nearly transparently.</para>

<para> Suppose you revise a column name, as with the SQL DDL <command>
alter table accounts alter column aid rename to cid; </command> This
revises the names of the columns in the table; it
<emphasis>simultaneously</emphasis> renames the names of the columns
in the primary key index.  The result is that the normal course of
things is that altering a column name affects both aspects
simultaneously on a given node.</para>

<para> The <emphasis>ideal</emphasis> and proper handling of this
change would involve using <xref linkend="stmtddlscript"> to deploy
the alteration, which ensures it is applied at exactly the right point
in the transaction stream on each node.</para>

<para> Interestingly, that isn't forcibly necessary.  As long as the
alteration is applied on the replication set's origin before
application on subscribers, things won't break irrepairably.  Some
<command>SYNC</command> events that do not include changes to the
altered table can make it through without any difficulty...  At the
point that the first update to the table is drawn in by a subscriber,
<emphasis>that</emphasis> is the point at which
<command>SYNC</command> events will start to fail, as the provider
will indicate the <quote>new</quote> set of columns whilst the
subscriber still has the <quote>old</quote> ones.  If you then apply
the alteration to the subscriber, it can retry the
<command>SYNC</command>, at which point it will, finding the
<quote>new</quote> column names, work just fine.
</para> </answer></qandaentry>


<qandaentry>
<question> <para> I had a network <quote>glitch</quote> that led to my
using <xref linkend="stmtfailover"> to fail over to an alternate node.
The failure wasn't a disk problem that would corrupt databases; why do
I need to rebuild the failed node from scratch? </para></question>

<answer><para> The action of <xref linkend="stmtfailover"> is to
<emphasis>abandon</emphasis> the failed node so that no more &slony1;
activity goes to or from that node.  As soon as that takes place, the
failed node will progressively fall further and further out of sync.
</para></answer>

<answer><para> The <emphasis>big</emphasis> problem with trying to
recover the failed node is that it may contain updates that never made
it out of the origin.  If they get retried, on the new origin, you may
find that you have conflicting updates.  In any case, you do have a
sort of <quote>logical</quote> corruption of the data even if there
never was a disk failure making it <quote>physical.</quote>
</para></answer>

<answer><para> As discusssed in <xref linkend="failover">, using <xref
linkend="stmtfailover"> should be considered a <emphasis>last
resort</emphasis> as it implies that you are abandoning the origin
node as being corrupted.  </para></answer>
</qandaentry>

<qandaentry> <question><para> After notification of a subscription on
<emphasis>another</emphasis> node, replication falls over on one of
the subscribers, with the following error message:</para>

<screen>
ERROR  remoteWorkerThread_1: "begin transaction; set transaction isolation level serializable; lock table "_livesystem".sl_config_lock; select "_livesystem".enableSubscription(25506, 1, 501); notify "_livesystem_Event"; notify "_livesystem_Confirm"; insert into "_livesystem".sl_event     (ev_origin, ev_seqno, ev_timestamp,      ev_minxid, ev_maxxid, ev_xip, ev_type , ev_data1, ev_data2, ev_data3, ev_data4    ) values ('1', '4896546', '2005-01-23 16:08:55.037395', '1745281261', '1745281262', '', 'ENABLE_SUBSCRIPTION', '25506', '1', '501', 't'); insert into "_livesystem".sl_confirm      (con_origin, con_received, con_seqno, con_timestamp)    values (1, 4, '4896546', CURRENT_TIMESTAMP); commit transaction;" PGRES_FATAL_ERROR ERROR:  insert or update on table "sl_subscribe" violates foreign key constraint "sl_subscribe-sl_path-ref"
DETAIL:  Key (sub_provider,sub_receiver)=(1,501) is not present in table "sl_path".
</screen>

<para> This is then followed by a series of failed syncs as the <xref
linkend="slon"> shuts down:</para>

<screen>
DEBUG2 remoteListenThread_1: queue event 1,4897517 SYNC
DEBUG2 remoteListenThread_1: queue event 1,4897518 SYNC
DEBUG2 remoteListenThread_1: queue event 1,4897519 SYNC
DEBUG2 remoteListenThread_1: queue event 1,4897520 SYNC
DEBUG2 remoteWorker_event: ignore new events due to shutdown
DEBUG2 remoteListenThread_1: queue event 1,4897521 SYNC
DEBUG2 remoteWorker_event: ignore new events due to shutdown
DEBUG2 remoteListenThread_1: queue event 1,4897522 SYNC
DEBUG2 remoteWorker_event: ignore new events due to shutdown
DEBUG2 remoteListenThread_1: queue event 1,4897523 SYNC
</screen>

</question>

<answer><para> If you see a &lslon; shutting down with
<emphasis>ignore new events due to shutdown</emphasis> log entries,
you typically need to step back in the log to
<emphasis>before</emphasis> they started failing to see indication of
the root cause of the problem.  </para></answer>

<answer><para> In this particular case, the problem was that some of
the <xref linkend="stmtstorepath"> commands had not yet made it to
node 4 before the <xref linkend="stmtsubscribeset"> command
propagated. </para>

<para>This demonstrates yet another example of the need to not do
things in a rush; you need to be sure things are working right
<emphasis>before</emphasis> making further configuration changes.
</para></answer>

</qandaentry>

<qandaentry>

<question><para>I just used <xref linkend="stmtmoveset"> to move the
origin to a new node.  Unfortunately, some subscribers are still
pointing to the former origin node, so I can't take it out of service
for maintenance without stopping them from getting updates.  What do I
do?  </para></question>

<answer><para> You need to use <xref linkend="stmtsubscribeset"> to
alter the subscriptions for those nodes to have them subscribe to a
provider that <emphasis>will</emphasis> be sticking around during the
maintenance.</para>

<warning> <para> What you <emphasis>don't</emphasis> do is to <xref
linkend="stmtunsubscribeset">; that would require reloading all data
for the nodes from scratch later.

</para></warning>
</answer>
</qandaentry>

<qandaentry>
<question><para> After notification of a subscription on
<emphasis>another</emphasis> node, replication falls over, starting
with the following error message:</para>

<screen>
ERROR  remoteWorkerThread_1: "begin transaction; set transaction isolation level serializable; lock table "_livesystem".sl_config_lock; select "_livesystem".enableSubscription(25506, 1, 501); notify "_livesystem_Event"; notify "_livesystem_Confirm"; insert into "_livesystem".sl_event     (ev_origin, ev_seqno, ev_timestamp,      ev_minxid, ev_maxxid, ev_xip, ev_type , ev_data1, ev_data2, ev_data3, ev_data4    ) values ('1', '4896546', '2005-01-23 16:08:55.037395', '1745281261', '1745281262', '', 'ENABLE_SUBSCRIPTION', '25506', '1', '501', 't'); insert into "_livesystem".sl_confirm      (con_origin, con_received, con_seqno, con_timestamp)    values (1, 4, '4896546', CURRENT_TIMESTAMP); commit transaction;" PGRES_FATAL_ERROR ERROR:  insert or update on table "sl_subscribe" violates foreign key constraint "sl_subscribe-sl_path-ref"
DETAIL:  Key (sub_provider,sub_receiver)=(1,501) is not present in table "sl_path".
</screen>

<para> This is then followed by a series of failed syncs as the <xref
linkend="slon"> shuts down:

<screen>
DEBUG2 remoteListenThread_1: queue event 1,4897517 SYNC
DEBUG2 remoteListenThread_1: queue event 1,4897518 SYNC
DEBUG2 remoteListenThread_1: queue event 1,4897519 SYNC
DEBUG2 remoteListenThread_1: queue event 1,4897520 SYNC
DEBUG2 remoteWorker_event: ignore new events due to shutdown
DEBUG2 remoteListenThread_1: queue event 1,4897521 SYNC
DEBUG2 remoteWorker_event: ignore new events due to shutdown
DEBUG2 remoteListenThread_1: queue event 1,4897522 SYNC
DEBUG2 remoteWorker_event: ignore new events due to shutdown
DEBUG2 remoteListenThread_1: queue event 1,4897523 SYNC
</screen>

</para></question>

<answer><para> If you see a &lslon; shutting down with
<emphasis>ignore new events due to shutdown</emphasis> log entries,
you'll typically have to step back to <emphasis>before</emphasis> they
started failing to see indication of the root cause of the problem.

</para></answer>

<answer><para> In this particular case, the problem was that some of
the <xref linkend="stmtstorepath"> commands had not yet made it to
node 4 before the <xref linkend="stmtsubscribeset"> command
propagated. </para>

<para>This is yet another example of the need to not do things too
terribly quickly; you need to be sure things are working right
<emphasis>before</emphasis> making further configuration changes.
</para></answer>

</qandaentry>

<qandaentry>
<question> <para> Is the ordering of tables in a set significant?</para>
</question>

<answer> <para> Most of the time, it isn't.  You might imagine it of
some value to order the tables in some particular way in order that
<quote>parent</quote> entries would make it in before their
<quote>children</quote> in some foreign key relationship; that
<emphasis>isn't</emphasis> the case since foreign key constraint
triggers are turned off on subscriber nodes.
</para>
</answer>

<answer> <para>(Jan Wieck comments) The order of table ID's is only
significant during a <xref linkend="stmtlockset"> in preparation of
switchover. If that order is different from the order in which an
application is acquiring its locks, it can lead to deadlocks that
abort either the application or <application>slon</application>.
</para>
</answer>

<answer><para> (David Parker) I ran into one other case where the
ordering of tables in the set was significant: in the presence of
inherited tables. If a child table appears before its parent in a set,
then the initial subscription will end up deleting that child table
after it has possibly already received data, because the
<command>copy_set</command> logic does a <command>delete</command>,
not a <command>delete only</command>, so the delete of the parent will
delete the new rows in the child as well.
</para>
</answer>
</qandaentry>

<qandaentry>

<question><para> If you have a <xref linkend="slonik"> script
something like this, it will hang on you and never complete, because
you can't have <xref linkend="stmtwaitevent"> inside a <xref
linkend="tryblock"> block. A <command>try</command> block is executed
as one transaction, and the event that you are waiting for can never
arrive inside the scope of the transaction.</para>

<programlisting>
try {
      echo 'Moving set 1 to node 3';
      lock set (id=1, origin=1);
      echo 'Set locked';
      wait for event (origin = 1, confirmed = 3);
      echo 'Moving set';
      move set (id=1, old origin=1, new origin=3);
      echo 'Set moved - waiting for event to be confirmed by node 3';
      wait for event (origin = 1, confirmed = 3);
      echo 'Confirmed';
} on error {
      echo 'Could not move set for cluster foo';
      unlock set (id=1, origin=1);
      exit -1;
}
</programlisting></question>

<answer><para> You <emphasis>must not</emphasis> invoke <xref
linkend="stmtwaitevent"> inside a <quote>try</quote>
block.</para></answer>

</qandaentry>

<qandaentry>
<question><para>Slony-I: cannot add table to currently subscribed set 1</para>

<para> I tried to add a table to a set, and got the following message:

<screen>
	Slony-I: cannot add table to currently subscribed set 1
</screen></para></question>

<answer><para> You cannot add tables to sets that already have
subscribers.</para>

<para>The workaround to this is to create <emphasis>ANOTHER</emphasis>
set, add the new tables to that new set, subscribe the same nodes
subscribing to "set 1" to the new set, and then merge the sets
together.</para>
</answer></qandaentry>

<qandaentry>
<question><para>
ERROR: duplicate key violates unique constraint "sl_table-pkey"</para>

<para>I tried setting up a second replication set, and got the following error:

<screen>
stdin:9: Could not create subscription set 2 for oxrslive!
stdin:11: PGRES_FATAL_ERROR select "_oxrslive".setAddTable(2, 1, 'public.replic_test', 'replic_test__Slony-I_oxrslive_rowID_key', 'Table public.replic_test without primary key');  - ERROR:  duplicate key violates unique constraint "sl_table-pkey"
CONTEXT:  PL/pgSQL function "setaddtable_int" line 71 at SQL statement
</screen></para></question>

<answer><para> The table IDs used in <xref linkend="stmtsetaddtable">
are required to be unique <emphasis>ACROSS ALL SETS</emphasis>.  Thus,
you can't restart numbering at 1 for a second set; if you are
numbering them consecutively, a subsequent set has to start with IDs
after where the previous set(s) left off.</para> </answer>
</qandaentry>

<qandaentry>
<question><para> One of my nodes fell over (&lslon; / postmaster was
down) and nobody noticed for several days.  Now, when the &lslon; for
that node starts up, it runs for about five minutes, then terminates,
with the error message: <command>ERROR: remoteListenThread_%d: timeout
for event selection</command> What's wrong, and what do I do? </para> 
</question>

<answer><para> The problem is that the listener thread (in
<filename>src/slon/remote_listener.c</filename>) timed out when trying
to determine what events were outstanding for that node.  By default,
the query will run for five minutes; if there were many days worth of
outstanding events, this might take too long.  </para> </answer>

<answer><para> On versions of &slony1; before 1.1.7, 1.2.7, and 1.3, a
solution was to increase the timeout in
<filename>src/slon/remote_listener.c</filename>, recompile &lslon;,
and retry. </para> </answer>

<answer><para> Another would be to treat the node as having failed,
and use the &lslonik; command <xref linkend="stmtdropnode"> to drop the
node, and recreate it.  If the database is heavily updated, it may
well be cheaper to do this than it is to find a way to let it catch
up.  </para> </answer>

<answer><para> In newer versions of &slony1;, there is a new
configuration parameter called <xref
linkend="slon-config-remote-listen-timeout">; you'd alter the config
file to increase the timeout, and try again.  Of course, as mentioned
above, it could be faster to drop the node and recreate it than it is
to require it to catch up across a week's worth of updates...  </para>
</answer>

</qandaentry>

</qandadiv>
<qandadiv id="faqperformance"> <title> &slony1; FAQ: Performance Issues </title>

<qandaentry id="longtxnsareevil">

<question><para> Replication has been slowing down, I'm seeing
<command> FETCH 100 FROM LOG </command> queries running for a long
time, &sllog1;/&sllog2; is growing, and performance is, well,
generally getting steadily worse. </para>
</question>

<answer> <para> There are actually a number of possible causes for
this sort of thing.  There is a question involving similar pathology
where the problem is that <link linkend="pglistenerfull"> &pglistener;
grows because it is not vacuumed. </link>
</para>

<para> Another <quote> proximate cause </quote> for this growth is for
there to be a connection connected to the node that sits <command>
IDLE IN TRANSACTION </command> for a very long time. </para>

<para> That open transaction will have multiple negative effects, all
of which will adversely affect performance:</para>

<itemizedlist>

<listitem><para> Vacuums on all tables, including &pglistener;, will
not clear out dead tuples from before the start of the idle
transaction. </para> </listitem>

<listitem><para> The cleanup thread will be unable to clean out
entries in &sllog1;, &sllog2;, and &slseqlog;, with the result that
these tables will grow, ceaselessly, until the transaction is
closed. </para>
</listitem>
</itemizedlist>
</answer>

<answer> <para> You can monitor for this condition inside the database
only if the &postgres; <filename> postgresql.conf </filename>
parameter <envar>stats_command_string</envar> is set to true.  If that
is set, then you may submit the query <command> select * from
pg_stat_activity where current_query like '%IDLE% in transaction';
</command> which will find relevant activity.  </para> </answer>

<answer> <para> You should also be able to search for <quote> idle in
transaction </quote> in the process table to find processes that are
thus holding on to an ancient transaction.  </para> </answer>

<answer> <para> It is also possible (though rarer) for the problem to
be a transaction that is, for some other reason, being held open for a
very long time.  The <envar> query_start </envar> time in <envar>
pg_stat_activity </envar> may show you some query that has been
running way too long.  </para> </answer>

<answer> <para> There are plans for &postgres; to have a timeout
parameter, <envar> open_idle_transaction_timeout </envar>, which would
cause old transactions to time out after some period of disuse.  Buggy
connection pool logic is a common culprit for this sort of thing.
There are plans for <productname> <link linkend="pgpool"> pgpool
</link> </productname> to provide a better alternative, eventually,
where connections would be shared inside a connection pool implemented
in C.  You may have some more or less buggy connection pool in your
Java or PHP application; if a small set of <emphasis> real </emphasis>
connections are held in <productname>pgpool</productname>, that will
hide from the database the fact that the application imagines that
numerous of them are left idle in transaction for hours at a time.
</para> </answer>

</qandaentry> 

<qandaentry id="faq17">
<question><para>After dropping a node, &sllog1;/&sllog2;
aren't getting purged out anymore.</para></question>

<answer><para> This is a common scenario in versions before 1.0.5, as
the <quote>clean up</quote> that takes place when purging the node
does not include purging out old entries from the &slony1; table,
<xref linkend="table.sl-confirm">, for the recently departed
node.</para>

<para> The node is no longer around to update confirmations of what
syncs have been applied on it, and therefore the cleanup thread that
purges log entries thinks that it can't safely delete entries newer
than the final <xref linkend="table.sl-confirm"> entry, which rather
curtails the ability to purge out old logs.</para>

<para>Diagnosis: Run the following query to see if there are any
<quote>phantom/obsolete/blocking</quote> <xref
linkend="table.sl-confirm"> entries:

<screen>
oxrsbar=# select * from _oxrsbar.sl_confirm where con_origin not in (select no_id from _oxrsbar.sl_node) or con_received not in (select no_id from _oxrsbar.sl_node);
 con_origin | con_received | con_seqno |        con_timestamp                  
------------+--------------+-----------+----------------------------
          4 |          501 |     83999 | 2004-11-09 19:57:08.195969
          1 |            2 |   3345790 | 2004-11-14 10:33:43.850265
          2 |          501 |    102718 | 2004-11-14 10:33:47.702086
        501 |            2 |      6577 | 2004-11-14 10:34:45.717003
          4 |            5 |     83999 | 2004-11-14 21:11:11.111686
          4 |            3 |     83999 | 2004-11-24 16:32:39.020194
(6 rows)
</screen></para>

<para>In version 1.0.5, the <xref linkend="stmtdropnode"> function
purges out entries in <xref linkend="table.sl-confirm"> for the
departing node.  In earlier versions, this needs to be done manually.
Supposing the node number is 3, then the query would be:

<screen>
delete from _namespace.sl_confirm where con_origin = 3 or con_received = 3;
</screen></para>

<para>Alternatively, to go after <quote>all phantoms,</quote> you could use
<screen>
oxrsbar=# delete from _oxrsbar.sl_confirm where con_origin not in (select no_id from _oxrsbar.sl_node) or con_received not in (select no_id from _oxrsbar.sl_node);
DELETE 6
</screen></para>

<para>General <quote>due diligence</quote> dictates starting with a
<command>BEGIN</command>, looking at the contents of
<xref linkend="table.sl-confirm"> before, ensuring that only the expected
records are purged, and then, only after that, confirming the change
with a <command>COMMIT</command>.  If you delete confirm entries for
the wrong node, that could ruin your whole day.</para>

<para>You'll need to run this on each node that remains...</para>

<para>Note that as of 1.0.5, this is no longer an issue at all, as it
purges unneeded entries from <xref linkend="table.sl-confirm"> in two
places:

<itemizedlist>
<listitem><para> At the time a node is dropped</para></listitem>

<listitem><para> At the start of each
<function>cleanupEvent</function> run, which is the event in which old
data is purged from &sllog1;, &sllog2;, and
&slseqlog;</para></listitem> </itemizedlist></para>
</answer>
</qandaentry>

<qandaentry>

<question><para>The <application>slon</application> spent the weekend out of
commission [for some reason], and it's taking a long time to get a
sync through.</para></question>

<answer><para> You might want to take a look at the tables &sllog1;
and &sllog2; and do a summary to see if there are any really enormous
&slony1; transactions in there.  Up until at least 1.0.2, there needs
to be a &lslon; connected to the origin in order for
<command>SYNC</command> events to be generated.</para>

<note><para> As of 1.0.2,
function <function>generate_sync_event()</function> provides an
alternative as backup...</para> </note>

<para>If none are being generated, then all of the updates until the
next one is generated will collect into one rather enormous &slony1;
transaction.</para>

<para>Conclusion: Even if there is not going to be a subscriber
around, you <emphasis>really</emphasis> want to have a
<application>slon</application> running to service the origin
node.</para>

<para>&slony1; 1.1 and later versions provide a stored procedure that
allows <command>SYNC</command> counts to be updated on the origin
based on a <application>cron</application> job even if there is no
<xref linkend="slon"> daemon running.</para> </answer></qandaentry>

<qandaentry> <question> <para> I have submitted a <xref
linkend="stmtmoveset"> / <xref linkend="stmtddlscript"> request, and
it seems to be stuck on one of my nodes.  &slony1; logs aren't
displaying any errors or warnings </para> </question>

<answer> <para> Is it possible that you are running
<application>pg_autovacuum</application>, and it has taken out locks
on some tables in the replication set?  That would somewhat-invisibly
block &slony1; from performing operations that require locking acquisition 
of exclusive locks. </para>

<para> You might check for these sorts of locks using the following
query: <command> select l.*, c.relname from pg_locks l, pg_class c
where c.oid = l.relation ; </command> </para>

<para> A <envar>ShareUpdateExclusiveLock</envar> lock will block the
&slony1; operations that need their own exclusive locks, which are
likely queued up, marked as not being granted. </para> </answer>
</qandaentry>

<qandaentry>

<question><para> I'm noticing in the logs that a &lslon; is frequently
switching in and out of <quote>polling</quote> mode as it is
frequently reporting <quote>LISTEN - switch from polling mode to use
LISTEN</quote> and <quote>UNLISTEN - switch into polling
mode</quote>. </para> </question>

<answer><para> The thresholds for switching between these modes are
controlled by the configuration parameters <xref
linkend="slon-config-sync-interval"> and <xref
linkend="slon-config-sync-interval-timeout">; if the timeout value
(which defaults to 10000, implying 10s) is kept low, that makes it
easy for the &lslon; to decide to return to <quote>listening</quote>
mode.  You may want to increase the value of the timeout
parameter. </para>
</answer>
</qandaentry>

</qandadiv>
<qandadiv id="faqbugs"> <title> &slony1; FAQ: &slony1; Bugs in Elder Versions </title>
<qandaentry>
<question><para>The &lslon; processes servicing my
subscribers are growing to enormous size, challenging system resources
both in terms of swap space as well as moving towards breaking past
the 2GB maximum process size on my system. </para> 

<para> By the way, the data that I am replicating includes some rather
large records.  We have records that are tens of megabytes in size.
Perhaps that is somehow relevant? </para> </question>

<answer> <para> Yes, those very large records are at the root of the
problem.  The problem is that &lslon; normally draws in
about 100 records at a time when a subscriber is processing the query
which loads data from the provider.  Thus, if the average record size
is 10MB, this will draw in 1000MB of data which is then transformed
into <command>INSERT</command> or <command>UPDATE</command>
statements, in the &lslon; process' memory.</para>

<para> That obviously leads to &lslon; growing to a
fairly tremendous size. </para>

<para> The number of records that are fetched is controlled by the
value <envar> SLON_DATA_FETCH_SIZE </envar>, which is defined in the
file <filename>src/slon/slon.h</filename>.  The relevant extract of
this is shown below. </para>
 
<programlisting>
#ifdef	SLON_CHECK_CMDTUPLES
#define SLON_COMMANDS_PER_LINE		1
#define SLON_DATA_FETCH_SIZE		100
#define SLON_WORKLINES_PER_HELPER	(SLON_DATA_FETCH_SIZE * 4)
#else
#define SLON_COMMANDS_PER_LINE		10
#define SLON_DATA_FETCH_SIZE		10
#define SLON_WORKLINES_PER_HELPER	(SLON_DATA_FETCH_SIZE * 50)
#endif
</programlisting>

<para> If you are experiencing this problem, you might modify the
definition of <envar> SLON_DATA_FETCH_SIZE </envar>, perhaps reducing
by a factor of 10, and recompile &lslon;.  There are two
definitions as <envar> SLON_CHECK_CMDTUPLES</envar> allows doing some
extra monitoring to ensure that subscribers have not fallen out of
SYNC with the provider.  By default, this option is turned off, so the
default modification to make is to change the second definition of
<envar> SLON_DATA_FETCH_SIZE </envar> from 10 to 1. </para> </answer>

<answer><para> In version 1.2, configuration values <xref
linkend="slon-config-max-rowsize"> and <xref
linkend="slon-config-max-largemem"> are associated with a new
algorithm that changes the logic as follows.  Rather than fetching 100
rows worth of data at a time:</para>

<itemizedlist>

<listitem><para> The <command>fetch from LOG</command> query will draw
in 500 rows at a time where the size of the attributes does not exceed
<xref linkend="slon-config-max-rowsize">.  With default values, this
restricts this aspect of memory consumption to about 8MB.  </para>
</listitem>

<listitem><para> Tuples with larger attributes are loaded until
aggregate size exceeds the parameter <xref
linkend="slon-config-max-largemem">.  By default, this restricts
consumption of this sort to about 5MB.  This value is not a strict
upper bound; if you have a tuple with attributes 50MB in size, it
forcibly <emphasis>must</emphasis> be loaded into memory.  There is no
way around that.  But &lslon; at least won't be trying
to load in 100 such records at a time, chewing up 10GB of memory by
the time it's done.  </para> </listitem>
</itemizedlist>

<para> This should alleviate problems people have been experiencing
when they sporadically have series' of very large tuples. </para>
</answer>

<answer> <para> In version 2.2, this is eliminated in a different way,
as the log data is transported using COPY rather than via querying
from a cursor, which may be expected to be a
<emphasis>massive</emphasis> improvement to the "flow control"
here. </para>

</qandaentry>

<qandaentry id="faqunicode"> <question> <para> I am trying to replicate
<envar>UNICODE</envar> data from &postgres; 8.0 to &postgres; 8.1, and
am experiencing problems. </para>
</question>

<answer> <para> &postgres; 8.1 is quite a lot more strict about what
UTF-8 mappings of Unicode characters it accepts as compared to version
8.0.</para>

<para> If you intend to use &slony1; to update an older database to 8.1, and
might have invalid UTF-8 values, you may be for an unpleasant
surprise.</para>

<para> Let us suppose we have a database running 8.0, encoding in UTF-8.
That database will accept the sequence <command>'\060\242'</command> as UTF-8 compliant,
even though it is really not. </para>

<para> If you replicate into a &postgres; 8.1 instance, it will complain
about this, either at subscribe time, where &slony1; will complain
about detecting an invalid Unicode sequence during the COPY of the
data, which will prevent the subscription from proceeding, or, upon
adding data, later, where this will hang up replication fairly much
irretrievably.  (You could hack on the contents of sl_log_1, but
that quickly gets <emphasis>really</emphasis> unattractive...)</para>

<para>There have been discussions as to what might be done about this.  No
compelling strategy has yet emerged, as all are unattractive. </para>

<para>If you are using Unicode with &postgres; 8.0, you run a
considerable risk of corrupting data.  </para>

<para> If you use replication for a one-time conversion, there is a risk of
failure due to the issues mentioned earlier; if that happens, it
appears likely that the best answer is to fix the data on the 8.0
system, and retry. </para>

<para> In view of the risks, running replication between versions seems to be
something you should not keep running any longer than is necessary to
migrate to 8.1. </para>

<para> For more details, see the <ulink url=
"http://archives.postgresql.org/pgsql-hackers/2005-12/msg00181.php">
discussion on postgresql-hackers mailing list. </ulink>.  </para>
</answer>
</qandaentry>

<qandaentry>
<question> <para> I am running &slony1; 1.1 and have a 4+ node setup
where there are two subscription sets, 1 and 2, that do not share any
nodes.  I am discovering that confirmations for set 1 never get to the
nodes subscribing to set 2, and that confirmations for set 2 never get
to nodes subscribing to set 1.  As a result, &sllog1;/&sllog2; grow
and grow, and are never purged.  This was reported as
&slony1; <ulink
url="http://gborg.postgresql.org/project/slony1/bugs/bugupdate.php?1485">
bug 1485 </ulink>.
</para>
</question>

<answer><para> Apparently the code for
<function>RebuildListenEntries()</function> does not suffice for this
case.</para>

<para> <function> RebuildListenEntries()</function> will be replaced
in &slony1; version 1.2 with an algorithm that covers this case. </para>

<para> In the interim, you'll want to manually add some <xref
linkend="table.sl-listen"> entries using <xref
linkend="stmtstorelisten"> or <function>storeListen()</function>,
based on the (apparently not as obsolete as we thought) principles
described in <xref linkend="listenpaths">.

</para></answer>
</qandaentry>

<qandaentry>
<question> <para> I am finding some multibyte columns (Unicode, Big5)
are being truncated a bit, clipping off the last character.  Why?
</para> </question>

<answer> <para> This was a bug present until a little after &slony1;
version 1.1.0; the way in which columns were being captured by the
<function>logtrigger()</function> function could clip off the last
byte of a column represented in a multibyte format.  Check to see that
your version of <filename>src/backend/slony1_funcs.c</filename> is
1.34 or better; the patch was introduced in CVS version 1.34 of that
file.  </para> </answer>
</qandaentry>

<qandaentry id="sequenceset"><question><para> <ulink url=
"http://gborg.postgresql.org/project/slony1/bugs/bugupdate.php?1226">
Bug #1226 </ulink> indicates an error condition that can come up if
you have a replication set that consists solely of sequences. </para>
</question>

<answer> <para> The  short answer is that having a replication set
consisting only of sequences is not a 
best practice. </para>
</answer>

<answer>
<para> The problem with a sequence-only set comes up only if you have
a case where the only subscriptions that are active for a particular
subscriber to a particular provider are for
<quote>sequence-only</quote> sets.  If a node gets into that state,
replication will fail, as the query that looks for data from
&sllog1;/&sllog2; has no tables to find, and the query will be
malformed, and fail.  If a replication set <emphasis>with</emphasis>
tables is added back to the mix, everything will work out fine; it
just <emphasis>seems</emphasis> scary.
</para>

<para> This problem should be resolved some time after &slony1;
1.1.0.</para>
</answer>
</qandaentry>

<qandaentry>
<question><para>I need to drop a table from a replication set</para></question>
<answer><para>
This can be accomplished several ways, not all equally desirable ;-).

<itemizedlist>

<listitem><para> You could drop the whole replication set, and
recreate it with just the tables that you need.  Alas, that means
recopying a whole lot of data, and kills the usability of the cluster
on the rest of the set while that's happening.</para></listitem>

<listitem><para> If you are running 1.0.5 or later, there is the
command SET DROP TABLE, which will "do the trick."</para></listitem>

<listitem><para> If you are still using 1.0.1 or 1.0.2, the
<emphasis>essential functionality of <xref linkend="stmtsetdroptable">
involves the functionality in <function>droptable_int()</function>.
You can fiddle this by hand by finding the table ID for the table you
want to get rid of, which you can find in <xref
linkend="table.sl-table">, and then run the following three queries,
on each host:</emphasis>

<programlisting>
  select _slonyschema.alterTableRestore(40);
  select _slonyschema.tableDropKey(40);
  delete from _slonyschema.sl_table where tab_id = 40;
</programlisting></para>

<para>The schema will obviously depend on how you defined the &slony1;
cluster.  The table ID, in this case, 40, will need to change to the
ID of the table you want to have go away.</para>

<para> You'll have to run these three queries on all of the nodes,
preferably firstly on the origin node, so that the dropping of this
propagates properly.  Implementing this via a <xref linkend="slonik">
statement with a new &slony1; event would do that.  Submitting the
three queries using <xref linkend="stmtddlscript"> could do that.
Also possible would be to connect to each database and submit the
queries by hand.</para></listitem> </itemizedlist></para>
</answer>
</qandaentry>

<qandaentry>
<question><para>I need to drop a sequence from a replication set</para></question>

<answer><para></para><para>If you are running 1.0.5 or later, there is
a <xref linkend="stmtsetdropsequence"> command in Slonik to allow you
to do this, parallelling <xref linkend="stmtsetdroptable">.</para>

<para>If you are running 1.0.2 or earlier, the process is a bit more manual.</para>

<para>Supposing I want to get rid of the two sequences listed below,
<envar>whois_cachemgmt_seq</envar> and
<envar>epp_whoi_cach_seq_</envar>, we start by needing the
<envar>seq_id</envar> values.

<screen>
oxrsorg=# select * from _oxrsorg.sl_sequence  where seq_id in (93,59);
 seq_id | seq_reloid | seq_set |       seq_comment				 
--------+------------+---------+-------------------------------------
     93 |  107451516 |       1 | Sequence public.whois_cachemgmt_seq
     59 |  107451860 |       1 | Sequence public.epp_whoi_cach_seq_
(2 rows)
</screen></para>

<para>The data that needs to be deleted to stop Slony from continuing to
replicate these are thus:

<programlisting>
delete from _oxrsorg.sl_seqlog where seql_seqid in (93, 59);
delete from _oxrsorg.sl_sequence where seq_id in (93,59);
</programlisting></para>

<para>Those two queries could be submitted to all of the nodes via
&funddlscript; / <xref
linkend="stmtddlscript">, thus eliminating the sequence everywhere
<quote>at once.</quote> Or they may be applied by hand to each of the
nodes.</para>

<para>Similarly to <xref linkend="stmtsetdroptable">, this is
implemented &slony1; version 1.0.5 as <xref
linkend="stmtsetdropsequence">.</para></answer></qandaentry>

<qandaentry>
<question><para> I set up my cluster using pgAdminIII, with cluster
name <quote>MY-CLUSTER</quote>.  Time has passed, and I tried using
Slonik to make a configuration change, and this is failing with the
following error message:</para>

<programlisting>
ERROR: syntax error at or near -
</programlisting>
</question>

<answer><para> The problem here is that &slony1; expects cluster names
to be valid <ulink url=
"http://www.postgresql.org/docs/8.3/static/sql-syntax-lexical.html">
SQL Identifiers</ulink>, and &lslonik; enforces this.  Unfortunately,
<application>pgAdminIII</application> did not do so, and allowed using
a cluster name that now causes <emphasis>a problem.</emphasis> </para> </answer>

<answer> <para> If you have gotten into this spot, it's a problem that
we mayn't be help resolve, terribly much.  </para>

<para> It's <emphasis>conceivably possible</emphasis> that running the
SQL command <command>alter namespace "_My-Bad-Clustername" rename to
"_BetterClusterName";</command> against each database may work.  That
shouldn't particularly <emphasis>damage</emphasis> things!</para>

<para> On the other hand, when the problem has been experienced, users
have found they needed to drop replication and rebuild the
cluster.</para> </answer>

<answer><para> A change in version 2.0.2 is that a function runs as
part of loading functions into the database which checks the validity
of the cluster name.  If you try to use an invalid cluster name,
loading the functions will fail, with a suitable error message, which
should prevent things from going wrong even if you're using tools
other than &lslonik; to manage setting up the cluster. </para></answer>
</qandaentry>

</qandadiv>

<qandadiv id="faqobsolete"> <title> &slony1; FAQ: Anachronisms -
Hopefully Obsolete </title>

<qandaentry>
<question><para> &lslon; does not restart after
crash</para>

<para> After an immediate stop of &postgres; (simulation of system
crash) in &pglistener; a tuple with <command>
relname='_${cluster_name}_Restart'</command> exists. slon doesn't
start because it thinks another process is serving the cluster on this
node.  What can I do? The tuples can't be dropped from this
relation.</para>

<para> The logs claim that <blockquote><para>Another slon daemon is
serving this node already</para></blockquote></para></question>

<answer><para> The problem is that the system table &pglistener;, used
by &postgres; to manage event notifications, contains some entries
that are pointing to backends that no longer exist.  The new <xref
linkend="slon"> instance connects to the database, and is convinced,
by the presence of these entries, that an old
<application>slon</application> is still servicing this &slony1;
node.</para>

<para> The <quote>trash</quote> in that table needs to be thrown
away.</para>

<para>It's handy to keep a slonik script similar to the following to
run in such cases:

<programlisting>
twcsds004[/opt/twcsds004/OXRS/slony-scripts]$ cat restart_org.slonik 
cluster name = oxrsorg ;
node 1 admin conninfo = 'host=32.85.68.220 dbname=oxrsorg user=postgres port=5532';
node 2 admin conninfo = 'host=32.85.68.216 dbname=oxrsorg user=postgres port=5532';
node 3 admin conninfo = 'host=32.85.68.244 dbname=oxrsorg user=postgres port=5532';
node 4 admin conninfo = 'host=10.28.103.132 dbname=oxrsorg user=postgres port=5532';
restart node 1;
restart node 2;
restart node 3;
restart node 4;
</programlisting></para>

<para> <xref linkend="stmtrestartnode"> cleans up dead notifications
so that you can restart the node.</para>

<para>As of version 1.0.5, the startup process of slon looks for this
condition, and automatically cleans it up.</para>

<para> As of version 8.1 of &postgres;, the functions that manipulate
&pglistener; do not support this usage, so for &slony1; versions after
1.1.2 (<emphasis>e.g. - </emphasis> 1.1.5), this
<quote>interlock</quote> behaviour is handled via a new table, and the
issue should be transparently <quote>gone.</quote> </para>

</answer></qandaentry>

<qandaentry><question><para> I tried the following query which did not work:</para> 

<programlisting>
sdb=# explain select query_start, current_query from pg_locks join
pg_stat_activity on pid = procpid where granted = true and transaction
in (select transaction from pg_locks where granted = false); 

ERROR: could not find hash function for hash operator 716373
</programlisting>

<para> It appears the &slony1; <function>xxid</function> functions are
claiming to be capable of hashing, but cannot actually do so.</para>


<para> What's up? </para>

</question>

<answer><para> &slony1; defined an XXID data type and operators on
that type in order to allow manipulation of transaction IDs that are
used to group together updates that are associated with the same
transaction.</para>

<para> Operators were not available for &postgres; 7.3 and earlier
versions; in order to support version 7.3, custom functions had to be
added.  The <function>=</function> operator was marked as supporting
hashing, but for that to work properly, the join operator must appear
in a hash index operator class.  That was not defined, and as a
result, queries (like the one above) that decide to use hash joins
will fail. </para> </answer>

<answer> <para> This has <emphasis> not </emphasis> been considered a
<quote>release-critical</quote> bug, as &slony1; does not internally
generate queries likely to use hash joins.  This problem shouldn't
injure &slony1;'s ability to continue replicating. </para> </answer>

<answer> <para> Future releases of &slony1; (<emphasis>e.g.</emphasis>
1.0.6, 1.1) will omit the <command>HASHES</command> indicator, so that
</para> </answer>

<answer> <para> Supposing you wish to repair an existing instance, so
that your own queries will not run afoul of this problem, you may do
so as follows: </para>

<programlisting>
/* cbbrowne@[local]/dba2 slony_test1=*/ \x     
Expanded display is on.
/* cbbrowne@[local]/dba2 slony_test1=*/ select * from pg_operator where oprname = '=' 
and oprnamespace = (select oid from pg_namespace where nspname = 'public');
-[ RECORD 1 ]+-------------
oprname      | =
oprnamespace | 2200
oprowner     | 1
oprkind      | b
oprcanhash   | t
oprleft      | 82122344
oprright     | 82122344
oprresult    | 16
oprcom       | 82122365
oprnegate    | 82122363
oprlsortop   | 82122362
oprrsortop   | 82122362
oprltcmpop   | 82122362
oprgtcmpop   | 82122360
oprcode      | "_T1".xxideq
oprrest      | eqsel
oprjoin      | eqjoinsel

/* cbbrowne@[local]/dba2 slony_test1=*/ update pg_operator set oprcanhash = 'f' where 
oprname = '=' and oprnamespace = 2200 ;
UPDATE 1
</programlisting>
</answer>

</qandaentry>
<qandaentry> <question><para> I can do a <command>pg_dump</command>
and load the data back in much faster than the <command>SUBSCRIBE
SET</command> runs.  Why is that?  </para></question>

<answer><para> &slony1; depends on there being an already existant
index on the primary key, and leaves all indexes alone whilst using
the &postgres; <command>COPY</command> command to load the data.
Further hurting performance, the <command>COPY SET</command> event (an
event that the subscription process generates) starts by deleting the
contents of tables, which leaves the table full of dead tuples.
</para>

<para> When you use <command>pg_dump</command> to dump the contents of
a database, and then load that, creation of indexes is deferred until
the very end.  It is <emphasis>much</emphasis> more efficient to
create indexes against the entire table, at the end, than it is to
build up the index incrementally as each row is added to the
table.</para></answer>

<answer><para> If you can drop unnecessary indices while the
<command>COPY</command> takes place, that will improve performance
quite a bit.  If you can <command>TRUNCATE</command> tables that
contain data that is about to be eliminated, that will improve
performance <emphasis>a lot.</emphasis> </para></answer>

<answer><para> &slony1; version 1.1.5 and later versions should handle
this automatically; it <quote>thumps</quote> on the indexes in the
&postgres; catalog to hide them, in much the same way triggers are
hidden, and then <quote>fixes</quote> the index pointers and reindexes
the table. </para> </answer>
</qandaentry>

<qandaentry id="dupkey">
<question><para>Replication Fails - Unique Constraint Violation</para>

<para>Replication has been running for a while, successfully, when a
node encounters a <quote>glitch,</quote> and replication logs are filled with
repetitions of the following:

<screen>
DEBUG2 remoteWorkerThread_1: syncing set 2 with 5 table(s) from provider 1
DEBUG2 remoteWorkerThread_1: syncing set 1 with 41 table(s) from provider 1
DEBUG2 remoteWorkerThread_1: syncing set 5 with 1 table(s) from provider 1
DEBUG2 remoteWorkerThread_1: syncing set 3 with 1 table(s) from provider 1
DEBUG2 remoteHelperThread_1_1: 0.135 seconds delay for first row
DEBUG2 remoteHelperThread_1_1: 0.343 seconds until close cursor
ERROR  remoteWorkerThread_1: "insert into "_oxrsapp".sl_log_1          (log_origin, log_xid, log_tableid,                log_actionseq, log_cmdtype,		log_cmddata) values	  ('1', '919151224', '34', '35090538', 'D', '_rserv_ts=''9275244''');
delete from only public.epp_domain_host where _rserv_ts='9275244';insert into "_oxrsapp".sl_log_1	  (log_origin, log_xid, log_tableid,		log_actionseq, log_cmdtype,		log_cmddata) values	  ('1', '919151224', '34', '35090539', 'D', '_rserv_ts=''9275245''');
delete from only public.epp_domain_host where _rserv_ts='9275245';insert into "_oxrsapp".sl_log_1	  (log_origin, log_xid, log_tableid,		log_actionseq, log_cmdtype,		log_cmddata) values	  ('1', '919151224', '26', '35090540', 'D', '_rserv_ts=''24240590''');
delete from only public.epp_domain_contact where _rserv_ts='24240590';insert into "_oxrsapp".sl_log_1	  (log_origin, log_xid, log_tableid,		log_actionseq, log_cmdtype,		log_cmddata) values	  ('1', '919151224', '26', '35090541', 'D', '_rserv_ts=''24240591''');
delete from only public.epp_domain_contact where _rserv_ts='24240591';insert into "_oxrsapp".sl_log_1	  (log_origin, log_xid, log_tableid,		log_actionseq, log_cmdtype,		log_cmddata) values	  ('1', '919151224', '26', '35090542', 'D', '_rserv_ts=''24240589''');
delete from only public.epp_domain_contact where _rserv_ts='24240589';insert into "_oxrsapp".sl_log_1	  (log_origin, log_xid, log_tableid,		log_actionseq, log_cmdtype,		log_cmddata) values	  ('1', '919151224', '11', '35090543', 'D', '_rserv_ts=''36968002''');
delete from only public.epp_domain_status where _rserv_ts='36968002';insert into "_oxrsapp".sl_log_1	  (log_origin, log_xid, log_tableid,		log_actionseq, log_cmdtype,		log_cmddata) values	  ('1', '919151224', '11', '35090544', 'D', '_rserv_ts=''36968003''');
delete from only public.epp_domain_status where _rserv_ts='36968003';insert into "_oxrsapp".sl_log_1	  (log_origin, log_xid, log_tableid,		log_actionseq, log_cmdtype,		log_cmddata) values	  ('1', '919151224', '24', '35090549', 'I', '(contact_id,status,reason,_rserv_ts) values (''6972897'',''64'','''',''31044208'')');
insert into public.contact_status (contact_id,status,reason,_rserv_ts) values ('6972897','64','','31044208');insert into "_oxrsapp".sl_log_1	  (log_origin, log_xid, log_tableid,		log_actionseq, log_cmdtype,		log_cmddata) values	  ('1', '919151224', '24', '35090550', 'D', '_rserv_ts=''18139332''');
delete from only public.contact_status where _rserv_ts='18139332';insert into "_oxrsapp".sl_log_1	  (log_origin, log_xid, log_tableid,		log_actionseq, log_cmdtype,		log_cmddata) values	  ('1', '919151224', '24', '35090551', 'D', '_rserv_ts=''18139333''');
delete from only public.contact_status where _rserv_ts='18139333';" ERROR:  duplicate key violates unique constraint "contact_status_pkey"
 - qualification was: 
ERROR  remoteWorkerThread_1: SYNC aborted
</screen></para>

<para>The transaction rolls back, and
&slony1; tries again, and again, and again.
The problem is with one of the <emphasis>last</emphasis> SQL
statements, the one with <command>log_cmdtype = 'I'</command>.  That
isn't quite obvious; what takes place is that
&slony1; groups 10 update queries together
to diminish the number of network round trips.</para></question>

<answer><para> A <emphasis>certain</emphasis> cause for this has been
difficult to arrive at.</para>

<para>By the time we notice that there is a problem, the seemingly
missed delete transaction has been cleaned out of &sllog1;, so there
appears to be no recovery possible.  What has seemed necessary, at
this point, is to drop the replication set (or even the node), and
restart replication from scratch on that node.</para>

<para>In &slony1; 1.0.5, the handling of purges of &sllog1; became
more conservative, refusing to purge entries that haven't been
successfully synced for at least 10 minutes on all nodes.  It was not
certain that that would prevent the <quote>glitch</quote> from taking
place, but it seemed plausible that it might leave enough &sllog1;
data to be able to do something about recovering from the condition or
at least diagnosing it more exactly.  And perhaps the problem was that
&sllog1; was being purged too aggressively, and this would resolve the
issue completely.</para>

<para> It is a shame to have to reconstruct a large replication node
for this; if you discover that this problem recurs, it may be an idea
to break replication down into multiple sets in order to diminish the
work involved in restarting replication.  If only one set has broken,
you may only need to unsubscribe/drop and resubscribe the one set.
</para>

<para> In one case we found two lines in the SQL error message in the
log file that contained <emphasis> identical </emphasis> insertions
into &sllog1;.  This <emphasis> ought </emphasis> to be impossible as
is a primary key on &sllog1;.  The latest (somewhat) punctured theory
that comes from <emphasis>that</emphasis> was that perhaps this PK
index has been corrupted (representing a &postgres; bug), and that
perhaps the problem might be alleviated by running the query:</para>

<programlisting>
# reindex table _slonyschema.sl_log_1;
</programlisting>

<para> On at least one occasion, this has resolved the problem, so it
is worth trying this.</para>
</answer>

<answer> <para> This problem has been found to represent a &postgres;
bug as opposed to one in &slony1;.  Version 7.4.8 was released with
two resolutions to race conditions that should resolve the issue.
Thus, if you are running a version of &postgres; earlier than 7.4.8,
you should consider upgrading to resolve this.
</para>
</answer>
</qandaentry>
<qandaentry> <question><para>I started doing a backup using
<application>pg_dump</application>, and suddenly Slony
stops</para></question>

<answer><para>Ouch.  What happens here is a conflict between:
<itemizedlist>

<listitem><para> <application>pg_dump</application>, which has taken
out an <command>AccessShareLock</command> on all of the tables in the
database, including the &slony1; ones, and</para></listitem>

<listitem><para> A &slony1; sync event, which wants to grab a
<command>AccessExclusiveLock</command> on the table <xref
linkend="table.sl-event">.</para></listitem> </itemizedlist></para>

<para>The initial query that will be blocked is thus:

<screen>
select "_slonyschema".createEvent('_slonyschema, 'SYNC', NULL);	  
</screen></para>

<para>(You can see this in <envar>pg_stat_activity</envar>, if you
have query display turned on in
<filename>postgresql.conf</filename>)</para>

<para>The actual query combination that is causing the lock is from
the function <function>Slony_I_ClusterStatus()</function>, found in
<filename>slony1_funcs.c</filename>, and is localized in the code that
does:

<programlisting>
  LOCK TABLE %s.sl_event;
  INSERT INTO %s.sl_event (...stuff...)
  SELECT currval('%s.sl_event_seq');
</programlisting></para>

<para>The <command>LOCK</command> statement will sit there and wait
until <command>pg_dump</command> (or whatever else has pretty much any
kind of access lock on <xref linkend="table.sl-event">)
completes.</para>

<para>Every subsequent query submitted that touches
<xref linkend="table.sl-event"> will block behind the
<function>createEvent</function> call.</para>

<para>There are a number of possible answers to this:
<itemizedlist>

<listitem><para> Have <application>pg_dump</application> specify the
schema dumped using <option>--schema=whatever</option>, and don't try
dumping the cluster's schema.</para></listitem>

<listitem><para> It would be nice to add an
<option>--exclude-schema</option> option to
<application>pg_dump</application> to exclude the &slony1; cluster
schema.  Maybe in 8.2...</para></listitem>

<listitem><para>Note that 1.0.5 uses a more precise lock that is less
exclusive that alleviates this problem.</para></listitem>
</itemizedlist></para>
</answer></qandaentry>

</qandadiv>


<qandadiv>
<qandaentry> <question> <para> I was trying to request <xref
linkend="stmtddlscript"> or <xref linkend="stmtmoveset">, and found
messages as follows on one of the subscribers:</para>

<screen>
NOTICE: Slony-I: multiple instances of trigger defrazzle on table frobozz
NOTICE: Slony-I: multiple instances of trigger derez on table tron
ERROR: Slony-I: Unable to disable triggers
</screen>
</question>

<answer> <para> The trouble would seem to be that you have added
triggers on tables whose names conflict with triggers that were hidden
by &slony1;. </para>

<para> &slony1; hides triggers (save for those <quote>unhidden</quote>
via <xref linkend="stmtstoretrigger">) by repointing them to the
primary key of the table.  In the case of foreign key triggers, or
other triggers used to do data validation, it should be quite
unnecessary to run them on a subscriber, as equivalent triggers should
have been invoked on the origin node.  In contrast, triggers that do
some form of <quote>cache invalidation</quote> are ones you might want
to have run on a subscriber.</para>

<para> The <emphasis>Right Way</emphasis> to handle such triggers is
normally to use <xref linkend="stmtstoretrigger">, which tells
&slony1; that a trigger should not get deactivated. </para> </answer>

<answer> <para> But some intrepid DBA might take matters into their
own hands and install a trigger by hand on a subscriber, and the above
condition generally has that as the cause.  What to do?  What to do?
</para>

<para> The answer is normally fairly simple: Drop out the
<quote>extra</quote> trigger on the subscriber before the event that
tries to restore them runs.  Ideally, if the DBA is particularly
intrepid, and aware of this issue, that should take place
<emphasis>before</emphasis> there is ever a chance for the error
message to appear.  </para>

<para> If the DBA is not that intrepid, the answer is to connect to
the offending node and drop the <quote>visible</quote> version of the
trigger using the <acronym>SQL</acronym> <command>DROP
TRIGGER</command> command.  That should allow the event to proceed.
If the event was <xref linkend="stmtddlscript">, then the
<quote>not-so-intrepid</quote> DBA may need to add the trigger back,
by hand, or, if they are wise, they should consider activating it
using <xref linkend="stmtstoretrigger">.</para>
</answer>
</qandaentry>

<qandaentry id="neededexecddl">

<question> <para> Behaviour - all the subscriber nodes start to fall
behind the origin, and all the logs on the subscriber nodes have the
following error message repeating in them (when I encountered it,
there was a nice long SQL statement above each entry):</para>

<screen>
ERROR remoteWorkerThread_1: helper 1 finished with error
ERROR remoteWorkerThread_1: SYNC aborted
</screen>
</question>

<answer> <para> Cause: you have likely issued <command>alter
table</command> statements directly on the databases instead of using
the slonik <xref linkend="stmtddlscript"> command.</para>

<para>The solution is to rebuild the trigger on the affected table and
fix the entries in &sllog1;/&sllog2; by hand.</para>

<itemizedlist>

<listitem><para> You'll need to identify from either the slon logs, or
the &postgres; database logs exactly which statement it is that is
causing the error.</para></listitem>

<listitem><para> You need to fix the Slony-defined triggers on the
table in question.  This is done with the following procedure.</para>

<screen>
BEGIN;
LOCK TABLE table_name;
SELECT _oxrsorg.altertablerestore(tab_id);--tab_id is _slony_schema.sl_table.tab_id
SELECT _oxrsorg.altertableforreplication(tab_id);--tab_id is _slony_schema.sl_table.tab_id
COMMIT;
</screen>

<para>You then need to find the rows in &sllog1;/&sllog2; that have
bad entries and fix them.  You may want to take down the slon daemons
for all nodes except the master; that way, if you make a mistake, it
won't immediately propagate through to the subscribers.</para>

<para> Here is an example:</para>

<screen>
BEGIN;

LOCK TABLE customer_account;

SELECT _app1.altertablerestore(31);
SELECT _app1.altertableforreplication(31);
COMMIT;

BEGIN;
LOCK TABLE txn_log;

SELECT _app1.altertablerestore(41);
SELECT _app1.altertableforreplication(41);

COMMIT;

--fixing customer_account, which had an attempt to insert a "" into a timestamp with timezone.
BEGIN;

update _app1.sl_log_1 SET log_cmddata = 'balance=''60684.00'' where pkey=''49''' where log_actionseq = '67796036';
update _app1.sl_log_1 SET log_cmddata = 'balance=''60690.00'' where pkey=''49''' where log_actionseq = '67796194';
update _app1.sl_log_1 SET log_cmddata = 'balance=''60684.00'' where pkey=''49''' where log_actionseq = '67795881';
update _app1.sl_log_1 SET log_cmddata = 'balance=''1852.00'' where pkey=''57''' where log_actionseq = '67796403';
update _app1.sl_log_1 SET log_cmddata = 'balance=''87906.00'' where pkey=''8''' where log_actionseq = '68352967';
update _app1.sl_log_1 SET log_cmddata = 'balance=''125180.00'' where pkey=''60''' where log_actionseq = '68386951';
update _app1.sl_log_1 SET log_cmddata = 'balance=''125198.00'' where pkey=''60''' where log_actionseq = '68387055';
update _app1.sl_log_1 SET log_cmddata = 'balance=''125174.00'' where pkey=''60''' where log_actionseq = '68386682';
update _app1.sl_log_1 SET log_cmddata = 'balance=''125186.00'' where pkey=''60''' where log_actionseq = '68386992';
update _app1.sl_log_1 SET log_cmddata = 'balance=''125192.00'' where pkey=''60''' where log_actionseq = '68387029';

</screen>
</listitem>

</itemizedlist>
</answer>

</qandaentry>

<qandaentry> 

<question><para> Node #1 was dropped via <xref
linkend="stmtdropnode">, and the &lslon; one of the
other nodes is repeatedly failing with the error message:</para>

<screen>
ERROR  remoteWorkerThread_3: "begin transaction; set transaction isolation level
 serializable; lock table "_mailermailer".sl_config_lock; select "_mailermailer"
.storeListen_int(2, 1, 3); notify "_mailermailer_Event"; notify "_mailermailer_C
onfirm"; insert into "_mailermailer".sl_event     (ev_origin, ev_seqno, ev_times
tamp,      ev_minxid, ev_maxxid, ev_xip, ev_type , ev_data1, ev_data2, ev_data3
   ) values ('3', '2215', '2005-02-18 10:30:42.529048', '3286814', '3286815', ''
, 'STORE_LISTEN', '2', '1', '3'); insert into "_mailermailer".sl_confirm
(con_origin, con_received, con_seqno, con_timestamp)    values (3, 2, '2215', CU
RRENT_TIMESTAMP); commit transaction;" PGRES_FATAL_ERROR ERROR:  insert or updat
e on table "sl_listen" violates foreign key constraint "sl_listen-sl_path-ref"
DETAIL:  Key (li_provider,li_receiver)=(1,3) is not present in table "sl_path".
DEBUG1 syncThread: thread done
</screen>

<para> Evidently, a <xref linkend="stmtstorelisten"> request hadn't
propagated yet before node 1 was dropped.  </para></question>

<answer id="eventsurgery"><para> This points to a case where you'll
need to do <quote>event surgery</quote> on one or more of the nodes.
A <command>STORE_LISTEN</command> event remains outstanding that wants
to add a listen path that <emphasis>cannot</emphasis> be created
because node 1 and all paths pointing to node 1 have gone away.</para>

<para> Let's assume, for exposition purposes, that the remaining nodes
are #2 and #3, and that the above error is being reported on node
#3.</para>

<para> That implies that the event is stored on node #2, as it
wouldn't be on node #3 if it had not already been processed
successfully.  The easiest way to cope with this situation is to
delete the offending <xref linkend="table.sl-event"> entry on node #2.
You'll connect to node #2's database, and search for the
<command>STORE_LISTEN</command> event:</para>

<para> <command> select * from sl_event where ev_type =
'STORE_LISTEN';</command></para>

<para> There may be several entries, only some of which need to be
purged. </para>

<screen> 
-# begin;  -- Don't straight delete them; open a transaction so you can respond to OOPS
BEGIN;
-# delete from sl_event where ev_type = 'STORE_LISTEN' and
-#  (ev_data1 = '1' or ev_data2 = '1' or ev_data3 = '1');
DELETE 3
-# -- Seems OK...
-# commit;
COMMIT
</screen>

<para> The next time the <application>slon</application> for node 3
starts up, it will no longer find the <quote>offensive</quote>
<command>STORE_LISTEN</command> events, and replication can continue.
(You may then run into some other problem where an old stored event is
referring to no-longer-existant configuration...) </para></answer>

</qandaentry>

<qandaentry>

<question> <para> I have a database where we have been encountering
the following error message in our application: </para>

<screen> permission denied for sequence sl_action_seq </screen>

<para> When we traced it back, it was due to the application calling
<function> lastval() </function> to capture the most recent sequence
update, which happened to catch the last update to a &slony1; internal
sequence. </para>

</question> 

<answer>
<para> &slony1; uses sequences to provide primary key values for log
entries, and therefore this kind of behaviour may (perhaps
regrettably!) be expected.  </para>

<para> Calling <function>lastval()</function>, to
<quote>anonymously</quote> get <quote>the most recently updated
sequence value</quote>, rather than using
<function>currval('sequence_name')</function> is an unsafe thing to do
in general, as anything you might add in that uses DBMS features for
logging, archiving, or replication can throw in an extra sequence
update that you weren't expecting.  </para>

<para> In general, use of <function>lastval()</function> doesn't seem
terribly safe; using it when &slony1; (or any similar trigger-based
replication system such as <application>Londiste</application> or
<application>Bucardo</application>) can lead to capturing unexpected
sequence updates. </para>

</answer> 

</qandaentry>

<qandaentry id="missingheaders">
<question><para> I tried building &slony1; 1.1 and got the following
error message:
<screen>
configure: error: Headers for libpqserver are not found in the includeserverdir.
   This is the path to postgres.h. Please specify the includeserverdir with
   --with-pgincludeserverdir=&lt;dir&gt;
</screen>
</para></question>

<answer><para> You are almost certainly running version &postgres; 7.4
or earlier, where server headers are not installed by default if you
just do a <command>make install</command> of &postgres;.</para>

<para> You need to install server headers when you install &postgres;
via the command <command>make install-all-headers</command>.

</para> </answer> </qandaentry>

<qandaentry id="pg81funs">

<question> <para> I'm trying to upgrade to a newer version of &slony1;
and am running into a problem with <xref
linkend="stmtupdatefunctions">.  When I run <xref
linkend="stmtupdatefunctions">, my
<application>postmaster</application> falls over with a Signal 11.
There aren't any seeming errors in the log files, aside from the
&postgres; logs indicating that, yes indeed, the postmaster fell
over.</para>

<para> I connected a debugger to the core file, and it indicates that
it was trying to commit a transaction at the time of the
failure. </para>

<para> By the way I'm on &postgres; 8.1.[0-3]. </para>
</question>

<answer> <para> Unfortunately, early releases of &postgres; 8.1 had a
problem where if you redefined a function (such as, say,
<function>upgradeSchema(text)</function>), and then, in the same
transaction, ran that function, the
<application>postmaster</application> would fall over, and the
transaction would fail to commit.  </para>

<para> The &lslonik; command <xref linkend="stmtupdatefunctions">
functions like that; it, in one transaction, tries to:

<itemizedlist>
<listitem><para> Load the new functions (from <filename>slony1_funcs.sql</filename>), notably including <function>upgradeSchema(text)</function>.  </para> </listitem>
<listitem><para> Run <function>upgradeSchema(text)</function> to do any necessary upgrades to the database schema. </para> </listitem>
<listitem><para> Notify &lslon; processes of a change of configuration.</para> </listitem>
</itemizedlist>
</para>

<para> Unfortunately, on &postgres; 8.1.0, 8.1.1, 8.1.2, and 8.1.3,
this conflicts with a bug where using and modifying a plpgsql function
in the same transaction leads to a crash. </para>

<para> Several workarounds are available. </para>

</answer>

<answer> <para> The preferred answer would be to upgrade &postgres; to
8.1.4 or some later version.  Changes between minor versions do not
require rebuilding databases; it should merely require copying a
suitable 8.1.x build into place, and restarting the
<application>postmaster</application> with the new version.  </para>
</answer>

<answer><para> If that is unsuitable, it would be possible to perform
the upgrade via a series of transactions, performing the equivalent of
what &lslonik; does <quote>by hand</quote>: </para>

<itemizedlist>
<listitem><para> Take <filename>slony1_funcs.sql</filename> and do three replacements within it: </para> 

<itemizedlist>
<listitem><para> Replace <quote>@CLUSTERNAME@</quote> with the name of the cluster </para> </listitem>
<listitem><para> Replace <quote>@MODULEVERSION@</quote> with the &slony1; version string, such as <quote>1.2.10</quote> </para> </listitem>
<listitem><para> Replace <quote>@NAMESPACE@</quote> with the <quote>double-quoted</quote> name of the cluster namespace, such as "_MyCluster" </para> </listitem>
</itemizedlist>
</listitem>
<listitem><para> Load that <quote>remapped</quote> set of functions into the database.</para> </listitem>
<listitem><para> Run the stored function via <command>select <function>upgradeSchema('1.2.7')</function>; </command>, assuming that the previous version of &slony1; in use was version 1.2.7. </para> </listitem>
<listitem><para> Restarting all &lslon; processes would probably be a wise move with this sort of <quote>surgery.</quote> </para> </listitem>
</itemizedlist>
</answer>
</qandaentry>

<qandaentry>
<question><para>Slonik fails - cannot load &postgres; library -
<command>PGRES_FATAL_ERROR load '$libdir/xxid';</command></para>

<para> When I run the sample setup script I get an error message similar
to:

<command>
stdin:64: PGRES_FATAL_ERROR load '$libdir/xxid';  - ERROR:  LOAD:
could not open file '$libdir/xxid': No such file or directory
</command></para></question>

<answer><para> Evidently, you haven't got the
<filename>xxid.so</filename> library in the <envar>$libdir</envar>
directory that the &postgres; instance is
using.  Note that the &slony1; components
need to be installed in the &postgres;
software installation for <emphasis>each and every one</emphasis> of
the nodes, not just on the origin node.</para>

<para>This may also point to there being some other mismatch between
the &postgres; binary instance and the &slony1; instance.  If you
compiled &slony1; yourself, on a machine that may have multiple
&postgres; builds <quote>lying around,</quote> it's possible that the
slon or slonik binaries are asking to load something that isn't
actually in the library directory for the &postgres; database cluster
that it's hitting.</para>

<para>Long and short: This points to a need to <quote>audit</quote>
what installations of &postgres; and &slony1; you have in place on the
machine(s).  Unfortunately, just about any mismatch will cause things
not to link up quite right. 
...</para> 

<para> Life is simplest if you only have one set of &postgres;
binaries on a given server; in that case, there isn't a <quote>wrong
place</quote> in which &slony1; components might get installed.  If
you have several software installs, you'll have to verify that the
right versions of &slony1; components are associated with the right
&postgres; binaries. </para> </answer></qandaentry>

<qandaentry>
<question> <para>I tried creating a CLUSTER NAME with a "-" in it.
That didn't work.</para></question>

<answer><para> &slony1; uses the same rules for unquoted identifiers
as the &postgres; main parser, so no, you probably shouldn't put a "-"
in your identifier name.</para>

<para> You may be able to defeat this by putting <quote>quotes</quote> around
identifier names, but it's still liable to bite you some, so this is
something that is probably not worth working around.</para>
</answer>
</qandaentry>

<qandaentry id="v72upgrade">
<question> <para> I have a &postgres; 7.2-based system that I
<emphasis>really, really</emphasis> want to use &slony1; to help me
upgrade it to 8.0.  What is involved in getting &slony1; to work for
that?</para>
</question>

<answer> <para> Rod Taylor has reported the following...
</para>

<para> This is approximately what you need to do:</para>
<itemizedlist>
<listitem><para>Take the 7.3 templates and copy them to 7.2 -- or otherwise
        hardcode the version your using to pick up the 7.3 templates </para></listitem>
<listitem><para>Remove all traces of schemas from the code and sql templates. I
        basically changed the "." to an "_". </para></listitem>
<listitem><para> Bunch of work related to the XID datatype and functions. For
        example, Slony creates CASTs for the xid to xxid and back -- but
        7.2 cannot create new casts that way so you need to edit system
        tables by hand. I recall creating an Operator Class and editing
        several functions as well. </para></listitem>
<listitem><para>sl_log_1 will have severe performance problems with any kind of
        data volume. This required a number of index and query changes
        to optimize for 7.2. 7.3 and above are quite a bit smarter in
        terms of optimizations they can apply. </para></listitem>
<listitem><para> Don't bother trying to make sequences work. Do them by hand
        after the upgrade using pg_dump and grep. </para></listitem>
</itemizedlist>
<para> Of course, now that you have done all of the above, it's not compatible
with standard Slony now. So you either need to implement 7.2 in a less
hackish way, or you can also hack up slony to work without schemas on
newer versions of &postgres; so they can talk to each other.
</para>
<para> Almost immediately after getting the DB upgraded from 7.2 to 7.4, we
deinstalled the hacked up Slony (by hand for the most part), and started
a migration from 7.4 to 7.4 on a different machine using the regular
Slony. This was primarily to ensure we didn't keep our system catalogues
which had been manually fiddled with.
</para>

<para> All that said, we upgraded a few hundred GB from 7.2 to 7.4
with about 30 minutes actual downtime (versus 48 hours for a dump /
restore cycle) and no data loss.
</para>
</answer>

<answer> <para> That represents a sufficiently ugly set of
<quote>hackery</quote> that the developers are exceedingly reluctant
to let it anywhere near to the production code.  If someone were
interested in <quote>productionizing</quote> this, it would probably
make sense to do so based on the &slony1; 1.0 branch, with the express
plan of <emphasis>not</emphasis> trying to keep much in the way of
forwards compatibility or long term maintainability of replicas.
</para>

<para> You should only head down this road if you are sufficiently
comfortable with &postgres; and &slony1; that you are prepared to hack
pretty heavily with the code.  </para> </answer>
</qandaentry>

<qandaentry id="morethansuper">
<question> <para> I created a <quote>superuser</quote> account,
<command>slony</command>, to run replication activities.  As
suggested, I set it up as a superuser, via the following query: 
<command>
update pg_shadow set usesuper = 't' where usename in ('slony',
'molly', 'dumpy');
</command>
(that command also deals with other users I set up to run vacuums and
backups).</para>

<para> Unfortunately, I ran into a problem the next time I subscribed
to a new set.</para>

<programlisting>
DEBUG1 copy_set 28661
DEBUG1 remoteWorkerThread_1: connected to provider DB
DEBUG2 remoteWorkerThread_78: forward confirm 1,594436 received by 78
DEBUG2 remoteWorkerThread_1: copy table public.billing_discount
ERROR  remoteWorkerThread_1: "select "_mycluster".setAddTable_int(28661, 51, 'public.billing_discount', 'billing_discount_pkey', 'Table public.billing_discount with candidate primary key billing_discount_pkey'); " PGRES_FATAL_ERROR ERROR:  permission denied for relation pg_class
CONTEXT:  PL/pgSQL function "altertableforreplication" line 23 at select into variables
PL/pgSQL function "setaddtable_int" line 76 at perform
WARN   remoteWorkerThread_1: data copy for set 28661 failed - sleep 60 seconds
</programlisting>

<para> This continues to fail, over and over, until I restarted the
<application>slon</application> to connect as
<command>postgres</command> instead.</para>
</question>

<answer><para> The problem is fairly self-evident; permission is being
denied on the system table, <envar>pg_class</envar>.</para></answer>

<answer><para> The <quote>fix</quote> is thus:</para>
<programlisting>
update pg_shadow set usesuper = 't', usecatupd='t' where usename = 'slony';
</programlisting>
</answer>

<answer><para> In version 8.1 and higher, you may also need the following:</para>
<programlisting>
update pg_authid set rolcatupdate = 't', rolsuper='t' where rolname = 'slony';
</programlisting>
</answer>
</qandaentry>

<qandaentry id="PGLISTENERFULL">
<question><para>Some nodes start consistently falling behind</para>

<para>I have been running &slony1; 1.2 on a node for a while, and am
seeing system performance suffering.</para>

<para>I'm seeing long running queries of the form:
<screen>
	fetch 100 from LOG;
</screen></para>

<para> By the way, I'm on &postgres; 8.3. </para>
</question>

<answer><para> This has been characteristic of &pglistener; (which is
the table containing <command>NOTIFY</command> data) having plenty of
dead tuples in it.  That makes <command>NOTIFY</command> events take a
long time, and causes the affected node to gradually fall further and
further behind.</para>

<para>You quite likely need to do a <command>VACUUM FULL</command> on
&pglistener;, to vigorously clean it out, and need to vacuum
&pglistener; really frequently.  Once every five minutes would likely
be AOK.</para>

<para> Slon daemons already vacuum a bunch of tables, and
<filename>cleanup_thread.c</filename> contains a list of tables that
are frequently vacuumed automatically.  In &slony1; 1.0.2,
&pglistener; is not included.  In 1.0.5 and later, it is regularly
vacuumed, so this should cease to be a direct issue.  In version 1.2,
&pglistener; will only be used when a node is only receiving events
periodically, which means that the issue should mostly go away even in
the presence of evil long running transactions...</para>

<para>There is, however, still a scenario where this will still
<quote>bite.</quote> Under MVCC, vacuums cannot delete tuples that
were made <quote>obsolete</quote> at any time after the start time of
the eldest transaction that is still open.  Long running transactions
will cause trouble, and should be avoided, even on subscriber
nodes.</para> </answer>

<answer><para> &postgres; 9.0 introduced a higher performance version
of the LISTEN/NOTIFY feature which eliminated the table &pglistener;
in favour of a memory-based queue.  An upgrade to 9.0 or better would
likely be useful to improve this. </para></answer>

<answer><para> Note that &slony1; 2.0 eliminated heavy usage of
LISTEN/NOTIFY, so this particular phenomenon is restricted to versions
older than that. </para></answer>
</qandaentry>

</qandadiv>

</qandaset>
</sect1>

<!-- Keep this comment at the end of the file
Local variables:
mode:sgml
sgml-omittag:nil
sgml-shorttag:t
sgml-minimize-attributes:nil
sgml-always-quote-attributes:t
sgml-indent-step:1
sgml-indent-data:t
sgml-parent-document:"slony.sgml"
sgml-exposed-tags:nil
sgml-local-catalogs:("/usr/lib/sgml/catalog")
sgml-local-ecat-files:nil
End:
-->
