<!-- doc/src/sgml/config.sgml -->

<sect1 id="runtime-online-recovery">
  <title>Online Recovery</title>

  <para>
    <productname>Pgpool-II</productname> can synchronize database nodes and attach
	a node without stopping the service.
    This feature is called <acronym>"online recovery"</acronym>.
  </para>
  <para>
    For online recovery, the recovery target node must be detached from <productname>Pgpool-II</>.
    If you wish to add a <productname>PostgreSQL</> server node dynamically, reload the
    <filename>pgpool.conf</filename> after adding the
    <xref linkend="guc-backend-hostname"> and its associated parameters.
    This will register the new server to <productname>Pgpool-II</productname> as a detached backend node.
  </para>
<!--
  <caution>
    <para>
      Make sure that <command>autovacuum</command> is stopped on the
      master node (the first node which is up and running) before starting the
      online recovery. Autovacuum can change the contents of the database which
      can cause the inconsistency after the online recovery.
    </para>
    <para>
      This applies only if you're recovering with a simple copy mechanism,
      such as the <command>rsync</command> and doesn't apply when using
      the PostgreSQL's PITR mechanism.
    </para>
  </caution>
-->
  <note>
    <para>
      The recovery target <productname>PostgreSQL</> server must not be running for performing the online recovery.
      If the target <productname>PostgreSQL</> server has already started, you must shut it down before
      starting the online recovery.
    </para>
  </note>

  <para>
    Online recovery is performed in two phases. Connections from cliens are
    not allowd only in second phase of online recovery while the data can be
    updated or retrieved during the first phase.
    <productname>Pgpool-II</productname> performs the follows steps in online recovery:
  </para>
  <itemizedlist>

    <listitem>
      <para>
        CHECKPOINT.
      </para>
    </listitem>
    <listitem>
      <para>
        First stage of online recovery.
      </para>
    </listitem>
    <listitem>
      <para>
        Wait until all client connections have disconnected.
      </para>
    </listitem>
    <listitem>
      <para>
        CHECKPOINT.
      </para>
    </listitem>
    <listitem>
      <para>
        Second stage of online recovery.
      </para>
    </listitem>
    <listitem>
      <para>
        Start up postmaster (perform <literal>pgpool_remote_start</literal>)
      </para>
    </listitem>
    <listitem>
      <para>
        Node attach
      </para>
    </listitem>

  </itemizedlist>
  <note>
    <para>
      There is a restriction in the online recovery. If
      <productname>Pgpool-II</productname> itself is installed
      on multiple hosts, online recovery does not work correctly,
      because <productname>Pgpool-II</productname> has to stop all
      the clients during the 2nd stage of online recovery.
      If there are several <productname>Pgpool-II</productname> hosts,
      only one of them will have received the online recovery command and will
      block the connections from clients.
    </para>
  </note>
  <variablelist>

    <varlistentry id="guc-recovery-user" xreflabel="recovery_user">
      <term><varname>recovery_user</varname> (<type>string</type>)
        <indexterm>
        <primary><varname>recovery_user</varname> configuration parameter</primary>
        </indexterm>
      </term>
      <listitem>
      <para>
        Specifies the <productname>PostgreSQL</> user name to perform online recovery.
      </para>
        <para>
          This parameter can be changed by reloading the <productname>Pgpool-II</> configurations.
        </para>
      </listitem>
    </varlistentry>

    <varlistentry id="guc-recovery-password" xreflabel="recovery_password">
      <term><varname>recovery_password</varname> (<type>string</type>)
        <indexterm>
        <primary><varname>recovery_password</varname> configuration parameter</primary>
        </indexterm>
      </term>
      <listitem>
        <para>
          Specifies the password for the <productname>PostgreSQL</> user name configured in
          <xref linkend="guc-recovery-user"> to perform online recovery.
        </para>
        <para>
          This parameter can be changed by reloading the <productname>Pgpool-II</> configurations.
        </para>
      </listitem>
    </varlistentry>

    <varlistentry id="guc-recovery-1st-stage-command" xreflabel="recovery_1st_stage_command">
      <term><varname>recovery_1st_stage_command</varname> (<type>string</type>)
        <indexterm>
        <primary><varname>recovery_1st_stage_command</varname> configuration parameter</primary>
        </indexterm>
      </term>
      <listitem>
        <para>
          Specifies a command to be run by master(primary) node at the
          first stage of online recovery. The command file must be placed in the
          database cluster directory for security reasons.
          For example, if <varname>recovery_1st_stage_command</varname> = <literal>
          'sync-command'</literal>, then <productname>Pgpool-II</productname> will
          look for the command scrit in <literal>$PGDATA</literal> directory and will
          try to execute <command>$PGDATA/sync-command</command>.
        </para>
        <para>
          <varname>recovery_1st_stage_command</varname> receives following 4 parameters:
        </para>

        <itemizedlist>
          <listitem>
            <para>
              Path to the database cluster of the master(primary) node.
            </para>
          </listitem>
          <listitem>
            <para>
              Hostname of the backend node to be recovered.
            </para>
          </listitem>
          <listitem>
            <para>
              Path to the database cluster of the node to be recovered.
            </para>
          </listitem>
          <listitem>
            <para>
              Port number of the master(primary) node.
            </para>
          </listitem>
        </itemizedlist>

        <note>
          <para>
            <productname>Pgpool-II</productname> accept connections and queries
            while <varname>recovery_1st_stage command</varname> is executed,
            so you can retrieve and update data.
          </para>
        </note>

        <caution>
          <para>
            <varname>recovery_1st_stage command</varname> runs as a <acronym>SQL</acronym>
            command from PostgreSQL's point of view. So <varname>recovery_1st_stage command
            </varname> can get prematuraly killed by PostgreSQL if the PostgreSQL's
            <varname>statement_time_out</varname> is configured with the value that is
            smaller than the time <varname>recovery_1st_stage_command</varname> takes for
            completion.
          </para>
          <para>
            Typical error in such case is
            <programlisting>
rsync used in the command is killed by signal 2 for example.
            </programlisting>
          </para>
        </caution>

        <para>
          This parameter can be changed by reloading the <productname>Pgpool-II</> configurations.
        </para>
      </listitem>
    </varlistentry>

    <varlistentry id="guc-recovery-2nd-stage-command" xreflabel="recovery_2nd_stage_command">
      <term><varname>recovery_2nd_stage_command</varname> (<type>string</type>)
        <indexterm>
        <primary><varname>recovery_2nd_stage_command</varname> configuration parameter</primary>
        </indexterm>
      </term>
      <listitem>

        <para>
          Specifies a command to be run by master(primary) node at the
          second stage of online recovery. The command file must be placed in the
          database cluster directory for security reasons.
          For example, if <varname>recovery_2nd_stage_command</varname> = <literal>
          'sync-command'</literal>, then <productname>Pgpool-II</productname> will
          look for the command scrit in <literal>$PGDATA</literal> directory and will
          try to execute <command>$PGDATA/sync-command</command>.
        </para>
        <para>
          <varname>recovery_2nd_stage_command</varname> receives following 4 parameters:
        </para>

        <itemizedlist>
          <listitem>
            <para>
              Path to the database cluster of the master(primary) node.
            </para>
          </listitem>
          <listitem>
            <para>
              Hostname of the backend node to be recovered.
            </para>
          </listitem>
          <listitem>
            <para>
              Path to the database cluster of the node to be recovered.
            </para>
          </listitem>
          <listitem>
            <para>
              Port number of database to be recovered.
            </para>
          </listitem>
        </itemizedlist>

        <note>
          <para>
            <productname>Pgpool-II</productname> <emphasis>does not</emphasis>
            accept client connections and queries during the execution
            of <varname>recovery_2nd_stage_command</varname> command, and waits
            for the existing clients to close their connections before executing the
            command.
            Therefore, the <varname>recovery_2nd_stage_command</varname> may not execute
            if the client stays connected for a long time.
          </para>
        </note>

        <caution>
          <para>
            <varname>recovery_2nd_stage command</varname> runs as a <acronym>SQL</acronym>
            command from PostgreSQL's point of view. Therefore, <varname>recovery_2nd_stage command
            </varname> can get prematuraly killed by PostgreSQL if the PostgreSQL's
            <varname>statement_time_out</varname> is configured with the value that is
            smaller than the time <varname>recovery_2nd_stage_command</varname> takes for
            completion.
          </para>
        </caution>

        <para>
          This parameter can be changed by reloading the <productname>Pgpool-II</> configurations.
        </para>
      </listitem>
    </varlistentry>

    <varlistentry id="guc-recovery-timeout" xreflabel="recovery_timeout">
      <term><varname>recovery_timeout</varname> (<type>integer</type>)
        <indexterm>
        <primary><varname>recovery_timeout</varname> configuration parameter</primary>
        </indexterm>
      </term>
      <listitem>
        <para>
          Specifies the timeout in seconds to cancel the online recovery if it
          does not completes within this time.
          Since <productname>Pgpool-II</productname> does not accepts the connections
          during the second stage of online recovery, this parameter can be used to cancel
          the online recovery to manage the service down time during the online recovery.
        </para>
        <para>
          This parameter can be changed by reloading the <productname>Pgpool-II</> configurations.
        </para>
      </listitem>
    </varlistentry>

    <varlistentry id="guc-client-idle-limit-in-recovery" xreflabel="client_idle_limit_in_recovery">
      <term><varname>client_idle_limit_in_recovery</varname> (<type>integer</type>)
        <indexterm>
        <primary><varname>client_idle_limit_in_recovery</varname> configuration parameter</primary>
        </indexterm>
      </term>
      <listitem>
        <para>
          Specifies the time in seconds to disconnect a client if it remains idle
          since the last query during the online recovery.
          <varname>client_idle_limit_in_recovery</varname> is similar to the
          <xref linkend="guc-client-idle-limit"> but only takes effect during the
          second stage of online recovery.
        </para>
        <para>
          This is useful for preventing the <productname>Pgpool-II</productname>
          recovery from being disturbed by the lazy clients or if the TCP/IP
          connection between the client and <productname>Pgpool-II</productname>
          is accidentally down (a cut cable for instance).
        </para>
        <para>
          If set to -1, all clients get immediately disconnected when the second
          stage of online recovery starts.
          The default is 0, which turns off the feature.
        </para>

        <para>
          This parameter can be changed by reloading the <productname>Pgpool-II</> configurations.
					You can also use <xref linkend="SQL-PGPOOL-SET"> command to alter the value of
					this parameter for a current session.
        </para>
      </listitem>
    </varlistentry>

  </variablelist>
</sect1>
