Update a timescale cluster

Update a timescale cluster#

In order to upgrade completely the cluster (change the machines and the software versions) different steps must be completed.

There are different strategies available, with more or less downtime, here is one that was used and is known to work in operation.

The goal of the update is to start from a cluster A with postgresql-xx and timescaledb-ii and end up with a cluster B with postgresql-yy and timescaledb-jj. The strategy will be:

Upgrade cluster A to the end versions.
Add machines from cluster B to cluster A.
Decommission cluster A.

For this guide we assume you have a cluster managed through patroni[etcd], and hosted on a machine running a debian os. The examples need to be adapted if you’re using a different system, but the strategy stays the same.

Upgrade old cluster#

As stated in timescaledb documentation, to upgrade a cluster, one need to first upgrade postgresql, then timescaledb.

Upgrade postgresql#

Postgresql will be updated using the pg_upgrade method. Only one node in the cluster needs to be upgraded.

Shut down the cluster: Stop all event subscribers, then stop the postgresql service on the machines.

On the primary node, perform the update of postgresql: Replace xx and yy with the proper versions.

apt install postgresql-yy
mv /path/to/data /path/to/data.old
mkdir /path/to/data
chown postgres:postgres /path/to/data
/usr/lib/postgresql/yy/bin/initdb -D /path/to/data -k
/usr/lib/postgresql/yy/bin/pg_upgrade --link --old-datadir /path/to/data.old --new-datadir /path/to/data --old-bindir /usr/lib/postgresql/xx/bin --new-bindir /usr/lib/postgresql/yy/bin

Now update postgresql.conf to retrieve the modifications you could have done on the old one.
Update /etc/patroni.yml, there are path to the bin dir for postgresql that need to be set up.
remove cluster information from the DCS (etcd), as the call to initdb reset the unique systemID, you need to restart the upgraded version in a fresh cluster. This is done via:
```
patronictl remove hdbpp
```
it will ask for many confirmations.
Fix the patroni startup script in /etc/systemd/system/patroni.service, there is a reference to postgresql version, so replace YY with the proper version.

ExecStartPre=-/bin/mkdir -m 2750 -p /var/run/postgresql/YY-main.pg_stat_tmp

restart patroni
upgrade timescaledb, see Upgrade timescaledb

If anything goes wrong at any point, one can restore /path/to/data.old and restart the old cluster. Otherwise we are now running the cluster with the new versions.

Upgrade timescaledb#

TimescaleDb supports in place updates. See this guide. Before performing the update, it is advisable to backup the database as a security measure.

To see the current version, log in as a admin user and display the extension table:

hdb_test=> \dx timescaledb
                                    List of installed extensions
    Name     | Version | Schema |                            Description
-------------+---------+--------+-------------------------------------------------------------------
 timescaledb | 1.3.2   | public | Enables scalable inserts and complex queries for time-series data
(1 row)

General process:

Backup database.

Update the package:

apt-get --only-upgrade install timescaledb-postgresql-yy

log into the system as a superuser and run:
```
ALTER EXTENSION timescaledb UPDATE;
```

Add machines to the old cluster#

This is simply done by starting patroni on the new machines. It will take some time for all the data to be copied, but once it’s done we’re almost there.

Decommission the old cluster#

The final step is to stop patroni on the old machines, and they will automatically be removed from the cluster. We are now running an updated cluster on new machines.