If you’ve installed BitBucket on a CentOS 7 server and use PostgreSQL as the back-end database, you’ve probably seen this nagging warning that support for PostgreSQL 9.2.24 has been deprecated and will be removed in an upcoming release.

While PostgeSQL 9.2 is quite old and they are already announcing version 12, mind that any of the Atlassian stack can’t support anything more than version 9.6 at the time of this writing.
So, I’ll explain how to upgrade PostgreSQL on CentOS from 9.2.24 to PostgreSQL 9.6. While this post is for CentOS, it might work for other distros too. There are some prerequisites though. First, I assume that you installed PostgreSQL using yum from the default repo. In this case, your database most likely resides under /var/lib/pgsql/data and the binaries are under /usr/bin/psql. If you use the default version, then you probably use systemctl start/stop postgresql to manage the start/stop of the database daemon. If you have something different, the following tutorial will probably work, but you’ll have to adjust some directories.
First thing first, let’s check the version of PostgreSQL that we use.

psql --version
psql (PostgreSQL) 9.2.24

Next, stop any services that use the database, such as BitBucket, Jira or Confluence or whatever you have there.
Log as the postgres user that you use to manage the database. By the default this user is postgres.
Do a full backup and move the backup somewhere because the home directory for the postgres user is where the database home is. You can compress the SQL file first. It’s just a plain text file.

pg_dumpall > backup.sql
mv backup.sql somewhere/

Once done, stop the postgreSQL daemon, but log as root first. You won’t be able to stop the daemon using the postgres user.

systemctl stop postgresql

Go to a temp folder and get the RPM package from the PostgreSQL site. In case the link doesn’t work, go to the website using a browser and see what’s the latest 9.6 version.

cd /tmp
wget https://download.postgresql.org/pub/repos/yum/9.6/redhat/rhel-7-x86_64/pgdg-centos96-9.6-3.noarch.rpm

Install the newer version.

rpm -ivh pgdg-centos96-9.6-3.noarch.rpm
yum install postgresql96-server

You’ll get some warning that symlinks can’t be created but you can ignore them. The new version installs the binaries under /usr/psql-9.6/bin.
Log as the postgres user and initialize the new database.

/usr/pgsql-9.6/bin/initdb -D /var/lib/pgsql/9.6/data/
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.

The database cluster will be initialized with locale "en_US.UTF-8".
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "english".

Data page checksums are disabled.

fixing permissions on existing directory /var/lib/pgsql/9.6/data ... ok
creating subdirectories ... ok
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting dynamic shared memory implementation ... posix
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok

WARNING: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.

Success. You can now start the database server using:

    /usr/pgsql-9.6/bin/pg_ctl -D /var/lib/pgsql/9.6/data/ -l logfile start

Ignore the line that says that we can start the server. We have to upgrade the current database first. Make sure that the directories below are the same as yours. If not, make changes.

/usr/pgsql-9.6/bin/pg_upgrade --old-datadir /var/lib/pgsql/data/ --new-datadir /var/lib/pgsql/9.6/data/ --old-bindir /usr/bin/ --new-bindir /usr/pgsql-9.6/bin/
Performing Consistency Checks
-----------------------------
Checking cluster versions                                   ok

*failure*
Consult the last few lines of "pg_upgrade_server.log" for
the probable cause of the failure.

connection to database failed: could not connect to server: No such file or directory
        Is the server running locally and accepting
        connections on Unix domain socket "/var/lib/pgsql/.s.PGSQL.50432"?


could not connect to old postmaster started with the command:
"/usr/bin/pg_ctl" -w -l "pg_upgrade_server.log" -D "/var/lib/pgsql/data/" -o "-p 50432 -b  -c listen_addresses='' -c unix_socket_permissions=0700 -c unix_socket_directory='/var/lib/pgsql'" start
Failure, exiting

Oh boy. If you check the error message, you’ll get some more info.

tail /var/lib/pgsql/pg_upgrade_server.log
server stopped


command: "/usr/bin/pg_ctl" -w -l "pg_upgrade_server.log" -D "/var/lib/pgsql/data/" -o "-p 50432 -b  -c listen_addresses='' -c unix_socket_permissions=0700 -c unix_socket_directory='/var/lib/pgsql'" start >> "pg_upgrade_server.log" 2>&1
waiting for server to start....FATAL:  unrecognized configuration parameter "unix_socket_directory"
 stopped waiting
pg_ctl: could not start server
Examine the log output.

The reason being that as of PostgreSQL version 9.3 the config parameter unix_socket_directory has been replaced with unix_socket_directories. There is a workaround thanks to this guy. The link for more info is here.
Pretty much, all you have to do is execute these lines first. Execute these lines as root or sudo user.

mv /usr/bin/pg_ctl{,-orig}
echo '#!/bin/bash' > /usr/bin/pg_ctl
echo '"$0"-orig "${@/unix_socket_directory/unix_socket_directories}"' >> /usr/bin/pg_ctl
chmod +x /usr/bin/pg_ctl

Now you can do the upgrade.

/usr/pgsql-9.6/bin/pg_upgrade --old-datadir /var/lib/pgsql/data/ --new-datadir /var/lib/pgsql/9.6/data/ --old-bindir /usr/bin/ --new-bindir /usr/pgsql-9.6/bin/
Performing Consistency Checks
-----------------------------
Checking cluster versions                                   ok
Checking database user is the install user                  ok
Checking database connection settings                       ok
Checking for prepared transactions                          ok
Checking for reg* system OID user data types                ok
Checking for contrib/isn with bigint-passing mismatch       ok
Checking for roles starting with 'pg_'                      ok
Checking for invalid "line" user columns                    ok
Creating dump of global objects                             ok
Creating dump of database schemas
                                                            ok
Checking for presence of required libraries                 ok
Checking database user is the install user                  ok
Checking for prepared transactions                          ok

If pg_upgrade fails after this point, you must re-initdb the
new cluster before continuing.

Performing Upgrade
------------------
Analyzing all rows in the new cluster                       ok
Freezing all rows on the new cluster                        ok
Deleting files from new pg_clog                             ok
Copying old pg_clog to new server                           ok
Setting next transaction ID and epoch for new cluster       ok
Deleting files from new pg_multixact/offsets                ok
Setting oldest multixact ID on new cluster                  ok
Resetting WAL archives                                      ok
Setting frozenxid and minmxid counters in new cluster       ok
Restoring global objects in the new cluster                 ok
Restoring database schemas in the new cluster
                                                            ok
Setting minmxid counter in new cluster                      ok
Copying user relation files
                                                            ok
Setting next OID for new cluster                            ok
Sync data directory to disk                                 ok
Creating script to analyze new cluster                      ok
Creating script to delete old cluster                       ok

Upgrade Complete
----------------
Optimizer statistics are not transferred by pg_upgrade so,
once you start the new server, consider running:
    ./analyze_new_cluster.sh

Running this script will delete the old cluster's data files:
    ./delete_old_cluster.sh

Start the new database now and check everything. As you can see, the startup script is different. This time we have to specify the version of PostgreSQL. As root do:

systemctl start postgresql-9.6

You can revert the change back.

mv -f /usr/bin/pg_ctl{-orig,}

Log as postgres user and do a check up. It might take some time if your database is large.

./analyze_new_cluster.sh

You can delete the old database if you want.

./delete_old_cluster.sh

If you check the new version, you’ll see that it’s the same.

psql --version
psql (PostgreSQL) 9.2.24

The database is upgraded and running 9.6, it’s just the psql binary that’s the old version. You have to tell the postgres user to use the new binaries. Delete the old binaries first, uninstall the old version, make sure it doesn’t start on boot and enable the new version. Also, if you have any specific configurations, make sure you compare postgresql.conf and pg_hba.conf under /var/lib/pgsql/data and /var/lib/pgsql/9.6/data. Log as root first. Make sure it says 9.2 when you run yum remove.

systemctl disable postgresql
yum remove postgresql

Enable postgreSQL to boot on start.

systemctl enable postgresql-9.6

Add the following line under .bash_profile for the postgres user.

export PATH=$PATH:/usr/pgsql-9.6/bin

If you don’t have a .bash_profile file, check if there is any .bash_profile.rpmsave. Rename this one as .bash_profile and add the above line. It should look like this.

[ -f /etc/profile ] && source /etc/profile
PGDATA=/var/lib/pgsql/9.6/data
export PGDATA
# If you want to customize your settings,
# Use the file below. This is not overridden
# by the RPMS.
[ -f /var/lib/pgsql/.pgsql_profile ] && source /var/lib/pgsql/.pgsql_profile
export PATH=$PATH:/usr/pgsql-9.6/bin

If you log off, log back on as the postgres user or execute source .bash_profile and check the version, you’ll see that you are all set.

psql --version
psql (PostgreSQL) 9.6.11

In the part 1, we discussed how to upgrade the OS from 8.1 to 9.1. But most of the applications that were installed are outdated at this point. In order to upgrade the port tree, use the following commands:

portsnap fetch extract
portsnap fetch update

These commands will download the current port tree, but won’t actually install the applications. At this point it is very important to check if you want to actually upgrade your applications. Some of them might now work, but if they work fine, you might leave them as is. Mind that this might cause some compatibility issues. So, in order to upgrade the installed application, let’s look what’s installed. The easiest way is to run:

portmaster -L --index-only| egrep '(ew|ort) version|total install'

The output will look similar to this.

If you want to upgrade an installed application, do:

portmaster openvpn

But, don’t do that without reading /usr/ports/UPDATING file first. As a matter of fact, this is a file that you have to read it before any port updates. For example, you can’t run:

portmaster pkg-config

According to the UPDATING file:

20120726:
AFFECTS: users of devel/pkg-config
AUTHOR: bapt@FreeBSD.org

devel/pkg-config has been replaced by devel/pkgconf

# portmaster -o devel/pkgconf devel/pkg-config
or
# portupgrade -fo devel/pkgconf pkg-config-\*

The same is valid for jpeg. Mind that some of the ports that you have installed are very important and used by many applications. So, spend some time and read the UPDATING file first. Pay special attention on all PHP installs.

What I do is to open three terminal sessions. The first one is where I execute:

portmaster -L --index-only| egrep '(ew|ort) version|total install'

The second terminal has /usr/ports/UPDATING file opened, so I can search for the ports remaining that needs to be upgraded, and the third terminal window is where I execute the portmaster command.

If you receive an error “pkg_version: corrupted record for…”, do:

portmaster --check-depends

You’ll be prompted to remove some of the obsolete ports, most likely some php* ports.

Links:
http://www.wonkity.com/~wblock/docs/html/portupgrade.html
https://www.freebsd.org/doc/handbook/ports-using.html

This post is based on the Chapter 25 of the official FreeBSD handbook. My version was the initial release of 8.2 using a generic kernel:

uname -a
FreeBSD acidjazz.chombe.org 8.2-RELEASE FreeBSD 8.2-RELEASE #0: Fri Feb 18 02:24:46 UTC 2011
root@almeida.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC  i386

First, make sure you start with a rebooted OS and make a copy of the /etc directory. Then do:

freebsd-update -r 9.1-RELEASE upgrade

After a second or two, I received a very funny error.

The update metadata is correctly signed, but failed an integrity check.
Cowardly refusing to proceed any further.

At this point, execute:

sed -i '' -e 's/=_/=%@_/' /usr/sbin/freebsd-update
freebsd-update -r 9.1-RELEASE upgrade

Say “y” and wait for some time for FreeBSD to inspect and download the necessary files.

Looking up update.FreeBSD.org mirrors... 3 mirrors found.
Fetching metadata signature for 8.2-RELEASE from update5.freebsd.org... done.
Fetching metadata index... done.
Inspecting system... done.

Fetching metadata signature for 9.1-RELEASE from update5.freebsd.org... done.
Fetching metadata index... done.
Inspecting system... done.
Fetching files from 8.2-RELEASE for merging... done.
Preparing to download files...

About 30 minutes later, on my VM, which is very low on resources (8GB HDD, 512MB RAM), you’ll be greeted with the “message of the day” /etc/motd. Just do “:q!”

You’ll get another prompt about new groups in /etc/groups. Say “y” to continue.

Same thing for /etc/hosts, /etc/manpath.config, /etc/master.passwd, /etc/motd, /etc/passwd, /etc/syslog.conf, /etc… That’s why it is important to make a backup of /etc as suggested earlier.
Hit “CTRL-C”, when you see the list of files that will be removed as part of the upgrade.
Then do:

freebsd-update install

Once everything is completed, do:

shutdown -r now

….and then do:

freebsd-update install

again…
After the update finishes, you’ll see this message.

The next part shows how to do the same.