Every 5 years, it's the same song and dance. The Debian LTS version is expiring, and it's time to update the server. It's also an opportune moment to upgrade to a newer machine to benefit from improved performance.
If we have a SATA hard drive, it's better to start with new disks regardless, since their lifespan is limited.
So we will proceed with a series of backups on the old server, transfer them to the new one, and reinstall everything as it was.
In our case, we were on Debian 8, which supports GitLab 13.3.9.
We are switching machines to Debian 10, which supports GitLab 14.x.x
Installing GitLab-ce is easy, as long as we don't have any repositories to migrate. The complication comes from ensuring we don't lose anything from the old instance. At partITech we have more than 100 repositories totaling about 20 GB of archives.
The archives are not compatible between versions. Therefore, if we have an archive created with version 13.3.9, we would need to install that same version and import it. The last step will involve migrating from 13.3.9 to the latest 14.x version
But to proceed with this major version upgrade, it's imperative to install the latest version of the 13.x series, which is 13.12.0
Installation of the base packages
apt-get install vim curl backup-manager mlocate libcurl4-openssl-dev libexpat1-dev gettext libz-dev libssl-dev libpcre2-dev build-essential git-core graphicsmagick libimage-exiftool-perl htop
We install vim and disable the mouse.
vi ~/.vimrc
#on ajoute set mouse-=a
We uncomment the aliases.
vi ~/.bashrc
. ~/.bashrc
On the old server
Services are stopped.
sudo gitlab-ctl stop unicorn
sudo gitlab-ctl stop sidekiq
A backup is created.
sudo gitlab-rake gitlab:backup:create
We prepare a directory to put all our files to be transferred.
mkdir ~/gitlab-old
We copy our configuration files from our directory. /etc/gitlab (gitlab.rb and gitlab-secrets.json) as well as the directory /etc/gitlab/ssl
sudo cp /etc/gitlab/gitlab.rb ~/gitlab-old
sudo cp /etc/gitlab/gitlab-secrets.json ~/gitlab-old
sudo cp -R /etc/gitlab/ssl ~/gitlab-old
sudo cp /etc/backup-manager.conf ~/gitlab-old/
sudo cp /etc/backup-manager.sh ~/gitlab-old/
On the new server
We install the GitLab apt source.
curl -s https://packages.gitlab.com/install/repositories/gitlab/gitlab-ce/script.deb.sh | bash
We install GitLab in exactly the same version as the old one to avoid installation problems. Then we will update to the latest version.
sudo apt-get install gitlab-ce=13.3.9-ce.0
We install our configuration files from our old instance.
sudo mv /etc/gitlab/gitlab.rb /etc/gitlab/_gitlab.rb
sudo cp gitlab-old/gitlab* /etc/gitLab
We copy the SSL directory from our old instance.
cp -R gitlab-old/ssl /etc/gitlab/
We initiate GitLab reconfiguration.
sudo gitlab-ctl reconfigure
We shut down the services.
sudo gitlab-ctl stop unicorn
sudo gitlab-ctl stop sidekiq
We place our first backup in GitLab's default backup directory (we have modified our default directory).
sudo cp gitlab-old/1624870826_2021_06_28_13.3.9_gitlab_backup.tar ~/gitlab-backups
And we import it.
sudo gitlab-rake gitlab:backup:restore BACKUP=1624870826_2021_06_28_13.3.9
We set the permissions.
sudo find /data/gitlab-storage -type f -exec chmod 0644 {} \;
sudo find /data/gitlab-storage/uploads -type d -not -path /data/gitlab-storage/uploads -exec chmod 0700 {} \;
We convert our repositories to hashes if necessary.
gitlab-rake gitlab:storage:migrate_to_hashed
We open ports 80 and 443 in the firewall if necessary.
iptables -A INPUT -p tcp --dport 80 -m conntrack --ctstate NEW,ESTABLISHED -j ACCEPT
iptables -A INPUT -p tcp --dport 443 -m conntrack --ctstate NEW,ESTABLISHED -j ACCEPT
The interface should now respond normally on our new server.
Upgrading versions
Now we need to update GitLab.
First step, we switch to version 13.5.0
apt-get install gitlab-ce=13.5.0-ce.0
And there we hit the following error: PG::DuplicateTable: ERROR: relation "postgres_indexes" already exists
To get the script to run, we will need to modify the upgrade procedure to add a drop of the indexes in the file: /opt/gitlab/embedded/service/gitlab-rails/db/migrate/20200922093004_add_postgres_index_view.rb
# frozen_string_literal: true
class AddPostgresIndexView < ActiveRecord::Migration[6.0]
DOWNTIME = false
def up
execute(<<~SQL)
DROP VIEW IF EXISTS postgres_indexes;
CREATE VIEW postgres_indexes AS
SELECT
pg_namespace.nspname || '.' || pg_class.relname as identifier,
pg_index.indexrelid,
pg_namespace.nspname as schema,
pg_class.relname as name,
pg_index.indisunique as unique,
pg_index.indisvalid as valid_index,
pg_class.relispartition as partitioned,
pg_index.indisexclusion as exclusion,
pg_indexes.indexdef as definition,
pg_relation_size(pg_class.oid) as ondisk_size_bytes
FROM pg_index
INNER JOIN pg_class ON pg_class.oid = pg_index.indexrelid
INNER JOIN pg_namespace ON pg_class.relnamespace = pg_namespace.oid
INNER JOIN pg_indexes ON pg_class.relname = pg_indexes.indexname
WHERE pg_namespace.nspname <> 'pg_catalog'
SQL
end
def down
execute(<<~SQL)
DROP VIEW postgres_indexes
SQL
end
end
Once the migration script is modified, we relaunch the reconfigure.
gitlab-ctl stop unicorn && gitlab-ctl stop sidekiq && gitlab-ctl reconfigure
We can now proceed to the next upgrade. Upgrade to version 13.6.0 and increment the installations up to version 13.12.0
apt-get install gitlab-ce=13.6.0-ce.0 && gitlab-ctl stop unicorn && gitlab-ctl stop sidekiq && gitlab-ctl reconfigure
apt-get install gitlab-ce=13.7.0-ce.0 && gitlab-ctl stop unicorn && gitlab-ctl stop sidekiq && gitlab-ctl reconfigure
apt-get install gitlab-ce=13.8.0-ce.0 && gitlab-ctl stop unicorn && gitlab-ctl stop sidekiq && gitlab-ctl reconfigure
apt-get install gitlab-ce=13.9.0-ce.0 && gitlab-ctl stop unicorn && gitlab-ctl stop sidekiq && gitlab-ctl reconfigure
apt-get install gitlab-ce=13.10.0-ce.0 && gitlab-ctl stop unicorn && gitlab-ctl stop sidekiq && gitlab-ctl reconfigure
apt-get install gitlab-ce=13.11.0-ce.0 && gitlab-ctl stop unicorn && gitlab-ctl stop sidekiq && gitlab-ctl reconfigure
apt-get install gitlab-ce=13.12.5-ce.0 && gitlab-ctl stop unicorn && gitlab-ctl stop sidekiq && gitlab-ctl reconfigure
For version 13.7.0, the migration script will need to be corrected.
/opt/gitlab/embedded/service/gitlab-rails/db/migrate/20201127170848_add_index_bloat_estimate_view.rb
# frozen_string_literal: true
class AddIndexBloatEstimateView < ActiveRecord::Migration[6.0]
DOWNTIME = false
def up
execute(<<~SQL)
DROP VIEW IF EXISTS postgres_index_bloat_estimates;
CREATE VIEW postgres_index_bloat_estimates AS
-- Originally from: https://github.com/ioguix/pgsql-bloat-estimation/blob/master/btree/btree_bloat.sql
-- WARNING: executed with a non-superuser role, the query inspect only index on tables you are granted to read.
-- WARNING: rows with is_na = 't' are known to have bad statistics ("name" type is not supported).
-- This query is compatible with PostgreSQL 8.2 and after
SELECT nspname || '.' || idxname as identifier,
CASE WHEN relpages > est_pages_ff
THEN bs*(relpages-est_pages_ff)
ELSE 0
END::bigint AS bloat_size_bytes
FROM (
SELECT
coalesce(1 +
ceil(reltuples/floor((bs-pageopqdata-pagehdr)*fillfactor/(100*(4+nulldatahdrwidth)::float))), 0
) AS est_pages_ff,
bs, nspname, tblname, idxname, relpages, is_na
FROM (
SELECT maxalign, bs, nspname, tblname, idxname, reltuples, relpages, idxoid, fillfactor,
( index_tuple_hdr_bm +
maxalign - CASE -- Add padding to the index tuple header to align on MAXALIGN
WHEN index_tuple_hdr_bm%maxalign = 0 THEN maxalign
ELSE index_tuple_hdr_bm%maxalign
END
+ nulldatawidth + maxalign - CASE -- Add padding to the data to align on MAXALIGN
WHEN nulldatawidth = 0 THEN 0
WHEN nulldatawidth::integer%maxalign = 0 THEN maxalign
ELSE nulldatawidth::integer%maxalign
END
)::numeric AS nulldatahdrwidth, pagehdr, pageopqdata, is_na
FROM (
We now install version 14.0.0.
apt-get install gitlab-ce=13.12.0-ce.0 && gitlab-ctl stop unicorn && gitlab-ctl stop sidekiq && gitlab-ctl reconfigure
Since version 14, it's imperative to use the hash filesystem. The installation will show a warning and stop the process.
root@git-partitech-sd-47766:/opt/gitlab# apt-get install gitlab-ce=14.0.0-ce.0 && gitlab-ctl stop unicorn && gitlab-ctl stop sidekiq && gitlab-ctl reconfigure
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages will be upgraded:
gitlab-ce
1 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Need to get 0 B/923 MB of archives.
After this operation, 14.5 MB of additional disk space will be used.
Reading changelogs... Done
(Reading database ... 119998 files and directories currently installed.)
Preparing to unpack .../gitlab-ce_14.0.0-ce.0_amd64.deb ...
gitlab preinstall: Checking for unmigrated data on legacy storage
gitlab preinstall:
gitlab preinstall: Legacy storage is no longer supported. Please migrate your data to hashed storage.
gitlab preinstall: Check https://docs.gitlab.com/ee/administration/raketasks/storage.html#migrate-to-hashed-storage for details.
gitlab preinstall:
gitlab preinstall: If you want to skip this check, run the following command and try again:
gitlab preinstall:
gitlab preinstall: sudo touch /etc/gitlab/skip-unmigrated-data-check
gitlab preinstall:
dpkg: error processing archive /var/cache/apt/archives/gitlab-ce_14.0.0-ce.0_amd64.deb (--unpack):
new gitlab-ce package pre-installation script subprocess returned error exit status 1
Errors were encountered while processing:
/var/cache/apt/archives/gitlab-ce_14.0.0-ce.0_amd64.deb
E: Sub-process /usr/bin/dpkg returned an error code (1)
So you will need to convert your repositories if not already done.
gitlab-rake gitlab:storage:migrate_to_hashed
And then launch the install specifying to no longer display the alert.
sudo touch /etc/gitlab/skip-unmigrated-data-check
apt-get install gitlab-ce=14.0.0-ce.0 && gitlab-ctl stop unicorn && gitlab-ctl stop sidekiq && gitlab-ctl reconfigure
The most critical step is behind us. We can now update to the very latest version. On our side, it's version 14.0.1
apt-get upgrade
Last operation. If, like us, you used Slack notifications, you will need to migrate the configuration. The services templates have been removed since version 14.x.x
You can reintegrate the notifications project by project, or like us, set it up globally.
For this, you will need to go to Admin Area >> System Hooks
