This is the first blog post in a series that will explain how to build a high availability instance of GitLab on the Linux on Z platform. The first step in that process is to build the GitLab installation package for the Linux on Z architecture.
GitLab has many different installation options as you can see in the official installation documentation:
https://about.gitlab.com/installation/
But the recommended process is to use the omnibus installer, as it packages all the pieces together nicely, and makes for a very easy installation.
The standard Package install that is offered by GitLab uses something called an Omnibus package. This is a type of installer that was created by the Chef Team. It works by creating a build script in a ruby like DSL, and then running that script to build the install package. This build script contains links to all the Gems and software packages that need to be download as prerequisites, as well as the source code for that software. The omnibus build takes care of downloading all the prerequisites, compiling them, bundling it all together and then packing into an RPM and a deb package. So in this system, the install package generally does not have any dependencies that need to be met by the package manager; everything is pre-bundled.
However, the rpm and deb packages offered in their official repositories are not compatible with the s390x architecture of our Linux on Z instances. Luckily, the building of the omnibus package is an open source project, so we can make a couple of tweaks to the source code, and rebuild the RPM and deb packages for our systems. The only current limitation is that the Mattermost chat client that comes bundled with GitLab is not currently built for s390x. When it is ported to the Linux on Z platform, it can be added to the installer.
It would also be possible to build from source instead, but there are a few problems with that. First, the source install process is difficult and prone to errors. It is actually much more difficult to build and debug a completely source install, then to correctly configure the omnibus installer. In addition, our goal is to build a High Availability implementation of GitLab. That will require multiple instances of the GitLab Rails front end to be installed on multiple machines. This would require multiple source installs, instead of doing one omnibus package creation, and simply installing the package on the desired machines. Finally, the omnibus install adds an extra layer of abstraction to the management of GitLab, using chef scripts to perform tasks through the 'gitlab-ctl' command. When installing from source, you lose that, and have to use more direct rake commands to manage the cluster. Overall, it is easier in the long run to take the time to build the RPM or Deb package.
Creating the Omnibus Environment
The first thing we need to do is build an instance to be our omnibus builder. It is best to have this environment specifically for these omnibus installs, so you can reuse it for rebuilding the omnibus packages for any upgrades. You can also run into problems later on in the build if there are extra packages installed. The Omnibus installer will only build the package for the system on which it is installed. That means it will build a deb on a Debian system and an RPM on a Red Hat based system. For this tutorial we are using Ubuntu, but the same general process applies to RHEL builds. Any major differences between the two will be outlined as we go along. Ubuntu has has released Ubuntu 16, Xenial LTS, for Z, which means there is a familiar and supported Debian Linux on Z version to use. In addition, Ubuntu's package repositories currently have some pre-compiled packages, like go1.6, that would currently need to be compiled manually on RHEL7.
Installing Prerequisites
There are a number of prerequisites that need to be installed. For the sake of simplicity, they will all be listed together, and I will revisit some for clarity afterwards. 'sudo apt-get install ...'
- curl
- wget
- golang
- chef
- git
- ruby
Chef Special instructions
The Chef-client install on Ubuntu would not complete successfully until I added /usr/sbin, /usr/local/sbin and /sbin to the PATH. This can with the command:
export PATH="$PATH":/usr/local/sbin:/usr/sbin:/sbin
That statement should also be added to the ~/.bashrc file for persistence across restarts and logouts.
Additionally, the Chef install on Ubuntu starts a configuration setup screen. You can use all the default values for this particular use case.
RHEL specific prerequisites
On RHEL 7.1, the Chef client and the golang 1.5+ packages do not currently exist in the the RHEL 7 repository, so you will need to do source build for both of them.
Chef-Client Instructions: https://github.com/linux-on-ibm-z/docs/wiki/Building-Chef-Client
Golang instructions: https://github.com/linux-on-ibm-z/docs/wiki/Building-Go
You also need to install the 'fakeroot' rpm with yum package manager.
Creating the Omnibus Build User and Environment
The first major step in building the OmniBus installer is creating an install user and environment according to the GitLab documentation. This process is used internally by the GitLab team to build the OmniBus Packages that they distributed, and we will be reusing to build our own. However, we will implement and kick off the build manually, while they have automated the process using Chef and GitLab Continuous Integration(CI). The first thing to do is to build the build user and environment using a modified version of the GitLab teams instructions listed below.
The default documentation can be found here: https://gitlab.com/gitlab-org/gitlab-omnibus-builder/blob/master/README.md#recipe-default
set -e
set -u
# Download the gitlab-omnibus-builder cookbook using Git and run its 'default' recipe
chef_root=/tmp/gitlab-omnibus-builder.$$
mkdir "$chef_root"
cd "$chef_root"
mkdir cookbooks
git clone https://gitlab.com/gitlab-org/gitlab-omnibus-builder.gitcookbooks/gitlab-omnibus-builder
git clone https://gitlab.com/gitlab-org/gitlab-attributes-with-secrets.git cookbooks/gitlab-attributes-with-secrets
sudo –E /usr/bin/chef-client -z -r 'recipe[gitlab-omnibus-builder::default]'
That series of commands will download and then kick off a Chef Script that should build our Omnibus build environment automatically. This process may take a few minutes. The user that is created for the build is called 'gitlab-ci-multi-runner'. To verify that it installed and ran correctly, switch to the newly created user:
sudo su - gitlab_ci_multi_runner
If you can login as that user, the environment creation was successful. For now, exit back to the standard user, as we will need to use sudo and gitlab_ci_multi_runner does not have sudo access.
exit
Building the Omnibus Package
Now that we have the build environment ready to go, it is time to start the omnibus build process. The base tutorial for this process is given in the GitLab documentation here: https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/doc/build/README.md
Using a Linux on Z system, we will have to take some extra steps to get the build process to work. Remember. the Omnibus install process works by downloading all the GitLab prerequisites, compiling them and bundling them into 1 package without dependencies. In the Omnibus builder, some of those downloads will list broken links, and sometimes the packages will be binaries compiled specifically for x86. For those compatibility issues, we will make the necessary changes to complete the build.
Special Go Note
After running the Chef script that created the build environment in the previous step, x86 golang binaries were installed in /usr/local/bin. You must delete the go binaries in /usr/local/bin and symlink /usr/local/bin/go to back to /usr/bin/go where dpkg installs s390x golang. You cannot use the gitlab_ci_multirunner to run these commands because it has no sudo access.
sudo rm /usr/local/bin/go
sudo rm /usr/local/bin/gofmt
sudo ln -s /usr/bin/go /usr/local/bin/go
sudo ln -s /usr/bin/gofmt /usr/local/bin/gofmt
Cloning the GitLab Omnibus Builder Code
To start, we follow the official documentation, setting some environment variables and getting the GitLab Omnibus builder code:
# Ubuntu/Debian only: ensure you have proper locale available
sudo locale-gen en_US.UTF-8
# Login as build user
sudo su - gitlab_ci_multi_runner
# Set git author
git config --global user.email "email@example.com"
git config --global user.name "Example name"
# Ensure you have proper locale in the environment, or build will fail
export LC_ALL="en_US.UTF-8"
# Clone the default omnibus repo
git clone https://gitlab.com/gitlab-org/omnibus-gitlab.git
This is where our process starts to deviate slightly. Next we will:
#Enter the omnibus directory
cd omnibus-gitlab
#Create a directory for the build files for the project, which is set in #omnibus.rb as the bas_dir
mkdir –p /home/gitlab_ci_multi_runner/omnibus-build
#Checkout the most recent stable release of the GitLab project(rc is not stable)
git checkout 8-7-stable
git checkout –b 8-7-stable
#Create the omnibus build settings file from the example
mv omnibus.rb.example omnibus.rb
Creating an omnibus.rb settings file
The omnibus.rb file is the settings for the build process. The content for this file is as follows:
#Omnibus.rb
base_dir '/home/gitlab_ci_multi_runner/omnibus-build'
use_git_caching false
Base_dir sets the output directory, as the build process can take up a bunch of space
use_git_caching is set to false to make sure we have a 100% fresh build every time, at the expense of the build speed. This is better for this use case because we are not doing frequent builds.
Fixing NodeJS
Now we will make some custom changes to some of the Omnibus prerequisite install scripts. Omnibus Builder uses ruby build scripts in the directory 'omnibus-gitlab/config/software/' to download and build GitLab software components. In the cases where those software components do not work, we can create or modify the build scripts in that directory to correct the process.
First, we will create a file called omnibus-gitlab/config/software/nodejs.rb:
touch config/software/nodejs.rb
vi config/software/nodejs.rb
The contents of the nodejs.rb should be set as follows:
#This is the content of the file nodejs.rb
name "nodejs"
default_version "0.10.36"
dependency "python"
version "0.10.36" do
source md5: "02de00cb56c976f71a5a9eb693511fe7"
end
source url: "http://github.com/andrewlow/node/archive/V8-3.14.5.9-Node.js-#{version}-201501281023.tar.gz"
relative_path "node-8-3.14.5.9-Node.js-#{version}-201501281023"
build do
env = with_standard_compiler_flags(with_embedded_path)
env["LINK"]="g++"
command "#{install_dir}/embedded/bin/python ./configure --dest-cpu=s390x --prefix=#{install_dir}/embedded", env: env
make "-j #{workers}", env: env
make "install", env: env
end
We have to do this because the default version of NodeJS is an x86 binary that will not work, so we download a version that is made to work with Linux on Z.
Note: Copying and pasting text from this document into the GitLab configuration files can cause errors if your system does not use UTF-8 encoding.
Fixing GitLab Repository Links
Generally, the GitLab RPM/deb package is built by the GitLab development team. As a result, the GitLab Omnibus builder points to internal GitLab repositories for the actual GitLab code. We have to change a few configuration files to point the publically accessible GitLab repositories. We will start with ‘version.rb’, where we will change every instance of the ‘dev@gitlab’ git path to the publically accessible http url.
vi lib/gitlab/version.rb
#This is the updated version file.
module Gitlab
class Version
def initialize(filename)
@filename = filename
filepath = File.expand_path(@filename, Omnibus::Config.project_root)
@read_version = File.read(filepath).chomp
rescue Errno::ENOENT
# Didn't find the file
@read_version = ""
end
def print
if @read_version.include?('.pre') || @read_version == "master"
"master"
elsif @read_version.empty?
nil
else
"v#{@read_version}"
end
end
def remote
case @filename
when "VERSION"
if @read_version.include?('-ee')
"git@dev.gitlab.org:gitlab/gitlab-ee.git"
else
"https://gitlab.com/gitlab-org/gitlab-ce.git"
end
when "GITLAB_SHELL_VERSION"
"https://gitlab.com/gitlab-org/gitlab-shell.git"
when "GITLAB_WORKHORSE_VERSION"
"https://gitlab.com/gitlab-org/gitlab-workhorse.git"
when "GITLAB_PAGES_VERSION"
"https://gitlab.com/gitlab-org/gitlab-pages.git"
else
nil
end
end
end
end
The highlighted area is where all the changes occur. Fixing that one file should take care of all the issues with internal GitLab links.
Fixing Ncurses
There is one package that can cause significant issues, not just on the Linux on Z platform, but any Omnibus build. That package is 'ncurses' and the issue is caused by a compilation error caused when using gcc 5 or above. On the Ubuntu 16.04 LTS install, the default gcc is a 5.XX install, so we will have to do a small workaround to get the install to work. The process to fix the issue is similar to that of nodejs.rb, where we add the modified ncurses.rb configuration file to the ‘omnibus-gitlab/config/software’ directory.
Special RHEL Note
If you are on RHEL 7.1 the default gcc version should be below 4.XX, and you can skip the ncurses step.
I have uploaded a working version of the ncurses.rb file that you can download directly from your omnibus build machine.
Updated ncurses.rb file:
https://raw.githubusercontent.com/rwbertsc/files/master/ncurses.rb
#Download the updated ncurses
wget https://raw.githubusercontent.com/rwbertsc/files/master/ncurses.rb
#Move ncurses to correct directory
mv ncurses.rb omnibus-gitlab/config/software/ncurses.rb
The original, unmodified version of the ncurses.rb file can be found in the omnibus-software repository. This repository can be found at https://github.com/chef/omnibus-software
This is where the Omnibus builder gets the needed software configuration files that are not in the local omnibus-gitab/config/software folder. To manually fix issues with software builds, you can clone the omnibus-software repository, and copy the ruby install file from omnibus-software to omnibus-gitlab/config/software. You can then modify the file to fix the install, and the Omnibus builder will use the local version of the software ruby file.
Kicking off the Build
Now that we have fixed all the issues, we are ready to continue with the rest of the build.
# Make sure you are in the root of the omnibus-gitlab project
cd omnibus-gitlab
# Install gem dependencies for omnibus-ruby
bundle install --path .bundle --binstubs
# Run the build command
bin/omnibus build gitlab
Package Download Workaround
There is an issue in the beginning of the omnibus build process where some software packages will fail to download. This happens with about 5 or 6 packages, and the build will fail within the first couple of minutes. However, there is a very simple workaround to the issue. Here is an example of an error I got, and how I fixed it.
[NetFetcher: libffi] I | Downloading from `ftp://sourceware.org/pub/libffi/libffi-3.2.1.tar.gz'
[NetFetcher: libffi] E | Download failed - OpenURI::HTTPError! 0% (0 KB/sec)
/usr/local/lib/ruby/2.1.0/open-uri.rb:353:in `open_http': 503 Service Unavailable (OpenURI::HTTPError)
from /usr/local/lib/ruby/2.1.0/open-uri.rb:733:in `buffer_open'
from /usr/local/lib/ruby/2.1.0/open-uri.rb:210:in `block in open_loop'
from /usr/local/lib/ruby/2.1.0/open-uri.rb:208:in `catch'
from /usr/local/lib/ruby/2.1.0/open-uri.rb:208:in `open_loop'
from /usr/local/lib/ruby/2.1.0/open-uri.rb:149:in `open_uri'
from /home/gitlab_ci_multi_runner/omnibus-gitlab-loz/.bundle/ruby/2.1.0/bundler/gems/omnibus-be6630690f17/lib/omnibus/core_extensions/open_uri.rb:51:in `open_uri'
from /usr/local/lib/ruby/2.1.0/open-uri.rb:704:in `open'
from /usr/local/lib/ruby/2.1.0/open-uri.rb:34:in `open'
from /home/gitlab_ci_multi_runner/omnibus-gitlab-loz/.bundle/ruby/2.1.0/bundler/gems/omnibus-be6630690f17/lib/omnibus/fetchers/net_fetcher.rb:184:in `download'
from /home/gitlab_ci_multi_runner/omnibus-gitlab-loz/.bundle/ruby/2.1.0/bundler/gems/omnibus-be6630690f17/lib/omnibus/fetchers/net_fetcher.rb:82:in `fetch'
from /home/gitlab_ci_multi_runner/omnibus-gitlab-loz/.bundle/ruby/2.1.0/bundler/gems/omnibus-be6630690f17/lib/omnibus/software.rb:644:in `fetch'
from /home/gitlab_ci_multi_runner/omnibus-gitlab-loz/.bundle/ruby/2.1.0/bundler/gems/omnibus-be6630690f17/lib/omnibus/project.rb:996:in `block (3 levels) in download'
from /home/gitlab_ci_multi_runner/omnibus-gitlab-loz/.bundle/ruby/2.1.0/bundler/gems/omnibus-be6630690f17/lib/omnibus/thread_pool.rb:64:in `call'
from /home/gitlab_ci_multi_runner/omnibus-gitlab-loz/.bundle/ruby/2.1.0/bundler/gems/omnibus-be6630690f17/lib/omnibus/thread_pool.rb:64:in `block (4 levels) in initialize'
from /home/gitlab_ci_multi_runner/omnibus-gitlab-loz/.bundle/ruby/2.1.0/bundler/gems/omnibus-be6630690f17/lib/omnibus/thread_pool.rb:62:in `loop'
from /home/gitlab_ci_multi_runner/omnibus-gitlab-loz/.bundle/ruby/2.1.0/bundler/gems/omnibus-be6630690f17/lib/omnibus/thread_pool.rb:62:in `block (3 levels) in initialize'
from /home/gitlab_ci_multi_runner/omnibus-gitlab-loz/.bundle/ruby/2.1.0/bundler/gems/omnibus-be6630690f17/lib/omnibus/thread_pool.rb:61:in `catch'
from /home/gitlab_ci_multi_runner/omnibus-gitlab-loz/.bundle/ruby/2.1.0/bundler/gems/omnibus-be6630690f17/lib/omnibus/thread_pool.rb:61:in `block (2 levels) in initialize'
As you can see, libffi failed to download correctly. The simplest way to deal with this problem is to use wget to download the package from the url (highlighted in the error), and move it into the ‘cache’ subdirectory of our build directory. This is the build directory that we specified in the omnibus.rb file. This is where the Omnibus builder looks for the packages, and if it already exists, in cache, it will skip the download.
wget ftp://sourceware.org/pub/libffi/libffi-3.2.1.tar.gz
mv libffi-3.2.1.tar.gz /home/gitlab_ci_multi_runner/omnibus-build/cache
This technique works for all the package that may experience this error. It may take a few build attempts to find all the packages that give download errors, but downloading packages is the first step of the build, so it will fail quickly. Each time the build fails, you can restart the process with the same command as you start it with:
# Kick off the build after a failure
bin/omnibus build gitlab
Compilation
After the packages are downloaded Omnibus will start compiling and building all the individual pieces of software that were just downloaded. Make sure you have at least 2GB between RAM and swap space, so the compiler doesn’t crash. It will most likely take 20 – 40 minutes for the build to complete, depending on the speed of your build machine.
Dealing with the Health Checker
At the end of the build process, there will be Health Checker that runs to verify that all the packages are built correctly, and check for unmet dependencies. One possible problem you may run into is the existence of shared libraries that are used in the Omnibus build process that are not part of the Omnibus package. This will throw an error because there are now dependencies in the package that are not guaranteed to be available when you install the Omnibus GitLab RPM.
If you start this build with a fresh Ubuntu instance and follow this guide, you should not have to deal with the Health Checker issue at all.
For example, I failed the Health Check on my first build with the following error:
[Project: gitlab-ce] I | Building version manifest
[HealthCheck] I | Running health on gitlab-ce
[HealthCheck] E | Failed!
[HealthCheck] E | The following libraries have unsafe or unmet dependencies:
--> /opt/gitlab/embedded/lib/python2.7/lib-dynload/_sqlite3.so
[HealthCheck] E | The following binaries have unsafe or unmet dependencies:
[HealthCheck] E | The following libraries cannot be guaranteed to be on target systems:
--> /usr/lib/s390x-linux-gnu/libsqlite3.so.0 (0x000003ff9ed80000)
[HealthCheck] E | The precise failures were:
--> /opt/gitlab/embedded/lib/python2.7/lib-dynload/_sqlite3.so
DEPENDS ON: libsqlite3.so.0
COUNT: 1
PROVIDED BY: /usr/lib/s390x-linux-gnu/libsqlite3.so.0 (0x000003ff9ed80000)
FAILED BECAUSE: Unsafe dependency
The easiest way to resolve this issue is to remove the packages that cause the Health Check error. In this case it was libsql3-dev, which I found by running some dpkg commands to locate the package that provided the library. First you must exit from the ‘gitlab_ci_multi_runner’ user, back to the standard user, because will need to use sudo.
sudo dpkg -S */libsqlite3.so.0
|---->libsqlite3-0:s390x: /usr/lib/s390x-linux-gnu/libsqlite3.so.0
It is usually the development software packages, (end in ‘-dev’ or ‘-devel’) that cause the issue because of the shared libraries they install. Now that I knew the package that caused the problem, I did a search to see if the dev version was installed.
sudo dpkg --get-selections | grep sqlite
|---->libsqlite3-0:s390x install
|---->libsqlite3-dev install
|---->ruby-sqlite3 install
I then uninstalled the dev version of the libsqlite3 package:
sudo apt-get remove libsqlite3-dev
I then switched back to the ‘gitlab_ci_multi_runner’ user, switched to the GitLab directory and restarted the build.
#Switch back to correct user
sudo su - gitlab_ci_multi_runner
#Switch to the omnibus directory
cd gitlab-omnibus
#Restart the build process
bin/omnibus build gitlab
Getting your Package
Once the installation is complete, the deb file (or RPM if you are using RHEL) is available at omnibus-gitlab/pkg/gitlab-ce_8.7.5+{some_build_number}.deb
You have now created a package that you can distribute to multiple systems, and use to easily install the entire GitLab package. At the most recent build, the build package needs about 850MB of free space to install, because of all the components involved in the install.
You can install the GitLab deb file using the command:
dpkg –i gitlab-ce_8.7.5+{some_build_number}.deb
Once installed, GitLab can be started as a standalone service, with all the components running locally with the command:
gitlab-ctl reconfigure
What’s Next?
This is the very first step in the installation process. You could run a small instance of GitLab using only this single deb package. All the required components are there, but our end goal is to create a high availability instance of GitLab. The next step in that process will be to building highly available PostgreSQL instance.