GitLab

From ArchWiki

From GitLab's homepage:

GitLab offers git repository management, code reviews, issue tracking, activity feeds and wikis. Enterprises install GitLab on-premise and connect it with LDAP and Active Directory servers for secure authentication and authorization. A single GitLab server can handle more than 25,000 users but it is also possible to create a high availability setup with multiple active servers.

An example live version can be found at GitLab.com.

Installation

GitLab requires Redis and a database backend. If you plan to run it on the same machine, first install PostgreSQL.

Install the gitlab package.

Finally, a web server has to be installed and configured. The configuration for GitLab will be discussed in the #Web server configuration section.

Configuration

Preliminary notes

GitLab is composed of multiple components, see the architecture overview page.

The gitlab package installs GitLab's files in a manner that more closely follow standard Linux conventions:

Description GitLab's Official gitlab
Application Code /home/git /usr/share/webapps/gitlab
Application Data /home/git /var/lib/gitlab
User (Home Directory) git (/home/git) gitlab (/var/lib/gitlab)
Configuration File GitShell /home/git/gitlab-shell/config.yml /etc/webapps/gitlab-shell/config.yml
Configuration File GitLab /home/git/gitlab/config/gitlab.yml /etc/webapps/gitlab/gitlab.yml
Logs /home/git/log /var/log/gitlab
Unix socket files / PID files /home/git/sockets /run/gitlab
Tip: If you are familiar with the Arch Build System you can edit the PKGBUILD and relevant files to change gitlab's home directory to a place of your liking.

GitLab

Edit /etc/webapps/gitlab/gitlab.yml and setup at least the following parameters:

Note: The hostname and port are used for the git clone http://hostname:port as example.

Hostname: In the gitlab: section set host: - replacing localhost to yourdomain.com (no http:// or trailing slash) - into your fully qualified domain name.

Port: port: can be confusing. This is not the port that the GitLab server (Puma) runs on; it is the port that users will initially access through in their browser. Basically, if you intend for users to visit yourdomain.com in their browser, without appending a port number to the domain name, leave port: as 80. If you intend your users to type something like yourdomain.com:3425 into their browsers, then you would set port: to 3425. You will also have to configure your webserver to listen on that port.

Timezone (optional): The time_zone: parameter is optional, but may be useful to force the zone of GitLab applications.

Based on the table in #Preliminary notes above, the following paths have to be configured in gitlab.yml:

  • repository_downloads_path: "/var/lib/gitlab/shared/cache/archive/"
  • gitlab_ci section: builds_path: "/var/lib/gitlab/builds/"
  • incoming_email section (if enabled): log_path: "/var/log/gitlab/mail_room_json.log"
  • artifacts section (if enabled): path: "/var/lib/gitlab/shared/artifacts"
  • external_diffs section (if enabled): storage_path: "/var/lib/gitlab/shared/external-diffs"
  • lfs section (if enabled): storage_path: "/var/lib/gitlab/shared/lfs-objects"
  • packages section (if enabled): storage_path: "/var/lib/gitlab/shared/packages"
  • dependency_proxy section (if enabled): storage_path: "/var/lib/gitlab/shared/dependency_proxy"
  • terraform_state section (if enabled): storage_path: "/var/lib/gitlab/shared/terraform_state"
  • pages section (if enabled): path: "/var/lib/gitlab/shared/pages"
  • registry section (if enabled): path: "/var/lib/gitlab/shared/registry"

Custom port for Puma

GitLab Puma is the main component which processes most of the user requests. By default, it listens on the /run/gitlab/gitlab.socket UNIX socket which can be changed in the /etc/webapps/gitlab/puma.rb file.

To configure Puma to listen on a TCP port as well as UNIX socket:

/etc/webapps/gitlab/puma.rb
bind 'unix:///run/gitlab/gitlab.socket'
bind 'tcp://127.0.0.1:8080'

If the Puma address is changed, the configuration of other components which communicate with Puma have to be updated as well:

  • For GitLab Shell, update the gitlab_url variable in /etc/webapps/gitlab-shell/config.yml and url in the [gitlab] section in /etc/gitlab-gitaly/config.toml.
Tip: UNIX socket path can be specified with URL-escaped slashes (i.e. http+unix://%2Frun%2Fgitlab%2Fgitlab.socket for the default /run/gitlab/gitlab.socket).
  • For GitLab Workhorse, edit the gitlab-workhorse.service and update the -authBackend option. See [1] for details.

Secret strings

Make sure that the files /etc/webapps/gitlab/secret and /etc/webapps/gitlab-shell/secret files contain something. Their content should be kept secret because they are used for the generation of authentication tokens etc.

For example, random strings can be generated with the following commands:

# hexdump -v -n 64 -e '1/1 "%02x"' /dev/urandom > /etc/webapps/gitlab/secret
# chmod 640 /etc/webapps/gitlab/secret
# hexdump -v -n 64 -e '1/1 "%02x"' /dev/urandom > /etc/webapps/gitlab-shell/secret
# chmod 640 /etc/webapps/gitlab-shell/secret

Also fill in (new) secret strings for secrets.yml:

/etc/webapps/gitlab/secrets.yml
production:
  secret_key_base: secret
  db_key_base: secret
  otp_key_base: secret
  openid_connect_signing_key: secret

Redis

In order to provide sufficient performance you will need a cache database. Install and configure a Redis instance, being careful to the section dedicated to listening via a socket.

Add the gitlab user to the redis user group and update this configuration file:

/etc/webapps/gitlab/resque.yml
development:
  url: unix:/run/redis/redis.sock
test:
  url: unix:/run/redis/redis.sock
production:
  url: unix:/run/redis/redis.sock

PostgreSQL database

A PostgreSQL database will be required before Gitlab can be run.

Login to PostgreSQL and create the gitlabhq_production database along with its user. Remember to change your_username_here and your_password_here to the real values:

# psql -d template1
template1=# CREATE USER your_username_here WITH PASSWORD 'your_password_here';
template1=# ALTER USER your_username_here SUPERUSER;
template1=# CREATE DATABASE gitlabhq_production OWNER your_username_here;
template1=# \q
Note: The reason for creating the user as a superuser is that GitLab is trying to be "smart" and install extensions (not just create them in its own userspace). And this is only allowed by superusers in Postgresql.

Try connecting to the new database with the new user to verify it works:

# psql -d gitlabhq_production

Open the new /etc/webapps/gitlab/database.yml and set the values for username: and password:. For example:

/etc/webapps/gitlab/database.yml
#
# PRODUCTION
#
production:
  adapter: postgresql
  encoding: unicode
  database: gitlabhq_production
  pool: 10
  username: your_username_here
  password: "your_password_here"
  # host: localhost
  # port: 5432
  socket: /run/postgresql/.s.PGSQL.5432
...

For our purposes (unless you know what you are doing), you do not need to worry about configuring the other databases listed in /etc/webapps/gitlab/database.yml. We only need to set up the production database to get GitLab working.

Initialize Gitlab database

Start the Redis server and the gitlab-gitaly.service before initializing the database.

Initialize the database and activate advanced features:

$ cd /usr/share/webapps/gitlab
$ sudo -u gitlab $(cat environment | xargs) bundle-2.7 exec rake gitlab:setup

Finally run the following commands to check your installation:

$ sudo -u gitlab $(cat environment | xargs) bundle-2.7 exec rake gitlab:env:info
$ sudo -u gitlab $(cat environment | xargs) bundle-2.7 exec rake gitlab:check
Note:
  • The gitlab:env:info and gitlab:check commands might show a fatal error related to git. This is OK.
  • The gitlab:check will complain about missing initscripts. This is nothing to worry about, as systemd service files are used instead (which GitLab does not recognize).

Adjust modifier bits

(The gitlab check will not pass if the user and group ownership is not configured properly)

# chmod -R ug+rwX,o-rwx /var/lib/gitlab/repositories/
# chmod -R ug-s /var/lib/gitlab/repositories
# find /var/lib/gitlab/repositories/ -type d -print0 | xargs -0 chmod g+s

Web server configuration

To access GitLab from an outside network, the upstream documentation recommends to use an established web server as a proxy. All queries from the web server to GitLab are processed by GitLab Workhorse, which decides how they should be processed. See [2] for details.

Nginx

See Nginx#Configuration for basic nginx configuration and Nginx#TLS for enabling HTTPS. The sample in this section also assumes that server blocks are managed with Nginx#Managing server entries.

Create and edit the configuration based on the following snippet. See the upstream GitLab repository for more examples.

/etc/nginx/sites-available/gitlab
upstream gitlab-workhorse {
  server unix:/run/gitlab/gitlab-workhorse.socket fail_timeout=0;
}

server {
  listen 80;                  # IPv4 HTTP
  #listen 443 ssl http2;      # uncomment to enable IPv4 HTTPS + HTTP/2
  #listen [::]:80;            # uncomment to enable IPv6 HTTP
  #listen [::]:443 ssl http2; # uncomment to enable IPv6 HTTPS + HTTP/2
  server_name example.com;

  access_log  /var/log/gitlab/nginx_access.log;
  error_log   /var/log/gitlab/nginx_error.log;

  #ssl_certificate ssl/example.com.crt;
  #ssl_certificate_key ssl/example.com.key;

  location ~ ^/(assets)/ {
    root /usr/share/webapps/gitlab/public;
    gzip_static on; # to serve pre-gzipped version
    expires max;
    add_header Cache-Control public;
  }

  location / {
      # unlimited upload size in nginx (so the setting in GitLab applies)
      client_max_body_size 0;

      # proxy timeout should match the timeout value set in /etc/webapps/gitlab/puma.rb
      proxy_read_timeout 60;
      proxy_connect_timeout 60;
      proxy_redirect off;

      proxy_set_header Host $http_host;
      proxy_set_header X-Real-IP $remote_addr;
      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
      proxy_set_header X-Forwarded-Proto $scheme;

      #proxy_set_header X-Forwarded-Ssl on;

      proxy_pass http://gitlab-workhorse;
  }

  error_page 404 /404.html;
  error_page 422 /422.html;
  error_page 500 /500.html;
  error_page 502 /502.html;
  error_page 503 /503.html;
  location ~ ^/(404|422|500|502|503)\.html$ {
    root /usr/share/webapps/gitlab/public;
    internal;
  }
}

Apache

Install and configure the Apache HTTP Server and Apache HTTP Server#TLS for enabling HTTPS. You can use these upstream recipes to get started with the configuration file for GitLab's virtual host.

Notice that the SSL virtual host needs a specific IP instead of generic. Also if you set a custom port for Puma, do not forget to set it at the BalanceMember line.

A working example with certbot flavor assuming that gitlab-workhorse.service contains -listenNetwork tcp -listenAddr 127.0.0.8181 on the ExecStart line:

/etc/httpd/conf/extra/gitlab.conf
<VirtualHost *:443>
    ServerName SERVERNAME
    ServerAlias SERVERNAME
    DocumentRoot /usr/share/webapps/gitlab/public
    <Directory /usr/share/webapps/gitlab/public>
        Options FollowSymlinks
        AllowOverride all
        Require all granted
    </Directory>
    <IfModule mod_alias.c>
        Alias / /usr/share/webapps/gitlab/public/
    </IfModule>

    <IfModule mod_headers.c>
        Header always set Strict-Transport-Security "max-age=15552000; includeSubDomains; preload"
    </IfModule>

    ProxyPreserveHost On
    SSLProxyEngine on
    AllowEncodedSlashes NoDecode
    <Location />
        Require all granted
        ProxyPassReverse http://127.0.0.1:8181
        ProxyPassReverse http://SERVERNAME
    </Location>
    #
    RewriteEngine on
    RewriteCond %{DOCUMENT_ROOT}/%{REQUEST_FILENAME} !-f [OR]
    RewriteCond %{REQUEST_URI} ^/uploads/.*
    RewriteRule .* http://127.0.0.1:8181%{REQUEST_URI} [P,QSA,NE]
    RequestHeader set X_FORWARDED_PROTO 'https'
    RequestHeader set X-Forwarded-Ssl on
    #
    ErrorDocument 404 /404.html
    ErrorDocument 422 /422.html
    ErrorDocument 500 /500.html
    ErrorDocument 503 /deploy.html
    ErrorLog /var/log/httpd/gitlab.lan.info-error_log
    CustomLog /var/log/httpd/gitlab.lan.info-access_log common
Include /etc/letsencrypt/options-ssl-apache.conf
SSLCertificateFile /etc/letsencrypt/live/SERVERNAME/fullchain.pem
SSLCertificateKeyFile /etc/letsencrypt/live/SERVERNAME/privkey.pem
</VirtualHost>

<VirtualHost *:80>
    ServerName SERVERNAME
    Redirect / https://SERVERNAME
RewriteEngine on
RewriteCond %{SERVER_NAME} =SERVERNAME
RewriteRule ^ https://%{SERVER_NAME}%{REQUEST_URI} [END,NE,R=permanent]
</VirtualHost>

Node.js

You can easily set up an HTTPS proxy on port 443 to proxy traffic to the GitLab Workhorse using http-master for Node.js. http-master is built on top of node-http-proxy.

HTTPS/SSL

Change GitLab configs

Modify /etc/webapps/gitlab/shell.yml so the URL to your GitLab site starts with https://. Modify /etc/webapps/gitlab/gitlab.yml so that https: setting is set to true.

Let's Encrypt

To validate your URL, the Let's Encrypt process will try to access your GitLab server via a URL such as your.domain.name/.well-known/acme-challenge/a_long_id. Hence, you need to make sure that requests to the .well-known subdirectory are not proxied to GitLab Workhorse. This can be done easily with the Certbot's "webroot" method, see Certbot#Webroot for details.

Firewall

If you want to give direct access to your Gitlab installation through an iptables firewall, you may need to adjust the port and the network address:

# iptables -A tcp_inbound -p TCP -s 192.168.1.0/24 --destination-port 80 -j ACCEPT

If you are behind a router, do not forget to forward this port to the running GitLab server host, if you want to allow WAN access.

Note: Allowing access to GitLab from hosts other than localhost is not recommended before completing the installation and setting the root password. See #Start and test GitLab.

Start and test GitLab

Make sure PostgreSQL and Redis are running and setup correctly.

Then start/enable gitlab.target.

Now test your GitLab instance by visiting http://localhost or http://localhost:port, where port is the port number on which your web server listens. You should be prompted to create a password:

username: root
password: You will be prompted to create one on your first visit.

See #Troubleshooting and log files inside the /var/log/gitlab/ directory for troubleshooting.

Upgrade database on updates

Manual method

After updating the gitlab package, it is required to upgrade the database:

$ cd /usr/share/webapps/gitlab
$ sudo -u gitlab $(cat environment | xargs) bundle-2.7 exec rake db:migrate

Afterwards, reload and restart gitlab-sidekiq.service, gitlab-puma.service, gitlab-workhorse.service and gitlab-gitaly.service.

Automatic method

You can create pacman hooks to automate database upgrades on gitlab package updates. Create the three following files, do not forget to set the executable flag on the shell script:

/etc/pacman.d/hooks/gitlab-pre.hook
[Trigger]
Operation = Upgrade
Type = Package
Target = gitlab

[Action]
Description = Stopping gitlab services
When = PreTransaction
Exec = /usr/bin/systemctl stop gitlab-gitaly.service gitlab-mailroom.service gitlab-puma.service gitlab-sidekiq.service gitlab-workhorse.service
/etc/pacman.d/hooks/gitlab-post.hook
[Trigger]
Operation = Upgrade
Type = Package
Target = gitlab

[Action]
Description = Upgrading gitlab database and restarting services
When = PostTransaction
Exec = /etc/pacman.d/hooks/gitlab-post.sh
/etc/pacman.d/hooks/gitlab-post.sh
#!/bin/sh

(cd /usr/share/webapps/gitlab || exit ; su -s /bin/sh gitlab -c "$(cat environment | xargs) bundle-2.7 exec rake db:migrate")
systemctl start gitlab.target

Advanced configuration

Basic SSH

After completing the basic installation, set up SSH access for users. Configuration for OpenSSH is described below. Other SSH clients and servers will require different modifications.

For tips on adding user SSH keys, the process is well-documented on the GitLab website. You can check the administrator logs at /var/lib/gitlab/log/gitlab-shell.log to confirm user SSH keys are being submitted properly. Behind the scenes, GitLab adds these keys to its authorized_keys file in /var/lib/gitlab/.ssh/authorized_keys.

The common method of testing keys (e.g. ssh -T git@your_server) requires a bit of extra configuration to work correctly. The user configured in /etc/webapps/gitlab/gitlab.yml (by default gitlab) must be added to the server's sshd configuration file, in addition to a handful of other changes:

/etc/ssh/sshd_config
PubkeyAuthentication   yes
AuthorizedKeysFile     %h/.ssh/authorized_keys

If your /etc/ssh/sshd_config contains the AllowUsers option, then the gitlab user should be added to the list:

/etc/ssh/sshd_config
AllowUsers gitlab other users...

After updating the configuration file, restart the sshd.service.

Test user SSH keys (optionally add -v to see extra information):

$ ssh -T gitlab@your_server

Custom SSH connection

If you are running SSH on a non-standard port, you must change the GitLab user's SSH config:

/var/lib/gitlab/.ssh/config
host localhost      # Give your setup a name (here: override localhost)
user gitlab         # Your remote git user
port 2222           # Your port number
hostname 127.0.0.1; # Your server name or IP

You also need to change the corresponding options (e.g. ssh_user, ssh_host, admin_uri) in the /etc/webapps/gitlab/gitlab.yml file.

Sending emails from GitLab

GitLab can send emails either using a local mail transfer agent (via sendmail) or using SMTP.

To use sendmail, edit /etc/webapps/gitlab/smtp_settings.rb and comment out all lines. Then mail delivery should work without any further configuration in GitLab, assuming that the local mail transfer agent is configured properly.

To use SMTP, configure the options in smtp_settings.rb according to your mail server. For example, to send via Gmail:

/etc/webapps/gitlab/smtp_settings.rb
if Rails.env.production?
  Gitlab::Application.config.action_mailer.delivery_method = :smtp

  ActionMailer::Base.delivery_method = :smtp
  ActionMailer::Base.smtp_settings = {
    address:              'smtp.gmail.com',
    port:                 587,
    domain:               'gmail.com',
    user_name:            '[email protected]',
    password:             'application password',
    authentication:       'plain',
    enable_starttls_auto: true
  }
end
Note: Gmail will reject mails received this way (and send you a mail that it did). You will need to disable secure authentication (follow the link in the rejection mail) to work around this. The more secure approach is to enable two-factor authentication for [email protected] and to set up an application specific password for this configuration file.

Useful tips

Rake tasks

A number of setup/maintenance/etc tasks are available through rake. To list them, go to Gitlab's home directory:

$ cd /usr/share/webapps/gitlab

and run:

$ sudo -u gitlab $(cat environment | xargs) bundle-2.7 exec rake -T | grep gitlab
rake gitlab:app:check                         # GITLAB | Check the configuration of the GitLab Rails app
rake gitlab:backup:create                     # GITLAB | Create a backup of the GitLab system
rake gitlab:backup:restore                    # GITLAB | Restore a previously created backup
rake gitlab:check                             # GITLAB | Check the configuration of GitLab and its environment
rake gitlab:cleanup:block_removed_ldap_users  # GITLAB | Cleanup | Block users that have been removed in LDAP
rake gitlab:cleanup:dirs                      # GITLAB | Cleanup | Clean namespaces
rake gitlab:cleanup:repos                     # GITLAB | Cleanup | Clean repositories
rake gitlab:env:check                         # GITLAB | Check the configuration of the environment
rake gitlab:env:info                          # GITLAB | Show information about GitLab and its environment
rake gitlab:generate_docs                     # GITLAB | Generate sdocs for project
rake gitlab:gitlab_shell:check                # GITLAB | Check the configuration of GitLab Shell
rake gitlab:import:all_users_to_all_groups    # GITLAB | Add all users to all groups (admin users are added as owners)
rake gitlab:import:all_users_to_all_projects  # GITLAB | Add all users to all projects (admin users are added as masters)
rake gitlab:import:repos                      # GITLAB | Import bare repositories from gitlab_shell -> repos_path into GitLab project instance
rake gitlab:import:user_to_groups[email]      # GITLAB | Add a specific user to all groups (as a developer)
rake gitlab:import:user_to_projects[email]    # GITLAB | Add a specific user to all projects (as a developer)
rake gitlab:satellites:create                 # GITLAB | Create satellite repos
rake gitlab:setup                             # GITLAB | Setup production application
rake gitlab:shell:build_missing_projects      # GITLAB | Build missing projects
rake gitlab:shell:install[tag,repo]           # GITLAB | Install or upgrade gitlab-shell
rake gitlab:shell:setup                       # GITLAB | Setup gitlab-shell
rake gitlab:sidekiq:check                     # GITLAB | Check the configuration of Sidekiq
rake gitlab:test                              # GITLAB | Run all tests
rake gitlab:web_hook:add                      # GITLAB | Adds a web hook to the projects
rake gitlab:web_hook:list                     # GITLAB | List web hooks
rake gitlab:web_hook:rm                       # GITLAB | Remove a web hook from the projects
rake setup                                    # GITLAB | Setup gitlab db

Backup and restore

Create a backup of the gitlab system:

$ cd /usr/share/webapps/gitlab
$ sudo -u gitlab $(cat environment | xargs) bundle-2.7 exec rake gitlab:backup:create

Restore the previously created backup file /var/lib/gitlab/backups/1556571328_2019_04_29_11.10.2_gitlab_backup.tar:

$ cd /usr/share/webapps/gitlab
$ sudo -u gitlab $(cat environment | xargs) bundle-2.7 exec rake gitlab:backup:restore BACKUP=1556571328_2019_04_29_11.10.2
Note: Backup folder is set in config/gitlab.yml. GitLab backup and restore is documented here.

Enable fast SSH key lookup

Enable Fast SSH Key Lookup as explained in this page: https://docs.gitlab.com/ee/administration/operations/fast_ssh_key_lookup.html

In short, edit /etc/ssh/sshd_config.

Revert all changes done following this wiki (or revert sshd_config from the openssh package) and only add:

AuthorizedKeysCommand /var/lib/gitlab/gitlab-shell/bin/gitlab-shell-authorized-keys-check gitlab %u %k
AuthorizedKeysCommandUser gitlab

Finally restart the sshd.service.

Troubleshooting

HTTPS is not green (gravatar not using https)

Redis caches gravatar images, so if you have visited your GitLab with http, then enabled https, gravatar will load up the non-secure images. You can clear the cache by doing

$ cd /usr/share/webapps/gitlab
$ sudo -u gitlab $(cat environment | xargs) bundle-2.7 exec rake cache:clear

as the gitlab user.

Errors after updating

After updating the package from the AUR, the database migrations and asset updates will sometimes fail. These steps may resolve the issue, if a simple reboot does not.

First, move to the gitlab installation directory.

$ cd /usr/share/webapps/gitlab

If every gitlab page gives a 500 error, then the database migrations and the assets are probably stale. If not, skip this step.

$ sudo -u gitlab $(cat environment | xargs) bundle-2.7 exec rake db:migrate

If gitlab is constantly waiting for the deployment to finish, then the assets have probably not been recompiled.

$ sudo -u gitlab $(cat environment | xargs) bundle-2.7 exec rake gitlab:assets:clean gitlab:assets:compile cache:clear

Finally, restart gitlab-puma.service, gitlab-sidekiq.service and gitlab-workhorse.service.

GitLab Puma cannot access non-default repositories directory

If a custom repository storage directory is set in /home, disable the ProtectHome=true parameter in the gitlab-puma.service (see systemd#Drop-in files and the relevant forum thread on gitlab.com).

Failed to connect to Gitaly

Sometimes, the Gitaly service will not get started, leaving GitLab unable to connect to Gitaly. The solution is simple: start gitlab-gitaly.service.

Failed to connect via SSH

If git operations (-T, pull, clone, etc.) fails using ssh try changing the shell:

# usermod -s /usr/share/webapps/gitlab-shell/bin/gitlab-shell-ruby gitlab

CSS or styles issue

If you have any issues with styles and CSS not working, you may try to edit /usr/share/webapps/gitlab/config/environments/production.rb and change:

 # Disable Rails's static asset server (Apache or nginx will already do this)
 config.public_file_server.enabled = false

to:

 # Disable Rails's static asset server (Apache or nginx will already do this)
 config.public_file_server.enabled = true

The server does not support push options

If you get an error like fatal: the receiving end does not support push options you might need to enable it for the GitLab Git user (gitlab) on the server. This can be done in the gitconfig:

/etc/webapps/gitlab-shell/.gitconfig
[receive]
         advertisePushOptions = true

Alternatively one can set this with:

$ sudo -u gitlab -H git config --global receive.advertisePushOptions true

See also