Set up a GitLab Docker Registry on a dedicated host within a Docker environment. - Jie's Adventures: Crafting Robust and Reliable Services

In this blog post, I’ll demonstrate the installation of a GitLab Docker Registry on a separate host.

Installation Information for GitLab Docker Registry

install docker, docker-compose, LVM
leave /data/gitlab-registry as a separate device to the docker repository, better to leave at least 500MB disk space.
install the certificate by using “lets’ encrypt”
configure docker-compose file as the following

registry:
  image: registry:2
  restart: always
  expose:
    - "5000"
  ports:
    - "5000:5000"
  volumes:
    - /data/gitlab-registry:/var/lib/registry
    - /certs:/certs
  environment:
    REGISTRY_AUTH_TOKEN_REALM: https://gitlab.example.com/jwt/auth
    REGISTRY_AUTH_TOKEN_SERVICE: container_registry
    REGISTRY_AUTH_TOKEN_ISSUER: omnibus-gitlab-issuer
    REGISTRY_AUTH_TOKEN_ROOTCERTBUNDLE: /certs/gitlab-registry.crt
    REGISTRY_STORAGE_DELETE_ENABLED: "yes"

config nginx setting to host ssl and redirect traffic to docker registry

nginx
  upstream gitlab-registry {
    server 127.0.0.1:5000;
  }

  ## Set a variable to help us decide if we need to add the
  ## 'Docker-Distribution-Api-Version' header.
  ## The registry always sets this header.
  ## In the case of nginx performing auth, the header is unset
  ## since nginx is auth-ing before proxying.
  map $upstream_http_docker_distribution_api_version $docker_distribution_api_version {
    '' 'registry/2.0';
  }

  server {
    listen 443 ssl http2;
    server_name registry.example.com;

    # SSL
    ssl_certificate /etc/letsencrypt/live/registry.example.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/registry.example.com/privkey.pem;

    # Recommendations from https://raymii.org/s/tutorials/Strong_SSL_Security_On_nginx.html
    ssl_protocols TLSv1.2 TLSv1.3;
    ssl_ciphers 'EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH';
    ssl_prefer_server_ciphers on;
    ssl_session_cache shared:SSL:10m;

    # disable any limits to avoid HTTP 413 for large image uploads
    client_max_body_size 0;

    # required to avoid HTTP 411: see Issue #1486 (https://github.com/moby/moby/issues/1486)
    chunked_transfer_encoding on;

    location /v2/ {
      proxy_pass                          http://gitlab-registry;
      proxy_set_header  Host              $http_host;   # required for docker client's sake
      proxy_set_header  X-Real-IP         $remote_addr; # pass on real client's IP
      proxy_set_header  X-Forwarded-For   $proxy_add_x_forwarded_for;
      proxy_set_header  X-Forwarded-Proto $scheme;
      proxy_read_timeout                  900;


    }
  }

restart nginx.
```
  nginx -t
  systemctl restart nginx
```

Installation Information on GitLab

Add the following change to the /etc/gilab/gitlab.rb

registry['enable'] = false

# enable registry used by application
registry_external_url 'https://registry.example.com/'

gitlab_rails['registry_enabled'] = true
gitlab_rails['registry_host'] = "registry.example.com"
gitlab_rails['registry_port'] = "443"
gitlab_rails['registry_api_url'] = "https://registry.example.com"

put prod repo certificate to /etc/gitlab/ssl. I actually don’t quite understand why it needs cert to work. The prod repo cert should be recognised everywhere. The problem is if I didn’t do this step, the gitlab-ctl reconfigure will fail for cert error.

root@aecilius:/etc/gitlab/ssl# ls -l | grep registry
-rw-r--r-- 1 root root 5621 Mar 24 10:50 registry.example.com.crt
-rw-r--r-- 1 root root 1705 Mar 24 10:50 registry.example.com.key

copy gitlab-registry.crt from research GitLab to prod-repo.

The location is at /var/opt/gitlab/registry/gitlab-registry.crt on Research GitLab. Be mindful that this file is generated by GitLab, and it is required to use this file, not using self-generated one.

Copy this file to /certs at prod-repo.

root@prod-repo:/certs# ls -l
total 4
-rw-r--r-- 1 root root 1806 Mar 24 12:10 gitlab-registry.crt

stop docker

systemctl stop docker

start Burp Suite or any proxy server you like

put proxy cert at /root/.docker/cert.pem

put config.json as the following.

{
	"auths": {
		"https://index.docker.io/v1/": {},
		"registry.example.com:443": {}
	},
	"credsStore": "pass"
}

start docker with command line through HTTPS PROXY

HTTPS_PROXY=127.0.0.1:8080 dockerd --debug

try to login registry.example.com:443

docker login registry.example.com:443

You should be able to see 3 GET HTTP requests.

first one got http 401

GET /v2/ HTTP/1.1
Host: registry.example.com:443
User-Agent: docker/20.10.21 go/go1.18.1 git-commit/20.10.21-0ubuntu1~20.04.1 kernel/5.15.0-52-generic os/linux arch/amd64 UpstreamClient(Docker-Client/20.10.21 \(linux\))
Accept-Encoding: gzip, deflate
Connection: close

HTTP/1.1 401 Unauthorized
Server: nginx
Date: Fri, 31 Mar 2023 04:13:24 GMT
Content-Type: application/json; charset=utf-8
Content-Length: 87
Connection: close
Docker-Distribution-Api-Version: registry/2.0
Www-Authenticate: Bearer realm="https://gitlab.example.com/jwt/auth",service="container_registry"
X-Content-Type-Options: nosniff

{"errors":[{"code":"UNAUTHORIZED","message":"authentication required","detail":null}]}

second one send auth to gitlab and get a token

GET /jwt/auth?account=u6076069&client_id=docker&offline_token=true&service=container_registry HTTP/1.1
Host: gitlab.example.com
User-Agent: docker/20.10.21 go/go1.18.1 git-commit/20.10.21-0ubuntu1~20.04.1 kernel/5.15.0-52-generic os/linux arch/amd64 UpstreamClient(Docker-Client/20.10.21 \(linux\))
Authorization: Basic <your name and password>
Accept-Encoding: gzip, deflate
Connection: close

HTTP/1.1 200 OK
Server: nginx
Date: Fri, 31 Mar 2023 04:13:24 GMT
Content-Type: application/json; charset=utf-8
Connection: close
Vary: Accept-Encoding
Cache-Control: max-age=0, private, must-revalidate
Content-Security-Policy: 
Etag: W/"92bc78ea502ee7802dbdbe6032209615"
Permissions-Policy: interest-cohort=()
Pragma: no-cache
X-Content-Type-Options: nosniff
X-Download-Options: noopen
X-Frame-Options: SAMEORIGIN
X-Permitted-Cross-Domain-Policies: none
X-Request-Id: 01GWTXXH4W6D9KJYGXQDQVRHDK
X-Runtime: 0.118600
X-Ua-Compatible: IE=edge
X-Xss-Protection: 1; mode=block
Strict-Transport-Security: max-age=63072000
Referrer-Policy: strict-origin-when-cross-origin
Content-Length: 1107

{"token":"eyJhbGciOiJSUzI1NiIsImtpZCI6IkVWVkQ6Q1Y2QzpBSVBLOkZONkw6WFkzNjpLREk3OkIyN1M6UlpCWDpVT0NROkZYSDc6VVpDSjpOM0ZGIiwidHlwIjoiSldUIn0.eyJhdXRoX3R5cGUiOiJnaXRsYWJfb3JfbGRhcCIsImFjY2VzcyI6W10sImp0aSI6IjYzMzZjZTg4LWNkYjgtNGViNC1hMTI3LTJlNmQ2NDY0YWRlNCIsImF1ZCI6ImNvbnRhaW5lcl9yZWdpc3RyeSIsInN1YiI6InU2MDc2MDY5IiwiaXNzIjoib21uaWJ1cy1naXRsYWItaXNzdWVyIiwiaWF0ssdxNjgwMjM2MDA0LCJuYmYiOjE2ODAyMzU5OTksImV4cCI6MTY4MDIzNjMwNH0.X8Q6GONfF0HM46Gxd_n-gxYru1Etxjxku8dGj7hThVuEIXDWx6YYqvEDV6O6N-RLNMFNhO4QeejWHpseKow6UpaViDp87fmqVlfTYqswYSjrOyLHI-6ocH-e2OfBdsFMiqn_B5529QjDQ8wSwtf6CkpItKS4pUOsuR9Y73LZ_Dx3jT47EtqB5jPWf9ryvup5TaBMMsdK9CKzdGnFEhEUnIWIRgUIfN3OYiCrVnxoR2AQdSqbtwqeUI_SNPJdVqDYg2z23b-7WH3m_HuYT3VNPnBkjCbjpBCN8DkPJB1xdmQCsL7Kj2wfbJkupT_gB7Q5NYQz6u1BnY1rfHSUuHQYtC2GlW0nzD1RDeQey8IVNJgNQEyMxJAda044M0VeMiNMvgR-E7ZuqNipGH5he_Yxuowgelh9AiFFXNk3OuK9bX8G2ikxuvQpBEOagOFmdILMbRP1dkhy3kBYivZV5NwXL198sYBRgMvXs6nahgu8P93PoP-JFtAwxwAzTSeoJjBZTi-genzNHCj6aLjs5puYxv9FeAFp9RNVi-aRFCCUrQqxbDy0dnUh4sfY3DQlOkUxDf8hwFj6KC6aJI7um3kK7JgsbpJT9yg1Lr9fGr4gWCzlnm51U79jG093fRnrOxW3QkRkn2Iol5GRWsZ3K20DikExAv8o9japdAj_4uNJP3g"}

third one send valid token to prod-repo, and this time, it should return 200

GET /v2/ HTTP/1.1
Host: registry.example.com:443
User-Agent: docker/20.10.21 go/go1.18.1 git-commit/20.10.21-0ubuntu1~20.04.1 kernel/5.15.0-52-generic os/linux arch/amd64 UpstreamClient(Docker-Client/20.10.21 \(linux\))
Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IkVWVkQ6Q1Y2QzpBSVBLOkZONkw6WFkzNjpLREk3OkIyN1M6UlpCWDpVT0NROkZYSDc6VVpDSjpOM0ZGIiwidHlwIjoiSldUIn0.eyJhdXRoX3R5cGUiOiJnaXRsYWJfb3JfbGRhcCIsImFjY2VzcyI6W10sImp0aSI6IjYzMzZjZTg4LWNkYjgtNGViNC1hMTI3LTJlNmQ2NDY0YWRlNCIsImF1ZCI6ImNvbnRhaW5lcl9yZWdpc3RyeSIsInN1YiI6InU2MDc2MDY5IiwiaXNzIjoib21uaWJ1cy1naXRsYWItaXNzdWVyIiwiaWF0IjoxNjgwMjdfDA0LCJuYmYiOjE2ODAyMzU5OTksImV4cCI6MTY4MDIzNjMwNH0.X8Q6GONfF0HM46Gxd_n-gxYru1Etxjxku8PGj7hsVuEIXDWx6YYqvEDV6O6N-RLNMFNhO4QeejWHpseKow6UpaViDp87fmqVlfTYqswYSjrOyLHI-6ocH-e2OfBdsFMiqn_B5529QjDQ8wSwtf6CkpItKS4pUOsuR9Y73LZ_Dx3jT47EtqB5jPWf9ryvup5TaBMMsdK9CKzdGnFEhEUnIWIRgUIfN3OYiCrVnxoR2AQdSqbtwqeUI_SNPJdVqDYg2z23b-7WH3m_HuYT3VNPnBkjCbjpBCN8DkPJB1xdmQCsL7Kj2wfbJkupT_gB7Q5NYQz6u1BnY1rfHSUuHQYtC2GlW0nzD1RDeQey8IVNJgNQEyMxJAda044M0VeMiNMvgR-E7ZuqNipGH5he_Yxuowgelh9AiFFs3OuK9bX8G2ikxuvQpBEOagOFmdILMbRP1dkhy3kBYivZV5NwXL198sYBRgMvXs6nahgu8P93PoP-JFtAwxwAzTSeoJjBZTi-genzNHCj6aLjs5puYxv9FeAFp9RNVi-aRFCCUrQqxbDy0dnUh4sfY3DQlOkUxDf8hwFj6KC6aJI7um3kK7JgsbpJT9yg1Lr9fGr4gWCzlnm51U79jG093fRnrOxW3QkRkn2Iol5GRWsZ3K20DikExAv8o9japdAj_4uNJP3g
Accept-Encoding: gzip, deflate
Connection: close

HTTP/1.1 200 OK
Server: nginx
Date: Fri, 31 Mar 2023 04:13:24 GMT
Content-Type: application/json; charset=utf-8
Content-Length: 2
Connection: close
Docker-Distribution-Api-Version: registry/2.0
X-Content-Type-Options: nosniff

{}

try to decode this base64 string at https://jwt.io/, and we should get head and payload like below.

{
  "alg": "RS256",
  "kid": "<the id of the key>",
  "typ": "JWT"
}

{
  "auth_type": "gitlab_or_ldap",
  "access": [],
  "jti": "6336ce88-cdb8-4eb4-a127-2e6d6464ade4",
  "aud": "container_registry",
  "sub": "u6076069",
  "iss": "omnibus-gitlab-issuer",
  "iat": 1680236004,
  "nbf": 1680235999,
  "exp": 1680236304
}

the key and cert that is used in gitlab.

They should be at /etc/gitlab/gitlab-secrets.json.

try to make a new token by using the key.

I am unsuccessful on this. The revelent code should be at https://github.com/jwt/ruby-jwt.

If someone (or futher me) knows how to do this, please upload the code in this repo.

← Previous Post Next Post →

Installation Information for GitLab Docker Registry

Installation Information on GitLab

Debug for docker login issues.