Configure a Kubernetes build farm to use self-signed certificates

CI build infrastructure pods can interact with servers using self-signed certificates. This option is useful for organizations that prefer to use internal certificates instead of certificates generated by a public Certificate Authority (CA).

info

• This topic assumes that you are familiar with how to implement SSL in Kubernetes. General information about implementing SSL is outside the scope of this topic.
• With a Kubernetes cluster build infrastructure, all Build and Push steps use kaniko. Kaniko reads certificates from /kaniko/ssl/certs/additional-ca-cert-bundle.crt.
• Harness uses a UBI image for the Git Clone step. UBI reads certificates from /etc/ssl/certs/ca-bundle.crt.
• Different base images use different paths as their default certificate location. For example, Alpine images use the path /etc/ssl/certs/ca-certificates.crt. For other images, make sure you verify the default certificate path.

warning

Custom certificates or self-signed certificates are only supported for linux nodes in a Kubernetes cluster. For Windows, directly mounting the certificate to the destination path will not work.

Enable self-signed certificates

1. Create a Kubernetes secret or config map with the required certificates in the same namespace used by the Harness Delegate. For example:

   apiVersion: v1
   kind: Secret
   metadata:
     name: addcerts
     namespace: harness-delegate-ng
   type: Opaque
   stringData:
     ca.bundle: |
       -----BEGIN CERTIFICATE-----
       XXXXXXXXXXXXXXXXXXXXXXXXXXX
       -----END CERTIFICATE-------
       -----BEGIN CERTIFICATE-----
       XXXXXXXXXXXXXXXXXXXXXXXXXXX
       -----END CERTIFICATE-------

2. Mount the secret as a volume on the delegate pod and add a volumeMount to your certificate files at /opt/harness-delegate/ca-bundle.

   In the delegate pod, do the following:

   1. Provide a comma-separated list of paths in the build pod where you want the certs to be mounted.
   2. Mount your certificate files to /opt/harness-delegate/ca-bundle.

   To mount the secret, go to the delegate Deployment YAML and update volumeMounts and volumes. For an example, expand the section below.

   info

   Secrets are retrieved by the delegate. For Kubernetes build infrastructure, secrets are set as Kubernetes secrets and passed to the pod.

   YAML example: Mount the SSL certificate

      apiVersion: apps/v1
      kind: Deployment
      metadata:
        labels:
          harness.io/name: abc-ca-kubernetes-delegate
        name: abc-ca-kubernetes-delegate
        namespace: abc-ca-kubernetes-delegate
      spec:
        replicas: 1
        minReadySeconds: 120
        selector:
          matchLabels:
            harness.io/name: abc-ca-kubernetes-delegate
        template:
          metadata:
            labels:
              harness.io/name: abc-ca-kubernetes-delegate
            annotations:
              # ...
          spec:
            terminationGracePeriodSeconds: 600
            restartPolicy: Always
            containers:
            - image: harness/delegate:yy.mm.verno
              imagePullPolicy: Always
              name: delegate
              securityContext:
                # ...
              ports:
                # ...
              resources:
                # ...
              livenessProbe:
                # ...
              startupProbe:
                # ...
              envFrom:
                # ...
              env:
                # ...
              volumeMounts:
              ### ADD THE VOLUME MOUNT HERE
              - name: certvol
                mountPath: /opt/harness-delegate/ca-bundle/ca.bundle
                subPath: ca.bundle
            volumes:
            ### ADD THE VOLUME HERE
            - name: certvol
              secret:
                secretName: addcerts
                items:
                  - key: ca.bundle
                    path: ca.bundle
   

   Both CI build pods and the SCM client on the delegate support this method.

   warning

   • Make sure the destination path is not same as the default CA certificate path of the corresponding container image.

   • If you override the default certificate file, make sure the Kubernetes secret or config map (from step one) includes all certificates required by the pipelines that will use this build infrastructure.

3. Restart the delegate. Once it is up and running, exec into the container and ensure that the volume exists at the mounted path and contains your certificates.

Enable self-signed certificates with a self-hosted image registry (advanced)

important

This workflow is applicable only if you're using a self-hosted registry to store your container images. For all other workflows, go to Enable self-signed certificates.

1. Create a Kubernetes secret or config map with the required certificates in the same namespace used by the Harness delegate. For example:

   apiVersion: v1
   kind: Secret
   metadata:
     name: addcerts
     namespace: harness-delegate-ng
   type: Opaque
   stringData:
     ca.bundle: |
       -----BEGIN CERTIFICATE-----
       XXXXXXXXXXXXXXXXXXXXXXXXXXX
       -----END CERTIFICATE-------
       -----BEGIN CERTIFICATE-----
       XXXXXXXXXXXXXXXXXXXXXXXXXXX
       -----END CERTIFICATE-------

2. Mount the secret as a volume on the delegate pod.

   In the delegate pod, do the following:

   1. Add DESTINATION_CA_PATH to the environment.
   2. Provide a comma-separated list of paths in the build pod where you want the certs to be mounted.
   3. Mount your certificate files to /opt/harness-delegate/ca-bundle.

   To add DESTINATION_CA_PATH and mount the secret, go to the delegate Deployment YAML and update env, volumeMounts, and volumes. For an example, expand the section below.

   YAML example: Add DESTINATION_CA_PATH and mount the SSL certificate

      apiVersion: apps/v1
      kind: Deployment
      metadata:
        labels:
          harness.io/name: abc-ca-kubernetes-delegate
        name: abc-ca-kubernetes-delegate
        namespace: abc-ca-kubernetes-delegate
      spec:
        replicas: 1
        minReadySeconds: 120
        selector:
          matchLabels:
            harness.io/name: abc-ca-kubernetes-delegate
        template:
          metadata:
            labels:
              harness.io/name: abc-ca-kubernetes-delegate
            annotations:
              # ...
          spec:
            terminationGracePeriodSeconds: 600
            restartPolicy: Always
            containers:
            - image: harness/delegate:yy.mm.verno
              imagePullPolicy: Always
              name: delegate
              securityContext:
                # ...
              ports:
                # ...
              resources:
                # ...
              livenessProbe:
                # ...
              startupProbe:
                # ...
              envFrom:
                # ...
              env:
              ### ADD DESTINATION_CA_PATH HERE
              - name: DESTINATION_CA_PATH
                value: "/etc/ssl/certs/ca-bundle.crt,/kaniko/ssl/certs/additional-ca-cert-bundle.crt"
              volumeMounts:
              ### ADD THE VOLUME MOUNT HERE
              - name: certvol
                mountPath: /opt/harness-delegate/ca-bundle/ca.bundle
                subPath: ca.bundle
            volumes:
            ### ADD THE VOLUME HERE
            - name: certvol
              secret:
                secretName: addcerts
                items:
                  - key: ca.bundle
                    path: ca.bundle

   Both CI build pods and the SCM client on the delegate support this method.

   warning

   • Make sure the destination path is not same as the default CA certificate path of the corresponding container image.

   • If you override the default certificate file, make sure the Kubernetes secret or config map (from step one) includes all certificates required by the pipelines that will use this build infrastructure.

   Legacy: CI_MOUNT_VOLUMES

   Prior to the introduction of DESTINATION_CA_PATH, you used ADDITIONAL_CERTS_PATH and CI_MOUNT_VOLUMES to mount certs.

   The legacy method is still supported, but Harness recommends DESTINATION_CA_PATH. If you include both, DESTINATION_CA_PATH takes precedence. If Harness can't resolve DESTINATION_CA_PATH, it falls back to CI_MOUNT_VOLUMES and ADDITIONAL_CERTS_PATH.

   You must specify both ADDITIONAL_CERTS_PATH and CI_MOUNT_VOLUMES.

   For ADDITIONAL_CERTS_PATH, provide the path to the certificates in the delegate, such as /tmp/ca.bundle.

   For CI_MOUNT_VOLUMES, provide a comma-separated list of source:destination mappings where source is the certificate path on the delegate, and destination is the path where you want to expose the certificates on the build containers. For example:

   /tmp/ca.bundle:/etc/ssl/certs/ca-bundle.crt,/tmp/ca.bundle:/kaniko/ssl/certs/additional-ca-cert-bundle.crt

   The CI_MOUNT_VOLUMES list must include all certificates that your build containers need to interact with external services.

         spec:
           containers:
             - image: harness/delegate:yy.mm.verno
               # ...
               env:
               # ADD ADDITIONAL_CERTS_PATH AND CI_MOUNT_VOLUMES HERE
                 - name: ADDITIONAL_CERTS_PATH
                   value: /tmp/ca.bundle
                 - name: CI_MOUNT_VOLUMES
                   value: "/tmp/ca.bundle:/etc/ssl/certs/ca-bundle.crt,/tmp/ca.bundle:/kaniko/ssl/certs/additional-ca-cert-bundle.crt"
               volumeMounts:
               # ADD VOLUME MOUNT HERE
                 - name: certvol
                   mountPath: /tmp/ca.bundle
                   subPath:  ca.bundle
           volumes:
             # ADD VOLUME HERE
             - name: certvol
               secret:
                 secretName: addcerts
                 items:
                 - key: ca.bundle
                   path: ca.bundle

3. If you're storing your certificates in a local registry and need to run Docker-in-Docker, specify the local certificate path on the delegate.

   For example, if your self-signed certs are stored at https://my-registry.local.org:799 and you log in with docker login my-registry.local.org:799, then you add the path to your DESTINATION_CA_PATH environment like this:

    spec:
      containers:
      - image: harness/delegate:yy.mm.verno
        env:
        - name: DESTINATION_CA_PATH
          value: "/etc/docker/certs.d/my-registry.local.org:799/ca.crt,/etc/ssl/certs/ca-bundle.crt,/kaniko/ssl/certs/additional-ca-cert-bundle.crt"
        volumeMounts:
          - name: certvol
            mountPath: /opt/harness-delegate/ca-bundle/ca.bundle
            subPath:  ca.bundle
      volumes:
      - name: certvol
        secret:
            secretName: addcerts
             items:
             - key: ca.bundle
               path: ca.bundle

4. Restart the delegate. Once it's up and running, exec into the container and ensure that the volume exists at the mounted path and contains your certificates.

Secret Name Creation for Certificate Bundles in Build Pods

When a customer mounts a certificate bundle in their build pods, the bundle is created as a secret and mounted in the pod.

The secret names follow a specific format: Podname-filename-hash.

For Exampple:- Pod Name: harnessci-buildpod-9rwcbkl6 Secret Name: harnessci-buildpod-9rwcbkl6-custom-cr-pem-0-pem-1667255857 Here, custom-cr-pem is the pem file name and 1667255857 is the random hash that is generated during secret creation.

The name is sanitized to comply with Kubernetes naming conventions. If the total length exceeds 63 characters, it is trimmed from the end to ensure compliance.

note

1. Previously, if the characters exceeded the limit, users had to manually shorten the file or stage names to avoid errors. Now, this change for automatic trimming of secret names beyond 63 characters is available starting from delegate release 821xx.
2. This behavior applies only to Kubernetes architecture.
3. To avoid potential failures in Kubernetes environments, ensure that all secret names are lowercase. Starting from delegate release 825xx, any uppercase characters in the secret name are automatically converted to lowercase to ensure compatibility with Kubernetes naming conventions.

Using Dockerfile builds with self-signed certificates (BuildKit-specific)

When using the Build and Push to Docker step (or any step that performs Dockerfile builds using BuildKit), mounting CA certificates to the Delegate is not enough. BuildKit must be explicitly configured to trust your private registry's certificate.

This workaround is only required for Build and Push to Docker steps using Docker BuildKit. Steps using Kaniko, like most default Kubernetes builds in Harness, do not require this additional configuration if the certificate is already mounted using DESTINATION_CA_PATH.

To make BuildKit trust your self-signed registry certs, you need to:

1. Dynamically prepare the BuildKit config and cert path in a Run step.

2. Share the cert and config path using sharedPaths and environment variables.

3. Concatenate multiple PEM files (if applicable) into a single CA bundle used by both Docker and BuildKit.

Step-by-step example

sharedPaths:
  - /etc/buildkit/certs/your.registry.com

envVariables:
  PLUGIN_BUILDER_CONFIG: /harness/buildkit.toml

Run step to prepare certs and BuildKit config

Here is an example Run step that uses CA certs already mounted into the Delegate and build pod (e.g., via volume mounts to /harness-shared-certs-path), prepares the BuildKit config file, and builds a Dockerfile

- step:
    type: Run
    name: Prepare BuildKit Config
    identifier: buildkit_config
    spec:
      connectorRef: account.harnessImage
      image: alpine
      shell: Sh
      command: |-
        # Registry address (update this to match your use case)
        REG="your.registry.com"
        CERT_PATH_BUILDX="/etc/buildkit/certs/${REG}"
        BUILDKIT_TOML="/harness/buildkit.toml"

        # Create cert paths
        mkdir -p "$CERT_PATH_BUILDX"

        # Concatenate all mounted PEMs into a single CA bundle
        for cert in /harness-shared-certs-path/*.pem; do
          cat "$cert" >> "$CERT_PATH_BUILDX/ca.crt"
          echo >> "$CERT_PATH_BUILDX/ca.crt"
        done

        # Write buildkit.toml
        echo "Writing buildkit.toml..."
        cat <<EOF > "$BUILDKIT_TOML"
        [registry."$REG"]
          ca = ["$CERT_PATH_BUILDX/ca.crt"]
        EOF

        ls -lr "$CERT_PATH_BUILDX"
        cat "$BUILDKIT_TOML"

Build step using the config

- step:
    type: BuildAndPushDockerRegistry
    name: Docker Build
    identifier: docker_build
    spec:
      ...
      envVariables:
        PLUGIN_BUILDER_CONFIG: /harness/buildkit.toml

note

• Certs should be mounted ahead of time (e.g., through Harness secrets or volume mounts).

• This approach supports multiple PEM files and ensures both Docker and BuildKit trust the self-signed certs.

• You must ensure that the sharedPaths include both cert directories so that they persist across steps.

This ensures that BuildKit inside your pipeline trusts the self-signed certificate used by your private registry.

Troubleshoot Kubernetes cluster build infrastructures

Go to the CI Knowledge Base for questions and issues related to Kubernetes cluster build infrastructures, including use of self-signed certificates, such as:

• SCM service connection issues
• How do I make internal CA certs available to the delegate pod?
• Where should I mount internal CA certs on the build pod?
• Certificate volumes aren't mounted to the build pod