Skip to main content

Managing Credentials

To manage the progression of freight from stage to stage, Kargo will often require read/write permissions on private GitOps repositories and read-only permissions on private container image and/or Helm chart repositories.

This section presents an overview of how these credentials can be managed.

Credentials as Kubernetes Secret Resources​

caution

Kargo formerly borrowed its general credential-management approach from Argo CD, but has since diverged.

Kargo expects any credentials it requires to have been stored as specially labeled Kubernetes Secret resources containing specially-formatted data. These Secrets take the following form:

apiVersion: v1
kind: Secret
metadata:
  name: <name>
  namespace: <project namespace>
  labels:
    kargo.akuity.io/cred-type: image
stringData:
  repoURL: <repo url>
  username: <username>
  password: <password>

The name of such a Secret is inconsequential and may follow any convention preferred by the user.

info

Kargo uses Kubernetes Namespaces to mark project boundaries. Secrets representing credentials will typically exist in the same Namespace as the Stage resources that will require them. There are exceptions to this, which are covered in the next section.

The label key kargo.akuity.io/cred-type and its value, one of git, helm, or image, is important, as it designates the Secret as representing credentials for a Git repository, a Helm chart repository, or a container image repository, respectively.

The Secret's data field (set above using plaintext in the stringData field), MUST contain the following keys:

  • repoURL: The full URL of the repository the credentials are for.

  • Either:

    • username: The username to use when authenticating to the repository.

    • password: A password or personal access token.

      info

      If the value of the password key is a personal access token, the value of the username field may be inconsequential. You should consult your repository's documentation for more information.

    OR:

    • sshPrivateKey: A PEM-encoded SSH private key. Applicable to Git repositories only.

Optionally, the following keys may also be included:

  • repoURLIsRegex: Set this to true if the value of the repoURL key is a regular expression. Any other value of this key or the absence of this key is interpreted as false.
note

When Kargo searches for repository credentials in a project Namespace, it first checks all appropriately labeled Secrets for a repoURL value matching the repository URL exactly. Only if no Secret is an exact match does it check all appropriately labeled Secrets for a repoURL value containing a regular expression matching the repository URL.

When searching for an exact match, and again when searching for a pattern match, appropriately labeled Secrets are considered in lexical order by name.

caution

Only username/password (or personal access token) authentication is supported at this time. Others are likely to be added in the future.

Global Credentials​

In cases where one or more sets of credentials are needed widely across all Kargo projects, the administrator/operator installing Kargo may opt-in to designating one or more Namespaces as homes for "global" credentials using the controller.globalCredentials.namespaces setting in Kargo's Helm chart. Refer to the advanced section of the installation guide for more details.

note

Any matching credentials (exact match or pattern match) found in a project's own Namespace take precedence over those found in any global credentials Namespace.

When Kargo searches for repository credentials in global credentials Namespaces, it first checks all appropriately labeled Secrets for a repoURL value matching the repository URL exactly. Only if no Secret is an exact match does it check all appropriately labeled Secrets for a repoURL value containing a regular expression matching the repository URL.

When searching for an exact match, and again when searching for a pattern match, appropriately labeled Secrets are considered in lexical order by name.

When Kargo is configured with multiple global credentials Namespaces, they are searched in lexical order by name. Only after no exact match and no pattern match is found in one global credentials Namespace does Kargo search the next.

caution

It is important to understand the security implications of this feature. Any credentials stored in a global credentials Namespace will be available to all Kargo projects.

Managing Credentials with the CLI​

The Kargo CLI can be used to manage credentials in a project's Namespace.

The following example creates credentials for a Git repository:

kargo create credentials --project kargo-demo my-credentials \
  --git --repo-url https://github.com/example/kargo-demo.git \
  --username my-username --password my-my-personal-access-token
secret/my-credentials created
caution

If you do not wish for your password or personal access token to be stored in your shell history, you may wish to omit the --password flag, in which case the CLI will prompt you to enter the password interactively.

Credentials can be listed or viewed with kargo get credentials:

kargo get credentials --project kargo-demo my-credentials
NAME             TYPE   REGEX   REPO                                        AGE
my-credentials   git    false   https://github.com/example/kargo-demo.git   8m25s

If requesting output as YAML or JSON, passwords and other potentially sensitive information will be redacted.

kargo get credentials --project kargo-demo my-credentials -o yaml
apiVersion: v1
kind: Secret
metadata:
  creationTimestamp: "2024-05-30T20:02:46Z"
  labels:
    kargo.akuity.io/cred-type: git
  name: my-credentials
  namespace: kargo-demo
  resourceVersion: "17614"
  uid: ca2660e4-867d-4709-b1a7-57fbb93fc6dc
stringData:
  password: '*** REDACTED ***'
  repoURL: https://github.com/example/kargo-demo.git
  username: my-username
type: Opaque

Credentials can be updated using the kargo update credentials command and the flags corresponding to attributes of the credential that you wish to modify. Other attributes of the credentials will remain unchanged.

The following example updates my-credentials with a regular expression for the repository URL:

kargo update credentials --project kargo-demo my-credentials \
  --repo-url '^http://github.com/' --regex
secret/my-credentials updated

And credentials can, of course, be deleted with kargo delete credentials:

kargo delete credentials --project kargo-demo my-credentials
secret/my-credentials deleted
note

While the CLI may be a fine way of managing project-level credentials whilst getting to know Kargo, it is unquestionably more secure to use other means to ensure the existence of these specially-formatted Secrets in the appropriate project Namespaces.

Git Provider-Specific Authentication Options​

This section provides Git provider-specific guidance on credential management.

GitHub​

Personal Access Token​

GitHub supports authentication using a personal access token, which can be used in place of a password. The corresponding username must be the GitHub handle of the user who created the token. These can be stored in the username and password fields of a Secret resource as described in the first section of this document.

info

This method of authentication may be best when wishing to rigorously enforce the principle of least privilege, as personal access tokens can be scoped to specific permissions on specific repositories.

A drawback to this method, however, is that the token is owned by a specific GitHub user, and if that user should lose their own access to the repositories in question, Kargo will also lose access.

GitHub App Authentication​

GitHub Apps can be used as an authentication method.

  1. Create a GitHub App:

    1. In the GitHub App name field, specify a unique name.
    2. Set the Homepage URL to any URL you like.
    3. Under Webhook, de-select Active.
    4. Under Permissions → Repository permissions → Contents, select whether the App will require Read-only or Read and write permissions. The App will receive these permissions on all repositories into which it is installed.
    5. Under Where can this GitHub App be installed?, leave Only on this account selected.
    6. Click Create GitHub App.
    7. Take note of the App ID.
    8. Scroll to the bottom of the page and click Generate a private key. The resulting key will be downloaded immediately. Store it securely.
    9. On the left-hand side of the page, click Install App.
    10. Choose an account to install the App into by clicking Install.
    11. Select Only select repositories and choose the repositories you wish to grant the App access to. Remember that the App will receive the permissions you selected earlier on all of these repositories.
    12. Click Install.
    13. In your browser's address bar, take note of the numeric identifier at the end of the current page's URL. This is the Installation ID.

  2. Create a Secret resource with the following structure:

    apiVersion: v1
    kind: Secret
    metadata:
      name: <name>
      namespace: <project namespace>
      labels:
        kargo.akuity.io/cred-type: git
    stringData:
      githubAppID: <app id>
      githubAppPrivateKey: <base64-encoded private key>
      githubAppInstallationID: <installation id>
      repoURL: <repo url>
      repoURLIsRegex: <true if repoURL is a pattern matching multiple repositories>
info

Compared to personal access tokens, a benefit of authenticating with a GitHub App is that the App's permissions are not tied to a specific GitHub user.

caution

It is all too easy to violate the principle of least privilege when authenticating using this method.

For convenience sake, it may be tempting to register a single GitHub App and select a broad set of repositories when installing that App into your organization. It may also be tempting to create a single set of global credentials such that all Kargo projects can use them to access their repositories, however, this will have the undesirable effect of granting all Kargo projects access to all of the repositories selected when the App was installed.

It is, instead, recommended to register a separate GitHub App for each Kargo project. When installing each App into your organization, only those repositories to which each Kargo project requires access should be selected.

GitHub organizations are limited to registering 100 GitHub Apps, however, so this approach may not be feasible for organizations with many Kargo projects.

caution

A second way in which authentication using GitHub Apps may violate the principle of least privilege involves the fact that the same permissions are granted to the App on all repositories that are selected when it is installed.

If a Kargo project requires read-only access to one repository and read/write access to another, it is not possible to grant the App different permissions on the two. This may then lead to granting broader permissions than are strictly necessary.

Image Registry-Specific Authentication Options​

While many container image registries support authentication using long-lived credentials, such as a username and password (or personal access token), some either require or offer more secure options.

This section provides registry-specific guidance on credential management and also covers options for gaining image repository access using workload identity on applicable platforms.

Amazon Elastic Container Registry (ECR)​

The authentication options described in this section are applicable only to container image repositories whose URLs indicate they are hosted in ECR.

Long-Lived Credentials​

Elastic Container Registries do not directly support long-lived credentials, however, an AWS access key ID and secret access key can be used to obtain an authorization token that is valid for 12 hours. Kargo can seamlessly obtain such a token and will cache it for a period of 10 hours.

To use this option, your Secret should take the following form:

apiVersion: v1
kind: Secret
metadata:
  name: <name>
  namespace: <project namespace>
  labels:
    kargo.akuity.io/cred-type: image
stringData:
  awsAccessKeyID: <access key id>
  awsSecretAccessKey: <secret access key>
  repoURL: <ecr url>
caution

Following the principle of least privilege, the IAM user associated with the access key ID and secret access key should be limited only to read-only access to the required ECR repositories.

caution

This method of authentication is a "lowest common denominator" approach that will work regardless of where Kargo is deployed. i.e. if running Kargo outside EKS, this method will still work.

If running Kargo within EKS, you may wish to either consider using EKS Pod Identity or IRSA instead.

EKS Pod Identity or IAM Roles for Service Accounts (IRSA)​

If Kargo locates no Secret resources matching a repository URL and is deployed within an EKS cluster, it will attempt to use EKS Pod Identity or IAM Roles for Service Accounts (IRSA) to authenticate. Both of these rely upon some external setup. Leveraging either eliminates the need to store ECR credentials in a Secret resource.

Follow this overview to set up EKS Pod Identity in your EKS cluster or this one to set up IRSA. For either, you will assign an IAM role to the kargo-controller ServiceAccount within the Namespace to which Kargo is (or will be) installed.

note

To use IRSA, you will additionally need to specify the ARN of the controller's IAM role as the value of the controller.serviceAccount.iamRole setting in Kargo's Helm chart. Refer to the advanced section of the installation guide for more details.

At this point, an IAM role will be associated with the Kargo controller, however, that controller acts on behalf of multiple Kargo projects, each of which may require access to different ECR repositories. To account for this, when Kargo attempts to access an ECR repository on behalf of a specific project, it will first attempt to assume an IAM role specific to that project. The name of the role it attempts to assume will always be of the form kargo-project-<project name>. It is this role that should be granted read-only access to applicable ECR repositories.

info

The name of the IAM role associated with each Kargo project is deliberately not configurable to prevent project admins from attempting to coerce Kargo into assuming arbitrary IAM roles.

caution

For optimal adherence to the principle of least permissions, the IAM role associated with the kargo-controller ServiceAccount should be limited only to the ability to assume project-specific IAM roles. Project-specific IAM roles should be limited only to read-only access to applicable ECR repositories.

info

If the Kargo controller is unable to assume a project-specific IAM role, it will fall back to using its own IAM role directly. For organizations without strict tenancy requirements, this can eliminate the need to manage a large number of project-specific IAM roles. While useful, this approach is not strictly recommended.

Once Kargo is able to gain necessary permissions to access an ECR repository, it will follow a process similar to that described in the previous section to obtain a token that is valid for 12 hours and cached for 10.

Google Artifact Registry​

The authentication options described in this section are applicable only to container image repositories whose URLs indicate they are hosted in Google Artifact Registry.

note

Google Container Registry (GCR) has been deprecated in favor of Google Artifact Registry. For authentication to repositories with legacy GCR URLs, the same options outlined here may be applied.

Long-Lived Credentials​

caution

Google Artifact Registry does directly support long-lived credentials as described here. The username _json_key_base64 and the base64-encoded service account key may be stored in the username and password fields of a Secret resource as described in the first section of this document. Kargo and Google both strongly discourage this method of authentication however.

Google documentation recommends using a service account key to obtain an access token that is valid for 60 minutes. Compared to the discouraged method of using the service account key to authenticate to the registry directly, this process does not transmit the service account key over the wire. Kargo can seamlessly carry out this process and will cache the access token for a period of 40 minutes.

To use this option, your Secret should take the following form:

apiVersion: v1
kind: Secret
metadata:
  name: <name>
  namespace: <project namespace>
  labels:
    kargo.akuity.io/cred-type: image
stringData:
  gcpServiceAccountKey: <base64-encoded service account key>
  repoURL: <ecr url>
note

Service account keys contain structured data, so it is important that the key be base64-encoded.

caution

Following the principle of least privilege, the service account associated with the service account key should be limited only to read-only access to the required Google Artifact Registry repositories.

caution

This method of authentication is a "lowest common denominator" approach that will work regardless of where Kargo is deployed. i.e. If running Kargo outside of GKE, this method will still work.

If running Kargo within GKE, you may wish to consider using Workload Identity Federation instead.

Workload Identity Federation​

If Kargo locates no Secret resources matching a repository URL, and if Kargo is deployed within a GKE cluster, it will attempt to use Workload Identity Federation to authenticate, but this relies upon some external setup. Leveraging this option eliminates the need to store credentials in a Secret resource.

First, follow these directions to provision a new GKE cluster with Workload Identity Federation enabled or these directions to enable Workload Identity Federation on an existing GKE cluster.

At this point, the kargo-controller ServiceAccount within the Namespace to which Kargo is (or will be) installed will be associated with an IAM principal identifier, which takes the following form:

principal://iam.googleapis.com/projects/<gcp project number>/locations/global/workloadIdentityPools/<gcp project name>.svc.id.goog/subject/ns/<kargo namespace>/sa/kargo-controller

Although associated with this one principal, the Kargo controller acts on behalf of multiple Kargo projects, each of which may require access to different Google Artifact Registry repositories. To account for this, when Kargo attempts to access a Google Artifact Registry repository on behalf of a specific project, it will first attempt to impersonate a Google service account specific to that project. The name of the service account it attempts to impersonate will always be of the form kargo-project-<kargo project name>@<gcp project name>.iam.gserviceaccount.com. It is this service account that should be granted read-only access to applicable Google Artifact Registry repositories.

info

The name of the Google service account associated with each Kargo project is deliberately not configurable to prevent Kargo project admins from attempting to coerce Kargo into impersonating arbitrary Google service accounts.

Once Kargo is able to impersonate the appropriate Google service account for a given project, it will follow a process similar to that described in the previous section to obtain a token that is valid for 60 minutes and cached for 40.

caution

Following the principle of least privilege, the IAM principal associated with the kargo-controller ServiceAccount should be limited only to the ability to impersonate project-specific Google service accounts. Project-specific Google service accounts should be limited only to read-only access to the applicable Google Artifact Registry repositories.

Azure Container Registry (ACR)​

Azure Container Registry directly supports long-lived credentials.

It is possible to create tokens with repository-scoped permissions, with or without an expiration date. These tokens can be stored in the username and password fields of a Secret resource as described in the first section of this document.

info

Support for authentication to ACR repositories using workload identity, on par with Kargo's support for ECR and Google Artifact Registry, is likely to be included in a future release of Kargo.