Akeyless K8s Secrets Injector

Overview

The Akeyless K8s Secrets Injector plugin enables K8s applications and workloads to use Static, Rotated, and Dynamic secrets as well as Certificates sourced from the Akeyless Platform.

This injector leverages the K8s MutatingAdmissionWebhook to intercept and augment specifically annotated pod configurations for secrets injection. By doing so, the user benefits as the applications remain ืดAkeyless unawareืด as the secrets are stored either as an environment variable or as a file at a filesystem path in their container.

Before the application starts, the injector deploys an init container to fetch and inject secrets at pod start-up, after which the init-container shuts down. To apply an automatic rollout restart to your deployments upon any change to your secrets, you can use the Injector with restart-rollout mode, which can track any changes of Static, Rotated and Certificates.

If the application consumes secrets which regularly change, an annotation can be used to deploy an additional Sidecar container which runs alongside the application to monitor changes in secrets. The Sidecar tracks and updates secrets within injected files inside the pods, according to specifically annotated pod configurations, and will remain up for the entire application lifecycle. Relevant for cases where the app can watch for live changes in files.

Although authorization in K8s is intentionally high level, you can configure the injector to support full and flexible segregation using K8s policies together with the Akeyless Platform's Role-based Access Control (RBAC).
For details, see Policy Segregation for Kubernetes.

๐Ÿ‘

Note

The documentation, configuration and examples for the plugin are also applicable to Red Hat OpenShift environment.

Prerequisites

  • Helm Installed.

  • K8s Auth or one of the supported Authentication Methods for Kubernetes.

  • K8s v1.19 and above.

  • For Azure Kubernetes Service (AKS), managed-identity is enabled on your AKS cluster.

  • For Google Kubernetes Engine (GKE) cluster, port 8443 is opened in your Google Cloud Platform (GCP) firewall rules.

Create a Secret in Akeyless

For example, the following command creates a static secret called my_k8s_secret inside K8s folder.

akeyless create-secret --name /K8s/my_k8s_secret --value myPassword

Alternatively, a secret can contain JSON structured data, for example:

akeyless create-secret --name /K8s/secret-json --value '{"aws_access_key":"1234","aws_key_id":"abcd"}'

๐Ÿ‘

Note

The following example uses a pre-defined K8s Auth called K8s_Auth in K8s folder i.e. K8s/K8s_Auth

Create an Access Role

Create an Access Role associate the role with an Auth Method and grant access to the secret.
For example, the following command creates K8s_role role, the role is associated to K8s_Auth Auth Method, and grant read and list access to all the secrets in K8s folder

akeyless create-role --name /K8s/K8s_Role
akeyless assoc-role-am --role-name /K8s/K8s_Role --am-name K8s/K8s_Auth
akeyless set-role-rule --role-name /K8s/K8s_Role --path /K8s/'*' --capability read --capability list

Install the Injector

  1. Add the Akelyess K8s Injector Helm repository from here and update your Helm repositories.
helm repo add akeyless https://akeylesslabs.github.io/helm-charts
helm repo update
  1. Fetch the values.yaml file locally:
helm show values akeyless/akeyless-secrets-injection > values.yaml

Modify the following values under the env section as follows:

  • Set AKEYLESS_ACCESS_ID to the Access ID of the Auth Method with access to the secret.

  • Set AKEYLESS_ACCESS_TYPE to k8s. Or with any other supported Authentication Methods for Kubernetes.

  • Set AKEYLESS_K8S_AUTH_CONF_NAME with your Gateway Kubernetes Auth name. Relevant only for Access type of k8s.

  • Set AKEYLESS_API_GW_URL with the URL of your Akeyless Gateway on port 8080.

  • Optional AKEYLESS_CRASH_POD_ON_ERROR Upon any failure, a pod that tries to fetch a secret and fails will crash. By default this option is disabled. Can be controlled globally or at the deployment level using a dedicated annotation.

  • Optional restartRollout - to apply automatic rollout restart to your deployments upon secret changes. Relevant only for the kinds of: Deployment, DaemonSet or StatefulSet. To control which deployments are not effected by the restart-rollout, you can use a dedicated annotation to disable this on the deployment level.

restartRollout:
  enabled: false
  interval: 1m
  
env:
  AKEYLESS_ACCESS_ID: "<AccessID>"
  AKEYLESS_ACCESS_TYPE: "k8s"
  AKEYLESS_K8S_AUTH_CONF_NAME: "K8s_Auth_Name"
  AKEYLESS_API_GW_URL: "https://Your-Gateway-URL:8080" 
 # AKEYLESS_CRASH_POD_ON_ERROR: "enable"

๐Ÿ‘

Note

  1. When working with Red Hat OpenShift, enable the OpenShift flag in the values.yaml chart file: openshiftEnabled: true

  2. Injecting secrets into the namespace where the k8s injector plugin is installed is unsupported.

  1. On your K8s cluster, create and label a namespace for Akeyless.
kubectl create namespace akeyless
kubectl label namespace akeyless name=akeyless

Alternatively, for Red Hat OpenShift:

oc create namespace akeyless
oc label namespace akeyless name=akeyless
  1. Deploy the Helm chart to the selected namespace.
helm install aks akeyless/akeyless-secrets-injection --namespace akeyless -f values.yaml
  1. Validate the deployment state.
kubectl get all -n akeyless

Alternatively, for Red Hat OpenShift:

oc get all -n akeyless

The following is an example of the output:

kubectl get all -n akeyless

NAME                                                        READY      STATUS         RESTARTS     AGE
pod/aks-akeyless-secrets-injection-77c857d496-r5xth         1/1        Running        1 (73s ago)  1d

NAME                                                        TYPE       CLUSTER-IP     EXTERNAL-IP  PORT(S)   AGE
service/aks-akeyless-secrets-injection                      ClusterIP  10.97.228.133  <none>       443/TCP   1d

NAME                                                        READY      UP-TO-DATE     AVAILABLE    AGE
deployment.apps/aks-akeyless-secrets-injection              1/1        1              1            1d

NAME                                                        DESIRED    CURRENT        READY        AGE
replicaset.apps/aks-akeyless-secrets-injection-77c857d496   1          1              1            1d

Launch an Application

The Akeyless Injector supports the following modes of operations, using Environment Variables, File Injection, and SideCar mode which can work only with File Injection.

Environment Variable

Set the following annotations in your deployment YAML files:

Enable the plugin under the annotations section, in your app deployment file:

akeyless/enabled: "true"

To inject your secret into your pod environment variable during the init phase, the value of your env should be set with akeyless:/Path/to/secret.

๐Ÿšง

Note

The Akeyless env mode does not support the docker ENTRYPOINT - it overrides it by default, meaning you have to explicitly provide it as part of the command inside your deployment YAML.

The following example demonstrates Akeyless secret injection as an Environment Variable into an alpine deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: test
spec:
  replicas: 1
  selector:
    matchLabels:
      app: hello-secrets
  template:
    metadata:
      labels:
        app: hello-secrets
      annotations:
        akeyless/enabled: "true"
    spec:
      containers:
      - name: alpine
        image: alpine
        command:
          - "sh"
          - "-c"
          - "echo $MY_SECRET && echo going to sleep... && sleep 10000"
        env:
        - name: MY_SECRET
          value: akeyless:/K8s/my_k8s_secret

Apply:

kubectl apply -f env.yaml

The following example demonstrates Akeyless secret injection as an Environment Variable into MySQL Database and WordPress server deployments:

apiVersion: v1
kind: Service
metadata:
  name: wordpress-mysql
  labels:
    app: wordpress
spec:
  ports:
    - port: 3306
  selector:
    app: wordpress
    tier: mysql
  clusterIP: None
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: mysql-pv-claim
  labels:
    app: wordpress
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 20Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: wordpress-mysql
  labels:
    app: wordpress
spec:
  selector:
    matchLabels:
      app: wordpress
      tier: mysql
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: wordpress
        tier: mysql
      annotations:
        akeyless/enabled: "true"
    spec:
      containers:
      - image: bitnami/mysql:5.7.33
        name: mysql
        command: ["/opt/bitnami/scripts/mysql/entrypoint.sh"]
        args: ["/opt/bitnami/scripts/mysql/run.sh"]
        env:
        - name: MYSQL_ROOT_PASSWORD
          value: akeyless:/K8s/my_k8s_secret
        ports:
        - containerPort: 3306
          name: mysql
        volumeMounts:
        - name: mysql-persistent-storage
          mountPath: /var/lib/mysql
      volumes:
      - name: mysql-persistent-storage
        persistentVolumeClaim:
          claimName: mysql-pv-claim
apiVersion: v1
kind: Service
metadata:
  name: wordpress
  labels:
    app: wordpress
spec:
  ports:
    - port: 80
  selector:
    app: wordpress
    tier: frontend
  type: LoadBalancer
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: wp-pv-claim
  labels:
    app: wordpress
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 20Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: wordpress
  labels:
    app: wordpress
spec:
  selector:
    matchLabels:
      app: wordpress
      tier: frontend
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: wordpress
        tier: frontend
      annotations:
        akeyless/enabled: "true"
    spec:
      containers:
      - image: wordpress:4.8-apache
        name: wordpress
        command: ["docker-entrypoint.sh"]
        args: ["apache2-foreground"]
        env:
        - name: WORDPRESS_DB_HOST
          value: wordpress-mysql
        - name: WORDPRESS_DB_PASSWORD
          value: akeyless:/K8s/my_k8s_secret
        ports:
        - containerPort: 80
          name: wordpress
        volumeMounts:
        - name: wordpress-persistent-storage
          mountPath: /var/www/html
      volumes:
      - name: wordpress-persistent-storage
        persistentVolumeClaim:
          claimName: wp-pv-claim

Apply:

kubectl apply -f MySQLWordPress.yaml
kubectl apply -f Wordpress.yaml

Another example demonstrates fetching secret specific versions for example version=2 of the secret my_k8s_secret in the K8s folder, decode in base64:

- name:  MY_SECRET
    value: 'akeyless:/K8s/my_k8s_secret|decode=base64|version=2'

Or to extract a specific key from a secret that contains a JSON structured data:

- name:  MY_JSON_SECRET
    value: 'akeyless:/K8s/secret-json|jq=.json_key'

Alternatively, you can parse the entire JSON keys automatically into environment variables using:

- name:  MY_JSON_SECRET
    value: 'akeyless:/K8s/secret-json|parse_json_secret=true'        

This will create an environment variable per each key that exists within the JSON using the following format:MY_JSON_SECRET_JSON_KEY_NAME.

๐Ÿ“˜

Note

The parse_json_secret flag is designed to handle flat JSON structures with single string-values (it does not support nested JSONs, nor array values).

File Injection

Enable the plugin under the annotations section, in your app deployment file:

akeyless/enabled: "true"

The default location of the Akeyless secrets folder inside your pod file system is /akeyless/secrets/.

To explicitly set a different location you can override this by adding |location=<path> after your secret name within the annotation.

For example, /K8s/my_k8s_secret and /K8s/my_k8s_secret2 will be saved inside your pod filesystem under the /tmp/ folder as secret1 and secret2 respectively.

akeyless/inject_file: "/K8s/my_k8s_secret|location=/tmp/secret1,/K8s/my_k8s_secret2|location=/tmp/secret2"

You can specify different destinations for file injection for multiple secrets using the following format: akeyless/inject_file_*.

In this example, we will inject different secrets into different locations:

akeyless/inject_file_s1: "/K8s/secret-json|jq=.object.first|location=/tmp/secret_json|permission=0777|decode=base64"
akeyless/inject_file_secret2: "/K8s/my_k8s_secret|location=/akeyless/secret2" 
akeyless/inject_file_secret1: "/K8s/my_k8s_secret2"

To inject an entire folder of secrets from Akeyless, for example, all secrets under /K8s/my-secrets-folder will be injected into the pod fs under /tmp/secrets/K8s/<secrets-full-name>:

akeyless/inject_folder: "/K8s/my-secrets-folder/|location=/tmp/secrets/"

To inject only the secrets from the source folder without the full folders structure from Akeyless, for example all secrets under /K8s/my-secret-folder use the following pipe command with the annotation:

akeyless/inject_folder: "/K8s/my-secrets-folder/|folder_location=/tmp/secrets/"

To modify the default target folder location of the Akeyless secrets on your pods' file, you can use this setting on the injector chart values.yaml file.

AKEYLESS_SECRET_DIR_NAME: "/My/New/Dir" #Path to save secrets inside pod's file systems

The following example demonstrates Akeyless secret injection with file injection into an alpine deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: test-file
spec:
  replicas: 1
  selector:
    matchLabels:
      app: hello-secrets-2
  template:
    metadata:
      labels:
        app: hello-secrets-2
      annotations:
        akeyless/enabled: "true"
        akeyless/inject_file: "/K8s/my_k8s_secret"
    spec:
      containers:
      - name: alpine
        image: alpine
        command:
          - "sh"
          - "-c"
          - "cat /akeyless/secrets/K8s/my_k8s_secret && echo going to sleep... && sleep 10000"

Apply:

kubectl apply -f Injectfile.yaml

Sidecar Mode

To keep track of secret changes while reflecting them into your pods during their lifetime, you can use the Sidecar mode, for example, while working with Dynamic or Rotated secrets.

Enable the plugin under the annotations section, in your app deployment file and enable the sidecar mode by adding the following annotations:

akeyless/enabled: "true"
akeyless/side_car_enabled: "true"

Set the desired refresh interval for the sidecar where the units are Int and the supported time units are "s", "m" or "h":

akeyless/side_car_refresh_interval: "30m"

To retrieve multiple versions of your secret simultaneously (only for Static and Rotated):

akeyless/side_car_versions_to_retrieve: "2"

The following example demonstrates secret injection with sidecar mode into an alpine deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: test-file-sidecar
spec:
  replicas: 1
  selector:
    matchLabels:
      app: file-secrets
  template:
    metadata:
      labels:
        app: file-secrets
      annotations:
        akeyless/enabled: "true"
        akeyless/inject_file: "/K8s/my_k8s_secret|location=/secrets/secretsVersion.json" 
        akeyless/side_car_enabled: "true"
        akeyless/side_car_refresh_interval: "30m"
        akeyless/side_car_versions_to_retrieve: "2"
    spec:
      containers:
      - name: alpine
        image: alpine
        command:
          - "sh"
          - "-c"
          - "while true; do [ ! -f /secrets/timestamp ] || [ /secrets/secretsVersion.json -nt /secrets/timestamp ] && touch /secrets/timestamp && cat /secrets/secretsVersion.json && echo ''; sleep 15; done"

Apply:

kubectl apply -f Akeyless_sidecar.yaml

Annotations List

The following table lists the available annotations:

AnnotationOptionsDescription
akeyless/enabled: "true""true" or "false"Enable the K8s plugin
akeyless/side_car_enabled: "true""true" or "false"Set the K8s plugin to work in sidecar mode
akeyless/disable_restart_rollout: "true""true" or "false"To disable the restart-rollout on a specific deployment
akeyless/side_car_refresh_interval: "30m"Int followed by
"s", "m"or "h" units
Set the desired refresh time interval for the Akeyless sidecar, by default set to 30m
akeyless/side_car_versions_to_retrieve: "2""2" or higherFetch the last X versions of your secret
akeyless/inject_file: "/mysecret/|location=/path to save secret name"location= /path to save secret nameSet the location for your secrets to be saved within your pod file system.

Note: Available for files only
akeyless/inject_file: "/mysecret|permission=0644"permission= 0644Set the permission of the file that contains your secret value

Default is 0644

Note: Available for files only
akeyless/inject_file: "/mysecret|version=1"version= {version number}Fetch a specific version of your secret

The default value is set to the latest version

Note: Available for Environment variables as well
akeyless/inject_file: "/mysecret|decode=base64"decode= none or base64Set the decoding for your encoded secret values

Default is none

Note: Available for Environment variables as well
akeyless/inject_file: "/mysecret|jq={jq-expresion}"jq expression to work with conventional JSON data formjq={jq-expresion} e.g. secret items that contain JSON structure, can be parsed directly
akeyless/inject_folder: "/prod/my-secrets-folder/|permission=0644"permission=0644Set the permission of the folder that contains your secret value

Default is 0644

Note: Available for files only
akeyless/inject_folder: "/prod/my-secrets-folder/|location=/tmp/secrets/|track-folder-changes=true"track-folder-changes= true or falseTrack injected folder changes to sync new secrets
akeyless/post_inject_script: |
#!/bin/bash
echo Hello > /akeyless/secrets/hello.txt
script to execute post-fetching the secretNote:
the execution occurs in the init container and at the sidecar container if set.
akeyless/crash_on_error: "true"Crash the pod on injection failuresCan be controlled globally for all deployments, or explicitly.
akeyless/agent_limits_cpuInt followed by m unitsLimit of CPU usage
e.g. 600m
where the unit suffix m stands for core thousandth (miliCPU)
akeyless/agent_requests_cpuInt followed by m unitsLimit of CPU request
e.g. 250m
where the unit suffix m stands for core thousandth (miliCPU)
akeyless/agent_limits_memInt followed by Mi unitsLimit of Memory usage, e.g. 64Mi
akeyless/agent_requests_memInt followed by MiunitsLimit of Memory request usage, e.g. 64Mi

Troubleshooting

When you are working with a GKE cluster, make sure that port 8443 is opened in your firewall rules, This port is needed by the Akeyless Secret Injection mutating webhook. Update your firewall rule as follows:

  1. Review the firewall rule for access:
gcloud compute firewall-rules list
  1. Replace the existing rule and allow access:
gcloud compute firewall-rules update <firewall-rule-name> --allow tcp:10250,tcp:443,tcp:8443

Tutorial

Check out our tutorial video on Injecting Secrets into a Kubernetes Cluster.