Provisioning a Secret to Your K8s Cluster

To provision a secret to your Kubernetes cluster, create the secret in the Akeyless Vault platform, along with an authentication method that has access to the secret. Then, configure K8s to get the secret.

Set Up a Secret in the Akeyless Vault Platform

👍

Prerequisites

  • Ensure that you have a K8s cluster and Helm installed before you begin.

  • A K8s authentication method.

  • K8s v1.19 and later (to use admissionregistration.k8s.io/v1).

  • MutatingAdmissionWebhook admission controllers enabled. This topic provides a recommended set of admission controllers to enable in general.

  • For Azure AKS, managed-identity enabled on your AKS cluster.

  1. Create a static or dynamic secret in the Akeyless Vault Platform. For example, the following command creates a static secret called my_k8s_secret in the K8s folder.
akeyless create-secret --name K8s/my_k8s_secret --value myPassword
  1. Create an authentication method. For details about the authentication methods that are supported for K8s, see Authentication Methods for K8s.

The following example uses a pre-defined Kubernetes Auth method called K8s_Auth in K8s folder i.e. K8s/K8s_Auth.

  1. Create an access role to provide the authentication method with access to the secret. For example, the following commands create the K8s_Role role in the K8s folder, associate the role with the K8s_Auth authentication method, and grant read and list access to all the secrets in the K8s folder.
akeyless create-role --name /K8s/K8s_Role

akeyless assoc-role-am --role-name /K8s/K8s_Role --am-name K8s/K8s_Auth

akeyless set-role-rule --role-name /K8s/K8s_Role --path /K8s/'*' --capability read --capability list

Configure the Akeyless K8s Plugin

  1. Get the Akeyless Helm chart from here.
helm repo add akeyless https://akeylesslabs.github.io/helm-charts
helm repo update
  1. Fetch the values.yaml file locally, and modify the access credential values in the file.
helm show values akeyless/akeyless-secrets-injection > values.yaml

👍

OpenShift

Enable the OpenShift flag in the values.yaml file: openshiftEnabled: true.

For example, if you are using a Kubernetes Auth, modify the values as follows:

  • Set AKEYLESS_ACCESS_ID to the access ID of the authentication method that has access to the secret.

  • Set AKEYLESS_ACCESS_TYPE to k8s.

  • Set AKEYLESS_K8S_AUTH_CONF_NAME with your Gateway Kubernetes Auth name.

  • Set the AKEYLESS_API_GW_URL with the URL of your Akeyless Gateway on port 8080.

  • Optional AKEYLESS_CRASH_POD_ON_ERROR upon any failure, a pod that tries to fetch a secret and fails will crash. By default this option is disabled.

AKEYLESS_URL: "https://vault.akeyless.io"
AKEYLESS_ACCESS_ID: "<acc_id>"
AKEYLESS_ACCESS_TYPE: "k8s"
AKEYLESS_K8S_AUTH_CONF_NAME: "K8s_conf_name"
AKEYLESS_API_GW_URL: "https://gateway.url:8080" 
#AKEYLESS_CRASH_POD_ON_ERROR: "enable"
  1. On your K8s cluster, create a new namespace for Akeyless.
kubectl create namespace akeyless
kubectl label namespace akeyless name=akeyless

For OpenShift:

oc create namespace akeyless
oc label namespace akeyless name=akeyless
  1. Deploy the Helm chart to the namespace.
helm install aks akeyless/akeyless-secrets-injection --namespace akeyless -f values.yaml
  1. Validate the deployment state.
kubectl get all -n akeyless

For OpenShift:

oc get all -n akeyless

The following is an example of the output:

kubectl get all -n akeyless
NAME                                                  READY   STATUS    RESTARTS       AGE
pod/aks-akeyless-secrets-injection-77c857d496-r5xth   1/1     Running   1 (73s ago)   1d

NAME                                     TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
service/aks-akeyless-secrets-injection   ClusterIP   10.97.228.133   <none>        443/TCP   1d

NAME                                             READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/aks-akeyless-secrets-injection   1/1     1            1           1d

NAME                                                        DESIRED   CURRENT   READY   AGE
replicaset.apps/aks-akeyless-secrets-injection-77c857d496   1         1         1       1d

Akeyless Annotations

To work with the Akeyless secret injecting webhook, you need to set the following annotations in your deployment YAML files:

  1. Enable the Akeyless webhook
akeyless/enabled: "true"
  1. The default location of the Akeyless secrets folder inside your pod file system is /akeyless/secrets/. You can modify the default directory in the values.yaml file. You can specify a different location by adding location=<path> after your secret name within the annotation. For example, secret 1 and secret 2 will be saved in the /tmp/ folder as s1 and s2 respectively.
akeyless/inject_file: "/secret1|location=/tmp/s1,/secret2|location=/tmp/s2"

Alternatively, you can specify different destinations for files injection for multiple secrets using the following format:

akeyless/inject_file_*

In this example we will inject different secrets to different files:

akeyless/inject_file_s1: "/my-dir/my-json-secret|jq=.object.first|location=/tmp/s1|permission=0777|decode=base64"
akeyless/inject_file_secret2: "/k8s-secrets/secret2|location=/akeyless/secret2" 
akeyless/inject_file_secret1: "/k8s-secrets/secret1"

To inject an entire folder, use the following annotation:

akeyless/inject_folder: "/prod/my-secrets-folder/|location=/tmp/secrets/"

In this example, all secrets under /prod/my-secrets-folder will be injected into your pod fs at /tmp/secrets/<full-secret-names>.

  1. To set a default location of the Akeyless secrets folder inside your pods' file system to inject your secret into your pod file system globally, use this setting on the values.yaml file.
AKEYLESS_SECRET_DIR_NAME: "/My/New/Dir" #Path to save secrets inside pods file systems

To inject your secret into your pod environment variable, the value of your env should have the
prefix akeyless:/Path/to/secret .

👍

The Akeyless env does not support docker ENTRYPOINT. Use your command inside your deployment yaml

The following examples demonstrate Akeyless secret injection for environment variable env.yaml and pod file system Injectfile.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: test
spec:
  replicas: 1
  selector:
    matchLabels:
      app: hello-secrets
  template:
    metadata:
      labels:
        app: hello-secrets
      annotations:
        akeyless/enabled: "true"
    spec:
      containers:
      - name: alpine
        image: alpine
        command:
          - "sh"
          - "-c"
          - "echo $MY_SECRET && echo going to sleep... && sleep 10000"
        env:
        - name: MY_SECRET
          value: akeyless:/K8s/my_k8s_secret
apiVersion: apps/v1
kind: Deployment
metadata:
  name: test-file
spec:
  replicas: 1
  selector:
    matchLabels:
      app: hello-secrets-2
  template:
    metadata:
      labels:
        app: hello-secrets-2
      annotations:
        akeyless/enabled: "true"
        akeyless/inject_file: "/K8s/my_k8s_secret"
    spec:
      containers:
      - name: alpine
        image: alpine
        command:
          - "sh"
          - "-c"
          - "cat /akeyless/secrets/K8s/my_k8s_secret && echo going to sleep... && sleep 10000"
apiVersion: v1
kind: Service
metadata:
  name: wordpress-mysql
  labels:
    app: wordpress
spec:
  ports:
    - port: 3306
  selector:
    app: wordpress
    tier: mysql
  clusterIP: None
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: mysql-pv-claim
  labels:
    app: wordpress
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 20Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: wordpress-mysql
  labels:
    app: wordpress
spec:
  selector:
    matchLabels:
      app: wordpress
      tier: mysql
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: wordpress
        tier: mysql
      annotations:
        akeyless/enabled: "true"
    spec:
      containers:
      - image: bitnami/mysql:5.7.33
        name: mysql
        command: ["/opt/bitnami/scripts/mysql/entrypoint.sh"]
        args: ["/opt/bitnami/scripts/mysql/run.sh"]
        env:
        - name: MYSQL_ROOT_PASSWORD
          value: akeyless:/K8s/my_k8s_secret
        ports:
        - containerPort: 3306
          name: mysql
        volumeMounts:
        - name: mysql-persistent-storage
          mountPath: /var/lib/mysql
      volumes:
      - name: mysql-persistent-storage
        persistentVolumeClaim:
          claimName: mysql-pv-claim
apiVersion: v1
kind: Service
metadata:
  name: wordpress
  labels:
    app: wordpress
spec:
  ports:
    - port: 80
  selector:
    app: wordpress
    tier: frontend
  type: LoadBalancer
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: wp-pv-claim
  labels:
    app: wordpress
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 20Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: wordpress
  labels:
    app: wordpress
spec:
  selector:
    matchLabels:
      app: wordpress
      tier: frontend
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: wordpress
        tier: frontend
      annotations:
        akeyless/enabled: "true"
    spec:
      containers:
      - image: wordpress:4.8-apache
        name: wordpress
        command: ["docker-entrypoint.sh"]
        args: ["apache2-foreground"]
        env:
        - name: WORDPRESS_DB_HOST
          value: wordpress-mysql
        - name: WORDPRESS_DB_PASSWORD
          value: akeyless:/K8s/my_k8s_secret
        ports:
        - containerPort: 80
          name: wordpress
        volumeMounts:
        - name: wordpress-persistent-storage
          mountPath: /var/www/html
      volumes:
      - name: wordpress-persistent-storage
        persistentVolumeClaim:
          claimName: wp-pv-claim

Sidecar Mode

Akeyless plugin for K8s can work as a sidecar, where an additional container will run side by side to your pod in order to track any changes within your secrets values while reflecting those changes into your running pod.

👍

Note:

Sidecar mode can work only with file injection mode.

To enable Akeyless as a sidecar on your K8s cluster add the following annotation to your deployment file:

akeyless/side_car_enabled: "true"

To set the desired refresh interval for Akeyless sidecar :

akeyless/side_car_refresh_interval: "300s"

Where supported time units are "s", "m", "h".

To retrieve multiple versions of your secret simultaneously (only for static-secret and rotated-secret)

akeyless/side_car_versions_to_retrieve: "2"

The following example demonstrate the Akeyless sidecar annotation:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: test-file
spec:
  replicas: 1
  selector:
    matchLabels:
      app: file-secrets
  template:
    metadata:
      labels:
        app: file-secrets
      annotations:
        akeyless/enabled: "true"
        akeyless/inject_file: "/K8s/my_k8s_secret|location=/secrets/secretsVersion.json" 
        akeyless/side_car_enabled: "true"
        akeyless/side_car_refresh_interval: "30s"
        akeyless/side_car_versions_to_retrieve: "2"
    spec:
      containers:
      - name: alpine
        image: alpine
        command:
          - "sh"
          - "-c"
          - "cat /secrets/secretsVersion.json && echo going to sleep... && sleep 10000"

The following table lists the available annotations:

AnnotationOptionsDescription
akeyless/enabled: "true"true or falseEnable akeyless K8s plugin.
akeyless/side_car_enabled: "true"true or falseSet Akeyless K8s plugin to work in side car mode.
akeyless/side_car_refresh_interval:"30s"Supported time units are
"s", "m", "h".
Set the desired refresh interval for Akeyless sidecar
akeyless/side_car_versions_to_retrieve: "2"'2' or higherFetch the last X versions of your secret.
akeyless/inject_file: "/mysecret/|location=/path/to/save/secret/name"location= /path/to/save/secretSet the location for your secrets to be saved within your pod file system.
Available for files only.
akeyless/inject_file: "/mysecret|permission=0644"permission=0777Set the permission of the file that contains your secret value.

Default is 0644.

Available for files only
akeyless/inject_file: "/mysecret|version=1"version={version of your secret}Fetch a specific version of your secret.

Default value is set to latest version.

Available for Environment variables as well.
akeyless/inject_file: "/mysecret|decode=base64"decode=base64Set default base64 decoding for your secrets values.

Default is none

Available for Environment variables as well.
akeyless/inject_file: "/mysecret|jq={jq-expresion}"jq expression to work with conventional JSON data formjq={jq-expresion} e.g. secret items which contains JSON structure, can be parsed directly.
akeyless/inject_folder: "/prod/my-secrets-folder/|location=/tmp/secrets/"permission=0644Set the permission of the folder that contains your secrets value.

Default is 0644.

Available for files only
akeyless/inject_folder: "/prod/my-secrets-folder/|location=/tmp/secrets/|track-folder-changes=true"track-folder-changes=true or falseTrack injected folder changes to sync new secrets
akeyless/post_inject_script: | #!/bin/bash echo "Hello World" script to execute post fetching the secretNote:
the execution occurs in the init container or/and at the sidecar container,
akeyless/agent_limits_cpuInt followed by m units e.g. 600m"Limit of CPU usage
e.g. 600m
where the unit suffix m stands for core thousandth.
akeyless/agent_limits_memInt followed by Mi units e.g. 64MiLimit of Memory usage.
akeyless/agent_requests_cpuInt followed by m units e.g. 250m"Limit of CPU request
e.g. 250m
where the unit suffix m stands for core thousandth.
akeyless/agent_requests_memInt followed by Mi units e.g. 64MiLimit of Memory requests usage.

The following examples demonstrate the usage of Akeyless annotations:

Files:

akeyless/inject_file: "/k8s-secrets/secret1|location=/tmp/s1|permission=0777|decode=base64,/k8s-secrets/secret2,/k8s-secrets/secret3|version=3|location=/tmp/s2"

Environment variable:

- name:  MY_SECRET
    value: 'akeyless:/k8s-secrets/secret1|decode=base64|version=2'

A secret that contains a JSON structured data for example:

akeyless create-secret --name k8s-secrets/secret-json --value '{"aws_access_key":"1234","aws_key_id":"abcd"}'

Can be fetched and parsed directly within the annotation for example to fetch only the aws_access_key key\value:

- name:  MY_JSON_SECRET
    value: 'akeyless:/k8s-secrets/secret-json|jq=.aws_access_key'

Troubleshooting

When you are working with a GKE cluster, make sure that port 8443 is opened in your firewall rules, This port is needed by the Akeyless Secret Injection mutating webhook. Update your firewall rule as follow:

  1. Review the firewall rule for master access:
gcloud compute firewall-rules list
  1. Replace the existing rule and allow master access:
gcloud compute firewall-rules update <firewall-rule-name> --allow tcp:10250,tcp:443,tcp:8443