Vault

From Christoph's Personal Wiki
Revision as of 17:31, 15 April 2020 by Christoph (Talk | contribs)

Jump to: navigation, search

HashiCorp Vault is a tool that provides secrets management, identity-based access, encrypting application data and auditing of secrets for applications, systems, and users.

Vault on Kubernetes using S3 Storage and KMS Auto-Unseal

This section will show how to set up and install HashiCorp Vault on Kubernetes using AWS S3 as the storage backend and AWS KMS to auto-unseal Vault.

Install Vault on Kubernetes

NOTE: The following will only work using Helm v3.

We will use the official vault-helm Helm Chart by HashiCorp.

  • Create a Kubernetes Secret to store your AWS access/secret keys:
$ kubectl create secret generic eks-creds \
    --from-literal=AWS_ACCESS_KEY_ID="${AWS_ACCESS_KEY_ID?}" \
    --from-literal=AWS_SECRET_ACCESS_KEY="${AWS_SECRET_ACCESS_KEY?}"

Make sure your vault-helm Helm Chart's values.yaml file has the following sections in it (add/update them if necessary): 

server:
...
  extraSecretEnvironmentVars:
    - envName: AWS_ACCESS_KEY_ID
      secretName: eks-creds
      secretKey: AWS_ACCESS_KEY_ID
    - envName: AWS_SECRET_ACCESS_KEY
      secretName: eks-creds
      secretKey: AWS_SECRET_ACCESS_KEY
...
  ha:
    enabled: true
    replicas: 3
    config: |
      ui = true

      listener "tcp" {
        tls_disable = 1
        address = "[::]:8200"
        cluster_address = "[::]:8201"
      }

      seal "awskms" {
        region     = "us-west-2"
        access_key = "UPDATE_ME"
        secret_key = "UPDATE_ME"
        kms_key_id = "UPDATE_ME"
      }

      storage "s3" {
        region     = "us-west-2"
        access_key = "UPDATE_ME"
        secret_key = "UPDATE_ME"
        bucket     = "UPDATE_ME"
      }
...
See: awskms and S3 Storage Backend for details.
  • Install Vault on your Kubernetes cluster:
$ helm3 install vault --set "server.ha.enabled=true" -f values.yaml .
  • Check on status of install:
$ helm3 ls
NAME 	NAMESPACE	REVISION	UPDATED                                	STATUS  	CHART      	APP VERSION
vault	default  	1       	2020-04-08 13:44:52.471357974 -0700 PDT	deployed	vault-0.4.0

$ kubectl get pods -l app.kubernetes.io/name=vault
NAME      READY   STATUS    RESTARTS   AGE
vault-0   0/1     Running   0          1m23
vault-1   0/1     Running   0          1m23
vault-2   0/1     Running   0          1m23

The above Pods should all report as "READY 0/1". You must first initialize Vault and unseal each member.

Setup Vault

  • Initialize Vault:
$ kubectl exec -ti vault-0 -- vault operator init
Recovery Key 1: r/rhD/ur9mx8rQHAYO3RSF/Y2/ceu340CVyvurwf2XFD
Recovery Key 2: A85k/b6N0LC5E2sXjBrq6FzrPmKa0//WWGEW3dQ56vry
Recovery Key 3: DS61ElZPPJFh947Jn6WQbj6Ny0SQEHI5TCg0CMU/+q3h
Recovery Key 4: gG3FYFWdum6D7z8hfG0H/S0Fq7NHmf369eV3ffQauQQM
Recovery Key 5: e8utfn95qEftFOEa2+k3aLAFPmKxqWk+25fFz1eBx9qK

Initial Root Token: s.AZ89jp5FafLGx2BPxc7OTT6G

Success! Vault is initialized
  • Unseal the Vault server with the key shares until the key threshold is met:
$ kubectl exec -ti vault-0 -- vault operator unseal # ... Unseal Key 1
$ kubectl exec -ti vault-0 -- vault operator unseal # ... Unseal Key 2
$ kubectl exec -ti vault-0 -- vault operator unseal # ... Unseal Key 3
  • When all Vault server Pods are unsealed, they will report as READY 1/1:
$ kubectl get pods -l app.kubernetes.io/name=vault
NAME      READY   STATUS    RESTARTS   AGE
vault-0   1/1     Running   0          1m54
vault-1   1/1     Running   0          1m54
vault-2   1/1     Running   0          1m54

Create a Secret

  • Start an interactive shell session in the vault-0 Pod:
$ kubectl exec -it vault-0 /bin/sh
/ $

and run the rest of the commands in this section from within that Pod.

  • Enable kv-v2 secrets at the path internal:
/ $ vault secrets enable -path=internal kv-v2
Success! Enabled the kv-v2 secrets engine at: internal/
  • Add a username and password secret at the path internal/exampleapp/config:
/ $ vault kv put internal/database/config username="db-readonly-username" password="db-secret-password"
Key              Value
---              -----
created_time     2020-04-13T23:27:50.503798447Z
deletion_time    n/a
destroyed        false
version          1
  • Verify that the secret is defined at the path internal/database/config:
/ $ vault kv get internal/database/config
====== Metadata ======
Key              Value
---              -----
created_time     2020-04-13T23:27:50.503798447Z
deletion_time    n/a
destroyed        false
version          1

====== Data ======
Key         Value
---         -----
password    db-secret-password
username    db-readonly-username

Make Kubernetes and Vault aware of each other

Vault provides a Kubernetes authentication method that enables clients to authenticate with a Kubernetes Service Account Token.

  • Enable the Kubernetes authentication method:
/ $ vault auth enable kubernetes
Success! Enabled kubernetes auth method at: kubernetes/

Vault accepts this service token from any client within the Kubernetes cluster. During authentication, Vault verifies that the service account token is valid by querying a configured Kubernetes endpoint.

  • Configure the Kubernetes authentication method to use the service account token, the location of the Kubernetes host, and its certificate:
/ $ vault write auth/kubernetes/config \
        token_reviewer_jwt="$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" \
        kubernetes_host="https://$KUBERNETES_PORT_443_TCP_ADDR:443" \
        kubernetes_ca_cert=@/var/run/secrets/kubernetes.io/serviceaccount/ca.crt
Success! Data written to: auth/kubernetes/config

The token_reviewer_jwt and kubernetes_ca_cert reference files written to the container by Kubernetes. The environment variable KUBERNETES_PORT_443_TCP_ADDR references the internal network address of the Kubernetes host. For a client to read the secret data defined in the previous step, at internal/database/config, requires that the read capability be granted for the path internal/data/database/config.

  • Write out the policy named internal-app that enables the read capability for secrets at path internal/data/database/config:
/ $ vault policy write internal-app - <<EOH
path "internal/data/database/config" {
  capabilities = ["read"]
}
EOH
Success! Uploaded policy: internal-app
  • Create a Kubernetes authentication role named internal-app:
/ $ vault write auth/kubernetes/role/internal-app \
        bound_service_account_names=internal-app \
        bound_service_account_namespaces=default \
        policies=internal-app \
        ttl=24h
Success! Data written to: auth/kubernetes/role/internal-app

The role connects the Kubernetes Service Account, internal-app, and Namespace, default, with the Vault policy, internal-app. The tokens returned after authentication are valid for 24 hours.

  • Lastly, exit the vault-0 Pod:
/ $ exit
$

Create Kubernetes Service Accounts for Vault

The Vault Kubernetes authentication role defined a Kubernetes Service Account named internal-app. This Service Account does not yet exist.

  • Create a Vault authentication Service Account:
$ cat <<EOF | kubectl create -f -
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: role-tokenreview-binding
  namespace: default
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:auth-delegator
subjects:
- kind: ServiceAccount
  name: vault-auth
  namespace: default
EOF
  • Describe the above Vault authentication Service Account:
$ kubectl describe serviceaccount vault-auth
Name:                vault-auth
Namespace:           default
Labels:              <none>
Annotations:         <none>
Image pull secrets:  <none>
Mountable secrets:   vault-auth-token-l8zcw
Tokens:              vault-auth-token-l8zcw
Events:              <none>
  • Create a Service Account for our test internal-app:
$ cat <<EOF | kubectl create -f -
apiVersion: v1
kind: ServiceAccount
metadata:
  name: internal-app
EOF

NOTE: The name of the service account here aligns with the name assigned to the bound_service_account_names field when creating the internal-app role when configuring the Kubernetes authentication.

Secret Injection from sidecar to application

  • Create a Deployment using a sample orgchart app:
$ cat <<EOF | kubectl create -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: orgchart
  labels:
    app: vault-agent-injector-demo
spec:
  selector:
    matchLabels:
      app: vault-agent-injector-demo
  replicas: 1
  template:
    metadata:
      annotations:
      labels:
        app: vault-agent-injector-demo
    spec:
      serviceAccountName: internal-app
      containers:
        - name: orgchart
          image: jweissig/app:0.0.1
EOF
The spec.template.spec.serviceAccountName defines the service account internal-app to run this container under.
  • Get all the Pods within the default Namespace:
$ kubectl get pods
NAME                                    READY   STATUS    RESTARTS   AGE
orgchart-6757598f54-vfkff               1/1     Running   0          31m
vault-0                                 1/1     Running   0          5d3h
vault-1                                 1/1     Running   0          5d3h
vault-2                                 1/1     Running   0          5d3h
vault-agent-injector-5b4445f78c-76vvk   1/1     Running   0          5d3h

The Vault-Agent injector looks for Deployments that define specific annotations. None of these annotations exists within the current Deployment. This means that no Secrets are present on the orgchart container within the orgchart-6757598f54-vfkff Pod.

  • Verify that no secrets are written to the orgchart container in the orgchart-6757598f54-vfkff Pod:
$ kubectl exec orgchart-6757598f54-vfkff --container orgchart -- ls /vault/secrets
ls: /vault/secrets: No such file or directory
command terminated with exit code 1

The Deployment is running the Pod with the internal-app Kubernetes Service Account in the default Namespace. The Vault Agent injector only modifies a deployment if it contains a very specific set of annotations. An existing deployment may have its definition patched to include the necessary annotations.

  • Create a Deployment patch file:
$ cat << EOF > inject_secrets.yaml
spec:
  template:
    metadata:
      annotations:
        vault.hashicorp.com/agent-inject: "true"
        vault.hashicorp.com/role: "internal-app"
        vault.hashicorp.com/agent-inject-secret-database-config.txt: "internal/data/database/config"
EOF

These annotations define a partial structure of the deployment schema and are prefixed with vault.hashicorp.com.

  • agent-inject enables the Vault Agent injector service
  • role is the Vault Kubernetes authentication role
  • role is also the Vault role created that maps back to the Kubernetes Service Account
  • agent-inject-secret-FILEPATH prefixes the path of the file, database-config.txt, written to /vault/secrets. The value is the path to the secret defined in Vault.
  • Patch the orgchart deployment defined in the inject_secrets.yaml manifest created above:
$ kubectl patch deployment orgchart --patch "$(cat inject-secrets.yml)"
deployment.apps/orgchart patched

This new Pod now launches two containers. The application container, named orgchart, and the Vault Agent container, named vault-agent. 

$ kubectl get pods
NAME                        READY   STATUS    RESTARTS   AGE
orgchart-6757598f54-vfkff   2/2     Running   0          31m
...
  • View the logs of the vault-agent container in the orgchart- Pod:
$ kubectl logs orgchart-6757598f54-vfkff --container vault-agent
==> Vault server started! Log data will stream in below:

==> Vault agent configuration:

                     Cgo: disabled
               Log Level: info
                 Version: Vault v1.3.2

2020-04-14T00:11:00.195Z [INFO]  sink.file: creating file sink
2020-04-14T00:11:00.195Z [INFO]  sink.file: file sink configured: path=/home/vault/.token mode=-rw-r-----
2020-04-14T00:11:00.196Z [INFO]  template.server: starting template server
2020/04/14 00:11:00.196068 [INFO] (runner) creating new runner (dry: false, once: false)
2020/04/14 00:11:00.196374 [INFO] (runner) creating watcher
2020-04-14T00:11:00.196Z [INFO]  auth.handler: starting auth handler
2020-04-14T00:11:00.196Z [INFO]  auth.handler: authenticating
2020-04-14T00:11:00.197Z [INFO]  sink.server: starting sink server
2020-04-14T00:11:00.313Z [INFO]  auth.handler: authentication successful, sending token to sinks
2020-04-14T00:11:00.313Z [INFO]  auth.handler: starting renewal process
2020-04-14T00:11:00.313Z [INFO]  sink.file: token written: path=/home/vault/.token
2020-04-14T00:11:00.313Z [INFO]  template.server: template server received new token
2020/04/14 00:11:00.313503 [INFO] (runner) stopping
2020/04/14 00:11:00.313548 [INFO] (runner) creating new runner (dry: false, once: false)
2020/04/14 00:11:00.313648 [INFO] (runner) creating watcher
2020/04/14 00:11:00.313701 [INFO] (runner) starting
2020-04-14T00:11:00.406Z [INFO]  auth.handler: renewed auth token
  • View the secret written to the orgchart container:
$ kubectl exec orgchart-6757598f54-vfkff --container orgchart -- cat /vault/secrets/database-config.txt
data: map[password:db-secret-password username:db-readonly-username]
metadata: map[created_time:2020-04-13T23:27:50.503798447Z deletion_time: destroyed:false version:1]

The secret is successfully present in the container.

External links

Retrieved from "http://wiki.christophchamp.com/index.php?title=Vault&oldid=7780"