Difference between revisions of "Vault"

Latest revision as of 21:16, 15 April 2020

HashiCorp Vault is a tool that provides secrets management, identity-based access, encrypting application data and auditing of secrets for applications, systems, and users.

Vault on Kubernetes using S3 Storage and KMS Auto-Unseal

This section will show how to set up and install HashiCorp Vault on Kubernetes using AWS S3 as the storage backend and AWS KMS to auto-unseal Vault.

Install Vault on Kubernetes

NOTE: The following will only work using Helm v3.

We will use the official vault-helm Helm Chart by HashiCorp.

Create a Kubernetes Secret to store your AWS access/secret keys:

$ kubectl create secret generic eks-creds \
    --from-literal=AWS_ACCESS_KEY_ID="${AWS_ACCESS_KEY_ID?}" \
    --from-literal=AWS_SECRET_ACCESS_KEY="${AWS_SECRET_ACCESS_KEY?}"

Make sure your vault-helm Helm Chart's values.yaml file has the following sections in it (add/update them if necessary):

server:
...
  extraSecretEnvironmentVars:
    - envName: AWS_ACCESS_KEY_ID
      secretName: eks-creds
      secretKey: AWS_ACCESS_KEY_ID
    - envName: AWS_SECRET_ACCESS_KEY
      secretName: eks-creds
      secretKey: AWS_SECRET_ACCESS_KEY
...
  ha:
    enabled: true
    replicas: 3
    config: |
      ui = true

      listener "tcp" {
        tls_disable = 1
        address = "[::]:8200"
        cluster_address = "[::]:8201"
      }

      seal "awskms" {
        region     = "us-west-2"
        access_key = "UPDATE_ME"
        secret_key = "UPDATE_ME"
        kms_key_id = "UPDATE_ME"
      }

      storage "s3" {
        region     = "us-west-2"
        access_key = "UPDATE_ME"
        secret_key = "UPDATE_ME"
        bucket     = "UPDATE_ME"
      }
...

See: awskms and S3 Storage Backend for details.

Install Vault on your Kubernetes cluster:

$ helm3 install vault --set "server.ha.enabled=true" -f values.yaml .

Check on status of install:

$ helm3 ls
NAME 	NAMESPACE	REVISION	UPDATED                                	STATUS  	CHART      	APP VERSION
vault	default  	1       	2020-04-08 13:44:52.471357974 -0700 PDT	deployed	vault-0.4.0

$ kubectl get pods -l app.kubernetes.io/name=vault
NAME      READY   STATUS    RESTARTS   AGE
vault-0   0/1     Running   0          1m23
vault-1   0/1     Running   0          1m23
vault-2   0/1     Running   0          1m23

The above Pods should all report as "READY 0/1". You must first initialize Vault and unseal each member.

Setup Vault

Initialize Vault:

$ kubectl exec -ti vault-0 -- vault operator init
Recovery Key 1: r/rhD/ur9mx8rQHAYO3RSF/Y2/ceu340CVyvurwf2XFD
Recovery Key 2: A85k/b6N0LC5E2sXjBrq6FzrPmKa0//WWGEW3dQ56vry
Recovery Key 3: DS61ElZPPJFh947Jn6WQbj6Ny0SQEHI5TCg0CMU/+q3h
Recovery Key 4: gG3FYFWdum6D7z8hfG0H/S0Fq7NHmf369eV3ffQauQQM
Recovery Key 5: e8utfn95qEftFOEa2+k3aLAFPmKxqWk+25fFz1eBx9qK

Initial Root Token: s.AZ89jp5FafLGx2BPxc7OTT6G

Success! Vault is initialized

Unseal the Vault server with the key shares until the key threshold is met:

$ kubectl exec -ti vault-0 -- vault operator unseal # ... Unseal Key 1
$ kubectl exec -ti vault-0 -- vault operator unseal # ... Unseal Key 2
$ kubectl exec -ti vault-0 -- vault operator unseal # ... Unseal Key 3

When all Vault server Pods are unsealed, they will report as READY 1/1:

$ kubectl get pods -l app.kubernetes.io/name=vault
NAME      READY   STATUS    RESTARTS   AGE
vault-0   1/1     Running   0          1m54
vault-1   1/1     Running   0          1m54
vault-2   1/1     Running   0          1m54

Create a Secret

Start an interactive shell session in the vault-0 Pod:

$ kubectl exec -it vault-0 /bin/sh
/ $

and run the rest of the commands in this section from within that Pod.

Enable kv-v2 secrets at the path internal:

/ $ vault secrets enable -path=internal kv-v2
Success! Enabled the kv-v2 secrets engine at: internal/

Add a username and password secret at the path internal/exampleapp/config:

/ $ vault kv put internal/database/config username="db-readonly-username" password="db-secret-password"
Key              Value
---              -----
created_time     2020-04-13T23:27:50.503798447Z
deletion_time    n/a
destroyed        false
version          1

Verify that the secret is defined at the path internal/database/config:

/ $ vault kv get internal/database/config
====== Metadata ======
Key              Value
---              -----
created_time     2020-04-13T23:27:50.503798447Z
deletion_time    n/a
destroyed        false
version          1

====== Data ======
Key         Value
---         -----
password    db-secret-password
username    db-readonly-username

Make Kubernetes and Vault aware of each other

Vault provides a Kubernetes authentication method that enables clients to authenticate with a Kubernetes Service Account Token.

Enable the Kubernetes authentication method:

/ $ vault auth enable kubernetes
Success! Enabled kubernetes auth method at: kubernetes/

Vault accepts this service token from any client within the Kubernetes cluster. During authentication, Vault verifies that the service account token is valid by querying a configured Kubernetes endpoint.

Configure the Kubernetes authentication method to use the service account token, the location of the Kubernetes host, and its certificate:

/ $ vault write auth/kubernetes/config \
        token_reviewer_jwt="$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" \
        kubernetes_host="https://$KUBERNETES_PORT_443_TCP_ADDR:443" \
        kubernetes_ca_cert=@/var/run/secrets/kubernetes.io/serviceaccount/ca.crt
Success! Data written to: auth/kubernetes/config

The token_reviewer_jwt and kubernetes_ca_cert reference files written to the container by Kubernetes. The environment variable KUBERNETES_PORT_443_TCP_ADDR references the internal network address of the Kubernetes host. For a client to read the secret data defined in the previous step, at internal/database/config, requires that the read capability be granted for the path internal/data/database/config.

Write out the policy named internal-app that enables the read capability for secrets at path internal/data/database/config:

/ $ vault policy write internal-app - <<EOH
path "internal/data/database/config" {
  capabilities = ["read"]
}
EOH
Success! Uploaded policy: internal-app

Create a Kubernetes authentication role named internal-app:

/ $ vault write auth/kubernetes/role/internal-app \
        bound_service_account_names=internal-app \
        bound_service_account_namespaces=default \
        policies=internal-app \
        ttl=24h
Success! Data written to: auth/kubernetes/role/internal-app

The role connects the Kubernetes Service Account, internal-app, and Namespace, default, with the Vault policy, internal-app. The tokens returned after authentication are valid for 24 hours.

Lastly, exit the vault-0 Pod:

/ $ exit
$

Create Kubernetes Service Accounts for Vault

The Vault Kubernetes authentication role defined a Kubernetes Service Account named internal-app. This Service Account does not yet exist.

Create a Vault authentication Service Account:

$ cat <<EOF | kubectl create -f -
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: role-tokenreview-binding
  namespace: default
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:auth-delegator
subjects:
- kind: ServiceAccount
  name: vault-auth
  namespace: default
EOF

Describe the above Vault authentication Service Account:

$ kubectl describe serviceaccount vault-auth
Name:                vault-auth
Namespace:           default
Labels:              <none>
Annotations:         <none>
Image pull secrets:  <none>
Mountable secrets:   vault-auth-token-l8zcw
Tokens:              vault-auth-token-l8zcw
Events:              <none>

Create a Service Account for our test internal-app:

$ cat <<EOF | kubectl create -f -
apiVersion: v1
kind: ServiceAccount
metadata:
  name: internal-app
EOF

NOTE: The name of the service account here aligns with the name assigned to the bound_service_account_names field when creating the internal-app role when configuring the Kubernetes authentication.

Secret Injection from sidecar to application

Create a Deployment using a sample orgchart app:

$ cat <<EOF | kubectl create -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: orgchart
  labels:
    app: vault-agent-injector-demo
spec:
  selector:
    matchLabels:
      app: vault-agent-injector-demo
  replicas: 1
  template:
    metadata:
      annotations:
      labels:
        app: vault-agent-injector-demo
    spec:
      serviceAccountName: internal-app
      containers:
        - name: orgchart
          image: jweissig/app:0.0.1
EOF

The spec.template.spec.serviceAccountName defines the service account internal-app to run this container under.

Get all the Pods within the default Namespace:

$ kubectl get pods
NAME                                    READY   STATUS    RESTARTS   AGE
orgchart-6757598f54-vfkff               1/1     Running   0          31m
vault-0                                 1/1     Running   0          5d3h
vault-1                                 1/1     Running   0          5d3h
vault-2                                 1/1     Running   0          5d3h
vault-agent-injector-5b4445f78c-76vvk   1/1     Running   0          5d3h

The Vault-Agent injector looks for Deployments that define specific annotations. None of these annotations exists within the current Deployment. This means that no Secrets are present on the orgchart container within the orgchart-6757598f54-vfkff Pod.

Verify that no secrets are written to the orgchart container in the orgchart-6757598f54-vfkff Pod:

$ kubectl exec orgchart-6757598f54-vfkff --container orgchart -- ls /vault/secrets
ls: /vault/secrets: No such file or directory
command terminated with exit code 1

The Deployment is running the Pod with the internal-app Kubernetes Service Account in the default Namespace. The Vault Agent injector only modifies a deployment if it contains a very specific set of annotations. An existing deployment may have its definition patched to include the necessary annotations.

Create a Deployment patch file:

$ cat << EOF > inject_secrets.yaml
spec:
  template:
    metadata:
      annotations:
        vault.hashicorp.com/agent-inject: "true"
        vault.hashicorp.com/role: "internal-app"
        vault.hashicorp.com/agent-inject-secret-database-config.txt: "internal/data/database/config"
EOF

These annotations define a partial structure of the deployment schema and are prefixed with vault.hashicorp.com.

agent-inject enables the Vault Agent injector service
role is the Vault Kubernetes authentication role
role is also the Vault role created that maps back to the Kubernetes Service Account
agent-inject-secret-FILEPATH prefixes the path of the file, database-config.txt, written to /vault/secrets. The value is the path to the secret defined in Vault.

Patch the orgchart deployment defined in the inject_secrets.yaml manifest created above:

$ kubectl patch deployment orgchart --patch "$(cat inject-secrets.yml)"
deployment.apps/orgchart patched

This new Pod now launches two containers. The application container, named orgchart, and the Vault Agent container, named vault-agent.

$ kubectl get pods
NAME                        READY   STATUS    RESTARTS   AGE
orgchart-6757598f54-vfkff   2/2     Running   0          31m
...

View the logs of the vault-agent container in the orgchart- Pod:

$ kubectl logs orgchart-6757598f54-vfkff --container vault-agent
==> Vault server started! Log data will stream in below:

==> Vault agent configuration:

                     Cgo: disabled
               Log Level: info
                 Version: Vault v1.3.2

2020-04-14T00:11:00.195Z [INFO]  sink.file: creating file sink
2020-04-14T00:11:00.195Z [INFO]  sink.file: file sink configured: path=/home/vault/.token mode=-rw-r-----
2020-04-14T00:11:00.196Z [INFO]  template.server: starting template server
2020/04/14 00:11:00.196068 [INFO] (runner) creating new runner (dry: false, once: false)
2020/04/14 00:11:00.196374 [INFO] (runner) creating watcher
2020-04-14T00:11:00.196Z [INFO]  auth.handler: starting auth handler
2020-04-14T00:11:00.196Z [INFO]  auth.handler: authenticating
2020-04-14T00:11:00.197Z [INFO]  sink.server: starting sink server
2020-04-14T00:11:00.313Z [INFO]  auth.handler: authentication successful, sending token to sinks
2020-04-14T00:11:00.313Z [INFO]  auth.handler: starting renewal process
2020-04-14T00:11:00.313Z [INFO]  sink.file: token written: path=/home/vault/.token
2020-04-14T00:11:00.313Z [INFO]  template.server: template server received new token
2020/04/14 00:11:00.313503 [INFO] (runner) stopping
2020/04/14 00:11:00.313548 [INFO] (runner) creating new runner (dry: false, once: false)
2020/04/14 00:11:00.313648 [INFO] (runner) creating watcher
2020/04/14 00:11:00.313701 [INFO] (runner) starting
2020-04-14T00:11:00.406Z [INFO]  auth.handler: renewed auth token

View the secret written to the orgchart container:

$ kubectl exec orgchart-6757598f54-vfkff --container orgchart -- cat /vault/secrets/database-config.txt
data: map[password:db-secret-password username:db-readonly-username]
metadata: map[created_time:2020-04-13T23:27:50.503798447Z deletion_time: destroyed:false version:1]

The secret is successfully present in the container.

Check the Vault Storage Backend

Since we are using AWS S3 as our Vault Storage Backend, we can view the contents of the S3 bucket we created/defined during the Vault installing process:

$ export AWS_PROFILE=default
$ export AWS_REGION=us-west-2

$ aws --profile ${AWS_PROFILE} \
      --region ${AWS_REGION} \
      s3 ls \
      s3://${MY_S3_BUCKET_NAME}

  PRE auth/
  PRE core/
  PRE logical/
  PRE sys/

#~OR~

$ aws --profile ${AWS_PROFILE} \
      --region ${AWS_REGION} \
      s3 ls \
      s3://${MY_S3_BUCKET_NAME}/auth/aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee/role/

2020-04-13 16:48:55        460 internal-app

#~OR~

$ aws --profile ${AWS_PROFILE} \
      --region ${AWS_REGION} \
      s3 ls \
      s3://${MY_S3_BUCKET_NAME}

2020-04-08 15:04:47       2055 auth/aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee/config
2020-04-13 16:48:55        460 auth/aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee/role/internal-app
2020-04-08 15:05:13        431 auth/aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee/role/myapp
2020-04-08 13:45:56         90 core/audit
2020-04-08 15:00:44        423 core/auth
2020-04-08 13:45:55        110 core/cluster/local/info
2020-04-08 13:45:55        349 core/hsm/barrier-unseal-keys
2020-04-08 13:45:55        227 core/keyring
2020-04-08 13:45:56         90 core/local-audit
2020-04-08 13:45:56         89 core/local-auth
2020-04-08 13:45:55        354 core/local-mounts
2020-04-08 13:45:55        147 core/master
2020-04-13 16:27:40        571 core/mounts
2020-04-08 13:45:56        116 core/recovery-config
2020-04-08 13:45:56        333 core/recovery-key
2020-04-08 13:45:55        116 core/seal-config
2020-04-08 13:45:55        534 core/wrapping/jwtkey
2020-04-13 16:27:40        480 logical/aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee/aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee/archive/metadata
2020-04-13 16:27:51        100 logical/aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee/aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee/metadata/p0CK...
2020-04-13 16:27:40        767 logical/aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee/aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee/policy/metadata
2020-04-13 16:27:51         69 logical/aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee/aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee/salt
2020-04-13 16:27:40         49 logical/aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee/aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee/upgrading
2020-04-13 16:27:51        116 logical/aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee/aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee/versions/6a2/c92...
2020-04-08 13:45:56         69 logical/aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee/casesensitivity
2020-04-13 16:48:57        469 logical/aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee/packer/buckets/94
2020-04-15 14:06:56         47 sys/counters/requests/2020/04
2020-04-15 03:32:50       1535 sys/expire/id/auth/kubernetes/login/h618...
2020-04-08 15:04:38        135 sys/policy/app
2020-04-08 13:45:56        249 sys/policy/control-group
2020-04-08 13:45:55       2553 sys/policy/default
2020-04-13 16:54:46        157 sys/policy/internal-app
2020-04-08 13:45:55        232 sys/policy/response-wrapping
2020-04-15 03:32:50        138 sys/token/accessor/0ce...
2020-04-08 13:45:56        138 sys/token/accessor/4bf...
2020-04-15 03:32:50        772 sys/token/id/h618...
2020-04-08 13:45:56        459 sys/token/id/hd1e...
2020-04-08 13:45:56         69 sys/token/salt

External links

Difference between revisions of "Vault"

Latest revision as of 21:16, 15 April 2020

Contents

Vault on Kubernetes using S3 Storage and KMS Auto-Unseal

Install Vault on Kubernetes

Setup Vault

Create a Secret

Make Kubernetes and Vault aware of each other

Create Kubernetes Service Accounts for Vault

Secret Injection from sidecar to application

Check the Vault Storage Backend

External links

Navigation menu

Personal tools

Namespaces

Variants

Views

More

Search

Navigation

Tools