Kubernetes/AWS

This article will cover topics related to Kubernetes running on AWS, whether running on EKS or stand-alone EC2 instances, etc.

Enable ELB Access Logs via Kubernetes Service

Setup details

• Kubernetes v1.17.3
• kubectl v1.17.3
• 1 x EC2 instance (Ubuntu 16.04) => k8s master+worker node

Initial steps

• First, setup some environment variables:

$ MY_ELB_LOGS_BUCKET=my-elb-logs
$ ELB_ACCOUNT_ID=797873946194  # <- us-west-2

You can find the appropriate ${ELB_ACCOUNT_ID} here.

• Create an S3 bucket in which to host your ELB logs:

$ aws s3 mb s3://${MY_ELB_LOGS_BUCKET}

• Make sure this S3 bucket as the following bucket policy (set under the permissions):

$ cat <<EOF >policy.json
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::${ELB_ACCOUNT_ID}:root"
            },
            "Action": "s3:PutObject",
            "Resource": "arn:aws:s3:::${MY_ELB_LOGS_BUCKET}/*"
        }
    ]
}
EOF

$ aws s3api put-bucket-policy --bucket ${MY_ELB_LOGS_BUCKET} --policy file://policy.json

Kubernetes setup

• Create test Nginx Deployment:

$ cat <<EOF | kubectl create -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  labels:
    app: nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.17.9
        ports:
        - containerPort: 80
EOF

• Create a Service to put in front of above Deployment:

$ cat <<EOF | kubectl create -f -
apiVersion: v1
kind: Service
metadata:
  name: frontdoor-service
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-access-log-enabled: "true"
    # The interval for publishing the access logs (can be 5 or 60 minutes).
    service.beta.kubernetes.io/aws-load-balancer-access-log-emit-interval: "5"
    service.beta.kubernetes.io/aws-load-balancer-access-log-s3-bucket-name: "${MY_ELB_LOGS_BUCKET}"
    service.beta.kubernetes.io/aws-load-balancer-access-log-s3-bucket-prefix: "logs/frontdoor"
  labels:
    app: frontdoor
spec:
  type: LoadBalancer
  ports:
  - name: frontdoorport
    port: 30010
    targetPort: 30010
  selector:
    app: nginx
EOF

• Get information on Service created:

$ kubectl get svc frontdoor-service
NAME                TYPE           CLUSTER-IP     EXTERNAL-IP                                                              PORT(S)           AGE
frontdoor-service   LoadBalancer   10.43.184.39   a371dfd887b56468fa65e126e0d03500-527425434.us-west-2.elb.amazonaws.com   30010:30526/TCP   62m

$ kubectl describe svc frontdoor-service
Name:                     frontdoor-service
Namespace:                default
Labels:                   app=frontdoor
Annotations:              service.beta.kubernetes.io/aws-load-balancer-access-log-emit-interval: 5
                          service.beta.kubernetes.io/aws-load-balancer-access-log-enabled: true
                          service.beta.kubernetes.io/aws-load-balancer-access-log-s3-bucket-name: my-elb-logs
                          service.beta.kubernetes.io/aws-load-balancer-access-log-s3-bucket-prefix: logs/frontdoor
...

AWS details

• Describe AWS Load Balancer (ELB) Kubernetes automatically created for us:

$ aws elb describe-load-balancer-attributes \
  --profile default \
  --region us-west-2 \
  --load-balancer-name a371dfd887b56468fa65e126e0d03500
{
    "LoadBalancerAttributes": {
        "ConnectionDraining": {
            "Enabled": false,
            "Timeout": 300
        },
        "CrossZoneLoadBalancing": {
            "Enabled": false
        },
        "ConnectionSettings": {
            "IdleTimeout": 60
        },
        "AccessLog": {
            "S3BucketPrefix": "logs/frontdoor",
            "EmitInterval": 5,
            "Enabled": true,
            "S3BucketName": "my-elb-logs"
        }
    }
}

• Interact with that ELB DNS name so we can generate some traffic for our access logs:

$ ab -c100 -n20000 http://a371dfd887b56468fa65e126e0d03500-527425434.us-west-2.elb.amazonaws.com:30010/

$ for i in $(seq 1 100); do curl -sI http://a371dfd887b56468fa65e126e0d03500-527425434.us-west-2.elb.amazonaws.com:30010/ | grep ^HTTP; done

• Check that the S3 bucket has ELB access logs:

$ aws s3 ls \
  --profile default \
  --recursive \
  s3://${MY_ELB_LOGS_BUCKET}/logs/frontdoor/
2020-03-04 16:05:12         86 logs/frontdoor/AWSLogs/<redacted>/ELBAccessLogTestFile
2020-03-04 16:25:16        156 logs/frontdoor/AWSLogs/<redacted>/elasticloadbalancing/us-west-2/2020/03/05/<redacted>_elasticloadbalancing_us-west-2_a371dfd887b56468fa65e126e0d03500_20200305T0025Z_54.39.161.151_4jmuxnr9.log
2020-03-04 16:25:31      15434 logs/frontdoor/AWSLogs/<redacted>/elasticloadbalancing/us-west-2/2020/03/05/<redacted>_elasticloadbalancing_us-west-2_a371dfd887b56468fa65e126e0d03500_20200305T0025Z_52.216.39.65_2tv1rd8u.log

• View the contents of one of those access logs:

$ aws --profile default s3 cp \
  s3://${MY_ELB_LOGS_BUCKET}/logs/frontdoor/AWSLogs/<redacted>/elasticloadbalancing/us-west-2/2020/03/05/<redacted>_elasticloadbalancing_us-west-2_a173dfd887b56468fa65e126e0d03500_20200305T0025Z_52.216.39.65_2tv1rd8u.log - | head -3
2020-03-05T00:22:25.152094Z a371dfd887b56468fa65e126e0d03500 70.104.137.198:35200 10.10.0.167:30526 0.000432 0.000006 0.000015 - - 141 238 "- - - " "-" - -
2020-03-05T00:22:25.243193Z a371dfd887b56468fa65e126e0d03500 70.104.137.198:22800 10.10.0.167:30526 0.000518 0.000007 0.000016 - - 141 238 "- - - " "-" - -
2020-03-05T00:22:25.282568Z a371dfd887b56468fa65e126e0d03500 70.104.137.198:22801 10.10.0.167:30526 0.000422 0.000005 0.000014 - - 141 238 "- - - " "-" - -

Related links

• Kubernetes Cloud Providers - AWS
• Enable Access Logs for Your Classic Load Balancer
• Rancher - Setting up Cloud Providers