AWS/DevOps

From Christoph's Personal Wiki
Revision as of 23:32, 18 February 2021 by Christoph (Talk | contribs) (Monitoring and Logging)

Jump to: navigation, search

This article will cover topics related to the AWS Certified DevOps Engineer - Professional exam and certification.

Domains

The 6 domains outlined in the AWS blueprint for the certification include:

  1. Software Development LifeCycle (SDLC) Automation [22%]
  2. Configuration Management and Infrastructure as Code [19%]
  3. Monitoring and Logging [15%]
  4. Policies and Standards Automation [10%]
  5. Incident and Event Response [18%]
  6. High Availability, Fault Tolerance, and Disaster Recovery [16%]

Domain 1: SDLC Automation

  • 1.1 Apply concepts required to automate a CI/CD pipeline
  • 1.2 Determine source control strategies and how to implement them
  • 1.3 Apply concepts required to automate and integrate testing
  • 1.4 Apply concepts required to build and manage artifacts securely
  • 1.5 Determine deployment/delivery strategies (e.g., A/B, Blue/Green, Canary, Red/Black) and how to implement them using AWS Services

Domain 2: Configuration Management and Infrastructure as Code

  • 2.1 Determine deployment services based on deployment needs
  • 2.2 Determine application and infrastructure deployment models based on business needs
  • 2.3 Apply security concepts in the automation of resource provisioning
  • 2.4 Determine how to implement lifecycle hooks on a deployment
  • 2.5 Apply concepts required to manage systems using AWS configuration management tools and services

Domain 3: Monitoring and Logging

  • 3.1 Determine how to set up the aggregation, storage, and analysis of logs and metrics
  • 3.2 Apply concepts required to automate monitoring and event management of an environment
  • 3.3 Apply concepts required to audit, log, and monitor operating systems, infrastructures, and applications
  • 3.4 Determine how to implement tagging and other metadata strategies

Domain 4: Policies and Standards Automation

  • 4.1 Apply concepts required to enforce standards for logging, metrics, monitoring, testing, and security
  • 4.2 Determine how to optimize cost through automation
  • 4.3 Apply concepts required to implement governance strategies

Domain 5: Incident and Event Response

  • 5.1 Troubleshoot issues and determine how to restore operations
  • 5.2 Determine how to automate event management and alerting
  • 5.3 Apply concepts required to implement automated healing
  • 5.4 Apply concepts required to set up event-driven automated actions

Domain 6: High Availability, Fault Tolerance, and Disaster Recovery

  • 6.1 Determine appropriate use of multi-AZ versus multi-region architectures
  • 6.2 Determine how to implement high availability, scalability, and fault tolerance
  • 6.3 Determine the right services based on business needs (e.g., RTO/RPO, cost)
  • 6.4 Determine how to design and automate disaster recovery strategies
  • 6.5 Evaluate a deployment for points of failure

SDLC Automation

Introduction

What is CI/CD?

The CI/CD Pipeline
  • AWS CodePipeline
    • Source Stage
      • AWS CodeCommit (think "git")
    • Deploy Stage - Development
      • AWS CodeDeploy -> EC2 instance
    • Deploy Stage - Production
      • AWS CodeDeploy -> EC2 instance

AWS CodeCommit

AWS CodeBuild

  • A fully managed build service
  • Compiles your code
  • Runs unit tests
  • Produces artifacts that are ready to deploy
  • Eliminates the need to provision/manage/scale your own build servers
  • Provides pre-packaged build environments
  • Allows you to build your own customized build environment
  • Scales automatically to meet your build requirements
Benefits of CodeBuild
  • It is fully managed
    • You do not have to set up any build servers, nor patch, update, or maintain them.
  • It is on-demand
    • It automatically scales to meet your requirements. No more migrating to larger EC2 servers because your builds are taking too long. You only pay for the minutes (seconds?) you consume.
  • It is preconfigured
    • It comes with many pre-configured build environments for the most popular programming languages. You just need to configure it to use your build script.
SEE: AWS - Troubleshooting CodeBuild

AWS CodeDeploy

What is CodeDeploy?
  • A fully managed deployment service that automates deployments to:
    • Amazon EC2 instances
    • On-premise instances
    • AWS Lambda functions
  • Makes it easier to:
    • Rapidly deploy new features
    • Update Lambda function versions
    • Avoid downtime during deployment
    • Handle the full complex deployment process without human intervention


AWS CodePipeline

CodePipeline is the "CD" of CI/CD.

Benefits
  • Automatic
    • From the check-in of your code to deployment on to your service, CodePipeline takes care of it all.
  • Easy to set up
    • CodePipeline has no servers to provision, it is dead simple to configure and get working. There are pre-built plugins or you can roll your own.
  • Configurable
    • You can create, configure, and modify all stages of your software release process with ease. You can implement automated testing and customize the deployment process.

Testing

Why do we test?
  • Meet the requirements defined
  • Ensure the code performs in an accepatble period of time
  • Ensure the code is usable
  • Ensure the code responds correctly to all kinds of inputs
  • Achieves the result the programmer desired
Types of testings (see Wikipedia)
Automated testing
  • Automatic execution oif test
  • Comparision of actual outcomes to predicted outcomes
  • Fast, continuos feedback
  • Immediate notifcation
  • Save resources
Unit test example
assert()

Artifacts

What are artifacts?

An artifact is a product or by-product produced during the software development process.

For example:

  • Compiled binaries
  • Source code
  • Documentation
  • Use cases
  • Class diagrams

Artifacts are stored in S3 (note: this has nothing to do with AWS Artifact!)

Deployment Strategies

Single Target Deployment (build -> target)
  • Use for small development projects, especially when legacy or non-highly-available infrastructure is involved.
  • When it is initiated, a new application version is installed on the target server.
  • A brief outage occurs during installation. There are no secondary servers, so testing is limited. Rollback involves removing the new version and install the previous.
All-at-Once Deployment (build -> x2 targets)
  • Deployment happens in one step, just like single target deployment.
  • With this method, the destination is multiple targets.
  • More complicated than single target; often requiring orchestration tooling.
  • Shares negatives of single target. No ability to test, still has deployment outages, and less than ideal rollback.
Minimum in-service Deployment (initial build stage -> t1 t2 t3 ...)
  • Deployment happens in multiple stages
  • Deployment happens to as many targets as possible while maintaining the minimum in-service targets.
  • A few moving parts, orchestration and health checks are required.
  • Allows automated testing, deployment targets are assessed and testsd prior to continuing.
  • Generally, no downtime.
  • Often quicker and less stages than a rolling deployment.
Rolling Deployment
  • Deployment happens in multiple stages. Number of targets per stage is user-defined.
  • Moving parts; orchestration and health-checks are required.
  • Overall applicable health is not necessariliy maintained.
  • Can be the leasat efficient deployment time based on time-taken.
  • Allows automated testing; deployment targets are assessed and tested prior to continuing.
  • Generally, no downtime, assuming number of targets per run is not large neough to impact the application.
  • Can be paused, allowing limited multi-version testing (combined with small targets per stage).
Blue/Green Deployment (aka Red/Black)
  • Requires advanced orchestration tooling
  • Carries significant cost - maintiang 2 environments for the duration of deployments.
  • Deployment process is rapid - entire environemnt (blue or green) is deployed all at once.
  • Cutover and migration is clean and controlled (e.g., DNS change)
  • Rollback is equally clean (e.g., DNS regression)
  • Health and performance of entire "green" environment can be tested prior to cutover.
  • using advanced template systems, such as CloudFormation, the entire process can be fully automated.
Canary Deployment
  • Like Blue/Green, but keep blue active and route percentage of traffic to green
  • In AWS, use Route53 w/weighted round-robin

Configuration Management and Infrastructure as Code

  • Introduction
  • AWS CloudFormation
  • AWS CloudFormation Intrinsic Functions
  • AWS CloudFormation Wait Conditions
  • AWS CloudFormation Nested Stacks
  • AWS CloudFormation Deletion Policies
  • AWS CloudFormation Stack Updates
  • AWS CloudFormation Change Sets
  • AWS CloudFormation Custom Resources
  • AWS Elastic Beanstalk
  • AWS Elastic Beanstalk extensions
  • AWS Config
  • Amazon ECS
  • AWS Managed Services
  • AWS Lambda
  • AWS Lambda Step Functions
  • AWS OpsWorks

Monitoring and Logging

  • Introduction

CloudWatch

$ aws cloudwatch put-metric-data \
    --metric-name randomNumber \
    --namespace Random \
    --value $(shuf -i 1-1000 -n1) \
    --region=us-west-2
  • CloudWatch Custom Metrics
  • CloudWatch Events

CloudWatch Logs

$ sudo dpkg -i amazon-cloudwatch-agent.deb
$ sudo vi /opt/aws/amazon-cloudwatch-agent/etc/common-config.toml
# create and edit config
$ sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -c file:cloudwatchconfig.cfg -s

AWS X-Ray

Think managed "Kiali + Jaeger"

Policies and Standards Automation

  • Introduction
  • AWS Service Catalog
  • AWS Trusted Advisor
  • AWS Systems Manager
  • AWS Organizations
  • AWS Secrets Manager
  • Amazon Macie
  • AWS Certificate Manager

Incident and Event Response

  • Introduction
  • Amazon GuardDuty
  • Amazon Inspector
  • Amazon Kinesis

High Availability, Fault Tolerance, and Disaster Recovery

  • Introduction
  • AWS Single Sign-On
  • Amazon CloudFront
  • AutoScaling and Lifecycle hooks
  • Amazon Route53
  • Amazon RDS
  • Amazon Aurora
  • Amazon DynamoDB
  • Amazon DynamoDB Keys and Streams

Other Services You Need to Know About

  • Introduction
  • Tagging
  • Amazon Elastic File System
  • Amazon ElastiCache
  • Amazon S3 Glacier
  • AWS Direct Connect
  • AWS Lambda Function Dead Letter Queues
  • Amazon CloudSearch
  • Amazon Elasticsearch Service
  • Amazon DynamoDB Accelerator
  • AWS Server Migration Service

External links