Difference between revisions of "AWS/DevOps"

From Christoph's Personal Wiki
Jump to: navigation, search
(Policies and Standards Automation)
(Incident and Event Response)
Line 297: Line 297:
 
==Incident and Event Response==
 
==Incident and Event Response==
  
* Introduction
+
===Introduction===
* Amazon GuardDuty
+
 
* Amazon Inspector
+
===Amazon GuardDuty===
* Amazon Kinesis
+
 
 +
GuardDuty is a threat-detection service that continuously monitors for malicious or unauthorized behaviour.
 +
 
 +
===Amazon Inspector===
 +
 
 +
Inspector is an automated service that assesses your applications for vulnerabilities and produces a security findings report.
 +
 
 +
===Amazon Kinesis===
 +
 
 +
Easily collect, process, and analyze video and data streams in real time.
 +
 
 +
; Services
 +
* Kinesis Data Analytics
 +
** Analyze streaming data
 +
** Respond in real-time
 +
** Query using SQL
 +
** Completely managed service (no servers required)
 +
** Pay-as-you-go for what you use
 +
** Powerful real-time processing
 +
* Kinesis Data Firehose
 +
** Deliver streaming data
 +
** No applications to write or manage
 +
** Just configure the producer
 +
** Data can be transformed
 +
** Destinations such as S3, Redshift, ElasiticSearch, and Splunk
 +
** Accepts records in chunks of up to 1,000 kb
 +
* Kinesis Data Streams
 +
** Collect streaming data
 +
** Massively scalable
 +
** Capture gigabytes per second (from thousands of sources)
 +
** Data is available in milliseconds
 +
** Durable (data in stored in 3 x DCs in a region)
 +
** Data is stored for 7 days
 +
** Elastic
 +
* Kinesis Video Streams
 +
** Collect streaming video
 +
** Can handle ingestion from millions of devices
 +
** Enables live and on-demand playback
 +
** Take advantage of Amazon Recognition Video and Machine Learning frameworks for video
 +
** Access your data through APIs
 +
** Build real-time video enabled applications
 +
 
 +
Review the tutorial: "Using AWS Lambda with Amazon Kinesis".
  
 
==High Availability, Fault Tolerance, and Disaster Recovery==
 
==High Availability, Fault Tolerance, and Disaster Recovery==

Revision as of 23:36, 18 February 2021

This article will cover topics related to the AWS Certified DevOps Engineer - Professional exam and certification.

Domains

The 6 domains outlined in the AWS blueprint for the certification include:

  1. Software Development LifeCycle (SDLC) Automation [22%]
  2. Configuration Management and Infrastructure as Code [19%]
  3. Monitoring and Logging [15%]
  4. Policies and Standards Automation [10%]
  5. Incident and Event Response [18%]
  6. High Availability, Fault Tolerance, and Disaster Recovery [16%]

Domain 1: SDLC Automation

  • 1.1 Apply concepts required to automate a CI/CD pipeline
  • 1.2 Determine source control strategies and how to implement them
  • 1.3 Apply concepts required to automate and integrate testing
  • 1.4 Apply concepts required to build and manage artifacts securely
  • 1.5 Determine deployment/delivery strategies (e.g., A/B, Blue/Green, Canary, Red/Black) and how to implement them using AWS Services

Domain 2: Configuration Management and Infrastructure as Code

  • 2.1 Determine deployment services based on deployment needs
  • 2.2 Determine application and infrastructure deployment models based on business needs
  • 2.3 Apply security concepts in the automation of resource provisioning
  • 2.4 Determine how to implement lifecycle hooks on a deployment
  • 2.5 Apply concepts required to manage systems using AWS configuration management tools and services

Domain 3: Monitoring and Logging

  • 3.1 Determine how to set up the aggregation, storage, and analysis of logs and metrics
  • 3.2 Apply concepts required to automate monitoring and event management of an environment
  • 3.3 Apply concepts required to audit, log, and monitor operating systems, infrastructures, and applications
  • 3.4 Determine how to implement tagging and other metadata strategies

Domain 4: Policies and Standards Automation

  • 4.1 Apply concepts required to enforce standards for logging, metrics, monitoring, testing, and security
  • 4.2 Determine how to optimize cost through automation
  • 4.3 Apply concepts required to implement governance strategies

Domain 5: Incident and Event Response

  • 5.1 Troubleshoot issues and determine how to restore operations
  • 5.2 Determine how to automate event management and alerting
  • 5.3 Apply concepts required to implement automated healing
  • 5.4 Apply concepts required to set up event-driven automated actions

Domain 6: High Availability, Fault Tolerance, and Disaster Recovery

  • 6.1 Determine appropriate use of multi-AZ versus multi-region architectures
  • 6.2 Determine how to implement high availability, scalability, and fault tolerance
  • 6.3 Determine the right services based on business needs (e.g., RTO/RPO, cost)
  • 6.4 Determine how to design and automate disaster recovery strategies
  • 6.5 Evaluate a deployment for points of failure

SDLC Automation

Introduction

What is CI/CD?

The CI/CD Pipeline
  • AWS CodePipeline
    • Source Stage
      • AWS CodeCommit (think "git")
    • Deploy Stage - Development
      • AWS CodeDeploy -> EC2 instance
    • Deploy Stage - Production
      • AWS CodeDeploy -> EC2 instance

AWS CodeCommit

AWS CodeBuild

  • A fully managed build service
  • Compiles your code
  • Runs unit tests
  • Produces artifacts that are ready to deploy
  • Eliminates the need to provision/manage/scale your own build servers
  • Provides pre-packaged build environments
  • Allows you to build your own customized build environment
  • Scales automatically to meet your build requirements
Benefits of CodeBuild
  • It is fully managed
    • You do not have to set up any build servers, nor patch, update, or maintain them.
  • It is on-demand
    • It automatically scales to meet your requirements. No more migrating to larger EC2 servers because your builds are taking too long. You only pay for the minutes (seconds?) you consume.
  • It is preconfigured
    • It comes with many pre-configured build environments for the most popular programming languages. You just need to configure it to use your build script.
SEE: AWS - Troubleshooting CodeBuild

AWS CodeDeploy

What is CodeDeploy?
  • A fully managed deployment service that automates deployments to:
    • Amazon EC2 instances
    • On-premise instances
    • AWS Lambda functions
  • Makes it easier to:
    • Rapidly deploy new features
    • Update Lambda function versions
    • Avoid downtime during deployment
    • Handle the full complex deployment process without human intervention


AWS CodePipeline

CodePipeline is the "CD" of CI/CD.

Benefits
  • Automatic
    • From the check-in of your code to deployment on to your service, CodePipeline takes care of it all.
  • Easy to set up
    • CodePipeline has no servers to provision, it is dead simple to configure and get working. There are pre-built plugins or you can roll your own.
  • Configurable
    • You can create, configure, and modify all stages of your software release process with ease. You can implement automated testing and customize the deployment process.

Testing

Why do we test?
  • Meet the requirements defined
  • Ensure the code performs in an accepatble period of time
  • Ensure the code is usable
  • Ensure the code responds correctly to all kinds of inputs
  • Achieves the result the programmer desired
Types of testings (see Wikipedia)
Automated testing
  • Automatic execution oif test
  • Comparision of actual outcomes to predicted outcomes
  • Fast, continuos feedback
  • Immediate notifcation
  • Save resources
Unit test example
assert()

Artifacts

What are artifacts?

An artifact is a product or by-product produced during the software development process.

For example:

  • Compiled binaries
  • Source code
  • Documentation
  • Use cases
  • Class diagrams

Artifacts are stored in S3 (note: this has nothing to do with AWS Artifact!)

Deployment Strategies

Single Target Deployment (build -> target)
  • Use for small development projects, especially when legacy or non-highly-available infrastructure is involved.
  • When it is initiated, a new application version is installed on the target server.
  • A brief outage occurs during installation. There are no secondary servers, so testing is limited. Rollback involves removing the new version and install the previous.
All-at-Once Deployment (build -> x2 targets)
  • Deployment happens in one step, just like single target deployment.
  • With this method, the destination is multiple targets.
  • More complicated than single target; often requiring orchestration tooling.
  • Shares negatives of single target. No ability to test, still has deployment outages, and less than ideal rollback.
Minimum in-service Deployment (initial build stage -> t1 t2 t3 ...)
  • Deployment happens in multiple stages
  • Deployment happens to as many targets as possible while maintaining the minimum in-service targets.
  • A few moving parts, orchestration and health checks are required.
  • Allows automated testing, deployment targets are assessed and testsd prior to continuing.
  • Generally, no downtime.
  • Often quicker and less stages than a rolling deployment.
Rolling Deployment
  • Deployment happens in multiple stages. Number of targets per stage is user-defined.
  • Moving parts; orchestration and health-checks are required.
  • Overall applicable health is not necessariliy maintained.
  • Can be the leasat efficient deployment time based on time-taken.
  • Allows automated testing; deployment targets are assessed and tested prior to continuing.
  • Generally, no downtime, assuming number of targets per run is not large neough to impact the application.
  • Can be paused, allowing limited multi-version testing (combined with small targets per stage).
Blue/Green Deployment (aka Red/Black)
  • Requires advanced orchestration tooling
  • Carries significant cost - maintiang 2 environments for the duration of deployments.
  • Deployment process is rapid - entire environemnt (blue or green) is deployed all at once.
  • Cutover and migration is clean and controlled (e.g., DNS change)
  • Rollback is equally clean (e.g., DNS regression)
  • Health and performance of entire "green" environment can be tested prior to cutover.
  • using advanced template systems, such as CloudFormation, the entire process can be fully automated.
Canary Deployment
  • Like Blue/Green, but keep blue active and route percentage of traffic to green
  • In AWS, use Route53 w/weighted round-robin

Configuration Management and Infrastructure as Code

  • Introduction
  • AWS CloudFormation
  • AWS CloudFormation Intrinsic Functions
  • AWS CloudFormation Wait Conditions
  • AWS CloudFormation Nested Stacks
  • AWS CloudFormation Deletion Policies
  • AWS CloudFormation Stack Updates
  • AWS CloudFormation Change Sets
  • AWS CloudFormation Custom Resources
  • AWS Elastic Beanstalk
  • AWS Elastic Beanstalk extensions
  • AWS Config
  • Amazon ECS
  • AWS Managed Services
  • AWS Lambda
  • AWS Lambda Step Functions
  • AWS OpsWorks

Monitoring and Logging

  • Introduction

CloudWatch

$ aws cloudwatch put-metric-data \
    --metric-name randomNumber \
    --namespace Random \
    --value $(shuf -i 1-1000 -n1) \
    --region=us-west-2
  • CloudWatch Custom Metrics
  • CloudWatch Events

CloudWatch Logs

$ sudo dpkg -i amazon-cloudwatch-agent.deb
$ sudo vi /opt/aws/amazon-cloudwatch-agent/etc/common-config.toml
# create and edit config
$ sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -c file:cloudwatchconfig.cfg -s

AWS X-Ray

Think managed "Kiali + Jaeger"

Policies and Standards Automation

Introduction

AWS Service Catalog

AWS Trusted Advisor

AWS Systems Manager

A management service that assists with:

  • Collecting software inventory
  • Applying OS patches
  • Creating system images
  • Configuring operating systems
  • Manage Hybrid Cloud systems from a single interface (AWS and on-prem)
  • Reducing costs
Run Command

Lets you run a given command(s) across all of your EC2 instances (or a group of them).

AWS Organizations

AWS Secrets Manager

Amazon Macie

Macie is a security server that uses machine learning to automatically discover, classify, and protect sensitive data in AWS.

  • Macie:
    • Can recognize any Personally Identifiable Information (PII)
    • Provides a dashboard
    • Monitors data access activity for anomalies
    • Generates detailed alerts when it detects risk of unauthorized access or accidental data leaks
    • As of February 2021 <CHECK>, it only protects data in S3, with more AWS data stores planned for the future.
    • It gives you superior visibility of data
    • Simple to set up and easy to manage

AWS Certificate Manager

Incident and Event Response

Introduction

Amazon GuardDuty

GuardDuty is a threat-detection service that continuously monitors for malicious or unauthorized behaviour.

Amazon Inspector

Inspector is an automated service that assesses your applications for vulnerabilities and produces a security findings report.

Amazon Kinesis

Easily collect, process, and analyze video and data streams in real time.

Services
  • Kinesis Data Analytics
    • Analyze streaming data
    • Respond in real-time
    • Query using SQL
    • Completely managed service (no servers required)
    • Pay-as-you-go for what you use
    • Powerful real-time processing
  • Kinesis Data Firehose
    • Deliver streaming data
    • No applications to write or manage
    • Just configure the producer
    • Data can be transformed
    • Destinations such as S3, Redshift, ElasiticSearch, and Splunk
    • Accepts records in chunks of up to 1,000 kb
  • Kinesis Data Streams
    • Collect streaming data
    • Massively scalable
    • Capture gigabytes per second (from thousands of sources)
    • Data is available in milliseconds
    • Durable (data in stored in 3 x DCs in a region)
    • Data is stored for 7 days
    • Elastic
  • Kinesis Video Streams
    • Collect streaming video
    • Can handle ingestion from millions of devices
    • Enables live and on-demand playback
    • Take advantage of Amazon Recognition Video and Machine Learning frameworks for video
    • Access your data through APIs
    • Build real-time video enabled applications

Review the tutorial: "Using AWS Lambda with Amazon Kinesis".

High Availability, Fault Tolerance, and Disaster Recovery

  • Introduction
  • AWS Single Sign-On
  • Amazon CloudFront
  • AutoScaling and Lifecycle hooks
  • Amazon Route53
  • Amazon RDS
  • Amazon Aurora
  • Amazon DynamoDB
  • Amazon DynamoDB Keys and Streams

Other Services You Need to Know About

  • Introduction
  • Tagging
  • Amazon Elastic File System
  • Amazon ElastiCache
  • Amazon S3 Glacier
  • AWS Direct Connect
  • AWS Lambda Function Dead Letter Queues
  • Amazon CloudSearch
  • Amazon Elasticsearch Service
  • Amazon DynamoDB Accelerator
  • AWS Server Migration Service

External links