Difference between revisions of "AWS/DevOps"
(→Configuration Management and Infrastructure as Code) |
(→AWS Config) |
||
Line 281: | Line 281: | ||
===AWS Config=== | ===AWS Config=== | ||
+ | |||
+ | * Delete Config service: | ||
+ | <pre> | ||
+ | $ aws configservice delete-configuration-recorder --configuration-recorder-name default | ||
+ | </pre> | ||
===Amazon ECS=== | ===Amazon ECS=== |
Revision as of 23:43, 18 February 2021
This article will cover topics related to the AWS Certified DevOps Engineer - Professional exam and certification.
Contents
- 1 Domains
- 2 SDLC Automation
- 3 Configuration Management and Infrastructure as Code
- 4 Monitoring and Logging
- 5 Policies and Standards Automation
- 6 Incident and Event Response
- 7 High Availability, Fault Tolerance, and Disaster Recovery
- 8 Other Services You Need to Know About
- 9 External links
Domains
The 6 domains outlined in the AWS blueprint for the certification include:
- Software Development LifeCycle (SDLC) Automation [22%]
- Configuration Management and Infrastructure as Code [19%]
- Monitoring and Logging [15%]
- Policies and Standards Automation [10%]
- Incident and Event Response [18%]
- High Availability, Fault Tolerance, and Disaster Recovery [16%]
Domain 1: SDLC Automation
- 1.1 Apply concepts required to automate a CI/CD pipeline
- 1.2 Determine source control strategies and how to implement them
- 1.3 Apply concepts required to automate and integrate testing
- 1.4 Apply concepts required to build and manage artifacts securely
- 1.5 Determine deployment/delivery strategies (e.g., A/B, Blue/Green, Canary, Red/Black) and how to implement them using AWS Services
Domain 2: Configuration Management and Infrastructure as Code
- 2.1 Determine deployment services based on deployment needs
- 2.2 Determine application and infrastructure deployment models based on business needs
- 2.3 Apply security concepts in the automation of resource provisioning
- 2.4 Determine how to implement lifecycle hooks on a deployment
- 2.5 Apply concepts required to manage systems using AWS configuration management tools and services
Domain 3: Monitoring and Logging
- 3.1 Determine how to set up the aggregation, storage, and analysis of logs and metrics
- 3.2 Apply concepts required to automate monitoring and event management of an environment
- 3.3 Apply concepts required to audit, log, and monitor operating systems, infrastructures, and applications
- 3.4 Determine how to implement tagging and other metadata strategies
Domain 4: Policies and Standards Automation
- 4.1 Apply concepts required to enforce standards for logging, metrics, monitoring, testing, and security
- 4.2 Determine how to optimize cost through automation
- 4.3 Apply concepts required to implement governance strategies
Domain 5: Incident and Event Response
- 5.1 Troubleshoot issues and determine how to restore operations
- 5.2 Determine how to automate event management and alerting
- 5.3 Apply concepts required to implement automated healing
- 5.4 Apply concepts required to set up event-driven automated actions
Domain 6: High Availability, Fault Tolerance, and Disaster Recovery
- 6.1 Determine appropriate use of multi-AZ versus multi-region architectures
- 6.2 Determine how to implement high availability, scalability, and fault tolerance
- 6.3 Determine the right services based on business needs (e.g., RTO/RPO, cost)
- 6.4 Determine how to design and automate disaster recovery strategies
- 6.5 Evaluate a deployment for points of failure
SDLC Automation
Introduction
What is CI/CD?
- The CI/CD Pipeline
- AWS CodePipeline
- Source Stage
- AWS CodeCommit (think "git")
- Deploy Stage - Development
- AWS CodeDeploy -> EC2 instance
- Deploy Stage - Production
- AWS CodeDeploy -> EC2 instance
- Source Stage
AWS CodeCommit
AWS CodeBuild
- A fully managed build service
- Compiles your code
- Runs unit tests
- Produces artifacts that are ready to deploy
- Eliminates the need to provision/manage/scale your own build servers
- Provides pre-packaged build environments
- Allows you to build your own customized build environment
- Scales automatically to meet your build requirements
- Benefits of CodeBuild
- It is fully managed
- You do not have to set up any build servers, nor patch, update, or maintain them.
- It is on-demand
- It automatically scales to meet your requirements. No more migrating to larger EC2 servers because your builds are taking too long. You only pay for the minutes (seconds?) you consume.
- It is preconfigured
- It comes with many pre-configured build environments for the most popular programming languages. You just need to configure it to use your build script.
AWS CodeDeploy
- What is CodeDeploy?
- A fully managed deployment service that automates deployments to:
- Amazon EC2 instances
- On-premise instances
- AWS Lambda functions
- Makes it easier to:
- Rapidly deploy new features
- Update Lambda function versions
- Avoid downtime during deployment
- Handle the full complex deployment process without human intervention
AWS CodePipeline
CodePipeline is the "CD" of CI/CD.
- Benefits
- Automatic
- From the check-in of your code to deployment on to your service, CodePipeline takes care of it all.
- Easy to set up
- CodePipeline has no servers to provision, it is dead simple to configure and get working. There are pre-built plugins or you can roll your own.
- Configurable
- You can create, configure, and modify all stages of your software release process with ease. You can implement automated testing and customize the deployment process.
Testing
- Why do we test?
- Meet the requirements defined
- Ensure the code performs in an accepatble period of time
- Ensure the code is usable
- Ensure the code responds correctly to all kinds of inputs
- Achieves the result the programmer desired
- Types of testings (see Wikipedia)
- Automated testing
- Automatic execution oif test
- Comparision of actual outcomes to predicted outcomes
- Fast, continuos feedback
- Immediate notifcation
- Save resources
- Unit test example
assert()
Artifacts
- What are artifacts?
An artifact is a product or by-product produced during the software development process.
For example:
- Compiled binaries
- Source code
- Documentation
- Use cases
- Class diagrams
Artifacts are stored in S3 (note: this has nothing to do with AWS Artifact!)
Deployment Strategies
- Single Target Deployment (build -> target)
- Use for small development projects, especially when legacy or non-highly-available infrastructure is involved.
- When it is initiated, a new application version is installed on the target server.
- A brief outage occurs during installation. There are no secondary servers, so testing is limited. Rollback involves removing the new version and install the previous.
- All-at-Once Deployment (build -> x2 targets)
- Deployment happens in one step, just like single target deployment.
- With this method, the destination is multiple targets.
- More complicated than single target; often requiring orchestration tooling.
- Shares negatives of single target. No ability to test, still has deployment outages, and less than ideal rollback.
- Minimum in-service Deployment (initial build stage -> t1 t2 t3 ...)
- Deployment happens in multiple stages
- Deployment happens to as many targets as possible while maintaining the minimum in-service targets.
- A few moving parts, orchestration and health checks are required.
- Allows automated testing, deployment targets are assessed and testsd prior to continuing.
- Generally, no downtime.
- Often quicker and less stages than a rolling deployment.
- Rolling Deployment
- Deployment happens in multiple stages. Number of targets per stage is user-defined.
- Moving parts; orchestration and health-checks are required.
- Overall applicable health is not necessariliy maintained.
- Can be the leasat efficient deployment time based on time-taken.
- Allows automated testing; deployment targets are assessed and tested prior to continuing.
- Generally, no downtime, assuming number of targets per run is not large neough to impact the application.
- Can be paused, allowing limited multi-version testing (combined with small targets per stage).
- Blue/Green Deployment (aka Red/Black)
- Requires advanced orchestration tooling
- Carries significant cost - maintiang 2 environments for the duration of deployments.
- Deployment process is rapid - entire environemnt (blue or green) is deployed all at once.
- Cutover and migration is clean and controlled (e.g., DNS change)
- Rollback is equally clean (e.g., DNS regression)
- Health and performance of entire "green" environment can be tested prior to cutover.
- using advanced template systems, such as CloudFormation, the entire process can be fully automated.
- Canary Deployment
- Like Blue/Green, but keep blue active and route percentage of traffic to green
- In AWS, use Route53 w/weighted round-robin
Configuration Management and Infrastructure as Code
Introduction
AWS CloudFormation
- Examples
see: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/template-anatomy.html
Resources: Parameters: # set of parameters Mappings: # set of mappings Resources: MyEC2Instance: Type: AWS::EC2::Instance Properties: ImageId: "ami-00000000" Outputs: # set of outputs
Resources: MyEC2Instance: Type: AWS::EC2::Instance Properties: ImageId: "ami-088ff0e3bde7b3fdf" InstanceType: "t2.micro"
AWS CloudFormation Intrinsic Functions
see: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/template-macros.html
- FindInMap YAML example
Mappings: RegionMap: us-east-1: HVM64: "ami-0000" HVMG2: "ami-0000" us-west-1: HVM64: "ami-0000" HVMG2: "ami-0000" --- Resources: myEC2Instance: Type: "AWS::EC2::Instance" Properties: ImageId: !FindInMap - RegionMap - !Ref 'AWS::Region' # intrinsic function - HVM64 InstanceType: m1.small
AWS CloudFormation Wait Conditions
AWS CloudFormation Nested Stacks
AWS CloudFormation Deletion Policies
AWS CloudFormation Stack Updates
AWS CloudFormation Change Sets
AWS CloudFormation Custom Resources
AWS Elastic Beanstalk
AWS Elastic Beanstalk extensions
AWS Config
- Delete Config service:
$ aws configservice delete-configuration-recorder --configuration-recorder-name default
Amazon ECS
AWS Managed Services
AWS Lambda
AWS Lambda Step Functions
AWS OpsWorks
Monitoring and Logging
- Introduction
CloudWatch
$ aws cloudwatch put-metric-data \ --metric-name randomNumber \ --namespace Random \ --value $(shuf -i 1-1000 -n1) \ --region=us-west-2
- CloudWatch Custom Metrics
- CloudWatch Events
CloudWatch Logs
$ sudo dpkg -i amazon-cloudwatch-agent.deb $ sudo vi /opt/aws/amazon-cloudwatch-agent/etc/common-config.toml # create and edit config $ sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -c file:cloudwatchconfig.cfg -s
AWS X-Ray
Think managed "Kiali + Jaeger"
Policies and Standards Automation
Introduction
AWS Service Catalog
AWS Trusted Advisor
AWS Systems Manager
A management service that assists with:
- Collecting software inventory
- Applying OS patches
- Creating system images
- Configuring operating systems
- Manage Hybrid Cloud systems from a single interface (AWS and on-prem)
- Reducing costs
- Run Command
Lets you run a given command(s) across all of your EC2 instances (or a group of them).
AWS Organizations
AWS Secrets Manager
Amazon Macie
Macie is a security server that uses machine learning to automatically discover, classify, and protect sensitive data in AWS.
- Macie:
- Can recognize any Personally Identifiable Information (PII)
- Provides a dashboard
- Monitors data access activity for anomalies
- Generates detailed alerts when it detects risk of unauthorized access or accidental data leaks
- As of February 2021 <CHECK>, it only protects data in S3, with more AWS data stores planned for the future.
- It gives you superior visibility of data
- Simple to set up and easy to manage
AWS Certificate Manager
Incident and Event Response
Introduction
Amazon GuardDuty
GuardDuty is a threat-detection service that continuously monitors for malicious or unauthorized behaviour.
Amazon Inspector
Inspector is an automated service that assesses your applications for vulnerabilities and produces a security findings report.
Amazon Kinesis
Easily collect, process, and analyze video and data streams in real time.
- Services
- Kinesis Data Analytics
- Analyze streaming data
- Respond in real-time
- Query using SQL
- Completely managed service (no servers required)
- Pay-as-you-go for what you use
- Powerful real-time processing
- Kinesis Data Firehose
- Deliver streaming data
- No applications to write or manage
- Just configure the producer
- Data can be transformed
- Destinations such as S3, Redshift, ElasiticSearch, and Splunk
- Accepts records in chunks of up to 1,000 kb
- Kinesis Data Streams
- Collect streaming data
- Massively scalable
- Capture gigabytes per second (from thousands of sources)
- Data is available in milliseconds
- Durable (data in stored in 3 x DCs in a region)
- Data is stored for 7 days
- Elastic
- Kinesis Video Streams
- Collect streaming video
- Can handle ingestion from millions of devices
- Enables live and on-demand playback
- Take advantage of Amazon Recognition Video and Machine Learning frameworks for video
- Access your data through APIs
- Build real-time video enabled applications
Review the tutorial: "Using AWS Lambda with Amazon Kinesis".
High Availability, Fault Tolerance, and Disaster Recovery
- Introduction
- AWS Single Sign-On
- Amazon CloudFront
- AutoScaling and Lifecycle hooks
- Amazon Route53
- Amazon RDS
- Amazon Aurora
- Amazon DynamoDB
- Amazon DynamoDB Keys and Streams
Other Services You Need to Know About
- Introduction
- Tagging
- Amazon Elastic File System
- Amazon ElastiCache
- Amazon S3 Glacier
- AWS Direct Connect
- AWS Lambda Function Dead Letter Queues
- Amazon CloudSearch
- Amazon Elasticsearch Service
- Amazon DynamoDB Accelerator
- AWS Server Migration Service