Difference between revisions of "Category:AWS"

Latest revision as of 16:44, 30 April 2021

This category/article is (currently) just a random collection of my notes on Amazon Web Services (AWS). I will organize each service-type into a separate article, when I find the time.

Identity and Access Management (IAM)

SEE: AWS IAM FAQs

AWS Identity and Access Management (IAM) enables you to securely control access to AWS services and resources for your users. Using IAM, you can create and manage AWS users and groups, and use permissions to allow and deny their access to AWS resources.

IAM is a feature of your AWS account offered at no additional charge. You will be charged only for use of other AWS services by your users.

• IAM Roles are more secure than storing your access key and secret access key on individual EC2 instances. This is consider a "best practice".
• Roles are easier to manage
• Roles can only be assigned when that EC2 instance is being provisioned (i.e., after provisioning an EC2 instance and adding an IAM Role to that instance, you are not able to delete the Role or add another Role. You can, however, add to or modify the existing Policies attached to the Role attached to the instance.)
• Roles are universal, you can use them in any AWS region.

Elastic Compute Cloud (EC2)

SEE: AWS/EC2

Elastic Load Balancer (ELB)

SEE: Amazon ELB FAQs

• Load Balancer types:
  • Application Load Balancer (ALB)

    Layer 7 Load Balancer

    Makes routing decisions at the application layer (HTTP/HTTPS), supports path-based routing, and can route requests to one or more ports on each EC2 instance or container instance in your VPC

  • Classic Load Balancer (ELB)

    Layer 4 Load Balancer

    Makes routing decisions at either the transport layer (TCP/SSL) or the application layer (HTTP/HTTPS), and support either EC2-Classic or a VPC.

• ELBs are not free; one is charged by the hour and on a per GB basis of usage

• ELB supported ports:
  • ec2-vpc: 1-65535
  • ec2-classic: 25, 80, 443, 465, 587, 1024-65535

• ELB supported protocols:
  • HTTP, HTTPS, TCP, SSL

Instances monitored by ELBs are reported as either:

• InService
• OutofService

Health Checks check the instance's health by simply "talking" to it over HTTP/HTTPS (looking for specific files on the instance)

NOTE: One can have multiple SSL certificates (for multiple domain names) on a single Elastic Load Balancer.

CloudWatch

SEE: CloudWatch FAQs

Amazon CloudWatch is a monitoring service for AWS cloud resources and the applications you run on AWS. You can use Amazon CloudWatch to collect and track metrics, collect and monitor log files, set alarms, and automatically react to changes to your AWS resources. Amazon CloudWatch can monitor AWS resources such as Amazon EC2 instances, Amazon DynamoDB tables, and Amazon RDS DB instances, as well as custom metrics generated by your applications and services, and any log files your applications generate. You can use Amazon CloudWatch to gain system-wide visibility into resource utilization, application performance, and operational health. You can use these insights to react and keep your application running smoothly.

• Services CloudWatch can monitor include: EC2, Classic ELB, ALB, EBS, S3, SNS, Lambda, DynamoDB, IoT, etc.

• Standard (free) monitoring = every 5 minutes
• Detailed (not free) monitoring = every 1 minute

• Default CloudWatch EC2 monitoring metrics
  • CPU (e.g., CPU utilization, credit usage, credit balance)
  • Disk (e.g., read/write bytes/ops)
  • Network (e.g., traffic in/out, packets in/out)
  • Status Checks (instance-level and host/hypervisor-level)
  • Able to create custom metrics

CloudWatch Dashboards allow you to create customizable dashboards to see what is happening within your AWS account.

• Bashboard widgets
  • Line (plot): compare metrics over time
  • Stacked area (plot): compare the total over time
  • Number: instantly see the latest value for a metric
  • Text: free text with markdown formatting. Example:

# Heading
## Sub-heading
Paragraphs are separated by a blank line. Text attributes *italic*, **bold**, ~~strikethrough~~ .

A [link](http://amazon.com). A link to this dashboard: [MyWebServer](#dashboards:name=MyWebServer).

[button:Button link](http://amazon.com) [button:primary:Primary button link](http://amazon.com)

Table | Header
----|-----
CloudWatch | Dashboards

```
Text block
ssh my-host
```
List syntax:

* CloudWatch
* Dashboards
  1. Graphs
  1. Text widget

CloudWatch Alarms allow you to set alarms that notify you (e.g., via email) when particular thresholds (you set) are hit.

Amazon EventBridge (formerly CloudWatch Events) helps you to respond to state changes in your AWS resources. When your resources change state they automatically send events into an event stream. You can create rules that match selected events in the stream and route them to targets to take action. You can also use rules to take action on a pre-determined schedule. For example, you can configure rules to:

• Automatically invoke an AWS Lambda function to update DNS entries when an event notifies you that Amazon EC2 instance enters the Running state
• Direct specific API records from CloudTrail to a Kinesis stream for detailed analysis of potential security or availability risks
• Periodically invoke a built-in target to create a snapshot of an Amazon EBS volume

CloudWatch Logs helps you to aggregate, monitor, and store logs. Note: You must install an agent on the EC2 instance to use this service. For example, you can:

• Monitor HTTP response codes in Apache logs
• Receive alarms for errors in kernel logs
• Count exceptions in application logs

Note the difference between CloudWatch and CloudTrail.

CloudTrail

AWS CloudTrail is a web service that records AWS API calls for your account and delivers log files to you. The recorded information includes the identity of the API caller, the time of the API call, the source IP address of the API caller, the request parameters, and the response elements returned by the AWS service.

With CloudTrail, you can get a history of AWS API calls for your account, including API calls made via the AWS Management Console, AWS SDKs, command line tools, and higher-level AWS services (such as AWS CloudFormation). The AWS API call history produced by CloudTrail enables security analysis, resource change tracking, and compliance auditing.

AWS Command Line Interface (CLI)

The AWS Command Line Interface (CLI) is a unified tool to manage your AWS services. With just one tool to download and configure, you can control multiple AWS services from the command line and automate them through scripts.

The AWS CLI introduces a new set of simple file commands for efficient file transfers to and from Amazon S3.

SDKs

• HTTP codes:
  • 200 - The request has succeeded
  • 3xx - Redirection
  • 4xx - Client error (think 404 not found)
  • 5xx - Server error (think Apache service not running, etc.)

• Available SDKs:
  • Android, iOS, JavaScript (browser)
  • Java
  • .Net
  • Node.js
  • PHP
  • Python
  • Ruby
  • Go
  • C++ (preview)

• SDK default regions:
  • default region (for most SDKs): us-east-1
  • Some SDKs have default regions set (Java)
  • Some SDKs do not (Node.js)

Services that are free include: CloudFormation, Elastic Beanstalk, Autoscaling, Opsworks, etc. (however, the resources they create are not free; e.g., EC2 instances, ELBs)

Simple Storage Service (S3)

SEE: AWS/S3

Lambda

SEE: AWS/Lambda

Databases

SEE: Amazon RDS FAQs

RDS 

Relational Database Server

• RDS (OLTP) Relational Database Types:
  • Aurora
  • MySQL Server
  • MariaDB
  • PostgreSQL
  • MS SQL Server
  • Oracle

RDS - Backups, multi-AZs, and read replicas

• There are two different types of RDS backups:

1. Automated Backups; and
2. Snapshots (manual)

• Automated Backups
  • Automated Backups allow you to recover your database to any point in time within a "retention period". The retention period can be between 1 and 35 days. Automated Backups will take a full daily snapshot and will also store transaction logs throughout the day. When you do a recovery, AWS will first choose the most recent daily backup, and then apply transaction logs relevant to that day. This allows you to do a point-in-time recovery down to a second, within the retention period.
  • Automated Backups are enabled by default. The backup data is stored in S3 and you get free storage space equal to the size of your database. Example: If you have an RDS instance of 10 GB, you will get 10 GB worth of storage.
  • Backups are taken within a defined window. During the backup window, storage I/O may be suspended while your data is being backed up and you may experience elevated latency.

• Snapshots
  • RDS Snapshots are done manually (i.e., they are user initiated). They are stored even after you delete the original RDS instance (unlike automated backups).

• Restoring backups
  • Whenever you restore either an Automated Backup or a manual Snapshot, the restored version of the database will be a new RDS instance with a new DNS endpoint.

• Encryption
  • Encryption at rest is supported for MySQL, Oracle, SQL Server, PostreSQL, and MariaDB. Encryption is done using the AWS Key Management Service (KMS). Once your RDS instance is encrypted, the data stored at rest in the underlying storage is encrypted, as are its automated backups, read replicas, and snapshots.
  • As of April 2017, encrypting an existing RDS instance is not supported. To use RDS encryption for an existing database, create a new instance with encryption enabled and migrate your data into it.

• Multi-AZ RDS
  • Allows you to have an exact copy of your production database in another Availability Zone (AZ). AWS handles the replication for you, so when you production database is written to, this write will automatically be synchronised to the standby database.
  • In the event of a planned database maintenance, instance failure, or an AZ failure, Amazon RDS will automatically failover to the standby so that database operations can resume quickly without administrative intervention.
  • This is meant for Disaster Recovery (DR) only. It is not primarily used for improving performance. For performance improvement, you need Read Replicas.
  • You cannot use the secondary database as an independent read node when you have deployed an RDS instance into multiple AZs (use Read Replicas instead).

• Read Replicas
  • Allow you to have a read-only copy of your production database. This is achieved by using asynchronous replication from the primary RDS instance to the read replica. You use read replica's primarily for very read-heavy database workloads.
  • Use for scaling! Not for DR!
  • You must have automatic backups turned on in order to deploy a read replica.
  • You can have up to 5 read replica copies of any database.
  • You can have read replicas of read replicas (but watch out for latency).
  • Each read replica will have its own DNS endpoint.
  • You cannot have Read Replicas that have Multi-AZ.
  • You can, however, create Read Replicas of Multi-AZ source databases.
  • Read Replicas can be promoted to be their own databases (but this breaks the replication).
  • Able to create a Read Replica in a second region for MySQL and MariaDB. Not for PostgreSQL.
  • There is no charge associated with data transfer when replicating data from your primary RDS instance to your secondary instance.
  • Supported databases:
    • MySQL
    • PostgreSQL
    • MariaDB

Provisioned IOPS volumes can range in size from 100 GB to 6 TB for MySQL, MariaDB, PostgreSQL, and Oracle DB engines. SQL Server Express and Web editions can range in size from 100 GB to 4 TB, while SQL Server Standard and Enterprise editions can range in size from 200 GB to 4 TB.

DynamoDB vs. RDS

• DynamoDB offers "push button" scaling (i.e., you can scale your database on-the-fly, without any down time).
• With RDS, it is not so easy and you usually have to use a bigger instance size or add a Read Replica.

DynamoDB

• A fast and flexible NoSQL database service for all applications that need consistent, single-digit millisecond latency at any scale. It is a fully managed database and supports both document and key-value data models. Its flexible data model and reliable performance make it a great fit for mobile, web, gaming, ad-tech, IoT, and many other applications.
• Stored on SSD storage.
• Spread across 3 geographically distinct data centres.
• Eventually Consistent Reads (default)
  • Consistency across all copies of data is usually reached within a second. Repeating a read after a short time should return the updated data.
  • Best read performance.
• Strongly Consistent Reads
  • Returns a result that reflects all writes that received a successful response prior to the read.

• Non-relational Databases:
  • DynamoDB (document-oriented)
    • Collection (=Table)
    • Document (=Row)
    • Key-value pairs (=Fields)

An example (JSON) NoSQL document:

{
  "_id": "345sdf45asdf",
  "firstname": "John",
  "surname": "Smith",
  "age": "23",
  "address": [
    {
      "street": "123 First Street",
      "suburb": "Wallingford"
    }
  ]
}

Data Warehousing

• Used for business intelligence (BI). Tools include: Cognos, Jaspersoft, SQL Server Reporting Services, Oracle Hyperion, SAP NetWeaver, etc.
• Used to pull in very large and complex datasets. Usually used by management to perform queries on data (e.g., current performance vs. targets, etc.).

• OLTP vs. OLAP
  • Online Transaction Processing (OLTP) differs from Online Analytics Processing (OLAP) in terms of the types of queries performed.
  • OLTP Example:
    • Return order number: 1234567
    • Pulls up a row of data (e.g., Name, Data, Address to deliver to, Delivery Status, etc.)
  • OLAP Example
    • Return net profit for EMEA and Pacific for the Digital Radio Product
    • Pulls in large numbers of records:

      Sum of radios sold in EMEA region

      Sum of radios sold in Pacific region

      Unit cost of radio in each region

      Sales price of each radio

      Sales price - unit cost

Data Warehousing databases use a different type of architecture, both from a database perspective and the infrastructure layer.

Redshift (OLAP)

Amazon Redshift is a fast and powerful, fully managed, petabyte-scale data warehouse service in the Cloud. Customers can start small for just $0.25 per hour with no commitment or upfront costs and scale to a petabyte or more for $1,000 per terabyte per year, less than a tenth of most other data warehousing solutions.

• Start out: Single Node (160 GB)
• Scale: Use Multi-Node, which consists of:
  • Leader Node (manages client connections and receives queries)
  • Compute Node (store data and perform queries and computations). Able to have up to 128 Compute Nodes.

• Columnar Data Storage
  • Instead of storing data as a series of rows, Redshift organizes the data by column. Unlike row-based systems, which are ideal for transaction processing, column-based systems are ideal for data warehousing and analytics, where queries often involve aggregates performed over large datasets. Since only the columns involved in the queries are processed and columnar data is stored sequentially on the storage media, column-based systems require far fewer I/Os, greatly improving query performance (up to 10x faster).

• Advanced Compression
  • Columnar data stores can be compressed much more than row-based data stores, because similar data is stored sequentially on disk. Redshift employs multiple compression techniques and can often achieve significant compression relative to traditional relational data stores. In addition, Redshift does not require indexes or materialized views and, as such, uses less space than traditional relational database systems. When loading data into an empty table, Redshift automatically samples your data and selects the most appropriate compression scheme.

• Massively Parallel Processing (MPP)
  • Redshift automatically distributes data and query load across all nodes.
  • Redshift makes it easy to add nodes to your data warehouse and enables you to maintain fast query performance as your data warehouse grows.

• Redshift pricing (you are charged for the following):
  • Compute Node Hours
    • Total number of hours you run across all your compute nodes for the billing period. You are billed for 1 unit per node per hour, so a 3-node data warehouse cluster running persistently for an entire month would incur 2,160 instance hours (60x24x30).
    • You are not charged for the Leader Node hours; only Compute Nodes will incur charges.
  • Backup
  • Data transfer (only within a VPC, not outside it)

• Redshift Security
  • Encrypted in transit using SSL
  • Encrypted at rest using AES-256 encryption
  • By default, Redshift takes care of key management. However you can also,
    • Manage your own keys through Hardware Security Module (HSM); or
    • Use AWS Key Management Service (KMS)

• Redshift Availability
  • Not designed to be multi-AZ. As of April 2017, only available in 1 AZ.
  • However, you can restore snapshots to new AZs in the event of an outage.

ElastiCache

• A web service that makes it easy to deploy, operate, and scale and in-memory cache in the Cloud. The service improves the performance of web applications by allowing you to retrieve information for fast, managed, in-memory caches, instead of relying entirely on slower disk-based databases.
• Amazon ElastiCache can be used to significantly improve latency and throughput for many read-heavy application workloads (e.g., social media, gaming, media sharing, and Q&A portals) or compute-intensive workloads (e.g., a recommendation engine).
• Caching improves application performance by storing critical pieces of data in memory for low-latency access. Cached information may include the results of I/O-intensive database queries or the results of computationally-intensive calculations.
• ElastiCache is a good choice if your database is particularly read-heavy and not prone to frequent changing.
• ElastiCache supports two open-source in-memory caching engines:
  • Memcached
    • A widely adopted memory object caching system. ElastiCache is protocol-compliant with Memcached, so popular tools that you use today with existing Memcached environments will work seamlessly with the service.
  • Redis
    • A popular open-source in-memory key-value store that supports data structures such as sorted sets and lists. ElastiCache supports Master / Slave replication and Multi-AZ, which can be used to achieve cross-AZ redundancy.

Database Migration Service (DMS)

• Announced at re:Invent 2015.
• Allows you to (live) migrate your production database to AWS.
• Once the migration has started, AWS manages all the complexities of the migration process like data type transformation, compression, and parallel transfer (for faster data transfer) while ensuring that data changes to the source database that occur during the migration process are automatically replicated to the target.
• The AWS schema conversion tool automatically converts the source database schema and a majority of the custom code, including views, stored procedures, and functions, to a format compatible with the target database.
• Example: convert Oracle to MySQL, etc.

Aurora

Amazon Aurora is a MySQL-compatible, relational database engine that combines the speed and availability of high-end commercial databases with the simplicity and cost-effectiveness of open source databases. Aurora provides up to five times better performance than MySQL at a price point 1/10th that of a commercial database while delivering similar performance and availability. It was announced at re:Invent 2014.

• Aurora scaling
  • Start with 10 GB, scales in 10 GB increments up to 64 TB (Storage Autoscaling; i.e., it autoscales for you)
  • Compute resources can scale up to 32 vCPUs and 244 GB of memory.
  • 2 copies of your data is maintained in each availability zone, with a minimum of 3 AZs. As such, Aurora maintains 6 copies of your data.
  • Aurora is designed to transparently handle the loss of up to two copies of data without affecting database write availability and up to three copies without affecting read availability.
  • Aurora storage is self-healing. Data blocks and disks are continuously scanned for errors and repaired automatically.

• Aurora Replicas
  • There are 2 types of Replicas:
    1. Aurora Replicas (currently 15). Automatic failover.
    2. MySQL Read Replicas (currently 5). No automatic failover.