Category:AWS

This category/article is (currently) just a random collection of my notes on Amazon Web Services (AWS). I will organize each service-type into a separate article, when I find the time.

Identity and Access Management (IAM)

SEE: AWS IAM FAQs

AWS Identity and Access Management (IAM) enables you to securely control access to AWS services and resources for your users. Using IAM, you can create and manage AWS users and groups, and use permissions to allow and deny their access to AWS resources.

IAM is a feature of your AWS account offered at no additional charge. You will be charged only for use of other AWS services by your users.

• IAM Roles are more secure than storing your access key and secret access key on individual EC2 instances. This is consider a "best practice".
• Roles are easier to manage
• Roles can only be assigned when that EC2 instance is being provisioned (i.e., after provisioning an EC2 instance and adding an IAM Role to that instance, you are not able to delete the Role or add another Role. You can, however, add to or modify the existing Policies attached to the Role attached to the instance.)
• Roles are universal, you can use them in any AWS region.

Elastic Compute Cloud (EC2)

SEE: AWS/EC2

Elastic Load Balancer (ELB)

SEE: Amazon ELB FAQs

• Load Balancer types:
  • Application Load Balancer (ALB)

    Layer 7 Load Balancer

    Makes routing decisions at the application layer (HTTP/HTTPS), supports path-based routing, and can route requests to one or more ports on each EC2 instance or container instance in your VPC

  • Classic Load Balancer (ELB)

    Layer 4 Load Balancer

    Makes routing decisions at either the transport layer (TCP/SSL) or the application layer (HTTP/HTTPS), and support either EC2-Classic or a VPC.

• ELBs are not free; one is charged by the hour and on a per GB basis of usage

• ELB supported ports:
  • ec2-vpc: 1-65535
  • ec2-classic: 25, 80, 443, 465, 587, 1024-65535

• ELB supported protocols:
  • HTTP, HTTPS, TCP, SSL

Instances monitored by ELBs are reported as either:

• InService
• OutofService

Health Checks check the instance's health by simply "talking" to it over HTTP/HTTPS (looking for specific files on the instance)

NOTE: One can have multiple SSL certificates (for multiple domain names) on a single Elastic Load Balancer.

CloudWatch

SEE: CloudWatch FAQs

Amazon CloudWatch is a monitoring service for AWS cloud resources and the applications you run on AWS. You can use Amazon CloudWatch to collect and track metrics, collect and monitor log files, set alarms, and automatically react to changes in your AWS resources. Amazon CloudWatch can monitor AWS resources such as Amazon EC2 instances, Amazon DynamoDB tables, and Amazon RDS DB instances, as well as custom metrics generated by your applications and services, and any log files your applications generate. You can use Amazon CloudWatch to gain system-wide visibility into resource utilization, application performance, and operational health. You can use these insights to react and keep your application running smoothly.

• Services CloudWatch can monitor include: EC2, Classic ELB, ALB, EBS, S3, SNS, Lambda, DynamoDB, IoT, etc.

• Standard (free) monitoring = every 5 minutes
• Detailed (not free) monitoring = every 1 mintue

• Default CloudWatch EC2 monitoring metrics
  • CPU (e.g., CPU utilization, credit usage, credit balance)
  • Disk (e.g., read/write bytes/ops)
  • Network (e.g., traffic in/out, packets in/out)
  • Status Checks (instance-level and host/hypervisor-level)
  • Able to create custom metrics

CloudWatch Dashboards allow you to create customizable dashboards to see what is happening within your AWS account.

• Bashboard widgets
  • Line (plot): compare metrics over time
  • Stacked area (plot): compare the total over time
  • Number: instantly see the latest value for a metric
  • Text: free text with markdown formatting. Example:

# Heading
## Sub-heading
Paragraphs are separated by a blank line. Text attributes *italic*, **bold**, ~~strikethrough~~ .

A [link](http://amazon.com). A link to this dashboard: [MyWebServer](#dashboards:name=MyWebServer).

[button:Button link](http://amazon.com) [button:primary:Primary button link](http://amazon.com)

Table | Header
----|-----
CloudWatch | Dashboards

```
Text block
ssh my-host
```
List syntax:

* CloudWatch
* Dashboards
  1. Graphs
  1. Text widget

CloudWatch Alarms allow you to set alarms that notify you (e.g., via email) when particular thresholds (you set) are hit.

CloudWatch Events helps you to respond to state changes in your AWS resources. When your resources change state they automatically send events into an event stream. You can create rules that match selected events in the stream and route them to targets to take action. You can also use rules to take action on a pre-determined schedule. For example, you can configure rules to:

• Automatically invoke an AWS Lambda function to update DNS entries when an event notifies you that Amazon EC2 instance enters the Running state
• Direct specific API records from CloudTrail to a Kinesis stream for detailed analysis of potential security or availability risks
• Periodically invoke a built-in target to create a snapshot of an Amazon EBS volume

CloudWatch Logs helps you to aggregate, monitor, and store logs. Note: You must install an agent on the EC2 instance to use this service. For example, you can:

• Monitor HTTP response codes in Apache logs
• Receive alarms for errors in kernel logs
• Count exceptions in application logs

Note the difference between CloudWatch and CloudTrail.

CloudTrail

AWS CloudTrail is a web service that records AWS API calls for your account and delivers log files to you. The recorded information includes the identity of the API caller, the time of the API call, the source IP address of the API caller, the request parameters, and the response elements returned by the AWS service.

With CloudTrail, you can get a history of AWS API calls for your account, including API calls made via the AWS Management Console, AWS SDKs, command line tools, and higher-level AWS services (such as AWS CloudFormation). The AWS API call history produced by CloudTrail enables security analysis, resource change tracking, and compliance auditing.

AWS Command Line Interface (CLI)

The AWS Command Line Interface (CLI) is a unified tool to manage your AWS services. With just one tool to download and configure, you can control multiple AWS services from the command line and automate them through scripts.

The AWS CLI introduces a new set of simple file commands for efficient file transfers to and from Amazon S3.

SDKs

• HTTP codes:
  • 200 - The request has succeeded
  • 3xx - Redirection
  • 4xx - Client error (think 404 not found)
  • 5xx - Server error (think Apache service not running, etc.)

• Available SDKs:
  • Android, iOS, JavaScript (browser)
  • Java
  • .Net
  • Node.js
  • PHP
  • Python
  • Ruby
  • Go
  • C++ (preview)

• SDK default regions:
  • default region (for most SDKs): us-east-1
  • Some SDKs have default regions set (Java)
  • Some SDKs do not (Node.js)

Services that are free include: CloudFormation, Elastic Beanstalk, Autoscaling, Opsworks, etc. (however, the resources they create are not free; e.g., EC2 instances, ELBs)

Simple Storage Service (S3)

SEE: AWS/S3

Lambda

SEE: AWS Lambda

AWS Lambda lets you run code without provisioning or managing servers. You pay only for the compute time you consume - there is no charge when your code is not running. With Lambda, you can run code for virtually any type of application or backend service - all with zero administration. Just upload your code and Lambda takes care of everything required to run and scale your code with high availability. You can set up your code to automatically trigger from other AWS services or call it directly from any web or mobile app.

AWS Lambda is a compute service where you can upload your code and create a Lambda function. Lambda takes care of provisioning and managing the servers that you use to run the code. You do not have to worry about operating systems, patching, scaling, etc. You can use Lambda in the following ways:

• As an event-driven compute service where Lambda runs your code in response to events. These events could be changes to data in an S3 bucket or a DynamoDB table.
• As a compute service to run your code in response to HTTP requests using Amazon API Gateway or API calls made using AWS SDKs.

Data Centre -> IaaS -> PaaS -> Containers -> Serverless (e.g., Lambda)

What is Lambda? It encapsulates all of the following for you:

• Data Centres
• Hardware
• Assembly code/protocols
• High-level languages
• Operating systems
• Application layer / AWS APIs

• Languages Lambda supports (as of February 2017):
  • Python
  • Node.js
  • Java
  • C#

• Pricing
  • Number of requests

    First 1 million requests are free. $0.20 per 1 million requests thereafter

  • Duration

    Duration is calculated from the time your code begins executing until it returns or otherwise terminates, rounded up to the nearest 100ms. The price depends on the amount of memory you allocate to your function. You are charged $0.00001667 for every GB-second used.

Databases

SEE: Amazon RDS FAQs

RDS 

Relational Database Server

• RDS (OLTP) Relational Database Types:
  • Aurora
  • MySQL Server
  • MariaDB
  • PostgreSQL
  • MS SQL Server
  • Oracle

• Non-relational Databases:
  • DynamoDB (document-oriented)
    • Collection (=Table)
    • Document (=Row)
    • Key-value pairs (=Fields)

E.g., JSON/NoSQL document

{
  "_id": "345sdf45asdf",
  "firstname": "John",
  "surname": "Smith",
  "age": "23",
  "address": [
    {
      "street": "123 First Street",
      "suburb": "Wallingford"
    }
  ]
}

• Redshift (OLAP)

• Elasticache
  • Memcached
  • Redis

DMS: (Online) Database Migration Service (convert Oracle to MySQL, etc.)

• Data Warehousing (business intelligence; e.g., Cognos, Jaspersoft, etc.)
• Online Transaction Processing (OLTP); vs.
• Online Analytics Processing (OLAP)

DynamoDB

SEE: Amazon DynamoDB FAQs (import to read for the exams!)

Amazon DynamoDB is a fast and flexible NoSQL database service for all applications that need consistent single-digit millisecond latency at any scale. It is a fully managed database and supports both document and key-value data models. Its flexible data model and reliable performance make it a great fit for mobile, web, gaming, ad-tech, IoT, and many other applications.

• Stored on SSD storage
• Spread across 3 geographically distinct data centres (note: not distinct AZs)
• Eventual Consistent READs (default)
  • Consistency across all copies of data is usually reached within a second. Repeating a read after a short time should return the update data. (best read performance.)
• Strongly Consistent READs
  • A strongly consistent read returns a result that reflects all writes that received a successful response prior to the read.

• The DynamoDB basics:
  • Tables
  • Items (think a row of data in the table)
  • Attributes (think of a column of data in a table)

Item 101 (has the following attributes):
{                                        
   Title = "Book 101 Title"
   ISBN = "111-1111111111"
   Authors = "Author 1"
   Price = "-2"
   Dimensions = "8.5 x 11.0 x 0.5"
   PageCount = "500"
   InPublication = true
   ProductCategory = "Book" 
}
Item 201 (has the following attributes):
{
   Title = "18-Bicycle 201"
   Description = "201 description"
   BicycleType = "Road"
   Brand = "Brand-Company A"
   Price = "100"
   Color = [ "Red", "Black" ]
   ProductCategory = "Bike"
}

An item can have any number of attributes, although there is a limit of 400 KB on the item size. An item size is the sum of lengths of its attribute names and values (binary and UTF-8 lengths); it helps if you keep the attribute names short.

• Supports up to 35 levels of nesting (JSON {foo,{bar,{baz,...)
• For any AWS account, there is an initial limit of 256 tables per region (one can, however, request an increase from Amazon)
• You can decrease the ReadCapacityUnits or WriteCapacityUnits settings for a table, but no more than four times per table in a single UTC calendar day. In a single operation, you can decrease the provisioned throughput for a table, for any global secondary indexes on that table, or for any combination of these.

• Pricing
  • Provisioned Throughput Capacity:
    • Write throughput $0.0065 per hour for every 10 units
    • Read throughput $0.0065 per hour for every 50 units
  • First 25GB stored per month is free
  • Storage costs of $0.25 per GB per month thereafter

1,000,000 (writes) / 24 (hours) / 60 (minutes) / 60 (seconds) = 11.6 writes per second

($0.0065/10) * 12 WCUs * 24 hours = $0.1872 per day

So, ($0.0065/50) * 12 RCUs * 24 hours = $0.0374 per day

(30 * ($0.1872 + $0.0374)) + $0.75 = $7.488

With the Free Tier you get:

• 25 Read Capacity Units
• 25 Write Capacity Units

• DynamoDB Indexes and Streams
• Primary Keys (two types of primary keys available):
  • Single Attribute (think uniqueID)
    • Partition Key (Hash Key) composed of one attribute
  • Composite (think uniqueID and a date range)
    • Partition Key and Sort Key (Hash & Range) composed of two attributes

Partition Key 

DynamoDB uses the partition key's value as input to an internal hash function. The output from the hash function determines the partition (this is simply the physical location in which the data is stored).

No two items in a table can have the same partition key value!

Composite Key (Partition Key and Sort Key) 

DynamoDB uses the partition key's value as input to an internal hash function. The output from the hash function determines the partition (this is simply the physical location in which the data is stored).

The two items can have the same partition key, but the must have a different sort key.

All items with the same partition key are stored together, in sorted order by the sort key value.

• Indexes
  • Local Secondary Index
    • an index that has the same hash key as the table, but a different range key. A local secondary index is "local" in the sense that every partition of a local secondary index is scoped to a table partition that has the same hash key.
    • has the same partition key and a different sort key.
    • can only be created at table creation. They cannot be removed or modified later.
    • maximum of 5 local secondary indexes per table
    • each table can have up to 20 projected non-key attributes, in total across all local secondary indexes within the table. Each index may also specify that all non-key attributes from the primary index are projected. (note: "projections" are the set of attributes that is copied into a local secondary index.)
  • Global Secondary Index
    • an index with a hash or a hash-and-range key that can be different from those on the table. A global secondary index is considered "global" because queries on the index can span all items in a table, across all partitions.
    • has a different partition key and a different sort key.
    • can be created at table creation or added later.
    • maximum of 5 global secondary indexes per table

• DynamoDB Streams
  • used to capture any kind of modification of the DynamoDB tables.
  • If a new item is added to the table, the stream captures an image of the entire item including all of its attributes.
  • If an item in updated, the stream captures the "before" and "after" image of any attributes that were modified in the item.
  • If an item was deleted from the table, the stream captures an image of the entire item before it was deleted.
  • DynamoDB Streams are stored for a maximum of 24 hours.
  • Can trigger a Lambda function (e.g., replicate table in another region and/or create an SES to send an email to the user for, say, when they first register to the website as a "welcome" email)

• One is able to export DynamoDB tables to a CSV file
• DyanmoDB allows for push-button scalability (with zero downtime)

DynamoDB Query 

find items in a table using only primary key attribute values. You must provide a partition attribute name and a distinct value to search for.

One can (optionally) provide a sort key attribute name and value, and use a comparison operator to refine the search results.

Be default, a Query returns all of the data attributes for items with the specified primary key(s), however, you can use the ProjectExpression parameter so that the query only returns some of the attributes, rather than all of them.

Query results are always sorted by the sort key. If the data type of the sort key is a number, the results are returned in numeric order; otherwise, the results are returned in order of ASCII character code values. By default, the sort order is ascending. To reverse the order, set the ScanIndexForward parameter to false.

By default, Queries are eventually consistent, but can be changed to be strongly consistent.

DynamoDB Scan 

A Scan operation examines every item in the table. By default, a Scan returns all of the data attributes for every item; however, you can use the ProjectExpression parameter so that the Scan only returns some of the attributes, rather than all of them.

What should one typically use: A Query or a Scan? Generally, a Query operation is more efficient than a Scan operation.

A Scan operation always scans the entire table, then filters out values to provide the desired result, essentially adding the extra step of removing data from the result set. Avoid using a Scan operation on a large table with a filter that removes many results, if possible. Also, as a table grows, the Scan operation slows. The Scan operation examines every item for the requested values, and can use up the provisioned throughput for a large table in a single operation.

For quicker response times, design your tables in a way that can use the Query, GetItem, or BatchGetItem APIs, instead. Alternatively, design your application to use Scan operations in a way that minimizes the impact on your table's request rate.

What happens if you exceed your throughput? 400 HTTP Status Code - ProvisionedThroughputExceededException You exceeded your maximum allowed provisioned throughput for a table or for one or more global secondary indexes.

DynamoDB Conditional Writes 

If item = $10, then update to $12 (conditional writes are idempotent)

E.g., two users try to update the same item at the same time

DynamoDB Atomic Counters 

DynamoDB supports Atomic Counters, where you use the UpdateItem operation to increment or decrement the value of an existing attribute (or "field" in a table) without interfering with other write requests. (All write requests are applied in the order in which the were received.) For example, a web application might with to maintain a counter per visitor to their site, the application would need to increment this counter regardless of its current value.

Atomic Counters are not idempotent. This mean that the counter will increment each time you call UpdateItem. If you suspect that a previous request was unsuccessful, your application could retry the UpdateItem operation; however, this would risk updating the counter twice. This might be acceptable for a web site counter, because you can tolerate slightly over- or under-counting visitors. However, in a banking application, it would be safer to use conditional updates rather than atomic counters.

If you application needs to read multiple items, you can use the BatchGetItem API. A single BatchGetItem request can retrieve up to 1 MB of data, which can contain as many as 100 items. In addition, a single BatchGetItem request can retrieve items from multiple tables.

DynamoDB API

Note: This is an incomplete list. The following are the main API calls one can expect to see on an exam.

CreateTable 

Creates a table and specifies the primary index used for data access.

UpdateTable 

Updates the provisioned throughput values for the given table.

DeleteTable 

Deletes a table.

DescribeTable 

Returns table size, status, and index information.

ListTables 

Returns a list of all tables associated with the current account and endpoint.

PutItem 

Creates a new item, or replaces an old item with a new item (including all the attributes). If an item already exists in the specified table with the same primary key, the new item completely replaces the existing item. You can also use conditional operators to replace an item only if its attribute values match certain conditions, or to insert a new item only if that item does not already exist.

BatchWriteItem 

Inserts, replaces, and deletes multiple items across multiple tables in a single request, but not as a single transaction. Supports batches of up to 25 items to Put or Delete, with a maximum total request size of 16 MB.

UpdateItem 

Edits an existing item's attributes. You can also use conditional operators to perform an update only if the item's attribute values match certain conditions.

DeleteItem 

Deletes a single item in a table by primary key. You can also use conditional operators to perform a delete an item only if the item's attribute values match certain conditions.

GetItem 

The GetItem operation returns a set of Attributes for an item that matches the primary key. The GetItem operation provides an eventually consistent read by default. If eventually consistent reads are not acceptable for your application, use ConsistentRead.

BatchGetItem 

The BatchGetItem operation returns the attributes for multiple items from multiple tables using their primary keys. A single response has a size limit of 16 MB and returns a maximum of 100 items. Supports both strong and eventual consistency.

Query 

Gets one or more items using the table primary key, or from a secondary index using the index key. You can narrow the scope of the query on a table by using comparison operators or expressions. You can also filter the query results using filters on non-key attributes. Supports both strong and eventual consistency. A single response has a size limit of 1 MB.

Scan 

Gets all items and attributes by performing a full scan across the table or a secondary index. You can limit the return set by specifying filters against one or more attributes.

A Scan operation on a table or secondary index has a limit of 1MB of data per operation. After the 1MB limit, it stops the operation and returns the matching values up to that point, and a LastEvaluatedKey to apply in a subsequent operation, so that you can pick up where you left off.

Using Web Identity Providers with DynamoDB

One can authenticate users using Web Identity providers (e.g., Facebook, Google, Amazon, or any other Open-ID Connect-compatible identity provider). This is done using AssumeRoleWithWebIdentity API.

You will need to create a role first.

1. Authenticate with Identity Provider (e.g., Facebook): Log into Facebook with your username + password
2. Facebook returns a Web Identity Token
3. AssumeRoleWithWebIdentity request (containing Web Identity Token, App ID of provider, and ARN of role) is sent to AWS Security Token Service
4. Amazon then issues Temporary Security Credentials (limit from 15 minutes to 1 hour; default 1 hour)

   Credentials contain:

   1. AccessKeyID, SecretAccessKey, SessionToken
   2. Expiration (time limit)
   3. AssumeRoleID
   4. SubjectFromWebIdentity Token (the unique ID that appears in an IAM policy variable for this particular identity provider)
5. Using the above credentials, the user is allowed to access DynamoDB

Example exam questions

• DynamoDB provisioned throughput calculations
  • Unit of Read provisioned throughput
    • All reads are rounded up to increments of 4 KB in size.
    • Eventually consistent Reads (default) consist of 2 reads per second.
    • Strongly consistent Reads consist of 1 read per second.
  • Unit of Write provisioned throughput
    • All writes are 1 KB in size.
    • All writes consist of 1 write per second.

• The Magic Formula

(size of Read rounded to nearest 4 KB chunk / 4 KB) x (number of items) = read throughput
# divide by 2 if eventually consistent.

Route53

DNS 101

• Root Zone Database — by the Internet Assigned Numbers Authority (IANA)

• The Start of Authority (SOA) record stores information about:
  • The name of the server that supplied the data for the zone;
  • The administrator of the zone;
  • The current version of the data file;
  • The number of seconds a secondary name server should wait before checking for updates;
  • The number of seconds a secondary name server should wait before retrying a failed zone transfer;
  • The maximum number of seconds that a secondary name server can use data before it must either be refreshed or expire; and
  • The default number of seconds for the time-to-live (TTL) file on resource records.

• Name Server (NS) records:
  • Used by Top Level Domain servers to direct traffic to the Content DNS server, which contains the authoritative DNS records.

• A Records:
  • An A record is the fundamental type of DNS record and the "A" in A record stands for "Address".
  • The A record is used by a computer to translate the name of the domain to the IP address (e.g., http://www.example.com => http://1.2.3.4).

• TTL
  • The length that a DNS record is cached on either the Resolving Server or the user's own local PC is equal to the value of the "Time To Live" (TTL) in seconds. The lower the TTL, the faster changes to DNS records take to propagate throughout the Internet.

• CNAMES
  • A Canonical Name (CName) can be used to resolve one domain name to another. For example, you may have a mobile website with the domain name http://m.example.com that is used for when users browse to your domain name on their mobile devices. You may also want the name http://mobile.example.com to resolve to this same address.

• Alias Records
  • Used to map resource record sets in your hosted zone to ELBs, CloudFront distributions, or S3 buckets that are configured as websites.
  • Alias records work like a CNAME record, in that you can map one DNS name (www.example.com) to another "target" DNS name (elb1234.elb.amazonaws.com).
  • The key difference: A CNAME cannot be used for naked domain names (zone apex; e.g., example.com, not www.example.com). You cannot have a CNAME for http://example.com, it must be either an A record or an Alias.
  • Alias resource record sets can save you time because Route53 automatically recognizes changes in the record sets that the alias resource record set refers to.
  • For example, suppose an alias resource record set for example.com points to an ELB at lb1-1234.us-west-2.elb.amazonaws.com. If the IP address of the load balancer changes, Route53 will automatically reflect those changes in DNS answers for example.com without any changes to the hosted zone that contains resource record sets for example.com.

Simple Queue Service (SQS)

SEE: Amazon SQS FAQs

Very first service offered by AWS

• SQS vs. RabbitMQ:
  • SQS is a managed service. So one does not have to worry about operational aspects of running a messaging system including administration, security, monitoring, etc. Amazon will do this for you and will provide support if something were to go wrong.
  • SQS is Elastic and can scale to very large rate/volumes (unlimited according to AWS)
  • Availability of SQS has a lot of 9's in it and is backed by Amazon, which is one less thing to worry about in your application.

Amazon SQS is a web service that gives you access to a message queue that can be used to store messages while waiting for a computer to process them.

SQS is a distributed queue system that enables web service applications to quickly and reliably queue messages that one component in the application generates to be consumed by another component. A queue is a temporary repository for messages that a awaiting processing.

Using SQS, you can decouple the components of an application so they run independently, with SQS easing message management between components. Any component of a distributed application can store messages in a fail-safe queue. Messages can contain up to 256 KB of text in any format. Any component can later retrieve the messages programmatically using the SQS API.

The queue acts as a buffer between the component producing and saving data, and the component receiving the data for processing. This mean that queue resolves issues that arise if the producer is producing work faster than the consumer can process it, or if the producer or consumer are only intermittently connected to the network.

Amazon SQS ensures delivery of each message at least once, and support multiple readers and writers interacting with the same queue. A single queue can be used simultaneously by many distributed application components, with no need for those components to coordinate with each other to share the queue.

SQS is engineered to always be available and deliver messages. One of the resulting trade-offs is that SQS does not guarantee first in, first out delivery of messages. For many distributed applications, each message can stand on its own, and as long as all messages are delivered, the order is not important. If your system requires that order be preserved, you can place sequencing information in each message, so that you can reorder the messages when the queue returns them.

To illustrate, suppose you have a number of images files to encode. In a SQS worker queue, you create a SQS message for each file specifying the command (jpeg-encode) and the location of the file in S3. A pool of EC2 instances running the needed image processing software does the following:

1. Asynchronously pulls the task messages from the queue;
2. Retrievers the named file;
3. Processes the conversion (e.g., create a thumbnail, add a watermark, etc.)
4. Write the image back to Amazon S3;
5. Writes a "task complete" message to another queue;
6. Deletes the original task message; and then
7. Checks for more messages in the worker queue

Visibility Timeout Clock only starts when the component server (i.e., EC2 instance) pulls the message from the queue

• SQS with Auto-scaling
• SQS does not offer FIFO (first in, first out)
• 12 hour visibility time out
• SQS is engineered to provide "at least once" delivery of all messages in its queues. Although most of the time each message will be delivered to your application exactly once, you should design your system so that processing a message more than once does not create any errors or inconsistencies.
• 256kb message size (as of May 2016)
• Billed at 64kb "chunks"
• A 256kb message will be 4 x 64kb "chunks"

• SQS Pricing
  • First 1 million SQS requests per month are free
  • $0.50 per 1 million SQS requests per month thereafter ($0.0000005 per SQS request)
  • A single request can have from 1 to 10 messages, up to a maximum total payload of 256kb
  • Each 64kb "chunk" of payload is billed as 1 request. For example, a single API call with a 256kb payload will be billed as four requests.

• If you see "decouple", think SQS

• SQS Delivery:
  • SQS messages can be delivered multiple times and in any order (no first in, first out; last in, last out)

• SQS - Default visibility timeout:
  • Default visibility timeout is 30 seconds
  • Maximum timeout is 12 hours
  • When you receive a message from a queue and begin processing it, you may find the visibility timeout for the queue is insufficient to fully process and delete that message. To give yourself more time to process the message, you can extend its visibility timeout by using the ChangeMessageVisibility action to specify a new timeout value. SQS restarts the timeout period using the new value.

• SQS Long Polling:
  • SQS long polling is a way to retrieve messages from your SQS queues. While the traditional SQ short polling returns immediately, even if the queue being polled is empty, SQS long polling does not return a response until a message arrives in the queue, or the long poll times out. SQS long polling makes it easy and inexpensive to retrieve messages from your SQS queue as soon as they are available.
  • Maximum Long Poll timeout is 20 seconds
  • Example exam question: Polling in tight loops is burning CPU cycles and costing the company money. How would you fix this? (Answer: Enable SQS Long Polling.)

• SQS - Fanning Out:
  • Create an SNS topic first using SNS. Then create a and subscribe multiple SQS queues to the SNS topic. Now whenever a message is sent to the SNS topic, the message will be fanned out to the SQS queues (i.e., SNS will deliver the message to all the SQS queues that are subscribed to the topic).

Simple Notification Service (SNS)

SEE: Amazon SNS FAQs

SNS is a web service that makes it easy to set up, operate, and send notifications from the cloud. It provides developers with a highly scalable, flexible, and cost-effective capability to publish messages from an application and immediately deliver them to subscribers or other applications.

SQS follows the "publish-subscribe" (pub-sub) messaging paradigm, with notifications being delivered to clients, using a "push" mechanism that eliminates the need to periodically check or "poll" for new information and updates. With simple APIs requiring minimal up-front development effort, no maintenance or management overhead and pay-as-you-go pricing, SNS gives developers an easy mechanism to incorporate a powerful notification system with their applications.

Push notifications to Apple, Google, Fire OS, and Windows devices, as well as Android devices in China with Baidu Cloud Push.

Besides pushing cloud notifications directly to mobile devices, SNS can also deliver notifications by SMS text message or email, to Amazon Simple Queue Service (SQS) queues or to any HTTP endpoint.

To prevent messages from being lost, all messages published to SNS are stored redundantly across multiple availability zones.

• SNS - Topics:
  • SNS allows you to group multiple recipients using "topics". A topic is an "access point" for allowing recipients to dynamically subscribe to identical copies of the same notification. One topic can support deliveries to to multiple endpoint types. For example, one can group together iOS, Android, and SMS recipients. When you publish once to a topic, SNS delivers appropriately formatted copies of your message to each subscriber.
  • Subscriptions via email require the receiving email owner to confirm the subscriptions in order to receive notifications from the given topic (prevents spam). Subscription requests expire after 3 days, if the receiving email owner does not confirm the subscription.

Example SNS email notification:

{
  "Type" : "Notification",
  "MessageId" : "436d9234-f427-5be8-aa54-dd98ae4e286dba0",
  "TopicArn" : "arn:aws:sns:us-west-2:01234:MyTestSNSTopic",
  "Subject" : "This is a test",
  "Message" : "Hello, world!",
  "Timestamp" : "2016-05-10T21:53:37.981Z",
  "SignatureVersion" : "1",
  "Signature" : "PInj15UDcwMI==",
  "SigningCertURL" : "https://sns.us-west-2.amazonaws.com/SimpleNotificationService-9390147a5624348ee.pem",
  "UnsubscribeURL" : "https://sns.us-west-2.amazonaws.com/?Action=Unsubscribe&SubscriptionArn=arn:aws:sns:us-west-2:01234:MyTestSNSTopic:1a4eead5-110c-4acb-894a-91cdf358aabc",
  "MessageAttributes" : {
    "AWS.SNS.MOBILE.MPNS.Type" : {"Type":"String","Value":"token"},
    "AWS.SNS.MOBILE.MPNS.NotificationClass" : {"Type":"String","Value":"realtime"},
    "AWS.SNS.MOBILE.WNS.Type" : {"Type":"String","Value":"wns/badge"}
  }
}

• SNS benefits:
  • Instantaneous, push-based delivery (no polling)
  • Simple APIs and easy integration with applications
  • Flexible message delivery over multiple transport protocols
  • Inexpensive, pay-as-you-go model with no up-front costs
  • Web-based AWS Management Console offers the simplicity of a point-and-click interface

• SNS vs. SQS:
  • Both messaging services in AWS
  • SNS => Push
  • SQS => Pulls (Polls)

• SNS Pricing:
  • Users pay $0.50 per 1 million SNS requests
  • $0.06 per 100,000 notification deliveries over HTTP
  • $0.75 per 100 notification deliveries over SMS
  • $2.00 per 100,000 notification deliveries over email

SNS data format = JSON

• SNS protocols include:
  • HTTP
  • HTTPS
  • Email
  • Email-JSON
  • Amazon SQS
  • Application

NOTE: Messages can be customized for each protocol

Simple Workflow Service (SWF)

SEE: Amazon SWF FAQs

Amazon Simple Workflow Server (Amazon SWF) is a web service that makes it easy to coordinate work across distributed application components. SWF enables applications for a range of use cases, including media processing, web application back-ends, business process work-flows, and analytics pipelines, to be designed as a coordination of tasks. Task represent invocations of various processing steps in an application, which can be performed by executable code, web service calls, human actions, and scripts.

SWF Workers 

programs that interact with SWF to get tasks, process received tasks, and return the results.

SWF Decider 

a program that controls the coordination of tasks, i.e., their ordering, concurrency, and scheduling according to the application logic.

The workers and the decider can run on cloud infrastructure, such as Amazon EC2, or on machines behind firewalls. SWF brokers the interactions between workers and the decider. It allows the decider to get consistent views into the progress of tasks and to initiate new tasks in an ongoing manner. At the same time, SWF stores tasks, assigns them to workers when they are ready, and monitors their progress. It ensures that a task is assigned only once and is never duplicated. Since SWF maintains the application's state durably, workers and deciders do not have to keep track of execution state. They can run independently and scale quickly.

SWF Domains 

Your work-flow, activity types, and the workflow execution itself are all scoped to a domain. Domains isolate a set of types, executions, and task lists from others within the same account. One can register a domain by using the AWS Management Console or by using the RegisterDomain action in the SWF API.

The parameters of an SWF Domain are specified in JSON format. E.g.:

https://swf.us-west-2.amazonaws.com
RegisterDomain
{
  "name": "123456789",
  "description": "images",
  "workflowExecutionRetentionPeriodInDays": "60"
}

Maximum Workflow can be 1 year and the value is always measured in seconds.

• SWF vs. SQS:
  • SWF presents a task-oriented API, whereas SQS offers a message-oriented API.
  • SWF ensures that a task is assigned only once and is never duplicated. What SQS, one needs to handle duplicate messages and may also need to ensure that a message is processed only once.
  • SWF keeps track of all the tasks and events in an application. With SQS, one needs to implement one's own application-level tracking, especially if one's application uses multiple queues.
  • Does the service require human interaction? If so, one should use SWF.
  • Does the service need to run for (much) more than 12 hours? If so, one should use SWF. If less than 12 hours, SQS might be the correct service to use.
  • Maintaining your application's execution state (e.g. which steps have completed, which ones are running, etc.) is a perfect use case for SWF.
  • Amazon SWF is useful for automating work-flows that include long-running human tasks (e.g. approvals, reviews, investigations, etc.) Amazon SWF reliably tracks the status of processing steps that run up to several days or months.

CloudFormation

SEE: Amazon CloudFormation FAQs

CloudFormation => Scripted infrastructure (IaaS)

Using the CloudFormation service is free. However, any resources it creates/consumes/provisions (e.g., EC2 instances, Load Balancers, etc.) are not free.

CloudFormation templates are written in JSON format. E.g., CloudFormation LAMP stack template (see here for full template):

{
  "AWSTemplateFormatVersion" : "2010-09-09",
  
  "Description" : "AWS CloudFormation Sample Template LAMP_Single_Instance:
Create a LAMP stack using a single EC2 instance and a local MySQL database for
storage. This template demonstrates using the AWS CloudFormation bootstrap
scripts to install the packages and files necessary to deploy the Apache web
server, PHP and MySQL at instance launch time. **WARNING** This template
creates an Amazon EC2 instance. You will be billed for the AWS resources used
if you create a stack from this template.",
  
  "Parameters" : {
      
    "KeyName": {
      "Description" : "Name of an existing EC2 KeyPair to enable SSH access to the instance",
      "Type": "AWS::EC2::KeyPair::KeyName",
      "ConstraintDescription" : "must be the name of an existing EC2 KeyPair."
    }, 
...

• CloudFormation "output":
  • SEE: http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/outputs-section-structure.html
  • Remember the "Fn::GetAtt" function

If a CloudFormation stack creation fails, the default is to terminate and roll-back all resources created on failure (i.e., delete all of the resources it was trying to create). One can disable roll back to leave all resources in their current state (failed or not). This is useful for troubleshooting your own templates.

Elastic Beanstalk

AWS Elastic Beanstalk is an easy-to-use service for deploying and scaling web applications and services developed with Java, .NET, PHP, Node.js, Python, Ruby, Go, and Docker on familiar servers such as Apache, Nginx, Passenger, and IIS.

You can simply upload your code and Elastic Beanstalk automatically handles the deployment, from capacity provisioning, load balancing, auto-scaling, to application health monitoring. At the same time, you retain full control over the AWS resources powering your application and can access the underlying resources at any time.

Using the AWS Elastic Beanstalk service is free. However, any AWS resources it creates/consumes/provisions to store and run your applications are not free.

• Environment tier:
  • Web Server Environment - Provides resources for an AWS Elastic Beanstalk web server in either a single instance or load-balancing, auto scaling environment.
  • Worker Environment - Provides resources for an AWS Elastic Beanstalk worker application in either a single instance or load-balancing, auto scaling environment.
• Environment type:
  • Single instance
  • Load balancing, auto-scaling
• Preconfigured platforms:
  • PHP, Node.js, Python, Ruby, Tomcat, IIS, Java, Go, Docker

Virtual Private Cloud (VPC)

SEE: Amazon VPC FAQs

Think of a VPC as a virtual data centre in the cloud.

• AWS definition of a VPC:
  • Amazon Virtual Private Cloud (VPC) lets you provision a logically isolated section of the AWS Cloud where you can launch AWS resources in a virtual network that you define. You have complete control over your virtual networking environment, including select of you own IP address range, creation of subnets, and configuration of route tables and network gateways.
  • You can easily customize the network configuration for your AWS virtual Private Cloud. For example, you can create a public-facing subnet for your webservers that has access to the Internet, and place backend systems, such as databases or application service in a private-facing subnet, with no Internet access. You can leverage multiple layers of security, including security groups and network access control lists, to help control access to EC2 instance in each subnet.
  • Additionally, one can create a Hardware Virtual Private Network (VPN) connection between your corporate datacentre and your VPC and leverage the AWS cloud as an extension of your corporate datacentre. "Hybrid Cloud".

• What can one do with a VPC?
  • Launch instances into a subnet of one's choosing
  • Assign custom IP address ranges in each subnet
  • Configure route tables between subnets
  • Create internet gateways and attach them to subnets (or not)
  • Much better security control over your AWS resources
  • Instance security groups
  • Create subnet network access control lists (ACLs)
  • Number of allowed VPCs in each AWS Region (by default): 5

• Default VPC vs. Custom VPC
  • Default VPC is user friendly (automatically created when one creates an AWS account). It allows one to immediately deploy instances.
  • All subnets in a default VPC have an internet gateway attached (i.e., all subnets are public)
  • Each EC2 instance has both a public and private IP address
  • If one were to delete the default VPC, the only way to get it back is to contact AWS

• VPC Peering
  • Allows one to connect one VPC with another via a direct network route using private IP addresses.
  • Instances behave as if they were on the same private network.
  • One can peer VPCs with other AWS accounts as well as with other VPCs in the same account.
  • One cannot create a VPC larger than /16
  • Peering is done in a "star configuration", i.e., 1 central VPC peers with 4 others. No transitive peering!

• A "star configuration" peering:

                 +-------+
                 | VPC C |
                 +-------+
                     ^
                     |
                     v
+-------+        +-------+        +-------+
| VPC B | <----> | VPC A | <----> | VPC E |
+-------+        +-------+        +-------+
                     ^
                     |
                     v
                 +-------+
                 | VPC D |
                 +-------+

In the above example, instances on VPC-B can not send/receive traffic on VPC-C via VPC-A. One would need to create a VPC peer directly from VPC-B and VPC-C. That is, with a star configuration (as shown above), there is no transitive peering.

• VPC tenancy:
  • Default - EC2 instances are created on shared hardware
  • Dedicated - EC2 instances are created on dedicated hardware (regardless of the tenancy attribute specified at launch). This the more expensive option.

By default, when one creates a VPC, a route tables is automatically created for the VPC.

NOTE: If one deletes one's account's default VPC (and/or the associated default subnets), the only way to get them back is to raise a ticket with Amazon.

• VPC subnets:
  • Use the CIDR format to specify your subnet's IP address block (e.g., 10.0.0.0/24). Note that block sizes must be between a /16 netmask and /28 netmask. Also, note that a subnet can be the same size as your VPC.
  • Subnets are always mapped to one availability zone (AZ). Subnets can not be mapped across multiple AZs. 1 subnet = 1 AZ.

• VPC Internet gateways:
  • By default, when one creates an Internet Gateway, it is detached. One must attach it to a VPC in order to use it.
  • One can only have 1 Internet Gateway per VPC.

Security Groups can span multiple VPCs and VPC subnets.

• Network Address Translation (NAT) Server:
  • Allow instances with only private IPs to reach the Internet via the NAT server (via, say, HTTP/HTTPS and all other protocols/ports closed, including SSH)
  • One must disable Source/Destination Check on NAT instances for them to work properly.

Network Access Control Lists (ACLs)

• A network ACL is an optional layer of security that acts as a firewall for controlling traffic in and out of a subnet.
• ACLs ~ "firewall"-like rules
• ACLs are a numbered list of rules, which are followed in order, starting with the lowest number first. They control network ingress/egress for all AWS resources within a given subnet.
• The highest ACL number allowed is 32,766
• ACLs have a default (editable) number list that allows all inbound/outbound traffic
• One can create a custom ACL, which starts out with no inbound/outbound traffic allowed, until one adds a rule
• ACLs are applied to an entire subnet (and override the security groups associated with a given instance on that subnet). For an example, if a security group applied to a given instance has port 80 allowed, but the ACL for the subnet the instance is on has port 80 denied, the ACL rule takes precedence (i.e., port 80 will be denied on all instances within that subnet, regardless of what the security group allows).
• Unless one creates a custom ACL and associates it with a given subnet, that ACL will use the default role and rules.
• One can not have multiple ACLs associated with the same subnet. However, a given ACL can be associated with multiple subnets.
• If one dis-associates a custom ACL from a given subnet(s), the subnet reverts back to the default ACL

Example labs

• VPC Lab:
  • Create a custom VPC
    • Define an IP Address Range (e.g., 10.0.0.0/16)
    • By default, this created a Network ACL and Routing Table
  • Create a custom Route Table
  • Create 3 subnets (e.g., 10.0.1.0/24, 10.0.2.0/24, and 10.0.3.0/24)
    • public subnet: 10.0.1.0/24; private subnets: 10.0.2.0/24 and 10.0.3.0/24
  • Create an Internet Gateway
  • Attach Internet Gateway to the custom VPC
  • Associate the public subnet with the custom Route Table
  • Provision an EC2 instance with an Elastic IP address (in the public subnet)
  • Provision an EC2 instance with only a private IP address (on the private subnet)

• NAT Lab:
  • Create a custom security group
  • Allow inbound traffic to 10.0.1.0/24 and 10.0.2.0/24 on HTTP and HTTPS
  • Allow outbound traffic on HTTP and HTTPS to anywhere
  • Provision a NAT instance inside the public subnet
  • Import! Make sure to select "Disabled Source/Destination Check" on this NAT instance!
  • Set up a route on the private subnet to route traffic through the NAT instance

AWS Shared Responsibility

Shared Responsibility Model for AWS Infrastructure Services

• Infrastructure services include: EC2, EBS, Auto-scaling, VPC, etc
• Amazon responsibility (i.e., managed by AWS):
  • AWS global infrastructure (regions, availability zones, edge locations)
  • Foundation services (compute, storage, databases, networking)
  • AWS endpoints
  • AWS IAM
• Customer responsibility (i.e., managed by AWD customers):
  • Server-side encryption, network traffic protection
  • Operating system, network, and firewall configuration(s)
  • Platform and application management
  • Customer data
  • Customer IAM

Shared Responsibility Model for AWS Container Services

• Container services include: Relational Database Services (RDS), Elastic Map Reduce (EMR), and Elastic Beanstalk
• AWS takes on more of the responsibility (e.g., operating system and network configuration; platform and application management)
• Customer still has the responsibility for firewall configuration

Shared Responsibility Model for AWS abstracted services

• Abstracted services include: S3, Glacier, DynamoDB, SQS, and Simple Email Service (SES), Lambda
• AWS takes on even more of the responsibility (e.g., network traffic protection provided by the platform; server-side encryption provided by the platform)
• Customer still has the responsibility for client-side data encryption, data integrity authentication, and customer data

Exams overview

AWS Certified Developer - Associate

• Time allotted: 80 minutes
• 55 questions on the exam
• $150 example registration fee
• Conducted Online at an approved centre

AWS Certified Solutions Architect - Associate

• Time allotted: 80 minutes
• 60 questions on the exam
• $150 example registration fee
• Conducted Online at an approved centre
• AWS platforms covered:
  • Security & Identity
  • Compute
  • Storage
  • Databases
  • Networking & Content Delivery
  • Messaging
  • Desktop & App Streaming (only at a very high-level)
  • Management Tools (only at a very high-level)
• AWS Global Infrastructure (what all of the above platforms/services reside in)
  • As of December 2016: 14 Regions and 38 Availability Zones (AZs)
  • In 2017: 4 more Regions and 11 more AZs
• Edge Locations are CDN Endpoints for CloudFront (as of December 2016, there are ~66 Edge Locations)

AWS Certifications

NOTE: All AWS certification exams are taken on-site and proctored.

• Associate Level ($150 each):
  • AWS Certified Developer - Associate
  • AWS Certified Solutions Architect - Associate
  • AWS Certified SysOps Administrator - Associate
• Professional Level ($300 each):
  • AWS Certified DevOps Engineer - Professional
  • AWS Certified Solutions Architect - Professional
• Specialty (Beta, as of January 2017)
  • AWS Certified Security - Specialty
  • AWS Certified Big Data - Specialty
  • AWS Certified Advanced Networking - Specialty

The AWS Partner Program

Glossary

see: Official AWS Glossary

AMI

Amazon Machine Image

ARN

Amazon Resource Name

EBS

Elastic Block Storage (victual disks for EC2 instances)

EC2

Elastic Compute Cloud

EC2 Container Service (ECS)

Elastic Beanstalk

EFS

Elastic File System

ELB

Elastic Load Balancer

Lambda

Serverless code

Lightsail

Out-of-the-box Cloud

STS

Security Token Service

VPC

Virtual Private Cloud

Route53

DNS + ability or register domain names

CloudFront

Content Delivery Network (CDN) / Edge Locations

DirectConnect

Storage

S3 

Simple Storage Service (object-based storage)

Glacier 

Data archival (for objects in S3). Low cost.

EFS

Elastic File Service (block-based storage; shareable)

Storage Gateway 

Connect S3 to on-premise DC

Databases

RDS 

Relational Database Service (e.g., MySQL, MariaDB, Aurora, Postgres, etc.)

DynamoDB 

NoSQL (non-relational database)

Redshift

Data warehousing

Elasticache 

Migration

Snowball 

Move large amounts of data into the Cloud (e.g., contents of a HDD)

DMS

Database Migration Service (e.g. in-house Oracle DB into AWS RDS:Aurora)

Server Migration Service 

Virtual machine migration (e.g., on-premise VMware VMs into AWS)

Analytics

Athena 

Run SQL queries on S3 (e.g., CSV/JSON files)

EMR 

Elastic MapReduce (process large amounts of data {Big Data}; e.g., log files)

CloudSearch 

Elastic Search 

Kinesis 

Stream and analyse live/real-time data (e.g., financial data, social media feeds, etc.)

Data Pipeline 

Allows moving data from one location to another (e.g., from S3 to DynamoDB or vice versa, etc.)

Quick Sight 

Business analytics tool

Security & Identity

IAM 

AWS Identity and Access Management

Inspector 

Agent-based service to inspect EC2 instances, etc.

Certificate Manager 

Free SSL certs

Directory Service 

Active Directory in the Cloud

WAF 

Web Application Firewall (e.g., protect against SQL injections, etc.)

Artifacts 

Compliance Reports (e.g., ISO 27001 certification, etc.)

Management Tools

Cloud Watch 

Monitor performance of AWS (e.g. EC2 => CPU/RAM util)

Cloud Formation 

Infrastructure as Code (document-based; JSON/YAML)

Cloud Trail 

Audit AWS resource usage

OpsWorks 

Chef for AWS

Config 

Monitor AWS environment (e.g., send alert if someone creates an IAM role that breaks company policy)

Trusted Advisor 

Automate performance, security, fault-tolerance, cost, etc.

Application Services

Step Functions

Visualize what is going on inside an application (and/or microservice)

SWF 

Simple Workflow Service (coordinate automated vs. human tasks)

API Gateway 

Create, publish, maintain APIs in the Cloud

AppStream 

Stream desktop applications to users

Elastic Transcoder 

Change video format (e.g., for viewing on different devices)

Developer Tools

CodeCommit 

GitHub in AWS

CodeBuild 

Compile code in the Cloud

CodeDeploy 

Deploy code to EC2 instances

CodePipeline 

Keep track of versions of code (e.g., dev, test, prod, UAT)

Mobile Services

Mobile Hub 

Cognito

Device Farm

Mobile Analytics

Pinpoint 

Google Analytics for mobile applications

Business Productivity

WorkDocs

WorkMail 

Exchange for AWS

Internet of Things (IoT)

IoT

Desktop & App Streaming

WorkSpaces 

Virtual Desktops in the Cloud / Virtual Desktop Infrastructure (VDI) solutions

AppStream 2.0 

Stream dekstop applications to users

Artificial Intelligence

Lex 

Think "Alexa in the Cloud" or Alexa on a RaspberryPi

Polly 

Text-to-Speech (text => mp3 in S3)

Machine Learning

Rekognition 

Analyse pictures with tagging and facial recognition

Messaging

SNS 

Simple Notification Service

SQS 

Message Queue Service

SES 

Simple Email Service

Links

AWS Whitepapers

• AWS Whitepapers
  • AWS Security Best Practices (PDF)
  • AWS Security Whitepaper (PDF)
  • AWS Storage Services Whitepaper (PDF)
  • AWS Well-Architected Framework (PDF)

Training and certification

• https://acloud.guru/courses
• https://aws.amazon.com/certification/certified-developer-associate/
• What you need to get AWS certified — by A Cloud Guru
• Register for AWS certification exams

Miscellaneous

• http://ec2price.com/
  • https://github.com/grosskur/ec2price
• AWS in Plain English