Difference between revisions of "Category:AWS"

Revision as of 19:31, 20 May 2016

This category/article is (currently) just a random collection of my notes on Amazon Web Services (AWS). I will organize each service type into a separate article, when I find the time.

Elastic Block Storage (EBS)

SEE: Amazon EBS FAQs

• EBS Volume types:
  • General purpose SSD (GP2): up to 10,000 IOPS
  • Provisioned IOPS SSD (IO1): more than 10,000 IOPS
  • Magnetic (Standard): infrequently accessed storage

NOTE: One can not mount 1 EBS volume to multiple EC2 instances; use EFS instead.

In order to enable encryption at rest using EC2 and Elastic Block Store, one needs to configure encryption when creating the EBS volume.

Elastic Compute Cloud (EC2)

SEE: Amazon EC2 FAQs

• EC2 provision types:
  • On-Demand
  • Reserved
    • Amazon EC2 Reserved Instances allow you to reserve Amazon EC2 computing capacity for 1 or 3 years, in exchange for a significant discount (up to 75%) compared to On-Demand instance pricing.
  • Spot
    • Amazon EC2 Spot instances allow you to bid on spare Amazon EC2 computing capacity. Since Spot instances are often available at a discount compared to On-Demand pricing, you can significantly reduce the cost of running your applications, grow your application's compute capacity and throughput for the same budget, and enable new types of cloud computing applications.
    • Example use cases: genomics, drug companies, etc.
    • If you terminate, you pay for the hour the instance was terminated
    • If Amazon terminates it, you do not pay for the hour the instance was terminated
    • EC2 Spot prices

• EC2 Instance Types:
  • D for Density
  • I for IOPS
  • R for RAM
  • T general purpose (think T2 Micro)
  • M for Main choice (for general purpose apps)
  • C for Compute
  • G for Graphics
  • mnemonic: DIRTMCG

• EC2 metadata:

$ curl http://169.254.169.254/latest/meta-data/public-ipv4 # returns the EC2 instance's public IPv4

NOTE: If one makes an Amazon Machine Image (AMI) public, this AMI is not immediately available across all regions, by default.

With EC2 you can have 2 types of storage: EBS storage and Instance Store. EBS is persistent and if an EC2 instance is stopped with an EBS volume attached, there will be no data lost. Instance Store is ephemeral and if the EC2 instance is stopped, all data will be lost.

Elastic Load Balancer (ELB)

SEE: Amazon ELB FAQs

• ELBs are not free; one is charged by the hour and on a per GB basis of usage

• ELB supported ports:
  • ec2-vpc: 1-65535
  • ec2-classic: 25, 80, 443, 465, 587, 1024-65535

• ELB supported protocols:
  • HTTP, HTTPS, TCP, SSL

NOTE: One can have multiple SSL certificates (for multiple domain names) on a single Elastic Load Balancer.

SDKs

• HTTP codes:
  • 200 - The request has succeeded
  • 3xx - Redirection
  • 4xx - Client error (think 404 not found)
  • 5xx - Server error (think Apache service not running, etc.)

• Available SDKs:
  • Android, iOS, JavaScript (browser)
  • Java
  • .Net
  • Node.js
  • PHP
  • Python
  • Ruby
  • Go
  • C++ (preview)

• SDK default regions:
  • default region (for most SDKs): us-east-1
  • Some SDKs have default regions set (Java)
  • Some SDKs do not (Node.js)

Services that are free include: CloudFormation, Elastic Beanstalk, Autoscaling, Opsworks, etc. (however, the resources they create are not free; e.g., EC2 instances, ELBs)

Simple Storage Service (S3)

SEE: Amazon S3 FAQs

Amazon Simple Storage Service (Amazon S3), provides developers and IT teams with secure, durable, highly-scalable cloud storage. Amazon S3 is easy to use object storage, with a simple web service interface to store and retrieve any amount of data from anywhere on the web. With Amazon S3, you pay only for the storage you actually use. There is no minimum fee and no setup cost.

• Simple key-value store:
  • key = name of the object;
  • value = the actual data (made up of a sequence of bytes);
  • version ID (important for versioning); and
  • metadata (data about the data you are storing)
  • Sub-resources
  • Access Control Lists (ACLs)
• S3 bucket URL: https://s3-<region>.amazonaws.com/<bucket_name> (e.g., https://s3-us-west-1.amazonaws.com/foobar)
• SLA:
  • availability: 99.99% (2 nines)
  • durability: 99.99999999999% (11 nines)
• Files can be from 1 byte to 5 TB in size (split files larger than 5 TB into pieces to upload)
  • Note that the largest size file you can transfer to S3 using a PUT operation is 5 GB
• Unlimited storage
• Files/objects are stored in "buckets"
• S3 is a universal namespace (i.e., bucket names must be unique globally; think domain names)
• Read-after-Write consistency for PUTs of new Objects
• Eventual Consistency for overwrite PUTs and DELETEs (can take some time to propagate)
• Lifecyle management
• Versioning
• Encryption (default Advanced Encryption Standard (AES) 256bit)
  • In transit (SSL/TLS)
  • At Rest:
    • Server Side Encryption (SSE):
      • S3 Managed Keys (SSE-S3; 256bit);
      • AWS Key Management Service, Managed Keys (SSE-KMS)
      • Server Siide Encryption with Customer Provided Keys (SSE-C)
  • Client Side Encryption (user encypts data on their local machine and then upload to AWS S3)
• Secure your data with Bucket Policies and ACLs
• Storage tiers/classes:
  • S3 99.99% (durable, immediately available, frequently accessed): stored across multiple devices in multiple facilities and is designed to sustain the loss of 2 facilities concurrently
  • S3 IA (Infrequently Accessed) (durable, immediately available, infrequently accessed): for data that is accessed less frequently, but requires rapid access when needed. Lower fee than S3, but you are charged a retrieval fee
  • S3 Reduced Redundancy Storage (RRS): designed to provide 99.99% availability/durability of objects over a given year (for objects where it is not critical if they are lost; e.g., thumbnails of images, as they can be easily regenerated). concurrent facility fault tollerance = 1
  • Glacier: Very cheap, but used for archival only. It takes 3-5 hours to restore from Glacier.

• Storage Gateways:
  • Gateway Stored Volumes (entire dataset is stored on site and is asynchronously backed up to S3)
  • Gateway Cached Volumes (entire dataset is stored on S3 and the most frequently accessed data is cached on site)
  • Gateway Virtual Tape Library (VTL) (used for backup and uses popular backup applications like NetBackup, Backup Exec, Veam, etc.)

• Import/Export Disk:
  • Import to EBS
  • Import to S3
  • Import to Glacier
  • Export from S3
• Import/Export Snowball (only available in North America)
  • Import to S3
  • Export from S3

• S3 bucket names must contain only lower case characters
• S3 stores data in alphabetical order (lexigraphical order)
• Largest file size one can transfer to S3 using a PUT operation: 5 GB (Use multi-part upload for files larger than 5GB)

• S3 Static Website:
  • http://foobar.s3-website-us-west-2.amazonaws.com/index.html (a static website link)
  • https://s3-us-west-2.amazonaws.com/foobar/demo.jpeg (not a static website link)
  • Static websites are always HTTP (not HTTPS, for now)

• S3 Cross Origin Resource Sharing (CORS)
  • One needs to enable CORS for one S3 bucket to reference objects in another S3 bucket

• S3 Versioning
  • Stores all versions of an object (including all writes, including deleting the object)
  • Great backup tool
  • Once enabled, versioning cannot be disabled, only suspended
  • Integrates with Lifecycle rules
  • Versioning's MFA (Multi-Factor Authentication) delete capability, which uses MFA, can be used to provide an additional layer of security.
  • Cross Region Replication, requires versioning enabled on the source bucket.

• S3 Lifecyle Management
  • Transition objects to Infrequent Access Storage Class (must wait a minimum of 30 days from initial upload for the object to transition to the new storage class; minimum object size 128KB) or Glacier Storage Class after x amount of days.
  • Infrequent retrieval: ~milliseconds
  • Glacier retrieval: 3 - 5 hours
  • With Versioning: Transition object versions as well (including deleting old/current versions)
  • Can not use Reduced Durability Storage Class with Lifecycle Management

• CloudFront (a content delivery network {CDN})
  • AWS Global CDN infrastructure
  • Edge Location: this is the location where content will be cached. This is different from an AWS Region/AZ (over 50 Edge Locations, as of April 2016)
  • Origin: This is the origin of all the files/objects that the CDN will distribute. This can be an S3 Bucket, an EC2 Instance, an Elastic Load Balancer, or Route53.
    • Possible to have multiple origin paths in the same distribution
  • Distribution: This is the name given to the CDN, which consists of a collection of Edge Locations.
    • Web Distribution: Typically used for websites.
    • RTMP: Used for media streaming (e.g., Adobe Flash)
  • Edge location are not just READ only, one can write to them as well (i.e., PUT and object to them)
  • Objects are cached for the life of the TTL (Time To Live)
  • One can clear the cache of an object stored on an Edge Location before the TTL expires, but one will be charged for that service.

• S3 Transfer Acceleration:
  • Instead of uploading object directly to an S3 bucket in a given region, one can upload a object directly to the nearest Edge Location and AWS will then transfer the object to the S3 bucket.
  • Example URL: <bucket_name>.s3-accelerate.amazonaws.com

Databases

SEE: Amazon RDS FAQs

RDS 

Relational Database Server

• RDS (OLTP) Relational Database Types:
  • MS SQL Server
  • Oracle
  • MySQL Server
  • PostgreSQL
  • Aurora
  • MariaDB

• Non-relational Databases:
  • DynamoDB (document-oriented)
    • Collection (=Table)
    • Document (=Row)
    • Key-value pairs (=Fields)

E.g., JSON/NoSQL document

{
  "_id": "345sdf45asdf",
  "firstname": "John",
  "surname": "Smith",
  "age": "23",
  "address": [
    {
      "street": "123 First Street",
      "suburb": "Wallingford"
    }
  ]
}

• Redshift (OLAP)

• Elasticache
  • Memcached
  • Redis

DMS: (Online) Database Migration Service (convert Oracle to MySQL, etc.)

• Data Warehousing (business intelligence; e.g., Cognos, Jaspersoft, etc.)
• Online Transaction Processing (OLTP); vs.
• Online Analytics Processing (OLAP)

DynamoDB

SEE: Amazon DynamoDB FAQs (import to read for the exams!)

Amazon DynamoDB is a fast and flexible NoSQL database service for all applications that need consistent single-digit millisecond latency at any scale. It is a fully managed database and supports both document and key-value data models. Its flexible data model and reliable performance make it a great fit for mobile, web, gaming, ad-tech, IoT, and many other applications.

• Stored on SSD storage
• Spread across 3 geographically distinct data centres (note: not distinct AZs)
• Eventual Consistent READs (default)
  • Consistency across all copies of data is usually reached within a second. Repeating a read after a short time should return the update data. (best read performance.)
• Strongly Consistent READs
  • A strongly consistent read returns a result that reflects all writes that received a successful response prior to the read.

• The DynamoDB basics:
  • Tables
  • Items (think a row of data in the table)
  • Attributes (think of a column of data in a table)

Item 101 (has the following attributes):
{                                        
   Title = "Book 101 Title"
   ISBN = "111-1111111111"
   Authors = "Author 1"
   Price = "-2"
   Dimensions = "8.5 x 11.0 x 0.5"
   PageCount = "500"
   InPublication = true
   ProductCategory = "Book" 
}
Item 201 (has the following attributes):
{
   Title = "18-Bicycle 201"
   Description = "201 description"
   BicycleType = "Road"
   Brand = "Brand-Company A"
   Price = "100"
   Color = [ "Red", "Black" ]
   ProductCategory = "Bike"
}

An item can have any number of attributes, although there is a limit of 400 KB on the item size. An item size is the sum of lengths of its attribute names and values (binary and UTF-8 lengths); it helps if you keep the attribute names short.

• Supports up to 35 levels of nesting (JSON {foo,{bar,{baz,...)
• For any AWS account, there is an initial limit of 256 tables per region (one can, however, request an increase from Amazon)
• You can decrease the ReadCapacityUnits or WriteCapacityUnits settings for a table, but no more than four times per table in a single UTC calendar day. In a single operation, you can decrease the provisioned throughput for a table, for any global secondary indexes on that table, or for any combination of these.

• Pricing
  • Provisioned Throughput Capacity:
    • Write throughput $0.0065 per hour for every 10 units
    • Read throughput $0.0065 per hour for every 50 units
  • First 25GB stored per month is free
  • Storage costs of $0.25 GB per month thereafter

1,000,000 (writes) / 24 (hours) / 60 (minutes) / 60 (seconds) = 11.6 writes per second

($0.0065/10) * 12 WCUs * 24 hours = $0.1872 per day

So, ($0.0065/50) * 12 RCUs * 24 hours = $0.0374 per day

(30 * ($0.1872 + $0.0374)) + $0.75 = $7.488

With the Free Tier you get:

• 25 Read Capacity Units
• 25 Write Capacity Units

• DynamoDB Indexes and Streams
• Primary Keys (two types of primary keys available):
  • Single Attribute (think uniqueID)
    • Partition Key (Hash Key) composed of one attribute
  • Composite (think uniqueID and a date range)
    • Partition Key and Sort Key (Hash & Range) composed of two attributes

Partition Key 

DynamoDB uses the partition key's value as input to an internal hash function. The output from the has function determines the partition (this is simply the physical location in which the data is stored).

No two items in a table can have the same partition key value!

Composite Key (Partition Key and Sort Key) 

DynamoDB uses the partition key's value as input to an internal has function. The output from the has function determines the partition (this is simply the physical location in which the data is stored).

The two items can have the same partition key, but the must have a different sort key.

All items with the same partition key are stored together, in sorted order by the sort key value.

• Indexes
  • Local Secondary Index
    • an index that has the same hash key as the table, but a different range key. A local secondary index is "local" in the sense that every partition of a local secondary index is scoped to a table partition that has the same hash key.
    • has the same partition key and a different sort key.
    • can only be created at table creation. They cannot be removed or modified later.
    • maximum of 5 local secondary indexes per table
    • each table can have up to 20 projected non-key attributes, in total across all local secondary indexes within the table. Each index may also specify that all non-key attributes from the primary index are projected. (note: "projections" are the set of attributes that is copied into a local secondary index.)
  • Global Secondary Index
    • an index with a hash or a hash-and-range key that can be different from those on the table. A global secondary index is considered "global" because queries on the index can span all items in a table, across all partitions.
    • has a different partition key and a different sort key.
    • can be created at table creation or added later.
    • maximum of 5 global secondary indexes per table

• DynamoDB Streams
  • used to capture any kind of modification of the DynamoDB tables.
  • If a new item is added to the table, the stream captures an image of the entire item including all of its attributes.
  • If an item in updated, the stream captures the "before" and "after" image of any attributes that were modified in the item.
  • If an item was deleted from the table, the stream captures an image of the entire item before it was deleted.
  • DynamoDB Streams are stored for a maximum of 24 hours.
  • Can trigger a Lambda function (e.g., replicate table in another region and/or create an SES to send an email to the user for, say, when they first register to the website as a "welcome" email)

• One is able to export DynamoDB tables to a CSV file
• DyanmoDB allows for push-button scalability (with zero downtime)

DynamoDB Query 

find items in a table using only primary key attribute values. You must provide a partition attribute name and a distinct value to search for.

One can (optionally) provide a sort key attribute name and value, and use a comparison operator to refine the search results.

Be default, a Query returns all of the data attributes for items with the specified primary key(s), however, you can use the ProjectExpression parameter so that the query only returns some of the attributes, rather than all of them.

Query results are always sorted by the sort key. If the data type of the sort key is a number, the results are returned in numeric order; otherwise, the results are returned in order of ASCII character code values. By default, the sort order is ascending. To reverse the order, set the ScanIndexForward parameter to false.

By default, Queries are eventually consistent, but can be changed to be strongly consistent.

DynamoDB Scan 

A Scan operation examines every item in the table. By default, a Scan returns all of the data attributes for every item; however, you can use the ProjectExpression parameter so that the Scan only returns some of the attributes, rather than all of them.

What should one typically use: A Query or a Scan? Generally, a Query operation is more efficient than a Scan operation.

A Scan operation always scans the entire table, then filters out values to provide the desired result, essentially adding the extra step of removing data from the result set. Avoid using a Scan operation on a large table with a filter that removes many results, if possible. Also, as a table grows, the Scan operation slows. The Scan operation examines every item for the requested values, and can use up the provisioned throughput for a large table in a single operation.

For quicker response times, design your tables in a way that can use the Query, GetItem, or BatchGetItem APIs, instead. Alternatively, design your application to use Scan operations in a way that minimizes the impact on your table's request rate.

What happens if you exceed your throughput? 400 HTTP Status Code - ProvisionedThroughputExceededException You exceeded your maximum allowed provisioned throughput for a table or for one or more global secondary indexes.

DynamoDB Conditional Writes 

If item = $10, then update to $12 (conditional writes are idempotent)

E.g., two users try to update the same item at the same time

DynamoDB Atomic Counters 

DynamoDB supports Atomic Counters, where you use the UpdateItem operation to increment or decrement the value of an existing attribute (or "field" in a table) without interfering with other write requests. (All write requests are applied in the order in which the were received.) For example, a web application might with to maintain a counter per visitor to their site, the application would need to increment this counter regardless of its current value.

Atomic Counters are not idempotent. This mean that the counter will increment each time you call UpdateItem. If you suspect that a previous request was unsuccessful, your application could retry the UpdateItem operation; however, this would risk updating the counter twice. This might be acceptable for a web site counter, because you can tolerate slightly over- or under-counting visitors. However, in a banking application, it would be safer to use conditional updates rather than atomic counters.

If you application needs to read multiple items, you can use the BatchGetItem API. A single BatchGetItem request can retrieve up to 1 MB of data, which can contain as many as 100 items. In addition, a single BatchGetItem request can retrieve items from multiple tables.

DynamoDB API

Note: This is an incomplete list. The following are the main API calls one can expect to see on an exam.

CreateTable 

Creates a table and specifies the primary index used for data access.

UpdateTable 

Updates the provisioned throughput values for the given table.

DeleteTable 

Deletes a table.

DescribeTable 

Returns table size, status, and index information.

ListTables 

Returns a list of all tables associated with the current account and endpoint.

PutItem 

Creates a new item, or replaces an old item with a new item (including all the attributes). If an item already exists in the specified table with the same primary key, the new item completely replaces the existing item. You can also use conditional operators to replace an item only if its attribute values match certain conditions, or to insert a new item only if that item does not already exist.

BatchWriteItem 

Inserts, replaces, and deletes multiple items across multiple tables in a single request, but not as a single transaction. Supports batches of up to 25 items to Put or Delete, with a maximum total request size of 16 MB.

UpdateItem 

Edits an existing item's attributes. You can also use conditional operators to perform an update only if the item's attribute values match certain conditions.

DeleteItem 

Deletes a single item in a table by primary key. You can also use conditional operators to perform a delete an item only if the item's attribute values match certain conditions.

GetItem 

The GetItem operation returns a set of Attributes for an item that matches the primary key. The GetItem operation provides an eventually consistent read by default. If eventually consistent reads are not acceptable for your application, use ConsistentRead.

BatchGetItem 

The BatchGetItem operation returns the attributes for multiple items from multiple tables using their primary keys. A single response has a size limit of 16 MB and returns a maximum of 100 items. Supports both strong and eventual consistency.

Query 

Gets one or more items using the table primary key, or from a secondary index using the index key. You can narrow the scope of the query on a table by using comparison operators or expressions. You can also filter the query results using filters on non-key attributes. Supports both strong and eventual consistency. A single response has a size limit of 1 MB.

Scan 

Gets all items and attributes by performing a full scan across the table or a secondary index. You can limit the return set by specifying filters against one or more attributes.

A Scan operation on a table or secondary index has a limit of 1MB of data per operation. After the 1MB limit, it stops the operation and returns the matching values up to that point, and a LastEvaluatedKey to apply in a subsequent operation, so that you can pick up where you left off.

Using Web Identity Providers with DynamoDB

One can authenticate users using Web Identity providers (e.g., Facebook, Google, Amazon, or any other Open-ID Connect-compatible identity provider). This is done using AssumeRoleWithWebIdentity API.

You will need to create a role first.

1. Authenticate with Identity Provider (e.g., Facebook): Log into Facebook with your username + password
2. Facebook return a Web Identity Token
3. AssumeRoleWithWebIdentity request (containing Web Identity Token, App ID of provider, and ARN of role) is sent to AWS Security Token Service
4. Amazon then issues Temporary Security Credentials (limit from 15 minutes to 1 hour; default 1 hour)

   Credentials contain:

   1. AccessKeyID, SecretAccessKey, SessionToken
   2. Expiration (time limit)
   3. AssumeRoleID
   4. SubjectFromWebIdentity Token (the unique ID that appears in an IAM policy variable for this particular identity provider)
5. Using the above credentials, the user is allowed to access DynamoDB

Example questions

• DynamoDB provisioned throughput calculations
  • Unit of Read provisioned throughput
    • All reads are rounded up to increments of 4 KB in size.
    • Eventually consistent Reads (default) consist of 2 reads per second.
    • Strongly consistent Reads consist of 1 read per second.
  • Unit of Write provisioned throughput
    • All write are 1 KB in size.
    • All writes consist of 1 write per second.

• The Magic Formula

(size of Read rounded to nearest 4 KB chunk / 4 KB) x (number of items) = read throughput
# divide by 2 if eventually consistent.

Simple Queue Service (SQS)

SEE: Amazon SQS FAQs

Very first service offered by AWS

• SQS vs. RabbitMQ:
  • SQS is a managed service. So one does not have to worry about operational aspects of running a messaging system including administration, security, monitoring etc. Amazon will do this for you and will provide support if something were to go wrong.
  • SQS is Elastic and can scale to very large rate/volumes (unlimited according to AWS)
  • Availability of SQS has a lot of 9's in it and is backed by Amazon, which is one less thing to worry about in your application.

Amazon SQS is a web service that gives you access to a message queue that can be used to store messages while waiting for a computer to process them.

SQS is a distributed queue system that enables web service applications to quickly and reliably queue messages that one component in the application generates to be consumed by another component. A queue is a temporary repository for messages that a awaiting processing.

Using SQS, you can decouple the components of an application so they run independently, with SQS easing message management between components. Any component of a distributed application can store messages in a fail-safe queue. Messages can contain up to 256 KB of text in any format. Any component can later retrieve the messages programmatically using the SQS API.

The queue acts as a buffer between the component producing and saving data, and the component receiving the data for processing. This mean that queue resolves issues that arise if the producer is producing work faster than the consumer can process it, or if the producer or consumer are only intermittently connected to the network.

Amazon SQS ensures delivery of each message at least once, and support multiple readers and writers interacting with the same queue. A single queue can be used simultaneously by many distributed application components, with no need for those components to coordinate with each other to share the queue.

SQS is engineered to always be available and deliver messages. One of the resulting trade-offs is that SQS does not guarantee first in, first out delivery of messages. For many distributed applications, each message can stand on its own, and as long as all messages are delivered, the order is not important. If your system requires that order be preserved, you can place sequencing information in each message, so that you can reorder the messages when the queue returns them.

To illustrate, suppose you have a number of images files to encode. In a SQS worker queue, you create a SQS message for each file specifying the command (jpeg-encode) and the location of the file in S3. A pool of EC2 instances running the needed image processing software does the following:

1. Asynchronously pulls the task messages from the queue;
2. Retrievers the named file;
3. Processes the conversion (e.g., create a thumbnail, add a watermark, etc.)
4. Write the image back to Amazon S3;
5. Writes a "task complete" message to another queue;
6. Deletes the original task message; and then
7. Checks for more messages in the worker queue

Visibility Timeout Clock only starts when the component server (i.e., EC2 instance) pulls the message from the queue

• SQS with Auto-scaling
• SQS does not offer FIFO (first in, first out)
• 12 hour visibility time out
• SQS is engineered to provide "at least once" delivery of all messages in its queues. Although most of the time each message will be delivered to your application exactly once, you should design your system so that processing a message more than once does not create any errors or inconsistencies.
• 256kb message size (as of May 2016)
• Billed at 64kb "chunks"
• A 256kb message will be 4 x 64kb "chunks"

• SQS Pricing
  • First 1 million SQS requests per month are free
  • $0.50 per 1 million SQS requests per month thereafter ($0.0000005 per SQS request)
  • A single request can have from 1 to 10 messages, up to a maximum total payload of 256kb
  • Each 64kb "chunk" of payload is billed as 1 request. For example, a single API call with a 256kb payload will be billed as four requests.

• If you see "decouple", think SQS

• SQS Delivery:
  • SQS messages can be delivered multiple times and in any order (no first in, first out; last in, last out)

• SQS - Default visibility timeout:
  • Default visibility timeout is 30 seconds
  • Maximum timeout is 12 hours
  • When you receive a message from a queue and begin processing it, you may find the visibility timeout for the queue is insufficient to fully process and delete that message. To give yourself more time to process the message, you can extend its visibility timeout by using the ChangeMessageVisibility action to specify a new timeout value. SQS restarts the timeout period using the new value.

• SQS Long Polling:
  • SQS long polling is a way to retrieve messages from your SQS queues. While the traditional SQ short polling returns immediately, even if the queue being polled is empty, SQS long polling does not return a response until a message arrives in the queue, or the long poll times out. SQS long polling makes it easy and inexpensive to retrieve messages from your SQS queue as soon as they are available.
  • Maximum Long Poll timeout is 20 seconds
  • Example exam question: Polling in tight loops is burning CPU cycles and costing the company money. How would you fix this? (Answer: Enable SQS Long Polling.)

• SQS - Fanning Out:
  • Create an SNS topic first using SNS. Then create a and subscribe multiple SQS queues to the SNS topic. Now whenever a message is sent to the SNS topic, the message will be fanned out to the SQS queues (i.e., SNS will deliver the message to all the SQS queues that are subscribed to the topic).

Simple Notification Service (SNS)

SEE: Amazon SNS FAQs

SNS is a web service that makes it easy to set up, operate, and send notifications from the cloud. It provides developers with a highly scalable, flexible, and cost-effective capability to publish messages from an application and immediately deliver them to subscribers or other applications.

SQS follows the "publish-subscribe" (pub-sub) messaging paradigm, with notifications being delivered to clients, using a "push" mechanism that eliminates the need to periodically check or "poll" for new information and updates. With simple APIs requiring minimal up-front development effort, no maintenance or management overhead and pay-as-you-go pricing, SNS gives developers an easy mechanism to incorporate a powerful notification system with their applications.

Push notifications to Apple, Google, Fire OS, and Windows devices, as well as Android devices in China with Baidu Cloud Push.

Besides pushing cloud notifications directly to mobile devices, SNS can also deliver notifications by SMS text message or email, to Amazon Simple Queue Service (SQS) queues or to any HTTP endpoint.

To prevent messages from being lost, all messages published to SNS are stored redundantly across multiple availability zones.

• SNS - Topics:
  • SNS allows you to group multiple recipients using "topics". A topic is an "access point" for allowing recipients to dynamically subscribe to identical copies of the same notification. One topic can support deliveries to to multiple endpoint types. For example, one can group together iOS, Android, and SMS recipients. When you publish once to a topic, SNS delivers appropriately formatted copies of your message to each subscriber.
  • Subscriptions via email require the receiving email owner to confirm the subscriptions in order to receive notifications from the given topic (prevents spam). Subscription requests expire after 3 days, if the receiving email owner does not confirm the subscription.

Example SNS email notification:

{
  "Type" : "Notification",
  "MessageId" : "436d9234-f427-5be8-aa54-dd98ae4e286dba0",
  "TopicArn" : "arn:aws:sns:us-west-2:01234:MyTestSNSTopic",
  "Subject" : "This is a test",
  "Message" : "Hello, world!",
  "Timestamp" : "2016-05-10T21:53:37.981Z",
  "SignatureVersion" : "1",
  "Signature" : "PInj15UDcwMI==",
  "SigningCertURL" : "https://sns.us-west-2.amazonaws.com/SimpleNotificationService-9390147a5624348ee.pem",
  "UnsubscribeURL" : "https://sns.us-west-2.amazonaws.com/?Action=Unsubscribe&SubscriptionArn=arn:aws:sns:us-west-2:01234:MyTestSNSTopic:1a4eead5-110c-4acb-894a-91cdf358aabc",
  "MessageAttributes" : {
    "AWS.SNS.MOBILE.MPNS.Type" : {"Type":"String","Value":"token"},
    "AWS.SNS.MOBILE.MPNS.NotificationClass" : {"Type":"String","Value":"realtime"},
    "AWS.SNS.MOBILE.WNS.Type" : {"Type":"String","Value":"wns/badge"}
  }
}

• SNS benefits:
  • Instantaneous, push-based delivery (no polling)
  • Simple APIs and easy integration with applications
  • Flexible message delivery over multiple transport protocols
  • Inexpensive, pay-as-you-go model with no up-front costs
  • Web-based AWS Management Console offers the simplicity of a point-and-click interface

• SNS vs. SQS:
  • Both messaging services in AWS
  • SNS => Push
  • SQS => Pulls (Polls)

• SNS Pricing:
  • Users pay $0.50 per 1 million SNS requests
  • $0.06 per 100,000 notification deliveries over HTTP
  • $0.75 per 100 notification deliveries over SMS
  • $2.00 per 100,000 notification deliveries over email

SNS data format = JSON

• SNS protocols include:
  • HTTP
  • HTTPS
  • Email
  • Email-JSON
  • Amazon SQS
  • Application

NOTE: Messages can be customized for each protocol

Simple Workflow Service (SWF)

SEE: Amazon SWF FAQs

Amazon Simple Workflow Server (Amazon SWF) is a web service that makes it easy to coordinate work across distributed application components. SWF enables applications for a range of use cases, including media processing, web application back-ends, business process work-flows, and analytics pipelines, to be designed as a coordination of tasks. Task represent invocations of various processing steps in an application, which can be performed by executable code, web service calls, human actions, and scripts.

SWF Workers 

programs that interact with SWF to get tasks, process received tasks, and return the results.

SWF Decider 

a program that controls the coordination of tasks, i.e., their ordering, concurrency, and scheduling according to the application logic.

The workers and the decider can run on cloud infrastructure, such as Amazon EC2, or on machines behind firewalls. SWF brokers the interactions between workers and the decider. It allows the decider to get consistent views into the progress of tasks and to initiate new tasks in an ongoing manner. At the same time, SWF stores tasks, assigns them to workers when they are ready, and monitors their progress. It ensures that a task is assigned only once and is never duplicated. Since SWF maintains the application's state durably, workers and deciders do not have to keep track of execution state. They can run independently and scale quickly.

SWF Domains 

Your work-flow, activity types, and the workflow execution itself are all scoped to a domain. Domains isolate a set of types, executions, and task lists from others within the same account. One can register a domain by using the AWS Management Console or by using the RegisterDomain action in the SWF API.

The parameters of an SWF Domain are specified in JSON format. E.g.:

https://swf.us-west-2.amazonaws.com
RegisterDomain
{
  "name": "123456789",
  "description": "images",
  "workflowExecutionRetentionPeriodInDays": "60"
}

Maximum Workflow can be 1 year and the value is always measured in seconds.

• SWF vs. SQS:
  • SWF presents a task-oriented API, whereas SQS offers a message-oriented API.
  • SWF ensures that a task is assigned only once and is never duplicated. What SQS, one needs to handle duplicate messages and may also need to ensure that a message is processed only once.
  • SWF keeps track of all the tasks and events in an application. With SQS, one needs to implement one's own application-level tracking, especially if one's application uses multiple queues.
  • Does the service require human interaction? If so, one should use SWF.
  • Does the service need to run for (much) more than 12 hours? If so, one should use SWF. If less than 12 hours, SQS might be the correct service to use.
  • Maintaining your application's execution state (e.g. which steps have completed, which ones are running, etc.) is a perfect use case for SWF.
  • Amazon SWF is useful for automating work-flows that include long-running human tasks (e.g. approvals, reviews, investigations, etc.) Amazon SWF reliably tracks the status of processing steps that run up to several days or months.

CloudFormation

SEE: Amazon CloudFormation FAQs

CloudFormation => Scripted infrastructure (IaaS)

Using the CloudFormation service is free. However, any resources it creates/consumes/provisions (e.g., EC2 instances, Load Balancers, etc.) are not free.

CloudFormation templates are written in JSON format. E.g., CloudFormation LAMP stack template (see here for full template):

{
  "AWSTemplateFormatVersion" : "2010-09-09",
  
  "Description" : "AWS CloudFormation Sample Template LAMP_Single_Instance:
Create a LAMP stack using a single EC2 instance and a local MySQL database for
storage. This template demonstrates using the AWS CloudFormation bootstrap
scripts to install the packages and files necessary to deploy the Apache web
server, PHP and MySQL at instance launch time. **WARNING** This template
creates an Amazon EC2 instance. You will be billed for the AWS resources used
if you create a stack from this template.",
  
  "Parameters" : {
      
    "KeyName": {
      "Description" : "Name of an existing EC2 KeyPair to enable SSH access to the instance",
      "Type": "AWS::EC2::KeyPair::KeyName",
      "ConstraintDescription" : "must be the name of an existing EC2 KeyPair."
    }, 
...

• CloudFormation "output":
  • SEE: http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/outputs-section-structure.html
  • Remember the "Fn::GetAtt" function

If a CloudFormation stack creation fails, the default is to terminate and roll-back all resources created on failure (i.e., delete all of the resources it was trying to create). One can disable roll back to leave all resources in their current state (failed or not). This is useful for troubleshooting your own templates.

Elastic Beanstalk

AWS Elastic Beanstalk is an easy-to-use service for deploying and scaling web applications and services developed with Java, .NET, PHP, Node.js, Python, Ruby, Go, and Docker on familiar servers such as Apache, Nginx, Passenger, and IIS.

You can simply upload your code and Elastic Beanstalk automatically handles the deployment, from capacity provisioning, load balancing, auto-scaling, to application health monitoring. At the same time, you retain full control over the AWS resources powering your application and can access the underlying resources at any time.

Using the AWS Elastic Beanstalk service is free. However, any AWS resources it creates/consumes/provisions to store and run your applications are not free.

• Environment tier:
  • Web Server Environment - Provides resources for an AWS Elastic Beanstalk web server in either a single instance or load-balancing, auto scaling environment.
  • Worker Environment - Provides resources for an AWS Elastic Beanstalk worker application in either a single instance or load-balancing, auto scaling environment.
• Environment type:
  • Single instance
  • Load balancing, auto-scaling
• Preconfigured platforms:
  • PHP, Node.js, Python, Ruby, Tomcat, IIS, Java, Go, Docker

Virtual Private Cloud (VPC)

SEE: Amazon VPC FAQs

Think of a VPC as a virtual data centre in the cloud.

• AWS definition of a VPC:
  • Amazon Virtual Private Cloud (VPC) lets you provision a logically isolated section of the AWS Cloud where you can launch AWS resources in a virtual network that you define. You have complete control over your virtual networking environment, including select of you own IP address range, creation of subnets, and configuration of route tables and network gateways.
  • You can easily customize the network configuration for your AWS virtual Private Cloud. For example, you can create a public-facing subnet for your webservers that has access to the Internet, and place backend systems, such as databases or application service in a private-facing subnet, with no Internet access. You can leverage multiple layers of security, including security groups and network access control lists, to help control access to EC2 instance in each subnet.
  • Additionally, one can create a Hardware Virtual Private Network (VPN) connection between your corporate datacentre and your VPC and leverage the AWS cloud as an extension of your corporate datacentre. "Hybrid Cloud".

• What can one do with a VPC?
  • Launch instances into a subnet of one's choosing
  • Assign custom IP address ranges in each subnet
  • Configure route tables between subnets
  • Create internet gateways and attach them to subnets (or not)
  • Much better security control over your AWS resources
  • Instance security groups
  • Create subnet network access control lists (ACLs)
  • Number of allowed VPCs in each AWS Region (by default): 5

• Default VPC vs. Custom VPC
  • Default VPC is user friendly (automatically created when one creates an AWS account). It allows one to immediately deploy instances.
  • All subnets in a default VPC have an internet gateway attached (i.e., all subnets are public)
  • Each EC2 instance has both a public and private IP address
  • If one were to delete the default VPC, the only way to get it back is to contact AWS

• VPC Peering
  • Allows one to connect one VPC with another via a direct network route using private IP addresses.
  • Instances behave as if they were on the same private network.
  • One can peer VPCs with other AWS accounts as well as with other VPCs in the same account.
  • One cannot create a VPC larger than /16
  • Peering is done in a "star configuration", i.e., 1 central VPC peers with 4 others. No transitive peering!

• A "star configuration" peering:

                 +-------+
                 | VPC C |
                 +-------+
                     ^
                     |
                     v
+-------+        +-------+        +-------+
| VPC B | <----> | VPC A | <----> | VPC E |
+-------+        +-------+        +-------+
                     ^
                     |
                     v
                 +-------+
                 | VPC D |
                 +-------+

In the above example, instances on VPC-B can not send/receive traffic on VPC-C via VPC-A. One would need to create a VPC peer directly from VPC-B and VPC-C. That is, with a star configuration (as shown above), there is no transitive peering.

• VPC tenancy:
  • Default - EC2 instances are created on shared hardware
  • Dedicated - EC2 instances are created on dedicated hardware (regardless of the tenancy attribute specified at launch). This the more expensive option.

By default, when one creates a VPC, a route tables is automatically created for the VPC.

NOTE: If one deletes one's account's default VPC (and/or the associated default subnets), the only way to get them back is to raise a ticket with Amazon.

• VPC subnets:
  • Use the CIDR format to specify your subnet's IP address block (e.g., 10.0.0.0/24). Note that block sizes must be between a /16 netmask and /28 netmask. Also, note that a subnet can be the same size as your VPC.
  • Subnets are always mapped to one availability zone (AZ). Subnets can not be mapped across multiple AZs. 1 subnet = 1 AZ.

• VPC Internet gateways:
  • By default, when one creates an Internet Gateway, it is detached. One must attach it to a VPC in order to use it.
  • One can only have 1 Internet Gateway per VPC.

Security Groups can span multiple VPCs and VPC subnets.

• Network Address Translation (NAT) Server:
  • Allow instances with only private IPs to reach the Internet via the NAT server (via, say, HTTP/HTTPS and all other protocols/ports closed, including SSH)
  • One must disable Source/Destination Check on NAT instances for them to work properly.

Network Access Control Lists (ACLs)

• A network ACL is an optional layer of security that acts as a firewall for controlling traffic in and out of a subnet.
• ACLs ~ "firewall"-like rules
• ACLs are a numbered list of rules, which are followed in order, starting with the lowest number first. They control network ingress/egress for all AWS resources within a given subnet.
• The highest ACL number allowed is 32,766
• ACLs have a default (editable) number list that allows all inbound/outbound traffic
• One can create a custom ACL, which starts out with no inbound/outbound traffic allowed, until one adds a rule
• ACLs are applied to an entire subnet (and override the security groups associated with a given instance on that subnet). For an example, if a security group applied to a given instance has port 80 allowed, but the ACL for the subnet the instance is on has port 80 denied, the ACL rule takes precedence (i.e., port 80 will be denied on all instances within that subnet, regardless of what the security group allows).
• Unless one creates a custom ACL and associates it with a given subnet, that ACL will use the default role and rules.
• One can not have multiple ACLs associated with the same subnet. However, a given ACL can be associated with multiple subnets.
• If one dis-associates a custom ACL from a given subnet(s), the subnet reverts back to the default ACL

Example labs

• VPC Lab:
  • Create a custom VPC
    • Define an IP Address Range (e.g., 10.0.0.0/16)
    • By default, this created a Network ACL and Routing Table
  • Create a custom Route Table
  • Create 3 subnets (e.g., 10.0.1.0/24, 10.0.2.0/24, and 10.0.3.0/24)
    • public subnet: 10.0.1.0/24; private subnets: 10.0.2.0/24 and 10.0.3.0/24
  • Create an Internet Gateway
  • Attach Internet Gateway to the custom VPC
  • Associate the public subnet with the custom Route Table
  • Provision an EC2 instance with an Elastic IP address (in the public subnet)
  • Provision an EC2 instance with only a private IP address (on the private subnet)

• NAT Lab:
  • Create a custom security group
  • Allow inbound traffic to 10.0.1.0/24 and 10.0.2.0/24 on HTTP and HTTPS
  • Allow outbound traffic on HTTP and HTTPS to anywhere
  • Provision a NAT instance inside the public subnet
  • Import! Make sure to select "Disabled Source/Destination Check" on this NAT instance!
  • Set up a route on the private subnet to route traffic through the NAT instance

AWS Shared Responsibility

Shared Responsibility Model for AWS Infrastructure Services

• Infrastructure services include: EC2, EBS, Auto-scaling, VPC, etc
• Amazon responsibility (i.e., managed by AWS):
  • AWS global infrastructure (regions, availability zones, edge locations)
  • Foundation services (compute, storage, databases, networking)
  • AWS endpoints
  • AWS IAM
• Customer responsibility (i.e., managed by AWD customers):
  • Server-side encryption, network traffic protection
  • Operating system, network, and firewall configuration(s)
  • Platform and application management
  • Customer data
  • Customer IAM

Shared Responsibility Model for AWS Container Services

• Container services include: Relational Database Services (RDS), Elastic Map Reduce (EMR), and Elastic Beanstalk
• AWS takes on more of the responsibility (e.g., operating system and network configuration; platform and application management)
• Customer still has the responsibility for firewall configuration

Shared Responsibility Model for AWS abstracted services

• Abstracted services include: S3, Glacier, DynamoDB, SQS, and Simple Email Service (SES), Lambda
• AWS takes on each more of the responsibility (e.g., network traffic protection provided by the platform; server-side encryption provided by the platform)
• Customer still has the responsibility for client-side data encryption, data integrity authentication, and customer data

Exams overview

AWS Certified Developer - Associate

• Time allotted: 80 minutes
• 55 questions on the exam
• $150 example registration fee
• Conducted Online at an approved centre

AWS Certifications

NOTE: All AWS certification exams are taken on-site and proctored

• Associate Level (each $150):
  • AWS Certified Developer - Associate
  • AWS Certified SysOps Administrator - Associate
  • AWS Certified Solutions Architect - Associate
• Professional Level (each $300):
  • AWS Certified DevOps Engineer - Professional
  • AWS Certified Solutions Architect - Professional

Glossary

see: Official AWS Glossary

AMI

Amazon Machine Image

ARN

Amazon Resource Name

EBS

Elastic Block Storage

EC2

Elastic Compute Cloud

EFS

Elastic File System

ELB

Elastic Load Balancer

IAM 

AWS Identity and Access Management

S3 

Simple Storage Service

Links

AWS Whitepapers

• AWS Whitepapers
  • AWS Security Best Practices (PDF)
  • AWS Security Whitepaper (PDF)
  • AWS Storage Services Whitepaper (PDF)
  • AWS Well-Architected Framework (PDF)

Training and certification

• https://acloud.guru/courses
• https://aws.amazon.com/certification/certified-developer-associate/
• What you need to get AWS certified — by A Cloud Guru
• Register for AWS certification exams

Miscellaneous

• http://ec2price.com/
  • https://github.com/grosskur/ec2price