Category:AWS

This category/article is (currently) just a random collection of my notes on Amazon Web Services (AWS). I will organize each service type into a separate article, when I find the time.

Elastic Block Storage (EBS)

• EBS Volume types:
  • General purpose SSD (GP2): up to 10,000 IOPS
  • Provisioned IOPS SSD (IO1): more than 10,000 IOPS
  • Magnetic (Standard): infrequently accessed storage

NOTE: One can not mount 1 EBS volume to multiple EC2 instances; use EFS instead.

In order to enable encryption at rest using EC2 and Elastic Block Store, one needs to configure encryption when creating the EBS volume.

Elastic Compute Cloud (EC2)

• EC2 provision types:
  • On-Demand
  • Reserved
    • Amazon EC2 Reserved Instances allow you to reserve Amazon EC2 computing capacity for 1 or 3 years, in exchange for a significant discount (up to 75%) compared to On-Demand instance pricing.
  • Spot
    • Amazon EC2 Spot instances allow you to bid on spare Amazon EC2 computing capacity. Since Spot instances are often available at a discount compared to On-Demand pricing, you can significantly reduce the cost of running your applications, grow your application's compute capacity and throughput for the same budget, and enable new types of cloud computing applications.
    • Example use cases: genomics, drug companies, etc.
    • If you terminate, you pay for the hour the instance was terminated
    • If Amazon terminates it, you do not pay for the hour the instance was terminated
    • EC2 Spot prices

• EC2 Instance Types:
  • D for Density
  • I for IOPS
  • R for RAM
  • T general purpose (think T2 Micro)
  • M for Main choice (for general purpose apps)
  • C for Compute
  • G for Graphics
  • mnemonic: DIRTMCG

• EC2 metadata:

$ curl http://169.254.169.254/latest/meta-data/public-ipv4 # returns the EC2 instance's public IPv4

NOTE: If one makes an Amazon Machine Image (AMI) public, this AMI is not immediately available across all regions, by default.

With EC2 you can have 2 types of storage, EBS storage or Instance Store. EBS is persistent and if an EC2 instance is stopped with an EBS volume attached, there will be no data lost. Instance Store is ephemeral and if the EC2 instance is stopped, all data will be lost.

Elastic Load Balancer (ELB)

• ELBs are not free, you are charged by the hour and on a per GB basis of usage

• ELB supported ports:
  • ec2-vpc: 1-65535
  • ec2-classic: 25, 80, 443, 465, 587, 1024-65535

• ELB supported protocols:
  • HTTP, HTTPS, TCP, SSL

NOTE: One can have multiple SSL certificates (for multiple domain names) on a single Elastic Load Balancer.

SDKs

• HTTP codes:
  • 200 - The request has succeeded
  • 3xx - Redirection
  • 4xx - Client error (think 404 not found)
  • 5xx - Server error (think Apache service not running, etc.)

• Available SDKs:
  • Android, iOS, JavaScript (browser)
  • Java
  • .Net
  • Node.js
  • PHP
  • Python
  • Ruby
  • Go
  • C++ (preview)

• SDK default regions:
  • default region (for most SDKs): us-east-1
  • Some SDKs have default regions set (Java)
  • Some SDKs do not (Node.js)

Services that are free include: CloudFormation, Elastic Beanstalk, Autoscaling, Opsworks, etc. (however, the resources they create are not free; e.g., EC2 instances, ELBs)

Simple Storage Service (S3)

Amazon Simple Storage Service (Amazon S3), provides developers and IT teams with secure, durable, highly-scalable cloud storage. Amazon S3 is easy to use object storage, with a simple web service interface to store and retrieve any amount of data from anywhere on the web. With Amazon S3, you pay only for the storage you actually use. There is no minimum fee and no setup cost.

• simple key-value store:
  • key = name of the object;
  • value = the actual data (made up of a sequence of bytes);
  • version ID (important for versioning); and
  • metadata (data about the data you are storing)
  • Sub-resources
  • Access Control Lists (ACLs)
• S3 bucket URL: https://s3-<region>.amazonaws.com/<bucket_name> (e.g., https://s3-us-west-1.amazonaws.com/foobar)
• SLA:
  • availability: 99.99% (2 nines)
  • durability: 99.99999999999% (11 nines)
• Files can be from 1 byte to 5 TB in size (split files larger than 5 TB into pieces to upload)
  • Note that the largest size file you can transfer to S3 using a PUT operation is 5 GB
• Unlimited storage
• files/objects are stored in "buckets"
• S3 is a universal namespace (i.e., bucket names must be unique globally; think domain names)
• Read-after-Write consistency for PUTs of new Objects
• Eventual Consistency for overwrite PUTs and DELETEs (can take some time to propagate)
• Lifecyle management
• Versioning
• Encryption (default Advanced Encryption Standard (AES) 256bit)
  • In transit (SSL/TLS)
  • At Rest:
    • Server Side Encryption (SSE):
      • S3 Managed Keys (SSE-S3; 256bit);
      • AWS Key Management Service, Managed Keys (SSE-KMS)
      • Server Siide Encryption with Customer Provided Keys (SSE-C)
  • Client Side Encryption (user encypts data on their local machine and then upload to AWS S3)
• Secure your data with Bucket Policies and ACLs
• Storage tiers/classes:
  • S3 99.99% (durable, immediately available, frequently accessed): stored across multiple devices in multiple facilities and is designed to sustain the loss of 2 facilities concurrently
  • S3 IA (Infrequently Accessed) (durable, immediately available, infrequently accessed): for data that is accessed less frequently, but requires rapid access when needed. Lower fee than S3, but you are charged a retrieval fee
  • S3 Reduced Redundancy Storage (RRS): designed to provide 99.99% availability/durability of objects over a given year (for objects where it is not critical if they are lost; e.g., thumbnails of images, as they can be easily regenerated). concurrent facility fault tollerance = 1
  • Glacier: Very cheap, but used for archival only. It takes 3-5 hours to restore from Glacier.

• Storage Gateways:
  • Gateway Stored Volumes (entire dataset is stored on site and is asynchronously backed up to S3)
  • Gateway Cached Volumes (entire dataset is stored on S3 and the most frequently accessed data is cached on site)
  • Gateway Virtual Tape Library (VTL) (used for backup and uses popular backup applications like NetBackup, Backup Exec, Veam, etc.)

• Import/Export Disk:
  • Import to EBS
  • Import to S3
  • Import to Glacier
  • Export from S3
• Import/Export Snowball (only available in North America)
  • Import to S3
  • Export from S3

• S3 bucket names must contain only lower case characters
• S3 stores data in alphabetical order (lexigraphical order)
• Largest file size one can transfer to S3 using a PUT operation: 5 GB (Use multi-part upload for files larger than 5GB)

• S3 Static Website:
  • http://foobar.s3-website-us-west-2.amazonaws.com/index.html (a static website link)
  • https://s3-us-west-2.amazonaws.com/foobar/demo.jpeg (not a static website link)
  • Static websites are always HTTP (not HTTPS, for now)

• S3 Cross Origin Resource Sharing (CORS)
  • One needs to enable CORS for one S3 bucket to reference objects in another S3 bucket

• S3 Versioning
  • Stores all versions of an object (including all writes, including deleting the object)
  • Great backup tool
  • Once enabled, versioning cannot be disabled, only suspended
  • Integrates with Lifecycle rules
  • Versioning's MFA (Multi-Factor Authentication) delete capability, which uses MFA, can be used to provide an additional layer of security.
  • Cross Region Replication, requires versioning enabled on the source bucket.

• S3 Lifecyle Management
  • Transition objects to Infrequent Access Storage Class (must wait a minimum of 30 days from initial upload for the object to transition to the new storage class; minimum object size 128KB) or Glacier Storage Class after x amount of days.
  • Infrequent retrieval: ~milliseconds
  • Glacier retrieval: 3 - 5 hours
  • With Versioning: Transition object versions as well (including deleting old/current versions)
  • Can not use Reduced Durability Storage Class with Lifecycle Management

• CloudFront (a content delivery network {CDN})
  • AWS Global CDN infrastructure
  • Edge Location: this is the location where content will be cached. This is different from an AWS Region/AZ (over 50 Edge Locations, as of April 2016)
  • Origin: This is the origin of all the files/objects that the CDN will distribute. This can be an S3 Bucket, an EC2 Instance, an Elastic Load Balancer, or Route53.
    • Possible to have multiple origin paths in the same distribution
  • Distribution: This is the name given to the CDN, which consists of a collection of Edge Locations.
    • Web Distribution: Typically used for websites.
    • RTMP: Used for media streaming (e.g., Adobe Flash)
  • Edge location are not just READ only, one can write to them as well (i.e., PUT and object to them)
  • Objects are cached for the life of the TTL (Time To Live)
  • One can clear the cache of an object stored on an Edge Location before the TTL expires, but one will be charged for that service.

• S3 Transfer Acceleration:
  • Instead of uploading object directly to an S3 bucket in a given region, one can upload a object directly to the nearest Edge Location and AWS will then transfer the object to the S3 bucket.
  • Example URL: <bucket_name>.s3-accelerate.amazonaws.com

Databases

1. SEE: RDS FAQ on AWS

RDS: Relational Database Server

• RDS (OLTP) Relational Database Types:
  • MS SQL Server
  • Oracle
  • MySQL Server
  • PostgreSQL
  • Aurora
  • MariaDB

• Non-relational Databases:
  • DynamoDB (document-oriented
    • Collection (=Table)
    • Document (=Row)
    • Key-value pairs (=Fields)

E.g., JSON/NoSQL document

{
  "_id": "345sdf45asdf",
  "firstname": "John",
  "surname": "Smith",
  "age": "23",
  "address": [
    {
      "street": "123 First Street",
      "suburb": "Wallingford"
    }
  ]
}

• Redshift (OLAP)

• Elasticache
  • Memcached
  • Redis

DMS: (Online) Database Migration Service (convert Oracle to MySQL, etc.)

• Data Warehousing (business intelligence; e.g., Cognos, Jaspersoft, etc.)
• Online Transaction Processing (OLTP); vs.
• Online Analytics Processing (OLAP)

DynamoDB

IMPORTANT! Read FAQ on AWS DynamoDB

Amazon DynamoDB is a fast and flexible NoSQL database service for all applications that need consistent single-digit millisecond latency at any scale. It is a fully managed database and supports both document and key-value data models. Its flexible data model and reliable performance make it a great fit for mobile, web, gaming, ad-tech, IoT, and many other applications.

• Stored on SSD storage
• Spread across 3 geographically distinct data centres (note: not distinct AZs)
• Eventual Consistent READs (default)
  • Consistency across all copies of data is usually reached within a second. Repeating a read after a short time should return the update data. (best read performance.)
• Strongly Consistent READs
  • A strongly consistent read returns a result that reflects all writes that received a successful response prior to the read.

• The DynamoDB basics:
  • Tables
  • Items (think a row of data in the table)
  • Attributes (think of a column of data in a table)

Supports up to 35 levels of nesting (JSON {foo,{bar,{baz,...)

• Pricing
  • Provisioned Throughput Capacity:
    • Write throughput $0.0065 per hour for every 10 units
    • Read throughput $0.0065 per hour for every 50 units
  • First 25GB stored per month is free
  • Storage costs of $0.25 GB per month thereafter

Pricing example:

Let's assume that your application needs to perform 1 million writes and 1 million reads per day, while storing 28 GB of data. First, you need to calculate how many writes and reads per second you need. 1 million evenly spread writes per data is equivalents to:

1,000,000 (writes) / 24 (hours) / 60 (minutes) / 60 (seconds) = 11.6 writes per second

A DynamoDB Write Capacity Unit (WCU) can handle 1 write per second, so you need 12 WCUs (round up 11.6 to 12). For write throughput, you are charged on $0.0065 per hour for every 10 units.

So, ($0.0065/10) * 12 WCUs * 24 hours = $0.1872 per day

Similarly, to handle 1 million strongly consistent reads per day, you need 12 Read Capacity Units (RCUs). For read throughput, one is charged $0.0065 per hour for every 50 units.

So, ($0.0065/50) * 12 RCUs * 24 hours = $0.0374 per day

Storage cost is $0.25 per GB per month. Let's assume our database is 28 GB. We get the first 25 GB for free, so we only pay for 3 GB of storage, which is $0.75 per month.

Total Cost = $0.1872 per day + $0.0374 per day + storage of $0.75 per month

(30 * ($0.1872 _ $0.0374)) + $0.75 = $7.488 /END

With the Free Tier you get:

• 25 Read Capacity Units
• 25 Write Capacity Units

• DynamoDB Indexes and Streams
• Primary Keys (two types of primary keys available):
  • Single Attrbute (think uniqueID)
    • Partition Key (Hash Key) composed of one attribute
  • Composite (think uniqueID and a date range)
    • Partition Key and Sort Key (Hash & Range) composed of two attributes

Partition Key 

DynamoDB uses the partition key's value as input to an internal hash function. The output from the has function determines the partition (this is simply the physical location in which the data is stored).

No two items in a table can have the same partition key value!

Composite Key (Partition Key and Sort Key) 

DynamoDB uses the partition key's value as input to an internal has function. The output from the has function determines the partition (this is simply the physical location in which the data is stored).

The two items can have the same partition key, but the must have a different sort key.

All items with the same partition key are stored together, in sorted order by the sort key value.

• Indexes
  • Local Secondary Index
    • Has the _same_ partition key, different sort key.
    • Can _only" be created when create a table. They cannot be removed or modified later.
  • Global Secondary Index
    • Has _different_ partition key and different sort key.
    • Can be created at table creation or added later.

• DynamoDB Streams
  • used to capture any kind of modification of the DynamoDB tables.
  • If a new item is added to the table, the stream captures an image of the entire item including all of its attributes.
  • If an item in updated, the stream captures the "before" and "after" image of any attributes that were modified in the item.
  • If an item was deleted from the table, the stream captures an image of the entire item before it was deleted.
  • DynamoDB Streams are stored for a maximum of 24 hours.
  • Can trigger a Lambda function (e.g., replicate table in another region and/or create an SES to send an email to the user for, say, when they first register to the website as a "welcome" email)

One is able to export DynamoDB tables to a CSV file DyanmoDB allows for push-button scalability (with zero downtown)

DynamoDB Query 

find items in a table using only primary key attribute values. You must provide a partition attribute name and a distinct value to search for.

One can (optionally) provide a sort key attribute name and value, and use a comparison operator to refine the search results.

Be default, a Query returns all of the data attributes for items with the specified primary key(s), however, you can use the ProjectExpression parameter so that the query only returns some of the attributes, rather than all of them.

Query results are always sorted by the sort key. If the data type of the sort key is a number, the results are returned in numeric order; otherwise, the results are returned in order of ASCII character code values. By default, the sort order is ascending. To reverse the order, set the ScanIndexForward parameter to false.

By default, Queries are eventually consistent, but can be changed to be strongly consistent.

DynamoDB Scan 

A Scan operation examines every item in the table. By default, a Scan returns all of the data attributes for every item; however, you can use the ProjectExpression parameter so that the Scan only returns some of the attributes, rather than all of them.

What should one typically use: A Query or a Scan? Generally, a Query operation is more efficient than a Scan operation.

A Scan operation always scans the entire table, then filters out values to provide the desired result, essentially adding the extra step of removing data from the result set. Avoid using a Scan operation on a large table with a filter that removes many results, if possible. Also, as a table grows, the Scan operation slows. The Scan operation examines every item for the requested values, and can use up the provisioned throughput for a large table in a single operation.

For quicker response times, design your tables in a way that can use the Query, Get, or BatchGetItem APIs, instead. Alternatively, design your application to use Scan operations in a way that minimizes the impact on your table's request rate.

• DynamoDB provisioned throughput calculations
  • Unit of Read provisioned throughput
    • All reads are rounded up to increments of 4 KB in size.
    • Eventually consistent Reads (default) consist of 2 reads per second.
    • Strongly consistent Reads consist of 1 read per second.
  • Unit of Write provisioned throughput
    • All write are 1 KB in size.
    • All writes consist of 1 write per second.

Example questions

• The Magic Formula

(size of Read rounded to nearest 4 KB chunk / 4 KB) x (number of items) = read throughput
# divide by 2 if eventually consistent.

What happens if you exceed your throughput? 400 HTTP Status Code - ProvisionedThroughputExceededException You exceeded your maximum allowed provisioned throughput for a table or for one or more global secondary indexes.

Using Web Identity Providers with DynamoDB

One can authenticate users using Web Identity providers (e.g., Facebook, Google, Amazon, or any other Open-ID Connect-compatible identity provider). This is done using AssumeRoleWithWebIdentity API.

You will need to create a role first.

1. Authenticate with Identity Provider (e.g., Facebook): Log into Facebook with your username + password
2. Facebook return a Web Identity Token
3. AssumeRoleWithWebIdentity request (containing Web Identity Token, App ID of provider, and ARN of role) is sent to AWS Security Token Service
4. Amazon then issues Temporary Security Credentials (limit from 15 minutes to 1 hour; default 1 hour)

   Credentials contain:

   1. AccessKeyID, SecretAccessKey, SessionToken
   2. Expiration (time limit)
   3. AssumeRoleID
   4. SubjectFromWebIdentity Token (the unique ID that appears in an IAM policy variable for this particular identity provider)
5. Using the above credentials, the user is allowed to access DynamoDB

DynamoDB Condition Writes : If item = $10, then update to $12 (conditional writes are idempotent) E.g., two users try to update the same item at the same time

DynamoDB supports Atomic Counters, where you use the UpdateItem operation to increment or decrement the value of an existing attribute (or "field" in a table) without interfering with other write requests. (All write requests are applied in the order in which the were received.) For example, a web application might with to maintain a counter per visitor to their site, the application would need to increment this counter regardless of its current value.

Atomic Counters are not idempotent. This mean that the counter will increment each time you call UpdateItem. If you suspect that a previous request was unsuccessful, your application could retry the UpdateItem operation; however, this would risk updating the counter twice. This might be acceptable for a web site counter, because you can tolerate slightly over- or under-counting vistors. However, in a banking application, it would be safer to use conditional updates rather than atomic counters.

If you application needs to read multiple items, you can use the BatchGetItem API. A single BatchGetItem request can retrieve up to 1 MB of data, which can contain as many as 100 items. In addition, a single BatchGetItem request can retrieve items from multiple tables.

Simple Queue Service (SQS)

Very first service offered by AWS

• SQS vs. RabbitMQ:
  • SQS is a managed service. So one does not have to worry about operational aspects of running a messaging system including administration, security, monitoring etc. Amazon will do this for you and will provide support if something were to go wrong.
  • SQS is Elastic and can scale to very large rate/volumes (unlimited according to AWS)
  • Availability of SQS has a lot of 9's in it and is backed by Amazon, which is one less thing to worry about in your application.

Amazon SQS is a web service that gives you access to a message queue that can be used to store messages while waiting for a computer to process them.

SQS is a distributed queue system that enables web service applications to quickly and reliably queue messages that one component in the application generates to be consumed by another component. A queue is a temporary repository for messages that a awaiting processing.

Using SQS, you can decouple the components of an application so they run independently, with SQS easing message management between components. Any component of a distributed application can store messages in a fail-safe queue. Messages can contain up to 256 KB of text in any format. Any component can later retrieve the messages programmatically using the SQS API.

The queue acts as a buffer between the component producing and saving data, and the component receiving the data for processing. This mean that queue resolves issues that arise if the producer is producing work faster than the consumer can process it, or if the producer or consumer are only intermittently connected to the network.

Amazon SQS ensures delivery of each message at least once, and support multiple readers and writers interacting with the same queue. A single queue can be used simultaneously by many distributed application components, with no need for those components to coordinate with each other to share the queue.

SQS is engineered to always be available and deliver messages. One of the resulting trade-offs is that SQS does not guarantee first in, first out delivery of messages. For many distributed applications, each message can stand on its own, and as long as all messages are delivered, the order is not important. If your system requires that order be preserved, you can place sequencing information in each message, so that you can reorder the messages when the queue returns them.

To illustrate, suppose you have a number of images files to encode. In a SQS worker queue, you create a SQS message for each file specifying the command (jpeg-encode) and the location of the file in S3. A pool of EC2 instances running the needed image processing software does the following:

1. Asynchronously pulls the task messages from the queue;
2. Retrievers the named file;
3. Processes the conversion (e.g., create a thumbnail, add a watermark, etc.)
4. Write the image back to Amazon S3;
5. Writes a "task complete" message to another queue;
6. Deletes the original task message; and then
7. Checks for more messages in the worker queue

Visibility Timeout Clock only starts when the component server (i.e., EC2 instance) pulls the message from the queue

• SQS with Auto-scaling
• SQS does not offer FIFO (first in, first out)
• 12 hour visibility time out
• SQS is engineered to provide "at least once" delivery of all messages in its queues. Although most of the time each message will be delivered to your application exactly once, you should design your system so that processing a message more than once does not create any errors or inconsistencies.
• 256kb message size (as of May 2016)
• Billed at 64kb "chunks"
• A 256kb message will be 4 x 64kb "chunks"

• SQS Pricing
  • First 1 million SQS requests per month are free
  • $0.50 per 1 million SQS requests per month thereafter ($0.0000005 per SQS request)
  • A single request can have from 1 to 10 messages, up to a maximum total payload of 256kb
  • Each 64kb "chunk" of payload is billed as 1 request. For example, a single API call with a 256kb payload will be billed as four requests.

• If you see "decouple", think SQS

• SQS Delivery:
  • SQS messages can be delivered multiple times and in any order (no first in, first out; last in, last out)

• SQS - Default visibility timeout:
  • Default visibility timeout is 30 seconds
  • Maximum timeout is 12 hours
  • When you receive a message from a queue and begin processing it, you may find the visibility timeout for the queue is insufficient to fully process and delete that message. To give yourself more time to process the message, you can extend its visibility timeout by using the ChangeMessageVisibility action to specify a new timeout value. SQS restarts the timeout period using the new value.

• SQS Long Polling:
  • SQS long polling is a way to retrieve messages from your SQS queues. While the traditional SQ short polling returns immediately, even if the queue being polled is empty, SQS long polling does not return a response until a message arrives in the queue, or the long poll times out. SQS long polling makes it easy and inexpensive to retrieve messages from your SQS queue as soon as they are available.
  • Maximum Long Poll timeout is 20 seconds
  • Example exam question: Polling in tight loops is burning CPU cycles and costing the company money. How would you fix this? (Answer: Enable SQS Long Polling.)

• SQS - Fanning Out:
  • Create an SNS topic first using SNS. Then create a and subscribe multiple SQS queues to the SNS topic. Now whenever a message is sent to the SNS topic, the message will be fanned out to the SQS queues (i.e., SNS will deliver the message to all the SQS queues that are subscribed to the topic).

Simple Notification Service (SNS)

SNS is a web service that makes it easy to set up, operate, and send notifications from the cloud. It provides developers with a highly scalable, flexible, and cost-effective capability to publish messages from an application and immediately deliver them to subscribers or other applications.

SQS follows the "publish-subscribe" (pub-sub) messaging paradigm, with notifications being delivered to clients, using a "push" mechanism that eliminates the need to periodically check or "poll" for new information and updates. With simple APIs requiring minimal up-front development effort, no maintenance or management overhead and pay-as-you-go pricing, SNS gives developers an easy mechanism to incorporate a powerful notification system with their applications.

Push notifications to Apple, Google, Fire OS, and Windows devices, as well as Android devices in China with Baidu Cloud Push.

Besides pushing cloud notifications directly to mobile devices, SNS can also deliver notifications by SMS text message or email, to Amazon Simple Queue Service (SQS) queues or to any HTTP endpoint.

To prevent messages from being lost, all messages published to SNS are stored redundantly across multiple availability zones.

• SNS - Topics:
  • SNS allows you to group multiple recipients usings "topics". A topic is an "access point" for allowing recipients to dynamically subscribe to identical copies of the same notification. One topic can support deliveries to to multiple endpoint types. For example, one can group together iOS, Android, and SMS recipients. When you publish once to a topic, SNS delivers appropriately formatted copies of your message to each subscriber.
  • Subscriptions via email require the receiving email owner to confirm the subscriptions in order to receive notifications from the given topic (prevents spam). Subscription requests expire after 3 days, if the receiving email owner does not confirm the subscription.

Example SNS email notification:

{
  "Type" : "Notification",
  "MessageId" : "436d9234-f427-5be8-aa54-dd98ae4e286dba0",
  "TopicArn" : "arn:aws:sns:us-west-2:01234:MyTestSNSTopic",
  "Subject" : "This is a test",
  "Message" : "Hello, world!",
  "Timestamp" : "2016-05-10T21:53:37.981Z",
  "SignatureVersion" : "1",
  "Signature" : "PInj15UDcwMI==",
  "SigningCertURL" : "https://sns.us-west-2.amazonaws.com/SimpleNotificationService-9390147a5624348ee.pem",
  "UnsubscribeURL" : "https://sns.us-west-2.amazonaws.com/?Action=Unsubscribe&SubscriptionArn=arn:aws:sns:us-west-2:01234:MyTestSNSTopic:1a4eead5-110c-4acb-894a-91cdf358aabc",
  "MessageAttributes" : {
    "AWS.SNS.MOBILE.MPNS.Type" : {"Type":"String","Value":"token"},
    "AWS.SNS.MOBILE.MPNS.NotificationClass" : {"Type":"String","Value":"realtime"},
    "AWS.SNS.MOBILE.WNS.Type" : {"Type":"String","Value":"wns/badge"}
  }
}

• SNS benefits:
  • Instantaneous, push-based delivery (no polling)
  • Simple APIs and easy integration with applications
  • Flexible message delivery over multiple transport protocols
  • Inexpensive, pay-as-you-go model with no up-front costs
  • Web-based AWS Management Console offers the simplicity of a point-and-click interface

• SNS vs. SQS:
  • Both messaging services in AWS
  • SNS => Push
  • SQS => Pulls (Polls)

• SNS Pricing:
  • Users pay $0.50 per 1 million SNS requests
  • $0.06 per 100,000 notification deliveries over HTTP
  • $0.75 per 100 notification deliveries over SMS
  • $2.00 per 100,000 notification deliveries over email

SNS data format = JSON

• SNS protocols include:
  • HTTP
  • HTTPS
  • Email
  • Email-JSON
  • Amazon SQS
  • Application

NOTE: Messages can be customized for each protocol

Simple Workflow Service (SWF)

Amazone Simple Workflow Server (Amazon SWF) is a web service that makes it easy to coordinate work across distributed application components. SWF enables applications for a range of use cases, including media processing, web application back-ends, business process workflows, and analytics piplines, to be designed as a coordination of tasks. Task represent invocations of various processing steps in an application, which can be performed by executable code, web service calls, human actions, and scripts.

SWF Workers 

programs that interact with SWF to get tasks, process received tasks, and return the results.

SWF Decider 

a program that controls the coordination of tasks, i.e., their ordering, concurrency, and scheduling according to the application logic.

The workers and the decider can run on cloud infrastructure, such as Amazon EC2, or on machines behind firewalls. SWF brokers the interactions between workers and the decider. It allows the decider to get consistent views into the progress of tasks and to initiate new tasks in an ongoing manner. At the same time, SWF stores tasks, assigns them to workers when they are ready, and monitors their progress. It encures that a task is assigned only once and is never duplicated. Since SWF maintains the application's state durably, workers and deciders do not have to keep track of execution state. They can run independently and scale quickly.

SWF Domains 

Your workflow, activity types, and the workflow execution itself are all scoped to a domain. Domains isolate a set of types, executions, and task lists from others within the same account. One can register a domain by using the AWS Management Console or by using the RegisterDomain action in the SWF API.

The parameters of an SWF Domain are specified in JSON format. E.g.:

https://swf.us-west-2.amazonaws.com
RegisterDomain
{
  "name": "123456789",
  "description": "images",
  "workflowExecutionRetentionPeriodInDays": "60"
}

Maximum Workflow can be 1 year and the value is always measured in seconds.

• SWF vs. SQS:
  • SWF presents a task-oriented API, whereas SQS offers a message-oriented API.
  • SWF ensures that a task is assigned only once and is never duplicated. What SQS, one needs to handle duplicate messages and may also need to ensure that a message is processed only once.
  • SWF keeps track of all the tasks and events in an application. With SQS, one needs to implement one's own application-level tracking, especially if one's application uses multiple queues.
  • Does the service require human interaction? If so, one should use SWF.
  • Does the service need to run for (much) more than 12 hours? If so, one should use SWF. If less than 12 hours, SQS might be the correct service to use.
  • Maintaining your application's execution state (e.g. which steps have completed, which ones are running, etc.) is a perfect use case for SWF.
  • Amazon SWF is useful for automating workflows that include long-running human tasks (e.g. approvals, reviews, investigations, etc.) Amazon SWF reliably tracks the status of processing steps that run up to several days or months.

CloudFormation

CloudFormation => Scripted infrastructure (IaaS)

Using the CloudFormation service is free. However, any resources it creates/consumes/provisions (e.g., EC2 instances, Load Balancers, etc.) are not free.

CloudFormation templates are written in JSON format. E.g., CloudFormation LAMP stack template (see here for full template):

{
  "AWSTemplateFormatVersion" : "2010-09-09",
  
  "Description" : "AWS CloudFormation Sample Template LAMP_Single_Instance:
Create a LAMP stack using a single EC2 instance and a local MySQL database for
storage. This template demonstrates using the AWS CloudFormation bootstrap
scripts to install the packages and files necessary to deploy the Apache web
server, PHP and MySQL at instance launch time. **WARNING** This template
creates an Amazon EC2 instance. You will be billed for the AWS resources used
if you create a stack from this template.",
  
  "Parameters" : {
      
    "KeyName": {
      "Description" : "Name of an existing EC2 KeyPair to enable SSH access to the instance",
      "Type": "AWS::EC2::KeyPair::KeyName",
      "ConstraintDescription" : "must be the name of an existing EC2 KeyPair."
    }, 
...

• CloudFormation "output":
  • SEE: http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/outputs-section-structure.html
  • Remember the "Fn::GetAtt" function

If a CloudFormation stack creation fails, the default is to terminate and rollback all resources created on failure (i.e., delete all of the resources it was trying to create). One can disable roll back to leave all resources in their current state (failed or not). This is useful for troubleshooting your own templates.

Elastic Beanstalk

Using the AWS Elastic Beanstalk service is free. However, any resources it creates/consumes/provisions are not free.

• Environment tier:
  • Web Server Environment - Provides resources for an AWS Elastic Beanstalk web server in either a single instance or load-balancing, auto scaling environment.
  • Worker Environment - Provides resources for an AWS Elastic Beanstalk worker application in either a single instance or load-balancing, auto scaling environment.
• Environment type:
  • Single instance
  • Load balancing, auto-scaling
• Preconfigured platforms:
  • PHP, Node.js, Python, Ruby, Tomcat, IIS, Java, Go, Docker

Virtual Private Cloud (VPC)

Think of a VPC as a virtual data centre in the cloud.

• AWS definition of a VPC:
  • Amazon Virtual Private Cloud (VPC) lets you provision a logically isolated section of the AWS Cloud where you can launch AWS resources in a virtual network that you define. You have complete control over your virtual networking environment, including select of you own IP address range, creation of subnets, and configuration of route tables and network gateways.
  • You can easily customize the network configuration for your AWS virtual Private Cloud. For example, you can create a public-facing subnet for your webservers that has access to the Internet, and place backend systems, such as databases or application service in a private-facing subnet, with no Internet access. You can leverage multiple layers of security, including security groups and network access control lists, to help control access to EC2 instance in each subnet.
  • Additionally, one can create a Hardware Virtual Private Network (VPN) connection between your corporate datacentre and your VPC and leverage the AWS cloud as an extension of your corporate datacentre. "Hybrid Cloud".

• What can one do with a VPC?
  • Launch instances into a subnet of your choosing
  • Assign custom IP address ranges in each subnet
  • Configure route tables between subnets
  • Create internet gateways and attach them to subnets (or not)
  • Much better security control over your AWS resources
  • Instance security groups
  • Create subnet network access control lists (ACLs)
  • Number of allowed VPCs in each AWS Region (by default): 5

• Default VPC vs. Custom VPC
  • Default VPC is user friendly (automatically created when one creates an AWS account). It allows one to immediately deploy instances.
  • All subnets in a default VPC have an internet gateway attached (i.e., all subnets are public)
  • Each EC2 instance has both a public and private IP address
  • If one were to delete the default VPC, the only way to get it back is to contact AWS

• VPC Peering
  • Allows one to connect one VPC with another via a direct network route using private IP addresses.
  • Instances behave as if they were on the same private network.
  • One can peer VPCs with other AWS accounts as well as with other VPCs in the same account.
  • One cannot create a VPC larger than /16
  • Peering is done in a "star configuration", i.e., 1 central VPC peers with 4 others. No transitive peering!

• A "star configuration" peering:

                 +-------+
                 | VPC C |
                 +-------+
                     ^
                     |
                     v
+-------+        +-------+        +-------+
| VPC B | <----> | VPC A | <----> | VPC E |
+-------+        +-------+        +-------+
                     ^
                     |
                     v
                 +-------+
                 | VPC D |
                 +-------+

In the above example, instances on VPC-B can not send/receive traffic on VPC-C via VPC-A. One would need to create a VPC peer directly from VPC-B and VPC-C. That is, with a star configuration (as shown above), there is no transitive peering.

• VPC tenancy:
  • Default - EC2 instances are created on shared hardware
  • Dedicated - EC2 instances are created on dedicated hardware (regardless of the tenancy attribute specified at launch). This the more expensive option.

By default, when one creates a VPC, a route tables is automatically created for the VPC.

NOTE: If one deletes one's account's default VPC (and/or the associated default subnets), the only way to get them back is to raise a ticket with Amazon.

• VPC subnets:
  • Use the CIDR format to specify your subnet's IP address block (e.g., 10.0.0.0/24). Note that block sizes must be between a /16 netmask and /28 netmask. Also, note that a subnet can be the same size as your VPC.
  • Subnets are always mapped to one availability zone (AZ). Subnets can not be mapped across multipl AZs. 1 subnet = 1 AZ.

• VPC Internet gateways:
  • By default, when one creates an Internet Gateway, it is detached. One must attach it to a VPC in order to use it.
  • One can only have 1 Internet Gateway per VPC.

Security Groups can span multiple VPCs and VPC subnets.

• Network Address Translation (NAT) Server:
  • Allow instances with only private IPs to reach the Internet via the NAT server (via, say, HTTP/HTTPS and all other protocols/ports closed, including SSH)
  • One must disable Source/Destination Check on NAT instances for them to work properly.

Network Access Control Lists (ACLs)

• A network ACL is an optional layer of security that acts as a firewall for controlling traffic in and out of a subnet.
• ACLs ~ "firewall"-like rules
• ACLs are a numbered list of rules, which are followed in order, starting with the lowest number first. They control network ingress/egress for all AWS resources within a given subnet.
• The highest ACL number allowed is 32,766
• ACLs have a default (editable) number list that allows all inbound/outbound traffic
• One can create a custom ACL, which starts out with no inbound/outbound traffic allowed, until one adds a rule
• ACLs are applied to an entire subnet (and override the security groups associated with a given instance on that subnet). For an example, if a security group applied to a given instance has port 80 allowed, but the ACL for the subnet the instance is on has port 80 denied, the ACL rule takes precedence (i.e., port 80 will be denied on all instances within that subnet, regardless of what the security group allows).
• Unless one creates a custom ACL and associates it with a given subnet, that ACL will use the default role and rules.
• One can not have multiple ACLs associated with the same subnet. However, a given ACL can be associated with multiple subnets.
• If one dis-associates a custom ACL from a given subnet(s), the subnet reverts back to the default ACL

Example labs

• VPC Lab:
  • Create a custom VPC
    • Define an IP Address Range (e.g., 10.0.0.0/16)
    • By default, this created a Network ACL and Routing Table
  • Create a custom Route Table
  • Create 3 subnets (e.g., 10.0.1.0/24, 10.0.2.0/24, and 10.0.3.0/24)
    • public subnet: 10.0.1.0/24; private subnets: 10.0.2.0/24 and 10.0.3.0/24
  • Create an Internet Gateway
  • Attach Internet Gateway to the custom VPC
  • Associate the public subnet with the custom Route Table
  • Provision an EC2 instance with an Elastic IP address (in the public subnet)
  • Provision an EC2 instance with only a private IP address (on the private subnet)

• NAT Lab:
  • Create a custom security group
  • Allow inbound traffic to 10.0.1.0/24 and 10.0.2.0/24 on HTTP and HTTPS
  • Allow outbound traffic on HTTP and HTTPS to anywhere
  • Provision a NAT instance inside the public subnet
  • Import! Make sure to select "Disabled Source/Destination Check" on this NAT instance!
  • Set up a route on the private subnet to route traffic through the NAT instance

AWS Shared Responsibility

Shared Responsibility Model for AWS Infrastructure Services

• Infrastructure services include: EC2, EBS, Auto-scaling, VPC, etc
• Amazon responsibility (i.e., managed by AWS):
  • AWS global infrastructure (regions, availability zones, edge locations)
  • Foundation services (compute, storage, databases, networking)
  • AWS endpoints
  • AWS IAM
• Customer responsibility (i.e., managed by AWD customers):
  • Server-side encryption, network traffic protection
  • Operating system, network, and firewall configuration(s)
  • Platform and application management
  • Customer data
  • Customer IAM

Shared Responsibility Model for AWS Container Services

• Container services include: Relational Database Services (RDS), Elastic Map Reduce (EMR), and Elastic Beanstalk
• AWS takes on more of the responsibility (e.g., operating system and network configuration; platform and application management)
• Customer still has the responsibility for firewall configuration

Shared Responsibility Model for AWS abstracted services

• Abstracted services include: S3, Glacier, DynamoDB, SQS, and Simple Email Service (SES), Lambda
• AWS takes on each more of the responsibility (e.g., network traffic protection provided by the platform; server-side encryption provided by the platform)
• Customer still has the responsibility for client-side data encryption, data integrity authentication, and customer data

AWS Certifications

NOTE: All AWS certification exams are taken on-site and proctored

• Associate Level (each $150):
  • AWS Certified Developer - Associate
  • AWS Certified SysOps Administrator - Associate
  • AWS Certified Solutions Architect - Associate
• Professional Level (each $300):
  • AWS Certified DevOps Engineer - Professional
  • AWS Certified Solutions Architect - Professional

Glossary

see: Official AWS Glossary

AMI

Amazon Machine Image

ARN

Amazon Resource Name

EBS

Elastic Block Storage

EC2

Elastic Compute Cloud

EFS

Elastic File System

ELB

Elastic Load Balancer

S3

Simple Storage Service

Links

AWS Whitepapers

• AWS Whitepapers
  • AWS Security Best Practices (PDF)
  • AWS Security Whitepaper (PDF)
  • AWS Storage Services Whitepaper (PDF)
  • AWS Well-Architected Framework (PDF)

Training and certification

• https://acloud.guru/courses
• https://aws.amazon.com/certification/certified-developer-associate/
• What you need to get AWS certified — by A Cloud Guru
• Register for AWS certification exams

Miscellaneous

• http://ec2price.com/
  • https://github.com/grosskur/ec2price