EC2 Zero to Hero

 

EC2 TheHungryFatCoder

EC2 or Elastic Compute Cloud, is Amazons primary webservice that provides reliable compute capacity on the cloud. Compute means the amount of computation power required by your workload.

You could acquire compute by launching virtual servers, and these are called Instance. When you launch an instance, you could use the compute as you wish. The best thing is that you only pay per hour, so as soon you stop your instance from running, you’ll no longer will be paying.

The 2 concepts that are key to launching an instance on AWS: These dimensions of the instances are controlled by the instance type and the AMI.

  1. The amount of virtual hardware dedicated to the instance.
  2. The software loaded on the instance.

What is an instance type –

It’s the virtual hardware supporting an Amazon EC2 instance. There are many such instance types and it has the following dimensions:

  • Virtual CPU (vCPU)
  • Memory
  • Storage (size and type)
  • Network performance

Instance Type’s are grouped into families and based on the ratio of the above dimensions.

The m4 family provides a balance on compute, memory and network resources. Amazon provides some good options for Instance Type’s within a family and they scale up linearly. Have a look at the AWS documentation to find out more.

When you consider a Family, Amazon has tilted the ratios so it can accommodate different types of work-loads.

Family Optimized for Purpose
c4 Compute optimized For workloads that require processing
r3 Memory optimized For memory intensive workloads
i2 Storage optimized For workloads that requires high amounts of SSD storage
g2 GPU based instance For graphics and general-purpose GPU compute workloads.

 

Note: C3,C4,d2,I2,M4,R3 families provide enhanced networking capabilities. Check the AWS documentation for an updated list.

With changing customer demands, AWS introduces new processing families. When choosing an Instance Type, consider the networking performance as well. AWS publishes a relative measure of networking performance as low, moderate or high. However, in some instance types it specifies the network performance of 10Gbps. A general observation is that the network performance increases within a family of the Instance Type.

For customers that require greater performance in their business use cases, AWS provides many instance types that support enhanced-networking.

What is enhanced-networking in an Instance Type

It reduces the impact of virtualization on network performance by enabling a capability called Single root I/O virtualization (SR-IOV). As a result, this increases packets per second (PPS), lower latency and less jitter.

Enabling enhanced networking on an instance involves ensuring the correct drivers are installed and modifying the instance attributes. Enhanced networking is available for instances that runs in an Amazon VPC.

Amazon Machine Image (AMI)

It defines the initial software that will be run on the instance when it is launched. All AMIs are x86, and comes in either Windows or Linux.

  • OS and configuration
  • The initial state of the patches
  • Application or system software

AMIs are provided by 4 sources

  1. Published by AWS –
    • This includes multiple distributions of Linux OS, like Ubuntu/ Redhat/Amazons own distribution, Or Windows OS like, Windows 2008/Windows 2012.
    • These AMIs will result in the default OS settings (Similar to installing an OS from ISO image)
    • Like in any OS installation you should apply all appropriate patches upon launch.
  2. AWS marketplace –
    • AWS partners have provided AMI with their software’s made available on it.
    • Helps customers, find, buy, and start using immediately.
    • Provides 2 benefits:
      • Customers do not need to install software’s.
      • License agreement is appropriate for the cloud.
    • Charged per hour, and an additional per-hour charge for software installed. No additional charges for Open Source AWS Marketplace packages.
  3. General form existing instance –
    • An AMI created from an existing EC2 instance. A customer, may configure the instance based on his corporate standards, considering security, etc.
    • This AMI can then be generated in the form of a generated from the configured instance and made use to launch all other instances guaranteeing No-Non-Conforming instances created in the company.
  4. Updated virtual servers –
    • Using the AWS VM Import/Export service, customers can create images from various virtualization formats. Like raw, VHD, VMDK and OVA.
    • Customers are responsible to remain compliant with the licensing terms of OS vendor.

There are several ways where you could access your instance securely. Ex: CLI, AWS management console, SDKs etc.

There are few ways an instance can be addressed over the web:

  1. Public Domain Name System (DNS) –
    • When you launch an instance AWS will create a DNS name for that instance and it could be used to access the instance.
    • The customer has no control over naming the DNS name, AWS takes care of it.
    • Only persist while the instance is running and can’t be transferred to another instance.
  2. Public IP –
    • When you launch an instance AWS will give it a public IP address.
    • IP addresses cannot be specified, and it’s assigned from a set of addresses reserved by AWS.
    • Only persist while the instance is running and can’t be transferred to another instance.
  3. Elastic IP –
    • It is a unique IP that you reserve and assign to a running EC2 instance.
    • You can transfer the IP address to another instance without coupling clients to an instance.

In addition to the above, private IP addresses and Elastic Network Interface (ENI) can be used to address instances that are available in Amazon VPC.

There’s always a first time

Amazon EC2 uses Public-key-cryptography to encrypt and decrypt login information. Public-key-cryptography uses the public-key to encrypt and private-key to decrypt. These 2 keys together are called a key-pair. A key-pair can be created in the AWS Management Console, CLI or API or else the customer can upload their own key-pairs.

For Linux instances, AWS stores the public key securely (~/.ssh/authorized_keys), and the customer is responsible to store the private key. The private-key can be used to securely access an instance for the first time.

For Windows instances, EC2 will generate a password for the local admin account and encrypt it with the public-key. Initial access is gained by decrypting the password with the private-key in the console or through the API. Using RDP the decrypted password can be used to login.

You could also change your local administrator password and it is considered to be a best practice.

Virtual Firewall Protection or Security-Groups

AWS has virtual firewalls and it is used to control traffic that goes in and out of your instances. And they are called Security-Groups. These Security-Groups allows you to control traffic using the following attributes:

  1. Port – The port number
  2. Protocol – The communication standard for traffic
  3. Source/destination – source or destination for traffic rules. Can be defined in 2 ways:
    • CIDR – x.x.x.x/x style that defines a particular range of IP address
    • Security group – includes any instance that is associated with the Security-Group. Helps prevent coupling Security-Group rules with Ip addresses.

Comparison between Security-Groups associated with VPC and EC2-Classic.

EC2-Classic SG Control outgoing instance traffic.
VPC SG Controls both outgoing and incoming instance traffic.

 

Every Instance is associated with at least one Security-Group. This means you could associate an instance with more Security-Groups.

A Security-Groups  is default deny. This means it does not allow any traffic that is not explicitly allowed by a Security-Groups rule.

A rule can be defined using the 3 attributes listed above: port, protocol and source/destination.

When there’s more than 1 Security-Group associated, then the Security-Groups will be aggregated. For example if Instance A allows SSH traffic at xx.xx.xx.xx/16 and also HTTP traffic from all 0.0.0.0/0 then the instance will allow both SSH and HTTP traffic.

Security-Groups Are stateful. This means if you specify a rule for outgoing request, it will allow to retrieve its response without having to define a rule for it.

Security-Groups are applied at instance level, so unlike traditional on-premises firewalls a hacker could breach a single perimeter to get access to all the instances in your Security-Group. However, since Security-Groups are applied at instance level, the hacker will have to penetrate instances individually.

 

Stay tuned for Part 2 of Amazon EC2. 

If you happen to be on Instagram please follow @thehungryfatcoder.

If you happen to like my blog, please subscribe and let your friends know about it.

7 Steps to host your static website on S3 – Part2

S3

Part 1 of the Amazon S3 beginner to hero series

You could leverage on the security, durability, availability and scalability by hosting it on S3. In 7 steps you could easily configure the bucket for website hosting and upload the content of your static website.

  1. Create a Bucket with the desired website host-name ex: TheHungryFatCoder
  2. Upload the static flies to the bucket.
  3. Make all files public. Ex: images, .html files etc.
  4. Enable static website hosting for Bucket. Includes specifying an index and error file.
  5. The website will be available at URL ex: bucketName.S3-website-AWS_Region.amazonaws.com
  6. Create a DNS name in your domain name for the website using a DNS CNAME or Amazon Route53 alias that resolves to the S3 website URL.
  7. http://thehungryfatcoder.com The website will be active and accessible.

 

Stay tuned for Part 3. 

If you happen to be on Instagram please follow @thehungryfatcoder.

If you happen to like my blog, please subscribe and let your friends know about it.

Amazon S3 beginner to expert : Part 1

S3 or Simple Storage Service, is one of the first services introduced by AWS. S3 and Glacier, is the core object storage in AWS. It provides a secure, durable and highly scalable cloud storage and consist of a simple web-service interface where it can be used to store or retrieve any amount of data. Further, as a plus you only require paying for the storage you have used.

There are many AWS service Amazon Elastic MapReduce (EMR) that depends on S3 as its target storage. Amazon Kinesis and are a few. Its also used as the storage for Amazon Elastic Block store(EBS) and Amazon Relational Database snapshots. In addition, it’s also used for data staging or loading mechanism for Amazon Redshift and DynamoDB.

AWS resources

Out from the many uses of S3, it can be used as a backup and archive facility for on premises data storage. Further, it could be used to store content, media, software and data for distribution. It is also useful in Bigdata analytics, static website hosting and cloud native mobile and internet application hosting. Also used in disaster recovery.

S3 offers 3 storage classes

  1. General Purpose
  2. Infrequent access
  3. Archive

By using lifecycle-policies, one can automatically migrate objects stored to the appropriate storage class. For example: If you are the administrator of a social media website, you could write a lifecycle policy that states to archive all profile pictures that’s being saved for than 12 months. (This may not be the best example but hope you understood what lifecycle policy is all about.)

S3, also provides functionality that provides a rich set of permission, access-control and encryption options which we’ll visit in this post.

On the other hand, Amazon Glaciers provides very cheap, data archiving and backup functionality, which is used to store ‘Cold Data’ or rarely accessible data. The retrieval time is usually 3 – 5 hours and can be used as a storage class for Amazon S3.

Traditional storage options

In traditional IT environments, there are 2 kinds of storage that are used.

  • Block storage – Operate at a lower level (Raw storage device level), stored in bits/bytes, Fixed block size
  • File Storage – Operates at a higher-level ex: OS level. Manages data as a hierarchy of folders and files.

S3, on the other hand provides ‘Object Storage’ feature on the cloud. It runs independent of any servers and accessed directly over the internet via an API. An Object consist of both data-stored and its meta-data. These Objects are stored in a Bucket, and it should have a unique user-specified key. It is said that these Buckets can hold an unlimited amount of files and a file size can range from 0 – 5TB in size.  These data can be replicated across multiple Availability Zones(AZs) and automatically replicate on multiple devices in multiple facilities in a region. In case the request rate of S3 goes up, it can partition buckets to support a high request rate thus, its highly scalable.

Buckets

AWS BucketIt’s a container, like a web-folder for objects (files). Bucket names are global. So, if you have named your bucket ‘TheHungryFatCoder’, you or anyone else in the world will not be able to create a bucket with the same name ever again.

A Bucket name can contain up to 62 lowercase letters, numbers, hyphen or periods. By default you can have up to 100 buckets per account. If you need additional buckets, you can increase your account bucket limit to a maximum of 1,000 buckets by submitting a service limit increase. By default, data in a S3 bucket are stored in a region, and you need to specify to copy it another region if required.

Objects

An unlimited number of Objects can be stored in a S3 Bucket, where an object can be of size between 0 and 5TB. These Objects stored in Buckets consist of:

  • Key – The name you give your object
  • Version Id – a key and version ID to uniquely identify an object
  • Value – the content that’s being stored
  • Metadata – Data about the file.
    • System data: Date modified, object size, MD5, HTTP content type
    • User data: Can be specified at the time of creating the object, Its optional. Can tag data with meaningful attributes.
  • Sub resource – additional information about objects
  • Access control information – control access to the objects that’s stored.

Note: S3 supports bit-torrents protocol, you can use the BitTorrent protocol to retrieve any publicly-accessible object in Amazon S3. You can only get a torrent file for objects that are less than 5 GBs in size.

The data is stored as a stream of bytes, and Amazon doesn’t know the type of data which is stored.

Keys

Every Object saved in a Bucket has a unique name called a Key. It can be considered as a filename. It needs to be a unique name within the bucket. It can have up to 1024 bytes Unicode UTF, and can consist of dashes, slashes, backlashes to dots.

In general practice a key will have the naming convention : bucket-key-versionId.

Tag

Bucket can be tagged, but the individual Objects within the Bucket doesn’t inherit the tagging. Individual Objects require to be tagged separately.

Object URL

All objects stored in a Bucket is unique. It consists of the following URL format.

http://bucketName.s3.amasonaws.com/object.doc

One can also name its object as /my/folder/structure/object.pdf. We discussed that a Key can contain dashes, slashes, backlashes to dots in it.

There’s a common misconception among individuals considering S3 to be a file system, and you could create a Bucket within a Bucket. However, S3 is not a file-system, yet you could navigate to the Bucket as in a folder hierarchy using the Amazon Console. You could learn more about this by visiting the AWS Link here.

Amazon S3 Operations

S3, can perform the following operations.

  • Create or delete bucket
  • Write an object
  • Read an object
  • Delete an object
  • List keys in bucket

REST interface

Amazon S3 is a REST API. One can use HTTP or HTTPS to request the above S3 operations.

  • Create – PUT or at times POST
  • Read – GET
  • Delete – DELETE
  • Update – POST or at times PUT

What is Durability? What is Availability?

Durability addresses the question “Will my data be there in the future”?

.

Availability address the question “Can I access the data right now?

S3, is designed so that it has 99.999999999% (9 9s) durability and 99.99% availability. This means if you store 10K objects, you’ll expect a loss of a single object every 10 million years. Durability is maintained by Amazon through automatically storing data redundantly on multiple devices in locations within a region.

In case, there’s a scenario where you want to store non-critical data or data that’s easily reproducible, for ex: thumbnail image. You can use Reduced-Redundancy-Storage (RRS) which offers 99.99% Durability at a lower cost for storage.

Despite all the high level of Durability provided at infrastructure level, it is your responsibility to protect the data from accidental deletion, overwriting etc. Features such as Versioning, Cross regional replication and MFA Delete can you used to prevent such unforeseen events.

How is data consistency maintained?

Data stored in S3, automatically gets replicated in multiple servers, in multiple locations within a region. However, it takes some time to propagate changes to all locations when a modification happens. If you have already used S3, you might have come across instances where you try to read an object after an immediate update, and it’ll return the old object. S3, provides ‘Eventual Consistency’ for PUTs to existing objects and for DELETEs.

For PUTs to a new object this is not a concern as S3 provides ‘Read after write consistency’.

What is Eventual Consistency and Read after write consistency?

Read-after-write consistency guarantees immediate visibility of new data to all clients. A newly created object / file / table row will immediately be visible without any delays.

Eventual consistency means that after an update, you may or not see the changes if you immediately read the data.

Access Control

S3 is secure by default where you only get access to it when a Bucket or Object is created. It provides the following access mechanisms:

  1. Coarse-grained access control – Amazon S3 ACL

Allows you to grant READ, WRITE or FULL-CONTROL at Objects or Bucket level. ACL was created before, IAM and it is a Legacy access control mechanism.

            ACLs are used today to enable bucket logging, or to provide World-Read permission for website hosting.

  1.  Fine grained access control – S3 bucket policies, IAM policies, query string authentication.

S3 bucket policies are the most recommended access control mechanism. It also provides much fine grain control. It is very similar to IAM policies but differs:

  • Associated with bucket resource
  • Include an explicit reference to IAM principal in the policy. This means that it can be associated with a different account making S3 bucket policies allow you to assign cross account access to Amazon S3 resources.
  • With Amazon S3 bucket policies, you can specify who can access the bucket, from where, and at what time of the day.
  • One can also use IAM policies to grant permission to Amazon S3, as it grants access to other AWS service and resources.

 

Stay tuned for for the second post : 7 Steps to host your static website on S3. 

If you happen to be on Instagram please follow @thehungryfatcoder.

If you happen to like my blog, please subscribe and let your friends know about it.

AWS Elastic Container Service – Everything you need to know | part 2

In order to understand and work with ECS it’s important to get to know other services it associates with. AWS Auto Scaling Group, AWS Elastic Load Balancer and AWS Virtual Private cloud are such services. Following is a nutshell of the functionality of these services required to understand ECS and we’ll discuss in details of these services in a later section.

If you have missed part 1 of the ECS tutorial series please view it here. AWS Elastic Container Service – Everything you need to know | part 1

Auto Scaling Group (ASG)

AWS ASGServices rents instances on your behalf to run docker containers.

One of the ways ECS makes you run Docker containers is by using Auto scaling groups. What ASG does is that it creates a group of identical clones of an instance – with the exact configuration. Auto scaling group (ASG) is not able to run a diverse set of configurations, but the exact same copy.  These instances are managed by a central service and almost all the time its controlled by another service and rarely used individually.

Imagine an occurrence, where the cluster is always instructed to run 6 instances. However, due to some unforeseen circumstance one of these instances crashed. The auto-scaling group is now responsible to maintain the proper number of healthy instances (in this case 6 instances), thereby, it fires another instance to maintain its equilibrium.

Elastic Load Balancer (ELB)

AWS ELBELB controls ASG. It provides 2 main functionalities:

            1. Connects incoming request to instances.
            2. Monitor instances (ASG), to look at its load. ELB uses AWS CloudWatch to monitor the load in ASG and instruct ASG if to fire up or fire down instances based on the load.

 

 

Virtual Private Cloud (VPC)

AWS VPCAWS VPC enables you to launch AWS resources into a virtual network that you’ve defined.

ECS depends on VPC. A group of instances otherwise isolated from the internet, can connect with each other using VPC. Instances can connect to one of this virtual networks that will allow to make direct connections to each other.

Instances in addition to being connected to a VPC, can also be connected to the outside internet, or they could even be connected to many VPCs all at once.

ELBs are able to forward connections into VPCs to allow machines on the outside to make use of the services offered within a VPC. In addition, ELBs are capable of connecting directly to a container running on an instance in a VPC.

AWS Elastic Container Service – Everything you need to know | part 1

ECS – Elastic Container Service! It might sound all greek to you, yet, I assure you that by the end of this tutorial series you’ll know lots about it. So bare with me. Subscribe to my channel. (wink)

First thing first. What is ECS ?

So this is what the documentation have to say: 

It’s a highly scalable, container-orchestrated service that supports Docker containers and will allow you to easily run and scale containerized application on AWS.

In other words, ECS allows you to run containerized apps on AWS, and that’s all what you need to know.

In the first ½ of this tutorial, lets cover the theoretical aspects of ECS and then complete a fully fledged practical session where you will containerize a dotnet-Core app with Docker and have it up and running on ECS. Finally, we’ll also look at how we could use AWS cloud formation to setup the infrastructure.

What you’ll need to know to follow this tutorial series:

  • Basic Docker understanding
  • Docker installed
  • Visual studio installed 
  • AWS account 

Docker in a nutshell

Docker URL - TheHungryFatCoder

Imagine an instance where you require several applications to collaborate with each other to perform a desired output. And, these applications needs to run on the same environment where each of them requires different Java versions installed. 

Since, the host machine is installed with java 6, there’ll be many issues running the other 2 application which requires Java 7 and 8. Docker solves this by allowing each application to have its own version of java installed, hence overcoming this issue.

Another example would be that the piece of code you wrote runs on your machine but, not on the test environment. Using Docker you could bundle all the dependencies thereby guaranteeing a smooth run.

Using docker container means you can abstract the software, OS, hardware config from the App and have a standard building block where you can run anywhere.

ECS in a nutshell

AWS ECS - TheHungryFatCoderECS Abstracts away which machine runs in which container. It treats a bunch of instances that you rent as a shared pool, and combines it together and lets the ECS decides which instance in the pool runs which container.

If an instance goes away ECS will notice it and monitors containers, maintain logs on information such as which container ran it etc.

While giving the develop full control, it allows you to store the images, monitor containers, and maintain logs.

 

Instance, Cluster, Task, Service

It’s all confusing! Prior to continuing with this tutorial lets understand what these terminologies are.

The following diagram illustrates what these really mean:

  • ECS Cluster – is nothing but a logical grouping of container instances.
  • Container instances – It’s simply an EC2 instance. However, you could uniquely identify a container instance with a plain EC2 instance as it has an ECS agent installed in it, then have Docker installed and its registered to an ECS Cluster.
  • Task definition – Its a blueprint that explains which Docker container to run and represent your application. It will contain information such as which container to use, CPU allocation, Memory to allocate, env variables, ports to expose.
  • Task – An instance of a Task definition, running the definition derived within it. Based on demand, multiple tasks can be created by a task definition.
  • Services – Defines the minimum and maximum Task from one Task-Definition at a given time, auto scaling and load balancing. A service is responsible to create Task. Services can be used to configure load balancer.

 

Stay tuned for Part 2.

If you are already on instagram, please follow my handle @thehungryfatcoder.