Skip to content

Latest commit

 

History

History
459 lines (232 loc) · 10.4 KB

04. EC2 Instance Storage.md

File metadata and controls

459 lines (232 loc) · 10.4 KB
Raw

EBS Volume

  • EBS -> Elastic Block Store

  • a network drive (not a physical drive) attached to the instances while running

    • to communicate b/w the instance and EBS Volume, network is used

    • there might be a bit of latency from one computer to reach to another server

    • can be detached from an EC2 instance and attached to another one quickly -> helps in 'failovers'

  • allows instances to persist data after termination

    • can recreate an instance and mount to the same EBS Volume from before and get back our data

  • There's 'multi-attach' feature for some EBS

    • an EBS Volume can be attached to multiple instances

  • bound to a specific AZ

    • snapshot can be used to move across AZ

  • have a provisioned capacity (size in GBs and IOPS)

    • get billed for the provisioned capacity

    • increase the capacity of drive over time (for better performance/ more size)

  • DELETE ON TERMINATION :

    • controls the EBS behaviour when EC2 instance is terminated

      • by default, root EBS volume is terminated

      • by default, any other EBS volume is not deleted on termination

    • controlled by AWS console/ AWS cli

    • USE CASE : to save data -> preserve root volume when instance terminated

EBS snapshots :

  • a backup of EBS volume at a point of time

  • can backup the state of EBS volume

  • can restore the EBS volume even after deletion

  • not necessary to detach the volume to do snapshot but recommended so that the volume is clean

  • can copy snapshots across AZ/ region to leverage AWS global infrastructure

  • EBS SNAPSHOT ARCHIVE :

    • move a snapshot to archive tier

    • 75% cheaper

    • takes 24 to 72 hours for restoring the archive

  • RECYCLE BIN :

    • setup rules to retain deleted snapshots, to recover them after accidental deletion

    • specify retention (1 day to 1 year)

EBS Volume Types : (6 types)

  • EBS volumes are characterised based on some factors like

    • size

    • throughput

    • IOPS

  • 6 types :

    • General Purpose SSD - gp2,gp3

    • High Performance SSD - io1,io2

    • HDD - st1,sc1

  • For EC2 instances -> gp2/gp3 , io1/io2 can only be used as boot volumes (root OS runs on them)

  • NOTE : if you want to get also over 32,000 IOPS, you need Ec2 Nitro with io1 or io2

General Purpose SSD

balances price and performance for wide variety of workloads

  • gp2 (SSD) : (important for exam)

    • older version of volume

    • SIZE -> 1 GB to 16 TB

    • IOPS :

      • small gp2 volumes -> max 3000 IOPS

      • size of volume and IOPS are linked (max 16000 IOPS)

      • 3 IOPS per GB -> at 5334 GB - max IOPS (16000) achieved

    • ADVANTAGES:

      • cost effective storage

      • low-latency

    • DISADVANTAGE :

      • IOPS and throughput can't be set independently (linked together)

    • USE CASES :

      • system boot volumes

      • virtual desktop

      • development

      • test environments

  • gp3 (SSD) :

    • newer generation of volume

    • gives a baseline of

      • 3000 IOPS

      • 125 MB/s of throughput

    • can increase to

      • 16000 IOPS

      • 1000 MB/s throughput

      independently

    • ADVANTAGES:

      • cost effective storage

      • low-latency

      • IOPS and throughput can be set independently

Provisioned IOPS - PIOPS (SSD) : (important for exam)

  • USE CASE :

    • critical business applications

    • applications that need more than 16000 IOPS

  • when you have a database workload that

    • really uses the storage

    • very sensitive to storage performance and consistency

    switch to io1/io2 from gp2/gp3

  • supports EBS multi-attach

  • io1 / io2 (SSD)

    • Highest performance SSD volume

    • used for

      • mission-critical

      • low-latency

      • high throughput

      workloads

    • SIZE : 4 GB to 16 TB

    • Max PIOPS :

      • 64000 -> EC2 Nitro

      • 32000 -> others

    • can increase PIOPS idependently from storage size

  • io2 BLOCK EXPRESS (SSD) :

    • SIZE : 4GB to 64 TB

    • sub-millisecond latency

    • Max PIOPS : 256000

    • 1000 IOPS per GB

  • NOTE : io2 has more durability and more IOPS per GB compared to io1

Hard Disk Drive (HDD)

  • THROUGHPUT OPTIMIZED HDD - st1 (HDD) :

    • Low cost HDD volume

    • used for frequently accessed and throughput intensive workloads

    • SIZE : 125 MB to 16 TB

    • USE CASE :

      • Big data

      • Data warehouses

      • log processing

    • MAX THROUGHPUT : 500 MB/s

    • MAX IOPS : 500

  • COLD HDD - sc1 (HDD) :

    • Lowest cost HDD volume

    • used for less frequently accessed workloads

    • SIZE : 125 MB to 16 TB

    • USE CASE:

      • scenarios where lowest cost is important

    • MAX THROUGHPUT : 250 MB/s

    • MAX IOPS : 250

EBS Multi Attach : (io1/io2 family)

  • attach same EBS volume to multiple EC2 instances in the same AZ

  • each EC2 instance has full read and write permissions to the volume.

  • the application must be able to manage concurrent write operations into the same volumes.

  • you must use a file system that is cluster-aware

  • used to achieve HIGHER APPLICATION AVAILABILTY

EBS Encryption :

  • When an encrypted EBS volume is created followinf are also encrypted

    • data at rest

    • data in flight moving between instance and volume

    • snapshots

    • volumes created from snapshots

  • has minimal impact on latency

  • copying an uncrypted EBS snapshot allows encryption (Snapshots of encrypted volumes are encrypted)

  • EBS Encryption leverages keys from KMS (AES-256)

  • HOW TO ENCRYPT AN UNENCRYPTED EBS VOLUME :

    • create EBS snapshot of the volume

    • encrypt EBS snapshot by copying

    • create new EBS volume from snapshot

    • the newly created encrypted EBS volume can be attached to some instance

AMI

  • AMI -> Amazon Machine Image

  • powers the EC2 instances

  • built for specific region

  • ADVANTAGES :

    • represent a customization of an EC2 instance

      • own software, configuration, operating system, monitoring

      • faster boot time

      • faster configuration time (because all the software to be installed onto the EC2 instance are pre-packaged through an AMI)

    • can be copied across regions to leverage AWS global infrastructiure

  • can build your own AMI

  • EC2 instances can be launched from different kinds of AMI

    • PUBLIC AMI :

      • AWS provided

      • Amazon linux 2 AMI -> popular AMI for AWS

    • YOUR OWN AMI :

      • make and maintain AMI yourself

    • AWS MARKETPLACE AMI :

      • AMI made by someone else

EC2 Instance Store

  • EBS -> network drives with limited performance

  • EC2 instance store -> extremely high performance hardware disk

  • They're a virtual machine but obviously attached to a real hardware server. Some of these servers have disk space that is attached directly with a physical connection onto the server.

  • ADVANTAGES :

    • better I/O performance

    • good throughput

  • DISADVANTAGE :

    • EC2 instance store lose their storage once EC2 instance is stopped / terminated

    • It's called an ephemeral storage

      • not feasible for long term storage (use EBS instead)

    • If online server of the EC2 instance fails, data loss occurs as the hardware attached to the instance will also fail

  • USE CASES :

    • buffer

    • cache

    • scratch data

    • temporary content

  • NOTE :

    • It's your responsibility to create backups and replicate data correctly based on your needs

EFS :

  • managed NFS (network file system)

  • can be mounted on many EC2 instances in multi AZ

  • 1000s of concurrent NFS clients; High throughput

  • ADVANTAGES : highly available, scalable, pay per use (no need of provisioning capacity)

  • DISADVANTAGES : expensive (3x times gp2 cost), slower

  • USE CASES :

    • content management

    • web serving

    • data sharing

    • wordpress

  • uses BFSv4.1 protocol

  • uses SG to control access to EFS

  • not compatible with windows (only with Linux based AMI because it is a POSIX file system)

  • Can be ecnrypted at rest using KMS

  • Automatic Backups available (Additional Pricing)

EFS Performance

Performance Mode :

  • General purpose (default): latency-sensitive use cases like webserver, CMS

  • Max IO: High latency, thoroughput, highly parallel workloads like big data

Throughput Mode :

  • Burst Mode: File system throughput scales with file system size

  • Provisioned: Ability to set throughput regardless of storage size

EFS - Storage Classes

Storage Tier :

  • Standard : Frequently accessed files

  • EFS-IA (Infrequent Acess):

    • Storage tier for less frequently accessed files

    • Lower price compared to default EFS storage to store files; costs only for retrieving files.

    • Lifecycle management feature (moves after N days)

Deployment :

  • Regional : Multi-AZ deployment - Used for production

  • One Zone : One Zone EFS/IA - Used for development / backup (90% cost savings)

EBS vs EFS

EBS :

  • Can be attached only to one instance at a time (except io1/io2)

  • Bound to specific AZ

  • Storage capacity must be privisioned in advance

  • General Purposed disks are cost optimized

EFS :

  • Can be mounted to multiple EC2 instance at the same time

  • Can be attached to EC2 instances in multiple AZs

  • Pay-per-use

  • 3x epensive than EBS gp2s

  • Runs only on Linux

NOTE :

  • EFS is a network file system to be mounted across multiple instances.

  • EBS is for a network volume, it should be mounted only on one instance and it is locked to an AZ.

  • An Instance Store is to get the maximum amount of IO onto an EC2 instance,but is something you lose, if you lose that instance, so it is an ephemeral drive