Skip to content

Latest commit

 

History

History
662 lines (338 loc) · 17.7 KB

06. RDS, Aurora & ElastiCache.md

File metadata and controls

662 lines (338 loc) · 17.7 KB
Raw

Amazon RDS

  • RDS -> Relational Database Service

  • managed DB service for DB that uses SQL as a query language

    • SQL is a structured query language and well-adapted

  • allows to create DB in the cloud managed by AWS

  • DB engines managed by AWS :

    • Postgres

    • MySQL

    • MariaDB

    • Oracle

    • Microsoft SQL Server

    • Aurora (AWS Proprietary DB)

Why RDS over deploying own DB on top of EC2 instance?

  • RDS is a managed service :

    • Automated provisioning and OS patching

    • Continuous backups and restore to specific timestamp (Point in Time Restore)

    • Has monitoring dashboards (to view performance of the DB)

    • Read Replicas for improved read performance

    • Setup Multi AZ for Disaster Recovery

    • Maintenance windows for upgrades

    • Scaling capability

      • vertical -> increasing instance type
      • horizontal -> increasing Read Replicas

    • storage backed by EBS (gp2 or io1)

  • Can't SSH into the RDS instances

RDS Backups

  • Backups are automatically enabled in RDS

  • Automated backups:

    • Daily full backup of the DB

    • Transaction logs are backed-up by RDS every 5 mins

    • ability to restore to any point in time(from oldest backup to 5 mins ago)

    • 7days retention(can be increased to 35 days)

  • DB Snapshots:

    • Manually triggered by the user

    • Retention of backup for as long as you want

    • useful when you need to store the state of the DB for 6 months

RDS - Storage Auto Scaling

  • Helps increasing storage on RDS DB instance dynamically

  • RDS scales automatically when running out of free database storage

  • Avoid manually scaling your database storage

  • You have to set Maximum Storage Threshold (maximum limit for DB storage)

  • Automatically modify storage if:

    • Free storage is less than 10% of allocated storage

    • Low-storage lasts at least 5 mins

    • 6 hrs passed since last modification

  • Useful for applications with unpredictable workloads

  • Supports all RDS database engines

RDS Read Replicas vs Multi AZ

RDS Read Replicas (for read scalability)

  • helps to scale your needs (read scalability)

  • Up to 5 Read Replicas

  • Read Replicas can be (important)

    • Within AZ

    • Cross AZ

    • Cross Region

  • Replication is Asynchronous b/w main RDS DB instance and Read Replicas, so reads are eventually consistent

    • Eventually consistent - if your application reads from the Read Replica before they had the chance to replicate the data then you may get all data.

  • Replicas can be promoted to their own DB

    • It's completely out of the replication mechanism after that

    • It lives and has its own life cycle afterwards.

  • Applications must update the connection string to leverage the list of all read replicas present in RDS cluster

  • USE CASE :

    • You have a production database that is taking on normal load (having read and writes to our main RDS database instance)

    • You want to run a reporting application to run some analytics on the data

    • If you plug in that reporting application onto the main RDS DB instance, it's going to overload the DB and possibly slow down the production application

    • So instead, you can create a Read Replica to run the new workload there

    • The Reporting application can just do reads from your Read Replica and run the analytics there. The production application is completely unaffected.

    • Read replicas are used for SELECT (=read)only kind of statements (not INSERT,UPDATE,DELETE)

  • Network Cost :

    • In AWS there's a network cost when data goes from one AZ to another. (There are exceptions for managed service)

    • For RDS Read Replicas (a managed service)

      • data transfer within the same region is free

      • cross region data transfer incurs a network fee

RDS Multi Az (Disaster Recovery)

  • synchronous replication

    • standby DB will replicate every single change in the Master synchronously.

  • One DNS name-automatic app failover to standby

  • Increase availability

  • Failover in case of loss of AZ /network /instance / storage failure (standby database will become the new Master)

  • No manual intervention in apps

  • Not used for scaling (No one can read to it. No one can write to it, it's just here as a failover)

  • Note: The Read Replicas be setup as Multi AZ for Disaster Recovery (common ques in exam)

How to make RDS instance go from Single AZ to Multi AZ ? (important)

  • Zero downtime operation(no need to stop the DB)

  • Just click on "modify" for the DB and enable Multi AZ

  • The following happens internally:

    • A snapshot is taken

    • A new standby DB is restored from the snapshot in a new AZ

    • Synchronization is established between the two DB (so the standby database will catch up to the main RDS database)

    and hence multi AZ is setup

RDS - Encryption

  • At rest encryption

    • Possibility to encrypt the master & read replicas with AWS KMS-AES-256 encryption

    • Encryption has to be defined at launch time

    • If the master is not encrypted,the read replicas cannot be encrypted (Important)

    • Transparent Data Encryption(TDE) available for Oracle and SQL Server

  • In-flight encryption

    • SSL certificates to encrypt data to RDS in flight (sent from client to DB)

    • Provide SSL options with trust certificate when connecting to database (this establishes SSL connection)

    • To enforce SSL:

      • PostgreSQL: set rds.force_ssl=1 in the AWS RDS Console (Parameter Groups)

      • MySQL: Within the DB: GRANT USAGE ON . TO'mysqluser'@'%'REQUIRE SSL;

RDS - Encryption Operations

  • Encrypting RDS backups:

    • Snapshots of un-encrypted RDS databases are un-encrypted

    • Snapshots of encrypted RDS databases are encrypted

    • Can copy a snapshot into an encrypted one

  • To encrypt an un-encrypted RDS database:

    • Create a snapshot of the un-encrypted database

    • Copy the snapshot and enable encryption for the snapshot

    • Restore the database from the encrypted snapshot

    • Migrate applications to the new database,and delete the old database

RDS Security-Network & IAM

  • Network Security :

    • RDS DB are deployed within a private subnet, not in a public one (make sure not to expose your DB to www)

    • RDS security works by leveraging security groups attached to the RDS instance

      • same as EC2 instance

      • it controls which IP/security group can communicate with RDS

  • Access Management : (User management)

    • IAM policies help control who can manage AWS RDS(through the RDS API)

      • who can create a DB

      • who can delete a DB

      • who can create a read replica, etc

    • Traditional Username and Password can be used to login into the database

    • IAM-based authentication can be used to login into RDS MySQL & PostgreSQL

  • DB security usually is from within the database.

Connecting to RDS using IAM Authentication :

  • IAM DB authentication works with MySQL and PostgreSQL

  • You don't need a password,just an authentication token obtained through IAM & RDS API calls

  • Auth token has a lifetime of 15 mins

  • How is connection made ?

    • Let's say we have our EC2 security group and our EC2 instance and then we have our MySQL RDS DB in RDS Security Group.

    • EC2 instance with an IAM Role will be able to issue an API call to the RDS service to get back an Authentication token.

    • The token is passed to RDS while connecting to the MySQL database and make sure the connection is encrypted (SSL encryption)

    • It will connect securely to MySQL database.

    EC2  ----- API call ------>   RDS
SG  <---- Auth Token ----- Service
|
|
| SSL Encryption
| Pass Auth Token
V
RDS SG

  • Benefits of this approach:

    • Network in/out must be encrypted using SSL

    • IAM to centrally manage users instead of DB (central type of authentication)

    • Can leverage IAM Roles and EC2 Instance profiles for easy integration

RDS Security-Summary

  • Encryption at rest:

    • Is done only when you first create the DB instance

    • or:unencrypted DB=>snapshot=>copy snapshot as encrypted=>create DB from snapshot

  • Your responsibility:

    • Check the ports/IP/security group inbound rules in DB's SG

    • In-database user creation and permissions or manage through IAM

    • Creatingadatabase with or without public access

    • Ensure parameter groups or DB is configured to only allow SSL connections

  • AWS responsibility:

    • No SSH access

    • No manual DB patching

    • No manual OS patching

    • No way to audit the underlying instance

Amazon Aurora

  • Aurora is a proprietary technology from AWS(not open sourced)

  • Postgres and MySQL are both supported as Aurora DB

    • That is, Aurora is compatible with Postgres and MySQL

    • Aurora DB has compatible drivers and works as if Aurora was a Postgres or MySQL database

  • Aurora is AWS cloud optimized and claims

    • 5x performance improvement over MySQL on RDS

    • over 3x the performance of Postgres on RDS

  • Aurora storage automatically grows in increments of 1OGB, up to 128 TB.

  • Aurora can have 15 replicas while MySQL has 5,and the replication process is faster(sub 10 ms replica lag)

  • Failover in Aurora is instantaneous.

    • It's faster than a failover from Multi AZ on MySQL or RDS

    • It's Cloud Native by default

    • We get High Availability

  • Aurora costs more than RDS(20%more)-but is more efficient

Aurora - High Availability and Read Scaling

  • 6 copies of your data across 3 AZ:

    • 4 copies out of 6 needed for writes

    • 3 copies out of 6 need for reads

    • Self healing with peer-to-peer replication

    • Storage is striped across 100s of volumes

  • One Aurora Instance takes writes(master)

    • It's like Multi AZ on RDS

  • Automated failover for master in less than 30 seconds

  • Master+up to 15 Aurora Read Replicas serve reads

  • Support for Cross Region Replication

  • Shared storage Volume -> Replication+ Self Healing +Auto Expanding

Aurora DB Cluster

  • Writer Endpoint

  • Reader Endpoint

image

Features of Aurora

  • Automatic fail-over

  • Backup and Recovery

  • Isolation and security

  • Industry compliance

  • Push-button scaling

  • Automated Patching with Zero Downtime

  • Advanced Monitoring

  • Routine Maintenance

  • Backtrack:restore data at any point of time without using backups

Aurora Security

  • Similar to RDS because uses the same engines

  • Encryption at rest using KMS

  • Automated backups,snapshots and replicas are also encrypted

  • Encryption in flight using SSL(same process as MySQL or Postgres)

  • Possibility to authenticate using IAM token(same method as RDS)

  • You are responsible for protecting the instance with security groups

  • You can't SSH

Aurora security is the exact same as RDS security.

Aurora Replicas - Auto scaling

  • we can set up replica auto scaling which will add ( according to the load )Amazon Aurora replicas

  • Automatically, the reader endpoint is extended to cover the new replicas.

  • And therefore these new replicas will start receiving some traffic and the reads will be happening in a more distributed fashion bringing down the overall CPU usage.

image

Aurora - Custom Endpoints

image

Aurora - Serverless

  • Automated database instantiation and auto scaling based on actual usage

  • Good for infrequent, intermittent or unpredictable workloads

  • No capacity planning needed

  • Pay per second -> more cost-effective

Aurora - Multi Master

  • When immediate failover for the writer nodes is required for high availability for the writer node.

  • In this case all the nodes in Aurora cluster does read and write (it is not like : if you only have one write node and it falls down it fails then you have to promote a read replica as the new master)

Global Aurora

  • Aurora Cross Region Read Replicas:

    • Useful for disaster recovery

    • Simple to put in place

  • Aurora Global Database(recommended):

    • 1 Primary Region(read/write)

    • Up to 5 secondary(read-only)regions

    • Replication lag is less than 1 sec

    • Up to 16 Read Replicas per secondary region

    • Helps for decreasing latency

    • In case you have a database outage in one region promoting another region for disaster recovery purposes has an RTO (recovery time objective) < 1 min.

Aurora Machine Learning

  • Enables adding ML-based predictions to the applications via SQL

  • Simple,optimized,and secure integration between Aurora and AWS ML services

  • 2 Supported services :

    • Amazon SageMaker(use with any ML model)

    • Amazon Comprehend(for sentiment analysis)

  • You don't need to have ML experience

  • Use cases:

    • fraud detection

    • ads targeting

    • sentiment analysis

    • product recommendations

ElastiCache

  • ElastiCache is used to get managed Redis or Memcached

  • Cache : in-memory DB with really high perforance, low latency

  • Helps reduce load off DB for read intensive workloads (for common queries, the DB need not be queried all the time. Cache can be used to access the frequently accessed records)

  • makes the application stateless by putting the state of the application into Amazon ElastiCache

  • Has the same benifits of RDS : AWS takes care of

    • OS maintenance / patching

    • Optimizations

    • Setup

    • Configuration

    • Monitoring

    • Failure Recovery

    • backups

  • Using ElastiCache invloves heavy application code changes

ElastiCache Solution Architecture

DB Cache

  • Application queries ElastiCache to check for availability

    • if available -> Cache Hit

    • if not available -> Cache Miss get from RDS and store in ElastiCache

  • Helps relieve load in RDS

  • Cache must have an INVALIDATION STRATEGY to make sure only the most current data is used in there

User Session Store

  • storing user session to make the application stateless

  • user logs into any of the application

  • application writes the session data into ElastiCache

  • if user redirected to another instance of the application, then the application can retrieve this session cache directly from ElastiCache, hence the user doesn't need to login again (already logged in)

ElastiCache - Redis vs Memcached (important)

Redis

  • Multi AZ with Auto Failover

  • Read replicas to scale reads and have high availabilty

  • Data durability (due to AOF persistence)

  • Backup and restore features

(Redis is more like RDS)

Memcached

  • Multi node for partitioning of data (sharing)

  • no high availability (replication)

  • Non persistent

  • No backup and Restore

  • Multi threaded architecture

  • Sharding (shard -> node group)

(pure case for distributed)

ElastiCache - Cache Security

  • All caches in ElastiCache :

    • Don't support IAM Authentication

    • IAM policies on ElastiCache are only used for AWS API level security like creating a cache, deleting a cache, etc

  • To authenticate Redis -> use Redis AUTH

    • can set a password/token whne you create a Redis Cluster

    • This is an extra level of security for your cache (on top of security groups)

    • support SSL in flight encryption

  • Memcached

    • supports SASL - based authentication (advanced)

Patterns for ElastiCache

  • Lazy Loading : all the read data is cached, data can become stale in cache

  • Write through : Adds or update data in the cache when written to a DB (no stale data)

  • Session Store : store temporary session data in a cache (using TTL (Time to live) features)

ElastiCache - Redis Use case

  • Gaming leaderboards (computationally complex)

  • Radis has a feature sorted sets which guarantees both

    • uniquness

    • element ordering

  • Each time a new element added, it's ranked in real time then added in correct order

List of ports

Important ports:

  • FTP: 21

  • SSH: 22

  • SFTP: 22 (same as SSH)

  • HTTP: 80

  • HTTPS: 443

vs RDS Databases ports:

  • PostgreSQL: 5432

  • MySQL: 3306

  • Oracle RDS: 1521

  • MSSQL Server: 1433

  • MariaDB: 3306 (same as MySQL)

  • Aurora: 5432 (if PostgreSQL compatible) or 3306 (if MySQL compatible)

NOTE

  • Multi-AZ keeps the same connection string regardless of which database is up.

  • Storing Session Data in ElastiCache is a common pattern to ensuring different EC2 instances can retrieve your user's state if needed.

  • Read Replicas will help as your analytics application can now perform queries against it, and these queries won't impact the main production RDS database.

  • Aurora Global Databases allows you to have an Aurora Replica in another AWS Region, with up to 5 secondary regions. This ensures you have a replica of your database available in another AWS Region if a disaster happens to your main AWS Region.

  • ElastiCache Redis Auth can enhance the security of your ElastiCache Redis Cluster by forcing users to enter a password when they connect

  • Creating Read Replica in different region and enabling Multi AZ on the Read Replica is

    • a disaster recovery strategy for your RDS PostgreSQL database so that in case of a regional outage the database can be quickly made available for both read and write workloads in another AWS Region. The DR database must be highly available.

  • You can not create encrypted Read Replicas from an unencrypted RDS DB instance.

  • You can have 5 Aurora Read Replicas in a single Aurora DB Cluster