Simple Storage Service (S3)

1. Overview

• Amazon S3 is one of the main building blocks of AWS
• It's advertised as "infinitely scaling" storage
• It's widely popular and deserves its own section
• Maybe websites use Amazon S3 as a backbone
• Many AWS services use Amazone S3 as an integration as well

2. Buckets

• Amazon S3 allows people to store objects (files) in "buckets" (directories)
• Buckets must have a globally unique name
• Buckets are defined at the region level
• Naming convention
  • No uppercase
  • No underscore
  • 3-63 characters long
  • Not an IP
  • Must start with a lowercase letter or number

3. Objects

• Objects (files) have a Key
• The key is the FULL path
  • s3://my-bucket/my_file.txt
  • s3://my-bucket/my_folder/another_folder/my_file.txt
• The key is composed of prefix + object name
  • s3://my-bucket/my_folder/another_folder/my_file.txt
    • prefix: my_folder/another_folder
    • object name: my_file.txt
• There's no concept of "directories" within buckets. (although the UI will trick you to think otherwise)
• Just keys with very long names that contain slashes ("/")
• Object values are the content of the body:
  • Max Object Size is 5TB
  • If uploading more than 5GB, must use "multi-part upload"
• Metadata (list of text key/value pairs - system or user metadata)
• Tags (Unicode key/value pair - up to 10) - useful for security/lifecycle
• Version ID (if versioning is enabled)

4. Versioning

• You can version your files in Amazon S3
• It is enabled at the bucket level
• The same key overwrite will increment the "version": 1, 2, 3, and so on
• It is best practice to version your buckets
  • Protect against unintended deletes (ability to restore a version)
  • Easy roll back to the previous version
• Any file that is not versioned prior to enabling versioning will have version "null"
• Suspending versioning does not delete the previous versions

5. Encryption for Objects

It's important to understand which ones are adapted to which situation

5.1 SSE-S3

• Encrypt S3 objects using keys handled & managed by AWS.
• The object is encrypted server-side
• AES-256 encryption type
• Must set header: "x-amz-server-side-encryption": :"AES256"

5.2 SSE-KMS (Key Management Service)

• Leverage AWS Key Management Service to manage encryption keys.
• KMS Advantages: user control + audit trail
• The object is encrypted server-side
• Must set header: "x-amz-server-side-encryption": "was:kms"

5.3 SSE-C

• When you want to manage your own encryption keys.
• Amazon S3 does not store the encryption key you provide
• HTTPS must be used
• The encryption key must be provided in HTTP headers, for every HTTP request made

5.4 Client-Side Encryption

• Client library such as the Amazon S3 Encryption Client
• Clients must encrypt data themselves before sending to S3
• Clients must decrypt data themselves when retrieving from S3
• The customer fully manages the keys and encryption cycle

6. S3 Security

6.1 User-based

IAM policies - which API calls should be allowed for a specific user from the IAM console

6.2 Resource-based

• Bucket Policies - bucket wide rules from the S3 console - allow cross-account
• Object Access Control List (ACL) - finer grain
• Bucket Access Control List (ACL) - less common

6.3 Example

An IAM principal can access an S3 object if the user IAM permissions allow it OR the resource policy ALLOWS it AND there's no explicit DENY

7. S3 Bucket Policies

7.1 JSON based policies

• Resources: buckets and objects
• Actions: Set of API to allow or deny
• Effect: Allow/ Deny
• Principal: The account or user to apply the policy to

7.2 Common Use cases S3 bucket for policy to:

• Grant public access to the bucket
• Force objects to be encrypted at upload
• Grant access to another account (Cross Account)

7.3 Bucket settings for Block Public Access

• Block public access to buckets and objects granted through
  • new access control lists (ACLs)
  • any access control lists (ACLs)
  • new public bucket or access point polices
• Block public and cross-account access to buckets and objects through any public bucket or access point policies
• These settings were created to prevent company data leaks
• If you know your bucket should never be public, leave these on
• Can be set at the account level

7.4 Other S3 Security

7.4.1 Networking

Support VPC Endpoints (for instances in VPC without www internet)

7.4.2 Logging and Audit

• S3 Access Logs can be stored in other S3 buckets
• API calls can be logged in AWS CloudTrail

7.4.3 User Security

• MFA Delete: MFA (Multi-Factor Authentication) can be required in versioned buckets to delete objects
• Pre-Signed URLs: URLs that are valid only for a limited time (ex: premium video service for logged in users)

8. S3 Websites

• S3 can host static websites and have them accessible on the www
• The website URL will be:
  • <bucket-name>.s3-website<AWS-region>.amazonneaws.com
  • <bucket-name>.s3-website.<AWS-region>.amazonaws.com
• If you get a 403 (Forbidden) error, make sure the bucket policy allows public reads

9. S3 CORS

9.1 Origin

• An origin is a scheme (protocol), host(domain), and port
  • E.g.: https://www.example.com (implied port is 443 for HTTPS, 80 for HTTP)
• CORS means Cross-Origin Resource Sharing
  • Web Browser based mechanism to allow requests to other origins while visiting the main origin
• Same-origin: HTTP://example.com/app1 & HTTP://example.com/app2
• Different origins: HTTP://www.example.com & HTTP://other.example.com
• The requests won't be fulfilled unless the other origin allows for the requests, using CORS Headers (ex: Access-Control-Allow-Origin)

9.1.1 Example CORS

The first visit server using the browser is the origin. And there is a second webserver called a cross-origin

9.2 S3 CORS

• If a client does a cross-origin request on our S3 bucket, we need to enable the correct CORS headers
• You can allow for a specific origin or for * (all origins)

10. Consistency Model

• Read after write consistency for PUTS of new objects
  • As soon as a new object is written, we can retrieve it. ex: (PUT 200 => GET 200)
• This is true, except if we did a GET before to see if the object existed
  • ex: (GET 404 => PUT 200 => GET 404) -eventually consistent
• Eventual Consistency for DELETES and PUTS of existing objects
  • If we read an object after updating, we might get the older version
    • ex: (PUT 200 => PUT 200 => GET 200 (might be older version))
  • If we delete an object, we might still be able to retrieve it for a short time
    • ex: (DELETE 200 => GET 200)
• There's no way to request "strong consistency"

11. Storage Class

12. Reference

aws.amazon.com/s3/

www.sankalpjonna.com/posts/hosting-your-entire-web-application-using-s3-cloudfront