s3.txt

_________________________________________________________________________________

S3 : Simple Storage Service :
-
Object based storage :  We cannot install/run any applications/os. : Dropbox / GDrive 
--> Designed to flat files.

Block based storage : Designed to run OS, We can install/run applications.. : EBS, Instance store volumes. : EC2 
Storage over the network : Storage runs over the network and it can be mounted to multiple devices. : EFS, FSx : EC2

--> Designed to store objects. We can store any flat files.
--> We store data in BUCKETS.
--> Bucket names should be unique across the globe.
--> Defualtly, we can create 100 buckets, This is a soft limit. use "Service Quota" to incease this limit.
--> We can store unlimited data in a bucket.
--> No Pre-provisioning required.
--> Bucket naming limitations.
	--> Min char 3, Max 63 char length
	--> Bucket name should be in small , and numeric
	--> Bucket name should not start with .
	--> Should not end with . , No adjesent ..
	--> Bucket name should not resemble IP address format. (192.168.100.1)

--> We can upload objects to this buckets.
--> Min obj size : 0 bytes.. Max Obj size: 5 TB
--> S3 is now required region selection. You first need to choose a region, where you want to create your s3 bucket.
--> While creating bucket, we need to choose the region. Data physically resides at this regions AZs. 

https://s3.ap-south-1.amazonaws.com/bucketname/objectname (standard URL)
https://bucketname.s3.ap-south-1.amazonaws.com/objectname (virtual path)
https://bucketname.s3.amazonaws.com/objectname (virtual path)

Virtual paths won't work if we have . in bucket names.

To make an object public : 

1. Block Public Access settings for this account : Account Level Blocking
2. Block public access (bucket settings) : Bucket level Blocking
--> We should make an object public, then only Object URLs works.
** Object Ownership : ACLs enabled
3. Make object public. (Select object --> actions --> last option --> make public using acl)

S3 Free Tier ELigibility : 
5 GB Standard Storage
--> 2000 PUT (Upload) / 20,000 GET (Download)
_______________________________________________________________________________

Task :  In Your Personal Accounts:
--> Enable "Free tier alerts". Login using root user/iam user with admin permissions..
Preferences --> Billing preferences --> Receive Free Tier Usage Alerts --> Enable and enter valid email address.

Task 2 : Create an s3 bucket, Upload an object, Access the Object as an IAM user.

Make it public and access same object using public URL.


==============================================================================================

D: 27/06/24

S3 Storage Classes : 

--> S3 Standard : Defualt storage class for all the data we upload to s3. Data will be available immedeatly.
Designed to store : Frequently accessed data. 
Availability : 99.99%,  Durability: 99.999999999 (11 9's) 
Data replicates across : >= 3 AZs

--> Standard-IA : Data will be available immedeatly. Designed to store : infrequently accessed data
Avai : 99.9%, Durability : 99.999999999 (11 9's) 
Data replicates across : >= 3 AZs

--> One Zone-IA : Data will be available immedeatly. Designed to store : infrequently accessed data. Store only non-critical data. Cost less than 40% compared to Standard-IA.
Avai : 99.5 %, Durability : 99.999999999 (11 9's) 
Data replicates across : 1 AZs

--> Glacier Flexible Retrieval (Formerly Glacier) : Long-term data archiving with retrieval times ranging from minutes to hours. Data won't be available immedeatly. We need to initiate a data restration to view the data. Restoration takes (1 min - 5 Hrs)
Avai : 99.99 %, Durability : 99.999999999 (11 9's) 
Data replicates across : >= 3 AZs

--> Glacier Instant Retrieval : Long-lived archive data accessed once a quarter with instant retrieval in milliseconds. 

--> Glacier Deep Archieve : Long-term data archiving with retrieval times ranging from 12 hrs. Data won't be available immedeatly. We need to initiate a data restration to view the data. Restoration takes (5 - 12 Hrs)
Avai : 99.99 %, Durability : 99.999999999 (11 9's) 
Data replicates across : >= 3 AZs
..

S3 Intelligent Tier : If we have unknown access pattern, we can choose. 
Avai : 99.9 %, Durability : 99.999999999 (11 9's) 
Data replicates across : >= 3 AZs

RRS : Redused Redundancy Storage : Not Recommanded : Less durability.. 
Durability : 99.99%, As we have less durability we have high chances to loss the data when compared to another storage classes.


S3 Pricing : 
--> How much data we are storing.
--> How many GET/PUT operations (uploads/downloaded object count)
--> Data retrivals

S3 FREE TIER :
--> 5 GB Standard Storage
--> 2000 PUT Operations (upload)
--> 20,000 GET operation (open/download)

Glacier data retrival : 
Bulk retrieval : Typically within 5-12 hours.
Standard retrieval : Typically within 3-5 hours.
Expedited retrieval : Typically within 1-5 minutes when retrieving less than 250 MB.

** Only S3 standard comes under free tier eligibility.


------------

--> versioning helps us to get our data to correct state, AWS maintains multiple copies of our data.
--> If we accedentally overwrite any data or delete any file, we can get it to correct state.

If you are typing "Delete" ==> you'll get delete Marker. 
If you are typing "perm Delete" ==> You wont get Delete marker.

Delete the "Delete Marker" to get your object back.
_________

Version : AWS maintains multiple copies in s3 bucket. 
--> Defaultly versioning will be in Suspended state.

--> When versioning is enabled on bucket, If we delete an object

Versioning : HIDE
--> S3 will show most recent uploaded object only / Current version objects.
--> We will get a Delete Marker. 
--> TO get object back / undo delete, Set Version to SHOW, Delete the delete marker.

Versioning : SHOW
--> If we delete an object, it deletes permanently.

======================================================================================================

D: 28/06/2024

Life Cycle Management Rule : 

--> We can apply Rule to entire bucket
--> We can limit rule to a prefix (Folder)
--> We can limit rule to objects by specifing a "Tag"


S3 Standard --> StandardIA/IT --> Glacier --> Delete
S3 Standard --> Glacier --> Delete
S3 Standard --> Delete


===========

S3 Static Website Hostic : 

--> We need to make data public read, then only our website will deliver. 
--> Purchased Domain name should be same as Bucket name. 

http://learnaws.co.in.s3-website.ap-south-1.amazonaws.com  ==> learnaws.co.in

http status codes :
2XX : OK/Success
3XX : Redirection
4XX : Client sider error
5XX : Server sider error

https://www.free-css.com/free-css-templates

Task : Configure life cycle management rule to transit data from S3Standard to "Delete" after 2 days.

Task : Download a template from Internet. Deliver the webiste using s3 - static webiste feature.

======================================================================================================

D: 01/07/2024

CRR / SRR : 
--> Source bucket and destination bucket must be enabled with versioning. 
--> Existing objects will not replicate to the destination bucket.
--> All future / subsequent uploads will replicate to destination bucket. 

RTC : Replication time control : Data replicates to destination bucket with in 15 Min. (99.99% of the data) : COST US

___

Data transfer from One region to another region "cost us". 
Data transfer within a region won't cost us anything.
_______________________

aws s3 sync s3://sourcebucket s3://Targetpath		==> 12:00AM   (task scheduler / cron job)

aws s3 sync localfolder s3://s3backupbucket

_______________________

Event notifications : 

--> SNS : Simple Notification Service : 1000 EMAILs/Month Free Tier eligibility
--> SQS : Simple Queue Service
--> Lambda 

--> Make sure you edit the SNS topic's "Access Policy to everyone".

______________________

Task 1 : Create a bucket in N.Virginia, Configure Versioning and test it. 

Task 2 :  Create another bucker in N Virginia region only.. configure same region replication between these buckets.

Task 3 : Configure to receive alerts to your email id when any Object deleted from an S3 bucket.

====================================================================================================

D: 02/07/2024

--> In-Transit Encryption : 
	SSL (Secure socket layer) / TLS (Transport layer secure)

--> SSE (Server Side Encryption) / At-rest : 

SSE-S3 : S3 platform generates and manages the key material. whoever have valid access on S3 platform can decrypt/view the data. (An encryption key that Amazon S3 creates, manages, and uses for you). AES-256
--> We need to use this to make any data public.

SSE-KMS : Key Management Service : 
	KMS (AWS managed keys) (aws/s3) DMK (Default Master Key) : KMS Service generates and manages the key material. whoever have valid access on S3 platform can decrypt/view the data. We cannot delete these keys. 
--> We cannot make data public using this keys.

	 CMK (Customer managed keys) : KMS Service generates and manages the keymaterial.
Along with S3 platform, the IAM user need to have valid permissions on ENCRYPTION KEY also. 
NO FREE TIER ELIGIBILITY. COST US.

	SSE-C (Customer Provided Key): Customer generates and upload the key material to KMS and KMS uses this key material to encrypt/decrypt the data. Customer can change the key material at any time. 
NO FREE TIER ELIGIBILITY. COST US



--> CSE (Client Side Encryption) : 
Before uploading data to s3 platform, we can use our own software/applications to encrypt the data. That encryptrd data we can send to s3. AWS is not completely responsible for this mechanism.

Encryption key generation : 
--> Choose key type : Symmetric /Asymetric : Symetric as s3 supports
--> Choose key Administrator
--> Choose Key Users
--> Review and create


SSE-S3 : S3 created key and key material, No Addl Permisisons on Encry key, Only S3 access is fine.
SSE-KMS DMK : KMS created key and key material, No Addl Permisisons on Encry key, Only S3 access is fine.
SSE-KMS CMK : Customer Creates key and KMS manages key material, Along with S3 access, User/role should have "Key Usage" permissions.
SSE-KMS Customer provided : Customer Create key and Customer provide key material, Along with S3 access, User/role should have "Key Usage" permissions.

---------------------------------------------

https://www.youtube.com/watch?v=r17Z75TqjU0&t=548s

Task : Try to access s3 using 3rd party applications. (S3 browser, Cyberduck, Cloudberry explorer, winscp)
Steps : Create an IAM user provide s3 readonly/fullaccess, generate Accesskey and SecretKey. Use this with 3rd party tool.

Task 2 : Enable Encryption on s3 bucket and test it. (SSE-S3/AWS Managed key KMS-DMK)

================================================================================================

D: 03/07/2024


Requirement : I want an IAM user, who can perform everything on s3 platform. But i dont want to allow this user to "DeleteObject" from one specific s3 bucket.

Requi: IAM user.. S3FullAccess, 


Deny upload object operation on a particular bucket. : Bucket policy

Bucket ARN : arn:aws:s3:::avinash.bucket
IAM User ARN / Principal : arn:aws:iam::501170964283:user/s3user
Effect : Allow / Deny : Deny
Action / Operation : PutObject, DeleteBucket


Object level / Inside bucket possible operation : arn:aws:s3:::avinash.bucket/*
Bucket level : arn:aws:s3:::avinash.bucket
if you dont know bucket level/object level ; give both
(arn:aws:s3:::BUCKETNAME,arn:aws:s3:::BUCKETNAME/*)


arn:aws:s3:::0105.avinash  Bucket : delete bucket / enable verisoni / enable policy/ acl
arn:aws:s3:::0105.avinash/*  Object : delete object/ uplaod obj/. over obj

arn:aws:s3:::0105.avinash,arn:aws:s3:::0105.avinash/*

--------------------

Task : Create an IAM user with "S3FullAccess". 
We need to restrict/DENY this user to perform "DeleteObject" opertion from a specific bucket (bucketname)


Task 2 : create a bucket policy, that allow "one spcific IAM USer", to perform anything on this bucket. Remaining all users should not able to perform anything on this bucket.
--> Refer the policy document

**
Task 3 : Create a bucket, Allocate this bucket to only one specific IAM user. Allow him to perform "GetObject", "PutObject" and "DeleteObject". This User should not able to view all the bucket we have in s3 platform.
--> Only with the help of a bucket URL, this guy should able to view.   --> "Perform using IAM POLICY"

** https://repost.aws/knowledge-center/s3-console-access-certain-bucket


Task 4 : Make a bucket public for eveyone using "BUCKET POLICY".
--> Refer the shared policy document.


=================================================================================


What is Presign URL :  Short lived urls. 
--> Without making bucket/object public, we can share an s3 object with presign url, this URL works only for sometime.


-------------------------

Object Lock : 

--> Versioning must be enabled.

Permanently allows objects in this bucket to be locked. Additional Object Lock configuration is required in bucket details after bucket creation to protect objects in this bucket from being deleted or overwritten.

Governance : Users with specific IAM permissions can overwrite or delete protected object versions during the retention period.

Compliance : No users can overwrite or delete protected object versions during the retention period.


========================================================================

D: 04/07/2024

Performance : 
3500 PUT Object per second per prefix : upload
5500 GET Object per second per prefix : Download

--> maintain/Increase the uniqueness for the object names.
--> Use more prefixes (folders)


My Req : i want 10,000 Uploads / Sec.. 

Bucket : 5500 uploads/sec.. 
--> folder1 --> 5500
--> folder2 --> 5500
--> folder3 --> 5500

-----

Cloudwatch : Monitoring

Monitoring S3 Bucket : 

Free Metrics:
Total number of objects
Total bucket size

We do have an option to enable paid metrics.

-----

Storage Class Analysis : Analyze storage access patterns to help you decide when to transition objects to the appropriate storage class.

_______

Transfer acceleration : Uses Edge locations for data transfer. (COST US)
S3 Transfer Acceleration is not supported for buckets with periods (.) in their names

https://s3-accelerate-speedtest.s3-accelerate.amazonaws.com/en/accelerate-speed-comparsion.html?region=ap-south-1&origBucketName=avinashspeedtest

-------------------

Server access logging : enable logging on bucket.

https://docs.aws.amazon.com/AmazonS3/latest/userguide/LogFormat.html

Cloudtrail : Data events : It allow us to log all s3 activities (Current buckets and future buckets).

----

Requester pays : the requester pays for requests and data transfer costs, and anonymous access to this bucket is disabled. Instead of bucket owner data requester need to pay for the data transfer & request.
--> Anonomus access disables once we enable this feature.

---

Cross-origin resource sharing (CORS)
The CORS configuration, written in JSON, defines a way for client web applications that are loaded in one domain to interact with resources in a different domain.

CORS : https://www.youtube.com/watch?v=YrfDLrSVxQ8&t=4s

==================================================================

If we have larger file to uploade to s3 paltform, We can initiate the "Multi Part Upload". 

How to Perform multiPart upload to S3 : 
https://www.youtube.com/watch?v=ZJaNTLKz334

---
Pricing : https://calculator.aws/
Free Tier : https://aws.amazon.com/free

========================================================================

S3 platform Consistency mechanism: 

For Put of new object, "read after write consistency". (If upload success, we can start accessing the data without any delay)
Eventual consistenacy For overwrite an object / delete an object. (After we delete the object, Some of the request may deliver the content (in some milli seconds)) (You have uploaded object with latest data and trying to access, you have chances to get old/previous data object)

_________________

AWS Snow Family : 

--> AWS snowcone : 8 TB
--> AWS Snowball : 100 TB
--> AWS Snowball edge : 40 - 80 TB
--> AWS Snow mobile : PB scale container

1 pb data, 1 gbps internet line with 80%.. 20-25 yrs


** Data transfering from Snow devices to S3 is free.
** Data from s3 to outer world/snow device, will cost us.
** We have to pay /job cost (apprx, 100$-150$)
** If we complete job within 10 daya, no addl cost, If extended, we need to pay /day extra.

___

AWS Direct Connect : Dedicated connection between on-premise/local environment to AWS environment. 3-4 weeks time required. We need to choose the required bandwidth. 
--> Airtel, jio, sify

Site-to-site VPN : Happens over the internet.

=================================================================================

Certification Topics : 
--> Storage classes
--> Life Cycle management rule
--> replication
--> encryption
--> versioning
--> Bucket policy
--> Data Migration (Snow Family)
--> Data consisntency
--> Data performance


Interview:
--> Storage classes
--> Life Cycle management rule
--> replication
--> encryption
--> versioning
--> Bucket policy