AWS Cross-Account S3 Copy

I recently had an issue when trying to copy files from an S3 bucket in an AWS account to another S3 bucket in a different account. Going through various AWS Documentation and StackOverflow issues that people were facing on the above, I decided to put this together so it could help those having similar issues. There are different options you can use to achieve the same result, but here is what I did(with the help of 2 other colleagues) to copy the files across from source account into destination account.


Requirements/Important Points

  1. An IAM role needs to be in source account with all the permissions needed to access the S3 bucket in both accounts and the destination account needs to be added in as a Trusted entity

  2. An IAM User needs to be created in destination account to assume the role above to read from source bucket and write to destination bucket

  3. IAM policy to be attached to the above user that has Assume Role permissions

  4. An S3 bucket in destination account with a bucket policy that allows the IAM role in source account to write to the bucket. The permissions given needs to be the same as the ones in the IAM role in source account

Process

1.In the source account, create an IAM role with following permissions. This will allow the role to read from source bucket and write to the dest. bucket

{

"Version": "2012-10-17",

"Statement": [

{

"Sid": "AllowReadAccess",

"Effect": "Allow",

"Action": [

"s3:GetObject",

"s3:ListBucket",

"s3:HeadBucket",

"s3:HeadObject"

],

"Resource": [

"arn:aws:s3:::<source_bucket>/*",

"arn:aws:s3:::<source_bucket>"

]

},

{

"Sid": "AllowWriteAccess",

"Effect": "Allow",

"Action": [

"s3:PutObject",

"s3:GetObject",

"s3:ListBucket",

"s3:HeadBucket",

"s3:HeadObject",

"s3:PutObjectAcl",

"s3:DeleteObject",

"s3:GetBucketLocation",

"s3:ListBucketMultipartUploads",

"s3:AbortMultipartUpload",

"s3:ListMultipartUploadParts"

],

"Resource": [

"arn:aws:s3:::<dest_bucket>/*",

"arn:aws:s3:::<dest_bucket>"

]

}

]

}


2. Once IAM role is created, click on Trust relationships and Edit trust relationship and add the destination account number to be trusted by the role. This is how it should look

{

"Version": "2012-10-17",

"Statement": [

{

"Effect": "Allow",

"Principal": {

"AWS": [

"arn:aws:iam::<destination_acct_number>:root",

"arn:aws:iam::<test_acct_number>:root"(if needed)

]

},

"Action": "sts:AssumeRole",

"Condition": {}

}

]

}


3. On the destination account, create an IAM user to assume role above and store the access keys in your ~/.aws/credentials file


4. Create an IAM Policy with Assume Role permissions and attach it to the user above

{

"Version": "2012-10-17",

"Statement": [

{

"Sid": "AllowAssumeRole",

"Effect": "Allow",

"Action": "sts:AssumeRole",

"Resource": "arn:aws:iam::<source_acct_number>:role/<source_acct_role>"

}

]

}


5. In the destination S3 bucket, input the following bucket policy to allow the IAM role in source account to write into the bucket. Need the permissions for destination bucket access in IAM role in source account to be the same

{

"Version": "2012-10-17",

"Statement": [

{

"Effect": "Allow",

"Principal": {

"AWS": "arn:aws:iam::<source_acct_number>:role/<source_role>"

},

"Action": [

"s3:PutObject",

"s3:PutObjectAcl",

"s3:GetObject",

"s3:GetObjectAcl",

"s3:DeleteObject",

"s3:GetBucketLocation",

"s3:ListBucketMultipartUploads",

"s3:AbortMultipartUpload",

"s3:ListMultipartUploadParts",

"s3:ListBucket"

],

"Resource": [

"arn:aws:s3:::<dest_bucket>",

"arn:aws:s3:::<dest_bucket>/*"

]

}

]

}


6. In your ~/.aws/config file, input the following. The profile name will be used as profile to run awscli command and the source_profile references the IAM user creds that was placed earlier in credentials file


[profile <profile_name>]

role_arn = arn:aws:iam::<source_acct_number>:role/<source_acct_role>

source_profile = <your IAM user>


7. Run the following commands to test in your account everything works as it is meant to


aws --profile <profile_name> s3 ls s3://<source_bucket_name>/

mkdir <dir_name> && cd <dir_name>

aws s3 --profile <profile_name> sync s3://<source_bucket_name/ .


8. Once everything is working as expected, run sync command on your terminal to copy files across from source bucket to destination bucket


aws s3 --profile <profile_name> sync s3://<source_bucket_name> s3://<dest_bucket_name>


So hopefully it works for you as well!!! Remember there are a lot of options out there with different ways of carrying out above task. Enjoy!!! Would appreciate any comments if this worked for you or how you would go about doing it.



107 views0 comments

Recent Posts

See All