How to connect to S3 using an AWS role

There’s different way to create role on AWS depending on your use case:

In this post we’ll explore a few ways Filestash can work with AWS role:

role of type AWS Service. Use case: you have an AWS role you want to attach to an EC2 instance that runs Filestash
role of type Another AWS Account. Use case: you were given a role you want to start using
role of type identity provider
role of type SAML

When role is of type AWS Service

Step 1: if this isn’t already done, create a role for EC2 to call AWS service. This role should be set with the appropriate policy for your use case. For example:

Step 2: attach this role to an EC2 instance. Assuming you have an EC2 instance running, modify the IAM role and add the role defined above, in my case FilestashTestOfTypeAWSService:

Step 3: connect to your EC2 instance via SSH and run the following test using the AWS CLI to ensure everything is configured properly:

[ec2-user@ip-172-31-31-239 ~]$ aws s3 ls
2018-07-18 13:27:57 example-bucket
2018-07-18 13:27:57 example-bucket2
2018-05-14 07:46:08 example-bucket3

Step 4: Install Filestash and run the S3 backend storage without providing neither the access key id nor the secret access key.

When role is of type Another AWS account

The idea here is to ask AWS to generate some temporary credentials through assumeRole

Assumption:

you have a ready to use role. For the sake of this guide, the role arn we will use is arn:aws:iam::194524073938:role/FilestashTestOfTypeAnotherAWSAccount.
The user is authorised to perform sts:AssumeRole on this role

Step 1: from the AWS CLI, we will generate some temporary credentials you can use to switch role through assume-role

~/$ export ROLE=arn:aws:iam::194524073938:role/FilestashTestOfTypeAnotherAWSAccount
~/$ aws sts assume-role --role-arn $ROLE --role-session-name foobar
{
    "Credentials": {
        "AccessKeyId": "ASIAS2SUJZ7JDUCIIUO7",
        "SecretAccessKey": "jrEIGf/VHmxhHeW348ZEPIqFrjx2tjcU+sugJIuJ",
        "SessionToken": "FwoGZXIvYXdzEMX//////////wEaDGFtPJzzBfdYrNhSCSKqAZ1SgufgGCV0RJpumI+rXAX9TM4GyE3ejvsdus7nX2DwS0fOz2ycdL/ejLZjixF8+PUVvbGha3Cpu952n2D4HEFY3irU/GD6d/FYUzXGcdQkoJlUoziFYWln6zqlwG4bCeY6oOspS2uxvYh4o9QH96Yl644dI7FwqgUyXXiFnTEXGilyvcIKeUT5TOBXFp0Bu1sxZl1X3CWrIZxrvCpLLbIRP79iyCcgT1GOKN/c9YkGMi0C4YiJaWJF1OqRRJgHyuUexCEWx/4t8Dx5UBASTwbhozvVzwQzvLrLtumDMUs=",
        "Expiration": "2021-09-12T04:05:03Z"
    },
    "AssumedRoleUser": {
        "AssumedRoleId": "AROAS2SUJZ7JG2EORNIKC:foobar",</br/>         "Arn": "arn:aws:sts::194524073938:assumed-role/FilestashTestOfTypeAnotherAWSAccount/foobar"
    }
}

Step2: you can now connect to S3 using the temporary credentials given by access_key_id, secret_access_key and the session_token:

Note:

Those credentials are temporary and only last 1 hour by default. You might be able to request credentials that last for a different duration depending on the setup of your role.
We can provide customers with plugins tailored for their uses that extends the base S3 and don’t require the user to manually use the AWS CLI.

Common Issue: when trying to assume the role, you might see the following error message:

An error occurred (AccessDenied) when calling the AssumeRole operation:
User: arn:aws:iam::194524073938:user/filestash is not authorized to perform: sts:AssumeRole
on resource: arn:aws:iam::194524073938:role/FilestashTestOfTypeAnotherAWSAccount

To fix it, you need to create a policy that looks like this:

Note on creating the role: if you are creating the role, AWS will ask for the Account ID that can use this role. This account ID is the 12 digit number you can find from:

~/$ aws sts get-caller-identity
{
    "UserId": "AIDAS2SUJZ7JPNCLVIWRT",
    "Account": "194524073938",
    "Arn": "arn:aws:iam::194524073938:user/filestash"
}

When role is of type Identity Provider

The flow is explained in the AWS doc:

~/$ aws sts assume-role-with-web-identity \
    --duration-seconds 3600 \
    --role-session-name "app1" \
    --provider-id "www.amazon.com" \
    --policy-arns "arn:aws:iam::123456789012:policy/q=webidentitydemopolicy1","arn:aws:iam::123456789012:policy/webidentitydemopolicy2" \
    --role-arn arn:aws:iam::123456789012:role/FederatedWebIdentityRole \
    --web-identity-token "Atza%7CIQEBLjAsAhRFiXuWpUXuRvQ9PZL3GMFcYevydwIUFAHZwXZXXXXXXXXJnrulxKDHwy87oGKPznh0D6bEQZTSCzyoCtL_8S07pLpr0zMbn6w1lfVZKNTBdDansFBmtGnIsIapjI6xKR02Yc_2bQ8LZbUXSGm6Ry6_BG7PrtLZtj_dfCTj92xNGed-CrKqjG7nPBjNIL016GGvuS5gSvPRUxWES3VYfm1wl7WTI7jn-Pcb6M-buCgHhFOzTQxod27L9CqnOLio7N3gZAGpsp6n1-AJBOCJckcyXe2c6uD0srOJeZlKUm2eTDVMf8IehDVI0r1QOnTV6KzzAI3OY87Vd_cVMQ"
{
    "AssumedRoleUser": {
        "AssumedRoleId": "AROA3XFRBF535PLBIFPI4:s3-access-example",
        "Arn": "arn:aws:sts::123456789012:assumed-role/xaccounts3access/s3-access-example"
    },
    "Credentials": {
        "SecretAccessKey": "9drTJvcXLB89EXAMPLELB8923FB892xMFI",
        "SessionToken": "AQoXdzELDDY//////////wEaoAK1wvxJY12r2IrDFT2IvAzTCn3zHoZ7YNtpiQLF0MqZye/qwjzP2iEXAMPLEbw/m3hsj8VBTkPORGvr9jM5sgP+w9IZWZnU+LWhmg+a5fDi2oTGUYcdg9uexQ4mtCHIHfi4citgqZTgco40Yqr4lIlo4V2b2Dyauk0eYFNebHtYlFVgAUj+7Indz3LU0aTWk1WKIjHmmMCIoTkyYp/k7kUG7moeEYKSitwQIi6Gjn+nyzM+PtoA3685ixzv0R7i5rjQi0YE0lf1oeie3bDiNHncmzosRM6SFiPzSvp6h/32xQuZsjcypmwsPSDtTPYcs0+YN/8BRi2/IcrxSpnWEXAMPLEXSDFTAQAM6Dl9zR0tXoybnlrZIwMLlMi1Kcgo5OytwU=,"
        "Expiration": "2016-03-15T00:05:07Z",
        "AccessKeyId": "ASIAJEXAMPLEXEG2JICEA"
    }
}

The given credentials can then be used like we did in this part 2

Note: Filestash can leverage this flow via a plugin that extends the base s3 plugin

When role is of type SAML

The flow is explained in the AWS doc:

~/$ aws sts assume-role-with-saml \
    --role-arn arn:aws:iam::123456789012:role/TestSaml \
    --principal-arn arn:aws:iam::123456789012:saml-provider/SAML-test \
    --saml-assertion "VERYLONGENCODEDASSERTIONEXAMPLExzYW1sOkF1ZGllbmNlPmJsYW5rPC9zYW1sOkF1ZGllbmNlPjwvc2FtbDpBdWRpZW5jZVJlc3RyaWN0aW9uPjwvc2FtbDpDb25kaXRpb25zPjxzYW1sOlN1YmplY3Q+PHNhbWw6TmFtZUlEIEZvcm1hdD0idXJuOm9hc2lzOm5hbWVzOnRjOlNBTUw6Mi4wOm5hbWVpZC1mb3JtYXQ6dHJhbnNpZW50Ij5TYW1sRXhhbXBsZTwvc2FtbDpOYW1lSUQ+PHNhbWw6U3ViamVjdENvbmZpcm1hdGlvbiBNZXRob2Q9InVybjpvYXNpczpuYW1lczp0YzpTQU1MOjIuMDpjbTpiZWFyZXIiPjxzYW1sOlN1YmplY3RDb25maXJtYXRpb25EYXRhIE5vdE9uT3JBZnRlcj0iMjAxOS0xMS0wMVQyMDoyNTowNS4xNDVaIiBSZWNpcGllbnQ9Imh0dHBzOi8vc2lnbmluLmF3cy5hbWF6b24uY29tL3NhbWwiLz48L3NhbWw6U3ViamVjdENvbmZpcm1hdGlvbj48L3NhbWw6U3ViamVjdD48c2FtbDpBdXRoblN0YXRlbWVudCBBdXRoPD94bWwgdmpSZXNwb25zZT4="
{
    "Issuer": "https://integ.example.com/idp/shibboleth",
    "AssumedRoleUser": {
        "Arn": "arn:aws:sts::123456789012:assumed-role/TestSaml",
        "AssumedRoleId": "ARO456EXAMPLE789:TestSaml"
    },
    "Credentials": {
        "AccessKeyId": "ASIAV3ZUEFP6EXAMPLE",
        "SecretAccessKey": "8P+SQvWIuLnKhh8d++jpw0nNmQRBZvNEXAMPLEKEY",
        "SessionToken": "IQoJb3JpZ2luX2VjEOz////////////////////wEXAMPLEtMSJHMEUCIDoKK3JH9uGQE1z0sINr5M4jk+Na8KHDcCYRVjJCZEvOAiEA3OvJGtw1EcViOleS2vhs8VdCKFJQWPQrmGdeehM4IC1NtBmUpp2wUE8phUZampKsburEDy0KPkyQDYwT7WZ0wq5VSXDvp75YU9HFvlRd8Tx6q6fE8YQcHNVXAkiY9q6d+xo0rKwT38xVqr7ZD0u0iPPkUL64lIZbqBAz+scqKmlzm8FDrypNC9Yjc8fPOLn9FX9KSYvKTr4rvx3iSIlTJabIQwj2ICCR/oLxBA==",
        "Expiration": "2019-11-01T20:26:47Z"
    },
        "Audience": "https://signin.aws.amazon.com/saml",
        "SubjectType": "transient",
    "PackedPolicySize": "6",
        "NameQualifier": "SbdGOnUkh1i4+EXAMPLExL/jEvs=",
        "Subject": "SamlExample"
}

The given credentials can then be used like we did in this part 2

Note: Filestash can leverage this flow via a plugin that extends the base s3 plugin

ref:

https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRoleWithSAML.html
https://docs.aws.amazon.com/cli/latest/reference/sts/assume-role-with-saml.html