Use AWS roles instead of service users

[yuvipanda]

We were creating AWS service users, giving them keys
and then checking those keys in with git-crypt. This isn't
good security practice. We should be creating roles with
minimal permissions instead. These roles can then be 'assumed'
by different entities - an EC2 instance running GitHub actions,
a local user on a computer, etc. This also removes need to
managed EC2 access credentials in a repo - dangerous, and bothersome
to rotate.

This needs corresponding changes in hubploy to use assumed
roles before it can work.

[cslocum]

aws-creds shouldn't be needed anymore right?

[cslocum]

I assume map_users should be removed here?

https://github.com/pangeo-data/terraform-deploy/blob/master/aws/your-cluster.tfvars.template#L6

[yuvipanda]

For EC2 instances, I'm guessing we need another role that can assume the ecr and eks roles. Then EC2 can use that role, and we can run hubploy there automatically?

[yuvipanda]

@cslocum yeah I think we should remove map_users. We might need to add map_roles (https://docs.aws.amazon.com/eks/latest/userguide/add-user-role.html) though - currently only the user that created the cluster can access it.

[yuvipanda]

I think aws-creds is unrelated - it is creating roles that can be used to run this terraform module itself. Equivalent to https://github.com/yuvipanda/aws-codecommit-secret#setting-up-iam-role-to-run-terraform

[yuvipanda]

Needs more docs! Overall the right thing to do, but needs docs on:

1. How do I give users permission to do deploys from computer?
2. How do I make credentials for my CI to deploy?

We should find ways to answer these questions before merging this.

[salvis2]

I like this! I think assuming roles has been tricky for us in the past but if we can streamline it with Terraform it should be much easier.

aws-creds should be changed, really. It had some Terraform blocks commented out as a starting point for assigning AWS policies to a user or a role. We can make sure the role setup in that file is correct and maybe uncomment those blocks as a suggested default.

After reading through the commits, it seems like aws/iam.tf is replacing the content in aws-creds. If you want to leave those options open, we could leave aws-creds. If we want to delete aws-creds, moving the data source in aws-creds/iam.tf would be helpful, since that is the minimum policy set for the terraform-aws-eks module.

[salvis2]

For your last comment:

1. How do I give users permission to do deploys from computer?

What do you mean by this? How to get other users permissions to mess with the cluster? Allow other users on your AWS account to deploy clusters in this manner? Would you not just make sure they are in the IAM group you are making?

2. How do I make credentials for my CI to deploy?

That might be a problem. There wouldn't be credentials for an IAM role, would there? How do we allow a CI machine that we don't always own access into our AWS account? If there is something we can give it, are we okay with git-crypting something into the repo as long as it isn't specific user's access keys?

No answers yet on my end, just hoping to drop some leading questions and generate discussion.

[salvis2]

This is the only thing I can think to change immediately. Otherwise, this looks great!

[yuvipanda]

Thanks for the review, @salvis2!

The stuff in iam.tf is not the same as in aws-creds/. aws-creds/ was enough stuff to run the whole terraform stuff in aws/, while iam.tf is just enough perms to run hubploy. A lot more restricted. @super-cob is working on cleaning up aws-creds/ in #34, so that should help.

For CI, there are two options:

1. Use a self-hosted runner in your own infra, an EC2 instance that can assume appropriate role. This is the more secure option
2. Create a service user, put it in appropriate group, and store access key in secret in your CI provider - not via git-crypt.

This also goes well with getting rid of git-crypt, the most secure version of that includes using codecommit.

[yuvipanda]

@salvis2 done!

[salvis2]

The stuff in iam.tf is not the same as in aws-creds/. aws-creds/ was enough stuff to run the whole terraform stuff in aws/, while iam.tf is just enough perms to run hubploy. A lot more restricted. @super-cob is working on cleaning up aws-creds/ in #34, so that should help.

Gotcha. Finding the minimum permission set for hubploy makes sense.

For CI, there are two options:

1. Use a self-hosted runner in your own infra, an EC2 instance that can assume appropriate role. This is the more secure option
2. Create a service user, put it in appropriate group, and store access key in secret in your CI provider - not via git-crypt.

This also goes well with getting rid of git-crypt, the most secure version of that includes using codecommit.

Ah I see.

For 1, I can kind of see how it would work but it seems like it would have poor integration and / or always have to be running. If there's a config for this, I'd love to see it.

For 2, this method is clearer to me. You could just put keys into CircleCI or GitHub Secrets manually. Makes sense.

These both look like good paths forward!

[yuvipanda]

@salvis2 for 1 I have had people use https://help.github.com/en/actions/hosting-your-own-runners/about-self-hosted-runners without too much trouble. But only in extreme cases haha.

How do you feel about hitting the merge button?