At Anghami, the leading music platform of the MENA region, we use Amazon Web Services — AWS. Our infrastructure has grown over the years to a point were we run on top of hundreds of EC2 instances, dozens of RDS databases, dozens of ElastiCache servers, etc… The list grows to cover most of AWS services.
Will share with you how we moved our workload from one account, more than 5 years old, into a multi-account setup where each team has its own private account. The decision was not easy, but we stepped into it hoping to:
- put more order within our infrastructure, giving each team a private account was no doubt a step in this direction.
- align with AWS guidelines.
- increase security within our growing infrastructure.
Not knowing what to expect along the road and not to put more delays into the project, we decided to go for centralized billing only. Once the project went live, we confirmed this decision. Enabling all features setup of AWS Organizations was without real benefit for us. Centralized billing alone fit well our workflow and team structure.
The multi-account setup brought us many advantages:
Isolation of teams
Each team has its own account and thus its own VPC, subnets, security groups. Each team has a separate pool of users, policies and roles. This gave them maximum control over their resources without affecting other teams or production workload.
Cost control
It was a bit surprising but yea, giving each team full control over their private account drove them to be more concerned about their budgeting and cost control strategies.
Clearer billing
When in single account, tagging was never enough to have clear view of cost per team. Grey areas in billing were around the corner all the time. With multi-account architecture, we have clearer billing per team. Coupled with tagging, it’s giving us better cost visibility and control.
Secure production environment
Giving each team a separate account, left the production environment with more room to review its security: VPC, subnets, security groups, flowlogs. It is confirmed that multi-account had great impact on our production environment.
Reservations and multi-account tango together
One awesome feature of multi-account is that we didn’t have to worry about existing reservations created couple of months back. once in main account, reservations propagated transparently into all linked accounts.
Apart from benefits of this project, many challenges were faced along the road. Network setup and Identity/Access Management were the hardest.
Network setup
Defining clearly all traffic flow between accounts was hard and time consuming but, this was definitely the only path to get in control and avoid a full mesh of VPC peering between accounts.
Policies/Roles/Users
This was a delicate process for a live environment. From one side, we had to review our existing policies and roles, from the other side, we had to define new policies and roles for each newly created account. Iterations was key for reaching a steady state. we started with a minimum of access policies for roles and incrementally brought adjustments till we reached a viable setup.
S3 writes
Writing objects, from one account, to an S3 bucket owned by another account was a headache at the early stages of the project. We were left with objects and parent buckets owned by different accounts. So if you are planing such move take good care to assume role of bucket owner when you write to S3 or give the bucket owner full access to object upon its creation.
Though we are already live with this setup, we are still facing some difficulties using efficiently the multi-account, in fact:
There is no central monitoring/alarming system except for billing this is really blocking for monitoring. Not having a one place to manage alarms and metrics is a weak point for AWS organizations.
No clean way to have production workload run on multiple accounts
The fact there is no way to send control events like autoscaling triggers or any other kind of trigger from one account to another, makes it impossible to have production workload run on multiple accounts.
Finally, multi-account with AWS is a must whenever you consider your workload is growing from small to medium. The earlier you step into this the better it is for your project and your team.
