This is a little story of how I discovered Anghami and how I ended up working with the company. I graduated in 2016 in Montréal and decided to come back to my home country of Lebanon after receiving my engineering degree. My plans definitely did not include working for a tech startup here I was actually coming back to focus on music as an aspiring musician, because I assumed that we were still in the stone age in this country and that opportunities did not really exist.
I had the wrong idea though, tech companies are around and they’ve been cooking up beautiful things over the years. So for you curious people in the field looking for an adventure, I’m here to show you that there are startups in the Middle East that are actually doing really advanced things, and that’s something I did not actually know before coming back to Lebanon.
At the core of it, Anghami is first and foremost an engineering company, thriving to become a tech hub. My main goal here is to expose that side of us.
One day I saw an ad for Anghami on Facebook if I recall correctly. I had never heard of the company before that moment and pressed on the link that took me to the website. I was interested immediately because the company combines two of my favorite things: music and programming. Of course, as any curious person, I decided to pop open the console on my browser to see what technologies they were using under the hood. I was greeted by some version of this lovely message,
and honestly I couldn’t have pressed fast enough on the link because this tiny subtle detail showed me that the people behind the company were geeks and that they expected curious people to snoop around. I ended up applying to every single engineering position at Anghami, ranging from Android, iOS, web, QA, backend engineering, devOps… Fate happened and I am currently a backend engineer mainly tinkering on the servers that power all the music that people can enjoy.
Now I would like to talk about one of our latest projects (Anghami Chats) in order to highlight a few important things that I think can drastically improve any project’s final quality, and what is involved in the process of going from the vaguest specs to a usable product. Obviously this article is written from the perspective of a server side engineer, but all of the concepts that I will mention can be applied to any kind of development, be it mobile, web or a microchip running in a toaster.
Specs:
Many programmers believe that the main part of our job is to write code. While that is somewhat true, I slowly discovered that it was literally 10% (exaggerating for effect) of the final effort. It still has to be done though, but before that, we need specs. Depending on the company you’re working at, this process can be the most chaotic one or the most organized one. Let’s just say that we are used to chaos, but the essential bit is to get to an agreed upon set of features and experiences that the product wants to implement, go from there, and then iterate ad infinitum. For our very first version of chats we wanted the users to be able to send messages to each other (you don’t say?) and to be able to share songs/artists/albums/playlists/mixtapes/videos to each other, and play them right there in the chat view.
Architecture:
So now you have a list of things that you need to build, what next? Start programming? Well, yes and no. You can start to tinker of course but your goal at this point should be to decide on the most appropriate tech stack for your specific problem. What databases will you use to store everything? What protocols will you implement in order to achieve the desired results? So many questions to ask. For those of you who are interested in databases, we ended up using DynamoDB for the biggest chunk of the data, and Redis for quick access to cached items that need to be served on the fly to users. Combining that with a few API endpoints and the power of web sockets, you get yourself a decent set of tools to implement instant messaging, with a musical twist.
Coding:
After the architecture has been designed, we can finally start coding. Now this is a very important bit, what you want to focus on is writing CLEAN and MAINTAINABLE code. Those two concepts might seem alien sometimes, and they are usually very relative. Different development teams use different conventions and your job is to adapt to the projects’ set of conventions. Write clear code, you should be able to understand it after your 13th beer, and more importantly your colleagues should be able to read what you wrote with ease. Notice I’m putting more focus on that aspect rather than on pushing you to write GREAT and CLEVER code. Always thrive for that of course but what’s more essential is working as a team.
We are not lone wolves as much as we’d like to believe it, even if we end up working long hours into the night without any light other than the dark terminal spitting out bits of data on the screen. You can’t do it all, and that’s why a collaborative effort will always lead to a better end result.
It took us a few weeks to set up a working version of the server side code. I thought we were touching the finish line at that point. Boy was I wrong. That was back in December, and we’re launching now in February. That tells you something. Now the real work had actually started. I will talk about a few different concepts briefly and I encourage everyone reading to research on how to apply these ideas in your own work and to always try to push yourself to learn new things.
Documentation
Everybody dreads writing doc files, but once you realize their importance it just becomes part of the job, something I personally enjoy.
Now that we had a usable set of API end-points, clients were ready to start implementing their side of the story, and breathe life into the feature. But how does that usually work? Well what I’m used to is people sitting together, for instance me sitting with a fellow iOS or Android engineer and hack things out together. And that works fine most of the time, and I love doing it.
But it’s not enough. You need to document everything you produce. Why? Because you want to limit the interactions that a client developer NEEDS to have with you in order to build a viable product. I’m not saying you shouldn’t collaborate with your colleagues, I’m saying that your colleagues should be able to build a client for your server simply by reading your documentation, and trust me they will be grateful for it.
When will this be really useful for us? When the web team starts implementing Anghami Chats. They’ve been very busy working on a rewrite of our web platform (be sure to check it out). The docs are ready and maintained. The team who will handle it has access to everything they need in order to make it happen. You want the process to feel like building legos, it should be that simple. Engineering is not supposed to be hard, it’s just meticulous sometimes.
Integration Testing
The specs involved so many features and edge cases from hell that I wasn’t confident in the code we produced for the most part. And by experience I know that when a programmer fixes a bug, he usually introduces 127 other bugs, and we couldn’t risk that every time we wanted to modify something in the software. The solution is to use integration testing.
We wrote down every single behaviour that the server is supposed to exhibit for everything to work properly, and then designed a test for it. These tests run on every single deployment of our APIs, and if something ever goes wrong, I won’t panic. Because we can just fix it, add a test for it, and move on. I felt much better after finalizing that aspect, I could sleep at night without worrying that things might explode suddenly.
A few quick notes about tests: a 5 year old should be able to read them so really focus on writing a set of helper tools that will make them more readable. They should be atomic and testing a single behaviour. One single complex behaviour can be composed of many simple ones. And a final important thing is to always spend some time on writing your clean up code. You want your tests to come and go and leave no trace on your live environment. Also, you want them to always start with the same initial conditions. Because in order to ensure predictability you need to set the initial conditions.
Since the integration tests now guarantee the correctness of our algorithms, adding new features is really a breeze and it feels so good to be able to add many interesting things to the product without jeopardizing everything every time.
Profiling:
Moving on to the next step. Is your code fast? Is it efficient? Does it consume too much memory? Too much network? Are you abusing your databases? Are you doing unnecessary things? Thankfully, people that are much smarter than I am have created tools for this. You should use them in order to tweak over and over again until you shave off precious running time off your wall time. We started profiling the messaging end points soon after the integration tests were roughly designed. And by the end of the process we were able to trim off many hundreds of milliseconds, and while doing that we found many hidden bugs / inefficiencies that were fixed. These issues might have never been found before a real user complained about performance and let us know. You have to try to NEVER get to that point. You want to catch ALL (as much of) the problems BEFORE they hit live servers.
Finally, we had working code, that was fully documented, that was constantly being challenged by integration tests, and that we iterated over by profiling it over and over again until we had quick and efficient end points.
Stress tests and simulations
The last step is stress testing and simulating real user load, because so far around 30 people were using it internally at Anghami, but that wasn’t nearly enough to assess if our servers and databases were not going to blow up once we went live to our users.
In order to validate that the system would hold, we replicated our live environments and generated many “fake” users and unleashed hell. You want 100 users sending a message to each other once a second? Fine do it, did anything break? No? Let’s try 1000 users sending a message every 500 milliseconds? Works? Keep going, keep trying to handle a load that is BIGGER than the one you expect from your users.
The simplest form of this is to fork one process per user, and then fire requests for each of those users in a rhythmical fashion (of course there are tools for this). This gives us the ability to tweak how many users are in our stress testing environment and how much traffic they are producing on our servers. And that was the cherry on top, seeing the whole system work as expected after all the effort put into it.
Of course nothing is perfect, and I expect many things to fail. Armed with all these tools, failures are just another day at the office. They’ll just make things better on the long run, I’m not afraid of them. And as we always say at Anghami, it’s better to fail fast.
If you are interested in joining the team please apply at https://anghami.com/jobs we are planning on greatly increasing our engineering roster and we are ready to offer the opportunity of a lifetime to anyone who is passionate, dedicated, and talented enough to work at Anghami.
I joined about 2 years ago, and I’m not looking back, simply having the time of my life.
