Backend, Distributed Systems Engineering at Uber, in Amsterdam
I’ve been working across the stack, before joining Uber. I built part of the backend system serving the 400M users of Outlook.com and the Skype for Web integration. Still, when I joined Uber three years ago to work on the Android app, I had no idea I would learn so much about large, distributed systems. My team ended up owning not only the mobile interface but the backbones to the payments powering all of Uber, processing more than $50B annually with services that handle hundreds of QPS.
We’re hiring backend engineers to build Uber Money and Uber Wallet. The press is asking, Can Uber Money compete with Apple Pay? With the team we are hiring - we certainly will be able to.
Career Growth
- Mentoring and mentorship is a big part of our culture. We have formal mentoring opportunities and encourage informal mentorship. Developers mentoring other developers and learning from each other is a big part of our culture. This culture allows us to hire less experienced people and then invest in them. It also keeps more experienced engineers a good way to further grow, by helping others grow.
- A never-ending firehose of learning about architecture. Teams write up their design approach before starting projects, then distribute it widely. This is our engineering planning review process that provides transparency, lots of learning opportunities, and the ability to give feedback to teams across the globe.
- Exposure to large, distributed systems. As someone who started as a mobile engineer, I learned a lot about distributed systems by reading design documents and talking with backend engineers. We’ve had mobile engineers fascinated by these systems move to the backend, broadening their skills.
- Promotions. We have a clear career ladder and a transparent promotions process that we keep tailoring based on feedback from engineers and managers. My team has had a pretty good run with promotions the past few years. Professional growth is something managers and peers are deeply invested in.
- Technical managers and a path into management. All engineering managers come from an engineering background, having been developers for a good time, and are technical. On backend-heavy teams, previous distributed systems experience is a common trait for managers. We have an apprentice manager program and occasional opportunities for moving from engineering to management. I moved into management at Uber, and three other engineers in Amsterdam have also done this transition.
- Principles of engineering growth. Dan Heller is a former engineering manager in Amsterdam, now a staff software engineer in San Francisco. His ten principles for growth as an engineer capture how we encourage engineers at Uber to take ownership of their growth - and the environment we strive to create to do so.
- A Silicon Valley company in Europe. There is a lot to like about this kind of environment, that merges good things from both worlds.
Engineering Culture, Architecture and Scale
As an engineer, I’ve had a hard time convincing stakeholders at other companies to adopt engineering best practices that benefit the team and the product, in the long term. At Uber, I had to push little: we are about pragmatic, sustainable, and world-class engineering.
- Scale and Reliability. We move billions of dollars each month, with systems that hit hundreds of QPS. At this scale, getting to 99.99% reliability is a challenge. This is especially true for the FinTech domain, where we need to build on less reliable financial systems. Operating large, distributed systems in a reliable way is something we practice day-in, day-out.
- Architecture. Designing distributed, reliable and extensible financial systems is hard. We document, share and debate our backend designs across engineering. I’ve picked up most of my distributed architecture knowledge by first listening, then participating in these discussions.
- Complexity. We have thousands of services. In this environment, making changes that are straightforward can be a lot more challenging than they seem. Modifying existing systems, migrating safely between systems, rolling out to millions of customers in production, making sure things continuously work - and doing all of this, quickly and with large autonomy make for quite the learning opportunities.
- Testing and quality are core to how we work. We continuously test, at scale. We test our code at the unit level, at integrations level. We invest heavily in advanced monitoring and alerting, SLOs, and oncall tooling.
- Blameless postmortems. We try to move fast, without breaking things. However, things do break, at times. When this happens, we don’t point fingers. Instead, we look for what systemic improvements we can make to not have the same type of incident happen before. Can we build automation to catch things like this, pre-commit? Tweak existing processes? Build better alerting or monitoring?
- Upping the bar. We are conscious of the areas we can get better, engineering quality-wise. We are moving towards all services having blackbox testing and automatic canary rollouts and rollbacks and continuous deployment for all services - but we are not yet there. We keep looking for other areas to improve how we build, deploy and maintain our code.
- Open source. We open source many of the tools and frameworks we build and use. Jaeger - a tool to monitor and troubleshoot complex transactions in distributed systems - is a well-known tool we built that has graduated to the Cloud Native Computing Foundation. Other projects include Cadence, H3, Kraken, Mamaray, Peloton and many others.
- Bold bets. We take on big, bold bets. As our systems quickly - just like Uber does - we are unafraid to do smart rewrites and efficient migrations to those new systems, reducing tech and architecture debt. We believe that migrations are the sole scalable fix to tech debt. We have rewritten most of our payments stack and most of our early Uber systems. While we used to be a NodeJS and Python up to 2016, all our services are now Java or Go.
Sounds interesting? Apply here and work with me in Amsterdam, or one of the teams sitting next to me.