The End of One Era
Well it is 2026 already. If you take a look at my last post here, it was four years ago when I started my first job at Amazon. Welp, a lot has happened. I just got laid off, but I still feel lucky. Here is the pièce de résistance: I enjoyed my time at Amazon, but it started to feel less interesting and less learning for me. So I actually feel lucky to have Amazon pay me to switch jobs. Anyway, here are all the interesting bits I learned from my past 4 years without violating my NDA.
I joined Amazon back on Valentine’s day, 2022, and I was assigned to a team called Your Orders, which is mostly known for developing the order history page on Amazon.com (and all the other marketplaces around the world besides US). We also owned other pages like Order Details, mobile Product Owner Page. I vividly recall my first 1 on 1 with my manager before the start date, and I asked her what I should do to prepare myself for the role for it being the first actual tech industry experience for me. She said, well, reading Effective Java would be a good start. So I bought the book and read it. Three months after I joined, most of the changes I have made are on Perl-Mason. My knowledge on Perl was on the same level as my knowledge on developing quantum computers - close to none.
Why did my manager ask me to learn Java while we worked on Perl? It’s because Amazon is old. Jeff Bezos was working on a door panel in 1994 when Amazon started selling books online. When Amazon started selling other stuff than books around 1997, they created an architecture called Obidos which is a monolithic architecture for the whole website. Actually ‘architecture’ isn’t the right word, as it was basically a giant giant package. The engineers soon found the architecture hard to scale (ofc) so they started a new architecture called Gurupa, named after the Amazon river in Pará. This new architecture is written solely in Perl-Mason which I was too young to learn from school. BTW there is a lawsuit between Amazon and IRS which interestingly adds a lot of technical details about Gurupa. Check it out here.
So what changed? (Speculation (said confidently) as I didn’t talk to Amazon CTO or Bezos) Why do we need to use Java if Gurupa is already an improvement? Well, first of all, it is Perl. It makes you feel old when you look back on what you have written 3 months ago - you simply have to stare at it for a while before realizing what you did there; It’s utterly unreadable if author doesn’t like comment in code and has a bad variable naming habit. I spent hours looking through all the Perl files just to understand what that $s is and why it can call functions not defined in the package. Because of this, you can see how it would not be the teacher’s pet in colleges or online classes. Almost twenty years later, new engineers like me do not know how to read or write Perl. So here is the second point - Amazon cannot find low level engineers who can code Perl anymore. Oh-if I didn’t mention it, Gurupa is a monolithic architecture too. Bro Bezos replaced Monolith with another Monolith (but better, people can own their services packages now). Every deployment had to be coordinated weekly with people cross checking their changes, making sure they were not the one who blocks the whole company’s deployment.
Now you understand why a change was needed. Most my time I had with Amazon was trying to migrate that big old Perl stack off to Java and the projects derived from that goal. The migration was long overdue, as it should have finished before I joined Amazon. It was hard also - we are serving 30+ marketplaces over the world, while new marketplaces are launching. We had to ensure customers doesn’t notice the change, and we had to ensure we support all the things we have launched since 1995 while launching new features.
Anyway, during the first 3 years my effort was mostly in building the new Your Orders page. I was exposed to so many cool things at the same time it overwhelmed me for the first few months and I had serious imposter syndrome. Build system (brazil, you can ask ChatGPT what it is. It’s the best, the BEST building system I have ever used ever), native pipelines, code-based monitoring system, ticketing, federation system, codeless microservices, you name it.
I started my own project the second year I joined Amazon, trying to migrate the payment alert logics from Perl to Java. Essentially, previously we had this page central controller which calls a payment dependency and then replace the text in-place on the page, and we wanted to move away from that. In-place replacement on the page basically means you have to keep all the payment business logics in the page controller. Like this:
page.allPreparations();
alert = page.callPaymentDependencyAndTransformIntoPresentableAlert()
page.getAllContent()
if (alert) page.order.text.replaceWith(alert);
page.render()
And that’s bad. Having the business logics in our frontend service is already bad, not to mention we don’t own or know the busines logics. We are lucky that our services are already designed as tiered services, separating business logics, display logics from presentation logics, so we don’t have business logics in presentation layer; also we are lucky that our awesome team has just built a federation system to our services. Basically, federation system allows us to build a separate application solely for the purpose of sending display-ready page customization based on different use cases. So I scoped the work, talked to different teams, traverse through different payment methods (there are a lot of ways to pay for Amazon orders around the world; e.g. you can go to 711 to pay for your order in Japan) and solved those edge cases. An interesting thing was, the project was actually created before I joined but because of its ownership complexity and people moving around, it didn’t gain traction until I picked it up. I am giving myself a little trophy here. One thing to mention is Datapath (listen here to get more idea on the platform), which is a fast data access and data join platform. I dived into their codebase with one question in mind: how did they resolve the logic dependency with super fast latency? I can only say the code I have read are just pure brilliant.
I have more fun development stories to tell but I feel like I need to mention the operational side of things before dragging too long; so I will probably save the stories for the future. I cannot stress how lucky I feel jumping into a team who owns one of the most important pages on Amazon platform. It was a bless for a fresh graduate to touch services that have more than 30k+ transactions per second (PER SECOND!). My site here has 0.0005787 transaction per second. I learned how to design and implement distributed systems correctly so it won’t break under 3x the traffic during Black Friday. You would assume such teams get a lot of tickets, especially sev-2+s, but I actually only got paged twice during my last oncall and none during Black Friday. That’s the outcome from good design, good code, proper monitoring, and proactive incident prevention. I learned so much while not getting overwhelmed during oncall week, which is insane if you compare to other companies or teams (AWS, wink wink). But we do get customer service tickets though, and some of them are just… fun. I did resolve a ticket which claimed a customer was “having trouble seeing his orders when the Wifi is not connected”. Another classic one was “customer not being able to see the tracking of the order” and the phone was on Android Lollipop (released Nov 2014). Can’t do much for you bro except helping you find a newer phone on Amazon.
Looking back, I did feel the pressure increasing as I know more and more about the services, frameworks, and tools. Mostly because I now know how to improve them. I spent a significant amount of time last year streamlining my workflow, while improving our team’s workflow. I did launch a bunch of stuff last year, and if you see the “Ask about your purchases” pills on your Amazon app, that’s my work too. One day before the layoff, I had my last 1 on 1 with my manager. She told me I had a good performance standing among the team, and the next morning our whole team was just gone. I guess performance and luck don’t always intersect.
My God, In God We Trust
But we never really know what got discussed
Click boom! Then it happened
But no one else was in the room where it happened