[intro]To help our company understand the way our web application works, we invented a fun, accessible analogy.[/intro]A few months ago, the Abacus Engineering team was scheduled to give a company-wide presentation at our All-Hands meeting. Rather than update everyone on our projects as usual, we decided to reach out and poll the group: what would you like to hear from us?
Right away, we had a winner.
Why does Abacus feel slow!?
Alas, they weren’t wrong. At the time, we had launched a feature that was slowing down the whole app. We had also onboarded some larger customers, and strains were showing.
Our engineering team had already identified the cause and was working on a fix, but we felt the company deserved a clear and thorough answer to their question. That meant formulating a jargon-free explanation that would make sense even without a detailed understanding of our technical infrastructure.
The truth is, web applications are complicated systems. We wanted to come up with a simple, engaging way to explain how Abacus works that would frame the larger issue of performance. We put some serious thought into an analogy:
Abacustard. It’s a restaurant that serves unique, boutique, found-materials bespoke craft artisanal conveyor-belt custard. Believe it or not, this custard-creating cafe can help you comprehend the complex communications systems that comprise cyberspace.
Abacustard. It’s a restaurant that serves unique, boutique, found-materials bespoke craft artisanal conveyor-belt custard. Believe it or not, this custard-creating cafe can help you comprehend the complex communications systems that comprise cyberspace.
Servers, data, and infrastructure
As you walk through the doors into Abacustard, you see a vast dining area with loads of customers sitting at tables. Servers dash to and from the kitchen, taking orders from customers. Chefs in the kitchen craft the custards for immediate consumption. Like a sushi bar, a conveyor belt delivers customers their orders. There’s a lot of commotion in our little cafeteria, but that’s our custom. As you might have guessed, the servers in our analogy are actually computer servers. The clientele are our users asking for information, represented by the custard we prepare in the kitchen—our backend. The kitchen fetches these bits of food from the storage pantry, our database. The conveyor belts are the web browsers that customers use to scroll through the data we return. Unfortunately, service at Abacustard sometimes gets slow. Customers have to wait a long time for their food! Let’s look at some of the reasons why this happens.Too many orders
One day, 10,000 new customers descend on Abacustard, all asking for different custards. We were only expecting a hundred or so, so we only have five servers working. It takes them a long time to take down everyone’s orders and handle the requests! This is called high load. A lot of restaurants around here have this problem, so there are temp agencies in town that will hire servers for us as needed. To fix this problem we reach out to our temp agency and ask them for a few more servers to keep each server’s load down. Sadly, there are times these temp agencies can't provide solutions. Not every problem can be solved by throwing servers at it.
Too much data
Sometimes the problem isn’t that we don’t have enough servers, but that the servers are doing too much. For example: One day our favorite customer, Custer, orders 10,000 of his favorite custard. Our kitchen wasn’t set up to make 10,000 of that custard, so we make each one by hand. It takes a while! The whole kitchen is busy working on Custer’s order and doesn’t have any time to prepare custards for anyone else. If just a few other people ordered like Custer, the whole operation would come screeching to a halt. There are a few problems here. The first is that the kitchen takes a long time to make each custard. That is a problem with the algorithms, or more simply, the process we use to make the custard. As owners (software developers) we can teach the chefs to cook much more quickly. For instance, instead of cooking similar custards one by one, chefs can use a big pot to make them all at once. Performing work on many pieces of information at once is known as batch processing. The second problem is that the conveyor belts, which show Custer his custards, start to get weighed down: the more custards we pile on, the heavier the belt and the slower it moves. In this case, slowness stems from having too many items on a web page at once. Custer can’t see, let alone eat, more than a cartload of custards at a time. One solution to both of these problems is to show Custer only a few of the custards he ordered at a time. This is called windowing or paging data, and is a standard practice on the web. Think of any online store. They only show about 25 results, then let you page through the rest. Another clever solution is to have the kitchen slide more custards on the conveyor belt right before Custer runs out of custards to look at. This is known as lazy loading, where we only fetch data when a user is about to need them.
Manager Reporting
At Abacus, we recently released a new feature called Manager Reporting. This feature did just fine in our test kitchen, but caused some performance issues in the field. Here’s how we took Manager Reporting from burning the Thanksgiving turkey to a regular Gordon Ramsay. First we had to diagnose the problem. Imagine trying to stand amid the busy bustling of Abacustard’s kitchen, clipboard in hand, trying to figure out what was delaying all those custards. Our clipboard is actually data visualization that we use to proactively monitor performance. While useful, these graphs only tell us the symptoms of a poorly functioning kitchen, not the cause. Our clipboard showed that the kitchen was absolutely filled to the brim with custard! With custards cluttering the counter space and covering the chopping blocks the chefs were confounded. With no room to operate, the kitchen slowed to a crawl. But where was all this custard coming from? The source of all these cumbersome custards was the new chef, Manager Reporting. (We knew that because we didn’t have this problem before we hired him.) He was working extremely inefficiently, making a whole cart of custards for each one he needed to serve! In his interview we didn’t notice because he only had to prepare one custard at a time. We told him to cut it out, and now the kitchen is back up to speed. In computer terms, this inefficient algorithm caused a memory issue, represented by the lack of space in the kitchen. If we had bought a bigger kitchen our chef might have been able to make the custards for now, but he would eventually fill up that kitchen too. The real problem with the situation wasn’t the size of the kitchen, but the process of the chef.
The value of custard-munication
When we came up with the idea for Abacustard, our goal wasn’t just to entertain our coworkers with whimsical stories of custard. It was to help them help us. As engineers, we often hear about performance issues from other teams. The customer-facing side of the organization lets us know when people are having issues, and we do our best to address them as they come up. This feedback is crucial to us. There are many issues we would never find otherwise. Likewise, the feedback we receive from Sales and Marketing helps us decide which features to build next. Communication allows us to prioritize our projects intelligently. At Abacus, we don’t believe in the trope of the isolated engineering team. Abacustard is just one example of our continual efforts to put our strongly held values on learning and teaching into practice. The Product team is not our translator. It’s not good enough to handwave or assume that our peers cannot understand. Communicating with our colleagues is as much a part of our job as building features. Communicating our skills is as integral to Engineering's success as the skills themselves. Since we introduced the Abacustard analogy, it has internally become a touchstone for describing and diagnosing performance issues. Customer Success now speaks the same language when explaining performance problems. When Sales asks if we can handle more clients, we respond in the language of Abacustard, and tell them about the hiring process for our servers. With the whole organization on the same page, we can make better, more informed decisions.