This post is part of a series previously posted in the public "The Platformer" channel on Mattermost's Community server. "The Platformer" is published on a weekly cadence sharing the latest updates from the Mattermost platform teams (web, server, mobile and QA). It's an insider look into how these teams operate, and what they're working on.

August is a month of onboarding for us. This Monday Mario joined the QA Platform team as a Senior Software Design Engineer (we don’t develop software at Mattermost, we design it). Next Monday we will have Tasos and Pantelis join Web Platform (as SDE II and Engineering Lead, respectively). I’m happy to have so many people join the party, especially in times where there is not so much “partying” going on across the industry. I’ve been following these industry lay-off and hiring freeze developments quite closely (I also have a bunch of former colleagues affected). My reading of the situation is that in most cases this seems to be correcting for over-optimistic “the sky is the limit” thinking for the last year or so. There seemed to be no end to explosive growth, an infinite pool of VC funding, and an infinite number of new coins to mine. But guess what? Reality is harsh. So, it’s time to cut back and focus on what matters most 🥁 Padum tss.

Before continuing our journey towards platform teams, let’s pick some of the fruits of our (in many parts of the world) sweaty labor.

Cherry picks

Joram posted another monthly update on the multi-product architecture.

On the mobile platform end, we released another beta build, squashed more bugs and are filling more gaps in functionality. GraphQL support is still eagerly awaiting to be activated. However, re-enabling GraphQL on community is still awaiting a few fixes on the web platform end. Speaking of which…

On the web platform end, beside being overwhelmed by anticipation of the upcoming new joiners, we’re making good progress in getting webpack federation ready to be merged. This will be a solid step towards the multi-product architecture on the front-end. The upgrade to React 17 is also almost there. Beyond that, as mentioned, we’re fixing a few issues in GraphQL before we enable it again on community.

On the desktop platform end, we’re making progress on building the desktop app in The Cloud ☁️. We also finally got native node modules working, squashing bugs, and are overwhelmed by anticipation of doubling the size of the desktop team next week.

On the server platform side, we’re still moving fast, but breaking fewer things (compared to last week). One could say we’re moving fast on stable infra. Refactoring of the multi-product architecture into a few smaller services continue, flushing out the “app” package bit by bit. By the end of Q3 you’ll all be confused Travoltas, you’ll see.

On the QA platform side, beside onboarding Mario, we’re also entering the wrap-up stage of running end-to-end tests of mobile v2 in The (Mac) Cloud ☁️. We’re running another weekly The Cloud ☁️ release. As well as making plans to move test plans over to a git repository (hosted in The Cloud ☁️), to make it more accessible to The Community ☁️ and version control it.

The Road to Platform Teams Episode II

I have to apologize to you all. I had not anticipated the amount of sleepless nights and marital problems I would cause to many of you by leaving you hanging with a cliffhanger like that, last week.

A few cleaned up responses:

WHAT HAPPENS WHEN WE GO VERTICAL!? I NEED TO KNOW! I CANNOT WAAAIT A WEEK TO FIND OUT GAAAH!!!!111one

And:

Dude, with all due respect, you cannot play with minds like this. My family needs me during the weekend. I was completely absent, drawing org charts on white boards in our basement for days and nights trying to figure out what would happen, and especially WHAT THE TRADE OFFS WOULD BE. Man! My wife is upset. My kids kept asking “where is daddy?” Stop toying with our minds.

I hear you. However, a disclaimer upfront: we’re not going to wrap this journey up this week either. My family also needs me, and I cannot be sitting here typing up absurd, made-up reader feedback day and night either. Just setting expectations here.

Before we proceed, and for those who have not been losing sleep over this, a quick TL;DR recap: Last week we started to look at how to structure an engineering organization effectively. We looked at structuring teams “horizontally” that is: per functional area, per role. A mobile team, a web team, a server team, QA team etc. We saw this has advantages, but comes at the cost of communication and coordination cost. Our hypothetical four-week project, turned into a hypothetical twelve-week project.

Then I wrote:

So next week, let’s flip this whole structure on its side, and go vertical. Sounds kinda random? This is management, we just throw stuff at the wall and see what sticks. YOLO! No seriously, there’s good reason to believe this may actually work, you’ll see.

So, let’s do this. Let’s take that radical turn and no longer organize around what people do (“Look at me, I’m a HTML developer!”), but what they work on (“Look at me, I build the Boards product!”).

So once again: reorg time!

Poof! gone are our web, mobile, server and QA teams. No more! We shall now have “product teams” internally staffed as follows:

• Frontend engineers
• Backend engineers
• Mobile engineers
• QA
• Product Manager
• UX
• Engineering Lead/Manager

That’s right. Everybody cosy put together. In our previous round we didn’t cover the product roles (product manager and UX) specifically, but I’ve listed them here. If we want to go all in, verticalization can even go deeper. You can also embed the “business people” into the team, such as sales and marketing. We’ll look at one company later that does this.

Why would you do this? To exactly solve the problem we observed in our previous setup: no more external communication required. In an organization, any time we have to reach across team boundaries and depend on them, we incur a cost. It’s expensive. By having all functions sit together in one (virtual) room, we severely limit the number of interactions we need with other teams, so we can go fast.

And guess what, this works! When you form autonomous units like this, each controlling their own destiny without much outside direction, you are effectively creating mini startups. And guess what startups are good at? Going fast, breaking things (without stable infra), and being innovative!

Wooh!

So, we got there, organizational nirvana, no? The silver bullet of company organization, yes? As we all know, there is no silver bullet (except for Silver Bullet), so what’s the catch this time?

To answer this question, let’s bring back an often forgotten old family-game classic. It’s a game that was highly popular for generations, but then brutally destroyed by uncle Marv, who we shall get to in a bit.

The game: guess the org chart!

This is how it works:

You’ll all be given a piece of paper and a pencil. Then, I’m going to call out the name of a company, and you’re all going to draw the organization chart.

Who gets closest to reality wins!

I can see the excitement in your eyes.

Since you will likely have limited experience with this game (for obvious reasons which, again, we’ll get to in a moment) let’s start with the “baby edition” version of this game. Don’t worry about the label, I sometimes play this with my five-year old twins as well. In this version you don’t have to draw the entire org chart, you just have to pick one of two options: horizontal or vertical? Is this company organized by function, or by product?

Alright. Ready?

We’ll start with two companies you will all know (and maybe even love).

1. Amazon
2. Apple

Think. Think some more. Got your answers? Perhaps you even got some org chart drawing you want to show? Great drawing!

Ok cool.

Before getting to the answers, let me tell you the story of why “guess the org chart” dropped significantly in popularity about 60 years ago.

In the 1960’s there was a fellow who was exceptionally good at this game. He’d win every time. You’d tell him the name of the company, show him their products, and he’d draw you an org chart. He was good. Annoyingly so. I think we all know people like this that just spoil it for everybody else, right?

The name of this party pooper was Melvin Conway.

What is the ultimate sign of success when you’re good at something? They name a law after you. In Melvin’s case: Conway’s law:

Any organization that designs a system (defined broadly) will produce a design whose structure is a copy of the organization's communication structure.

What does this mean in non-Wikipedian? You ship your org chart. Your products betray how you are organized. They’re a tell, as they’d say in poker (although “guess the org chart” predates poker by about a decade).

Once you figure this out, “guess the org chart” is no fun anymore. Thanks Uncle Melv!

Alright, back to Amazon and Apple.

Amazon is a massive company, but the amount of stuff you put out doesn’t generally scale with the number of people you employ. However, at Amazon, and this is especially visible in AWS, there seems to be a much stronger correlation between number of employees and features shipped than in other places. If you ever witnessed their yearly re:Invent conference, or ever checked the AWS services page your mind will have been blown. AWS had a bit of a head start in the cloud computing world, but the amount of stuff they keep adding to their offering is shocking. They are moving super fast and only seem to be accelerating.

Their secret? Amazon is probably the most extreme example in the tech industry that organizes vertically. While us, mere mortals are pretty braggy about “doing DevOps.” AWS has implemented BusProdDevOps, integrating not just development and operations (DevOps), but adding Product and Business as well into a single units with a large degree of autonomy. Whenever AWS sees an opportunity to add another product, they just “instantiate” another BusProdDevOps team and off they go.

So what’s the catch? There are at least two obvious ones that come to mind:

The first you’ll hear about the second you first talk to any senior person at Amazon. “Oh my, there’s so much duplication here, you have to learn to accept it.” Since each team is autonomous, they get to pick their own tech stack. Of course, teams use other team’s services. AWS Lambda runs on AWS EC2, they wouldn’t build their own data centers again. However, in many cases teams pick their own path, facing similar problems and solving them their own way, over and over again. People tend to talk within their teams, less so with people outside. So if a frontend engineer needs a button, they’d ask the team: “Do we have one? No, cool, I’ll just build it.” “But that other team has a button!” Yeah, but it doesn’t do x, y or z, and we don’t want to create a feature request and be blocked on waiting for it. We’ll just build our own. How many implementations of a button are there at Amazon? I don’t want to know. This is also externally visible, which brings us to the second catch.

Incoherence in product line integration and user experience. If Melvin Conway would be around today, he’d able to pretty accurately draw you an org chart simply by clicking through the Amazon website. You often find yourself on a page wondering “Why does this look so different? Why is this listed here, whereas that other thing is all the way over there?” Org structure! Two different teams that don’t coordinate.

Are AWS services seamlessly integrated? Is it super clear when you would use this service over that other one, because they have cleanly cut boundaries? No spoilers, ask any AWS customer to get the answer.

I think this may have recently improved, but until recently the AWS Console was also a massive tell on AWS’s internal organization. The console for each service looked ridiculously different. Some seemed to have been implemented with jQuery UI, where others looked pretty modern. There was no consistency in UX at all. Why? Again: team boundaries. I’m sure there’s an AWS Console team, but what they offer is likely just the skeleton that other teams can plug their iframe UIs into.

Now, let’s turn to Apple. When Steve Jobs returned to Apple in 1997 he made a big change:

When Jobs arrived back at Apple, it had a conventional structure for a company of its size and scope. It was divided into business units, each with its own P&L responsibilities. General managers ran the Macintosh products group, the information appliances division, and the server products division, among others.

In other words: Apple was organized primarily vertically before.

Believing that conventional management had stifled innovation, Jobs, in his first year returning as CEO, laid off the general managers of all the business units (in a single day), put the entire company under one P&L, and combined the disparate functional departments of the business units into one functional organization.

Apple (apparently ever since) has primarily been organized horizontally: by functional area.

Alright, so let’s have a look at Apple’s products, can we tell this is the case? I would say so. Even to the casual observer, this is actually quite visible.

Apple’s products are in fact a remarkable feat in consistency. They look and feel similar, and do not overlap all that much. Also if you look at the tech level that underlies it, they share chip sets and CPUs and manifacturing pipelines. At the software level, they’re effectively all powered by the roughly the same operating system (ranging from Macbooks to iPads to iPhones to Apple TVs to Apple Watches and Homepods) and same underlying development frameworks (UI Kit, C, Objective C, Swift).

When you think Apple, you will also think “design” and indeed, you’ll also find that in the Jobs era, Jonathan Ive (Apple’s lead designer) reported directly to Jobs.

Oh Conway...

So what’s the catch here, then? Apple is the most valuable company in the world, right?

Not surprisingly, I’d say the catch is speed.

Apple is also a massive company, and they sure do a lot, but my sense is they don’t put out the same amount of “features per employee” that Amazon does. Knowing what the challenges are of a horizontal organization like Apple it’s not hard to understand why. If you’re working on the next gen Airpods and need some changes in your design, you may have to wait in line, because designers are currently working on the box design of the next iPhone.

The way Apple managed to still be successful is through relentless focus. Given their size, their product line is actually rather small.

So let’s sum this up:

• Horizontal organizations are great at producing consistent, coherent products, with a lot of reuse of knowledge and software/hardware components. However, this comes at the cost of speed.
• Vertical organizations are great at moving fast and scaling up. This comes at the cost of replication of work and a _less _coherent suite of products.

Oh my. None of them are perfect. Why can we not have nice things? Welcome to life. However, there’s something we can do that strikes a reasonable balance.

And you will never guess what is is... More next week!

Have a great weekend! Try to get some sleep over this.