Reasoning about Leverage in Engineering Organisations
Leverage is an important concept for an engineering organisation. When we are debating whether to standardise and on what, which technology tools and stacks to use, whether to add new technology, replace an old system with a new one, which programming languages to use and how many, under the surface we are ultimately talking about or around, the topic of leverage of technology. It’s one I don’t think we talk about enough in our day to day work, compared to topics like productivity and efficiency or even the technologies themselves. So first, let’s build an intuition for what leverage is.
Circumscribing Leverage
Having leverage is having means to optimise the impact and performance of you and your colleagues’ work relative to effort invested. I’m not going to spend much if any time here identifying what impact is, other than to say it’s useful to do so. It invariably helps to clearly understand the value of the solution we’re engineering, something that I’ll summarise as the wish to solve real problems.
Leverage is not a new idea. It goes at least as far back as Andy Grove’s 1983 book, High Output Management, which concentrates on managerial activity—“A manager’s output is the output of his or her organization plus the output of the neighboring organizations under his influence.” Edmond Lau’s more recent definition of leverage from his book The Effective Engineer, focuses on the ratio of time invested to impact produced—“To be effective engineers, we need to be able to identify which activities produce more impact with smaller time investments. Not all work is created equal.” High Output Management and The Effective Engineer identify general activities and practices that give individuals leverage, such as prioritisation or iteration speed and are highly recommended. In the the rest of this post we’ll look at obtaining leverage through technology engineering and at the level of multiple teams.
A valuable activity then is identifying and creating ways and means to increase leverage. You might also have come across the term force multiplier—it’s a complementary idea, in that a force multiplier is something that gives us increased leverage. One observation, obvious but easy to leave unstated is we want to avoid introducing technical approaches that impair us or end up as a net negative. We should be looking for and weeding out what we are using that’s not helping as much as finding things that will help.
Lift
Using a technology or service should provide some benefit in kind to its users. We’ll call that benefit lift. Why introduce a second concept and not just talk about leverage in itself? It’s because we want a way to frame and identify the benefit not just the activity. For example, if we write a tool that 50 engineers use, instead of having each engineer write their own tool, that tool is being leveraged via re-use, but we haven’t identified the benefit. What if the tool slowed those 50 engineers down, or slowed down 20 of them? It’s very easy to get trapped here and assume simply the act of leveraging something is sufficient.
One reason lift is worth calling out is because we can have negative lift, where a technology or choice is hampering the organisation. Without considering lift while presuming efficiencies, organisations can end up in strange, uncomfortable places where those tasked with providing leverage are claiming wins, perhaps in the form of efficiencies (eg cost savings) and rationalisations (eg simplicity and removing heavy lifting), but the broader engineering organisation doesn’t appear to be getting much value or seeing things speed up.
On the other hand, if we can get it right, the benefits can be immense. Using our toy example again, imagine a lift per engineer of 4% from the tool. That’s as if we created 2 more engineers, less 1 for the engineer occupied with the tool, to magick an engineer out of thin air. While that example is unrealistic, it serves to make the point that lift is what we as engineers want to establish is happening and more broadly is how we want to think about obtaining leverage via technology.
We can think about this equationally for a generalisation of the developer tools case—a team working on technical levers for other engineers—as the number of engineers less the engineers working on technical levers force multiplied by a lift factor:
(E - T) x (1 + T x L) This is a simplification of a model [1] created by Peter Seibel in his excellent post Let a 1000 Flowers Bloom. The goal being less to obtain a precise model as having enough shape to run a Fermi-like estimation [2].
Points of Leverage
There are a number of strategies we can use to obtain technical leverage. We can build and create tools and services. We can buy or rent those tools and services. We can introduce technologies—in particular for software, we can add languages and data stores. We can agree to apply common idioms, standards, or principles. The argument is that all of these can be better examined using the idea of lift.
When it comes to introducing technology, two things need to happen. First we want to establish there will be some kind of lift, and do so in the context of our organisation. What works elsewhere may not work here. At the most simplistic level—is there expertise in-house or, will we have train up or hire in? Second, we want to look at classes of technology and not assess something in isolation. This is particularly important for programming language and database technologies as they tend to be foundational engineering choices that exhibit diminishing returns. The 5th programming language an organisation uses is not going to provide the same level of differentiated impact as the 2nd did. The 10th language is going to be a net-negative. For databases the diminishing return tends to be operational—to paraphrase Tolstoy, every unhappy database is unhappy in its own way. As expertise gets spread across ever smaller silos, it’s easy to get to a point where the engineering organisation itself starts to struggle under the burden of variation.
The technologies a company uses are path dependent. We can debate the merits of say, Go vs Python, but if the company has a sunk investment and expertise in one, that acts as a natural moat to introducing the other. This comes back to the point about differentiation—a shop with an existing investment in Go for microservices, wanting to fully leverage machine learning, may want to look at Python as that unlocks opportunities. For building a new round of microservices? Not so much.
Because of the path dependent nature of technical choices, the early engineers in a company, or the engineers around during a major phase transition, get to pick winners. This can be frustrating for engineers arriving at a company who are productive or expert in a different technology set, and doubly so when each new proposed option is held to an increasingly higher bar because of diminishing returns we mentioned earlier. That said, it’s extremely easy to overweight local or short term productivity benefits, relative to depth of expertise in a small cohort of technologies known to a larger engineering population. Local impact is often much more tangible to us and easier to attribute than a statistical benefit to a larger group.
On the other hand, highly restricting choice has its own constraints. For example it can require dedicated investment to enter new problem domains, especially if the technology does not have a broad industry backing or ecosystem. A good example is how Jane Street have invested in OCaml to allow it to be used as general purpose engineering language. Facebook’s investment on performance and gradual typing for PHP via HipHop and Hack has allowed it to grow without getting stuck on a rewrite (interpreted languages being the poster child for justifying major rewrites and technical swapouts—Square, Gilt, Twitter are good examples). There isn’t an absolute right or wrong approach here, but having a mental model where these technical choices are required to consider lift helps with decision making and tradeoffs.
Standardisation is another common approach to leverage. This can be anything from internal guidelines, to de jure/de facto industry standards, to an all-in strategy on a vendor. The key to standardization is adoption. A standard used by 80% of the engineering org has more opportunity for lift than one adopted by 20%. This works as long as the standard provides lift. If it doesn’t, the costs can be extremely high to the organisation as more and more teams experience the drag imposed by the approach. Arguing for standards as an intrinsic good in themselves is common. Standards provide a feeling of simplicity and rationality; there’s a surface elegance to having one way of doing things that has strong appeal. My own bias on this is local options are preferable unless the standard can be clearly demonstrated to provide benefit or quantified in some way, even if the collective of local approaches seems wasteful, or inefficient.
Efficiency in the sense of resource utilisation is worth touching on, as it’s a common concern for engineers and those that fund engineering organisations. The most important thing to note is efficiency by itself does not provide leverage. It may allow other things to happen, notably reduced expenditure allowing capital to be redirected to other efforts, or margin improvements. One example of this I’ve seen repeatedly over the last decade are cloud cost initiatives and ongoing observations that cloud is expensive [3]. This misses a point of using a cloud, being not so much to save money, but to obtain lift by not having to build as well as better meeting the underlying demand for compute and data services, resulting in something akin to a Jevons’ effect. As engineers we want to be efficient and frugal, but organisationally we want to assess the effort in the context of other goals, such as revenue growth.
Aspects and Consequences of Leverage
Let’s look at three general aspects of technical leverage: indirectness, dependencies, and shelf-life.
One point about leverage is that it’s indirect. It impacts how we solve a problem but does not directly solve a problem [4] . Because it’s indirect, there can be a tendency to undervalue the impact of working on things that don’t have an immediate and obvious first order benefit, and defer or miss out on the compounding effects. Technologies that afford high leverage, I think invariably, are applicable to multiple problems. Because there can be a concern from management that people are not working on the direct customer or product problem (witness the entire Lean canon). All other things being equal, we want to be sure to have enough problems to justify indirect investment, and lift is a useful way to frame that analysis.
Technology leverage introduces dependencies. Renting a service from AWS or Azure is a dependency. Using tools from the tools team is a dependency. Adding a library for circuit breaking is a dependency. We can almost define leverage as the adoption of useful dependencies. It’s an obvious and almost trivial point, but easy to overlook its implications. Of course what we want are “good“ dependencies 😀. These tend to appear in the form of paid services, healthy technical communities and engineering teams where healthy implies a track record of shipping and iterating on feedback—developing with rather than developing at other teams. There are real risks to dependency introduction but the upsides are just as real. In particular it can help to assess lift in terms of what we’re not doing as well as the perceived gain—it’s quite normal to not measure what isn’t happening. A good example of paying attention to this is Adrian Cockcroft’s “what we don’t do slide” from his time moving Netflix to AWS.
There is a time component to technology. What offered leverage in the past might not in the future. In particular an engineering organisation will want to be able to replace internal software builds with open source or services as they reach maturity and commoditise. We can think of this as an un-platforming exercise that allows the engineering team to redirect its focus on more differentiating problems and technologies. This is related to the organisation’s ability to manage the technology portfolio— a long tail of old systems lying around, and or, a slash and burn approach to greenfield work are indicators the engineering organisation is (or eventually will be) in quagmire. Sensing when to switch and being able to act on that is a superpower for any engineering organisation that wants to sustain its ability to move quickly and support business growth. The exception is moving things back in house—typically this is done on the basis of unit economics, where the organisation is at sufficient scale to build for itself. Less common is to want strategic ownership of technology or a vertical integration. In either case, using the idea of lift helps with understanding the internal investment, commitment needed to succeed, as well as timing.
Conclusion
In retrospect the idea that technical investment should have benefit seems almost too obvious to write about, and I’ll confess to having sat on this topic for a long time. There’s also really nothing new being said here. Unfortunately as engineers we don’t always seem to frame discussions around technical choices well; it’s easy to take over-simplified positions that I’ve see many times over the two decades I’ve worked in software result in either-or debates, polarisation and engineering organisations generally getting stuck. The emphasis on the organisation is deliberate—recent literature and thinking about software I think has tended to emphasise team level concerns over the organisational ones.
If there’s a takeaway, it’s this—it is useful to have a way to identify the return on our engineering investment first for practical economic reasons, and second as a way to frame decision making for an engineering organisation. Lift, thinking in terms of technical benefits and not just technical properties, has been useful to ground my thinking, and I imagine there are other viable intuition pumps we can apply. Regardless of how we choose to frame things, I think what’s important is to think about technical investment beyond individual or per team preferences, and in terms optimal overall outcomes for engineering organisation itself.
1. Peter Seibel's original has a scaling factor for each engineer added to the Team (T) that is tasked with building for others.
2. Although, trying to establish a general purpose model that described the benefits of technical leverage to an organisation could be an interesting line of research, not unlike how we assess technology more broadly with respect to GDP.
3. The broader dynamics of cloud adoption is a fascinating topic in its own right; for that I recommend this post by Subbu Allamaraju.
4. Of course leverage is not a thing, it’s just convenient to speak of it as an entity.