Categories
Miscellaneous Scalability

Pay Off Your Technical Debt

The first thing I do when I get my hands on a client’s code is to figure out the size of the code base. I execute something like this to determine the number of lines of code (LoC) using a fresh checkout from trunk:

$ find . -name “*.java” -exec cat {} \; | wc -l

At one recent client, the measurement was over 180,000 lines of Java source code in the project. I was a bit taken aback as the project didn’t seem to need nearly so much code for what they were doing. After all, this was a fairly straightforward web application.

In a search to find what all those 180,000 lines of Java source code were doing, I found several tens of thousands of lines of code dedicated to custom implementations of the following:

  • Database connection pool
  • Transaction layer
  • Multiple ORMs
  • Caching layers

This was technical debt, pure and simple.

The phrase technical debt is attributed to Ward Cunningham from the OOPSLA conference in 1992:

Shipping first time code is like going into debt. A little debt speeds development so long as it is paid back promptly with a rewrite. . . . The danger occurs when the debt is not repaid. Every minute spent on not-quite-right code counts as interest on that debt. Entire engineering organizations can be brought to a stand-still under the debt load of an unconsolidated implementation, object-oriented or otherwise.

The inclusion of transaction management, caching layers, a connection pool, and ORMs represents technical debt in the form of infrastructure code, not application logic. The infrastructure code represents the foundation and plumbing atop which the application logic rests. In and of itself infrastructure code provides no value to the code and company. In fact, quite the opposite — it requires a disproportionate amount of time to design, implement, test, and maintain.

It is common for engineers to want to write infrastructure code. It’s a nice respite from the sometimes tedium of writing business applications. However, it is easy to understand why an in-house implementation will be lacking in robustness. Here are some oft-repeated causes for the poorly-performing infrastructure code:

  • There was only one engineer tackling the “problem”
  • The engineer lacked in-depth domain knowledge of the internals of such systems
  • Infrastructure wasn’t the engineer’s “real” job, the projects only received part-time attention
  • The use cases for which the infrastructure code was designed were too narrow in scope; the code often implemented a set of features needed by only the then current set of application requirements; when new application requirements are mandated by product management, the infrastructure code needed to be augmented again ad infinitum

Companies have several stages of growth, and it is my opinion that using off-the-shelf solutions will prove adequate even into the hundreds of thousands of users. As a company grows, there will be a point at which off-the-shelf tools will no longer support the load that its users generate. I would argue that the load would have to be measured in the millions of users to warrant infrastructure code implementation. And then, it’s a must that detailed rationale, research, and measurements before entering into such a responsibility. Don’t have a hunch or a gut feeling – have data.

Remember Lord Kelvin who famously said:

If you can not measure it, you can not improve it.

The bottom line is this:

Unless the company is in the business of offering infrastructure or middleware AND/OR has millions of users and terabytes of data, it is best that the company focus on its core technology differentiator and leverage existing for-cost or open source solutions for all of their infrastructure code needs.

For companies like Google, Facebook, Amazon, their needs go beyond what off-the-shelf tools provide, and thus customized tools and implementations are a legitimate effort for such organizations. But they arrive at such systems after very close scrutiny in their correct uses of the current systems before implementing their in-house systems. Even then, they will often open source these solutions and solicit contributions from the community to help in alleviating this maintenance burden.