All notes

Reading a system you didn't build

Before you can change a production system safely, you have to understand what it is actually doing — not what the documentation says it does. Here is the order I work in.

Most of the work I get called in for is not greenfield. There is already a system, it already has users, and something about it has started to hurt — latency, cost, an audit, or a team that has stopped being able to ship.

The temptation is to start with the fix. I have learned to start with the map.

The documentation is a hypothesis

Every system I have joined had documentation that was true at some point. Services get split, a caching layer gets added in a hurry before a launch, someone works around a bug and the workaround becomes load-bearing. The architecture diagram on the wiki is the system somebody intended to build.

So I treat docs as a set of claims to verify, not as ground truth. The things that do not lie:

  • Traffic. What actually gets called, how often, and by whom.
  • The database. Schemas and query patterns tell you what the system really cares about.
  • Logs and traces. Where time goes, and where it goes wrong.
  • Deploy history. What changes often is what is fragile.

Find the load-bearing walls

In any system there are a handful of paths that matter and a long tail that does not. On a customer onboarding platform, the load-bearing path is the one a new customer walks through to get an account. Everything else — admin screens, reporting, back-office tooling — can degrade for an hour and the business survives. That path cannot.

I want to know, early:

  1. Which flows would cost real money if they broke for ten minutes?
  2. Which of those have no fallback?
  3. Which are also the slowest?

The overlap is where the work is. It is rarely where the loudest complaints are.

Constraints before solutions

In regulated environments — banking especially — the interesting constraints are not technical. When I architected the KYC module for a customer onboarding platform, the shape of the solution was set by compliance long before any design decision: identity verification steps, document retention, and audit trails aligned to FATF and Central Bank of Egypt requirements.

You cannot design that backwards. If you pick the architecture first and discover the compliance requirements second, you rebuild.

So I ask early and bluntly: what are we legally required to do, what are we contractually required to do, and what does the team merely believe we have to do? The third category is usually the largest, and clearing it out is often the cheapest win available.

Measure before you touch anything

You need a number from before, or you cannot prove you helped.

This sounds obvious and is skipped constantly. On one onboarding platform, the work that produced roughly a 35% cut in average API response latency was mostly Redis caching and query strategy — but the only reason anyone could say "35%" is that the baseline had been captured first. Without it, the same work is an anecdote.

Capture the baseline even when it is embarrassing. Especially then.

Change in increments that can be reversed

The migration plan matters more than the target architecture. Almost any reasonable target state will work. What kills projects is a plan where nothing is valuable until everything is finished.

I look for a sequence where each step:

  • ships on its own,
  • is visible in the metrics,
  • and can be rolled back without a data migration.

If a step fails those tests, it is too big. Split it.

Leave it owned

The last step is the one people skip: the system has to be owned by the team after I go. That means runbooks for the failure modes we actually hit, dashboards someone other than me understands, and at least one engineer who has walked the critical path end to end with me.

A system nobody on the team can debug is not finished. It is just deployed.


This is roughly the order I work in, though real engagements are messier than a list. If you have a system that has started to hurt and want a second pair of eyes on it, get in touch.