Eighteen months at DigiMantra Labs. Three systems still running, and one duplicate-payment incident I can walk you through.
Four systems, what I was responsible for on each, and what changed in production because of it.
R-01
Owned the backend across authentication, tenant isolation, payments and deployment. Responsible for what the system did in production, not only for what it did in review.
Reliable systems are designed around failure before failure occurs.
R-02
Owned the Lambda-based runtime, its deployment pipeline and its operational behaviour — cold paths, retries, timeouts, and what the on-call engineer sees at 3am.
Infrastructure is a product decision that happens to be written in configuration.
R-03
Owned the payment integration end to end, including the duplicate-charge incident it produced and the idempotency model that closed it. Accountable for correctness under retry, not only under test.
A payment system is judged by what it does the second time it receives the same event.
R-04
Owned the backend for automated workflows built on model output, including the parts that decide what happens when the model is wrong. Responsible for the boundary between a probabilistic step and a system of record.
The engineering work in an AI feature is almost entirely in the failure path.
Four decisions taken in production systems, with the alternatives that were real at the time and the costs that came with the choice.
D-01
Needed capacity and deploys that nobody had to operate. Traffic was bursty and impossible to forecast, and the engineer who owned deployment also owned the product work.
Deploys stop needing a release owner, failures stay inside one route instead of taking a host with them, and capacity stops being a decision anyone revisits weekly.
Managed functions
SELECTEDNo hosts to patch, per-route isolation, cost follows traffic.
Container service
NOT CHOSENBetter local parity, but someone has to own scaling policy and base images.
Long-running VMs
NOT CHOSENFull control, and a patching schedule the team could not staff.
Accepted cold starts on rarely-hit routes, runtime coupling to one provider, worse local reproduction, and a separate home for long-running jobs.
Engineering is usually the process of choosing which problems deserve to exist.
D-02
Needed correctness under repeated delivery. Duplicates are normal traffic rather than an error, and we learned that from a duplicate charge in production, not from the documentation.
Retries arrive at the boundary, so that is where correctness under retry belongs. One choke point can be audited; correctness spread across every handler cannot, and it decays the moment someone adds a handler.
Event ledger at the edge
SELECTEDEvery event id recorded once, before any handler runs. Replay becomes safe by construction.
Guards inside each handler
NOT CHOSENCheaper to write, and correctness then depends on every future author remembering.
Accepted one extra write on the hot path, a ledger that grows until someone owns its expiry policy, and a boundary that is now first to fail under load.
A system that cannot be safely retried is not finished, it is only untested.
D-03
Needed sign-in without owning stored credentials. Business users sign in occasionally, across tenants, often on shared machines, and a password flow means owning storage, rotation, reset and lockout.
A credential that is never stored cannot leak, and the remaining risk moves to email delivery — measurable, owned, and failing visibly rather than silently.
Managed email link
SELECTEDNo stored credential, no reset flow, session policy in one place.
Password + reset
NOT CHOSENFamiliar to every user, and the largest surface area of the three.
Enterprise SSO only
NOT CHOSENRight answer eventually; most tenants had nothing to federate with yet.
Accepted sign-in latency that depends on someone else's mail queue, shared inboxes that complicate attribution, an expiry window that has to be tuned against real behaviour, and a separate route in for automated tests.
The cheapest secret to protect is the one you decided not to store.
D-04
Needed automation that could be wrong safely. Model output was being written into a system of record, and the wrong parts were wrong with complete confidence — the failure mode that does not announce itself.
A confidence gate bounds the blast radius and makes mistakes visible instead of quiet. The review queue is also the measurement that says when automation can safely widen.
Confidence gate + queue
SELECTEDAutomatic where it is reliable, reviewed where it is not, measured everywhere.
Fully automatic writes
NOT CHOSENSimplest system, and errors reach the record silently.
Review everything
NOT CHOSENSafest, and it removes the reason the feature exists.
Accepted two code paths to keep correct, a queue that becomes a backlog the moment nobody owns it, and a threshold that is never finally right.
Automation earns trust in proportion to how visible its mistakes are.
Three systems that run in production, documented as the problems they exist to solve.
Needed one codebase to serve tenants who must never see each other's data, and one deployment path that a single engineer can operate safely. Each tenant arrived with its own pricing rules, its own onboarding and its own idea of what an order is.
Tenant data is isolated at the query boundary rather than by convention. Onboarding a tenant is configuration, not a release. One deployment path covers every environment, and any request raised in support can be traced by id.
Broke at the payment boundary. Duplicate webhook delivery charged an account twice before idempotency existed at the edge; the repair was a decision about where correctness lives, recorded as D-02, rather than a patch to a handler.
Multi-tenancy is a data-access decision that everything else in the system inherits.
CS-02
See ownership record 1 and decision 3.
WALKTHROUGHNeeded sign-in that works for infrequent business users across several tenants, without the system storing a credential it could lose. Sessions also had to mean the same thing to a browser, a background job and a support engineer looking at a log.
No password is stored anywhere in the system, so there is no reset flow to secure and no credential to leak. Session rules live in one place, and a support engineer can answer whether a person is signed in from a single source.
Broke at token expiry. Long-lived browser tabs held sessions that had quietly expired and failed on the next write instead of at the boundary; refresh moved to the edge and the failure became a redirect rather than an error.
Identity is the one subsystem where a small convenience becomes everyone else's constraint.
CS-03
See ownership record 4 and decision 4.
WALKTHROUGHNeeded a workflow that uses model output to do real work without letting a confident mistake reach a system of record. The work itself was repetitive enough to automate and consequential enough that being wrong quietly was unacceptable.
Model output is validated before it reaches persistent state, and low-confidence results wait in a review queue instead of being written. The queue depth is the signal that says whether automation can widen or should narrow.
Broke at the retry path. A failing downstream write put the same job back on the queue repeatedly and the review backlog absorbed the noise silently; the queue gained a visible depth metric and a dead-letter path.
A queue with no one watching its depth is a place where failures go to be forgotten.
Seven boundaries down one spine — what each is responsible for, what crosses it, and where it fails in production.
CS-01
Multi-tenant commerce backend. One request path, seven boundaries, and the places it fails.
Systems like this one are not built by an architecture. They are built by whoever is still holding the pager after the diagram is finished.
Failure points 3 · Decisions D-01 D-02 D-03
Three stages of production responsibility, newest first, and what became possible after each one.
EX-03
Multi-tenant products in production with real money moving through them, operated by a team small enough that every engineer is on the failure path.
Trusted with the deployment path, the payment boundary and the on-call response for backend services. Architecture decisions were made here rather than received.
Became able to design for failure before it occurs rather than after, and to decline work whose operational cost was not yet understood.
Responsibility grows fastest in teams too small to hide behind.
EX-02
Several client platforms with different constraints and no shared runtime, where the same problem kept arriving in three different shapes.
Trusted with service-level work end to end, including the authentication boundary and the release that carried it.
Learned to tell a client's constraint from the system's, which is the distinction that makes a decision reusable on the next platform.
Working across several products teaches faster than working deeply on one, and worse.
EX-01
Existing services with real users, where most work meant reading code someone else had already shipped and making a small change safely.
Trusted with scoped changes behind review, and with the question of whether a change was safe to ship on a Friday.
Became able to read an unfamiliar system faster than rewriting it, which is the habit every later stage depended on.
Most production code is read far more often than it is written.
Eight groups of tools, organised by the production responsibility they carry rather than by category.
TK-01
Carries the request. Everything a caller waits for happens here.
One language across services, scripts and tooling, so context-switching costs nothing. Types exist for the boundaries where a tenant id must never be optional. NestJS where structure earns its overhead, Express where it does not — the choice is per service, not per résumé.
Cold-path startup cost is real, measurable, and accepted per D-01.
TK-02
Holds the truth, and holds tenants apart.
Relational where correctness and constraints are the point, key-value where the access pattern is known before the data exists, document storage where the shape is genuinely the client's rather than the system's, and cache only where a miss is harmless.
Tenant isolation is enforced at the query boundary, never by convention above it. Concurrent writes to one tenant are the remaining contention risk.
TK-03
Decides what runs, where, and who is woken when it does not.
Two tools, chosen together so that no engineer owns a host. Failure is isolated to a route rather than taking a machine with it, and capacity stops being a weekly conversation.
D-01 for the decision and its cost; CS-01 boundaries 02 and 06 for where these sit in the request path.
TK-04
Gets a change into production without a person remembering how.
One pipeline definition for every environment, so parity is a property of the system rather than of somebody's discipline. Containers are used where a runtime needs pinning, not as a deployment philosophy.
A deploy is a merge. Rollback is a redeploy of the previous version, because schema changes are kept out of releases.
TK-05
Decides who the caller is, once, for everyone else.
One managed component rather than a hand-built flow, because the system then stores no credential it could lose and the session rules live in one place. Email-link sign-in is a property of this choice, not an addition to it.
Sign-in latency now depends on mail delivery, and token refresh had to move to the edge after expiry surfaced at the write instead of the boundary.
TK-06
Moves money, and stays correct when the same event arrives twice.
One provider, integrated end to end including the webhook boundary — which is the part that actually decides whether a payment system is correct. At-least-once delivery is treated as normal traffic rather than as an edge case.
Every event id is recorded once before any handler runs. The ledger grows and needs an expiry policy with an owner. See D-02.
TK-07
Does repetitive work, and fails visibly when it is wrong.
Grouped as automation rather than as "AI" because the engineering is in the workflow around the model, not in the model. Orchestration, validation before persistence, and a review path are the parts that make the output usable.
Two code paths to keep correct, and a confidence threshold that is never finally right. See D-04.
TK-08
Renders what the backend decided.
Owned enough to build a working product end to end and to know what the frontend needs from an API before being asked. Not claimed as a speciality, and listed last for that reason.
Sufficient for internal tools and admin surfaces. A dedicated frontend engineer will do better work than this, and should.
One responsibility with no tool behind it, stated rather than hidden.
Observability
WHAT IS TRUE: Platform-native logs and traces, keyed on one request id. No dedicated observability platform has been operated in production.
WHAT THAT MEANS FOR A TEAM: Investigation works and dashboards do not exist. Adopting a platform would be new ground, and the request-id discipline is the part that makes it easy.
Orchestration
WHAT IS TRUE: No Kubernetes in production. Serverless was chosen precisely so that no one had to run a cluster — D-01.
WHAT THAT MEANS FOR A TEAM: Container orchestration is a gap, and a deliberate one. It would be learned rather than recalled.
Scale
WHAT IS TRUE: These systems serve tenants, not millions of concurrent users. Nothing here was hardened against a traffic profile it never saw.
WHAT THAT MEANS FOR A TEAM: The judgement is transferable; the specific tuning is not. Stated here so it is never inferred from the architecture.
A toolkit is a record of what has been maintained, which makes it the wrong place to look for what someone is curious about.
Six areas currently worth time, and the question each one is being read for.
EXP-01
Building small production experiments to understand where orchestration boundaries belong.
A model call is a step in a workflow, not a feature, and the interesting engineering is in what surrounds it — retries that do not duplicate work, state that survives a partial run, and a boundary a human can inspect. That is the same problem payments and background processing already posed, which is why it is worth the time rather than merely current.
Where should orchestration stop and application code begin?
EXP-02
Reading how teams instrument systems they did not design; no platform operated in production yet.
Request-id tracing has answered every production question these systems have raised so far, which is precisely why the next class of question is unfamiliar — the ones asked about behaviour in aggregate rather than about one request. The gap named in the toolkit is the strongest argument for spending time here.
How observable does a system need to be before it is trusted to run unattended?
EXP-03
Working through consistency trade-offs on paper before they arrive in a system that needs them.
Tenant isolation inside one database is a solved problem in CS-01. Isolation across services that each own writes is not, and it is the next constraint any of these systems would meet if they grew — so it is better understood before that happens than during it.
When is eventual consistency a design decision and when is it a deferred bug?
EXP-04
Running one personal environment defined entirely in code, to find where the definition stops being readable.
One pipeline definition already gives environment parity, but a definition that nobody can read is a different kind of risk from a manual step. The interesting question is not how to automate more, it is how much of a system's shape should be legible from its configuration alone.
How much of an architecture should be readable from its configuration?
EXP-05
Moving small pieces of one side project to the edge to see what genuinely belongs there.
Authentication refresh already moved to the edge to fix a real failure, which raised a more general question about what else is cheap to do early and expensive to do late. Most of what is written about the edge is about latency; the part worth understanding is about responsibility.
What work belongs at the edge for reasons other than latency?
EXP-06
Revisiting two production decisions a year on, to see which simplifications aged well.
Every decision in Scene 03 chose the simpler operational shape, and simple choices are the ones most likely to be outgrown quietly rather than loudly. Reading them again with a year of production behind them is cheaper than learning the same thing from an incident.
When does simplicity become technical debt?
The systems above are documented. The rest is a conversation.
The best engineering conversations usually begin with a production problem.