I keep seeing the same argument in microservice estates, and it usually gets framed as a tooling choice. Should we pin every NuGet version in one place with central package management, or lean on feature flags so each service can ship new behaviour whenever it likes? Wrong framing. They don't compete. They control two different kinds of change, and most of the pain I see comes from using one to do the other's job.

Here is the split. Central package management, the `Directory.Packages.props` file with `ManagePackageVersionsCentrally` switched on, decides **what code is compiled into your binaries**. A feature flag decides **which compiled code runs right now**. One acts at build time, the other at runtime. Obvious when stated that way, and yet teams routinely try to roll out a risky library upgrade by hiding it behind a flag, which can't work.

Why not? Because the flag can't un-load an assembly. If service A upgrades a shared client library from 4.x to 5.x, the 5.x types are in the process whether the flag is on or off. Static initialisers run. Transitive dependencies get resolved. If 5.x drags in a newer `System.Text.Json` that changes a default, every serializer call in that service sees it, flag or no flag. You can gate the call site. You cannot gate the dependency graph.

So I'd say this plainly: **a feature flag is not a safe way to adopt a package upgrade.** It is a safe way to adopt a behaviour change that lives in code you own.

That sounds like a point for central package management, and for a single repo it mostly is. One file, one version of each package, a dependency bump is one diff that touches every project at once. Drift between projects disappears. Nobody ends up with three services on three patch levels of the same logging package, each failing in a subtly different way.

The cost is the blast radius. One edit to `Directory.Packages.props` changes the build of every service under that root. In a monorepo of, say, thirty services, a bump that breaks two of them blocks a merge that the other twenty-eight wanted. Central pinning trades per-service autonomy for consistency, and in microservices autonomy was the whole reason you split things up.

Which brings me to the part where I take a side. For services that deploy independently and are owned by different teams, I'd put the central file **per team or per domain, not per repository tree**. Shared across the things that genuinely upgrade together, separate across the things that don't. A single global props file for forty services is a monolith wearing a microservice costume. It just moved the coupling from the runtime to the restore step.

Feature flags have their own version of this trap. They are cheap to add and expensive to remove. Each flag doubles the paths through a function, and a service with ten live flags has, in principle, 1,024 configurations of which you tested maybe four. The flags that stay around past their rollout are the dangerous ones, because nobody remembers which combination production actually runs.

My rule of thumb: a flag gets an owner and an expiry date at creation, and the build warns when it's past due. A flag with no expiry is configuration pretending to be a release mechanism.

Now the interesting case. What about when you need both?

Say you're moving a service onto a new major version of a library and the new version changes behaviour you depend on. The sequence that has worked for me goes like this:

1. Bump the package in the narrowest central file that covers the service. Nothing else changes yet.
2. Keep the old code path and add the new one beside it, selected by a flag defaulting to old.
3. Ship. At this point the new library is loaded but the behaviour is unchanged. That is the entire point of splitting the step in two.
4. Flip the flag for a slice of traffic, watch, widen.
5. Delete the flag and the old path in the next release.

Notice that step 1 and step 4 are different risks on different timelines. If something goes wrong at step 3, a rollback is a redeploy. If something goes wrong at step 4, a rollback is a config change taking seconds. Mixing them into one release means a bad outcome can't be attributed to either, and you end up bisecting in production.

This only works when the library lets old and new behaviour coexist in one process. When it doesn't, say a wire-format change on a shared contract package, no flag will save you. That needs a compatibility window: publish a version that reads both formats, upgrade every consumer, and only then start writing the new one. Slower, yes. Still the only honest option.

Shared contract packages are where central management earns its keep in a way people undervalue. If services exchange messages through a common contracts library, then the version of that library is effectively part of your API. Letting each service float on its own version there is how you get a producer writing a field the consumer's older copy silently drops. Pin those centrally and bump them deliberately. Let leaf libraries, the ones that touch only internal code, drift per service if teams want.

One more thing I'd push back on. People say flags give you "independent deployability." They give you independent *activation*. Deployability is about the build and the artifact, and it's dependency versions that tie services together there. A fleet where every service is flag-driven but all share one pinned dependency set still has to move in lockstep for anything that matters.

So the decision isn't flags versus central management. It's two questions you ask separately. Does this change alter what gets compiled in? Then it's a package decision, scoped as narrowly as your ownership boundaries allow. Does it alter what runs? Then it's a flag, with an owner and a date.

Get those the right way round and the tradeoff mostly dissolves. Get them swapped and you'll be debugging a "flagged-off" feature that is still breaking production, because the thing that broke was never behind the flag.

I keep seeing the same argument in microservice estates, and it usually gets framed as a tooling...

Central package management and feature flags solve the same problem at opposite speeds

Tummala Krishna Kishore

Someone on the team asked me this week whether we should swap our Service Bus trigger for an HTTP one, so a single request could tell us where a file is. The file leaves Azure, lands on-premises, and gets forwarded from there to a customer's site. Compliance says the report has to come from the on-prem machine, not from us in the cloud. Fair. But the question hid something more interesting than a trigger choice.

Ask which Well-Architected pillar gets cut first under deadline pressure and the honest answer is operational excellence. Nobody schedules it. Nobody demos it. Cost gets a dashboard, security gets a review gate, reliability gets a diagram with two regions on it. "Can I see where this file is?" gets a shrug and a promise for phase two.

Then phase two arrives as a single question in a chat thread.

## Why the HTTP idea is tempting

I understand the pull. One request in, one answer out. The caller asks "what happened to file X?" and the response walks the whole chain: Azure side, on-prem hop, customer delivery. No extra storage. No eventual anything. It feels honest.

It is also the wrong shape, and I'd say so even before looking at latency. The request would have to stay open across a link we don't control, to a machine we don't operate, delivering to a customer whose site may be down for maintenance on a Tuesday afternoon. Every timeout you configure becomes a lie. Set it at 30 seconds and a slow transfer looks like a failed one. Set it at 10 minutes and you're holding threads and gateway connections hostage to a customer's network.

The real cost isn't slow responses. It's that the status question and the transfer have been welded together, so you can no longer answer "what happened?" without redoing the waiting.

## Keep the events, add a status record

The Service Bus design isn't the problem. It's missing a read side.

Give every file a correlation ID at the moment it enters the flow. Each hop, when it finishes or fails, writes one small record: ID, stage, timestamp, outcome. Put those in something cheap and queryable. Table Storage or Cosmos DB will do; I wouldn't reach for a relational database here, because the access pattern is "all records for this ID, in order" and nothing else.

Now the HTTP endpoint you wanted exists after all. It just reads the record instead of chasing the file. `GET /files/{id}/status` returns in milliseconds and says "delivered to customer at 14:02" or "on-prem received, forward pending since 13:40". The expensive part, waiting, stays asynchronous where it belongs. If a caller really needs to block, give them the standard 202 Accepted plus a status URL and let them poll it. That's boring. Boring is what you want from a status API.

## The compliance hop changes who writes the last record

Here is where the original constraint matters, and where I'd push back on treating it as a nuisance. If reporting must originate from the on-prem side, then the on-prem agent is the authority on the final two stages. The cloud shouldn't guess. So the on-prem component pushes its own status events back to Azure, outbound only, over HTTPS to a small ingest endpoint or onto a Service Bus queue of its own. Outbound-only is also the easier conversation with the customer's network team, which counts for more than people admit.

What the cloud side must never do is infer success. "Sent to on-prem" is not "delivered". Our status page should say exactly which of those it knows, and for how long it has known nothing newer.

That last part is the one I'd defend hardest. A status record with a stale timestamp is a signal. A file that has sat at "on-prem received" for four hours is a support ticket you can open before the customer calls. Add an alert on stage age, per stage, with thresholds you can argue about: ten minutes for the on-prem forward, an hour for customer delivery, say. You will tune them wrong the first time. Tune them anyway.

## The counterpoint that actually matters

The status record is a second source of truth, and second sources of truth drift. An agent crashes after forwarding but before writing its record, and now the log says pending for a file the customer already has. Make the writes idempotent, make the on-prem agent re-report on restart, and accept that for a short window the status can lag reality. If a file's status could ever cost someone money or a compliance finding, add a reconciliation job that compares against the customer-side acknowledgement. I would not skip it for a regulated flow.

I still prefer that drift to the synchronous alternative. Drift is bounded and repairable. A request chain that spans three organizations' networks fails in ways you cannot even enumerate.

## What I'm doing about it

We're keeping the Service Bus trigger. Correlation ID at ingress, a stage-record table, an outbound status feed from the on-prem agent, and one GET endpoint over the top. The whole read side is maybe two days of work, which is the part that stings: we only got to it because someone asked in a chat thread, not because anyone planned it.

If operational excellence is the pillar that gets cut first, the cheapest defence is to treat "where is it now?" as a feature with a ticket, written in the first sprint, not a question you answer in the last one.

Someone on the team asked me this week whether we should swap our Service Bus trigger for an HTTP...

Operational excellence is the Well-Architected pillar that loses first, and file-transfer status shows why

Here's the design question that keeps coming back on our SaaS platform: one queue, shared by every customer, sitting between the cloud product and whatever the customer runs on-prem. Is that one queue enough? Scalable enough, reliable enough? I think the answer is yes for a surprisingly long time. What I don't think survives is the dashboard most teams put on top of it.

Start with the plumbing, because the on-prem link is where the decision actually gets made.

There are two ways to get the cloud talking to a customer's data centre. You can have the cloud call in over HTTP, a trigger that hits an endpoint on their side. Or you can put a broker in the middle, Service Bus say, and let the on-prem agent reach out and pull. I'm firmly in the second camp. Customers' firewall teams will say no to inbound holes. They rarely object to an outbound connection on 443 from one agent. And an HTTP trigger has an ugly failure mode: when the customer's site is down for maintenance, your caller either blocks, retries in a tight loop, or drops the call. A queue just holds the message.

That last point is the whole value. The queue turns "the customer is offline" from an incident into a backlog.

Now the multi-tenant part. A single queue for all customers is cheap and simple, and the usual objection is noisy neighbours: one customer bulk-loads fifty thousand messages and everyone behind them waits. That's real. But it's a fairness problem, not a capacity problem, and the two get different fixes. Capacity you buy: Premium tier, more messaging units, partitioning. Fairness you design: session IDs keyed on tenant, so ordering holds per customer and a slow consumer for one tenant doesn't block the others' sessions.

Sessions have a cost. A session-enabled queue means every receiver has to accept a session before it reads anything, and a tenant's session is locked to one receiver at a time. That caps single-tenant throughput at what one receiver can chew through. For most B2B integration traffic that's fine. For a tenant who really does send in bursts, give them their own queue. I'd rather have a shared default and a handful of promoted exceptions than start with queue-per-tenant and own a thousand entities in the deployment pipeline.

So far so ordinary. Here's where I think teams get hurt.

They wire up an alert on active message count. Over 5,000, page someone. It feels right. It's wrong in a multi-tenant queue, and wrong in a specific way: it measures the sum of everybody, and the sum hides the one person who is actually broken.

Picture it. Customer A's on-prem agent lost its credentials on Tuesday afternoon. Their messages pile up, 300 of them. Total queue depth is 300 on a queue that normally runs at 2,000 during business hours. No alert. Nothing looks unusual. Customer A finds out their data hasn't synced for two days when they phone support on Thursday.

Depth fails you in both directions. A big batch from a healthy tenant trips the alarm while every message is being consumed on time. A stuck tenant stays under the threshold forever because the rest of the queue is quiet.

The metric I want instead is the age of the oldest unprocessed message, per tenant. A stuck tenant shows up in minutes, because their oldest message keeps getting older while every other tenant's resets constantly. A bulk load from a healthy tenant shows high depth but a low, flat age. Age separates "busy" from "broken", and depth cannot.

Getting it isn't free. Service Bus doesn't hand you per-session oldest-age as a built-in metric, so you stamp the enqueue time and the tenant ID on every message, and emit the delay from the consumer when it completes one. That gives you processing latency per tenant, which is already useful. For the stuck case, where nothing is completing at all, you need a second signal: a small scheduled check that peeks the head of the queue and flags any tenant with a message older than your threshold. Peek doesn't lock anything, so it's safe to run every minute. Messy, yes. Still much cheaper than the Thursday phone call.

Dead-letter needs the same treatment. A total dead-letter count of 40 means nothing until you know 38 of them belong to one tenant whose payloads started failing schema validation after a release. Tag it, group by tenant, alert on a tenant crossing a number rather than the queue crossing one.

I'll admit the counterpoint with some weight behind it: per-tenant tracking adds cardinality, and metrics platforms bill on it. If you have ten thousand tenants, per-tenant time series is its own cost problem. Fine. Track the top offenders, or sample. But even at that scale, I'd take the oldest-age-by-tenant on the worst twenty over a perfect global depth chart.

On reliability, the answer to "one queue for all customers" comes down to three choices, and none is exotic. Turn on duplicate detection, since the on-prem agent will retry and you don't want a customer's order processed twice. Set max delivery count deliberately, because the default of 10 means a poison message cycles ten times before dead-lettering, and in a session queue that stalls the whole tenant while it does. And make the on-prem consumer idempotent regardless, because duplicate detection only covers the window you configured.

Where does that leave the original question? One queue is fine to start. HTTP triggers into customer networks are the thing I'd avoid. But the part that deserves your design time isn't the topology at all. It's whether, on a quiet Tuesday, you would find out that one customer has been silently stuck since lunch.

If your current alert can't answer that, the queue is not the thing to fix.

Here's the design question that keeps coming back on our SaaS platform: one queue, shared by every...

Queue depth lies when one Service Bus queue serves every customer

Someone asked me last week to stop users typing backticks into a product-name field. Fine. Then someone asked for the same thing on a host-address field, then a description field. Three tickets, same shape, three weeks apart. That's the moment a form-validation ask stops being a bug fix and starts being a design question: do I bolt a fourth one-off regex onto a fourth page, or do I finally admit this is a pattern and give it a name?

We'd already answered that question once, for a different reason, and built a generic field-validation engine instead of scattering regex checks across pages. It lives in `fieldValidation.ts` as a rule registry, plus a `useFieldValidation` hook every form imports. Each field gets a small config object (`required`, `minLength`, `pattern`, `restrictedChars`, whatever it needs), and the hook hands back live sanitization, blur-time errors, and a submit-time gate. No new regex logic per page. That part isn't the interesting decision. The interesting decision is the rule we set for *adding to* the registry, and it's the same rule that governs a much bigger call: when do you carve a new bounded context out of an existing service, versus keep extending the one you have?

Here's the rule, stated plainly, because plain rules are falsifiable and vague ones aren't: **don't add a new named preset to the registry until the same logic is needed in two or more places.** Before that threshold, you route the requirement through `custom`, an escape hatch that takes a function and returns an error string or null. One-off logic goes through `custom`. Only once you've written functionally the same `custom` function twice do you graduate it into a first-class rule kind with its own key in the table.

This sounds obvious written down. It is not obvious in practice, because the pressure runs the other way. The moment you build one generic engine, every future request looks like it deserves to live inside it. That's the whole point of building the thing. Someone will ask for phone-number formatting and you'll be tempted to add a `phoneNumber` preset on request one, because you already have the registry open in your editor and adding a case feels cheap. It isn't cheap. Every named preset is a permanent addition to the table other developers now have to read, understand, and choose correctly between, forever. A registry with nine well-chosen rule kinds is a tool. A registry with thirty barely-distinguished rule kinds, half of them used exactly once, is a maze. The `custom` escape hatch exists specifically to absorb that pressure: it lets you say yes to the requirement today without saying yes to permanent API surface today.

Restricted characters were the exception that proves the rule, and it's worth being honest about why. `restrictedChars` shipped as a named preset from day one, before any second request existed, because we already knew, with high confidence, that every field touching a name or free-text description in this app was one incident away from someone pasting a backtick or a quote into it. That's not premature abstraction. That's abstraction based on a claim you can actually defend: "every text field in this category needs this," not "this one field might need this again someday." The distinction that matters isn't first-use-versus-second-use as a mechanical counter. It's whether you can name the population of future callers with a straight face. If you can, build the preset now. If you're guessing, wait for the second ticket.

That's the exact same test that decides whether a new feature deserves its own bounded context or belongs inside an existing one. Nobody splits out a reporting service the first time someone asks for a report. You extend the orders service, ship it, move on. The split becomes obviously correct the moment reporting's actual shape (its scaling profile, its data-access pattern, its release cadence) diverges from the service it's living inside, and that divergence is almost never visible from request one. It shows up on the second or third time you're fighting the host service's deploy schedule to ship a reporting-only change. Same test, wildly different stakes: a validation preset used wrong costs you a slightly messier table. A bounded context extracted wrong, or extracted too late, costs you months of migration and a service boundary drawn around the wrong seam.

The part people get backwards is thinking restraint is the safe default and formalizing is the risky move. It's the opposite, in both directions, depending which side you're guessing on. Refusing to name a pattern you can already see clearly (insisting on three more `custom` callbacks that are secretly the same function with different variable names) is its own tax, paid in duplicated logic and inconsistent error messages across pages. Naming a pattern before you can see it is a different tax, paid in an API surface nobody asked for and a table entry that quietly becomes dead weight. Both failure modes look identical from the outside: an engineer added complexity. Only the timing tells you which one happened.

One thing the `custom` hook made easier than I expected: cross-field validation. A `confirmPassword` check needs `ctx.allValues.password`, and because `custom` receives the whole values object, not just its own field's value, that fell out of the escape hatch for free — no separate cross-field validation path bolted on later. That's a small tell that the escape hatch was designed with enough headroom to actually defer decisions, rather than just deferring them until you're forced to special-case them anyway. An escape hatch that can't handle the requirements you're deferring isn't an escape hatch, it's a delay.

So the actual discipline isn't "prefer simplicity" or "prefer generality," which is the kind of answer that fits on a poster and helps nobody at 4pm on a Thursday. It's: know which side of the two-or-more-instances line you're standing on, for the thing in front of you, right now. If you're not sure, that uncertainty is itself the answer. Route it through the escape hatch and find out.

Someone asked me last week to stop users typing backticks into a product-name field. Fine. Then...

Don't add a new rule kind until you've written the same check twice

I approved a pull request last month in about four minutes. Tests passed, the diff was clean, the checklist was all green. By every mechanical measure it was a fine review. 

It was also a rubber stamp, and I didn't notice until three weeks later, when someone tried to reuse the same "quick regex check" pattern for a fourth field and the file had turned into a pile of copy-pasted validation logic.

That's the gap I want to talk about.

## The Test for Real Design

A design review that only asks "does this work" isn't reviewing the design. It's reviewing the implementation. Those are different questions, and conflating them is how teams end up with codebases full of individually reasonable decisions that add up to a mess.

Here's the test I use now, and it comes straight from a validation engine I built recently for a set of React forms that kept accumulating one-off regex checks. 

The engine's rule registry currently has ten kinds:
* required
* minLength
* maxLength
* pattern
* forbiddenPattern
* restrictedChars
* numeric
* email
* custom
* sanitize

Every field a page needs validated gets one entry in a rules object, and a single hook wires up live sanitization, blur-time errors, and a submit-time gate off that same object. 

On paper, that's a nice piece of generalized infrastructure. The interesting part isn't the registry, though. It's the rule I attached to growing it: **a new named preset only gets added once the same logic is needed in at least two places.** Until then, you reach for the `custom` escape hatch and write it inline.

Nobody asked me to write that rule down. I added it after watching the opposite happen on a different project, where a "flexible" config object grew a new top-level key for every caller's pet requirement until it had thirty optional fields, and reading any one form's config meant mentally filtering out the twenty-six that didn't apply. 

Generality that isn't load-bearing yet is just clutter with extra steps.

## Catching Premature Generalization

This is exactly where most design reviews go quiet. Someone proposes a new rule kind — say, a `crossFieldMatch` preset because a signup form needs "confirm password." 

The review checks whether the regex works, whether it handles empty strings, whether there's a test. All reasonable. None of it asks the question that actually matters: 
> Is this the second caller, or the first? 

If it's the first, `custom` already does the job, and adding a named preset for one consumer is premature generalization dressed up as good architecture. It reads fine in the PR. Eighteen months later it reads as bloat, because nobody remembers why there are three ways to express the same rule.

A rubber-stamp review can't catch this. Catching it requires context the diff doesn't contain — how many other places in the codebase actually need this, and whether "two" is real or aspirational. That's not something a linter flags. It's judgment, and supplying it instead of a checklist is the entire point of having a design review at all.

I'll grant the obvious objection: *doesn't gatekeeping every new preset behind "prove you need it twice" slow teams down?* 

A little, sometimes. Someone writes the same three-line custom validator twice instead of importing a preset once. I think that cost is worth paying, because the alternative — a registry with fifteen presets, six of which exist for a single caller each — is worse and much harder to undo. 

You can always promote a duplicated `custom` block into a named preset once the duplication is real. You can't un-invent a preset that three other teams have already started depending on for reasons that have nothing to do with why you built it.

## Separating Logic from Framework Glue

There is a second question a real review should ask, and almost never does: **where does the pure logic live relative to the framework glue?**

In this engine, `fieldValidation.ts` holds nothing but rule definitions and pure validator functions — no React, no hooks, no component state. `useFieldValidation.ts` is the only file that touches React at all, and its entire job is wiring those pure functions into a component's lifecycle: `errors`, `sanitizeOnChange`, `validateField`, `validateAll`. 

That split wasn't originally deliberate. It happened because the pure functions were easier to unit test without a rendering harness. Only later did it become the reason the validation logic could survive a framework migration untouched. 

A rubber-stamp review doesn't ask "will this still work if the framework changes," because that question doesn't affect whether today's PR merges. A real design review asks it anyway. Today's PR is never just today's PR — it's a small vote on the shape of the next fifty PRs that get built on top of it.

None of this shows up in a green CI run.

That's the actual argument for keeping design review distinct from code review, rather than collapsing both into a single "LGTM" gate. Code review answers whether something is correct. Design review answers whether it's the right shape, and whether you're generalizing at the right moment — and that second question needs someone in the room asking it out loud, because the diff itself will never volunteer the answer.



I approved a pull request last month in about four minutes. Tests passed, the diff was clean, the...

The difference between a design review and a rubber stamp

*What infrastructure drift, frontend validation, and shared engineering patterns have in common.*

Somewhere in your infrastructure right now, a resource doesn't match what your Terraform state file claims.

I'd put money on it.

Maybe it's a firewall rule someone added through the portal during an incident at 2 a.m. and never ticketed. Maybe it's a tag that got renamed by a policy remediation task outside the pipeline.

Doesn't matter which.

The shape of the bug is always the same:

> There is a declared source of truth, and there is an easier path around it. Eventually, someone takes the easier path.

I spent last month staring at the exact same bug in a codebase that has nothing to do with infrastructure.

## The Same Problem, Wearing a Different Hat

A teammate and I built a generic field-validation engine for the React forms in one of our applications.

The idea was straightforward: a rule registry.

Each form declares a small configuration object:

- This field is required.
- This one has a maximum length of 100.
- This one only allows digits.
- This one needs a custom cross-field check against another field's value.

We ended up with around ten rule types covering almost everything we needed:

- Required
- Minimum/maximum length
- Pattern matching
- Forbidden patterns
- Restricted-character presets
- Numeric validation
- Email validation
- Custom validation
- Explicit sanitization for live input stripping

A hook wires everything together:

- Errors
- Sanitize-on-change
- Validate-on-blur
- Validate-all-on-submit

Any page that needs field validation imports the hook, defines its rule configuration, and it works.

It's a good pattern.

I'd build it again.

The whole point was to stop every page from reinventing its own bespoke regex checks scattered throughout the codebase — each slightly different, each with its own bugs.

Except there is one problem.

The escape hatch that makes the engine flexible is the same door that lets it drift.

## The Escape Hatch Is the Problem

Nothing stops a developer, six months from now, in the middle of a sprint, from writing a raw regex check directly inside a field's `onChange` handler instead of routing it through the rule registry.

It compiles.

It works.

For that one field, in that one moment.

Nobody's build breaks.

QA doesn't flag it because the field still validates correctly from the user's point of view.

But something important has happened.

There is now a validation rule living outside the registry.

The registry doesn't know about it.

The next developer won't find it when they look at the shared configuration to understand what's enforced and where.

It won't automatically get the live-sanitization behavior or the shared error-display wiring.

The declared source of truth — *"these are the validation rules for this application"* — quietly stopped being true for one field.

Nothing announced it.

That's drift.

Not a compile error.

Not a test failure.

A fact that becomes false without anyone deciding it should.

## Terraform Drift Is the Same Problem

Terraform drift works the same way, mechanically, even though the domain couldn't be more different.

You declare desired state in `.tf` files.

Someone with portal access — an on-call engineer under pressure, or someone doing *"just this one quick thing"* — changes a resource directly.

Your infrastructure has now diverged from its declared configuration.

In a clean environment, `terraform plan` should show zero unexpected differences.

The moment it doesn't, your declared infrastructure state no longer completely describes what is actually running.

And just like the validation engine, the manual portal edit usually works fine in isolation.

Nothing breaks that day.

It breaks three months later.

Someone runs an apply based on stale assumptions, and the plan output doesn't match what's actually running.

Or worse, the apply "corrects" a change someone made deliberately — and nobody remembers why that change was made anymore.

That's the dangerous part of drift.

The original change wasn't necessarily wrong.

The problem is that the system lost track of the fact that the change existed.

## Why Process Alone Doesn't Work

Here's the part I think most teams get wrong about drift, in both the infrastructure sense and the code sense:

**They try to solve it with process.**

Code review.

A wiki page that says:

> "Please use the shared validation hook."

A runbook that says:

> "Please don't edit resources manually. Always go through the pipeline."

But review didn't catch the inline regex.

It looked like a normal, working field check.

Nobody reviewing that PR was necessarily thinking about the registry's completeness as a system property.

The runbook doesn't stop the 2 a.m. portal edit either.

In the moment, going through a slower deployment pipeline felt like the wrong call to the person holding the pager.

That's understandable.

And that's exactly why the process fails.

**Process asks people to consistently make the right choice under pressure.**

Engineering should make the right choice easier — and the wrong choice visible.

## Process Detects Drift Only by Accident

Process catches drift only when someone happens to be looking at the right diff at the right time.

That's luck with a policy document stapled to it.

What actually works is either removing the choice altogether or, at minimum, making drift loud and automatic instead of quiet and manual.

For infrastructure, that means scheduled drift detection.

For example:

```bash
terraform plan -detailed-exitcode
```

Run it on a schedule against every managed environment, not just before a deliberate deployment.

The important part isn't simply running the command.

The important part is what happens when it detects a difference.

A non-zero drift result shouldn't become a dashboard tile that nobody opens.

It should trigger an actionable workflow.

Depending on the environment, that might mean:

1. Create an incident or ticket.
2. Notify the owning team.
3. Capture the plan output.
4. Identify whether the change was intentional.
5. Either reconcile the Terraform configuration or revert the manual change.

The goal isn't to prevent humans from ever making mistakes.

The goal is to reduce the time between **drift occurring** and **someone knowing about it**.

## We Had the Same Problem in Frontend Code

For our validation engine, the fix was considerably less glamorous.

A lint rule.

We flag an `onChange` handler in a form component when it contains a raw regex literal.

The reasoning is simple:

> If you're writing pattern-matching logic directly inside a form component, you've probably reinvented something the shared validation registry already does.

That doesn't mean every regex inside a component is inherently wrong.

It means bypassing the shared mechanism should be deliberate.

The lint rule makes the bypass visible.

And that's the important part.

## You Don't Need to Eliminate the Escape Hatch

Neither fix is complete.

The lint rule can be silenced with an inline comment.

A developer can still find ways around infrastructure controls.

Terraform itself has mechanisms such as targeted operations that can be used to bypass parts of a normal workflow.

But that's not necessarily a failure.

The escape hatch exists for a reason.

Sometimes you genuinely need it.

The mistake is making the escape hatch completely frictionless.

If bypassing the shared path costs nothing, people will eventually bypass it whenever the local problem feels easier to solve that way.

The useful middle ground is:

> **Make friction proportional to how much a bypass should hurt to justify.**

Total lockdown kills the flexibility that made the registry or pipeline worth building in the first place.

Total permissiveness turns "declared state" into a suggestion.

The engineering sweet spot is somewhere between the two.

## Look for Drift Beyond Infrastructure

This is the part I think is worth taking beyond Terraform.

If your team runs drift detection on infrastructure but nowhere else that has this shape:

**A declared shared source of truth + a legitimate escape hatch**

then you're only watching for the bug in the one place where you've already been burned by it.

Look around your systems.

### Feature flags

Is there a central feature-flag system, but developers can quietly introduce local configuration switches?

### Configuration

Is configuration supposed to come from a centralized registry, but individual services maintain their own hidden defaults?

### Permissions

Is there a central permissions model, but individual endpoints implement additional authorization logic outside it?

### Validation

Is there a shared validation framework, but components can bypass it with custom checks?

### API contracts

Is there a canonical schema, but services add undocumented assumptions that aren't represented in it?

### Deployment configuration

Is the pipeline the source of truth, but someone occasionally changes production configuration manually?

These are all variations of the same problem.

The technology changes.

The failure mode doesn't.

## The Bigger Architectural Pattern

The pattern is worth naming because once you recognize it, you start seeing it everywhere.

You have:

**1. A declared source of truth**

Something says:

> "This is how the system is supposed to behave."

**2. A convenient escape hatch**

Something allows someone to say:

> "I'll just change this one thing directly."

**3. No automatic reconciliation**

Nothing checks whether reality still matches the declaration.

**4. Delayed consequences**

The change works today.

The problem appears weeks or months later.

That's drift.

And drift isn't fundamentally a Terraform problem.

Terraform just happens to make the problem particularly visible.

## Build Systems That Notice When Reality Changes

The strongest systems aren't necessarily the ones that prevent every deviation.

They're the ones that make deviations observable.

A developer might still need a custom validation rule.

An operator might still need to make an emergency infrastructure change.

A production incident might still require bypassing the normal deployment path.

That's reality.

The architectural question is:

**What happens after the bypass?**

Does the system know it happened?

Does someone get notified?

Does the change become part of the declared source of truth?

Is there an explicit reconciliation step?

Or does the system simply move on and hope everyone remembers?

That's the difference between a controlled escape hatch and silent drift.

## The 2 a.m. Test

Here's a simple test you can apply to your architecture.

Find a system with a central source of truth.

Then ask:

> **"What happens if someone bypasses it at 2 a.m.?"**

If the answer is:

*"They shouldn't do that."*

you don't have a control.

You have a policy.

If the answer is:

*"The system will detect the deviation, make it visible, and force reconciliation,"*

you have an engineering control.

That's a much stronger place to be.

Because people will take the fast path.

Especially when they're under pressure.

The goal isn't to pretend they won't.

The goal is to make sure the fast path doesn't quietly become the new source of truth.

Go find the other places in your system where declared state can quietly become fiction.

Before they find you at 2 a.m.

What infrastructure drift, frontend validation, and shared engineering patterns have in...

The same drift bug that breaks your Terraform state also breaks your login form

Three boxes. That's what our C4 container diagram showed for the EPCIS batch upload flow in ALSC. Upload API. Processing Service. Notification Service. Clean. Presentable. Wrong in the way that actually costs you time during an incident.

Here's what one upload really does. A user drops an EPCIS batch file in the browser. That request lands on a Node.js Function. The Function writes the file to blob storage and drops a message on Service Bus. A .NET Function picks that up, does its work, and hands off to a Hangfire background job. The Hangfire job does more processing and puts more messages back on Service Bus for the next stage. Count them: browser request, Node Function, blob write, Service Bus message, .NET Function, Hangfire job, another Service Bus hop. Seven independent executions for one click. Three boxes on the diagram.

Nobody drew that. Nobody was going to, either. Not because the team was lazy. Because C4's container level was never built to answer "what actually runs, in what order, for this one business action." It answers a different question: what are the deployable units, and how do they relate to each other structurally. That's a fine question. It's just not the question you need answered at 2am when a batch upload fails somewhere in the middle of seven hops and nobody can say which one.

I want to be fair to C4 here, because the framework gets blamed for a gap it never claimed to fill. Dynamic diagrams exist precisely for this: sequence-style views of a specific runtime scenario, layered on top of the static container view. That's the textbook answer. In practice, on this project, we never drew one until an incident forced the question. I'd guess most teams doing async, queue-heavy work are in the same position. The container diagram gets maintained because it's the artifact everyone points to in design reviews. The dynamic view, the one that would have actually shown the seven hops, doesn't get the same attention because nothing forces it. Until something breaks.

And when something breaks, the cost of not having it is specific and boring: a support ticket references a failed batch, and resolving it means pulling in whichever four teams own a piece of that chain, each one grepping their own logs around a rough timestamp, hoping the clocks are close enough to correlate a batch ID by eye. That's not a hypothetical. That's what used to happen here before we fixed it.

The fix was distributed tracing with OpenTelemetry, and the part worth writing down isn't "we added tracing," it's the specific hop that made it necessary rather than nice-to-have. Hangfire's Redis-backed queue does not carry request context by default. A job dequeued by the Hangfire server has no idea which upload triggered it unless that link gets made explicit, in code, on purpose, by whoever wires the enqueue call. Every other hop in the chain (HTTP call, Service Bus message) has some natural place to smuggle a correlation ID through. Hangfire's queue doesn't hand you that for free. It's the kind of gap that only shows up once you actually try to trace something end to end, which is exactly why it never showed up on the container diagram either.

Once the trace ID actually propagates across all seven hops, the same incident resolves differently. Instead of four people grepping four log stores against a guessed time window, you pull one trace ID, open Application Map, and read the path the request actually took. What used to burn half a day now takes the time it takes to type an ID into a search box.

Worth saying plainly, though: turning on distributed tracing everywhere, always, is not free, and I'd push back on any advice that pretends otherwise. Trace volume maps directly to Application Insights ingestion cost. Blanket always-on tracing across every service, every environment, every request, is a line item, not a checkbox. We ended up building the ability to switch tracing on for a specific service only when investigating something, without a redeploy, rather than leaving it running everywhere by default. That's a governance decision as much as a technical one, and it belongs in the same conversation as the tracing rollout itself, not bolted on after finance asks about the Application Insights bill.

So here's the actual claim, not the softened version: a C4 container diagram that stops at "these are our services and how they call each other" is systematically going to undercount the real failure surface the moment you introduce a queue, and queues are exactly where async architectures live. Three boxes versus seven runtime hops isn't a rounding error — it's the difference between a diagram that looks complete and one that would have actually helped during the incident that eventually forced us to build tracing anyway.

If I were redoing the design review for this system today, I'd make one change to how we document it: no container diagram for an async pipeline gets signed off without a companion flow diagram showing at least one real end-to-end scenario, hop by hop, including the ones that don't carry context automatically. Not every scenario. One. The one that actually matters — usually the longest chain, or the one with the least obvious hop, like a Redis-backed queue that quietly drops your correlation ID on the floor unless you catch it.

The diagram wasn't wrong because it lied about anything on it. It was wrong because it stopped one layer too early, right where the actual failure surface starts. Draw the hops. The boxes were never the hard part.

Three boxes. That's what our C4 container diagram showed for the EPCIS batch upload flow in ALSC....

My container diagram had three boxes. The actual pipeline had seven hops.

"Property with the same name already exist." That's the whole error. No stack trace worth reading, no line number, just APIM refusing a PUT and a pipeline stuck red at 2pm.

The instinct was to blame the script. It's always the instinct. Somebody had touched the apiops publisher config two days earlier, so that got the first look, and it was clean. The real cause was quieter and, honestly, more interesting than a script bug: Azure API Management named values carry two separate identifiers, and almost nobody designs around the fact that only one of them is actually enforced.

There's `id`: whatever apiops derives from the artifact folder name during extraction. And there's `name`, a display property living inside that resource, which APIM enforces must be unique across the *entire service*, independent of `id`. Two axes, one enforced, one not. Somewhere in one environment, a named value already existed with `name: key1`. It had been created manually through the portal months earlier, sitting under a completely different `id` than the one our publisher was trying to write. APIM saw the collision on `name` and rejected the write. From the pipeline's point of view, this looked exactly like corruption. It wasn't corruption. It was two people, at two different times, both being reasonable, and the platform not telling either of them about the other.

I've now seen this shape of bug in three different systems this year, and every time it traces back to the same design-review gap.

## The question nobody asks

When a design doc introduces a resource, reviewers reach for the obvious questions. Does it scale? What's the failure mode? Who owns it in prod? Fine questions, all of them, and none of them would have caught this.

The question that would have caught it is smaller and gets skipped almost every time: **does this resource have more than one identifier, and if so, which one is actually unique, and at what scope?**

Most engineers assume "the ID is unique" and stop there. But plenty of platforms (APIM named values being one, but far from the only one) split identity into a system-assigned key and a human-facing name, and enforce uniqueness on the one you weren't watching. Git branch names inside a protected-branch rule. DNS record names inside a zone. Feature-flag keys versus their internal flag IDs. Environment variable names across shared app-service slots. Same pattern every time: one field is globally unique, the other is scoped or arbitrary, and the design doc almost never says which is which.

That gap survives review because it doesn't fail during review. It fails three weeks later, in a different environment, when someone who's never read the design doc creates something through a portal instead of a pipeline. By then nobody's connecting the failure back to a design decision at all. It just looks like an ops incident.

## What actually would have helped

Not more process. A one-line addition to the design doc template: for every resource with a compound identity, state explicitly which field is unique, at what scope, and what happens on collision.

That's it. It's a sentence, sometimes two. But writing that sentence forces the author to actually go check — which means they find out, before shipping, that `name` is global and `id` is local, instead of finding out from a failed pipeline run with a cryptic 409.

I'd also push back on the instinct to "fix" this by locking down the portal so nobody can create named values manually again. We talked about it. It's the wrong fix. Manual creation isn't the bug. Undocumented shared-identity scope is. Lock the portal and you've removed one path to the collision while leaving the actual landmine in place for the next person who imports a named value some other way, or renames a folder that changes the derived `id`. Governance theater instead of a fix.

What we actually did: added an idempotency check to the publisher step. Look up by `name` before attempting the PUT. If a named value with that name exists under a different `id`, fail loudly with the actual conflicting `id` in the error, not APIM's generic message. Twenty minutes of work. It would have turned a 45-minute production debugging session into a five-second, self-explanatory pipeline failure.

## Why this belongs in a Node/Python conversation, not just an Azure one

It's tempting to file this under "Azure quirks" and move on. Don't. The exact same failure mode shows up constantly in backend services outside Azure entirely. A Python service that treats a Stripe customer ID and an internal user ID as interchangeable until a webhook arrives out of order. A Node service that keys a cache on a slug that's supposed to be unique but is only enforced unique per-tenant, not globally, until two tenants pick the same slug and one silently overwrites the other's cache entry.

Compound identity is one of the most common shapes a design doc's blind spot takes, across every stack I've worked in. It's not an Azure problem wearing an Azure name. It's a resource-modeling problem that happens to have surfaced there this time.

If you review backend design docs, yours or someone else's, this is worth adding as a fixed line item, not something you trust yourself to remember to ask ad hoc under deadline pressure. "What's unique, and at what scope, for every identifier this resource exposes." One question. It would have saved us a debugging session, and it's cheap enough to ask on every doc, not just the ones that feel risky.

"Property with the same name already exist." That's the whole error. No stack trace worth reading, no...

The one design-review question that would have caught our APIM naming collision

Health checks. Structured logging. Retry policies with jitter. Everyone who writes a "production readiness checklist" for a .NET Core service reaches for the same list, and none of it is wrong. But it's the wrong list to reach for first if your services live inside a shared domain model.

I spent a chunk of this month wiring up something that's not on anyone's checklist: turning our domain projects into versioned NuGet packages, pushing them to a private Artifactory feed, and building a bot whose entire job is bumping the version number and rippling that bump out to every microservice repo that consumes it.

It sounds like plumbing. It is plumbing. But it's the plumbing that decides whether "production-ready" means anything across a fleet of services instead of just one.

Here's the problem it solves. Once you factor shared domain logic — entities, value objects, business rules that more than one service needs — out of a monolith and into a library, you've created a dependency graph you didn't have before. Every consuming service now has an opinion, implicit or not, about which version of that library it's running. Before we packaged it properly, that opinion was enforced by copy-paste and tribal memory. Nobody actually knew which service had which version of which rule.

NuGet fixes the packaging half of that. Artifactory gives you a private feed so the package doesn't leak outside the org and you get an audit trail of who pulled what, when. Neither of those pieces is interesting on its own. The interesting part, and the part that actually determines whether this approach helps you or quietly wrecks you, is the bot.

The bot watches the domain package repo. When a new version ships, it opens a PR against every downstream microservice repo, bumping the reference. That's the whole job. No cleverness, just a scripted dependency bump that used to be a manual chore somebody forgot to do half the time.

And here's where I'll disagree with the take you see in half the "microservices done right" posts: automating that bump is good, but only if you admit you've just built an internal Dependabot, and internal Dependabot needs the same discipline the external kind does — discipline most teams skip because "it's our own code, we'll catch it."

You won't catch it. Not reliably.

If the bot opens the PR and someone rubber-stamps merges on domain package bumps the way they rubber-stamp a patch bump on a logging library, you are one mislabeled semver bump away from a breaking change landing in a dozen-plus services on the same afternoon. A shared domain library isn't a leaf dependency. It's load-bearing. A major-version bump there deserves the same scrutiny as a schema migration, because functionally that's what it is.

So the real production-readiness question isn't "do we have a bot that bumps versions." It's "does every consuming service have a CI gate that runs its own integration suite against the new package version before that PR is mergeable — not after." We didn't have that gate on day one. We had the bot open the PR, a green build meaning "it compiles," and a merge button sitting right there. That's not readiness. That's speed dressed up as readiness.

Getting the gate right meant slowing the bot down on purpose. It still opens the PR immediately, but it no longer auto-merges past a minor version bump until the consuming service's full integration suite passes against the new package — not just its unit tests, the whole suite, including the slow ones nobody likes running. That one change moved the failure mode from silent breakage in production to a red CI check somebody has to look at. Not exciting. Correct.

There's a real trade-off here, and I won't pretend it away: this slows down how fast a domain fix reaches every service. In the old copy-paste world, a bug fix in a shared value object reached whichever service someone remembered to update, whenever they remembered — slow, but nobody's build broke overnight because of it. In the new world, the fix propagates fast to every service wired up to the bot, and any service where the fix collides with a local assumption finds out immediately, in CI, instead of three weeks later in an incident channel.

I'd take the second failure mode every time. Fast and visible beats slow and invisible. But the cost is real, and pretending otherwise is how these systems lose the team's trust the first time a bump breaks a build at 5pm on a Friday.

One more thing worth saying plainly: none of this replaces the usual checklist. A service still needs health checks, a sane retry policy, and logs that actually correlate across a request. What packaging and version-bump automation buys you is a production-readiness checklist that's true across the whole fleet, not just true for the one service you happened to test carefully. A service can pass every item on that list in isolation and still go down the moment a shared library changes underneath it in a way nobody flagged.

If you're building shared .NET domain libraries and haven't worked out who reviews a major version bump, what test suite has to pass before that bump merges, and who's accountable when the bot's PR breaks something two services downstream, you don't have a production-ready fleet. You have a production-ready service, once, and a bot quietly deciding when that stops being true.

Health checks. Structured logging. Retry policies with jitter. Everyone who writes a "production...

What a .NET Core service needs before it's actually production-ready

Every landing zone conversation eventually lands on the same fork: subscription-per-team or subscription-per-environment. Pick one, defend it in the design review, move on. I used to think that fork was the decision. It isn't. It's a proxy for a question nobody actually asks, which is whether your environments are things you *have* or things you *make*.

We hit this a few weeks back building out a branching strategy for a product that needed to ship in phases — not one big release, several overlapping ones, each needing its own slice of infrastructure to validate against before merging up the chain. The instinct in every Azure reference architecture I'd read was to stand up a fixed set of subscriptions — dev, qa, uat, prod — and let teams queue for access to the shared ones. Clean org chart. Terrible fit for what we were actually doing, which was running three or four release branches at once, each wanting its own isolated environment for a week or two and then wanting to disappear.

So, we didn't fix the environments. We templated them.

The branching strategy and the environment strategy ended up being the same decision, which surprised me at the time. Every release branch got a corresponding environment spun up from Terraform, wired into the pipeline automatically the moment the branch existed, torn down automatically once it merged or went stale. No ticket. No "can I borrow QA for the afternoon." The subscription-per-environment debate assumes environments are scarce and long-lived enough to be worth governing carefully as static things. Once provisioning is cheap and repeatable, that assumption just stops holding.

Here's the number that made the case for me, not some theoretical justification. Before this, we were running four non-prod subscriptions, always on, mostly idle outside business hours, because nobody wanted to be the one who tore down an environment someone might still need. After moving to on-demand provisioning tied to branch lifecycle, we typically had one to two environments alive at any given moment instead of four sitting there permanently. That's not a rounding error. That's roughly half the standing footprint gone, for infrastructure that was doing the same job.

I won't pretend this is free. Ephemeral environments push a lot of discipline requirements onto your IaC that a static model lets you get lazy about. If your Terraform templates aren't actually idempotent — if there's some manual step, some hand-configured firewall rule, some secret someone pasted in once and forgot to script — dynamic provisioning will find that gap for you, repeatedly, at the worst time. We found ours in a Key Vault access policy that had been set by hand eight months earlier and never made it into the template. First time a fresh environment failed a smoke test, it took an afternoon to figure out why "prod-like" environment #3 didn't behave like the original.

There's also a real counterpoint here, and it's not a strawman: audit and compliance people like fixed environments. A subscription that's existed for two years with a known history is easier to reason about for a SOC 2 review than one that spins up Tuesday and dies Friday. If your organization's compliance posture depends on long-lived resource identity — not just "the config was correct" but "this specific resource has been continuously monitored since provisioning" — ephemeral environments genuinely complicate your story. That's a real cost. I'm not going to hand-wave it away with better logging. You need to decide whether your compliance framework cares about resource continuity or just about proving the right controls were in place at time of use and design your audit trail around whichever one it actually is.

For us, it wasn't. The controls mattered, the specific subscription's birthdate didn't. So, the trade-off was worth making.

What I'd push back on is treating subscription-per-team vs subscription-per-environment as the primary architectural decision in the first place. It's a structural question about naming and boundaries. The actual lever — the one that changes your cost profile, your pipeline design, and how fast a team can validate a branch before it merges — is whether "environment" means a durable thing you provision once and then manage, or a disposable output of your pipeline that happens every time you need it. Landing zone docs spend pages on the former and barely mention the latter, probably because it's harder to draw on a slide. A static subscription topology is easy to diagram. "Environments are ephemeral outputs of the release pipeline" doesn't fit neatly into an org chart, so it gets left out of the reference architecture even when it's the more honest answer for a fast-phased release model.

If you're mid-debate on subscription topology right now, my actual advice is to shelve that question for a week and instead ask whether your team could provision a working non-prod environment from scratch, unattended, in under half an hour. If the answer is no, fix that first — the subscription boundary question becomes almost irrelevant once you can. If the answer is yes, you've probably already made the topology decision without realizing it, because ephemeral environments don't care much which subscription boundary they sit inside. They just need somewhere to land and a template that doesn't lie about what production actually looks like.

Every landing zone conversation eventually lands on the same fork: subscription-per-team or...