Someone on the team asked me this week whether we should swap our Service Bus trigger for an HTTP one, so a single request could tell us where a file is. The file leaves Azure, lands on-premises, and gets forwarded from there to a customer's site. Compliance says the report has to come from the on-prem machine, not from us in the cloud. Fair. But the question hid something more interesting than a trigger choice.

Ask which Well-Architected pillar gets cut first under deadline pressure and the honest answer is operational excellence. Nobody schedules it. Nobody demos it. Cost gets a dashboard, security gets a review gate, reliability gets a diagram with two regions on it. "Can I see where this file is?" gets a shrug and a promise for phase two.

Then phase two arrives as a single question in a chat thread.

## Why the HTTP idea is tempting

I understand the pull. One request in, one answer out. The caller asks "what happened to file X?" and the response walks the whole chain: Azure side, on-prem hop, customer delivery. No extra storage. No eventual anything. It feels honest.

It is also the wrong shape, and I'd say so even before looking at latency. The request would have to stay open across a link we don't control, to a machine we don't operate, delivering to a customer whose site may be down for maintenance on a Tuesday afternoon. Every timeout you configure becomes a lie. Set it at 30 seconds and a slow transfer looks like a failed one. Set it at 10 minutes and you're holding threads and gateway connections hostage to a customer's network.

The real cost isn't slow responses. It's that the status question and the transfer have been welded together, so you can no longer answer "what happened?" without redoing the waiting.

## Keep the events, add a status record

The Service Bus design isn't the problem. It's missing a read side.

Give every file a correlation ID at the moment it enters the flow. Each hop, when it finishes or fails, writes one small record: ID, stage, timestamp, outcome. Put those in something cheap and queryable. Table Storage or Cosmos DB will do; I wouldn't reach for a relational database here, because the access pattern is "all records for this ID, in order" and nothing else.

Now the HTTP endpoint you wanted exists after all. It just reads the record instead of chasing the file. `GET /files/{id}/status` returns in milliseconds and says "delivered to customer at 14:02" or "on-prem received, forward pending since 13:40". The expensive part, waiting, stays asynchronous where it belongs. If a caller really needs to block, give them the standard 202 Accepted plus a status URL and let them poll it. That's boring. Boring is what you want from a status API.

## The compliance hop changes who writes the last record

Here is where the original constraint matters, and where I'd push back on treating it as a nuisance. If reporting must originate from the on-prem side, then the on-prem agent is the authority on the final two stages. The cloud shouldn't guess. So the on-prem component pushes its own status events back to Azure, outbound only, over HTTPS to a small ingest endpoint or onto a Service Bus queue of its own. Outbound-only is also the easier conversation with the customer's network team, which counts for more than people admit.

What the cloud side must never do is infer success. "Sent to on-prem" is not "delivered". Our status page should say exactly which of those it knows, and for how long it has known nothing newer.

That last part is the one I'd defend hardest. A status record with a stale timestamp is a signal. A file that has sat at "on-prem received" for four hours is a support ticket you can open before the customer calls. Add an alert on stage age, per stage, with thresholds you can argue about: ten minutes for the on-prem forward, an hour for customer delivery, say. You will tune them wrong the first time. Tune them anyway.

## The counterpoint that actually matters

The status record is a second source of truth, and second sources of truth drift. An agent crashes after forwarding but before writing its record, and now the log says pending for a file the customer already has. Make the writes idempotent, make the on-prem agent re-report on restart, and accept that for a short window the status can lag reality. If a file's status could ever cost someone money or a compliance finding, add a reconciliation job that compares against the customer-side acknowledgement. I would not skip it for a regulated flow.

I still prefer that drift to the synchronous alternative. Drift is bounded and repairable. A request chain that spans three organizations' networks fails in ways you cannot even enumerate.

## What I'm doing about it

We're keeping the Service Bus trigger. Correlation ID at ingress, a stage-record table, an outbound status feed from the on-prem agent, and one GET endpoint over the top. The whole read side is maybe two days of work, which is the part that stings: we only got to it because someone asked in a chat thread, not because anyone planned it.

If operational excellence is the pillar that gets cut first, the cheapest defence is to treat "where is it now?" as a feature with a ticket, written in the first sprint, not a question you answer in the last one.

Someone on the team asked me this week whether we should swap our Service Bus trigger for an HTTP...

Operational excellence is the Well-Architected pillar that loses first, and file-transfer status shows why

Tummala Krishna Kishore

Here's the design question that keeps coming back on our SaaS platform: one queue, shared by every customer, sitting between the cloud product and whatever the customer runs on-prem. Is that one queue enough? Scalable enough, reliable enough? I think the answer is yes for a surprisingly long time. What I don't think survives is the dashboard most teams put on top of it.

Start with the plumbing, because the on-prem link is where the decision actually gets made.

There are two ways to get the cloud talking to a customer's data centre. You can have the cloud call in over HTTP, a trigger that hits an endpoint on their side. Or you can put a broker in the middle, Service Bus say, and let the on-prem agent reach out and pull. I'm firmly in the second camp. Customers' firewall teams will say no to inbound holes. They rarely object to an outbound connection on 443 from one agent. And an HTTP trigger has an ugly failure mode: when the customer's site is down for maintenance, your caller either blocks, retries in a tight loop, or drops the call. A queue just holds the message.

That last point is the whole value. The queue turns "the customer is offline" from an incident into a backlog.

Now the multi-tenant part. A single queue for all customers is cheap and simple, and the usual objection is noisy neighbours: one customer bulk-loads fifty thousand messages and everyone behind them waits. That's real. But it's a fairness problem, not a capacity problem, and the two get different fixes. Capacity you buy: Premium tier, more messaging units, partitioning. Fairness you design: session IDs keyed on tenant, so ordering holds per customer and a slow consumer for one tenant doesn't block the others' sessions.

Sessions have a cost. A session-enabled queue means every receiver has to accept a session before it reads anything, and a tenant's session is locked to one receiver at a time. That caps single-tenant throughput at what one receiver can chew through. For most B2B integration traffic that's fine. For a tenant who really does send in bursts, give them their own queue. I'd rather have a shared default and a handful of promoted exceptions than start with queue-per-tenant and own a thousand entities in the deployment pipeline.

So far so ordinary. Here's where I think teams get hurt.

They wire up an alert on active message count. Over 5,000, page someone. It feels right. It's wrong in a multi-tenant queue, and wrong in a specific way: it measures the sum of everybody, and the sum hides the one person who is actually broken.

Picture it. Customer A's on-prem agent lost its credentials on Tuesday afternoon. Their messages pile up, 300 of them. Total queue depth is 300 on a queue that normally runs at 2,000 during business hours. No alert. Nothing looks unusual. Customer A finds out their data hasn't synced for two days when they phone support on Thursday.

Depth fails you in both directions. A big batch from a healthy tenant trips the alarm while every message is being consumed on time. A stuck tenant stays under the threshold forever because the rest of the queue is quiet.

The metric I want instead is the age of the oldest unprocessed message, per tenant. A stuck tenant shows up in minutes, because their oldest message keeps getting older while every other tenant's resets constantly. A bulk load from a healthy tenant shows high depth but a low, flat age. Age separates "busy" from "broken", and depth cannot.

Getting it isn't free. Service Bus doesn't hand you per-session oldest-age as a built-in metric, so you stamp the enqueue time and the tenant ID on every message, and emit the delay from the consumer when it completes one. That gives you processing latency per tenant, which is already useful. For the stuck case, where nothing is completing at all, you need a second signal: a small scheduled check that peeks the head of the queue and flags any tenant with a message older than your threshold. Peek doesn't lock anything, so it's safe to run every minute. Messy, yes. Still much cheaper than the Thursday phone call.

Dead-letter needs the same treatment. A total dead-letter count of 40 means nothing until you know 38 of them belong to one tenant whose payloads started failing schema validation after a release. Tag it, group by tenant, alert on a tenant crossing a number rather than the queue crossing one.

I'll admit the counterpoint with some weight behind it: per-tenant tracking adds cardinality, and metrics platforms bill on it. If you have ten thousand tenants, per-tenant time series is its own cost problem. Fine. Track the top offenders, or sample. But even at that scale, I'd take the oldest-age-by-tenant on the worst twenty over a perfect global depth chart.

On reliability, the answer to "one queue for all customers" comes down to three choices, and none is exotic. Turn on duplicate detection, since the on-prem agent will retry and you don't want a customer's order processed twice. Set max delivery count deliberately, because the default of 10 means a poison message cycles ten times before dead-lettering, and in a session queue that stalls the whole tenant while it does. And make the on-prem consumer idempotent regardless, because duplicate detection only covers the window you configured.

Where does that leave the original question? One queue is fine to start. HTTP triggers into customer networks are the thing I'd avoid. But the part that deserves your design time isn't the topology at all. It's whether, on a quiet Tuesday, you would find out that one customer has been silently stuck since lunch.

If your current alert can't answer that, the queue is not the thing to fix.

Here's the design question that keeps coming back on our SaaS platform: one queue, shared by every...

Queue depth lies when one Service Bus queue serves every customer

Every landing zone conversation eventually lands on the same fork: subscription-per-team or subscription-per-environment. Pick one, defend it in the design review, move on. I used to think that fork was the decision. It isn't. It's a proxy for a question nobody actually asks, which is whether your environments are things you *have* or things you *make*.

We hit this a few weeks back building out a branching strategy for a product that needed to ship in phases — not one big release, several overlapping ones, each needing its own slice of infrastructure to validate against before merging up the chain. The instinct in every Azure reference architecture I'd read was to stand up a fixed set of subscriptions — dev, qa, uat, prod — and let teams queue for access to the shared ones. Clean org chart. Terrible fit for what we were actually doing, which was running three or four release branches at once, each wanting its own isolated environment for a week or two and then wanting to disappear.

So, we didn't fix the environments. We templated them.

The branching strategy and the environment strategy ended up being the same decision, which surprised me at the time. Every release branch got a corresponding environment spun up from Terraform, wired into the pipeline automatically the moment the branch existed, torn down automatically once it merged or went stale. No ticket. No "can I borrow QA for the afternoon." The subscription-per-environment debate assumes environments are scarce and long-lived enough to be worth governing carefully as static things. Once provisioning is cheap and repeatable, that assumption just stops holding.

Here's the number that made the case for me, not some theoretical justification. Before this, we were running four non-prod subscriptions, always on, mostly idle outside business hours, because nobody wanted to be the one who tore down an environment someone might still need. After moving to on-demand provisioning tied to branch lifecycle, we typically had one to two environments alive at any given moment instead of four sitting there permanently. That's not a rounding error. That's roughly half the standing footprint gone, for infrastructure that was doing the same job.

I won't pretend this is free. Ephemeral environments push a lot of discipline requirements onto your IaC that a static model lets you get lazy about. If your Terraform templates aren't actually idempotent — if there's some manual step, some hand-configured firewall rule, some secret someone pasted in once and forgot to script — dynamic provisioning will find that gap for you, repeatedly, at the worst time. We found ours in a Key Vault access policy that had been set by hand eight months earlier and never made it into the template. First time a fresh environment failed a smoke test, it took an afternoon to figure out why "prod-like" environment #3 didn't behave like the original.

There's also a real counterpoint here, and it's not a strawman: audit and compliance people like fixed environments. A subscription that's existed for two years with a known history is easier to reason about for a SOC 2 review than one that spins up Tuesday and dies Friday. If your organization's compliance posture depends on long-lived resource identity — not just "the config was correct" but "this specific resource has been continuously monitored since provisioning" — ephemeral environments genuinely complicate your story. That's a real cost. I'm not going to hand-wave it away with better logging. You need to decide whether your compliance framework cares about resource continuity or just about proving the right controls were in place at time of use and design your audit trail around whichever one it actually is.

For us, it wasn't. The controls mattered, the specific subscription's birthdate didn't. So, the trade-off was worth making.

What I'd push back on is treating subscription-per-team vs subscription-per-environment as the primary architectural decision in the first place. It's a structural question about naming and boundaries. The actual lever — the one that changes your cost profile, your pipeline design, and how fast a team can validate a branch before it merges — is whether "environment" means a durable thing you provision once and then manage, or a disposable output of your pipeline that happens every time you need it. Landing zone docs spend pages on the former and barely mention the latter, probably because it's harder to draw on a slide. A static subscription topology is easy to diagram. "Environments are ephemeral outputs of the release pipeline" doesn't fit neatly into an org chart, so it gets left out of the reference architecture even when it's the more honest answer for a fast-phased release model.

If you're mid-debate on subscription topology right now, my actual advice is to shelve that question for a week and instead ask whether your team could provision a working non-prod environment from scratch, unattended, in under half an hour. If the answer is no, fix that first — the subscription boundary question becomes almost irrelevant once you can. If the answer is yes, you've probably already made the topology decision without realizing it, because ephemeral environments don't care much which subscription boundary they sit inside. They just need somewhere to land and a template that doesn't lie about what production actually looks like.

Every landing zone conversation eventually lands on the same fork: subscription-per-team or...