Update, October 2026: the pre-computed counts described at the end of this post are now in production. Here’s what shipped.
The question nobody actually answers
In March we wrote about why the hard part of a financial data hub isn’t the pipeline — it’s the accounting. Since then the hub has grown a ten-prong automated validation framework, native row-level security, an operations dashboard, and a period-balance engine that reconciled against decades of accounting history.
It’s now in production.
But before you take a financial data platform live, there’s a question every vendor waves away with the phrase “near real-time”:
When someone posts a transaction, exactly how long until the data hub knows?
Not roughly. Not “typically.” Exactly — at every hop, under load, with evidence.
Yes, you can load-test a general ledger safely
That sentence stops most conversations, so let’s deal with it first.
We built a reversible load generator. It pushed sustained change traffic, with deliberate bursts, at five to ten times the busiest hour ever observed in production. Every change it made to the financial data was automatically reversed and verified afterward. The test environment ended the campaign byte-for-byte identical to how it started.
Alongside it we built a latency harness that timed each stage of the journey — source commit, change-data-capture ingestion, warehouse consolidation, automated processing, finished query-ready data — without requiring elevated privileges in the warehouse.
Two instruments, one week, a full copy of the production pipeline.
What the numbers said
Ingestion is not the bottleneck. Changes left the source system and landed in the warehouse’s receiving area in 4 to 17 seconds, even at ten times normal load.
Freshness has exactly one dial. Nearly all of the end-to-end wait came from a single consolidation setting in the replication layer — a number we control. Typical commit-to-queryable time: 2 to 4 minutes under controlled load. In production, where traffic is burstier and less predictable than anything we could synthesize, we observe 2 to 8 minutes. Need it faster? That’s a configuration change, not a redesign.
The pipeline is self-healing under pressure. At peak overload, work queued briefly — then the system drained its own backlog and returned to normal speed. Two campaigns, zero failures, zero lost records, zero human intervention.
The smallest warehouse size held. No congestion, no upsizing required. Performance headroom is budget headroom.
Why this should matter to you
If your organization is being sold a “near real-time” data platform, ask for the hop-by-hop numbers.
If nobody can produce them, the SLA conversation you’ll eventually have with your business stakeholders is being deferred to the worst possible moment: after go-live, during an incident.
Our take: measure before you promise. A one-week testing campaign turned three open architecture debates into closed decisions, gave the client’s testers realistic expectations in writing, and documented exactly how the platform behaves on its worst day — before production ever saw it.
What’s next, and what we’ll publish about it
With the hub in production, the next milestone inverts the problem. Instead of asking how fast data can get in, we’re looking at pre-computing answers on the way in — so the application’s most expensive screen, a login-time summary that today fires off a couple of dozen live queries, reads a handful of already-counted rows instead.
We’ll publish the numbers for that one too, once we’ve measured them.
Which is the entire point of this post.
Ready to know your numbers?
You don’t need more dashboards claiming “real-time.” You need architects who can tell you — with evidence — how many seconds sit between a keystroke and an insight, and which dial to turn when the business asks for fewer of them.
If that conversation is overdue at your organization, let’s talk.