There is a specific kind of meeting that every technology leader wishes they didn’t have to have. It is one in which a client informs you that the platform cannot do what their business actually needs. It is not a feature request or a wish list item, but a structural limitation that no amount of configuration changes or workaround can resolve!
At FUTUROOT, we had such a meeting last year. A mid-market manufacturer had uploaded roughly 140 million process events into our platform. The volume is predictable as shift changes, machine state transitions, quality checkpoints, logistics handoffs: modern factories generate event data at a blistering pace, even for mid-sized businesses.
However, what happened next caught us off guard. Query response times measured in seconds were stretching toward minutes. Dashboards that process owners relied on for daily operational decisions began timing out.
It was a clear call to action for improvement, and our decision to tackle it set in motion 21 days of the most concentrated engineering work we have done as a company.
A Common Assumption About DB Architectures
Process mining emerged primarily in academic computing environments. In early enterprise deployments, data volumes were modest, and query patterns were predictable. Therefore, the approach was straightforward — storing event logs in relational tables, indexing on case ID and timestamp, running conformance and performance queries across those indexes. This approach, although elegant in theory, is prone to failure at scale!
The problem is not the relational model itself, but the fact that event data has a structure that conflicts with relational assumptions at scale.
Process events are append-only in practice. They are queried almost always by case journey, not by individual attribute. And critically, the analytical patterns that generate business value, such as cycle time distributions, bottleneck identification, and variant analysis, require repeatedly reconstructing sequences across tens or hundreds of millions of rows.
Standard columnar warehouses handle this better than row-oriented databases. However, even columnar storage encounters friction when the query is for drill-down visibility into a case’s journey rather than an aggregated view.
The industry is not unfamiliar with this limitation. The acknowledgement is that the architecture will scale well in most cases. However, in the mid-market context, where data volumes are large, and budgets are too constrained to absorb the cost of expensive workarounds, things might go sideways.
The global process mining market is growing at a 34% CAGR, with accelerated adoption in the mid-market. The customers arriving now will not be bringing clean, curated datasets, but rather the raw operational exhaust of genuinely complex businesses. The architecture decisions made today will determine whether those customers stay or leave within eighteen months.
How FUTUROOT Made a Breakthrough in Three Weeks
Let me explain what made those three weeks defining for Team FUTUROOT. We had a mandate with a constraint: the client’s data could not move to a different storage system. They had compliance requirements, data residency considerations, and an IT governance framework that made migration off their existing infrastructure a challenge on any reasonable timeline.
However, that constraint, which initially felt like a wall, turned out to be an innovation pathway!
With no option to change the storage, we turned our focus to reimagine the query layer, and we started asking questions about pre-computation we had never asked before — which analytical results should be materialised at ingest time rather than computed at query time? In process mining terms: instead of reconstructing case journeys on every query, what if the journey graph were assembled incrementally as events arrive, and stored in a structure optimised for traversal rather than for tabular retrieval?
What emerged over those three weeks was not a new database but a new layer of intelligence sitting between the event log and the query engine.
The core idea is straightforward: as events are loaded, they are simultaneously used to update a compact, traversable representation of the case graph. Queries for variant analysis, bottleneck detection, and path frequency do not access the raw event log. Instead, they traverse the graph. While the raw log serves as the source of truth for auditability, the graph serves as the source of performance.
The results were measurable and immediate. Queries that had been running for over two minutes returned in sub-seconds, and dashboard load time dropped by more than 80% —outcomes made possible simply with the arithmetic of not recomputing what we already know!
Customer Reality Guiding Innovation
As a technology leader, I believe that the phrase ‘customer-driven innovation’ is highly generalised (and sometimes misused) in the industry today. More often than not, it is used to describe a process where you build what you planned to build, then confirm that customers wanted it.
Genuine customer-driven innovation, on the other hand, is messier, more disruptive, and frankly more valuable.
The distinction is important because it determines the source of inspiration. Internal roadmaps are optimised for plausibility. They reflect what the engineering team believes is achievable within a sprint cycle, what the product team believes customers will respond to positively, and what the commercial team believes can be positioned competitively.
While all of those filters are useful, none of them generates the kind of pressure that forces a rethink of foundational assumptions. It is where client reality plays its part. A business in pain is not asking to reconsider the architecture. They are indicating, through their operational distress, exactly where the assumptions failed to anticipate the real world!
The 2023 State of Data Engineering report found that 67% of enterprise data teams cited ‘query performance at scale’ as their most persistent infrastructure challenge. The problem that our manufacturing client ran into also pivots on performance. In fact, it is the central experience of organisations that have committed to data-driven operations and are now discovering that the infrastructure beneath them was not designed for the volumes they are generating.
Here, the mid-market companies are particularly exposed because they lack the engineering resources to build bespoke solutions and the leverage to demand custom architecture from enterprise vendors. Here, the gap between vendor promises and operational reality is widest. Addressing that gap with genuine architectural responses rather than configuration workarounds is both a commercial opportunity and a responsibility that comes with serving this segment.
The Lessons Learned
The capabilities built during the engagement went into general availability on FUTUROOT six months after the initial sprint. But the real takeaway of those three weeks was a set of principles that have since reshaped how we approach architectural decisions across the platform. I would list them as follows:
- Optimise for the query pattern, not the storage model.
While this sounds obvious, in practice, it requires resisting the massive gravitational pull of convention! Columnar storage became the default for analytical workloads because it is genuinely excellent for aggregate queries across large datasets. However, it is not the right tool for graph traversal at the case level. Knowing that distinction and being willing to innovate around it is the difference between a platform that performs and one that merely functions.
- Pre-computation is a first-class design decision, not a performance optimisation.
In most development cultures, caching and pre-computation are tools for solving a performance problem. However, in a system where query patterns are known and data arrives in predictable structures, pre-computation should be built in from the beginning, not as an afterthought.
- Client escalations are architecture reviews in disguise.
When a client tells us something is not working, they are not just reporting a bug but telling us about assumptions hard-coded in the system that do not match operational reality. Treating those conversations as architectural input, rather than as support tickets to be resolved and closed, changes what can be learned from them.
The Process Mining Infrastructure That the Mid-market Truly Deserves
Mid-market companies have, for too long, been treated as a segment that needs enterprise capability but must accept enterprise compromise, scaled down. The assumption is that architecture designed for large enterprises will accommodate mid-market volumes by default. However, that is not the case, and that is why the mid-market company often lacks the technical resources to diagnose the failure, the commercial leverage to demand a fix, and the financial runway to absorb the disruption.
Here, process mining platforms like FUTUROOT serving this segment have an obligation to build an architecture from scratch that is honest about where it will break. Mid-market operations generate substantial data volumes and deserve an architecture that accounts for failure modes and strategies to manage them.
The Conversation We Must be Having
The argument here is not that every process mining vendor has hidden architectural debt they are refusing to acknowledge. However, the industry conversation needs to adequately acknowledge the distinction between scale at the enterprise tier, where dedicated engineering teams, significant infrastructure budgets, and bespoke deployments absorb the friction, and scale in the mid-market, where none of those buffers exists.
FUTUROOT’s experience with the mid-market manufacturing client allowed us to meet head-on a standing assumption: that our event log architecture would handle the volume growth of even the most data-intensive clients without architectural intervention. Uncomfortable as it was, it allowed FUTUROOT to rethink and evolve our database architecture at market speed.
A product team that learns from a client’s operational pain and builds it into the architecture is doing something fundamentally different from one that routes that pain through the support queue. While one offers a solution, the other is merely adding to the workaround backlog!
FUTUROOT has and will always choose the former approach to treat customer pain at its root, rather than offering a band-aid.
Related Blogs
