Try It Now

Author: Vinay Pardeshi

  • When the Customer Is The R&D Lab: Rethinking Database Architecture at Market Speed

    When the Customer Is The R&D Lab: Rethinking Database Architecture at Market Speed

    There is a specific kind of meeting that every technology leader wishes they didn’t have to have. It is one in which a client informs you that the platform cannot do what their business actually needs. It is not a feature request or a wish list item, but a structural limitation that no amount of configuration changes or workaround can resolve!  

    At FUTUROOT, we had such a meeting last year. A mid-market manufacturer had uploaded roughly 140 million process events into our platform. The volume is predictable as shift changes, machine state transitions, quality checkpoints, logistics handoffs: modern factories generate event data at a blistering pace, even for mid-sized businesses.

    However, what happened next caught us off guard. Query response times measured in seconds were stretching toward minutes. Dashboards that process owners relied on for daily operational decisions began timing out.

    It was a clear call to action for improvement, and our decision to tackle it set in motion 21 days of the most concentrated engineering work we have done as a company.

    A Common Assumption About DB Architectures

    Process mining emerged primarily in academic computing environments. In early enterprise deployments, data volumes were modest, and query patterns were predictable. Therefore, the approach was straightforward — storing event logs in relational tables, indexing on case ID and timestamp, running conformance and performance queries across those indexes. This approach, although elegant in theory, is prone to failure at scale!

    The problem is not the relational model itself, but the fact that event data has a structure that conflicts with relational assumptions at scale.

    Process events are append-only in practice. They are queried almost always by case journey, not by individual attribute. And critically, the analytical patterns that generate business value, such as cycle time distributions, bottleneck identification, and variant analysis, require repeatedly reconstructing sequences across tens or hundreds of millions of rows.

    Standard columnar warehouses handle this better than row-oriented databases. However, even columnar storage encounters friction when the query is for drill-down visibility into a case’s journey rather than an aggregated view.

    The industry is not unfamiliar with this limitation. The acknowledgement is that the architecture will scale well in most cases. However, in the mid-market context, where data volumes are large, and budgets are too constrained to absorb the cost of expensive workarounds, things might go sideways.

    The global process mining market is growing at a 34% CAGR, with accelerated adoption in the mid-market. The customers arriving now will not be bringing clean, curated datasets, but rather the raw operational exhaust of genuinely complex businesses. The architecture decisions made today will determine whether those customers stay or leave within eighteen months.

    How FUTUROOT Made a Breakthrough in Three Weeks

    Let me explain what made those three weeks defining for Team FUTUROOT. We had a mandate with a constraint: the client’s data could not move to a different storage system. They had compliance requirements, data residency considerations, and an IT governance framework that made migration off their existing infrastructure a challenge on any reasonable timeline.

    However, that constraint, which initially felt like a wall, turned out to be an innovation pathway!

    With no option to change the storage, we turned our focus to reimagine the query layer, and we started asking questions about pre-computation we had never asked before — which analytical results should be materialised at ingest time rather than computed at query time? In process mining terms: instead of reconstructing case journeys on every query, what if the journey graph were assembled incrementally as events arrive, and stored in a structure optimised for traversal rather than for tabular retrieval?

    What emerged over those three weeks was not a new database but a new layer of intelligence sitting between the event log and the query engine.

    The core idea is straightforward: as events are loaded, they are simultaneously used to update a compact, traversable representation of the case graph. Queries for variant analysis, bottleneck detection, and path frequency do not access the raw event log. Instead, they traverse the graph. While the raw log serves as the source of truth for auditability, the graph serves as the source of performance.

    The results were measurable and immediate. Queries that had been running for over two minutes returned in sub-seconds, and dashboard load time dropped by more than 80% —outcomes made possible simply with the arithmetic of not recomputing what we already know!

    Customer Reality Guiding Innovation

    As a technology leader, I believe that the phrase ‘customer-driven innovation’ is highly generalised (and sometimes misused) in the industry today. More often than not, it is used to describe a process where you build what you planned to build, then confirm that customers wanted it.

    Genuine customer-driven innovation, on the other hand, is messier, more disruptive, and frankly more valuable.

    The distinction is important because it determines the source of inspiration. Internal roadmaps are optimised for plausibility. They reflect what the engineering team believes is achievable within a sprint cycle, what the product team believes customers will respond to positively, and what the commercial team believes can be positioned competitively.

    While all of those filters are useful, none of them generates the kind of pressure that forces a rethink of foundational assumptions. It is where client reality plays its part. A business in pain is not asking to reconsider the architecture. They are indicating, through their operational distress, exactly where the assumptions failed to anticipate the real world!

    The 2023 State of Data Engineering report found that 67% of enterprise data teams cited ‘query performance at scale’ as their most persistent infrastructure challenge. The problem that our manufacturing client ran into also pivots on performance. In fact, it is the central experience of organisations that have committed to data-driven operations and are now discovering that the infrastructure beneath them was not designed for the volumes they are generating.

    Here, the mid-market companies are particularly exposed because they lack the engineering resources to build bespoke solutions and the leverage to demand custom architecture from enterprise vendors. Here, the gap between vendor promises and operational reality is widest. Addressing that gap with genuine architectural responses rather than configuration workarounds is both a commercial opportunity and a responsibility that comes with serving this segment.

    The Lessons Learned

    The capabilities built during the engagement went into general availability on FUTUROOT six months after the initial sprint. But the real takeaway of those three weeks was a set of principles that have since reshaped how we approach architectural decisions across the platform. I would list them as follows:

    • Optimise for the query pattern, not the storage model.

     While this sounds obvious, in practice, it requires resisting the massive gravitational pull of convention! Columnar storage became the default for analytical workloads because it is genuinely excellent for aggregate queries across large datasets. However, it is not the right tool for graph traversal at the case level. Knowing that distinction and being willing to innovate around it is the difference between a platform that performs and one that merely functions.

    • Pre-computation is a first-class design decision, not a performance optimisation.

    In most development cultures, caching and pre-computation are tools for solving a performance problem. However, in a system where query patterns are known and data arrives in predictable structures, pre-computation should be built in from the beginning, not as an afterthought.

    • Client escalations are architecture reviews in disguise.

    When a client tells us something is not working, they are not just reporting a bug but telling us about assumptions hard-coded in the system that do not match operational reality. Treating those conversations as architectural input, rather than as support tickets to be resolved and closed, changes what can be learned from them.

    The Process Mining Infrastructure That the Mid-market Truly Deserves

    Mid-market companies have, for too long, been treated as a segment that needs enterprise capability but must accept enterprise compromise, scaled down. The assumption is that architecture designed for large enterprises will accommodate mid-market volumes by default. However, that is not the case, and that is why the mid-market company often lacks the technical resources to diagnose the failure, the commercial leverage to demand a fix, and the financial runway to absorb the disruption.

    Here, process mining platforms like FUTUROOT serving this segment have an obligation to build an architecture from scratch that is honest about where it will break. Mid-market operations generate substantial data volumes and deserve an architecture that accounts for failure modes and strategies to manage them.

    The Conversation We Must be Having

    The argument here is not that every process mining vendor has hidden architectural debt they are refusing to acknowledge. However, the industry conversation needs to adequately acknowledge the distinction between scale at the enterprise tier, where dedicated engineering teams, significant infrastructure budgets, and bespoke deployments absorb the friction, and scale in the mid-market, where none of those buffers exists.

    FUTUROOT’s experience with the mid-market manufacturing client allowed us to meet head-on a standing assumption: that our event log architecture would handle the volume growth of even the most data-intensive clients without architectural intervention. Uncomfortable as it was, it allowed FUTUROOT to rethink and evolve our database architecture at market speed.

    A product team that learns from a client’s operational pain and builds it into the architecture is doing something fundamentally different from one that routes that pain through the support queue. While one offers a solution, the other is merely adding to the workaround backlog!

    FUTUROOT has and will always choose the former approach to treat customer pain at its root, rather than offering a band-aid.

  • The Lean Signal: Why Mid-market Businesses Should Prioritise Lesser Data for Better Insights

    The Lean Signal: Why Mid-market Businesses Should Prioritise Lesser Data for Better Insights

    There is an irony at the heart of modern process mining: most platforms that promise clarity have been delivering confusion! Sounds outrageous? Let me explain.

    For years, the implicit sales pitch has been maximalist —more event logs, more connectors, more dashboards, more granularity. Enterprises warehoused billions of processed events, built data lakes that resembled data swamps, and paid enterprise-grade licensing bills priced by the terabyte.

    However, in return, many of them received something they did not expect. It’s best termed as analytical paralysis — a state of inertia where decision making slows down, cutting progress.

    The mid-market cannot afford a repetition of that in their process intelligence story. But more importantly, it shouldn’t want to.

    Data obesity is not a sign of analytical maturity. It is a symptom of strategic immaturity

    I have spent a significant portion of my career building process intelligence tools for organisations that operate between the agility of the mid-market and the complexities of the large enterprises. More often than not, we come across lean organisations where the CFO also runs IT, the ops director also serves as the process analyst, and the CTO also serves as the data steward. For such companies, the most valuable thing to offer is not comprehensive but precision and the luxury of saved time!

    The Weight of What We Collect

    The process mining market is projected to reach USD 9.49 billion by 2030, and the velocity of that growth has pulled vendor incentives in a predictable direction: towards volume. Today, most process mining platforms are built to ingest everything. Pricing is structured to reward scale. Implementation timelines can stretch up to 24 months, with a single process analysis in year one costing between USD 50,000 and USD 100,000 when internal resource allocation is factored in. These are enterprise economics applied to mid-market realities.

    The result is what I would describe as data obesity: organisations that have consumed far more process data than they can meaningfully digest, and whose analytics infrastructure is heavy, slow, and expensive to maintain. The condition is widespread. This research shows that 42% of data models built by data scientists are never used within their organisations.

    And yet we keep feeding the machine!

    The cost is not just financial, though the financial burden is real. The deeper implications are cognitive. Leaders experiencing information overload are, according to published research by The Harvard Business Review, 7.4 times more likely to regret their decisions and 2.6 times more likely to avoid making decisions altogether. Process mining tools that surface a thousand inefficiencies at once don’t empower operations teams — they immobilise them.

    What Minimum Viable Data Actually Means

    Minimum Viable Data is neither a compromise nor a consolation prize for organisations that cannot afford an enterprise-grade process mining deployment. It is a philosophy that holds that the right signal, cleanly captured, consistently acted upon, outperforms a noisy dataset by an order of magnitude!

    Consider what process mining fundamentally does: it reconstructs how work actually flows through an organisation, versus how it was designed to flow. The gap between those two realities is where value lives. To identify that gap, you do not need every event log from every system over the last five years, but rather the right events from the right systems across a timeframe that reflects current operational reality.

    The broader shift happening in analytics, from big data to right data, reflects this exact maturation. As one leading industry insider recently noted, “organisations are realising they don’t need to bring all their data to solve a problem — they need to bring the right data.” Indeed, the overwhelming abundance of data has only made it harder to extract the insights that matter.

    On the other hand, if you are a mid-market manufacturer running an ERP, here’s what Minimum Viable Data might mean for you:

    Order-to-ship event logs for the last 18 months, filtered to your top three product lines by revenue and have its conformance checked against your baseline lead-time commitment.

    That is a tractable problem, solving which yields decision-ready insights within hours and not weeks. Therefore, for a mid-market business, it is worth doing even on a busy Tuesday morning rather than keeping it shelved for the next month.   

    The right signal, cleanly captured, consistently acted upon, outperforms a noisy dataset by an order of magnitude.

    Summaries Over Raw Feeds

    There is a related discipline that has been largely undervalued in conversations: the power of the intelligent summary. Most platforms today are built on the assumption that every user wants to explore the full event graph — to drill down, pivot, filter, and slice. While this is valuable for career data professionals and experts, it is of little use to the decision-makers who are making hard choices under pressure!

    For instance, the CFO of a USD 200 million revenue distribution company does not have time to roleplay as a process analyst to get the numbers she needs. She needs hands-on insights on: where is my Order-to-Cash cycle losing more than three days, and what is the most likely root cause? That is a summary which allows her to walk into a conversation with her COO with something actionable rather than something exploratory.

    The discipline of building summary-first outputs requires a fundamentally different product design philosophy. It means pre-computing the conclusions that matter most to specific roles and industries, instead of building an infinitely flexible exploration environment that assumes expertise the user may not have. It needs an intelligence layer alongside the data layer.

    For the mid-market, this fusion is existential. These organisations do not have extensive process mining centres of excellence and in-house data teams. Most of them have a few sharp operators who need to pick up the pace from the very beginning, with minimal handholding and knowledge transfer.

    The Cost Equation That’s Mostly Overlooked

    Enterprise process mining pricing was built for a world where massive data volume is a badge of credibility. There, even an entry-level process mining offering can cost over USD 3000 per month for a single analyst and millions of events. Clearly, this is a domain built for enterprise-scale data appetites.

    However, there is a second cost dimension that rarely surfaces in vendor conversations: compliance costs. GDPR’s data minimisation principle, laid down in Article 5(1)(c), is not just a bureaucratic shackle upon businesses.

    It is an architectural constraint that should be reshaping how every European mid-market company thinks about what process data it stores, for how long, and for what purpose. In 2025, according to IBM, the average cost of a data breach hovered around USD 4.4 million.

    Yes, the figure represents a 9% decrease from the previous year.

    However, organisations holding excessive operational data, much of which contains personally identifiable information embedded in process logs, are still carrying undisclosed regulatory and reputational risk on their balance sheets.

    Therefore, data minimisation, when practised properly in process mining, is ethically sound and financially prudent.

    Smaller, more purposefully curated event logs cost less to store, easier to secure, simpler to audit, and require less explanation to a regulator. They also perform better analytically: reducing redundant, obsolete, and trivial data produces more accurate and reliable insights.

    A Question You Can Lead With

    The conversation the mid-market leaders need to have right now is not “what data should we collect?” but “what decisions do we need to make, and what is the minimum data footprint needed to make them confidently?”

    That pivot changes everything.

    It gives a refreshed look at what you deploy, what you pay, how quickly your team adopts the tool, and how quickly you see a return. It also changes the nature of your vendor relationship from a data maximisation exercise to a machine that churns out actionable insights for your decision-makers.

    At FUTUROOT, we designed our platform around this conviction from the start.

    Our architecture is built on the premise that mid-market business leaders should be able to get a meaningful process diagnostic running in days, not months, and that the outputs should cater to their priorities, not an analyst’s curiosity. For us, the question driving every product decision is: what is the minimum viable data surface that allows this organisation to act with confidence on its most important operational questions?

    The next frontier in process intelligence for the mid-market is not in the watered-down enterprise platforms that retain complex connectors and the biggest event log capacity. Instead, it is in their ability to see with a clearer lens and with the sharpest editorial discipline. The future belongs to mid-market organisations that can curate and consume fast rather than accumulate and wait!

    The next wave of process mining will not come from those with the most connectors. It will come from those with the sharpest lens.

    The Competitive Advantage of Restraint

    Restraint in data collection is not a limitation, but a competitive posture. Mid-market companies that are serious about defining their Minimum Viable Data footprint and building their process intelligence practice around it will move faster, spend less, comply more easily, and make better decisions than their peers who are still waiting to put together all the data.

    The intelligence advantage has never been about who has the most. It has always been about who extracts the most meaning from what they have. Process mining, at its most powerful, is exactly that: the discipline of reading what your organisation’s data is actually telling you, without the noise of everything it isn’t.

    In a world obsessed with volume, that is the genuinely radical position to hold and it is one that the mid-market is uniquely positioned to lead.

  • FUTUROOT’s July 2024 Release: Here’s What We Have in Store

    FUTUROOT’s July 2024 Release: Here’s What We Have in Store

    The Modern Chief Financial Officer is Navigating a Perfect Storm.

    Geopolitical instability is disrupting supply chains, persistent inflation is eroding margins, and a complex web of regulations, from ESG reporting to new data privacy laws, creates a minefield of business risk. In this environment, the traditional tools, like spreadsheets and business intelligence (BI) dashboards, are no longer enough.

    These tools are like a car’s rearview mirror. They tell you that margins dropped last quarter or that audit findings spiked. But they can’t tell you why. They can’t show you the traffic jam building up ahead.

    To navigate today’s landscape, CFOs need a windshield, a forward-looking, real-time view into the operational DNA of the business. This is precisely what process mining delivers. It’s no longer a niche analytics tool, but an essential co-pilot for strategic financial leadership.

    From Post-Mortem Audits to Proactive Assurance

    For decades, risk management and compliance have operated on a cycle of retrospection and assumption. We conduct sample-based audits, find issues months after they occur, and then scramble to fix them.

    However, in a world where a single compliance failure can result in millions of pounds in fines and irreparable reputational damage, this lag is a critical vulnerability. According to a recent study, organisations lose an estimated 5% of their revenue to fraud annually, often due to internal control weaknesses!

    Process mining shifts the paradigm from reactive detection to continuous assurance.

    By tapping directly into the event logs of your core systems (ERPs, CRMs, etc.), it reconstructs every single process as it actually happens, not as it has been drawn on a flowchart.

    For instance, imagine being able to monitor 100% of your transactions for Segregation of Duties (SoD) violations in real-time. Instead of waiting for an auditor to discover that the same employee created a vendor and approved their invoice, you get an alert the moment it happens.

    Process mining platforms doesn’t just flag the Policy Violation Rate. It also quantifies the value-at-risk for each deviation and pinpoints the root cause, whether it’s a system misconfiguration, a training gap, or intentional malpractice.

    This transforms compliance from a periodic, manual checklist into a dynamic, automated, and always-on defence mechanism.

    Turbocharging the Financial Close with Surgical Precision

    The record-to-report (R2R) process, culminating in the month-end close, is the heartbeat of the finance function. Yet for many organisations, it’s a source of chronic pain characterised by a chaotic scramble of late journals, manual reconciliations, and intercompany breaks that extends the Days to Close and inflates the Manual Adjustment Rate

    Traditional reporting might indicate that the close is delayed. However, it can’t tell you that 30% of the delay is caused by one specific team struggling with intercompany reconciliations in a newly acquired subsidiary.

    Process mining provides this control tower or a birds-eye view. It dissects the entire R2R cycle with surgical precision, revealing the hidden bottlenecks and inefficiencies that create friction, like:

    • Late & Back-Dated Journals: Instantly identify entries that bypass the closing calendar, pinpointing their origin and impact.
    • Intercompany Breaks: Flag mismatches the moment they occur and track Intercompany Difference Ageing, allowing teams to resolve high-value breaks before they stall consolidation.
    • Control Overrides: Surface every instance where standard approval workflows or authority limits were bypassed, turning hidden exceptions into transparent, data-backed facts.

    By making the entire process transparent, you can transition from month-end hustle to a controlled, predictable, and ultimately faster close, paving the way for the strategic goal of a near-continuous financial close.

    The Next Frontier: AI-Powered Finance and the Autonomous Enterprise

    The true power of process mining is unlocked when it’s combined with AI and Machine Learning. This is where the finance function moves from diagnosis to prognosis and, eventually, to self-healing.

    • Predictive Process Monitoring: The future isn’t just about seeing what went wrong; it’s about predicting what will go wrong. AI models trained on your process data can identify patterns that signal a future problem. For instance, the system could predict with 95% confidence that a specific high-value invoice will miss its payment deadline based on its initial attributes, allowing the accounts payable team to intervene proactively.
    • Prescriptive Recommendations: The next logical step is for the system not only to predict a problem but also recommend or even automate the solution. Imagine the platform detecting a bottleneck in the approval workflow and automatically rerouting tasks to an available, authorised manager to keep the process moving. It is the foundation of the autonomous enterprise, where processes can dynamically adapt to changing conditions without human intervention.
    • Conversational Intelligence: With the rise of Generative AI, we are stepping into an era where a CFO can simply ask their system about the primary driver of increase in rework during a certain quarter and flag the automation initiatives with the highest ROI. The system would respond not with a dashboard, but with a full root-cause analysis with data-backed recommendations.

    This is the next logical evolution of financial management underpinned by process mining and AI. The organisations that embrace this will build a level of operational resilience and strategic agility that their competitors can only dream of matching!

    For the modern CFO, whose role has expanded to that of a strategic partner and driver of enterprise-wide transformation, relying on outdated tools is akin to flying blind. Process mining provides the necessary instrumentation to visualise, understand, and optimise the intricate processes that underpin financial performance and corporate integrity

    Harnessing this power requires more than just software. It demands a platform built for the complexities of modern finance, implemented by a team cross-skilled in the art of data science, enterprise applications, process automation and AI.

    At FUTUROOT, we are such a team. Our mission is to provide the process intelligence that translates complex operational data into the clear, actionable insights CFOs need to navigate today’s challenges and build the resilient, autonomous finance function of tomorrow.