When BI Hits the Memory Wall: Rethinking Performance in an Era of Scarce RAM
What happens to Qlik Sense’s in-memory model when RAM is no longer cheap or endlessly available? We compare it to TURBOARD’s hybrid architecture using Qlik’s own documentation and our granted patent on intelligent caching.
For most of the last decade, business intelligence platforms were built on a comfortable assumption: memory would keep getting cheaper, denser, and easier to throw at problems. That assumption is quietly breaking. AI infrastructure buildouts are absorbing an unprecedented share of global DRAM and HBM supply, and major memory manufacturers have warned that constraints are likely to deepen through 2026. For enterprise BI teams, the practical effect is simple — the RAM your analytics platform depends on is becoming more expensive, more contested, and harder to scale on demand.
This changes the question that matters.
The traditional BI performance question was: how fast is the dashboard when the dataset fits in memory? It’s the question vendors love, because the answer is almost always “very fast.” But it’s no longer the question that decides whether a platform will hold up in production. The real question now is: how does the platform behave when data volumes grow, concurrent users multiply, and memory is the resource you can’t simply buy more of?
That question has two very different answers in the market today. Qlik Sense represents the mature in-memory school — load everything into RAM, keep it there, and let an associative engine deliver speed through proximity. TURBOARD represents a hybrid school — treat memory as an acceleration layer for the workloads that benefit most, and route everything else to columnar storage or live sources designed for the job.
Both philosophies work. The interesting question is which one ages better as the conditions around it change. This post walks through what actually happens to each architecture as data grows, users increase, and RAM gets tight — using Qlik’s own documentation and TURBOARD’s published architecture as the evidence.
Two architectures, briefly
Qlik Sense is built around the QIX associative engine. When an application loads, the unaggregated dataset is held in RAM, along with the associative structures that link every value to every other value. User selections are answered by traversing those in-memory structures, and calculation results are cached in RAM so subsequent identical requests return instantly.
Qlik’s own QVD format accelerates the loading layer dramatically — Qlik documentation reports QVD reads as ten to one hundred times faster than reads from other sources — but the runtime model still requires the working dataset to live in physical memory.
TURBOARD takes a different path. Data can be accessed through optimized live connections using native database drivers, or imported into a high-performance columnar store such as MariaDB ColumnStore, ClickHouse, or Vertica. Frequently used query results are cached in Redis, the in-memory data structure store.
A cache-trigger mechanism decides when cached results are still valid and when the source needs to be re-queried. Memory is used deliberately, for the workloads where it produces the most acceleration — not as the default home for all analytical data.
The diagram above shows how TURBOARD’s layers fit together. The rest of the post examines what each architecture does as the conditions around it get harder.
The fair scenario: when in-memory wins
It’s worth saying clearly: when the dataset fits comfortably in RAM, when concurrency is modest, and when the environment is sized for the application, Qlik Sense is genuinely fast. The associative engine is a mature piece of engineering, and the user experience of clicking through filters and watching aggregations recalculate in milliseconds is, on its day, excellent. For departmental deployments, focused analytical applications, and workloads where the data model is stable and well-understood, the in-memory approach is a real strength.
This is the scenario most product demonstrations are built around. It’s also the scenario that tells you the least about how a platform will behave a year into production, after the data has grown, the user base has expanded, and three more applications have been deployed onto the same server.
The scaling scenario: where the architectures diverge
Qlik’s own support documentation describes the memory model in detail. The engine operates against two configurable thresholds known as Working Set Low and Working Set High, with default values of 70% and 90% of physical RAM respectively. Below the low threshold, the engine retains cached calculation results aggressively, on the principle that unused memory is wasted memory. Once the low threshold is crossed, the engine begins evicting cached results to make room for new ones, prioritizing eviction by age, size, and original calculation time. Once the high threshold is crossed, caching effectively stops, and the operating system’s pagefile may be engaged to keep the engine alive. Qlik’s documentation notes that pagefile use can degrade performance significantly.
The mechanics of what happens between those thresholds matter more than the thresholds themselves. Every cached result that gets evicted is a calculation that will need to be recomputed the next time a user requests it. As eviction accelerates, computational load shifts from memory to CPU. In a high-concurrency environment, where many users are simultaneously triggering calculations that no longer have cached answers, the CPU becomes the new bottleneck — and unlike memory pressure, CPU pressure manifests directly as user-visible latency: dashboards that hang, filters that take seconds to apply, sessions that time out.
TURBOARD’s hybrid approach distributes the load differently. The unaggregated dataset does not need to occupy RAM, because the columnar store is designed to answer analytical queries efficiently from disk — reading only the columns a query touches, exploiting heavy compression, and serving aggregations at speeds that would have required in-memory processing a decade ago. Incremental loading keeps the columnar store fresh without forcing repeated full reloads, which means users experience near-live data without the memory cost of holding the full dataset in RAM. The Redis cache then accelerates the queries that benefit most from acceleration — which, in any real enterprise deployment, is a small subset of total query volume. The next section explains why.
Live data: the Direct Discovery problem
Qlik’s answer to workloads that exceed available memory is Direct Discovery, a mode in which dimension fields are loaded into memory while measure fields remain in the source database and are queried on demand. The design intent is reasonable; the documented limitations are significant.
Documented limitations of Direct Discovery (per Qlik’s own help materials):
- Single shared database connection for all users of an application — can flood the source database under high concurrency.
- Generated SQL is not optimized; joins between in-memory tables and Direct Discovery tables can produce very large IN clauses that strain database query buffers.
- Several core analytical capabilities are unsupported, including Set Analysis, Insight Advisor, and Dynamic Views.
- No further development is planned to address these limitations, per Qlik’s own statement.
For TURBOARD, live connectivity is not a workaround for memory pressure. It’s a first-class part of the architecture. Native database drivers — rather than generic ODBC connections — provide direct, high-performance access to source systems, including big data platforms like Apache Kudu. The cache-trigger mechanism prevents the source database from being hit on every user click: TURBOARD watches a designated trigger field (such as a last-updated timestamp) and serves cached results from Redis when the underlying data has not changed, querying the source only when the trigger indicates fresh data is available. Advanced analytical features remain fully available whether a widget is reading from cache, from the columnar store, or from a live source. A single dashboard can mix all three without compromise.
The Pareto insight, and the patent that protects it
In any enterprise BI deployment, user behavior follows a predictable pattern. Roughly 80% of users return repeatedly to the same core dashboards — executive summaries, monthly performance reports, regional rollups. The remaining 20% are power users running ad-hoc queries that may never be run again in the same form. The implication for memory management is direct: not all cached results are equally valuable, and a system that treats them as equivalent is wasting its most expensive resource.
Qlik’s eviction model is behavior-blind. Cached results are removed based on age, size, and original calculation time — not based on how often actual users actually request them. When memory fills, the engine cannot distinguish the dashboard the entire executive team checks every morning from a one-off query that happened to be cached recently.
TURBOARD takes the opposite approach. Every time a report is viewed, its popularity score is incremented in a Redis Sorted Set — a data structure designed for exactly this kind of ranked-access pattern. The system maintains a continuous leaderboard of which reports are actually being used, and uses that leaderboard to decide what stays in cache when memory is tight. The most-requested reports are protected in Redis, where they return with zero latency to the majority of users who depend on them. Power-user queries that miss the cache are routed to the columnar store or the live source — both of which are engineered to answer those queries quickly without displacing high-value cached content.
Artificial Intelligence-Powered Rapid Search Result Display System with Adaptation to User Habits
This mechanism is the subject of a granted patent. The system was filed by E-Kalite (TURBOARD’s parent company) in 2022 and granted in 2025. The patent covers the method of accelerating data display on the user interface, shortening input/output loading times, and reducing the frequency of access to data-storing and data-transmitting hardware — in other words, the combination of usage-based scoring, intelligent cache refresh, and trigger-based staleness detection that lets TURBOARD serve enterprise workloads on memory budgets that would push purely in-memory architectures into pagefile territory.
The strategic point is straightforward: when memory is abundant, behavior-blind caching costs you efficiency. When memory is scarce, it costs you performance.
Side by side
| Question | Qlik Sense Approach | TURBOARD Approach |
|---|---|---|
| Where does the data live at runtime? | In RAM, in full | In columnar store or live source; results cached selectively in Redis |
| What happens when memory fills? | Cache eviction by age/size, then pagefile | Behavior-scored retention; overflow served from columnar or live source |
| How is repeated usage handled? | In-memory result cache, evicted blindly | Redis cache, retained by usage score (patented) |
| How is data freshness maintained? | Full or QVD-incremental reloads into RAM | Incremental columnar loads + cache triggers |
| How are live queries handled? | Direct Discovery, with documented limits | Native drivers, full analytics retained |
| Scaling philosophy | Provision more memory | Use memory selectively, route the rest |
Conclusion
Qlik Sense remains a capable in-memory analytics platform, and in the conditions it was designed for — bounded datasets, predictable concurrency, generous memory provisioning — it performs well. The question this post has tried to answer is what happens as those conditions become harder to meet. Data volumes are not shrinking. Concurrency expectations are not relaxing. And the cost and availability of memory, for the first time in a long time, are moving in the wrong direction.
TURBOARD’s architecture is a bet that the next decade of BI performance will be won by platforms that use memory intelligently rather than abundantly. Hybrid data access, columnar storage, native live connectivity, and a patented usage-scored cache let TURBOARD deliver enterprise-scale performance on memory budgets that purely in-memory architectures struggle to live within.
For organizations facing the collision of growing data, rising concurrency, and tightening hardware economics, that difference in approach is no longer academic. It’s the difference between a platform that scales with the business and a platform that scales with the procurement budget.
Ready to See the Difference?
If you’re evaluating Qlik Sense — or already running it and beginning to feel the memory pressure described in this post — we’d love to show you how TURBOARD’s hybrid architecture handles the same workloads.
Book a walkthrough with our team and see the platform in action with your own data. No pressure — we’d rather you find the right fit for your specific needs.
You can also explore our Full BI Comparison page to see how we stack up across even more enterprise scenarios.
Latest Blogs
Stop building dashboards, and start getting answers. Experience the next era of BI with our leading GenAI capabilities ‒ request your personalized free demo today.