Foundations
What is the essence of a good platform in this brave new world of AI? This is a question that doesn't stray far from my mind as we consider what to build, how to achieve that goal, and why we should expend the effort. This are the sorts of questions that resurfaced as i reflected on my previous article in the series, The Mirror Ai Stack: High-Level Architecture, where I introduced out platform at a high level.
I've previously described Mirror Ai as a classical big data problem, focused on solving what we believe to be a key issue in the blockchain space to this day. The traditional definition of big data problems can be distilled into three Vs:
* Volume: There is a lot of it, how do you store it all?
Velocity: It's coming at you so fast, how do you process it all?
* Variety:* There there are so many different types of information, how do you fuse it all into a unified picture?
The truth is, the essence of any good platform is in the data and how it's managed. No amount of AI-driven capability or Data Visualisation brilliance can replace the need for quality data. You can tell a good story with AI and visualisation... but the truth that powers effective decisions lies with the data.
The Data Landscape
When you ask Mirror Ai a question, the answer arrives in seconds: sourced, cited, and ready to use. From the user's perspective, it looks simple. Type a question, get an answer. The same thing with our dashboard, the data materialises at your fingertips, ready with the information you need to make your next decision.
Behind that simplicity is a data layer that ingests from 10+ sources, manages over 150 actively maintained tables across our database storage engines, and runs continuous monitoring to catch problems before they reach the user. The AI is the visible intelligence. The data layer is the invisible foundation. This post is about that foundation.
Mirror Ai pulls data from a wide range of sources and brings it together in one place to give you quality information to help you better understand the blockchain space to help you quickly figure out what you what you want to do next:
Exchange data such as price, volume and project metadata from sources like CoinMarketCap, CoinGecko, and Binance to give you a factual picture of market conditions as part of a broader research process. Understanding whether a project's market presence is growing or shrinking is one input among many when evaluating it.
Blockchain analytics and DeFi data such as block transactions, protocols, liquidity and token holders from providers like Moralis and Etherscan, GoPlus, Forta, DeFiLlama and DexScreener to explore beyond the centralised participants to better understand the industry as a whole, and to better understand individual projects of interest.
Regulatory and news data* to help you to understand what's affecting the industry at large and the narratives that are shaping how the digital economy evolves.
Why should you care?
Each source has its own format, update cadence, and reliability profile. Some update in real time. Others are polled periodically. Some require API keys and rate limiting. Others are free but less reliable.
The challenge is not collecting the data. The challenge is keeping it consistent, fresh, and trustworthy across all those sources. A single stale pipeline cascades through the system. We learned this the hard way recently when a pipeline stall left price history empty despite millions of rows of raw price data existing. The data was there, but the system could not connect it to the historical view.
This is also why general-purpose Generative AI like ChatGPT or Claude cannot do what Mirror Ai does. They have broad knowledge but no integrated, real-time data pipeline. They cannot tell you whether a protocol's total value locked (TVL) changed in the last hour, or whether a regulatory filing was published this morning, because they do not have a data layer pulling from those sources continuously.
Why not subscribe to the sources yourself? The short answer is that most people would use a fraction of what is available, and the cost adds up fast. The cheapest paid tiers across our core data providers: CoinMarketCap, CoinGecko, Moralis, and Etherscan,can set you north of $230 per month before you write a single line of integration code, depending on what you are looking for. Worse still, you are likely leaving a lot of value on the table as you probably won't use the entire API to the full. The free sources (DeFiLlama, DexScreener, GoPlus, Forta) are valuable, but each requires custom integration, monitoring, and failover handling. In short, making data from multiple sources useful is a full-time, highly technical endeavour.
| Source | Cheapest paid tier (USD) |
|--------|-------------------|
| CoinMarketCap | $79/mo |
| CoinGecko | $79/mo |
| Moralis | $49/mo |
| Etherscan | $29/mo |
| Total | $236/mo |
Please note that these numbers are indicative, bearing in mind that inflation is doing no favours.
This, all before you even write a single line of code!
At Mirror Ai, we ingest all of our sources simultaneously, normalise them into a consistent format, and make the combined result available through a single interface. This means every user benefits from the full dataset without managing multiple subscriptions or maintaining individual pipelines. And because we are constantly adding new sources and finding new ways to derive insights from the data we already have, so does the potential for insights that can be derived from the Mirror Ai data platform grow over time without additional effort on your part.
This is the reality of data infrastructure at production scale. It is not a one-time setup. It is ongoing.
Three Engines, One Platform
Not all data behaves the same way. Some data needs ACID transactions and complex joins across dozens of tables. Some data needs to be stored at petabyte scale and queried analytically. Some data needs sub-millisecond response times.
Choosing one database for everything would mean compromising on all three. So Mirror Ai runs three storage engines, each chosen for the job it does best:
| Engine | What it handles | Why it is the right tool |
|---|---|---|
| PostgreSQL | Transactional data, compliance, structured workflows | Mature, reliable, ACID-compliant, excellent for complex queries |
| Apache Hudi (Data Lake) | Analytical data at scale, e.g. OHLCV candles, market snapshots, DeFi history | Cost-effective object storage, medallion architecture, compaction for performance |
| Redis | Caching, session state, real-time operations | In-memory speed, sub-millisecond latency |
The Core Data Platform
PostgreSQL: The Reliable Foundation
PostgreSQL is a relational database that has been the industry standard for decades. It is mature, flexible, well understood, and backed by some of the most robust open-source development in the world.
At Mirror Ai, PostgreSQL handles the structured data that keeps the platform running, for example:
* Token research cases: every compliance investigation, regulatory analysis, and whitepaper evaluation lives in PostgreSQL. Research analysts run complex queries across 20+ related tables, pulling together issuer data, evidence items, domain scores, and compliance checklists. ACID compliance means the results are always consistent, even if multiple analysts are working simultaneously.
Compliance workflows:* regulatory obligations, jurisdiction scores, and policy rules are stored and queried relationally. These workflows need the ability to join across tables, aggregate results, and maintain audit trails.
Where the reliability requirements are high, PostgreSQL delivers!
Why it adds power to Mirror Ai: Without PostgreSQL, many of the core workflows that collect and manage our data would be far less efficient. When we have complex transactional data queries that need ACID guarantees at speed, PostgreSQL comes in clutch!
The Data Lake: Scale Without the Cost Blowout
Some data is too vast to practically store in a traditional database or doesn't need to meet the same performance requirements that PostgreSQL provides, and some need a system that can flexibly handle heterogeneous data types in one place comfortably. This is true for some of the datasets that we have, including our OHLCV candles (Open, High, Low, Close, Volume) across 200+ trading pairs, years of market snapshots, and historical DeFi protocol data, just to name a few. These represent the kind of data that would be prohibitively expensive to store in a transactional relational database.
A data lake is storage designed for huge volumes of data, built on object storage (S3-compatible storage) rather than traditional database disks. Apache Hudi is an open-source framework that adds database-like features to data lakes, such as transactions, upserts, time travel, and compaction. It was pioneered at Uber and is now used by Amazon, ByteDance, Robinhood, and others. We have chosen this database to power many of our analytical workloads due to its balance of scalability and speed, making the additional maintenance load an acceptable cost of doing business
The Medallion Architecture
Data in the data lakes is typically managed using the medallion architecture , a design pattern that organises data into three progressively refined layers:
* Bronze: Raw data as it arrives. Exactly as received, with metadata about when and how it was ingested. Nothing is modified. This preserves the original data for auditing, reprocessing, and lineage tracing.
* Silver: Cleansed, validated, and conformed. Data from different sources is matched, merged, and de-duplicated. Ready for analysis.
* Gold Curated, aggregated, and optimised for specific consumption. This is where dashboards query from and where AI agents look for market context.
This means data can be stored at source quality and refined as needed, without paying database prices for every byte. The Bronze layer is cheap object storage. The Gold layer is a smaller, curated, high-performance dataset.
The Learning Curve
We built our infrastructure in-house from robust open-source components. This gives us control and cost efficiency, but it also means we own the operational complexity.
The most visible example: when you append data every 5 minutes (as OHLCV data does), the data lake accumulates thousands of small files. Over time, these degrade read performance. The solution is compaction, a background process that merges small files into larger ones. We recently ran a compaction job on a single table that had accumulated over one million small files. The job ran for days.
This is the tradeoff. The data lake lets us store both transactional and analytical data at scale on a budget, allowing us to manage datasets that would be cost-prohibitive in a traditional database. But we had to build the tooling to manage it: automated compaction jobs, table management schedules, and monitoring to catch stalls before they cascade. We manage thousands of small files through regular automated processes, because the alternative, paying for a managed solution that abstracts this away, would mean losing the cost advantage that makes the data lake worthwhile.
Why it adds power to Mirror Ai: Without the data lake, the breadth of market coverage would be economically non-viable. The medallion architecture gives us the flexibility to store everything at source quality, then refine it as needed. The compaction work is simply the natural cost of doing data at scale on a budget with open-source tools.
Redis: The UX Layer
Between the data lake and the user sits Redis. It plays a simple but critical role: caching data so that the outputs of complex database queries can be served seamlessly, with minimal impact on the user experience.
For example, one of the patterns that we use to drive our dashboards in Mirror Ai is called "good until replaced". Input data is cached with a safety TTL (time-to-live). The cache is never evicted by time alone t is only overwritten when a fresh read from the data lake succeeds. If the data lake pipeline has a transient delay, the user sees the cached data instead of a blank screen. This way, the dashboard stays responsive and the user stays productive.
This is a small detail in the overall architecture, but it is the kind of detail that defines the user experience. The data lake does the heavy lifting. PostgreSQL does the complex querying. Redis makes sure the user never has to wait for either of them.
How They Work Together
The three engines do not operate in isolation. They are designed to complement each other:
As the diagram above shows, data flows systematically through the platform in a systematic, structured way that makes the data both usable and manageable. The data lake is the source of truth for analytical data, able to manage different types of data in the same accessible place. PostgreSQL is the backbone for reliable, flexible management of transactional and structural data. Redis keeps everything snappy.
Each engine plays its role. The integration between them is where the real value lives.
The TL;DR
Data is the foundation that makes AI useful. Mirror Ai's data layer is 150+ actively managed tables across three storage engines, each chosen for a specific job:
- PostgreSQL for reliability and flexibility — the transactional backbone that powers compliance, research, and user management
- Apache Hudi for cost-effective scale — the medallion architecture (Bronze, Silver, Gold) enables enterprise-grade data management without the enterprise price tag, built in-house from open-source components
- Redis for speed — the performance layer that makes everything feel instant
The integration between them is where the real value lives. And keeping them healthy is ongoing work — monitoring, compaction, freshness checks, and table management.
Because data quality is not a one-time investment. It is the product.
Published on the Mirror Ai Knowledge Hub. Follow us on LinkedIn for more insights into building production AI systems.
References
- PostgreSQL Documentation — ACID Compliance. https://www.postgresql.org/docs/current/acid.html
- PostgreSQL Documentation — Transaction Isolation. https://www.postgresql.org/docs/current/transaction-iso.html
- PostgreSQL Documentation — WAL (Write-Ahead Logging). https://www.postgresql.org/docs/current/wal-intro.html
- Apache Hudi — Overview. https://hudi.apache.org/docs/overview/
- Apache Hudi — Compaction. https://hudi.apache.org/docs/compaction/
- Apache Hudi — Use Cases. https://hudi.apache.org/docs/use_cases/
- Databricks — What is Medallion Architecture? https://www.databricks.com/blog/what-is-medallion-architecture
- Redis Documentation — About. https://redis.io/docs/about/
- Redis Documentation — Data Types. https://redis.io/docs/data-types/
- Mirror Ai Dashboard — Try the platform out now!
