What We Found When We Audited Our Own AI

Working with AI is a humbling exercise. You go from "oh my God, I am a genius" to "oh wow, this thing is stupid" in less time than it takes to process your request.

We build Mirror AI - a compliance intelligence platform that reads regulatory documents, extracts obligations, and scores how crypto-friendly (or hostile) different jurisdictions are. It sounds impressive. Then last week we audited our own database and found out how much of that impressiveness was... aspirational.

Here's what the audit actually turned up.

The Problem With Scoring by Volume

Our jurisdiction scoring system had a blind spot. It measured regulatory coverage by document count - the more documents a jurisdiction had, the higher it scored. On paper, Australia scored 68.1 and the UAE scored 48.6.

The problem? Australia's count was inflated by hundreds of reports, bulletins, and consultation papers. The UAE's 19 documents were mostly binding regulations. We were rewarding volume, not relevance.

For a crypto compliance platform, that's not just a math problem. It's a truth problem. We were telling people "Australia is more regulated for crypto than Dubai" when anyone who works in this space knows that's backwards. VARA's entire rulebook is virtual-asset-specific. Australia's is mostly general finance with crypto bolted on.

The score was measuring the wrong thing. That's on us.

The Part Where It Gets Worse

The audit found 11 issues. The headline ones:

• 20,628 regulatory obligations existed but weren't linked to source documents. The data was there, but you couldn't trace an obligation back to verify it. In compliance, untraceable data is just a rumour with a timestamp.
• 48 documents had garbage jurisdiction labels - topics and filenames sitting in the jurisdiction field. Somebody's spreadsheet habits leaked into our database.
• 114 document type variants when there should have been about 20. We had "regulatory_guide", "Regulatory Guide", and "guide" all meaning the same thing. AI is great at finding patterns. It's also great at inventing them.
• 209 documents registered with no source file. Placeholder records. Ghosts in the machine.
• VARA (Dubai) had 19 documents and zero obligations extracted. The PDFs were sitting in storage. We just... never processed them. Nobody noticed until we looked.

The uncomfortable truth: we built a platform that tells people how to comply with regulation, and we couldn't point at our own obligations table and say "this came from that document." That's not a bug report. That's a character flaw.

What We Changed

1. Three New Scoring Metrics

We stopped scoring by document count and started scoring by what the documents actually are:

• Authority weight - how much legal force does a document carry? Regulations and legislation score 3x. Guidance and rulebooks score 2x. Reports and press releases score 1x or 0.5x. A binding law is not the same as a press release, and our score no longer pretends it is.
• Crypto-specific obligation density - what percentage of a jurisdiction's obligations are actually about crypto? Not general consumer protection. Not generic AML. Crypto-asset rules, custody requirements, market integrity for digital assets. The stuff that actually matters to a crypto business.
• Composite score - the two above combined with obligation volume into a single crypto-relevance number.

The result: the UAE now ranks highest for crypto-specific regulatory relevance. Which is correct - VARA is the only jurisdiction in our database whose entire framework is virtual-asset-specific. Australia and the US rank lower on crypto-specific metrics. Also correct - they have more total regulation, but less of it is about crypto.

The ranking didn't change because we got smarter. It changed because we started measuring the right thing.

2. Obligations Now Link to Source Documents

Every one of the 20,919 obligations now links back to the document it came from. You can trace an obligation to its source and verify it yourself. This should have been true from day one. It wasn't. Now it is.

3. VARA (UAE) Obligations Added

Dubai's VARA framework now has 291 obligations extracted across 8 topics - governance, AML/CTF, custody, disclosure, consumer protection, market integrity, crypto-assets, and capital requirements. 57% of these are crypto-specific, the highest proportion of any jurisdiction in our database.

We had the PDFs the whole time. The gap was ours. It's closed now.

4. Data Quality Cleanup

48 mislabeled jurisdiction records cleaned. Document types normalized from 114 variants to consistent categories. The ghosts have been flagged for sourcing or exorcism.

The Honest Bit

I keep hearing the word "authenticity" a lot these days. I reckon I'll be hearing it a lot more in the future. Mostly because everyone is going to be auditing everyone else's AI output, and the ones who can trace their data back to a source are going to be the only ones still standing.

This isn't a one-time fix. The scoring methodology is designed to be rerun as new jurisdictions and documents are added. The audit process is repeatable. We'll keep running it, and we'll keep publishing what we find - including the parts that make us look bad.

The full audit report is available on request. If you're using our compliance intelligence and want to understand how the scores are calculated - or if you want to see the raw data behind a specific jurisdiction - ask. We'll show you. We can trace it now.

• Mirror AI Compliance Intelligence