How accurate is AI compliance review? Our methodology explained

When we tell prospective customers that RegOak achieves 99.2% rule coverage accuracy, the first question is always: how do you measure that? It's a fair question. Accuracy claims in AI products are often vague or misleading. Here's exactly how we measure ours.

Our accuracy measurement is based on a test dataset of 10,000 content samples — a mix of real marketing content (anonymised) and synthetic examples designed to test edge cases. Each sample has been manually reviewed by qualified compliance professionals and labelled with the correct compliance assessment.

We run RegOak's review engine against this dataset quarterly and measure two things: recall (the percentage of genuine compliance issues that RegOak identifies) and precision (the percentage of RegOak's flagged issues that are genuine compliance issues). Our 99.2% figure refers to recall — we identify 99.2% of genuine compliance issues in our test dataset.

Precision is slightly lower, at 94.7%. This means that approximately 5% of RegOak's flagged issues are false positives — issues that a human reviewer would not consider genuine compliance problems. We believe this is an acceptable trade-off: it's better to flag a potential issue that turns out to be fine than to miss a genuine compliance problem.

Where AI review is less reliable is in the assessment of context and intent. RegOak can identify that a performance claim doesn't include a risk warning. It's less reliable at assessing whether a complex investment strategy description is genuinely accessible to a retail audience — that requires human judgement.

This is why we're clear that RegOak is a compliance assistance tool, not a compliance decision-making tool. The AI handles the rule-checking; the human compliance professional makes the final judgement. This combination is more accurate and more efficient than either approach alone.