CFA Mock Exam
A CFA exam practice platform with computer-based-test style mocks, drills, and an on-screen BA II Plus calculator. Operated by the Phi team, with AI agents running content audits, security fixes, and QA.
Visit cfamockexam.com- Merged PRs (22 Aug – 24 Oct 2026)
- 124
- Commits with AI agent co-author
- 290 / 412
- Items scanned in answer-key audit
- 1,356
- Detector failures left after fix
- 0
The Challenge
An exam-prep product lives or dies on trust: every answer key must be right, paid content must stay paid, and scores must be real. With a large, partly generated question bank and a small team, the problem was how to keep content quality and security high while still shipping quickly.
Tech Stack
The Solution
Agents run structured audits of the question bank with a CI gate, and own security fixes end to end: moving question delivery and grading server-side, locking Firestore rules, and tagging bot accounts so they stay out of metrics.
The Results
Live at cfamockexam.com. 124 merged PRs between 22 Aug and 24 Oct 2026, and 290 of 412 commits on main carry an AI agent co-author. The product itself does not use an LLM; the agents work on building and operating it.
The problem
CFA candidates pay for practice that mirrors the real exam. That means correct, unambiguous questions, a realistic test runner, and scores that mean something. A small team cannot hand-check a large bank, and a client-side quiz app is easy to game.
The production system
- Computer-based-test style runner, full mock exams, topic drills, a study guide, and a glossary, with math rendered through KaTeX and an on-screen BA II Plus calculator.
- Firebase Auth and Firestore with emulator-tested security rules, Netlify Functions for server logic, and Stripe for paid access.
- Admin rights decided server-side from a configured allowlist, never shipped to the browser.
Judgment calls
- Fix at the source, not in the UI. When a client-side weakness was found, the fix moved delivery and grading to the server rather than hiding fields.
- No production writes during audits. Content corrections ship as an ID-preserving overlay with a dry-run-only migration script, so every change is reviewable and reversible.
- Keep metrics honest. Headless-browser signups are tagged as bots and excluded from KPIs.
Failures we caught and fixed
- Security: monitoring caught an automated scorer, so grading moved server-side, suspicious high scores are held for review, and the question bank was locked to admins. Agent review also found and closed paid-access and account-integrity gaps before exploitation.
- PR #44: after a learner reported ambiguous "most accurate" items, an audit scanned 1,356 items, flagged 342, applied 349 overlay corrections, and left 0 detector failures, with a CI gate to keep it that way.
- PR #27: audited Level I content, retired ambiguous generated templates, and repaired the affected items.
What the agent fleet does
Agents pick up issues from the tracker, write the fix, run unit, emulator, and live read-only checks, attach screenshots, and open a PR for human approval. Our CFA agent works from pollen/ branches; Codex, Claude Code, and Cursor work alongside it.
How we count
Numbers come from gh pr list --state merged on our private product repositories, measured 24 Oct 2026. A PR counts as agent work only if its branch is named for an AI coding agent (for example codex/, claude/, cursor/, hermes/, fizz/, pollen/, honey/, systems/). Agent work on other branches is not counted, so these are floors. For CFA Mock Exam we also count commits on the main branch that carry an AI agent Co-authored-by trailer. We do not publish user, activation, or retention numbers here.
Ready to build something like this?
Start with a free audit, or the S$3,500 assessment (paid by PayNow invoice, credited in full against your build).