THE CLIENT
About the organization
The assurance practice and AI leadership team of a U.S. public accounting firm.
THE CHALLENGE
The problem
Staff answered methodology questions by searching dozens of manuals, templates and workbooks, or by asking a senior colleague. General-purpose AI assistants could produce an answer, but the firm could not limit them to its approved library, see which page an answer came from, replay later what the tool had said, or measure how often it was right. In a regulated practice, an answer nobody can trace is a risk.
OUR APPROACH
How we solved it
Test set first
Wrote 100 practitioner-style questions with expected answers and source passages from the firm's own documents, plus 10 out-of-scope questions the system should decline.
Ingestion and retrieval
Parsed PDF, Word and Excel files into size-capped pieces tagged by page or sheet, then combined meaning-based and keyword search with a reranking step.
Answer controls
Answers draw only on retrieved passages and return their sources. When the library does not cover a question, the system says so. Each question, its sources, model and cost are logged for replay.
Measurement and deployment
Scored each change with two evaluation frameworks and a judge model separate from the answering model, recording code version, settings and dataset for every run. Deployed a secured web application where staff ask questions, check sources and rate answers.
A knowledge assistant earns trust in a regulated firm by showing its sources, declining when it should, and proving its accuracy with a test anyone can rerun.
More results