Professional Services / Public Accounting

RAG · AI Evaluation · Knowledge Management

A measured, cited knowledge assistant for a public accounting firm

Built and deployed a private retrieval-augmented generation (RAG) system that answers staff questions from the firm's approved methodology library, with page-level sources and a test harness that scores every change.

110

Firm-specific test questions scored on every change, including 10 the system is expected to decline

90+

Out of 100 on each of four quality scores: faithfulness, answer relevance, retrieval precision and retrieval recall

THE CLIENT

About the organization

The assurance practice and AI leadership team of a U.S. public accounting firm.

THE CHALLENGE

The problem

Staff answered methodology questions by searching dozens of manuals, templates and workbooks, or by asking a senior colleague. General-purpose AI assistants could produce an answer, but the firm could not limit them to its approved library, see which page an answer came from, replay later what the tool had said, or measure how often it was right. In a regulated practice, an answer nobody can trace is a risk.

OUR APPROACH

How we solved it

Test set first

Wrote 100 practitioner-style questions with expected answers and source passages from the firm's own documents, plus 10 out-of-scope questions the system should decline.

Ingestion and retrieval

Parsed PDF, Word and Excel files into size-capped pieces tagged by page or sheet, then combined meaning-based and keyword search with a reranking step.

Answer controls

Answers draw only on retrieved passages and return their sources. When the library does not cover a question, the system says so. Each question, its sources, model and cost are logged for replay.

Measurement and deployment

Scored each change with two evaluation frameworks and a judge model separate from the answering model, recording code version, settings and dataset for every run. Deployed a secured web application where staff ask questions, check sources and rate answers.

A knowledge assistant earns trust in a regulated firm by showing its sources, declining when it should, and proving its accuracy with a test anyone can rerun.

More results

50+
Methodology documents in the first library, across PDF, Word and Excel
450+
Automated code tests run on every change
Page-level
Sources returned with every answer, down to the worksheet for Excel files

Facing a similar challenge?

Let's discuss how EvoNova can deliver similar results for your organization.