Support assistant that answers from yourown documents
A retrieval assistant that answers staff questions straight from the company’s own documents, with every answer traceable to its source.
- Client
- Fieldnote
- Type
- AI
- Year
- 2025
- Delivered
- AI engineering · Retrieval pipeline · Evaluation · UI build
- Method
- Retrieval + evaluation
- Behaviour
- Escalates when unsure
- Answers
- Cited to source
- Rollout
- Measured first
01 /
The challenge
A generic chatbot sounds impressive in a demo and then confidently answers with something that is not true for this company. The team needed an assistant that only answers from documents it can actually cite, and says “I don’t know” when the documents do not cover it.
02 /
What we did
- 01
Chunked the company’s documents into searchable pieces and indexed them, so answers come from real material rather than guesswork.
- 02
Built the retrieval step to return the source alongside every answer, so staff can check the original rather than trusting the assistant.
- 03
Set a confidence threshold: below it, the assistant escalates to a person instead of guessing, which is the only safe default.
- 04
Measured accuracy on a test set of real questions before anything reached the team, and kept the harness so regressions are caught.
03 /
Outcome
An assistant the support team actually uses. Routine questions are answered instantly with a citation, anything uncertain goes to a person, and the team can rerun the evaluation whenever documents change.
Built with
- Python
- FastAPI
- pgvector
- React