The problem with asking one model
The simple way to answer a security questionnaire is to hand a large language model (LLM) your documents and the questionnaire and ask it to fill in the answers. It works well enough in a demo. In practice it has two problems:
- It costs a lot. Every question sends your whole set of documents through an LLM, so a long questionnaire runs up a large bill. It's one of the reasons questionnaire tools are often priced at several times what DeliverTrust charges.
- It doesn't reliably show its evidence. The line it points to isn't always the one that supports the answer, and when your documents don't cover a question, it often writes something plausible anyway.
So I built it differently. A keyword search and TypeSafe's Jev, a small and cost-effective judgment model, collect the evidence. Only then does an LLM write a short answer from those lines alone. That fixes both problems: the LLM never reads your whole library, only a handful of lines per question, and every answer is tied to lines you can check.
Step one: reading the questionnaire
Before anything is answered, the customer's file has to be read correctly: which cells are questions and which column the answers go in. Real files are messy, so three readers look at every row: a script that parses the file, TypeSafe's Jev, and Anthropic's Claude Sonnet. The majority decides whether a row is kept, and any row they disagree on is marked for you to check before import.
Step two: answering, with the line behind every answer
Once the questions are enumerated, each one goes through the steps below. Your reused answers are checked first; if none fits, the evidence is found and scored before Sonnet writes anything, and every claim is checked before you see it.
- 1Check your reused answersJev and Sonnetboth must agree an answer you marked for reuse asks the same thing
- 2Find the linesKeyword search, then Jevnarrow your sources to likely lines, then score each one
- 3Write the answerSonneta short answer from those lines alone
- 4Check every claimJevagainst the lines, before you see it
The quoted lines sit beside every answer, so the person reviewing it checks it against the evidence instead of taking it on faith. Each answer is also marked Drafted, Needs review or Insufficient evidence:
- Drafted when the lines clearly answer it.
- Needs review when the evidence only partly answers it, or the readers don't agree. If Sonnet says the lines don't state the answer, that counts as a vote against, even when the evidence looked strong.
- Insufficient evidence when your sources don't cover it. The answer is left for you instead of invented.
How I picked the model that writes the answers
I tested Claude Sonnet against the smaller Claude Haiku in a blind side-by-side: 48 questions answered from a real company's documents, each pair shown without the model's name. Sonnet won clearly and was faithful to the evidence on every answer. Haiku would have cut the model cost by about 70%, but eight unfaithful answers out of 48 is not a trade I'll make on a security questionnaire. I didn't move up to the larger, pricier Opus either. Sonnet was already faithful to the evidence on all 48 questions, and its job is a few sentences from lines that have already been chosen and checked, so a bigger model had nothing left to improve.
Why answers cost cents
Splitting the work this way is what keeps the price low. The narrow judgments, which are most of the calls, go to a small model that costs under a tenth of a cent per question. The larger model's main job is writing the short answers, and only for questions that have evidence.
Answering the CSA CAIQ, 283 questions, cost $0.96 in model calls, about a third of a cent per question. Jev was a quarter of that and Sonnet the rest. That's why DeliverTrust can offer unlimited questionnaires, under fair use, for a flat $149 a month instead of charging per questionnaire or per seat.
What I don't do
- No training on your data. Neither TypeSafe, Anthropic nor DeliverTrust trains models on what you upload.
- No logging of the text being judged. Questionnaires and documents can carry NDA content, so the logs record counts, scores and timings, never the words.
- No AI until you turn it on. An Administrator turns on AI answering once for the account before anything is sent to either provider.
- Nothing goes out on its own. Answers reach your customer only when you export the file yourself, after you have reviewed it.
See how DeliverTrust Questionnaires work, or what security questionnaire software should cost.