Questions to ask an AI development company before hiring.
The questions that show whether an AI development company has run systems in production or only built demos, grouped by stage, with what a good answer and a red-flag answer sound like.
Before you hire an AI development company, ask five questions. How will you measure quality before launch, and can we see an evaluation report from a past project? What does the system do when the model is wrong, slow or unavailable? How do you defend against prompt injection, and what can the system do if an attack works? What do we own at the end: the code, prompts, test sets and documentation? And who runs the system in the six months after launch, including when the model provider changes or retires the model?
A good answer is specific, names the people who do the work, and comes with something you can look at. A red-flag answer leads with the model’s name, replays the demo, or promises accuracy before anyone has seen your data. Below, every question is grouped by stage with why it matters, a good answer and a red flag, then a scoring table and a list you can copy.
How to use these questions
Send the list before the first call so the firm brings evidence, ask for the engineers who will build it to join, and record every firm’s answers in the same format. Pick the groups that match your risk: a tool that drafts emails for staff to check needs fewer security questions than an agent that changes customer orders. Most of the questions check the Measure and Manage functions of the NIST AI Risk Management Framework (AI RMF 1.0): whether a firm measures quality and manages risk as a habit.
The first call: the problem and the people
Listen for whether the firm asks about your work or starts describing its technology.
| Question | Why it matters | Good answer | Red flag |
|---|---|---|---|
| What problem do you think we have, and is AI the right tool for it? | Some problems are better solved with a rule, a form or a process fix. | They restate the problem in your terms, name the number that should move, and say which parts do not need AI. | They agree with everything and start talking about which model they will use. |
| Who will build it, and can we meet them? | The people on the sales call often do not write the code. | Named engineers, their roles, how much of their week is yours, and a call with them before you sign. | “We will assign the team after signing.” |
| What do you need from us? | AI projects stall on access to data, systems and the person who knows the work. | Sample data, system access, a few hours a week from someone who does the job today, and one decision maker. | “Nothing, we handle everything.” |
| How will we see progress each week? | Working software is the one progress report that cannot be dressed up. | A demo of the working system every week or two, on your data, with open problems listed. | Slide decks and percentages complete. |
| How many working hours do our days overlap? | Questions come up daily during a build, and delays add up. | A clear number of shared hours, and who answers questions in them. | Vague answers, or one account manager passing messages along. |
Proof they have run AI in production
A demo shows the happy path worked once. Production shows the system kept working for months, with real users, model updates and bad inputs.
Can you show us a system that has been live for at least six months?
Why it matters: most of what goes wrong with AI shows up after launch, when real users find the cases nobody tested.
Good answer: they describe the system, how long it has run and what changed since launch. Ideally you can use it, and talk to the person who runs it day to day.
Red flag: demos and prototypes, or case studies with no dates, no numbers and no sign that anything is still running.
What went wrong in production, and what did you change?
Why it matters: every live system has incidents, and a team that has run one can describe them.
Good answer: a specific story: what failed, how they noticed, how long the fix took, and what stops it happening again.
Red flag: “Nothing has gone wrong.” Either the system is not live, or nobody is watching it.
Which numbers do you track on your live systems?
Why it matters: the numbers a team watches show what it thinks quality means.
Good answer: task measures, such as questions resolved without a person, correct citations, handovers, uptime and response time.
Red flag: only public model benchmarks, or “the client is happy”.
Quality and evaluation
In our experience, this group predicts success best. A team that measures quality first can tell you on any day whether the system is getting better or worse. Our note on building the evaluation set before the features shows what this looks like in practice.
How will we measure quality before launch?
Why it matters: without an agreed measure, “done” means whatever the last demo looked like.
Good answer: an evaluation set (eval set) of real examples from your work, each with an expected answer or grading rule, agreed before the build. There is a score the system must reach before each release, and the failures that would hurt you most, such as a wrong refund amount, are cases that must always pass.
Red flag: “We will test it thoroughly” or “we will look at the outputs”. If they cannot describe an eval set, they do not have one.
Can we keep some test cases back?
Why it matters: a team that sees every test case can tune the system to pass them, deliberately or not.
Good answer: yes, and they help you design the held-back set. The US government’s acquisition memo M-25-22, Driving Efficient Acquisition of Artificial Intelligence in Government (April 2025) says that when agencies run independent evaluations, they must use test data they have defined themselves, kept away from the vendor and as close as possible to real use.
Red flag: pushback, or a claim that the system must see every example to work.
How do you check that a change does not make things worse?
Why it matters: every prompt edit, model upgrade and new data source can fix one thing and break another.
Good answer: the full eval set runs on every change, results are compared with the last release, and a drop on a critical case blocks the release. Rolling back takes minutes.
Red flag: changes go straight to production after a quick look.
What does the system do when it does not know?
Why it matters: a system that always answers will sometimes be wrong with full confidence. The NIST Generative AI Profile (NIST AI 600-1) calls this confabulation: “the production of confidently stated but erroneous or false content”.
Good answer: it declines, says it is unsure or hands over to a person, and the eval set tests exactly that. When it answers from your documents, it cites the source.
Red flag: “The model is very accurate now, so hallucinations are not really a problem.”
Data, security and privacy
An AI system reads text from users, documents and websites, and some of it will try to give the system instructions. It may also send your data to a model provider you did not choose. If security arrives late as a change request, the team has probably not run AI where it mattered.
Where does our data go, and who can use it for training?
Why it matters: your data may pass through the firm, a model provider, a host and logging tools, and each can leak or reuse it.
Good answer: a list of every service that sees your data, where it is stored and for how long, and written confirmation that neither the firm nor its providers train models on it. M-25-22 tells agencies to make contracts permanently prohibit training publicly or commercially available AI on their non-public inputs and outputs without explicit consent. You can ask for the same term.
Red flag: “The model provider handles that”, or no clear answer on training.
How do you handle personal data in prompts and logs?
Why it matters: logs are where personal data piles up unnoticed.
Good answer: personal data kept out of prompts when the task does not need it, logs masked or deleted after a set period, access limited and recorded.
Red flag: full conversations kept forever in a shared tool “in case we need them”.
How do you defend against prompt injection?
Why it matters: prompt injection is the first entry in the OWASP Top 10 for LLM Applications (2025), which describes it as user prompts that alter the model’s behaviour or output in unintended ways. NIST AI 600-1 separates direct injection, typed in by an attacker, from indirect injection, planted in data the system is likely to retrieve. Any system that reads outside content is exposed.
Good answer: outside text is treated as untrusted input and kept separate from instructions. The system gets only the permissions the task needs, a person approves high-risk actions, and attacks are run against it before launch. These match the mitigations OWASP lists, and our note on prompt injection as an input-validation problem covers the design.
Red flag: “Our system prompt tells the model to ignore malicious instructions.” A prompt cannot be the only control.
If someone does trick the system, what is the worst it can do?
Why it matters: no defence stops every injection, so the useful question is how much damage one can cause.
Good answer: a clear list of what the system can read and change, with limits enforced in code, such as refund ceilings and read-only access where possible. Actions above a limit wait for a person.
Red flag: the agent has administrator access “so it can be helpful”.
Could we reconstruct why it gave a bad answer?
Why it matters: when a customer complains, you need to see what the system saw.
Good answer: each answer traces back to the input, the documents retrieved, the model version and the prompt version.
Red flag: only the final answer is stored, or nothing is.
Ownership, IP and lock-in
The test is simple. If the firm disappeared tomorrow, could you keep the system running and hire someone else to change it? M-25-22 lists protections agencies should seek against lock-in, including knowledge transfer, clear data and model portability, rights to the code and models produced under the contract, and clear licensing terms. NIST AI 600-1 suggests contracts and service level agreements that specify content ownership, usage rights, quality standards and security requirements. Our guide to writing an AI statement of work shows how to put these terms in your contract.
| Question | Why it matters | Good answer | Red flag |
|---|---|---|---|
| What do we own at the end? | Prompts and eval sets are as much the product as the code. | Everything built for you: code, prompts, eval sets, any fine-tuned models, data pipelines, infrastructure definitions and documentation. | “You get a licence to use our platform.” |
| Whose accounts does it run in? | Whoever holds the accounts holds the system. | Your cloud and model-provider accounts, with access granted to the firm. | The firm’s accounts, moved to yours only “on request”. |
| What is in the handover package? | Code alone does not tell the next team why it works. | Architecture decisions and their reasons, runbooks for common failures, the eval set and how to run it, and training for your team. | A code repository and a promise to answer questions. |
| Can we switch model provider? | Providers change terms, models and quality. | The model sits behind one interface, and switching means rerunning the eval set, not rebuilding. | The system depends on one provider’s special features, and nobody mentions it. |
| Do you reuse anything from other projects in ours? | Reused parts may stay theirs. | Existing components listed in writing, with who owns them and what rights you get. | Silence until after signing. |
| Which third-party tools will we depend on? | Each one is a contract you inherit. | A full list, with what each does and how it could be replaced. | Surprises after launch. |
Delivery and change
AI projects change shape more than ordinary software, because you only learn what the model can do on your data.
| Question | Why it matters | Good answer | Red flag |
|---|---|---|---|
| How do you start, and what does discovery produce? | A plan made before anyone tests your data is a guess. | A short discovery on your data and workflow, ending in a written plan, a first eval set and the number that defines success. | A promised accuracy figure in the first meeting, before anyone has seen your data. |
| What happens when we learn something that changes the plan? | It happens on almost every AI project. | A written change process covering the effect on the timeline and the eval set, agreed before work goes ahead. | Everything is a change request, or nothing is written down. |
| How do releases and rollbacks work? | Releases are where regressions slip in. | A test environment, the eval gate, a small first group of users, and a rollback that has been tried. | Edits made directly in production. |
| What happens when the model or a connected system is down? | Providers and internal systems fail. | A planned fallback for each failure: a backup model, a clear message, or a handover to a person. | “The provider has very high uptime.” |
| What does done mean? | Without a definition, projects drift. | The eval score is met, the handover package is delivered and your team has run the system for an agreed period. | Done means the demo worked. |
After launch: monitoring, model changes and support
If you only ask one group, ask these. The first months after launch are when real users find the gaps, providers update or retire models, and the data starts to drift. The UK government’s Guidelines for AI procurement (June 2020) put it plainly: “Testing the model on an ongoing basis is necessary to maintain its accuracy.” Our note on the six months after the demo covers what tends to break.
Who watches the system after launch, and what do they watch?
Why it matters: quality drops quietly. Nobody files a ticket when answers get slightly worse.
Good answer: named owners, a dashboard for the numbers agreed at launch, alerts when they move, and a regular review of real conversations. M-25-22 asks agencies to keep the right to review performance and risks regularly, for example quarterly or twice a year.
Red flag: “Call us if something goes wrong.”
What happens when the provider changes or retires the model?
Why it matters: providers change models on their own schedule, and a new version can change answers you depend on.
Good answer: the model version is pinned, the eval set is rerun on any new version before a switch, and they can tell you how the last switch went.
Red flag: the system calls whatever the latest model is, and nobody checks.
What happens when something breaks at night?
Why it matters: a slow response turns a small fault into a public one.
Good answer: a written incident process: who is alerted, how fast they respond, how the system falls back safely, and a write-up afterwards. NIST AI 600-1 suggests incident response plans for third-party AI with defined owners, rehearsed at a regular cadence.
Red flag: no process, or one person’s phone number.
How does the system get better after launch?
Why it matters: the most useful improvements come from real failures, not new features.
Good answer: failed conversations become new eval cases, fixes are measured against them, and you can see the trend.
Red flag: improvement only ever means starting a new project.
What support do we get, and how does it end?
Why it matters: every support arrangement ends, so you need to be able to run the system yourselves.
Good answer: a defined support period, response times by severity, and a plan for moving day-to-day running to your team.
Red flag: support with no end date and no handover plan.
Ask them to show you
Anyone can give good answers in a meeting. These four things are hard to fake, and a firm that has run production AI will have them ready, with client details removed.
- A live system. One you can use yourself. Try to break it: go off-topic, ask for a person, hide an instruction in a document.
- An eval report. The eval set from a past project, its scores across several releases, and a release held back because a score dropped.
- An incident write-up. What happened, how they found out, what users saw, and what changed afterwards.
- Handover documents. A runbook and an architecture decision record your team could actually work from.
If a firm cannot show any of the four, give that more weight than anything said in the meeting.
How to compare firms afterwards
Score each firm from 0 to 2 on the same areas, straight after each call. Score the evidence they showed, not how confident they sounded. Agree the commercial terms separately, once you know which firms can do the work. For a formal tender, our AI RFP template puts these questions in writing and adds a proof stage on your own data.
| Area | 0: weak | 1: partial | 2: strong |
|---|---|---|---|
| Production evidence | Demos only | Live systems described, little detail | A live system you used, plus an incident write-up |
| Evaluation | No eval set | Plans one, none to show | Showed an eval report and a release gate |
| Security | Relies on the prompt | Some controls, untested | Least privilege, human approval, attack testing |
| Data handling | Unclear where data goes | Clear data flow, training use unconfirmed | Full data list and no training on your data, in writing |
| Ownership and handover | A licence to their platform | You own the code only | You own code, prompts, eval sets and documents, in your accounts |
| After launch | “Call us” | Monitoring, no plan for model changes | Monitoring, pinned models and an incident process |
| Team and delivery | Met sales only | Met the lead, not the builders | Met the engineers, saw a working demo |
Add up the scores, but read the zeros first. A zero in evaluation or security rules a firm out for anything customer-facing. If two firms are close, give each the same small sample of your data and test cases, and compare the results.
Red flags that end the conversation
- No evaluation story. Quality means “we’ll look at the outputs”.
- Demo-only case studies. No numbers and no time in production.
- Model-name marketing. The model they use, not what they measure.
- Lock-in by design. You could not run or move the system without them.
- Instant certainty. Promised outcomes before anyone has seen your data.
The one-question shortcut
If you only ask one thing of every firm on your shortlist, ask this:
“Walk me through what happens in the six months after launch.”
Teams that have run production AI will talk about monitoring, drift, model changes and incidents. Teams that have not will talk about the demo again.
The full question list
Copy this into an email or a request for proposals, and send it to every firm before the first call.
QUESTIONS TO ASK AN AI DEVELOPMENT COMPANY
The first call
1. What problem do you think we have, and is AI the right tool for it?
2. Who will build it, and can we meet them before we sign?
3. What do you need from us: data, access and people?
4. How will we see working progress each week?
5. How many working hours do our days overlap?
Proof of production work
6. Can you show us a system that has been live for six months, and can we talk to the person who runs it?
7. What went wrong in production, and what did you change?
8. Which numbers do you track on your live systems?
Quality and evaluation
9. How will we measure quality before launch?
10. Can we keep some test cases back from you?
11. How do you check that a change does not make things worse?
12. What does the system do when it does not know?
Data, security and privacy
13. Which services see our data, where is it stored, and for how long?
14. Will you or any provider train models on our data?
15. How do you handle personal data in prompts and logs?
16. How do you defend against prompt injection?
17. If someone tricks the system, what is the worst it can do?
18. Could we reconstruct why it gave a bad answer?
Ownership, IP and lock-in
19. What do we own at the end?
20. Whose accounts does it run in?
21. What is in the handover package?
22. Can we switch model provider without a rebuild?
23. Do you reuse anything from other projects in ours?
24. Which third-party tools will we depend on?
Delivery and change
25. How do you start, and what does discovery produce?
26. What happens when we learn something that changes the plan?
27. How do releases and rollbacks work?
28. What happens when the model or a connected system is down?
29. What does done mean?
After launch
30. Who watches the system after launch, and what do they watch?
31. What happens when the provider changes or retires the model?
32. What happens when something breaks at night?
33. How does the system get better after launch?
34. What support do we get, and how does it end?
Ask them to show you
35. A live system we can use ourselves.
36. An eval report across several releases.
37. A real incident write-up.
38. A runbook and an architecture decision record.
To see how we answer these ourselves, our services page sets out what we build, how we measure it and what you own at the end.
Questions buyers ask
What questions should I ask an AI development company before hiring?
Ask how they will measure quality before launch, what happens when the model fails, how they defend against prompt injection, what you will own, and who runs the system after launch. Then ask to see a live system, an eval report, an incident write-up and handover documents.
How do I evaluate an AI vendor’s past work?
Ask for a system that has been live for at least six months, use it yourself, and ask what went wrong in production and what they changed. Case studies with no dates, no numbers and no running system are demos, not evidence.
What should a contract with an AI development company cover?
Ownership of the code, prompts, eval sets and any models. Where your data goes, and no training on it without consent. The quality score required before release, monitoring and incident duties after launch, and a handover plan so you could move to another firm.
Who should own the eval set, us or the AI company?
You should, and you should keep some test cases back to check the system independently. The firm can build and maintain it with you, but what counts as a correct answer is a business decision.
What is the biggest red flag when hiring an AI company?
No clear answer to how quality will be measured. If a firm cannot describe its eval set and its release gate, it cannot tell you whether the system works, before launch or after.