How to choose an AI agent agency
Choose the agency whose normal project looks like yours, then test it on four topics: how it evaluates agents, how it limits what they can do, how it handles your data, and what you hold at handover. A firm that answers those four with specifics from past work is a safer choice than one with a better demo.
Write down the task before you talk to anyone
Describe one task in plain terms: who does it today, how often, which systems they open, what a good result looks like and what a mistake costs. A page is enough. This brief lets agencies give comparable answers, and it exposes early whether you need an agent at all or a fixed workflow with one model step.
Add the constraints that will not move: where data must be hosted, which model providers your security team accepts, and who inside the company will own the agent after launch.
Match the agency type to your situation
| Your situation | Agency type that fits | Why |
|---|---|---|
| Small or mid-size company, first agent, no AI team | Boutique build studio | One senior team handles design, build and integration without programme overhead |
| You have engineers but lack LLM experience | AI engineering consultancy | Embedded specialists build with your team and transfer skills |
| Many business units, strict governance, global rollout | Enterprise integrator | Capacity, change management and a governed runtime |
| You have standardised on one framework, platform or model | Platform-led or lab-backed team | Direct access to the people who build the underlying technology |
Questions that separate good agencies from the rest
- Evaluation. How do you build the test set, who labels it, and what pass rate did your last agent need before launch? Ask to see an anonymised evaluation report.
- Guardrails. Which actions can the agent take without approval, how are permissions scoped per tool, and what happens when it is unsure?
- Data access. Does the agent use each user's own permissions or a shared service account? Where are prompts, documents and logs stored, and for how long?
- Failure handling. How does the agent hand a case to a person, and how do you find out it has started failing after a model or system change?
- Monitoring. Which tracing tool do you use, who reads the traces each week, and which numbers go on the dashboard?
- Handover. Do we receive the repository, prompts, evaluation sets and runbooks? Can our team change a prompt and re-run the tests without you?
- Team. Who exactly will work on this, and have those people shipped an agent that is still in use?
Red flags
- The proposal contains no evaluation plan, or treats testing as a final step.
- The pitch promises full autonomy from day one for a task that changes records or contacts customers.
- Every problem you describe gets the same multi-agent architecture as the answer.
- The agency cannot name a deployed agent with real users, only prototypes and workshops.
- Ownership of code, prompts and logs is vague, or the system only runs on a platform you cannot leave.
- Run costs are missing from the quote.
- Senior people sell the work and an unnamed team delivers it.
Use a paid discovery as the real interview
A short, paid discovery phase tells you more than any pitch. You see how the agency asks questions, how it treats your data, and whether its plan is specific. Keep it small and make the output yours to use with any supplier.
- Shortlist two or three agencies of the right type.
- Send each the same one-page task brief.
- Commission discovery from the one or two with the most concrete answers.
- Ask for a written plan with success measures, an evaluation approach, an access list and a price for the pilot.
- Decide on the pilot only after you have read that plan.