The Questions to Ask Before Buying Any AI Tool

Every demonstration works. That is what it is for, and it tells you almost nothing about whether the thing will work in your business.

What tells you is how a supplier answers a small number of specific questions, most of which have nothing to do with the technology.

1. What does this connect to, and who maintains that

The most important question and the one most often left until implementation.

A system producing good output that somebody then moves by hand has not removed work. Ask exactly which of your systems it writes to, whether that is a supported connection or a custom build, and who is responsible when the connection breaks.

If integration is described as straightforward without anybody asking what you currently run, that is an answer in itself.

2. What happens when it fails

Not whether it fails. When.

The answer you want involves retrying, then escalating to a named person with the reason attached. The answer that should worry you is anything implying you would notice. Silent failure is how systems get abandoned, because the discovery usually comes from a client.

3. What are we measuring, and what is it today

Ask the supplier what number this should move, then establish where that number currently stands before anything is installed.

Without a baseline you cannot demonstrate a result afterwards, and a project with no demonstrable result does not get renewed regardless of how well it works. Suppliers who are confident welcome this. Suppliers who deflect are telling you something.

4. Where does our data go, and is it used to train anything

Two separate questions that frequently get one vague answer.

Where the information is processed and under whose jurisdiction. And whether what you submit may be used to improve the supplier’s models. Consumer tiers of the same product often reserve that right where business agreements do not, and the interface is identical, so people using it have no idea which they are on.

5. What can it see, and does that match what the user is entitled to

The most commonly missed control.

A system with access to everything will answer questions about everything, to whoever asks it. If permissions do not carry through to the AI layer, you have created a way for anybody to query material they should not reach.

6. What can we show if a client challenges an output

Ask what is logged. What was asked, what material it drew on, what it produced, who approved it.

That record costs nothing at the start and cannot be reconstructed later. In regulated work it is the difference between an awkward conversation and a serious one.

7. Who owns this in eighteen months

Not who supports it. Who inside your business notices when it has quietly stopped being right.

Processes change and systems drift. A tool nobody owns degrades silently and is eventually switched off. If you cannot name a person, the purchase has an expiry date regardless of the contract length.

8. What happens if we stop working together

Ask early, while the answer is cheap.

Who owns the configuration and the data. Can you export it in a usable form. Is the documentation good enough for somebody else to take over. A good answer costs the supplier nothing, and a vague one is the most reliable warning available at this stage.

The question to ask yourself

Separately from the supplier: what does this do that a person was not going to do anyway?

If the honest answer is that it makes an existing task somewhat faster, the effort of doing all of the above may not be justified. If it covers hours nobody was covering, or reads volume nobody had time to read, the case is clear and the controls are worth building properly.

Stanford’s AI Index for 2024 found that assistance of this kind both speeds task completion and improves output quality, with the largest gains among people less experienced at the task. That is a useful filter. The strongest cases are usually where capability is currently limited by one person’s availability rather than where an already efficient process could be slightly faster.

What a good supplier sounds like

They ask about your process before describing their product. They name something this will not do. They talk about detection rather than only accuracy. They want a baseline. And they answer the exit question without hesitating.

None of that is about the technology, which is rather the point. The technology is broadly comparable across serious suppliers. The difference is entirely in whether somebody has thought about your business.

Run a real pilot, not a demonstration

The single most useful thing you can insist on is a trial using your own material and your own awkward cases.

Not the tidy examples a supplier prepared. The document that arrives in a strange format, the enquiry that does not fit any category, the client whose arrangement is an exception to everything. Those are what the system will meet in week two, and they are what a demonstration is carefully constructed to avoid.

Agree in advance what result would count as success, and set a time limit. A pilot with no defined end becomes a permanent state where nobody has to decide.

Ask who else is running it in a business your size

Reference customers are usually offered from the largest and happiest clients. Ask instead for one at roughly your scale, in a comparable industry, that has been live for over a year.

The gap between what a system does at enterprise scale and what it does with a twenty person team is substantial, and it is mostly about who maintains it. A reference that has been running twelve months will tell you what breaks, which is the useful information.

Price the whole thing, not the subscription

The licence is rarely the largest number.

Add the implementation, the integration work, the time your team spends being trained and answering questions during setup, and the ongoing ownership. Then compare against what the process costs you today, honestly measured.

Businesses that only compare subscription against nothing conclude everything is cheap. Businesses that price it properly make better decisions and negotiate better, because they know what the thing is actually worth to them.

Sources

• Stanford Institute for Human-Centered Artificial Intelligence, Artificial Intelligence Index Report 2024, 15 April 2024.

AI Optimize answers all eight of these before a build starts, and will tell you when the honest answer is that you should not buy anything. That work sits under Custom AI Integrations.