Companies have spent heavily on AI pilots, and many of those pilots never reach daily use.

Gartner: in July 2024, Gartner predicted that at least 30% of generative AI projects would be abandoned after the proof-of-concept stage by the end of 2025. It named poor data quality, weak risk controls, rising costs and unclear business value as the causes.
MIT: a 2025 report from MIT’s Project NANDA found that 95% of the organizations it studied saw no measurable return from generative AI pilots. The sample was small, with 153 leaders, 52 interviews and about 300 public projects, so the exact number needs caution.

In many of these cases, the model itself is not the problem. The problems come from two places: running the model after launch, and company data the model cannot read.

Problem 1: The model is the small part

In 2015, Google engineers published a paper called “Hidden Technical Debt in Machine Learning Systems”. Its best-known figure shows a real machine learning system as many boxes:

The machine learning code is one small box in the middle.
Around it sit data collection, data checks, feature extraction, serving, monitoring and configuration.

A pilot needs only the small box. One data scientist trains a model in a notebook, and the demo goes well. Production needs everything around the box, and that work is rarely in the pilot’s budget.

Three things that break after launch

Quality drops without an error. A model learns from data collected at one point in time. Customers, prices and products change after that, so its answers slowly drift away from reality. No error message appears, so someone has to measure quality.
Nobody can rebuild the model. A model depends on its training data, its training settings and its library versions. If those were not recorded, the team cannot retrain it or explain a decision to an auditor.
Training code and live code differ. A data scientist prepares data one way in a notebook. An engineer rewrites that step for the live system, often in another language. Small differences change the model’s input, and accuracy drops for no visible reason.

The fix: MLOps

The practice that handles these problems is called MLOps, short for machine learning operations. It covers how teams version, test, release, monitor and retrain models. A guide to MLOps walks through each stage and its tools. Google’s MLOps guide describes three levels:

Level 0: every step is manual.
Level 1: a pipeline retrains the model automatically.
Level 2: the pipeline itself is tested and released like any other software.

One test shows a team’s level. Can a colleague rebuild the production model from the code repository and retrain it on this month’s data?

When MLOps is worth the cost

Not every model needs the full setup. A single model that runs once a night does not need a feature store, an orchestrator and a model registry. The investment makes sense when:

the model’s output feeds a product or a decision,
the model must be retrained,
more than one person works on it.

Problem 2: Company data is not just text

Many company AI projects are assistants that answer questions from internal documents. Most use retrieval-augmented generation, or RAG:

The system searches the company’s documents for passages that match a question.
A language model reads those passages and writes the answer.

Most RAG systems search text only. Company knowledge often sits elsewhere:

Charts: the values are the heights of bars, not words.
Scanned PDFs: contracts and forms often have no text layer.
Diagrams: technical manuals explain things in pictures.
Slides: strategy lives in decks.

When a text-only system reads these files, the chart values and table layouts are lost. The assistant then answers from what is left, or finds nothing.

Three designs for multimodal search

Systems that search across text, images and scanned pages are called multimodal RAG systems. There are three main designs:

Convert everything to text. An image model describes each chart, and optical character recognition reads scanned pages. Each description is written before anyone asks a question, so a detail it leaves out cannot be found later.
Put images and text in one search space. Models such as OpenAI’s CLIP, trained on 400 million image and caption pairs, place a photo and a sentence about it close together. A text question can then find a matching image.
Search page images directly. Methods such as ColPali treat each page as an image and store many small vectors per page. They keep layout and charts, but they need much more storage than text search.

The right choice depends on the documents. Knowledge stored mostly in scanned forms is a different problem from knowledge stored in charts and slides.

Four questions before the next pilot

Who runs the model after launch? Name the team and agree on a budget for that work.
How will we know when quality drops? Agree on the measure and how often someone checks it.
Can we rebuild and retrain it? Record the data, the code and the settings.
What does our data look like? Count how much knowledge sits in charts, scans and slides before choosing a text-only design.

A pilot shows that a model can do a task. These questions decide whether the company can keep it doing that task for years.