<p>In India's hospitality sector, procurement aggregators face a critical challenge: small vendors frequently fail to deliver promised quantities when payments are delayed. This white paper highlights the scope of this issue and introduces proven solutions that are transforming supply chain reliability.</p>
<p>Here’s how some forward-thinking companies are tackling these challenges to streamline operations and enhance efficiency:</p>
<h3 id="directdealerengagementinthebatterysector">Direct Dealer Engagement in the Battery Sector</h3>
<p>Companies like Exide and Amara Raja Batteries have shifted from traditional distributor-led models to direct interactions with dealers, allowing them to offer higher margins. This strategic shift has paid off, with both companies now commanding over 70% of the market share in India’s battery industry. This closer relationship with dealers not only improves margins but also enhances supply chain responsiveness and reliability.</p>
<h3 id="advancedcontractingandsupplierengagement">Advanced Contracting and Supplier Engagement</h3>
<p>McDonald's and Zomato Hyperpure are deepening their relationships with farmers and suppliers through advanced contracts and engagement strategies.</p>
<p>{{quote_1}}</p>
<ul>
<li>McDonald's provides training, quality seeds, and assured purchase agreements at fair prices, ensuring a steady and reliable supply.</li>
<li>Both companies employ sophisticated inventory and supply chain management software that offers real-time visibility and predictive analytics on vendor operations.</li>
</ul>
<p>For example, this technology can forecast potential delivery failures when external factors—like a 10% rise in chicken prices—occur.</p>
<p>{{quote_2}}</p>
<h3 id="tatamotorsinnovativesupplychainfinancing">Tata Motors’ Innovative Supply Chain Financing</h3>
<p>Tata Motors has launched a Supply Chain Financing (SCF) program named "PayEarly" through the CashInvoice platform. This program offers on-demand early payments to MSME vendors, alleviating the common issue of payment delays.</p>
<p>By ensuring vendors receive their payments promptly, Tata Motors enhances their ability to meet supply obligations, thus strengthening the entire supply chain.</p>
<p>{{metric_1}}</p>
<h3 id="thewayforward">The Way Forward</h3>
<p>The whitepaper explores how leading companies are combining different strategies to create resilient supply chains even when working with the smallest vendors.</p>
<p>Learn the specific implementation tactics that are proving successful in India's unique market conditions, and discover how your organization can transform procurement challenges into competitive advantages.</p>


A vendor demonstration searches for an exact title and finds it. So does every other vendor. The dataset that separates them is the one built from your own catalogue.

<blockquote>
<p><strong>The short answer:</strong> no public dataset predicts how an enterprise search tool will behave on your content, because the hard part of your search is your vocabulary and your users' mistakes. Generate the dataset yourself: sample a few hundred items you know exist, produce the ways people actually ask for them in 20 variation classes, add a nonsense set that must return nothing, and add your real failed queries. Score every candidate per class, never only overall.</p>
</blockquote>

This is the method ITMTB uses to evaluate its own search engine on every change and to compare it with whatever a customer had before. It pairs with [how to measure the success of an enterprise search tool](/blog/how-to-measure-enterprise-search-success), which covers what to measure once a tool is live.

<nav>
<p><strong>In this article:</strong></p>
<ol>
  <li><a href="#public">Why public benchmarks do not answer the question</a></li>
  <li><a href="#principle">The principle: known items, realistic damage</a></li>
  <li><a href="#classes">The 20 variation classes</a></li>
  <li><a href="#patterns">Classes that come from live complaints</a></li>
  <li><a href="#nonsense">The nonsense guardrail</a></li>
  <li><a href="#replay">Real failed searches</a></li>
  <li><a href="#scoring">How to score</a></li>
  <li><a href="#fair">Comparing vendors fairly</a></li>
  <li><a href="#build">Building the dataset</a></li>
  <li><a href="#faq">FAQ</a></li>
  <li><a href="#takeaways">Key takeaways</a></li>
</ol>
</nav>

<h2 id="public">Why public benchmarks do not answer the question</h2>

Retrieval research has good public datasets: <a href="https://microsoft.github.io/msmarco/" target="_blank" rel="noopener">MS MARCO</a>, the <a href="https://github.com/beir-cellar/beir" target="_blank" rel="noopener">BEIR</a> collection, the <a href="https://trec.nist.gov/" target="_blank" rel="noopener">TREC</a> tracks. They are the right way to compare retrieval models in general and the wrong way to choose an enterprise search tool, for three reasons.

* **They do not contain your vocabulary.** The failures that matter in a specialist catalogue are specialist: a general dictionary correcting a valid technical term, a two-letter abbreviation missing from the index, a hyphenated product name split into two halves.
* **They do not contain your users' mistakes.** People searching a particular catalogue misspell its words, drop its convention words and reorder its titles in patterns visible only in its logs.
* **They reward a different task.** Most public benchmarks score ranked relevance over long documents for question-style queries. Most enterprise search is known-item retrieval: the user wants one specific thing and cannot name it exactly.

<h2 id="principle">The principle: known items, realistic damage</h2>

Take an item you know exists. Damage the query the way a real person would, keeping the intent intact. Ask whether the item still comes back in the top five.

That is the whole method. The correct answer is known by construction, so no human relevance judgement is needed, and the set can be regenerated at any size on any catalogue.

<h2 id="classes">The 20 variation classes</h2>

These are the classes the harness generates from each sampled title. Some apply only to titles with enough words, a plural or a hyphen, so their counts will be lower than the sample.

<table>
  <thead>
    <tr><th>Class</th><th>What it does to the title</th><th>What it tests</th></tr>
  </thead>
  <tbody>
    <tr><td>exact_title</td><td>Nothing</td><td>Baseline; should be close to 100%</td></tr>
    <tr><td>subject</td><td>Keeps the subject, drops the catalogue's convention words</td><td>Convention handling</td></tr>
    <tr><td>lowercase</td><td>All lower case</td><td>Case handling</td></tr>
    <tr><td>typo_delete</td><td>Deletes one letter</td><td>Fuzzy matching</td></tr>
    <tr><td>typo_insert</td><td>Inserts one letter</td><td>Fuzzy matching</td></tr>
    <tr><td>typo_substitute</td><td>Replaces one letter with a keyboard neighbour</td><td>Realistic typos</td></tr>
    <tr><td>typo_transpose</td><td>Swaps two adjacent letters</td><td>Realistic typos</td></tr>
    <tr><td>typo_plural_noise</td><td>Adds or removes a trailing s</td><td>Stemming</td></tr>
    <tr><td>typo_phonetic</td><td>Spells a word as it sounds</td><td>Phonetic correction</td></tr>
    <tr><td>typo_two_words</td><td>Misspells two words at once</td><td>Correction under load</td></tr>
    <tr><td>order_reverse</td><td>Reverses word order</td><td>Order independence</td></tr>
    <tr><td>order_rotate</td><td>Moves the first word to the end</td><td>Order independence</td></tr>
    <tr><td>order_shuffle</td><td>Random order</td><td>Order independence</td></tr>
    <tr><td>drop_word</td><td>Removes one content word</td><td>Partial recall</td></tr>
    <tr><td>nl_phrasing</td><td>Wraps the title in natural language</td><td>Mandatory-word handling</td></tr>
    <tr><td>punct_strip</td><td>Removes punctuation</td><td>Tokenisation</td></tr>
    <tr><td>hyphen_join</td><td>Joins or splits hyphenated terms</td><td>Tokenisation</td></tr>
    <tr><td>plural_flip</td><td>Singular to plural or back</td><td>Stemming</td></tr>
    <tr><td>combo_order_typo</td><td>Reorders and misspells together</td><td>Compound damage</td></tr>
    <tr><td>abbrev</td><td>Abbreviates a multi-word term</td><td>Abbreviation knowledge</td></tr>
  </tbody>
</table>

The per-class table is what finds the work. On the catalogue where this harness runs in production, the overall top-5 figure is 98.8%, and the class that drove the most engineering was abbreviations, which started around 90%. An overall figure would have hidden it.

<h2 id="patterns">Classes that come from live complaints</h2>

Generated classes test what you anticipated. Once a search is live, people complain about things you did not. Each complaint should become a new class, generated the same way and scored outside the overall figure so it cannot be averaged away.

Classes that have been added this way include: a query made entirely of the catalogue's convention words plus a subject; a single wrong vowel in a technical term; one word more than the title contains; very short subjects that must rank the exact item first; hyphenated compounds that must not match either half on its own; and a geography or year qualifier the catalogue does not use in its titles. Real traffic contained few of those queries, which is why no generated class had caught them and why a dataset needs a channel for complaints.

<h2 id="nonsense">The nonsense guardrail</h2>

A search that returns something for everything has not solved retrieval; it has hidden the zero. Keep a set of queries that must return nothing: invented words, and phrases that combine two unrelated subjects. Score the number of results returned. Good is zero.

This set matters most once a meaning-based component is in the path, because embeddings will find the nearest neighbour of nonsense. A reasonable rule is that the meaning-based channel may only answer when the lexical engine found nothing, and only above a confidence floor; the nonsense set is what calibrates the floor.

<h2 id="replay">Real failed searches</h2>

The third part of the dataset is not generated. Every query from your logs that returned nothing, and every query a user complained about, goes in with the item it should have found. It runs on every change, versioned, and grows with every incident. A fix for one complaint that breaks another shows up here first.

If you are comparing vendors and have no logs yet, collect them first. A week of real queries says more than any amount of generated data about which classes your users actually hit.

<h2 id="scoring">How to score</h2>

<table>
  <thead>
    <tr><th>Score</th><th>Definition</th><th>Rule</th></tr>
  </thead>
  <tbody>
    <tr><td>top-5, top-10 per class</td><td>Share of cases where the known item appears in the first 5 / 10 results</td><td>Report per class. A class dropping while overall holds is a regression.</td></tr>
    <tr><td>top-1 for exact classes</td><td>For exact titles and exact subjects, the item must be first</td><td>Below about 95% means ranking, not recall, is the problem</td></tr>
    <tr><td>p50 latency per class</td><td>Median engine time per class</td><td>Record on the same hardware for every candidate; watch for classes that cost ten times others</td></tr>
    <tr><td>Nonsense leaks</td><td>Results returned for queries that should return none</td><td>Zero</td></tr>
    <tr><td>Replay pass rate</td><td>Real failed queries that now find their item</td><td>100%, and the corpus must grow</td></tr>
  </tbody>
</table>

Two things to know before trusting the numbers:

* **Ties flicker.** Items with identical relevance scores can come back in a different order after an index rebuild, and a case sitting on the top-5 boundary flips. Record the index build with every score, sort ties deterministically if you need stability to a decimal, and treat differences below the noise as noise.
* **Overall hides the class you care about.** A change that lifts long queries can cost the drop-a-word class and move the overall figure by less than the flicker. Only the per-class table shows it.

<h2 id="fair">Comparing vendors fairly</h2>

1. **Same corpus.** Give every candidate the identical export of your items, same fields, same count.
2. **Same dataset.** Generate it once, from one seed, and hand the same file to each. Keep a second seed back to check the result is not an artefact of the first.
3. **Same depth and budget.** Ten results, same hardware class, same timeout. A tool that is excellent at two seconds and poor at 300 milliseconds is a different tool.
4. **Per class, side by side.** Put the candidates' per-class tables next to each other and weight the classes your logs say matter.
5. **Nonsense and replay last.** A candidate that leaks on nonsense or fails your real queries is out, whatever its overall score.
6. **Ask how failures are counted.** The tool you choose will fail sometimes. How a failed, degraded or timed-out search is counted and shown decides whether you will know.

<h2 id="build">Building the dataset</h2>

<table>
  <thead>
    <tr><th>Step</th><th>Output</th></tr>
  </thead>
  <tbody>
    <tr><td>Export item identifiers and titles, plus the one or two fields users search by</td><td>A CSV of your corpus</td></tr>
    <tr><td>Sample a few hundred items at random with a fixed seed</td><td>The known-item list</td></tr>
    <tr><td>Generate the 20 classes per item with a script; the typo classes need only a keyboard-neighbour map and a vowel table</td><td>Query, expected item and class, several thousand rows</td></tr>
    <tr><td>Write a few dozen nonsense queries</td><td>The guardrail set</td></tr>
    <tr><td>Take recent zero-result queries from your logs and find the intended item by hand for the most frequent</td><td>The replay set</td></tr>
    <tr><td>Run each candidate; record the rank of the expected item and the latency per query</td><td>Raw results</td></tr>
    <tr><td>Aggregate per class: top-5, top-10, p50; count leaks; count replay passes</td><td>The comparison table</td></tr>
  </tbody>
</table>

Once the engine is live, the same dataset becomes the gate on every later change, and the live measures in [how to measure enterprise search success](/blog/how-to-measure-enterprise-search-success) take over from there. Where the engine also serves AI agents as a tool, run the dataset through the tool interface as well; an agent should see exactly what the search box sees.

ITMTB generates this dataset from a prospective customer's catalogue and runs it against their current search before scoping any build, so the decision rests on a per-class table rather than a demonstration; see the [enterprise search](/enterprise-search) page. [Contact us](/contact-us).

<h2 id="faq">Frequently asked questions</h2>

<h3 id="faq-public">Is there a public dataset for comparing enterprise search solutions?</h3>

Public retrieval benchmarks exist, but they score an engine on general documents and question-style queries. They do not contain your items, vocabulary or your users' mistakes, so they cannot predict behaviour on your catalogue.

<h3 id="faq-size">How big does it need to be?</h3>

A few hundred items generating several thousand queries across 20 classes detects a one-point change in a class on a catalogue of tens of thousands of items. Use two seeds.

<h3 id="faq-contain">What should it contain?</h3>

Known-item queries in realistic variation classes, a nonsense set that must return nothing, and your real failed queries with the item each should have found.

<h3 id="faq-fair">How do you compare vendors fairly?</h3>

Same corpus, dataset, depth, hardware budget and day. Score top-5, top-10 and latency per class, leaks on nonsense, passes on replay. Compare per class.

<h3 id="faq-exact">Why not test with exact titles?</h3>

Because users rarely type them, and every engine passes. The realistic classes are where engines differ.

<h3 id="faq-semantic">Should semantic or AI search use the same dataset?</h3>

Yes, with the nonsense guardrail weighted heavily, because meaning-based matching tends to answer everything.

<h2 id="takeaways">Key takeaways</h2>

* No public dataset predicts your enterprise search outcome. Generate one from your own items, in realistic variation classes, plus nonsense and your real failures.
* A few hundred items, several thousand queries, two seeds and 20 classes is enough to see one-point changes.
* Score per class. A class dropping while overall holds is a regression.
* Zero leaks on nonsense. A search that answers everything has hidden the zero, not solved it.
* Add every real complaint as a new class.
* Compare vendors on the same corpus, dataset, depth and budget, per class, and ask how they count their own failures.

<h2 id="source-note">Source note</h2>

The class list and scores are from the evaluation harness of ITMTB's enterprise search for The Business Research Company, described in [the case study](/blog/enterprise-search-case-study-business-research-company).



Someone asks whether the new search is better. Everyone has an opinion. Nobody has a number that would survive a second question.

<blockquote>
<p><strong>The short answer:</strong> ITMTB measures enterprise search on three layers. <strong>Offline:</strong> a known-item test where realistic variations of items you know exist must return that item in the top five, tracked per variation class. <strong>Live:</strong> zero-result rate deduplicated by intent, fallback and degraded counters, latency against a budget, and clicks attributed to the search that produced them. <strong>Business:</strong> catalogue reach, the share of items that search ever surfaces, and the demand visible in honest zero results. Most of the difficulty is in the counting.</p>
</blockquote>

This article sets out the twelve measures ITMTB uses for an enterprise search deployment: what each one is for, how it is computed, what good looks like, and where the reading was wrong before the counting was fixed. The figures come from one live deployment on a market research publisher's catalogue.

<nav>
<p><strong>In this article:</strong></p>
<ol>
  <li><a href="#three-layers">Why one metric is never enough</a></li>
  <li><a href="#offline">Layer 1: offline measures, before anything ships</a></li>
  <li><a href="#live">Layer 2: live measures, the first month and after</a></li>
  <li><a href="#business">Layer 3: business measures</a></li>
  <li><a href="#scorecard">The scorecard on one page</a></li>
  <li><a href="#traps">Five ways the numbers were wrong</a></li>
  <li><a href="#plan">A first-month measurement plan</a></li>
  <li><a href="#faq">FAQ</a></li>
  <li><a href="#takeaways">Key takeaways</a></li>
</ol>
</nav>

<h2 id="three-layers">Why one metric is never enough</h2>

Every single metric for search has a way of being satisfied while search gets worse.

* A zero-result rate can be driven to nothing by returning something for every query.
* An offline accuracy score can stay high while a whole class of real queries fails, because the test set never contained that class.
* Click-through rate rises when fewer results are shown, and falls when more good ones are.
* Latency improves when the engine gives up early and falls back to the old search.

So the measures come as a set spanning three layers: what the engine does with queries you control, what it does with the queries real people type, and what that changes for the business.

<h2 id="offline">Layer 1: offline measures, before anything ships</h2>

Offline measures are the only ones readable on day one, and the only ones that say whether a change made search better or merely different.

<h3 id="m1">Measure 1: known-item accuracy, per variation class</h3>

Take a sample of items you know exist. For each, generate the ways a real person would ask for it badly: a letter deleted, two letters swapped, a word dropped, the words reordered, an abbreviation, a plural flipped, the subject without the catalogue's title convention, a natural-language phrasing. Run every variation and record whether the intended item appears in the top five and top ten.

Report the result per class, not only overall. The baseline on that catalogue is 450 sampled titles producing about 7,100 test queries per random seed, scored across 20 classes. Overall top-5 is 98.8%. The weakest class, abbreviations, began around 90% top-5, and that one row drove more engineering than the overall figure did.

The rule: a class dropping while the overall figure holds is a regression. A change that improved long queries by a few points quietly cost the drop-a-word class, and the overall number moved by less than the noise.

Building the test set, including the class list, is covered in [a realistic dataset for comparing search solutions](/blog/enterprise-search-evaluation-dataset).

<h3 id="m2">Measure 2: real failed queries, replayed</h3>

Generated variations test what you imagined. Historical failures test what happened. Every query from the logs that returned nothing, or that a customer complained about, goes into a second corpus with the item it should have found, versioned, and runs on every change alongside the generated set.

Good is 100% on this set and a steadily growing count of cases.

<h3 id="m3">Measure 3: the nonsense guardrail</h3>

A search that never returns zero returns something for nonsense. Keep a set of queries that should return nothing: made-up words, phrases that combine two unrelated subjects. Score the number of results returned. Good is zero. This matters more once any meaning-based or fuzzy matching is in the path, because those components tend to answer everything.

<h3 id="m4">Measure 4: engine latency at the median and the tail</h3>

Measure the engine on its own, per class, at p50 and p95. On the catalogue above most classes answer in 6 to 20 milliseconds at the median. Recording this offline is how a ranking improvement that costs time gets noticed before it meets the live budget: one relaxation change pushed title-shaped queries to 24 milliseconds, which was fine, and a later one pushed hyphenated terms past 300 milliseconds on the production box, which was not.

<h2 id="live">Layer 2: live measures, the first month and after</h2>

<h3 id="m5">Measure 5: zero-result rate, by intent</h3>

The share of real searches that returned nothing. The denominator matters: count intents, one query within a ten-minute window, not events. The first version of the console counted events and overstated searches by about a quarter.

What good looks like depends on where you start. The catalogue began at 53.4% zero results with the previous search. A random sample of 500 of those failures found that 470 had a relevant existing item, so the achievable floor was low. In the first days after the new engine went live, 5.1% of search intents returned nothing. The two figures have different denominators, raw searches before and intents after, and on either basis the fall is an order of magnitude.

Split the honest zeros out and keep them. A query that genuinely has no answer in the catalogue is demand.

<h3 id="m6">Measure 6: a result quality label on every search</h3>

Zero or non-zero is too coarse. Label each live search: zero, weak (results came back but none shares a subject word with the query), ok, or rescued (answered by a secondary channel such as spelling correction or meaning-based matching after the primary matching found nothing). The weak label finds the cases that embarrass: a three-word query that returned unrelated reports because the engine relaxed to its most common word. That one was found by reading the explorer, and fixed by making the rarest word in a query mandatory.

Good is a weak share that falls week on week and a rescued share you understand.

<h3 id="m7">Measure 7: fallback, degraded and dropped counters</h3>

Every swallowed error in the search path is a counter on the health endpoint: fallbacks (the proxy gave up on the new engine and served the old one), degraded searches (a component failed but the search still answered), dropped telemetry, dropped clicks. Good is all zero. Non-zero counters are the difference between "search seems fine" and "search was fine for everyone except the people who typed hyphenated terms during the hour the timeout was too short."

Treat the timeout as a measure in its own right. The type-ahead budget on this deployment started at 300 milliseconds and moved to 600 after live data showed legitimate queries being cut off into fallback.

<h3 id="m8">Measure 8: latency per surface, against the budget</h3>

Type-ahead and a results page are different products with different budgets, so report them separately, at p50 and p95, per day, with the sample count next to every percentile. On this deployment the results-page rows carried no latency for the first week, because the path that logged them never recorded a timing; the tiles showed the type-ahead figures, which were excellent. A missing measure looks like a good one unless the sample size is printed.

<h3 id="m9">Measure 9: clicks, attributed to the search, per surface</h3>

A click is only meaningful when it can be joined to the search that produced it. Give every search an identifier when it is answered, hand it to the page, and send it back with each click together with the position clicked and the surface. Clicks that arrive with an identifier the engine never issued are kept but never counted, which also keeps automated traffic out of the click figures.

Then read three things: click rate per surface, position of the click, and searches that showed results nobody clicked. The last is the clearest "results did not convince" signal available.

<h2 id="business">Layer 3: business measures</h2>

<h3 id="m10">Measure 10: catalogue reach</h3>

The share of items that search surfaced at all in a period. In a sampled week before the change, the previous search had surfaced under 15% of the catalogue. Testing the corrected engine against the sample of failed queries alone surfaced more than a thousand reports that had not appeared in any result that week. A catalogue that cannot be reached through search is inventory being paid for and not sold.

<h3 id="m11">Measure 11: demand visible in honest zeros</h3>

Once zero results are mostly honest, the list of what people searched for and did not find is a product signal. Rank it by distinct intents. The console shows it for any window, and the publisher reads it as a commissioning list.

<h3 id="m12">Measure 12: downstream conversion from search</h3>

The measure everyone wants first and should take last, because it needs the other eleven to be interpretable. Join searches to the actions that matter in your business: an item opened, a sample requested, an order, a ticket resolved. Compare searches that converted with searches that did not, by quality label and by click position. Expect weeks of data before per-query figures mean anything on a site with modest search volume; aggregate figures are useful sooner. The conversion events live in commerce or service systems, not in the search engine, so this is an analytics join rather than a search feature; ITMTB's [data and AI practice](/services/analytics-ai-automation) treats it that way.

<h2 id="scorecard">The scorecard on one page</h2>

<table>
  <thead>
    <tr><th>#</th><th>Measure</th><th>Layer</th><th>What good looks like</th><th>Read it</th></tr>
  </thead>
  <tbody>
    <tr><td>1</td><td>Known-item accuracy, per class</td><td>Offline</td><td>&gt;98% top-5 overall; no class falls on a change</td><td>Every change</td></tr>
    <tr><td>2</td><td>Real failed queries replayed</td><td>Offline</td><td>100%, corpus growing</td><td>Every change</td></tr>
    <tr><td>3</td><td>Nonsense guardrail</td><td>Offline</td><td>0 leaks</td><td>Every change</td></tr>
    <tr><td>4</td><td>Engine latency p50 / p95 per class</td><td>Offline</td><td>Well inside the live budget</td><td>Every change</td></tr>
    <tr><td>5</td><td>Zero-result rate, by intent</td><td>Live</td><td>About 5%, remainder honest</td><td>Daily</td></tr>
    <tr><td>6</td><td>Quality label (zero / weak / ok / rescued)</td><td>Live</td><td>Weak share falling</td><td>Weekly</td></tr>
    <tr><td>7</td><td>Fallback, degraded, dropped counters</td><td>Live</td><td>All zero</td><td>Continuously</td></tr>
    <tr><td>8</td><td>Latency per surface vs budget, with sample size</td><td>Live</td><td>p95 inside budget, no empty samples</td><td>Daily</td></tr>
    <tr><td>9</td><td>Attributed clicks, per surface and position</td><td>Live</td><td>Unclicked-with-results share falling</td><td>Weekly, after a few weeks</td></tr>
    <tr><td>10</td><td>Catalogue reach</td><td>Business</td><td>Rising share of items ever surfaced</td><td>Monthly</td></tr>
    <tr><td>11</td><td>Demand in honest zeros</td><td>Business</td><td>A ranked list someone acts on</td><td>Monthly</td></tr>
    <tr><td>12</td><td>Downstream conversion from search</td><td>Business</td><td>Converting share rising by label</td><td>Monthly, after a quarter</td></tr>
  </tbody>
</table>

When evaluating a tool rather than running one, measures 1 to 4 can be run on your own data before any contract, and the console that shows 5 to 9 should exist on day one.

ITMTB runs the offline layer against a prospective customer's catalogue and real failed queries before scoping a build; the offer is described on the [enterprise search](/enterprise-search) page. [Contact us](/contact-us).

<h2 id="traps">Five ways the numbers were wrong</h2>

<h3 id="trap-double">1. Every Enter was counted twice</h3>

A person types "car" in the header, sees the dropdown, presses Enter and lands on a results page. That is one search and two events, sometimes from two IP addresses because the type-ahead travelled over IPv4 and the page request over IPv6. In one week's sample, most results-page rows had a type-ahead twin within ten minutes and few shared its IP; about a quarter of events were being counted twice. The fix was to define a search as a distinct query within a ten-minute window and to drop the IP from that key.

<h3 id="trap-bots">2. Automated traffic inflates searches but never clicks</h3>

Bots hit search endpoints. They do not pick up the identifier the engine issues and send it back with a click. So search counts carry some noise and click joins carry none, which is one reason to read click rate as a ratio with care and the unclicked-with-results list with more confidence.

<h3 id="trap-flicker">3. The offline score moved without a code change</h3>

The same test set scored 98.8% one run and 98.6% the next with nothing changed in the engine. After an index rebuild, items with identical relevance scores came back in a different order, and a few cases sat exactly on the top-5 boundary. Treat differences below the noise as noise, record the index build with each score, and sort ties deterministically if the figure needs to be stable to a decimal.

<h3 id="trap-rescued">4. Rescued is not the same as answered</h3>

Once a meaning-based channel answers queries the lexical engine could not, the zero-result rate falls. That is good only if the rescued answers are right. Label them, count them separately, and read them in the explorer weekly. Clicks on rescued results, once click attribution exists, are the evidence that the channel is worth its cost.

<h3 id="trap-missing">5. A missing measure looked like a good one</h3>

The results page had no latency figure for a week, and the percentile tiles showed the type-ahead numbers. Nothing was wrong except the reading. Every percentile now carries its sample size, and a surface with zero samples shows a dash.

<h2 id="plan">A first-month measurement plan</h2>

<table>
  <thead>
    <tr><th>When</th><th>Do</th><th>Decide</th></tr>
  </thead>
  <tbody>
    <tr><td>Before launch</td><td>Build the known-item set from your items and your failed queries. Run measures 1 to 4. Fix anything under 95% in a class. Set latency budgets per surface.</td><td>Go or no-go on retrieval quality</td></tr>
    <tr><td>First week</td><td>Read counters hourly on day one, then daily. Read every weak and rescued search by hand. Confirm intent dedup and sample sizes on the console.</td><td>Timeout and relaxation adjustments</td></tr>
    <tr><td>Rest of the month</td><td>Add every real failure to the replay corpus. Start click attribution on every surface. Read zero-result and weak shares weekly.</td><td>Which failure classes need engine work and which a business rule</td></tr>
    <tr><td>Second month onward</td><td>Catalogue reach, honest-zero demand, conversion join. Clicks become test cases and, bounded, a ranking signal.</td><td>Where search is leaving money</td></tr>
  </tbody>
</table>

The same measures serve a search used by AI agents as a retrieval tool, with one addition: track how often the agent's query returned nothing or something weak, because the agent reasons on whatever it gets back and the failure looks like a model problem. That dependency is covered in [AI agents in business workflows](/blog/ai-agents-business-workflow-automation).

<h2 id="faq">Frequently asked questions</h2>

<h3 id="faq-single">What is the single best metric for enterprise search success?</h3>

There is no single one. The pair that works is a known-item test score measured offline, together with the live zero-result rate deduplicated by intent. Either alone can be gamed; together they test relevance and coverage.

<h3 id="faq-zero">What is a good zero-result rate?</h3>

Around 5% of distinct intents is achievable on a well-indexed catalogue. One catalogue went from 53.4% to 5.1% in the first days live. The remainder should be honest zeros, which are demand signals rather than failures.

<h3 id="faq-before">How do you measure before launch?</h3>

With an offline evaluation: items you know exist, realistic variations of how people ask for them, scored top-5 and top-10 per variation class on every change, with any class dropping treated as a regression.

<h3 id="faq-count">Why is a raw search count a poor measure?</h3>

One person's search produces many events: a type-ahead request per keystroke burst, then a results-page request, sometimes from two IP addresses. Counting events overstated searches by roughly a quarter. Count distinct intents.

<h3 id="faq-ctr">Should click-through rate be used?</h3>

Yes, per surface, and only once a click can be attributed to the search that produced it. The most useful click signal is searches that returned results nobody clicked.

<h3 id="faq-time">How long before live metrics are trustworthy?</h3>

A few weeks on a site with modest search volume for aggregate measures. Per-query click rates stay noisy longer.

<h3 id="faq-tool-vs-deploy">Tool metric or deployment metric?</h3>

Tool metrics describe retrieval in isolation: offline accuracy, latency, corrections. Deployment metrics describe the tool inside your traffic: zero-result rate, fallbacks, counters, index freshness, clicks. A good tool can be a poor deployment.

<h2 id="takeaways">Key takeaways</h2>

* Measure on three layers: offline (known-item accuracy per class, replayed failures, nonsense guardrail, latency), live (zero-result by intent, quality label, counters, latency per surface, attributed clicks) and business (catalogue reach, honest-zero demand, conversion).
* Report offline accuracy per variation class. A class dropping while the overall figure holds is a regression.
* Count intents, not events. One search produced two events and sometimes two IP addresses.
* Print sample sizes next to percentiles. A missing measure looks like a good one.
* Attribute clicks to the search that produced them, per surface. Read unclicked-with-results before click-through rate.
* Measures 1 to 4 can be run on your own data before choosing a tool; the console for 5 to 9 should exist on day one.

<h2 id="source-note">Source note</h2>

The deployment behind these figures is ITMTB's enterprise search for The Business Research Company, described in [the case study](/blog/enterprise-search-case-study-business-research-company).



The business sees the catalogue. The user sees the search box. When the two disagree, the user is the one who is right about what exists.

<blockquote>
<p><strong>The short answer:</strong> enterprise search is important because it is the point at which everything an organisation owns either becomes usable or stays invisible, and the cost of getting it wrong is rarely visible to the business, because the business never sees the result the user did not get. On one market research catalogue ITMTB worked on, 53.4% of searches in a sampled week returned nothing, and a random sample of 500 of those failures found an existing relevant report for 470 of them. After the retrieval was fixed, the figure was 5.1%. The same dependency now sits under AI assistants and agents, which retrieve before they reason.</p>
</blockquote>

This article gives six reasons, each with the evidence behind it, and a short way to find out whether your own search is a problem. The retrieval failures it draws on are described in [Improving enterprise search for people and AI agents](/blog/enterprise-search-real-world-queries).

<nav>
<p><strong>In this article:</strong></p>
<ol>
  <li><a href="#reason-1">1. The user decides what exists</a></li>
  <li><a href="#reason-2">2. Most "no results" are retrieval failures, not content gaps</a></li>
  <li><a href="#reason-3">3. Unfindable inventory is inventory you pay for</a></li>
  <li><a href="#reason-4">4. Search is where customers say what they want</a></li>
  <li><a href="#reason-5">5. AI agents retrieve before they reason</a></li>
  <li><a href="#reason-6">6. Search quality is measurable, so its cost is avoidable</a></li>
  <li><a href="#who">Who should care most</a></li>
  <li><a href="#cost">Putting a number on it for your business</a></li>
  <li><a href="#check">A short check</a></li>
  <li><a href="#faq">FAQ</a></li>
  <li><a href="#takeaways">Key takeaways</a></li>
</ol>
</nav>

<h2 id="reason-1">1. The user decides what exists</h2>

No buyer phones to ask whether the empty result was a bug. They conclude the product, report or document does not exist and leave, or they try again with different words. In the sampled week, hundreds of sequences of three or more consecutive failed searches showed people reformulating, failing, reformulating again.

For an employee the conclusion is the same with a different cost: they stop searching and ask a colleague, or rebuild the thing that already exists.

Search is the organisation's answer to the question "do you have this?" When the answer is wrong, nothing downstream corrects it.

<h2 id="reason-2">2. Most "no results" are retrieval failures, not content gaps</h2>

The reflex when zero-result searches are high is to assume people search for things you do not sell. The data said otherwise. A random sample of 500 distinct failed searches from the week was run against an index built around the same catalogue. 470 returned a relevant report that already existed. For that sample, 94% of failures were the search failing to connect how people asked with how content was stored.

The causes were small and specific: words such as "latest" or "report" treated as mandatory, longer queries punished because every word had to match, specialist terms "corrected" into something else by a general dictionary, and two-character terms such as AI, 5G and EV absent from the index because of a database default. None of them required a new technology. All of them required measurement.

<h2 id="reason-3">3. Unfindable inventory is inventory you pay for</h2>

A catalogue only earns from the items that can be reached. In the same week, under 15% of the catalogue appeared in any result for anyone. Testing the corrected engine against the sample of failed queries alone surfaced more than a thousand reports that had not appeared at all that week.

The rest of the catalogue was being written, updated, priced and hosted for readers who could not get to it. This measure, catalogue reach, is tracked monthly; it is one of the [twelve measures used for enterprise search success](/blog/how-to-measure-enterprise-search-success).

<h2 id="reason-4">4. Search is where customers say what they want</h2>

Once retrieval failures are fixed, the searches that still return nothing become honest: the catalogue genuinely lacks what was asked for. That list, ranked by how many distinct people asked, is a commissioning list. On this deployment it appears on a console tab and the publisher reads it as demand.

Two more signals live in the same logs. Which results people clicked, and from which position, says whether ranking matches intent. Searches that showed results nobody clicked say the results did not convince.

<h2 id="reason-5">5. AI agents retrieve before they reason</h2>

Every assistant or agent that answers from your content, whether called RAG or a tool call, does two things in order: it retrieves a subset of your content, then it reasons about that subset. If retrieval returns the wrong subset, or nothing, the model produces a confident answer about the wrong thing, and the failure is diagnosed as a model problem. It is a search problem.

This is why retrieval is treated here as a foundation rather than a feature, and why it appears among [the technology shifts expected to matter most in 2026](/blog/technology-trends-india-2026). On that deployment, the same engine that serves the website's search box is exposed to AI assistants as a tool through the Model Context Protocol, so a person and an agent asking the same question get the same answer. Where several agents share a retrieval layer in production, the sharing becomes an orchestration concern, which is what <a href="https://orchestrik.ai" target="_blank" rel="noopener">Orchestrik</a> handles in ITMTB's agent deployments.

An organisation that improves search for people improves every agent it builds afterwards. One that adds agents on top of poor search automates the wrong answer.

<h2 id="reason-6">6. Search quality is measurable, so its cost is avoidable</h2>

Search is one of the few parts of an enterprise system that can be scored objectively before and after a change. Take items you know exist, generate the ways people would ask for them badly, and measure how often the right item appears in the top five. On the catalogue above that figure is 98.8% across twenty classes of misspelling, reordering, dropped words and abbreviations, tracked per class on every change. Live, the share of searches returning nothing is read daily, deduplicated so one person's typing does not count as five searches.

Because the measures exist, the cost of poor search is a choice rather than a condition. The method is in [how to measure the success of an enterprise search tool](/blog/how-to-measure-enterprise-search-success); the test set that makes it repeatable is in [a realistic dataset for comparing enterprise search solutions](/blog/enterprise-search-evaluation-dataset).

<h2 id="who">Who should care most</h2>

<table>
  <thead>
    <tr><th>Organisation</th><th>Where search decides value</th><th>The failure it hides</th></tr>
  </thead>
  <tbody>
    <tr><td>Publishers, research houses, data vendors</td><td>A buyer knows the market, not the report title</td><td>Sales lost to "we don't have that"</td></tr>
    <tr><td>Product and parts catalogues</td><td>Model numbers, abbreviations, half-remembered names</td><td>Orders placed elsewhere, or for the wrong part</td></tr>
    <tr><td>Documentation-heavy companies</td><td>Specifications, policies, procedures by colloquial name</td><td>Rework and non-compliance</td></tr>
    <tr><td>Support operations</td><td>Agents searching a knowledge base under time pressure</td><td>Longer handling time, inconsistent answers</td></tr>
    <tr><td>Internal knowledge</td><td>Employees searching from memory with project shorthand</td><td>Duplicated work, asking colleagues</td></tr>
    <tr><td>Anyone building AI agents on their content</td><td>Retrieval feeds every answer</td><td>Confident wrong answers blamed on the model</td></tr>
  </tbody>
</table>

The common factor is a user who knows roughly what they want but not its exact name. Most of the content that matters sits inside the [enterprise applications](/services/enterprise-applications) already in use, which is where the indexing question starts.

<h2 id="cost">Putting a number on it for your business</h2>

No revenue figure is published for the deployment above, because the conversion join runs on the customer's commerce data. Your own exposure can be estimated with four inputs you probably already have.

<table>
  <thead>
    <tr><th>Input</th><th>Where to get it</th></tr>
  </thead>
  <tbody>
    <tr><td>Searches per month</td><td>Your analytics or server logs</td></tr>
    <tr><td>Share returning nothing</td><td>Same logs, or a short period of sampling</td></tr>
    <tr><td>Share of those with an existing answer</td><td>Replay a random sample of 100 failures by hand</td></tr>
    <tr><td>Value of a successful search</td><td>Conversion rate of users who find a result, times order or ticket value</td></tr>
  </tbody>
</table>

Multiply the first three to get the number of monthly searches that failed although the answer existed. Multiply by the fourth for a monthly figure. On the catalogue above the second and third were 53.4% and 94%: half of all searches ended with a buyer told the answer did not exist.

<h2 id="check">A short check</h2>

1. Pull recent zero-result searches. If you cannot, that is the first finding.
2. Take 50 at random. For each, look for the item by hand. Count how many you find.
3. Pick ten things you know exist. Search for each badly: misspell it, drop a word, reorder it, use the words a customer would use. Count how many still appear in the top five.
4. Search for your shortest important terms: two-letter abbreviations, hyphenated names.

If step 2 finds answers for most failures, or step 3 loses more than one in ten, the search is hiding content you own. The fix is a retrieval project scoped against a measure, not a platform replacement.

If this is relevant to something you are working on, ITMTB runs the check above on a sample of a catalogue's real queries; the full offer is on the [enterprise search](/enterprise-search) page. [Contact us](/contact-us).

<h2 id="faq">Frequently asked questions</h2>

<h3 id="faq-why">Why is enterprise search important?</h3>

Because it decides whether the content an organisation already owns can be found by the people and systems that need it. When search fails, the user concludes the thing does not exist. On one catalogue, 53.4% of searches returned nothing while 470 of 500 sampled failures had an existing answer; after the fix, 5.1%.

<h3 id="faq-cost">What does poor enterprise search cost?</h3>

Lost sales, employee time, support and research work started from incomplete information, inventory never surfaced, and AI assistants reasoning on the wrong context. Most of it is invisible because the business never sees the result the user did not get.

<h3 id="faq-ai">Is enterprise search still important now that AI assistants exist?</h3>

More so. Assistants and agents retrieve first and reason second. Poor retrieval produces confident wrong answers that look like model problems.

<h3 id="faq-box">Enterprise search versus a website search box?</h3>

The box is one surface. Enterprise search is the retrieval capability behind every surface, including agents calling it as a tool. One engine, one index, one vocabulary.

<h3 id="faq-who">Who needs it most?</h3>

Publishers, catalogues, documentation-heavy companies, support operations, internal knowledge, and anyone building agents on their own content. The common factor is a user who knows roughly what they want but not its exact name.

<h3 id="faq-know">How do I know whether mine is a problem?</h3>

Measure the share of searches returning nothing and sample them for existing answers. Then search for things you know exist, badly. If owned content keeps disappearing, the problem is measurable and usually fixable.

<h2 id="takeaways">Key takeaways</h2>

* The user, not the catalogue, decides what exists. A failed search is a "no" the business never hears.
* Most zero results are retrieval failures: 470 of 500 sampled failures had an existing answer.
* Unfindable inventory is paid for and never sold: under 15% of one catalogue surfaced in a week.
* Honest zero results, clicks and unclicked results are customer demand data you already collect.
* AI agents retrieve before they reason. Fix search for people and every agent improves.
* Search quality is measurable before and after a change, so its cost is a choice.

<h2 id="source-note">Source note</h2>

The deployment behind these figures is ITMTB's enterprise search for The Business Research Company, described in [the case study](/blog/enterprise-search-case-study-business-research-company).


A Realistic Dataset for Comparing Enterprise Search Solutions

How to Measure the Success of an Enterprise Search Tool: 12 Measures From a Live Deployment

Why Enterprise Search Is Important: The Cost of Content Nobody Can Find

Technology Trends in India 2026: 8 Shifts Enterprise Leaders Should Act On

Agents for Procurement Workflows: Handling the Exceptions

Improving Enterprise Search for People and AI Agents

AWS Cost Optimization: How to Find Cloud Resources You Are Paying For but No Longer Need

MCP for Business: Can AI Agents Become a New Software Distribution Channel?

Cloud Cost Comparison: How to Estimate Public Cloud Cost for Your Business