One thing I’ve noticed from many years attending search conferences is that the return on investment (ROI) in search technology is hard to calculate: this is particularly difficult when considering intranet and/or enterprise search, as users can usually find another way to answer their question. In most cases, some slightly tired numbers are trotted out from years-old studies, based on how much employee time a better search engine might save. I’ve never really believed this to be a sensible metric; however if you’re trying to sell a search solution (especially an overpriced, closed source, magic black box that ‘understands your content’) it may be all you can rely on.
We’ve been working recently with a couple of clients who sell significant volumes of products via their websites: in both cases the current search solution is underperforming. In this situation it’s far easier to justify an investment in better search: if customers can’t find products they simply won’t be buying them. Simple changes to the search algorithms can have huge impacts, costing or saving the company millions in revenue. However, installing a new search engine is just the beginning – it’s vital to consider a solid test strategy. To start with, the new engine should be at least as good as the old one in terms of relevance. The search logs will show what terms your users have typed into the search box, and these can be used to construct a set of queries that can be tested against both engines, with the results being scored by content experts, beta testers and even small groups of real customers. The results of this scoring can be used to inform how the new engine can be tuned – and of course, this should be an ongoing process, as search should never be built as a ‘fire and forget’ project.
There’s sadly little available on the subject of real-world relevance testing, although there’s a forthcoming book by Doug Turnbull and and John Berryman. Based on our work with the clients above, we hope later this year to be able to talk further about relevance testing and tuning – and how to do it right for e-commerce, avoiding significant financial risk.
UPDATE: Doug has been kind enough to give our readers a discount code for his book – “39turnbull” – not sure how long this will last but it gives you 39% off the price so worth a try!
Also I’ve encounter some companies that only think that they should invest in better search (technologies, monitoring, infrastructure) if they were search-focused companies, when in reality almost everyone needs search.