Let’s start by turning the clock back a few years. Ten or fifteen years ago search engines were usually closed source, mysterious black boxes, costing five or six-figure sums for even relatively modest installations (let’s say a couple of million documents – small by today’s standards). Huge amounts of custom code were necessary to integrate them with other systems and projects would take many months to demonstrate even basic search functionality. The trick was to get search working at all, even if the eventual results weren’t very relevant. Sadly even this was sometimes difficult to achieve.

Nowadays, search technology has become highly commoditized and many developers can build a functioning index of several milion documents in a couple of days with off-the-shelf, open source, freely available software. Even the commercial search firms are using open source cores – after all, what’s the point of developing them from scratch? Relevance is often ‘good enough’ out of the box for non business-critical applications.

A relevance engineer is required when things get a little more complicated and/or when good search is absolutely critical to your business. If you’re trading online, search can be a major driver of revenue and getting it wrong could cost you millions. If you’re worried about complying with the GDPR, MiFID or other regulations then ‘good enough’ simply isn’t if you want to prevent legal issues. If you’re serious about saving the time and money your employees waste looking for information or improving your business’ ability to thrive in a changing world then you need to do search right.

So what search engine should you choose before you find a relevance engineer to help with it? I’m going to go out on a limb here and say it doesn’t actually matter that much. At Flax we’re proponents of open source engines such as Apache Lucene/Solr and Elasticsearch (which have much to recommend them) but the plain fact is that most search engines are the same under the hood. They all use the same basic principles of information retrieval; they all build indexes of some kind; they all have to analyze the source data and user queries in much the same way (ignore ‘cognitive search’ and other ‘AI’ buzzwords for now, most of this is marketing rather than actual substance). If you’re using Microsoft Sharepoint across your business we’re not going to waste your time trying to convince you to move wholesale to a Linux-based open source alternative.

Any modern search engine should allow you the flexibility to adjust how data is ingested, how it is indexed, how queries are processed and how ranking is done. These are the technical tools that the relevance engineer can use to improve search quality. However, relevance engineering is never simply a technical task – in fact, without a business justification, adjusting these levers may make things worse rather than better.

In the next post I’ll cover how a relevance engineer can engage with a business to discover the why of relevance tuning. In the meantime you can read Doug Turnbull’s chapter in the free Search Insights 2018 report by the Search Network (the rest of the report is also very useful) and you might also be interested in the ‘Think like a relevance engineer’ training he is running soon in the USA. Of course, feel free to contact us for details of similar UK or EU-based training or if you need help with relevance engineering.

The post Defining relevance engineering, part 1: the background appeared first on Flax.

There’s also a small group of people who tend to be quieter about their expertise, perhaps because as independent practitioners or small business owners they’re not supported by the marketing budgets of large companies. These people survive on their reputation, which has been built steadily on a record of solid advice, honesty and neutrality. I’m now lucky enough to be part of this group – an informal network of experts in subjects as diverse as search for Sharepoint, intranet strategy and taxonomy management. Occasionally we collaborate on projects, often we recommend each other to our clients and it’s always hugely enjoyable to meet in person and discuss the latest trends and industry landscape. This informal network means Flax can offer more services to our clients – and if we can’t help, we probably know someone we trust who can.

So I’m very proud to announce that this group – the Search Network – are releasing a joint publication, Search Insights 2018. In this 70-page collection of essays you can learn how to research, procure, choose, budget, plan and run a search project in the best way for your business and your users.

Unlike some other industry reports, we’re not charging for this report, you won’t have to register or give us your email address, and it’s Creative Commons licensed so you can even redistribute it if you like (with attribution). There’s no sponsorship, no plotting of vendors on confusing trend diagrams, no marketing buzzwords or direct recommendations – after all, we’re independent. We welcome any feedback you have of course.

My personal thanks to Martin White who has led this effort and who has also written about the Network and the report.

The post Search Insights 2018 – a free, independent report on search appeared first on Flax.

Our intention was to combine our skills and experience, drawn both from academia and industry, and create something that would be useful for both those interested in researching enterprise search (perhaps for a PhD) and those developing actual search systems. Once you’ve read the book itself there are over 20 pages of references (derived both from academia and industry) to follow up. Hopefully the book will complement Martin White’s seminal Enterprise Search (Martin has kindly reviewed it for us here).

Huge thanks must go of course to my co-author Udo, Mark Sanderson at Now Publishing, those who reviewed early drafts including Martin White and David Hawking and all those who have provided help and information. The book will be available in printed form at the SIGIR 2017 conference in Tokyo and the publishers have also made the entire book available for free download until July 29th. We would of course be very grateful for any feedback! We’re also planning some joint presentations this autumn to introduce the book.

We’d also like to hope that our book goes a little way towards explaining what is still a much misunderstood, over-hyped and over-sold field – as Martin writes today.

The post Announcing our new book, Searching the Enterprise appeared first on Flax.