This process is slightly different each time, but usually includes the following steps, shown with some examples of questions we might ask.

• A review of any specifications created for the search function – these might include design documents, requirements lists and external references. Were these documents complete and well-constructed? Did they take into account best practice, competitors’ implementations, refer to literature? Most importantly, were they followed as the search function was built?
• A series of interviews with users both within and if at all possible external to the organisation. What is your experience of search? Does it help or hinder your job? What do you think could be done better?
• A comparison with competitors/peers in the same market area or sector. Who else does search well and why?
• Detailed tests of the search function. How relevant are the results presented? Is the user experience (UX) good? Does it handle wildcards, mis-spellings, quoted phrases, foreign language? Are the facets useful?
• A review of the technical implementation. Is indexing efficient and complete? Is the underlying software up to date? How is configuration managed and changed? How is testing performed?
• What’s the overall picture? What is done well and done badly? What are the key  prioritised recommendations for improving the search function?

Obviously, the more access we are given to people, software and documents during this process, the more detailed our examination can be. We realise this isn’t always easy! It’s also important to realise that we wouldn’t be doing our job if we pulled our punches – we will tell you exactly what we think is good and bad and back this up with detailed references.

We’ve carried out a number of reviews of this type over the years (a good example is the U.K.’s Office for National Statistics – you can even read the report we created). Most recently, we reviewed the website for a global consulting firm, ably assisted by intranet & search guru Martin White of our partners Intranet Focus. 

If you’re interested in this kind of review please do let us know – with over 15 years experience in search we’re confident we can tell you exactly how to improve your search function.

The post Recipe for a strategic search review appeared first on Flax.

Kafka has been described as ‘TiVo for data’ – you can put pretty much any streaming data into Kafka, store it in a distributed and resilient way and then play it out again from any point. It’s highly scalable and integrates well with other Big Data tools such as Apache Hadoop. We’ve used Kafka and its sister project Apache Samza to develop prototype high-performance media monitoring systems and we’re also using it along with Elasticsearch, Logstash and Kibana (the ELK stack) to develop log monitoring and analysis systems. We’re hearing about many other potential uses of Kafka in the Big Data and Internet of Things ecosystems.

Our partnership with Confluent will allow us to work more closely together to provide a foundation for delivering better solutions faster for our customers based on Kafka and Confluent Platform, a complete and fully supported streaming data system based on Kafka and Hadoop.

“Kafka is creating a new paradigm for organizations and allowing businesses across industries to make informed, timely decisions from their data in real time” said Jabari Norton, VP Business Development at Confluent. “We are excited to include Flax among the ranks of a growing landscape of diverse partners and systems integrators committed to unlocking the potential of streaming data for their customers.”

We’ll be talking at the London Kafka meetup on April 13th if you’d like to find out more or discuss a potential Kafka project – if you can’t make it do get in touch.

The post Flax announces partnership with Apache Kafka creators Confluent appeared first on Flax.

The week after I spoke at the Elasticsearch London Meetup with our client Westcoast on how we helped them build a better product search. Westcoast are the UK’s largest privately owned IT supplier and needed a fast and scalable search engine they could easily tune and adjust – we helped them build administration systems allowing boosts and editable synonym lists and helped them integrate Elasticsearch with their existing frontend systems. However, integrating with legacy systems is never a straightforward task and in particular we had to develop our own custom faceting engine for price and stock information. You can find out more in the slides here.

Search Solutions, my favourite search event of the year, was the next day and I particularly enjoyed hearing about Google’s powerful voice-driven search capabilities, our partner UXLab‘s research into complex search strategies and Digirati and Synaptica‘s complimentary presentations on image search and the International Image Interoperability Framework (a standard way to retrieve images by URL). Tessa Radwan of our client NLA media access spoke about some of the challenges in measuring similar news articles (for example, slightly rewritten for each edition of a daily newspaper) as part of the development of the new version of their Clipshare system, a project we’ve carried out over the last year of so. I also spoke on Test Driven Relevance, a theme I’ll be expanding on soon: how we could improve how search engines are tested and measured (slides here).

Thanks to the organisers of all these events for all their efforts and for inviting us to talk: it’s great to be able to share our experiences building search engines and to learn from others.

The post Out and about in search & monitoring – Autumn 2015 appeared first on Flax.

• Building an open source search team is hard – let us help with training & mentoring on Solr and Elasticsearch
• RS Components: Flax & Quepid help us to make “crucial” data driven decisions for tuning search
• 40x faster indexing with Elasticsearch for Hadoop – over a gigabyte per second!

The post Flax Newsletter November 2015 appeared first on Flax.

I’ve known the guys at OpenSource Connections (OSC) for several years now – working out of Charlottesville, Virginia, like Flax they provide expertise in search and related technologies. Last year they’d shown me an early version of Quepid, a browser-based tool for recording relevance judgements. This tool seemed like the perfect fit and we began to work with OSC to add various enterprise features for the aforementioned client. Along the way, Quepid gained compatibility with both Elasticsearch and Solr and many user interface improvements and is now in daily use at the client’s site. As it can be used by both business users (to rate searches) and developers (to adjust search configuration and to instantly see the effect on those ratings, across the board) it helps to develop a fast feedback loop for improving relevance.

I’m very glad to annnounce that we’re now announcing a full partnership with OSC and will be offering Quepid to all our clients (let me know if you want a demo!). We’ll also be talking about test-driven relevance tuning over the next few months – I’m particularly looking forward to the publication of this book co-written by Quepid developer Doug Turnbull.

If you’re not measuring how good your search is performing, you simply have no idea if your search engine is correctly configured. Too often, changes to search are driven by the HiPPO, by reacting to customer feedback without considering the effects of this across the whole system, or simply by dropping in a new technology and assuming this will fix everything. We can change this, by introducing test-driven relevance tuning.

The post Quepid & Flax – if you’re not testing your search, you’re doing it wrong! appeared first on Flax.

The EBI, located on the Wellcome Trust Genome Campus near Cambridge, maintains the world’s most comprehensive range of freely available and up-to-date molecular databases and is already using Apache Lucene/Solr extensively, for example in the Protein Databank in Europe which indexes over 100,000 items derived from experimental research – but this is just one of the many complex collections they provide. The BioSolr project will run for a full year, during which members of the Flax team will work directly with the EBI team to run workshops, demonstrate and document best practises in search application design, create, improve and extend open source software and learn a lot about the specialist search requirements of bioinformatics. This is a fantastic opportunity for us to push the boundaries of what is possible with Solr and associated software, to work with some incredibly rich data and to do all of this in the open to encourage collaboration from the wider software and biology communities.

We’ll be creating various open resources (software repositories, Wikis, blogs) to support the project later this year – do let us know if you would like to be involved and we will keep you informed.

The post BioSolr – building better search for bioinformatics appeared first on Flax.

Gartner correctly explains that Hadoop isn’t just one project, but an ecosystem comprising an increasing number of open source projects (and some closed source distributions and add-ons). Once you’ve got your Big Data in a HDFS-shaped pile, there are many ways to make sense of it – and one of those is a search engine, so there’s been a lot of work recently trying to add Lucene-powered search engines such as Apache Solr and Elasticsearch into the mix. There’s also been some interesting partnerships.

I’m thus wondering whether this could signal a significant boost to the development of these search projects: there are already Lucene/Solr committers working at Hadoop-flavoured companies who have been working on distributed search and other improvements to scalability. Let’s hope some of the investment cash goes to search!

The post As Hadoop gains, does Lucene benefit? appeared first on Flax.

What we’re seeing here is the further growth of an ecosystem based around what has almost become the default choice for new and migrating search applications. Some clients will want a packaged distribution of Solr, some will be happy to download the source from Apache, some will need help getting started and some will just need help when things get complicated, or support for a running application. We’ve seen all of these requirements and more in the last year.

Next week the largest conference on open source search, Lucene Revolution is held in Dublin, and four of the Flax team are attending. Do let us know if you’d like to meet up – I don’t think there’s going to be a lack of things to talk about!

The post Solr and the changing landscape of search appeared first on Flax.

Those I’ve spoken to who have attempted to build applications on Hadoop have said that it’s very much a kit of parts rather than an integrated platform, so not that easy to get started with – which has led to the emergence of various vendors providing ‘curated’ distributions and support, much as Lucidworks does for Apache Lucene/Solr. Cloudera, Hortonworks, and MapR are just some of the best-known of these vendors. With everyone jumping on the BigData bandwagon these days some of these vendors have attracted significant interest and funding.

As you might expect full-text search is often required for these distributed systems and there have been various attempts to bring Hadoop and search closer together. Hortonworks support integration with Elasticsearch, although this currently appears to mean that you can use Hive or Pig to move data from Hadoop on or off a separate Elasticsearch cluster, rather than the search engine running on the cluster itself. Cloudera’s integration of Hadoop with Solr appears to be tighter, with Solr storing its indexes on HDFS directly (perhaps not surprising considering Lucene/Solr committer Mark Miller, who is responsible for most recent SolrCloud development, works for Cloudera). Cloudera even has its own data conditioning framework Flume (yes, it seems we need yet another data conditioning/pipelining solution!) and allows for distributed indexing. MapR have partnered with LucidWorks and integrated LucidWorks Search into their distribution. All these vendors are heavy contributors to Hadoop of course and most also contribute to Lucene/Solr or Elasticsearch.

Since Hadoop has been linked with search from the beginning one can hope that these integration efforts will continue – applications that require distributed search are becoming increasingly common and Hadoop, despite its nature as a kit of parts requiring assembly, is a good foundation to build on.

The post Finding the elephant in the room: open source search & Hadoop grow closer together appeared first on Flax.

Here in the slightly calmer waters of open source search, we’ve seen a huge rise in enquiries from often blue-chip companies, no longer needing persuasion that open source is a serious contender for even the largest search and content projects. Often these companies have considered large commercial solutions but are put off by both the price and high-pressure marketing tactics – in a world of reduced budgets you simply can’t sell magic beans for a pile of gold. We’ve also seen increased interest in related technologies such as machine learning and automatic categorisation – search really isn’t just about search any more.

At Flax we’re busier than we have ever been and we’re expected the trend to continue. We’re looking forward to running more Cambridge Search Meetups, visiting and helping organise conferences such as Enterprise Search Europe and Lucene Revolution, building our network of carefully chosen partners and of course working on exciting and cutting-edge development projects.

As the storms in our sector continue to rage overhead we’ll simply be getting on with what we do best, building effective search.

The post New Year predictions: further search storms ahead! appeared first on Flax.