Details of search events in 2012 are beginning to appear already, here’s a few to start with:
- 1-5 April 2012 – European Conference on Information Retrieval (ECIR) in Barcelona, Spain. An academic conference featuring new developments in IR.
- 7-10 May 2012 – Lucid Imagination’s Lucene Revolution in Boston, USA. The largest conference on open source search – this event has a great buzz as the Lucene/Solr community continues to grow.
- 30/31 May 2012 – Enterprise Search Europe in London, after a successful first event last year. Great for those planning or working on enterprise search projects.
More to come as we hear about them – we’ll also be running another Cambridge Search Meetup soon.
On the twelfth day of (Search) Christmas my inbox brought to me:
Twelve users searching,
Eleven pages found,
Ten facets shown,
Nine Search Meetups,
Eight entity extractors,
Seven SOLR servers,
Six Xapian patches,
Five Open Source,
Four cloud apps,
Three Lucid partners,
Two big acquisitions,
And a Mike Lynch on board at HP.
Have a great Christmas and New Year from everyone at Flax.
I spent yesterday at the British Computer Society Information Retrieval Specialist Group’s annual Search Solutions conference, which brings together theoreticians and practitioners to discuss the latest advances in search.
The day started with a talk by John Tait on the challenges of patent search where different units are concerned – where for example a search for a plastic with a melting point of 200°C wouldn’t find a patent that uses °F or Kelvin. John presented a solution from max.recall, a plugin for Apache Solr that promises to solve this issue. We then heard from Lewis Crawford of the UK Web Archive on their very large index of 240m archived webpages – some great features were shown including a postcode-based browser. The system is based on Apache Solr and they are also using ‘big data’ projects such as Apache Hadoop – which by the sound of it they’re going to need as they’re expecting to be indexing a lot more websites in the future, up to 4 or 5 million. The third talk in this segment came from Toby Mostyn of Polecat on their MeaningMine social media monitoring system, again built on Solr (a theme was beginning to emerge!). MeaningMine implements an iterative query method, using a form of relevance feedback to help users contribute more useful query information.
Before lunch we heard from Ricardo Baeza-Yates of Yahoo! on moving beyond the ‘ten blue links’ model of web search, with some fascinating ideas around how we should consider a Web of objects rather than web pages. Gabriella Kazai of Microsoft Research followed, talking about how best to gather high-quality relevance judgements for testing search algorithms, using crowdsourcing systems such as Amazon’s Mechanical Turk. Some good insights here as to how a high-quality task description can attract high-quality workers.
After lunch we heard from Marianne Sweeney with a refreshingly candid treatment of how best to tune enterprise search products that very rarely live up to expectations – I liked one of her main points that “the product is never what was used in the demo”. Matt Taylor from Funnelback followed with a brief overview of his company’s technology and some case studies.
The last section of the day featured Iain Fletcher of Search Technologies on the value of metadata and on their interesting new pipeline framework, Aspire. (As an aside, Iain has also joined the Pipelines meetup group I set up recently). Next up was Jared McGinnis of the Press Association on their work on Semantic News – it was good to see an openly available news ontology as a result. Ian Kegel of British Telecom came next with a talk about TV program recommendation systems, and we finished with Kristian Norling’s talk on a healthcare information system that he worked on before joining Findwise. We ended with a brief Fishbowl discussion which asked amongst other things what the main themes of the day had been – my own contribution being “everyone’s using Solr!”.
It’s rare to find quite so many search experts in one room, and the quality of discussions outside the talks was as high as the quality of the talks themselves – congratulations are due to the organisers for putting together such an interesting programme.
The theme of Big Data continued at the next conference I attended, the first Enterprise Search Europe held in London. There was a good mix of presentations ranging from the academic to the practical, my favourite probably being Martin Belam and colleague’s talk about using Solr to dynamically generate content for the new Guardian Books site. I was lucky enough to be able to talk about the real business benefits of open source search along with one of our customers, Stephen Wicks, CTO of Gorkana Group, which drew some interesting questions. We also ran a combined Meetup on the Monday evening, combining Enterprise Search Cambridge with Enterprise Search London.
There did seem to be a rather negative spin on search from many presenters – saying that search technology is misunderstood, more costly than expected, rarely works and hasn’t seen much recent innovation. Some of this is true – but I see this as an opportunity rather than a problem. There is more focus on the world of search now than before due to some high-profile acquisitions; people are questioning the value and capability of search technology. Those of us working at the cutting edge, delivering real working solutions, should perhaps take this opportunity to say that yes, it can be done, at a sensible cost, and it can deliver real business benefit. Perhaps as we move further into the world of Big Data we’ll realise the true value of effective search.
It’s been an interesting and busy few weeks this autumn – starting with Lucene Eurocon in Barcelona. ‘Big Data’ was a main theme, with some great presentations including the keynote from Grant Ingersoll and the talk from Eric Baldeschwieler of Hortonworks, showing how Lucene fits with other Apache projects such as Hadoop, Mahout and HBase. I also enjoyed the presentations from Andrzej Bialecki on a portable index format for Lucene, Jan Høydahl of Cominvent AS on the Solr Update Chain and James Alexander of the Open University on building a Solr-powered search of their video archives. Luckily this year the presentations were videoed – so I can catch up on the presentations I missed – you’ll also be able to see me talk about our recent work with Reed Specialist Recruitment.
Of course, one of the major reasons for attending an event like this is the networking and talks outside the main event, and it was great to catch up with others in the field – one meeting between a number of us with an interest in pipelining and data conditioning led to the creation of an informal group to discuss how we might better share ideas, code and best practises.
While we were at the conference the announcement that search vendor Endeca had been bought by Oracle - and yes, this is also probably about Big Data. These are fascinating times – is search becoming the enabling technology for a revolution in how we deal with digital information?
We’re pleased to announce our work with Reed Specialist Recruitment, one of the UK’s largest recruitment companies, where we helped them implement an Apache Solr powered application to allow their 3000+ staff to search for and match candidates to jobs. We built an innovative indexing framework, a configuration tool and performance monitoring system for Reed and the system launched on time and under budget, a great testament to the flexibility and power of this open source software. The new system responds in under a second – a massive improvement on the previous response time of several minutes. You can read the press release here.
If you’d like to hear more I’ll be giving a presentation on the project at Lucene Eurocon in Barcelona tomorrow – Wednesday 19th October at 1.30 p.m. – slides and a video will be online after the event.
If you can’t make it to Barcelona I’ll also be talking in London, on the business benefits of open source search, at around 10am on Tuesday 25th October with our client Stephen Wicks, CTO of Gorkana Group as part of Enterprise Search Europe – there are still tickets available and you can even get a 20% discount if you join the Cambridge or London Enterprise Search Meetups, who are hosting a joint event on the Monday evening of the conference.
The diary is filling up quickly already after the summer break (which turned out not to be much of a break at all, what with the HP/Autonomy news and everything). Here’s where you can hear us speak over the next few months:
Hope to meet some of you at these exciting events (do get in touch if you’d like to arrange something more formal). There’s certainly a lot to talk about!
Today marks 10 years since we formed Flax (originally as Lemur Consulting Ltd.). We had an idea that search based on open source software was going to be increasingly important (indeed, our original business model was consultancy based on Xapian) and I think we’ve been proved right over the decade. Today, in the depths of a recession, we’re seeing significant growth in the business and some fascinating opportunities: the sector is still going through rapid change and it will be very interesting to see what the next few years bring.
Thanks to all of those who have worked with us and for us over the last decade – we look forward to the next ten years in this exciting field!
We’ve been helping to organise a new conference to be held in London this October, Enterprise Search Europe. This two-day event promises to give a ‘European perspective on the technology, selection, implementation and optimisation of enterprise-scale search’ and features speakers from 3i plc, Logica, The Guardian and a number of search providers such as Findwise, Funnelback and ourselves (I’ll be talking on ‘Building a Strong Business Foundation with Open Source Search’ on the second day).
It’s going to be a busy time as I’m also chairing a panel on the first day and helping run the evening reception, which is co-hosted by the London and Cambridge Search Meetups – this is likely to be one of the largest Search Meetups ever and is sure to be a fascinating evening, featuring speakers from the conference in an informal setting (i.e., a pub!).
Hope to see some of you there.
The Cambridge Enterprise Search Meetup last night featured Francis Rowland of the European Bioinformatics Institute and Rob Stacey of TrueKnowledge, in a newly refurbished venue. Thanks to all those who came and it was good to meet some new faces.
Francis talked about how search user interfaces should try not to restrict the user’s ‘flow’ of activity, as search is after all only a means to and end. Among the wealth of material he mentioned was the Endeca User Interface Design Pattern Library and what is sure to be a very useful upcoming book, Search Analytics for Your Site.
Rob told us about how TrueKnowledge provides a semantic question answering system – trying to understand the goal(s) of someone asking the system a question such as “is Madonna single?”. He also mentioned how this kind of technology might be applied to an enterprise environment, for example to answer questions like “has the invoice for last Thursday’s job been paid?”. Rob’s talk sparked off a very active Q&A session, with the audience raising issues such as how TrueKnowledge’s method might be applied to languages other than English and how to model the trustworthiness of their sources, which include Wikipedia.
Francis’ slides are now online – with some great sketchnotes of Rob’s talk as well! Thanks to both our speakers.
