Archive for September, 2010

Autumn events

Autumn seems to be conference season: first is the Lucene Revolution event in Boston, USA from October 7th-8th, where I’ll be on the closing panel whose subject is “Data Crossroads – At The Intersection Of Search And Open Source”.

Next is the British Computer Society’s Search Solutions 2010 in London on October 21st, where I’m giving a presentation titled “What’s the story with open source? – Searching and monitoring news media with open-source technology”.

Both events feature a wide range of other speakers from organisations such as Cisco, LinkedIn, Twitter, Google and Microsoft.

Tags: , , , ,

Posted in events

September 10th, 2010

No Comments »

Open source search engines and programming languages

So you’re writing a search-related application in your favourite language, and you’ve decided to choose an open source search engine to power it. So far, so good – but how are the two going to communicate?

Let’s look at two engines, Xapian and Lucene, and compare how this might be done. Lucene is written in Java, Xapian in C/C++ – so if you’re using those languages respectively, everything should be relatively simple – just download the source code and get on with it. However if this isn’t the case, you’re going to have to work out how to interface to the engine.

The Lucene project has been rewritten in several other languages: for C/C++ there’s Lucy (which includes Perl and Ruby bindings), for Python there’s PyLucene, and there’s even a .Net version called, not surprisingly, Lucene.NET. Some of these ‘ports’ of Lucene are ‘looser’ than others (i.e. they may not share the same API or feature set), and they may not be updated as often as Lucene itself. There are also versions in Perl, Ruby, Delphi or even Lisp (scary!) – there’s a full list available. Not all are currently active projects.

Xapian takes a different approach, with only one core project, but a sheaf of bindings to other languages. Currently these bindings cover C#, Java, Perl, PHP, Python, Ruby and Tcl – but interestingly these are auto-generated using the Simplified Wrapper and Interface Generator or SWIG. This means that every time Xapian’s API changes, the bindings can easily be updated to reflect this (it’s actually not quite that simple, but SWIG copes with the vast majority of code that would otherwise have to be manually edited). SWIG actually supports other languages as well (according to the SWIG website, “Common Lisp (CLISP, Allegro CL, CFFI, UFFI), Lua, Modula-3, OCAML, Octave and R. Also several interpreted and compiled Scheme implementations (Guile, MzScheme, Chicken)”) so in theory bindings to these could also be built relatively easily.

There’s also another way to communicate with both engines, using a search server. SOLR is the search server for Lucene, whereas for Xapian there is Flax Search Service. In this case, any language that supports Web Services (you’d be hard pressed to find a modern language that doesn’t) can communicate with the engine, simply passing data over the HTTP protocol.

Tags: , , , , , , ,

Posted in Technical

September 3rd, 2010

1 Comment »

Revolutions and interviews

This October I’ve been invited to speak at Lucene Revolution, a conference on open source search to be held in Boston, USA. I’ll be part of the closing panel on October 8th, together with speakers from Lucid Imagination and Exalead. It looks like a very interesting event, with speakers from IBM, Cisco, LinkedIn and the Smithsonian.

As part of the run-up to the conference Stephen Arnold has interviewed me – we discussed the wider picture of open source search, why a strong community is important and why flexibility can be the key to successful integration.

Tags: , ,

Posted in events

September 1st, 2010

No Comments »