When even the commercial vendors are using it, has open source search won?

There have been some interesting announcements recently which may point to an increasing realisation amongst commercial search firms that an open source model is an essential advantage in today's search market. Coveo have announced that their enterprise search engine can run on an Elasticsearch core, an interesting move for a previously decidedly closed source company. BA Insight, who have previou...Continue reading

No, Elastic X-Pack is not going to be open source – according to Elastic themselves

Elastic are the company founded by the creator of Elasticsearch, Shay Banon. At this time of year they have their annual Elasticon conference in San Francisco and as you might expect a lot of announcements are made during the week of the conference. The major ones to appear this time are that Swiftype, which Elastic acquired last year, has reappeared as Elastic Site Search and that Elastic are Continue reading

Inspiring students to work in Open Source Search

I've recently been asked to join the Industrial Advisory Board for the School of Computer Science and Electronic Engineering at the University of Essex and will be talking to students there on Monday 5th February, repeating a similar talk I did last year. The subject is 'Working in Open Source Search' and I'll describe how we founded Flax back in 2001, how we've built, tuned and implemented open source search engines and some of the client projects we've worked on. It's been a fascinating journe...Continue reading

Search Solutions 2017 review

Search Solutions is one of my favourite search events of the year - small, focused and varied, with presentations from both the largest and smallest players in the world of search, drawn from both industry and academia. This year's event started with Edgar Meij of Bloomberg, who Flax have helped in the past with their larg...Continue reading

Elastic acquires Swiftype and broadens its offering to include enterprise search

The news today that Elastic (the company behind the open source Elasticsearch software) has acquired Swiftype will have surprised a few people, even though Elastic has already acquired a good number of other companies. Swiftype have a couple of products that deliver cloud-based site and enterprise search and under the hood, all of this is built on Elasticsearch.  Swiftype are part of a new ...Continue reading

Elastic London Meetup: Rightmove & Signal Media and a new free security plugin for Elasticsearch

I finally made it to a London Elastic Meetup again after missing a few of the recent events: this time Rightmove were the hosts and the first speakers. They described how they had used Elasticsearch Percolator to run 3.5 million stored searches on new property listings as part of an overall migration from the Exalead search engine and Oracle database to a new stack bas...Continue reading

Better performance with the Logstash DNS filter

We've been working on a project for a customer which uses Logstash to read messages from Kafka and write them to Elasticsearch. It also parses the messages into fields, and depending on the content type does DNS lookups (both forward and reverse.) While performance testing I noticed that adding caching to the Logstash DNS filter actually reduced performance, contrary to expectations. With four filter worker threads, and the following configuration:

dns { 
  resolve => [ ...Continue reading

Elasticsearch, Kibana and duplicate keys in JSON

JSON has been the lingua franca of data exchange for many years. It's human-readable, lightweight and widely supported. However, the JSON spec does not define what parsers should do when they encounter a duplicate key in an object, e.g.:

  "foo": "spam",
  "foo": "eggs",
Implementations are free to interpret this how they like. When different systems have different interpretations this can cause problems. We recently encounter...Continue reading