Better performance with the Logstash DNS filter

We've been working on a project for a customer which uses Logstash to read messages from Kafka and write them to Elasticsearch. It also parses the messages into fields, and depending on the content type does DNS lookups (both forward and reverse.) While performance testing I noticed that adding caching to the Logstash DNS filter actually reduced performance, contrary to expectations. With four filter worker threads, and the following configuration:

dns { 
  resolve => [ ...Continue reading

Elasticsearch London Meetup – Exploring the Graph API & SearchKit UI components

This month's Elasticsearch Meetup was hosted by Argos at their Victoria Digital Hub with a relatively small crowd this time - I suspect quite a few who registered didn't actually turn up or release their tickets, which is a shame as there was a waiting list. Mark Harwood of Elastic was first with a talk about the new Graph API and visua...Continue reading

Elastic London User Group Meetup – scaling with Kafka and Cassandra

The Elastic London User Group Meetup this week was slightly unusual in that the talks focussed not so much on Elasticsearch but rather on how to scale the systems around it using other technologies. First up was Paul Stack with an amusing description of how he had worked on scaling the logging infrastructure for a major restaurant booking website, to cope with hundreds ...Continue reading