I had been planning not to continue with these posts, but after Matt Weber pointed out the github pull requests (which to my embarrassment I’d not even noticed) he’d made to address some methodological flaws, another attempt was the least I could do.
For Solr there was a slight reduction in mean search time, from 39ms (for my original, suboptimal query structure) to 34ms and median search time from 27ms to 25ms – see figure 1. Elasticsearch, on the other hand, showed a bigger improvement – see figure 2. Mean search time went down from 39ms to 27ms and median from 36ms to 24ms.
Comparing Solr with Elasticsearch using Matt’s changes, we get figure 3. The medians are close, at 25ms vs 24ms, but Elasticsearch has a significantly lower mean, at 27ms vs 34ms. The difference is even greater at the 99th percentile, at 57ms vs 126ms.
These results seem to confirm that Elasticsearch still has the edge over Solr. However, the QPS measurement (figure 4) now gives the advantage to Solr, at nearly 80 QPS, vs 60 QPS for Elasticsearch. The latter has actually decreased since making Matt’s changes. This last result is very unexpected, so I will be trying to reproduce both figures as soon as I have the chance (as well as Matt’s suggestion of trying the new-ish filter() operator in Solr).
Our sincere thanks to Matt for his valuable input.
Figure 1: Solr search times, no indexing, original vs Matt
Figure 2: ES search times, no indexing, original vs Matt
Figure 3: ES vs Solr search times, no indexing, Matt’s changes
Figure 4: QPS, original vs Matt’s changes
