Indexing and Searching Logs with Elasticsearch or Solr

In this talk, Radu Gheorghe, from SemaText, talks about using Elasticsearch or Solr to index your logs, so you can search and also analyze them in real-time. The term “logs” can range from server logs and application events to metrics or even social media information.  This talk was recorded at the NYC Search, Discovery and Analytics meetup at Pivotal Labs.  


In the first part of the talk, Radu will discuss how Logstash, Apache Flume or rsyslog can help you parse, buffer and ship your logs to Elasticsearch or Solr. He will show the pros and cons of each tool, so you can decide what’s best for your use case. Then Radu will put Elasticsearch and Solr side by side and show how to make them handle write-heavy applications like those presented in the first part. Radu will also talk about how to tweak various buffers, merge policies, how to structure indices/collections, and more. And when one machine isn’t enough, Radu looks at how Elasticsearch and Solr (through SolrCloud) can be scaled horizontally to handle more and more data. In the demo part of the presentation, Radu takes one end-to-end solution from all the possible combinations that he discusses and shows how to index, search and analyze a high flow of logs.