Etsy - A Deep Dive into Monitoring with Skyline

Data Engineer at EtsyAbe Stanway, talks about Skyline, a real-time anomaly detection tool. The talk was recorded at eBay NYC. Abe goes over Skyline's architecture and design, taking a deep dive into the architecture and design of Skyline. This talk was recorded at the NYC Data Engineering meetup at Ebay NYC.

Skyline is a real-time anomaly detection system, built to enable passive monitoring of hundreds of thousands of metrics, without the need to configure a model/thresholds for each one, as you might do with Nagios. It is designed to be used wherever there are a large quantity of high-resolution time-series which need constant monitoring. Once a time metrics stream is set up (from StatsD or Graphite or other source), additional metrics are automatically added to Skyline for analysis. Skyline's easily extendible algorithms allow you to define what each metrics baseline should be, thereby also defining anomalous behavior. After Skyline detects an anomalous metric, it surfaces the entire time-series to the web app, where the anomaly can be viewed and acted upon.

github.com/etsy/skyline

Get updates of upcoming tech talks and presentations

If you'd like to be notified when we post new tech talks, developer presentations and opensource updates, you can subscribe to our newsletter, or YouTube channel.

Want to hear from more top engineers?

Our weekly email contains the best software development content and interviews with top CTOs. Enter your email address now to stay in the loop.