Here at SumAll, we provide analytics from over 40 different sources that range from social media data to health and fitness tracking. We collect and connect metrics from a wide range of 3rd party API data sources for our clients to help them make better business decisions. One of engineering's biggest challenge has been on building and maintaining a data pipeline that is both performant and reliable.
We currently have a few different implementations that are running in production. One set uses Esper at its core while another set uses an in-house aggregation library written in Java. The use of Esper makes the aggregation piece of the processing asynchronous unlike the pipeline that uses the in-house aggregation library. This has been one of the debate points in discussions amongst our engineering team over which technology to ultimately use for our aggregator services. As the number of additional platforms we integrate with continues to increase, we have found that having different approaches to solving the same problem fragments the engineering team and leads to code overhead.