Small thumb speaker nevilleli Neville Li on

Learn about Scio, a Scala API for Google Cloud Dataflow (incubated as Apache Beam). Apache Beam offers a simple, unified programming model for both batch and streaming data processing while Scio brings it much closer to the high level API many data engineers are familiar with, e.g. Spark and Scalding. Neville will cover design and implementation of the framework, including features like typesafe BigQuery macros, REPL, and serialization. There will also be a live coding demo.

Neville is a software engineer at Spotify who works mainly on data infrastructure and tools for machine learning and advanced analytics. In the past few years he has been driving the adoption of Scala and new data tools for music recommendation, including Scalding, Spark, Storm and Parquet. Before that he worked on search quality at Yahoo! and old school distributed systems like MPI.

This talk was given at the NYC Data Engineering meetup in June 2016.

Continue
Small 128247 Adam Denenberg on

Here at iHeartRadio we have made a significant investment in choosing Scala and Akka for our MicroService backend. We have also recently made an investment in moving a lot of our infrastructure over to AWS to give us a lot more freedom and flexibility into how we manage our infrastructure and deployments.


One of the really exciting technologies coming out of AWS is Lambda. Lambda allows you to listen to various “events” in AWS, such as file creation in S3, stream events from Kinesis, messages from SQS and then invoke your custom code to react to those events. Additionally the applications you deploy that react to these events, require no infrastructure and are completely auto-scaled by Amazon (on what seems like a cluster of containers). Currently Lambda supports writing these applications in Python, Node and Java.

Continue

Small 515577 Owein Reese on

Owein Reese, Senior Engineer at Mediamath, gives an overview of Autolifts, an open source dependently typed library for auto lifting and auto mapping of functions built in Scala.

Autolifts takes advantage of Scala’s advanced type system to yield a set of abstractions for working with complex objects. We’ll introduce the concept of lifting and why you might want to incorporate this pattern in your code. Then we’ll show how the library takes that concept, mixes it with dependent types and implicit extensions to automatically lift in a type safe manner. Finally, we’ll show how using these extensions simplifies code, reducing boilerplate while making code more easily understood and maintained.

Owein Reese has been a full-time Scala developer for over five years and has spoken at several Scala meetups and conferences. He has several open source projects and leads several engineering teams at Media Math.

 

 

Continue
Small 83257 Kailuo Wang on

Announcing a new iHeartRadio open source project Kanaloa. Kanaloa is a set of work dispatchers implemented using Akka actors. These dispatchers sit in front of your service and dispatch received work to them. They make your service more resilient through the following means:


  1. Auto scaling - it dynamically figures out the optimal number of concurrent requests your service can handle, and make sure that at any given time your service handles no more than that number of concurrent requests. This algorithm was also ported and contributed to Akka as Optimal Size Exploring Resizer (although with some caveats).

  2. Back pressure control - this control is Little’s law inspired. It rejects requests when estimated wait time of which exceeds a certain threshold.

  3. Circuit breaker - when error rate from your service goes above a certain threshold, kanaloa dispatcher stops all requests for a short period of time to give your service a chance to “cool down”.

  4. Real-time monitoring - a built-in statsD reporter allows you to monitor a set of critical metrics (throughput, failure rate, queue length, expected wait time, service process time, number of concurrent requests, etc) in real time. It also provides real-time insights into how kanaloa dispatchers are working. An example on Grafana:Dashboard


Continue
Unknown author on

In this talk Adam Gibson from skymind.io presents the ND4J framework with an iScala notebook.
Combined with Spark's dataframes, this is making real data science viable in Scala. ND4J is "Numpy for Java." It works with multiple architectures (or backends) that allow for run-time-neutral scientific computing as well as chip-specific optimizations -- all while writing the same code. Algorithm developers and scientific engineers can write code for a Spark, Hadoop, or Flink cluster while keeping underlying computations that are platform-agnostic. A modern runtime for the JVM with the capability to work with GPUs lets engineers leverage the best parts of the production ecosystem without having to pick which scientific library to use.

This video was recorded at the SF Bay Area Machine Learning meetup.

Continue
Small thumb speaker nevilleli Neville Li on

There has been a lot of debate over Scala lately, including criticisms like this, this, this, and defenses like this and this. Most of the criticisms seem to focus on the language's complexity, performance, and integration with existing tools and libraries, while some praise its elegant syntax, powerful type system, and good fit for domain-specific languages.

However most of the discussions seem based on experiences building production backend or web systems where there are a lot of other options already. There are mature, battle tested options like Java, Erlang or even PHP, and there are Go, node.js, or Python for those who are more adventurous or prefer agility over performance.

Here I want to argue that there's a best tool for every job, and Scala shines for data processing and machine learning, for the following reasons:

-good balance between productivity and performance
-integration with big data ecosystem
-functional paradigm


Interested in learning more about data engineering and data science? Don't miss our 2 day DataEngConf with top engineers in San Francisco, April 2016.


Continue
Small adam warski Adam Warski on

Spray, once a stand-alone project, now part of Akka, is a toolkit for building and consuming REST services. SoftwareMill CTO and Co-founder Adam Warski demos how to build a simple REST service with Spray, and then consume it with a Spray-based client. He shows that new routes can be added very quickly, how to use type-safe query and path parameters, as well as how to create custom directives, reusing existing code.

01:02:48

This talk was given at the Scala Bay meetup hosted at SumoLogic in SF.

Continue
Small 0a1aeb3 Chris Sachs on

Type members and path-dependent types breathe new life into objects. This talk focusses on type abstraction, and how to wield it effectively to create simple, robust, performant designs. Chris Sachs (Senior Software Engineer, Sensity Systems) designed the experimental Scala Basis library, which implements simple collection interfaces with macro optimized extensions providing all auxiliary functions, efficient container implementations, and more.

01:05:22

This talk was given at the Scala Bay meetup hosted by Box. Be sure to check out the Scala Bay meetup page for video of  talks you might have missed.

Continue
Small c25d674f69bb9c155f265a7482712be4 Paul Phillips on

In this talk, Paul Phillips, co-founder of Typesafe, will talk about collections for Scala. Based on Paul's extensive experience with scala collections, he decided to write his own. According to Paul, "The focus is much tighter: immutable, performant, predictable, correct." His talk will "will alternate between why the scala collections manage none of those things, and how I hope to do better." This talk was recorded at the Scala Bay meetup at LinkedIn.

01:28:59

Continue
Small twitter Bill Venners on

In this talk, Bill Venners talks about  ScalaTest 2.0, and how it is a vast enough forest that it is hard to see all the useful trees. Bill will guide you through ScalaTest 2.0 by pointing out lesser-known features of the testing toolkit that can help you get stuff done. You'll gain insight into what ScalaTest offers, why ScalaTest is designed the way it is, and how you can get more out of it. This talk was recorded at the Scala Bay meetup at Box.

01:36:08

Continue

Join Us