TensorFlow Dev Summit 2019

Right around the time Seth was composing his assessment of the state of deep learning tools available today, the third annual TensorFlow Dev Summit was held. The entire first day of talks was live-streamed, and videos are available on the TensorFlow YouTube channel. I stayed up late (from the UK) watching a few of the talks, and here’s an entirely idiosyncratic description of three things I find exciting. A broader recap is provided by the TensorFlow team themselves.

TensorFlow Federated

TensorFlow Federated is a library for federated learning. Given the title of our last report, you already know we think this is an exciting area. It was a group of Google researchers who coined the term “federated learning” with the release of the Federated Averaging algorithm for learning from decentralized data, so it makes sense that Federated Learning is getting the TensorFlow treatment.

The library includes both the high-level “Federated Learning” API designed for wrapping existing TensorFlow models, and a low-level “Federated Core” API, which is intended to the development of new federated learning algorithms. The project is nascent, but particular attention has been paid to separating the various concerns of building a federated learning system. For instance, without some thought, it would easy to start entangling model operations and network communication. By providing high level abstractions, TensorFlow Federated allows experts in each area to contribute their respective knowledge, which could substantially lower the bar to entry to participating in federated learning research. The ability to wrap an existing Keras model is particularly appealing.

TensorFlow Federated currently includes only a simulated runtime (where all federated nodes actually live on the same, local, device), so it does not out-the-box “solve” federated learning. Eventually, other runtimes will be supported with the same API, meaning that the gap between researching novel federated approaches and deploying them to real federated networks is minimised.

TensorFlow Probability

Likewise to federated learning, probabilistic programming is something we’ve written about before at Fast Forward Labs. Since our probabilistic programming report was released, the ecosystem has evolved. PyMC and Stan both continue to develop, but perhaps the most notable change has been the emergence of so called “deep probabilistic programming” libraries. These libraries combine the ability of deep neural networks to learn complex functions with the probabilistic paradigm of computing with distributions. Pyro.ai runs atop PyTorch, and until recently, Edward was the go-to library for TensorFlow.

TF Probability takes a broader approach to probabilistic programming than Edward, introducing layers of abstraction into the “probprog” stack. At its base, it has complete interoperability with TensorFlow, which means all of TF’s linear algebra operations are available. On top of this is a set of “statistical building blocks” (their language): probability distributions and bijectors (composable transformations of random variables). There are also model building tools, including probabilistic programming language Edward2 - unsurprisingly, the successor to Edward - and a “probabilistic layers” component that can be used with high level APIs like Keras. Finally, there are a suite of probabilistic inference methods including Monte Carlo and Variational Inference.

The library has been evolving rapidly over the last year, but seems to be converging to a useful suite of tools. The TF Dev Summit demo of TensorFlow Probability included a simple and compelling walk through of building a regression model then including aleatoric (observation noise) and epistemic (model) uncertainty. It is encouraging to see probabilistic programming continue to mature and gain traction with existing software ecosystems.

Swift for TensorFlow

Anyone in CFFL will tell you this particular newsletter author has a bit of a soft spot for probabilistic programming. Despite developments in TF Probability, of all the developments announced at the TF Dev Summit, I find the progress in Swift for TensorFlow (which has moved into beta) the most promising.

Swift is an open source language from Apple, first designed to provide a better development experience for iOS and OSX (but now also supporting Linux, less a few niceties like automatic test discovery). Chris Lattner, who implemented much of the Swift language while at Apple, is now working on bringing TensorFlow to Swift. However, this is not a simple wrapper around the existing TensorFlow codebase, but rather a re-imagining of what tools for machine learning can be. Swift for TensorFlow integrates automatic differentiation into the Swift language itself, allowing easy definition of performant custom operations. Swift is fast, expressive and has an advanced type system. We love Python in CFFL, and the Swift for TensorFlow team have also provided an almost native experience for interoperability with Python.

The watch words are “differentiable computing”, and we eagerly await more demo applications in this ecosystem. One advantage to Swift, being designed as a replacement for C family languages is that it is not necessary to wrap C libraries for performant numerical code. This means one can move up and down the numerical computing stack in the same language, which is also great for pedagogy. Speaking of pedagogy, Jeremy Howard of Fast.ai has noticed Swift too, and at the TF Dev Summit, it was announced that the next iteration of the Fast.ai course will include a Swift component from Chris Lattner himself.

Also a snazzy new logo.