Understanding what a model doesn’t know is important both from the practitioner’s perspective and for the end users of many different machine learning applications. In our previous blog post we discussed the different types of uncertainty. We explained how we can use it to interpret and debug our models. In this post we’ll discuss different ways to obtain uncertainty in Deep Neural Networks. Let’s start by looking at neural networks from a Bayesian perspective. Bayesian learning 101 Bayesian statistics allow us to draw conclusions based on both evidence (data) and our prior knowledge about the world. This is often contrasted with frequentist statistics which only consider evidence. The prior knowledge captures our belief on which model generated the data, or what the weights of that model are. We can represent this belief using a prior distribution p(w) over the model’s weights. As we collect more data we update the […]
As deep neural networks (DNN) become more powerful, their complexity increases. This complexity introduces new challenges, including model interpretability. Interpretability is crucial in order to build models that are more robust and resistant to adversarial attacks. Moreover, designing a model for a new, not well researched domain is challenging and being able to interpret what the model is doing can help us in the process. The importance of model interpretation has driven researchers to develop a variety of methods over the past few years and an entire workshop was dedicated to this subject at the NIPS conference last year. These methods include: LIME: a method to explain a model’s prediction via local linear approximation Activation Maximization: a method for understanding which input patterns produce maximal model response Feature Visualizations Embedding a DNN’s layer into a low dimensional explanation space Employing methods from cognitive psychology Uncertainty estimation methods – the focus of […]
Now that more than a year has passed since our first deep learning project emerged, we have had to keep moving forward and delivering the best models we can. Doing so has involved a lot of research, trying out different models, from as simple as bag-of-words, LSTM and CNN, to the more advanced attention, MDN and multi-task learning. Even the simplest model we tried has many hyperparameters, and tuning these might be even more important than the actual architecture we ended up using – in terms of the model’s accuracy. Although there’s a lot of active research in the field of hyperparameter tuning (see 1, 2, 3), implementing this tuning process has evaded the spotlight. If you go around and ask people how they tune their models, their most likely answer will be “just write a script that does it for you”. Well, that’s easier said than done… Apparently, there […]
Do you know that feeling when you’ve finished working on a feature, pushed the code, but then your CI system refuses to respond? Lagging or slow responsiveness is very common among Jenkins users, and you can find many reported issues on it. Slow CI systems are frustrating, they make you develop slower and waste your time. I have worked on several Jenkins systems with various versions (1.x and 2.x), tens of slaves and hundreds of builds per day. I have managed to improve the performance of those systems using a few simple guidelines. In the following post I will share 5 tips that can make your Jenkins better, and put a smile on your developers’ faces. Tip 1: Minimize the amount of builds on the master node The master node is where the application is actually running, this is the brain of your Jenkins and, unlike a slave, it […]
Back in 2012, when neural networks regained popularity, people were excited about the possibility of training models without having to worry about feature engineering. Indeed, most of the earliest breakthroughs were in computer vision, in which raw pixels were used as input for networks. Soon enough it turned out that if you wanted to use textual data, clickstream data, or pretty much any data with categorical features, at some point you’d have to ask yourself – how do I represent my categorical features as vectors that my network can work with? The most popular approach is embedding layers – you add an extra layer to your network, which assigns a vector to each value of the categorical feature. During training the network learns the weights for the different layers, including those embeddings. In this post I will show examples of when this approach will fail, introduce category2vec, an alternative method […]
One pleasant morning I got to work, thinking this day couldn’t get any better. But as Murphy would have it, there was my boss walking frantically toward me. It turned out that almost over night one of the main data pipeline systems had become a major bottleneck for the company, and a solution was needed, Fast! Usually in a startup, let alone a company moving as fast as Taboola, these things can occur on a weekly basis. I needed to find some quick wins to relieve some of the bottlenecks inside the system. Luckily Re2 was there to the rescue – in this post I will share how to find the bottlenecks using Gprof2dot beautiful image rendering, and of course, what Re2 is and how to use it. * Note that this article addresses a pain I had in a Python framework, but because there are Re2 implementations to all […]
This is a tale of heroism, of overcoming obstacles and hardships. This is a tale of ingenuity, of originality and thinking outside the box. This is a tale… of how I was too lazy to go and look if someone was already playing table tennis in the game room. Hiking Across the Office is a Drag Taboola’s Israeli office in Tel Aviv, houses about 350 people, spread over five large floors. The game room, however is smack dab in the middle of them. In smaller companies, if you wanted to know if the game room was available, all you had to do was to look slightly over your monitor and you would have your answer. Here, it takes 60 seconds and 110 steps, including one flight of stairs, to get from my workstation all the way to the game room – believe me, I counted. Unfortunately, due […]
Hello Git user. In this blog post I will discuss a technique for a unique version calculation for every Git commit. You may ask why we need this, after all every commit in Git is identified by a unique sha1 hash. That’s right, let’s take 2 commits, 4bd92c9 and f5fc029, use their sha1 hash as a version and perform a simple A/B test. The test showed that 4bd92c9 is preferred to f5fc029. If this is the case, how can we tell: Which version is newer? If 4bd92c9 is included in f5fc029, or vice versa? What branch they were built from? It seems we need an alternative. The common standard for the versioning is a SemVer scheme. We will use its parts as follows: Major – manual increment Minor – every released feature will increment the minor Patch – will always be 0 Now let’s take a look at our […]
As part of the optimization team at Taboola, we are constantly working with publishers and conducting AB tests, to find the best user interface (UI) for our widget. In doing so, we improve sponsored content (SC) revenue per mille (RPM). When we dealt with a very large news site in India, as part of an ongoing optimization, we needed to get a bit creative. We were already implementing our best practices for our widget on the publisher homepage — so we came up with a different solution. We tested a native UI, one that would help blend the widget into the design of the page. User Behaviour on the Homepage Unlike article pages, where users come to read an article, and then sometimes leave after finish reading it, a user who enters a homepage directly will sometimes scan it from top to bottom to find something to read. When they […]
The existing CI processes in Taboola are quite demanding – a full product build includes 150 maven modules accounting for about 20000 unit tests. Beside running builds from master branch there are also builds for all feature branches and patch releases. All in all accumulating to more than a hundred builds per day. Some of the executed tests are pretty heavy on CPU and memory, performing resource-intensive data crunching. As you can imagine – all this requires substantial CI infra horse power. Which it definitely possesses: Taboola’s Jenkins cluster currently has 35+ Jenkins slaves, each with 20 to 40 CPU cores and at least 100 Gb memory. Each slave runs 5 to 10 executors. All in all – a powerful CI/CD factory. But even with all this power – there are limitations. On the day of the weekly release the volume of builds peaks and we sometimes find ourselves starving […]