All Stories
Something strange happened while I worked with Kafka. While adding a new consumer from Kafka to one of our services, the service stopped consuming from ALL other existing consumers. As part of my job at Taboola as a team leader on a production team in the Infrastructure group, we’re supposed to remove bottlenecks, not create them. This post will describe how I investigated the issue, explain what I discovered, and share my insights into the whole situation. Some background Before I get into the rest of the story, here’s some background on how we use Kafka at Taboola’s events handling pipeline and why it’s critical to our infrastructure. Taboola’s recommendations appear on tens of thousands of web pages and mobile apps every second. As users engage with the content, multiple events are fired to signal that recommendations are rendered, opened, clicked, and so on. Each event triggers one or more Kafka messages, […]
As a team member in the Scale Performance Data group of Taboola’s R&D, I had the opportunity to develop a mechanism which prioritizes the consumption of Kafka topics. I did so to tackle a challenge we had of handling messages that are being sent from hundreds of frontend servers around the world by our time-series based backend servers. In this post, I will focus on three things. Why we needed such a mechanism – the problem Code snippets showing how I put said mechanism in place – the solution Issues I faced with Kafka – bumpers on the way to the solution Prerequisites Assume you have basic knowledge in: Java multithreading multithreading in Java Kafka Data coming from all over the world needs to be divided by minutes Taboola has a few hundred frontend servers. They serve more than half a million requests per minute – and this number grows steadily. […]