Join us at Kafka Summit London 2018
We are excited to announce that our CTO, Nicki Watt and Lead Consultant, Pergerto Fernandez are presenting at Kafka Summit London 2018! Kafka Summit is the premier event for data architects, engineers, devops professionals, and developers who want to learn about streaming data. It brings the Apache Kafka community together to share best practices, write code, and discuss the future of streaming technologies.
“Make Your Connections Count” – A Journey Building Kafka Connectors by Nicki Watt & Pergerto Fernandez
As Kafka keeps rising in popularity, people are increasingly adopting it as a backbone to store and process large streams of immutable events (Event Streaming as the Source of Truth). Also, we need to maintain integration between new data sources and the existing IT systems, as well as evaluate and use new technologies to analyse the captured events. We can observe this trend by the growing number of Kafka connectors. In this talk we focus on the Kafka Connect architecture and the practicalities around building a connector. Not all data sources are created equal and different scenarios require different approaches.
Having assisted customers develop their own connectors to integrate with a number of different data end points, this talk will dive under the hood and look at:
- Key patterns to handle different loads across the Kafka connect distributed framework.
- Impact of different offset management strategies on message reliability.
- Challenges and pitfalls to be aware of.
Using existing connectors as examples, as well as sample connector code, this pragmatic talk aims to deliver practical insight into this process. If you are a Kafka enthusiast looking to develop a better understanding of existing connectors, or contribute to new ones, this talk is for you.