Join us for the Applied Data Engineering Meet Up #6!
The Applied Data Engineering Meet Up is back with its first meetup of the year on April 18th! This time we have the CTO from Humio joining us to discuss data processing and SecOps. As always the venue and beer/pizza is kindly provided by OpenCredo!
6.30pm – Arrive; Beers, Pizza and Socialising
7pm – Data Processing and SecOps at Scale (Case Study) by Kresten Thorup, CTO @ Humio
Humio is a log aggregation and data processing system designed for sec-ops and dev-ops users. Fundamentally designed as a streaming timeseries-text data engine, it is perfectly suited for high-volume log processing.
In this talk, we will walk through how one of our customers use Humio as a central component in their security and incident response infrastructure, doing live-processing of live logs from some 30.000 desktop PCs to identify malware and bad actors in a changing environment.
This case study will explore the impact of processing data where the vast majority of the stored data is never actually directly retrieved, instead operating with most of the data being processed in-flight on arrival. This affords a data processing architecture that combines stream processing of live data and aggressive compression and time-only indexing of stored data. We explore how this trade off provides the ability to vastly outperform indexing-heavy solutions in both cpu and disk capacity load.