#118 - Data Engineering (with Gal Shpantzer)

Our systems generate fantastic amounts of information, but do we have a complete understanding of how we collect, analyze, manage, store, and retrieve possibly petabytes a day? Gal Shpantzer has been doing InfoSec for over 20 years and has managed some huge data engineering projects, and offers a lot of actionable insights in this CISO Tradecraft episode.

Gal's LinkedIn Page - https://www.linkedin.com/in/riskmanagement/

Gal's Twitter Page - https://twitter.com/Shpantzer

Full Transcript - https://docs.google.com/document/d/14RXnsVttvKlRi6VL94BTrItCjOAjgGem/

Chapters

• 00:00 Introduction
• 02:00 How do you Architect Big Data Data Infrastructure
• 03:33 Are you taking a look at Ransomware?
• 06:11 Web Scale Technologies are used mostly in Marketing & Fraud Detection
• 08:11 Data Engineering - The Mindset Shift
• 10:51 The Iron Triangle of Data Engineering
• 13:55 Can I Outsource My Logging Pipeline to a Vendor
• 15:37 Kafka & Flink - Data Engineering in the Pipeline
• 18:12 Streaming Analytics & Kafka
• 22:08 How to Enable Data Science Analytics with Streaming Analytics
• 26:33 Streaming Analytics
• 30:25 Data Engineering - Is there a Security Log
• 32:30 Streaming Analytics is a Weird Thing
• 35:50 How to Get a Handle on a Big Data Pipeline
• 39:11 Data Engineering Hacks for Big Data Analytics