←Back to Blogs

Outline of Designing Data-Intensive Applications

November 24, 2024

Data EngineeringSystem DesignDistributed SystemsBook Review

Embark on an In-Depth Journey into the Design of Data-Intensive Applications

In this series, we'll unravel the complexities of building systems that are reliable, scalable, and maintainable—core principles every engineer should master.

This book, Designing Data-Intensive Applications by Martin Kleppmann, is a treasure trove of insights, and I'll be sharing my learnings through a series of articles as I read it. Here's a sneak peek at what's coming:

Part I: Core Foundations of Data-Intensive Applications

We begin with the fundamental principles that shape the design of robust systems.

  • Chapter 1: Explore Reliability, Scalability, and Maintainability—why they matter and how to achieve them effectively.
  • Chapter 2: Compare different data models and query languages to understand when and where each shines.
  • Chapter 3: Dive into storage engines, discovering how databases structure data for efficient retrieval.
  • Chapter 4: Learn about data encoding formats and the evolution of schemas over time to manage change gracefully.

Part II: Tackling the Challenges of Distributed Systems

Scaling out to multiple machines introduces unique hurdles. This section addresses these challenges head-on.

  • Chapter 5: Replication: Ensuring data availability and durability across systems.
  • Chapter 6: Partitioning/Sharding: Breaking data into manageable chunks for scalability.
  • Chapter 7: Understanding transactions to maintain consistency in distributed environments.
  • Chapter 8: A deep dive into the quirks and challenges of distributed systems.
  • Chapter 9: Achieving consistency and consensus, the holy grail of distributed computing.

Part III: Derived Data and Integration in Heterogeneous Systems

No single database can handle all use cases, and derived datasets are key in integrating diverse systems.

  • Chapter 10: The role of batch processing in transforming and deriving data.
  • Chapter 11: Stream processing as a real-time complement to batch workflows.
  • Chapter 12: Bringing it all together—strategies for building systems that are future-proof and robust.

Stay Tuned!

This series will delve into each topic, unpacking concepts and sharing actionable insights for engineers, architects, and anyone fascinated by the underpinnings of backend systems.

References:

  • Martin Kleppmann, Designing Data-Intensive Applications: The Big Ideas Behind Reliable, Scalable, and Maintainable Systems