Designing Data Architecture for AI Systems

Here is a fact that should reshape how your organization invests in AI: the model is a rental, not an asset.

The frontier model you are building on today will be a baseline within a year, and its successor is already in training. You rent it by the token, and so does your competitor — the same model, available to everyone, improving on someone else's schedule. Betting your AI strategy on a specific model is building on sand. What you cannot rent, what actually determines whether any model delivers value, and what compounds over time, is the data architecture underneath it: the governed, high-quality, well-lineaged, appropriately-fresh, cost-aware foundation that any model — this year's or next year's — can run on.

That reframing is the spine of this book, and it explains a pattern anyone who has shipped AI in an enterprise has seen. The demo is dazzling. The model answers questions, retrieves documents, scores well on the test set. Then it goes to production and falls apart — and the honest postmortem never blames the model. It blames the data: six systems with three definitions of "customer," an index nobody kept fresh, a vector store that died at scale, a governance review that happened after launch. AI, at the scale where it either creates value or embarrasses everyone, is a data architecture problem wearing a machine-learning costume. This is the guide to the body underneath the costume.

A Design-Level Guide, Not a Product Tour

Most writing about AI data is either vendor marketing dressed as thought leadership or a tutorial for a tool that will be obsolete before the ink dries. This book is deliberately neither. It is vendor-neutral by design: lakes, warehouses, lakehouses, vector stores, and feature stores are treated as architectural capabilities with properties, guarantees, tradeoffs, and exit costs you can evaluate — not brands to buy — because the products will change and the architecture has to outlive them. The specific tools you use in five years may not exist yet; the design reasoning here is meant to still apply when they arrive.

It is also organized around what AI workloads specifically demand, rather than being a general data-engineering book with an AI chapter bolted on. A central, hard-won insight runs through it: training pipelines, inference pipelines, and RAG pipelines are three different things with three different sets of requirements — freshness, latency, lineage, reproducibility — and treating them as one undifferentiated "data pipeline" is the single most common source of production incidents. The book takes that distinction apart carefully, then builds outward: data quality as the number-one cause of AI failure and the structural remedies for it; embeddings and vector infrastructure past the demo cliff, where a hundred million vectors behave nothing like ten thousand; the honest real-time-versus-batch decision the industry keeps getting wrong; governance, lineage, and privacy written from inside a regulated financial institution; the data side of fine-tuning and evaluation; the MLOps and LLMOps touchpoints an architect must design for; and cost, treated as the first-class dimension it is, because the second postmortem after quality is always the bill. It ends with the payoff architects buy books for: complete, walked-through reference architectures and honest migration roadmaps.

What's Inside

• Why AI breaks the data architecture you already have, and the five recurring failure modes behind dead AI projects
• The workload taxonomy — training, inference, retrieval, agents — and the five dimensions that drive every design decision
• The modern data stack read vendor-neutrally: lakes, warehouses, lakehouses, and open table formats as capabilities
• The three pipelines, one chapter each, and why conflating them causes incidents
• Vector infrastructure at scale: recall/latency/cost tradeoffs, filtering, multi-tenancy, and budgeting for re-embedding
• Governance, lineage, and privacy that survives an examination — from an author who answers to examiners
• Cost architecture with worked numbers, and the economics of freshness
• Reference architectures for a RAG platform, agent memory, and the analytics-plus-AI hybrid estate
• Migration roadmaps: foundations first, platforms second, AI workloads third — and why reversing the order fails

Why I Wrote This

I architect data-intensive integration platforms inside a heavily regulated financial institution, where lineage, classification, retention, and the questions examiners ask are not research topics — they are my Tuesday. I have watched enough AI initiatives succeed and fail from the architect's seat to be certain of one thing: the organizations that win are not the ones with the best model. They are the ones whose data foundation lets any decent model work, and keeps working when the model changes.

I also sit unusually squarely on both sides of this book's intersection. My series flagship, Designing Solutions Architecture for Enterprise Integration, established the enterprise-architecture side; my hands-on AI titles — on training LLMs, fine-tuning, and building agents — established command of the workloads this architecture has to serve. A data-architecture-for-AI book needs exactly that dual fluency, and most authors have only one half: the data-engineering books predate the AI workload shift, and the ML books assume the data platform already exists. I wrote the book for the person actually responsible for building that platform — the architect who has to defend their choices in front of engineers, executives, and auditors — because that book did not exist and the postmortems kept piling up without it.

Frequently Asked Questions

Is this a hands-on coding book or a design book?

It's a design and architecture book. It includes original illustrative diagrams and small pseudocode/config snippets to make concepts concrete, but it's about design decisions, patterns, and tradeoffs — not a tutorial for any specific tool or framework. For the hands-on model work, it points to the complementary practitioner titles.

Is it tied to a particular vendor or cloud?

No — deliberately. It treats platforms as capabilities to evaluate (on guarantees, operational burden, and exit costs) rather than products to buy, so it stays useful as the vendor landscape churns. You'll be better at choosing and defending tools after reading it, without it choosing them for you.

Do I need to be an AI/ML expert?

No. You need to be technically literate — an architect, platform engineer, data engineer, or technical leader. AI-specific concepts (embeddings, RAG, feature skew, fine-tuning data) are explained clearly. If you can read an architecture diagram, you can read this book.

How is it different from a data-engineering book like Designing Data-Intensive Applications?

Those books give you the physics of data systems and predate the AI-workload era; this one applies that thinking to the specific demands of training, inference, retrieval, and agents — vector infrastructure, the three pipelines, AI governance, and the cost of freshness. It's the AI-workload-specific layer that the classics don't cover.

Does it help with regulated industries?

Especially. The governance, lineage, and privacy chapter is written from inside a regulated financial institution — how to prove what a model was trained on, keep sensitive data out of an index, handle a deletion request against a trained model, and design so the architecture produces its own audit evidence.

If You Liked This, You Might Like

• Designing Solutions Architecture for Enterprise Integration — The series flagship; the integration foundation this data book builds alongside.
• The Fundamentals of Training an LLM — The hands-on complement to this book's data-supply-chain view of training.
• Build Your Own AI Agent From Scratch — The agent-building companion to this book's agent-memory reference architecture.
• The CIO's Guide to MCP — Connecting AI to enterprise data and systems, from the leadership seat.

This volume completes the three-book "Designing X" architecture set alongside Designing Solutions Architecture for Enterprise Integration and Designing Event-Driven Architecture — integration, events, and data.

The models will keep changing. Build the foundation that makes each new one immediately useful, and you build the one asset in your AI stack that actually appreciates.

Third volume in the "Designing X: A Comprehensive Guide" series.