DOCSEARCHbyAlgolia
Submit Docs

AI Engineering (AIE) Book Resources

repository·main·Indexed 12 days ago

https://github.com/chiphuyen/aie-book

Supplementary materials and study tools for Chip Huyen's 'AI Engineering' book. This resource provides a framework for transitioning from AI demos to production-ready systems using foundation models (LLMs and LMMs), covering evaluation, prompt engineering, RAG, AI agents, fine-tuning (including PEFT and LoRA), and data management.

Tokens
12.5K
Snippets
13
Records
43
Agent score
96%

What's inside AI Engineering

  1. Overview of AI Engineering (AIE) resources

    main

    This repository serves as a central hub for resources related to the AI Engineering book. It includes supplementary materials to help developers and engineers master the end-to-end process of adapting foundation models (LLMs and LMMs) to real-world problems.

    Key resources available in this repository include:

    • Table of Contents: ToC.md
    • Chapter Summaries: chapter-summaries.md
    • Study Notes: study-notes.md
    • AI Engineering Resources: resources.md
    • Prompt Examples: prompt-examples.md
    • Case Studies: case-studies.md
    • Misalignment AI: misalignment.md
    • Appendix: appendix.md
    • Fun Tools: A ChatGPT and Claude conversation heatmap generator located at scripts/ai-heatmap.ipynb.

  2. Explore AI Engineering resources by topic

    main

    The resources.md file provides a curated list of learning materials for AI Engineering, organized by book chapters and technical domains. Key topics include:

    • ML Theory Fundamentals: Neural networks, probability, and calculus.
    • Foundation Models: Planning applications, understanding architectures, training, sampling, and context efficiency.
    • Methodology: Evaluation, Prompt Engineering (including defensive techniques), and RAG (Retrieval-Augmented Generation).
    • Advanced Engineering: Agents, Finetuning, Dataset Engineering, Inference Optimization, and AI Architecture.

    You can also find a list of 1000+ generative AI GitHub repos to track current tools and implementations.

  3. Target audience for AI Engineering

    main

    The book is a technical resource intended for roles involved in the AI development lifecycle, including:

    • Engineers: AI Engineers, ML Engineers, and Data Scientists.
    • Management: Engineering Managers and Technical Product Managers.
    • Specialized Roles: Tool developers (identifying underserved areas), Researchers (understanding use cases), and Job Candidates (clarifying AI engineering skills).

    It is specifically useful for those building or optimizing AI applications, moving from demo to production, or attempting to streamline AI development processes within an organization.

  4. What is covered in the AI Engineering book

    main

    The AI Engineering (AIE) book provides a framework for adapting foundation models (Large Language Models and Large Multimodal Models) to specific applications. It focuses on fundamentals rather than specific tools or APIs to ensure longevity of knowledge.

    Core topics addressed include:

    • Application Feasibility: Deciding whether to build an AI application.
    • Evaluation: Methods for evaluating applications and using AI to evaluate AI outputs.
    • Hallucinations: Detecting and mitigating hallucinations.
    • Prompt Engineering: Best practices for prompting.
    • RAG (Retrieval-Augmented Generation): Understanding why it works and various strategies.
    • Agents: Building and evaluating AI agents.
    • Fine-tuning: Determining when to fine-tune versus when not to.
    • Data Management: Data requirements and quality validation.
    • Optimization: Making models faster, cheaper, and more secure.
    • Feedback Loops: Creating continuous improvement cycles for applications.

  5. Inference optimization techniques

    main

    Inference optimization can be applied at two different levels: the model level and the inference service level.

    Model-level Optimization

    These techniques involve changing the model itself, which may alter its behavior:

    • Quantization: Reducing the precision of model weights (generally effective across most models).
    • Distillation: Training a smaller model to mimic a larger one.
    • Attention Mechanism Optimization: Improving the efficiency of the attention mechanism (e.g., KV cache management, writing custom attention kernels).
    • Architecture-specific optimizations: Tailoring techniques to specific model types (e.g., addressing the autoregressive decoding bottleneck).

    Inference Service-level Optimization

    These techniques keep the model intact and focus on how the model is served:

    • Batching and Parallelism Strategies: Including Tensor Parallelism (reduces latency and enables larger models) and Replica Parallelism (scales out by running multiple copies of the model).
    • Prefilling/Decoding Decoupling: Separating the two main phases of inference.
    • Prompt Caching: Crucial for workloads with long, overlapping prompt segments or multi-turn conversations.
    • KV Caching: Significantly improves performance for workloads with long contexts.

  6. Observability and User Feedback in AI Architecture

    main

    Building AI applications requires a system-wide approach to monitoring and data collection:

    Observability

    Observability involves designing metrics and alerts to detect and trace failures. While traditional software engineering practices apply, foundation models introduce unique failure modes that require specific design considerations and metrics.

    User Feedback and the Data Flywheel

    Conversational interfaces provide unique feedback loops. Engineers should design applications to effectively collect conversational feedback to drive:

    • Analytics: Understanding usage patterns.
    • Product Improvement: Refining the user experience.
    • The Data Flywheel: Using collected feedback as high-quality data to continuously improve the underlying AI models.

  7. Implement RAG and Agentic patterns

    main

    RAG (Retrieval-Augmented Generation) and Agents are prompt-based methods that influence model quality via inputs without modifying the model weights.

    RAG (Retrieval-Augmented Generation)

    RAG overcomes context window limitations by retrieving relevant information from external memory before generating a response.

    • Two-step process: 1. Retrieve relevant information $\rightarrow$ 2. Generate response.
    • Term-based retrievers: (e.g., Elasticsearch, BM25) Lighter to implement, good for baselines.
    • Embedding-based retrievers: Use vector search; more computationally intensive but can outperform term-based methods.

    AI Agents

    An agent uses a model as a planner to analyze tasks, consider solutions, and pick the best path.

    • Components: An agent is defined by its environment and the tools it can access.
    • Augmentation: A model's planning ability can be improved using reflection and a memory system to track progress.
    • Risks: Increased automation and tool use increase the risk of catastrophic failures and security vulnerabilities.

  8. Core design decisions in foundation model development

    main

    When determining which foundation models to use or how to build them, consider these key factors:

    Training Data

    Large models require massive datasets. Because model providers often use whatever data is available, models may not perform well on specific domains or low-resource languages without curated training data.

    Architecture and Scale

    • Architecture: The dominant architecture for language-based foundation models is the transformer.
    • Scale: Measured by the number of parameters, the number of training tokens, and the FLOPs required for training. Scaling laws help determine the optimal balance of parameters and tokens relative to a compute budget.

    Post-training (Alignment)

    To ensure model outputs align with user preferences and mitigate issues from self-supervision during pre-training, two post-training steps are used:

    1. Supervised Finetuning (SFT)
    2. Preference Finetuning (often involving RLHF)

    Sampling

    Sampling is the process by which a model generates output tokens. This makes models probabilistic, which is beneficial for creativity but can lead to inconsistency and hallucinations.

  9. Core criteria for dataset design

    main

    When designing datasets for training models (whether for pre-training, instruction finetuning, or preferred finetuning), you should evaluate your data based on three core criteria:

    1. Quality: High-quality data can outperform larger amounts of noisy data.
    2. Coverage: Ensuring the dataset covers the necessary behaviors and diversity required for the use case.
    3. Quantity: The total amount of data available.

    Increasing dataset diversity is a key lever for improving model performance.

  10. Inference efficiency metrics

    main

    To measure the efficiency of language model-based inference, use the following metrics:

    Latency Metrics

    • Time to First Token (TTFT): Influenced by the prefilling phase. This measures how quickly the model starts responding.
    • Time per Output Token (TPOT): Influenced by the decoding phase. This measures the speed of the generation process after the first token.

    Throughput and Utilization

    • Throughput: Directly related to the cost of inference. There is a fundamental tradeoff: reducing latency often increases cost, while reducing cost often increases latency.

  11. Relationship between AI Engineering (AIE) and Designing Machine Learning Systems (DMLS)

    main

    The AI Engineering (AIE) book is designed to be a companion to Designing Machine Learning Systems (DMLS). While they are self-contained and can be read independently, they cover different focuses within the ML lifecycle:

    FeatureDesigning Machine Learning Systems (DMLS)AI Engineering (AIE)
    Core FocusBuilding applications on top of traditional ML models.Building applications on top of foundation models (LLMs/LMMs).
    Key TechniquesTabular data annotations, feature engineering, and model training.Prompt engineering, context construction, and parameter-efficient fine-tuning.
    OverlapCovers general ML concepts relevant to foundation models.Covers general ML concepts, but focuses more on foundation-model-specific engineering.

    Real-world systems often require knowledge from both domains, as they frequently combine traditional ML models with foundation models.

  12. Approaches to evaluating foundation models

    main

    Evaluating foundation models is more challenging than traditional ML due to their open-ended nature. Evaluation strategies include:

    Language Modeling Metrics

    Used to measure how well a model predicts text. Key metrics include:

    • Perplexity
    • Cross entropy

    Open-ended Response Evaluation

    1. Exact Evaluation: Focuses on functional correctness (e.g., exact matches).
    2. Similarity Scores: Hand-designed measurements of how close a response is to a reference.
    3. AI-as-a-judge: Using a powerful AI model to evaluate the quality of an answer. This is subjective and depends heavily on the specific judge used. AI judges should be iterated upon and supplemented with exact or human evaluation.
    4. Comparative Evaluation: Ranking models by asking which of two responses is better. This is often used in conjunction with preference models (specialized AI judges that predict user preference).