DOCSEARCHbyAlgolia
Submit Docs

Generative AI for Beginners - A Curriculum

repository·main·Indexed 11 days ago

https://github.com/microsoft/generative-ai-for-beginners

A comprehensive 21-lesson course by Microsoft Cloud Advocates teaching the fundamentals of building Generative AI applications using Python and TypeScript. The curriculum covers LLM mechanics, tokenization, prompt engineering, Retrieval Augmented Generation (RAG), fine-tuning, and responsible AI mitigation strategies. It includes guidance on selecting models by modality (text, image, audio) and using Microsoft Foundry for model lifecycle management.

Tokens
249.4K
Snippets
429
Records
1K
Agent score
97%

What's inside Generative AI for Beginners

  1. Overview of Mistral Models

    main

    This lesson explores three primary Mistral models available via Microsoft Foundry Models:

    1. Mistral Large 2: The flagship enterprise model. It features a 128k context window, high performance in math and coding, and native function calling (sequential or parallel). It is ideal for Retrieval Augmented Generation (RAG), complex code generation, and multilingual tasks.
    2. Mistral Small: A Small Language Model (SLM) designed for cost savings (approx. 80% cheaper than LLMs) and low latency. It is best suited for text-based tasks like summarization, sentiment analysis, translation, and low-latency code suggestions.
    3. Mistral NeMo: An Apache2 licensed model that serves as an upgrade to Mistral 7B. It uses the efficient Tekken tokenizer (better for code and multiple languages) and supports native function calling and fine-tuning.

  2. Introduction to Generative AI and Large Language Models

    main

    Generative AI is a subset of deep learning capable of creating text, images, and other content using natural language prompts. This lesson introduces the core concepts of Generative AI, its evolution from machine learning and neural networks, and the specific architecture of Transformers that powers modern Large Language Models (LLMs).

    Key concepts covered:

    • Generative AI: AI that can generate new content.
    • Machine Learning: Algorithms that learn patterns from data without explicit programming.
    • Neural Networks (Deep Learning): Advanced algorithms (like RNNs) that model human-like understanding.
    • Transformers: A modern architecture using an attention mechanism to process long sequences of text by weighing the importance of different parts of the input.

  3. Overview of Advanced Prompt Engineering

    main

    Advanced Prompt Engineering is the process of guiding a Large Language Model (LLM) toward more relevant responses by providing useful instructions or context. The process consists of two main stages:

    1. Prompt Creation: Providing the initial instructions and relevant context.
    2. Optimization: The iterative method of gradually improving the prompt to achieve better results.

    This lesson focuses on moving beyond basic prompting to understand why certain prompts perform better than others and how to apply specific techniques that are applicable to any LLM.

  4. Overview of Integrating with Function Calling

    main

    This lesson focuses on enhancing Generative AI applications by using Function Calling. Function calling addresses two primary limitations of standard Large Language Model (LLM) responses:

    1. Inconsistent Response Formats: Standard LLM outputs can be unstructured, requiring complex validation logic. Function calling provides a consistent response format that is easy to integrate with other systems.
    2. Lack of External Data: LLMs are limited to their training data. Function calling allows the application to access real-time or external data sources (e.g., weather APIs, databases, or specific catalogs) to enrich the chat context.

    By the end of this lesson, you will be able to set up function calls using Azure OpenAI and design effective function call patterns for your applications.

  5. Overview of Meta Llama Model Family

    main

    The Meta Llama model family includes Llama 3.1 and Llama 3.2, which are available via the Microsoft Foundry Models catalog.

    Available Variants:

    • Llama 3.1: 70B Instruct, 405B Instruct (Optimized for large context, native function calling, and RAG).
    • Llama 3.2: 11B Vision Instruct, 90B Vision Instruct (Multimodal capabilities for text and image processing).

    Note: GitHub Models is scheduled to end in late July 2026. For prototyping, use Microsoft Foundry Models.

  6. Overview of the Microsoft Phi-3 / Phi-3.5 Family

    main

    The Microsoft Phi-3 and Phi-3.5 family consists of efficient models optimized for text, vision, and Mixture of Experts (MoE) applications. These models are designed to compete with much larger models while remaining small enough for edge deployment.

    Phi-3 / 3.5 Instruct (Text & Chat)

    Focuses on text generation, chat completion, and information extraction.

    • Phi-3-mini (3.8B): High performance for its size; available via Microsoft Foundry, Hugging Face, and Ollama.
    • Phi-3-small & medium: 7B parameter models that outperform GPT-3.5 in reasoning, coding, and math.
    • Phi-3.5-mini (3.8B): An upgrade to Phi-3-mini with support for over 20 languages and improved long-context handling.

    Phi-3 / 3.5 Vision (Visual Reasoning)

    Enables models to understand visual inputs like text, images, and diagrams.

    • Phi-3-Vision (4.2B): Performs well on OCR, table understanding, and general visual reasoning.
    • Phi-3.5-Vision: An upgrade capable of handling multiple images and multi-frame inputs (video reasoning).

    Phi-3.5-MoE (Mixture of Experts)

    Uses a Mixture of Experts architecture to scale model quality without a proportional increase in compute.

    • Phi-3.5-MoE: Features 16x3.8B expert modules. It uses only 6.6B active parameters to achieve reasoning and math capabilities comparable to much larger dense models.

  7. Overview of Using Generative AI Responsibly

    main
    This lesson focuses on the necessity of prioritizing responsible AI when building Generative AI applications. It covers how to ensure outputs are fair, non-harmful, and reliable. Developers learn to identify potential risks such as hallucinations, harmful content, and lack of fairness, and how to implement practical strategies and tools to mitigate these issues.

  8. Overview of Generative AI for Power Platform

    main

    Generative AI enhances the Power Platform by allowing users to build AI-powered apps, sites, dashboards, and automated processes using natural language, without requiring data science expertise. This is achieved through two primary mechanisms:

    1. Copilot: An AI assistant available across Power Platform products (Power Apps, Power Automate, Power BI, Power Pages, and Copilot Studio). It allows you to build solutions by describing what you want in conversational steps.
    2. AI Builder: A low-code AI tool available for Power Apps and Power Automate. It enables the use of AI models to automate processes and predict results by connecting to data in Dataverse or cloud sources like SharePoint, OneDrive, or Azure.

  9. Introduction to Prompt Engineering Fundamentals

    main

    This module covers the essential concepts and techniques for creating effective prompts for generative AI models.

    Key Concepts:

    • Generative AI: Capable of creating new content (text, images, audio, code, etc.) in response to user requests using Large Language Models (LLMs).
    • Prompt: The text input sent to an LLM that serves as the primary programming interface, instructing the model on what to do and directly affecting the quality of the response.
    • Completion: The response returned by the AI based on the prompt.
    • Prompt Engineering: The practice of designing and optimizing prompts to deliver consistent, high-quality outputs at scale.
    • Tokenization: The process of converting text into smaller units called 'tokens' that the model can process.
    • Instruction-Tuned LLMs: Models specifically fine-tuned to follow instructions, improving the accuracy and relevance of their responses.

  10. Overview of the Phi-3 and Phi-3.5 Model Families

    main

    The Phi-3 and Phi-3.5 families are Small Language Models (SLMs) designed for text, vision, and Agent (Mixture of Experts) application scenarios. They are optimized for high performance relative to their parameter size and can be deployed on edge devices.

    Phi-3 / 3.5 Instruct (Text-based)

    Used for text generation, full chat capabilities, and information extraction.

    • Phi-3-mini (3.8B): High performance for its size; available via Microsoft Foundry, Hugging Face, and Ollama. Outperforms models twice its size.
    • Phi-3-small (7B): Outperforms GPT-3.5T in language, reasoning, coding, and mathematics.
    • Phi-3-medium (14B): Outperforms Gemini 1.0 Pro.
    • Phi-3.5-mini (3.8B): An upgrade to Phi-3-mini with improved multi-language support (20+ languages including English, Chinese, Arabic, etc.) and robust long-context support.

    Phi-3 / 3.5 Vision (Multimodal)

    Provides visual understanding capabilities (OCR, diagrams, tables).

    • Phi-3-Vision (4.2B): Outperforms larger models like Claude-3 Haiku and Gemini 1.0 Pro V in visual reasoning, OCR, and table/diagram understanding.
    • Phi-3.5-Vision: An upgrade supporting multi-frame input (multiple images or video), outperforming Claude-3.5 Sonnet and Gemini 1.5 Flash in OCR and chart understanding.

    Phi-3.5-MoE (Mixture of Experts)

    Uses a Mixture of Experts architecture to achieve high quality with less compute during pretraining.

    • Phi-3.5-MoE: Composed of 16x3.8B expert modules. It uses only 6.6B active parameters while achieving reasoning, language, and math capabilities comparable to much larger dense models.

  11. Overview of Taskweaver

    main

    Taskweaver is a "code-savvy" agent framework designed for data analysis and generation. Unlike standard agents that work primarily with strings, Taskweaver can manipulate Python DataFrames, create graphs, and generate charts.

    Core Concepts

    • Planner: An LLM that receives a user request and maps out the necessary tasks to complete it.
    • Plugins: A collection of tools (Python classes or code interpreters) that the Planner uses. Plugins are stored as embeddings so the LLM can retrieve the correct one.
    • Experience: A feature that allows the agent to store conversation context long-term in a YAML file, enabling the LLM to improve on specific tasks over time based on previous interactions.

  12. Overview of Exploring and Comparing Different LLMs

    main

    This lesson focuses on navigating the current Large Language Model (LLM) landscape to identify the best models for specific use cases. It covers the different types of LLMs available, how to test, iterate, and compare models within Azure, and the strategies for deploying LLMs in a production environment.

    Key Learning Objectives:

    • Selecting the appropriate model for a specific use case.
    • Understanding how to test, iterate, and improve model performance.
    • Learning how enterprises deploy LLMs.