Groq API Cookbook

The JigsawStack Prompt Engine is an AI feature designed to integrate into backend infrastructures. It automates tasks like web scraping, OCR, and translation using custom fine-tuned models. Its primary capability is automatically selecting the optimal Large Language Model (LLM) for a given prompt to deliver high-speed results powered by Groq.

Key features include:

• Prompt caching: Speeds up repeated prompt runs.
• Automatic prompt optimization: Improves model performance.
• Response schema validation: Ensures accuracy and consistency of outputs.
• Reusable prompts: Streamlines workflows.
• Multi-agent LLM support: Access to over 50 models for application flexibility.
• Managed Infrastructure: No need to manage virtual rate limits, tokens, or GPUs.

LlamaChat is an application that integrates the LlamaIndex framework with Meta's Llama3 model via the Groq API. It supports three primary interaction modes:

• Simple Chat: Standard request-response interaction.
• Streaming Chat: Real-time response streaming.
• Customizable Chat: Uses system prompts to define the chatbot's persona or behavior.

To run the application locally, use the command python main.py.

Phoenix is an open-source AI observability platform designed to help you trace, evaluate, and experiment with Groq-powered applications. It provides four core capabilities:

• Tracing: Monitor your Groq application's runtime using OpenTelemetry-based instrumentation.
• Evaluation: Use Groq to benchmark application performance through response and retrieval evaluations.
• Datasets: Manage versioned datasets of examples for use in experimentation, evaluation, and fine-tuning.
• Experiments: Track and compare changes made to prompts, LLMs, and retrieval mechanisms.

Phoenix can be deployed in various environments, including Jupyter notebooks, local machines, containerized environments, or in the cloud.

OpenLIT is an open-source, OpenTelemetry-native platform designed to monitor the performance of LLMs, AI agents, and GPUs. It allows you to track critical parameters such as cost, latency, interactions, and task sequences using a single line of code.

Key features include:

• Analytics Dashboard: Visualizes application health, costs, and user interactions.
• OpenTelemetry-native SDKs: Vendor-neutral SDKs for sending traces and metrics to existing observability tools.
• Cost Tracking: Supports custom and fine-tuned models via custom pricing files.
• Exceptions Monitoring: Dedicated dashboard for tracking and resolving errors.
• Prompt Management: Versioning and management via Prompt Hub.
• Secrets Management: Centralized handling of API keys and secrets.

This application demonstrates how to use Groq's function calling capabilities to bridge natural language questions with pre-verified SQL queries. Instead of allowing the LLM to generate arbitrary SQL (which can be risky or incorrect), the application uses function calling to semantically match a user's question to a specific, pre-verified SQL query stored in YAML files. The selected query is then executed against a DuckDB database.

Core Workflow:

1. Semantic Mapping: The LLM receives a list of available functions (representing verified queries) and their descriptions.
2. Function Selection: The LLM selects the most relevant function based on the user's intent.
3. Execution: The application executes the corresponding SQL query against the DuckDB instance and returns the results.

python main.py

The Groq Chat Streamlit App provides a high-performance chat interface with the following capabilities:

• Model Selection: Users can switch between different models, such as mixtral-8x7b-32768, llama2-70b-4096, and gemma-7b-it.
• Chat History: Maintains session-based history to support continuous conversation flow.
• Dynamic Response Generation: Uses generator functions to stream responses from the Groq API for a seamless experience.
• Error Handling: Includes error handling for API call failures.

In a multi-agent LangGraph system, define a TypedDict to represent the state passed between nodes. A common pattern is to track a sequence of messages and the current sender.

from typing import Annotated, Sequence, TypedDict
import operator
from langchain_core.messages import BaseMessage

class AgentState(TypedDict):
    # Annotated with operator.add allows new messages to be appended to the list
    messages: Annotated[Sequence[BaseMessage], operator.add]
    sender: str

The Browserbase MCP server provides two distinct modes of operation:

1. Single Session (Sequential): Uses traditional session management tools (browserbase_session_create, browserbase_session_close) to perform tasks one after another in a single browser context.
2. Multi-Session (Parallel): Uses advanced tools (multi_browserbase_stagehand_session_create, etc.) to manage multiple independent sessions simultaneously. This allows for parallel workflows such as:
   • Parallel Data Collection: Scraping multiple sites at once.
   • A/B Testing: Comparing user flows across different sessions.
   • Cross-Site Operations: Coordinating actions across different websites.
   • Backup Sessions: Maintaining fallback sessions.

Portkey uses a JSON configuration object to define how requests are handled. You can pass this config object during client initialization to enable features like retries, fallbacks, caching, and routing.

portkey = Portkey(
    api_key="YOUR_PORTKEY_API_KEY",
    virtual_key="YOUR_GROQ_VIRTUAL_KEY",
    config=your_config_object,
    model="llama-3.1-8b-instant"
)

The base llm object in Langroid does not maintain conversation state; you must manually provide the message history for each call. To automatically maintain conversation history, use the ChatAgent abstraction with a ChatAgentConfig.

import langroid as lr
import langroid.language_models as lm

llm_config = lm.OpenAIGPTConfig(
    chat_model="groq/llama3-70b-8192",
    chat_context_length=8192,
)

agent_config = lr.ChatAgentConfig(
    llm=llm_config,
    system_message="""Be nice but concise""",
)

agent = lr.ChatAgent(agent_config)
response = agent.llm_response("Capital of France?")
# The agent maintains history, so follow-up questions work:
response = agent.llm_response("What about Congo?")

The E2B MCP Gateway acts as a unified interface for multiple MCP servers, hosting them all in one gateway so an AI agent can use various tools (search, browser automation, etc.) through a single endpoint.

When paired with Groq, the system benefits from high-speed inference (up to 500+ tokens/second), which allows for near-instant tool orchestration and response generation during multi-step workflows like research or data extraction.