Learn Data With Mark

The project provides several stages of chatbot implementation, ranging from simple echo bots to advanced RAG-capable assistants:

• chat_echo.py: A basic bot that echos back any messages sent to it.
• chat_single.py: Answers questions using an LLM but does not maintain conversation history.
• chat_history.py: Answers questions and maintains chat history for context.
• chat.py: The complete implementation that answers questions, remembers history, and supports answering questions based on uploaded text files.

DuckDB 1.2 supports predicate pushdown on nested struct fields. This means when querying datasets (like Parquet files or JSON files), filters applied to nested fields (e.g., json.field.subfield = 'value') can be pushed down to the storage layer to optimize performance. This is particularly effective when querying Parquet files exported from JSON data.

-- Example of filtering on nested struct fields in a Parquet file
FROM 'events.parquet'
SELECT json.pass.recipient.name, count()
WHERE
  json.pass.height.name = 'High Pass'
AND
  json.pass.outcome.name = 'Incomplete'
AND 
  json.possession_team.name = 'Japan'
AND
  json.pass.recipient.name NOT NULL
GROUP BY ALL
ORDER BY count() DESC
LIMIT 5;

To launch the Jupyter Lab environment for this project, use poetry run to ensure all dependencies are managed within the Poetry virtual environment. Use the --notebook-dir flag to specify the notebooks directory as the root for your workspace.

poetry run jupyter lab --notebook-dir notebooks

Use the Guidance library to control and constrain text generation from large language models. This implementation integrates with the llama CPP library and the Mistral 7B model. Key capabilities include:

• Using select to restrict generation to a specific array of values.
• Using gen to control generation via regular expressions.
• Creating reusable components and forcing outputs into JSON format.

This project demonstrates how to implement Retrieval Augmented Generation (RAG) using LangChain, Chroma (a vector database), and OpenAI/GPT models. The goal is to provide an LLM with additional context (e.g., data about the Wimbledon 2023 tennis tournament) to improve answer accuracy.

To use this project, you must configure your OpenAI API key and install the necessary dependencies using Poetry.

In DuckDB 1.2, you can define structs using the SET VARIABLE command with a JSON-like dictionary syntax. This allows you to store complex objects in session variables for later use.

SET VARIABLE struct1 = {'name': 'DuckDB', 'firstRelease': 2019};
SET VARIABLE struct2 = {'description': 'in-process SQL database'};

To run this transcription and analysis pipeline, you must have the following tools installed:

• insanely-fast-whisper: The transcription engine.
• Ollama: For running local LLMs.
• jq: A command-line JSON processor.

You can use the docling tool via uv to convert a Wikipedia page (or any URL) into a structured Markdown file. This is useful for preparing data for LLM analysis.

uv run --with docling --with numpy==1.26.4 \
docling \
https://en.wikipedia.org/wiki/Novak_Djokovic_career_statistics