Content Core

Content Core resolves configuration using the following priority order (highest to lowest):

1. Command flags (e.g., --engine)
2. Environment variables (prefixed with CCORE_, e.g., export CCORE_URL_ENGINE=firecrawl)
3. Config file (~/.content-core/config.toml)
4. Defaults

Environment Variables: To override config file values for a specific session, use the CCORE_ prefix. For example:

export CCORE_URL_ENGINE=firecrawl
export OPENAI_API_KEY=sk-...

Content Core automatically routes input to the appropriate processor based on the input type:

• URL provided: Routes to a URL processor (or the YouTube processor specifically for YouTube links).
• File path provided: The system detects the file type using binary signatures and content analysis (no libmagic required), then routes it to the matching document or media processor.
• Text provided: Handled by the text processor.

File type detection is performed in pure Python by reading the first 512 bytes for binary signatures or 1024 bytes for text-based content analysis, making it independent of file extensions.

Content Core distinguishes between a genuinely empty source and a processing failure. It is critical to handle exceptions correctly based on this distinction:

• Total Failure (Raises Exception): Any failure related to routing, input, configuration, network, or parsing will raise a typed exception from common/exceptions.py. You should not treat an exception as an empty string.
• Degradation (Internal to Source): Degradation is only permitted within a single source. For example, if a single page in a PDF fails to parse, the processor may log the error and skip that unit, but the overall file processing should continue. If the entire file cannot be processed, it must raise an error.
• Empty Content: A return value of content="" signifies that the source was successfully processed but contained no text. It does not mean an error occurred.

In version 2.0, the clean/cleanup functionality has been removed. The library is now focused on two primary core operations:

1. extract: Detects the input type and routes it to the appropriate processor to retrieve content.
2. summarize: Processes existing content to generate a summary.

The guiding principle of the architecture is a simple pipeline: detect input type $\rightarrow$ route to the right processor $\rightarrow$ return content.

The media pipeline handles audio and video files by extracting audio streams and performing Speech-to-Text (STT).

Workflow:

1. Video: Uses ffprobe to identify audio streams, selects the highest quality stream, and then transcribes it.
2. Audio: Transcribes the file using the configured audio_provider and audio_model (defaulting to settings in ContentCoreConfig).

Concurrency: Audio transcription can be parallelized. The audio_concurrency setting (default: 3, max: 10) controls how many segments are processed simultaneously using a semaphore. Results are returned in the original order.

Content Core provides an MCP (Model Context Protocol) server that exposes two primary tools for AI agents and LLM environments. Both tools return plain text results.

Available Tools

1. extract_content: Extracts content from a given source.
2. summarize_content: Summarizes provided text.

You can launch the server via the CLI using the content-core mcp command.

When configuring engines (such as url_engine or document_engine), the system follows a strict rule regarding explicit vs. automatic selection:

1. Explicit Selection: If you specify a particular engine, the system will honor it. If that engine is unavailable or fails, the operation will raise an error rather than switching to a different one. This ensures you get the specific quality level you requested.
2. Automatic Selection (auto): If you set the engine to auto, the system is permitted to fall back to alternative engines if the preferred one fails. This is the only mode that allows for graceful degradation.

This behavior applies to all implementation selectors, including LLM or STT providers.

Content Core automatically detects Reddit URLs and extracts data via the public .json endpoint. This requires no API key or authentication for public posts. It extracts:

• Post title and body
• Metadata (author, score, subreddit)
• Full comment tree with nested replies

If JSON extraction fails, it falls back to the configured url_engine.

The orchestrator automatically routes inputs to the appropriate processor based on the input type or MIME type:

Errors:

• InvalidInputError: Raised when no source is provided.
• UnsupportedTypeException: Raised when the MIME type is unknown.

Content Core separates dependencies into two categories:

• Core Formats: Libraries required to read standard file formats (e.g., python-docx, openpyxl, pdfplumber) are included in the main project dependencies. You can expect these to work upon installation.
• Optional Engines: Heavyweight libraries used as alternative engines (e.g., docling, crawl4ai) are treated as optional extras.

If an optional engine is missing, the system will raise an ImportError that includes the specific installation command required to enable that feature.

When extracting from audio or video files, Content Core automatically extracts the audio track and transcribes it using a Speech-to-Text (STT) engine.

Key Features:

• Parallel Transcription: Long audio files are split into segments and transcribed in parallel to improve speed.
• Concurrency Control: Use audio_concurrency (1-10, default 3) to manage simultaneous transcriptions. Higher values increase speed but may trigger API rate limits.
• Custom STT: You can override the STT provider and model for specific calls using audio_provider and audio_model in the ContentCoreConfig object.

# Custom STT model for a specific call
from content_core import ContentCoreConfig

config = ContentCoreConfig(audio_provider="openai", audio_model="whisper-1")
result = await content_core.extract_content(file_path="audio.mp3", config=config)