Scalene Python Profiler

Scalene is a high-performance Python profiler that uses sampling to minimize overhead (typically 10-20%). It provides line-level and function-level granularity for several metrics:

• CPU Profiling: Separates time spent in Python from native code (C/C++ libraries) and identifies system time (I/O bottlenecks).
• GPU Profiling: Reports GPU time (currently limited to NVIDIA-based systems).
• Memory Profiling: Tracks memory usage per line, identifies likely memory leaks, separates Python vs. native memory consumption, and profiles copying volume (MB/s) to detect inefficient data conversions.
• Async Profiling: Attributes await time to specific lines, including mean and peak concurrency of coroutines.
• Stack Views: When --stacks is enabled (default), it provides stitched Python + native call stacks, memory-weighted flame charts, and a wall-clock timeline.

Scalene uses three distinct renderers for profile output. If you are extending Scalene or debugging output discrepancies, note that these are separate systems:

• JSON output: Handled by scalene_json.py:output_profiles() via output_profile_line().
• CLI viewer: Handled by scalene_parseargs.py:_display_profile_cli(). This is used when running scalene view --cli and is completely separate from the HTML/GUI output logic.
• HTML/GUI output: Handled by scalene_output.py:output_profiles() (used by scalene view --html).
• GUI (Browser): Uses scalene-gui.ts:makeProfileLine() which embeds Vega-Lite charts via vegaEmbed().
• Standalone HTML: Generated by scalene_utility.py:generate_html(standalone=True), which embeds all assets inline.

Scalene is a high-performance profiler that provides detailed insights across several dimensions:

• CPU Profiling: Separates time spent in Python from time in native code (C/C++ libraries) and identifies system/IO bottlenecks.
• GPU Profiling: Reports GPU time (currently limited to NVIDIA systems).
• Memory Profiling: Provides per-line memory usage, identifies memory leaks, separates Python vs. native memory consumption, and tracks "copy volume" (megabytes copied per second).
• Granularity: Performs profiling at both the function level and the specific line level.
• Reduced Profiles: Can use --reduced-profile to only report lines consuming >1% CPU or performing >100 allocations.

Scalene profiles provide several key metrics to help triangulate performance issues:

• Time Python: Time spent in Python code.
• native: Time spent in non-Python code (e.g., C/C++ libraries).
• system: Time spent in the system (e.g., I/O).
• GPU: Time spent on the GPU (if an NVIDIA GPU is present).
• Memory Python: Memory allocation on the Python side.
• net: Net memory allocation in MB (positive for allocation, negative for reclamation).
• timeline / %: Visualized memory consumption per line over runtime and its percentage of total activity.
• Copy (MB/s): The rate of megabytes being copied per second.

Scalene can suggest code optimizations using AI providers like Amazon Bedrock, Microsoft Azure, OpenAI, or local models via Ollama.

To use this feature:

1. Run your profile.
2. In the web GUI, locate the "AI Optimization Options" box.
3. Select your provider and enter required credentials.
4. Click the lightning bolt (⚡) next to a specific line or the explosion (💥) for a code region to generate a suggestion.
5. Click a suggestion to copy it to your clipboard.

The Scalene GUI follows a specific asynchronous rendering lifecycle for charts:

1. Placeholder Creation: makeProfileLine() generates the HTML string containing <span id="chart_name${index}"> placeholders.
2. Spec Collection: Chart specifications are collected and pushed into arrays (e.g., cpu_bars, gpu_pies, await_pies) during the profile loop.
3. DOM Insertion: The HTML containing the placeholders is inserted into the DOM.
4. Asynchronous Rendering: Once the DOM is ready, embedCharts(array, "prefix") is called, which invokes vegaEmbed() for every specification in the array.

Note for Testers: Because SVGs render asynchronously, Selenium tests must implement explicit waits to verify the presence and content of SVG elements.

When implementing pie charts in the Scalene GUI, follow these best practices:

• Complete Circles: Always provide two data values (the filled value and the remaining value) to ensure a complete circle. Using a single value with scale: { domain: [0, 100] } results in partial arcs with gaps.
• Rotating Pies: To create a continuous effect where each row's wedge starts where the previous one ended, use the theta encoding with a dynamic range:
  • Set scale: { range: [startAngle, startAngle + 2*PI] }.
  • Track the cumulative angle in TypeScript:

    pieAngles.await += (pct / 100) * 2 * Math.PI;

• State Management: Reset the angle state per table. The line profile and function profile tables must use separate pieAngles objects.

You can start a process with profiling disabled and then enable/disable it remotely using the PID. This is useful for controlling profiling during specific phases of a long-running process.

# 1. Start the program with profiling OFF
scalene run --off prog.py

# 2. Resume profiling for a specific PID
python3 -m scalene.profile --on --pid <PID>

# 3. Suspend profiling for a specific PID
python3 -m scalene.profile --off --pid <PID>

To profile a Python program, run Scalene from the command line. By default, the profiling results are saved to scalene-profile.json.

To pass arguments to your actual program (the code you are profiling) rather than to Scalene itself, use the -- separator. Everything following the -- will be passed directly to your script.

scalene my_script.py -- --my_arg value

When using scalene run, you can pass arguments to your target script by using the --- separator. Everything following the separator is passed directly to your program.

scalene run your_prog.py --- --arg1 --arg2

You can install Scalene using either pip or conda.

# Using pip
python3 -m pip install -U scalene

# Using conda
conda install -c conda-forge scalene