DOCSEARCHbyAlgolia
Submit Docs

Jack Pan's Blog

website·Indexed 30 days ago

https://www.jackpan.me/

Technical blog by Jack Pan, a data platform engineer, featuring deep dives into computer vision data pipelines, human-in-the-loop systems using Label Studio, and write-ups from CIT CTF 2026.

Tokens
16.8K
Snippets
47
Records
117
Agent score
48%

What's inside Jack Pan

  1. Overview of the video pre-annotation pipeline architecture

    The pipeline transforms raw MP4 videos into Label Studio task JSONs for human verification. It uses a modular approach where four atomic subcommands are managed by a thin orchestrator (process). This allows for independent re-running of failing steps and faster testing.
    # Subcommands and their roles:
# handmark: ego inference (hands, arms, operated objects)
# posemark: exo inference (body skeleton)
# segment: per-episode action segmentation
# export-ls: sample review frames, build LS task JSON, run QC
# process: run all four end-to-end

  2. Overview of the video pre-annotation pipeline

    The pipeline converts raw MP4 videos (ego and exo camera footage) into Label Studio task JSONs for human verification. It consists of four atomic subcommands and an orchestrator to ensure independent re-runnability and fast testing.

    Pipeline Subcommands

    SubcommandJob
    handmarkEgo inference: hands, arms, operated objects
    posemarkExo inference: body skeleton
    segmentPer-episode action segmentation
    export-lsSample review frames, build LS task JSON, run QC
    processOrchestrator: runs all four end-to-end

  3. Understand RSC payload deserialization RCE in Next.js 15.0.4

    Next.js version 15.0.4 (with React 19.0.0) is vulnerable to RCE via RSC payload deserialization (CVE-2025-55182 / CVE-2025-66478). The vulnerability exists because the RSC wire format allows strings to reference other chunks' properties using the $N:path:to:prop syntax. When the parser handles a resolved_model chunk's _response object, it fails to validate if _prefix or _formData.get are user-controllable. By setting the then field to $N:__proto__:then, an attacker can walk the prototype chain to Object.prototype.then, eventually causing _prefix to be passed into Function(_prefix), leading to arbitrary JavaScript execution.

  4. Manage large-scale Label Studio imports via per-episode aggregation

    When dealing with large datasets (e.g., many frames per video episode), use a two-step import process to improve reliability and observability:

    1. Per-episode JSONs: Write individual task JSON files for each episode. This ensures resumability (if one episode fails, others are saved) and allows for easy manual inspection of small files.
    2. Aggregate Import: Concatenate these per-episode JSONs into a single large import file for Label Studio.

    This approach prevents the need to re-process the entire dataset if a single episode fails during the pre-annotation phase.

  5. Prevent sensitive data leaks in Flask applications

    To avoid leaking environment variables or internal configuration, keep the .env file outside of the web root (e.g., in /etc/ or a parent directory). Use python-dotenv to read the configuration once at application startup rather than serving it via a route.

  6. Implement lazy imports for heavy inference libraries

    To keep CI costs low and test suites fast, move imports of heavy inference-related libraries (e.g., mediapipe, ultralytics, cv2) from the module top-level into the specific function bodies that use them. This allows the module to be imported without loading heavy dependencies or downloading model weights, which is critical for testing pure-python logic like schema validation or data transformation.
    # pipeline.py
from .layout import EpisodeLayout
from .schema import EpisodePredictions
# No top-level mediapipe / ultralytics / cv2 imports here

def run_handmark_episode(ego_video: Path, layout: EpisodeLayout) -> None:
    import mediapipe as mp  # lazy import
    import cv2               # lazy import

    # ... actual inference logic ...

  7. Manage large-scale Label Studio imports via per-episode aggregation

    When dealing with large datasets, it is recommended to write task JSONs on a per-episode basis before aggregating them into a single project import file. This approach provides:

    1. Resumability: If a process crashes, only the failing episode needs to be re-processed.
    2. Inspection: Individual episode files are easier to debug in a text editor than multi-megabyte aggregate files.
    3. Independent Reruns: Specific episodes can be re-rendered and then re-aggregated without reprocessing the entire dataset.

  8. Analyze audio for custom duration-based encoding

    When dealing with audio challenges where signals (like barks) have near-uniform spacing but varying lengths, the payload may be encoded in the duration of the sounds rather than the silence between them. To extract this data:

    1. Apply a moving-average envelope (e.g., 20ms) to the signal.
    2. Threshold the envelope to segment individual sounds.
    3. Measure the duration of each segment and create a histogram of these durations.
    4. Identify distinct duration clusters to map them to bits or control characters (e.g., short = 0, long = 1, medium = byte separator).

  9. Use two-person agreement to measure label quality

    For dense pixel-level work (like keypoints or bounding boxes), the disagreement rate between two labelers on the same frame serves as a direct quality signal. This can be used to tier training samples:

    • Disagreement < 5px: High quality; keep both or average them.
    • Disagreement 5-20px: Moderate quality; keep one and flag for spot-check.
    • Disagreement > 20px: Poor quality (due to ambiguity or error); discard the frame entirely.

  10. Refresh the evaluation set

    Refresh the evaluation set only under two conditions: distribution drift (e.g., new cameras, tasks, or environments) or eval saturation (model reaches ~99% and improvements are indistinguishable from noise).

    Critical Rule: Refresh by replacement, never by extension. Build a completely new clean eval set and archive the old one. Extending an existing set makes it impossible to compare model versions accurately because the measurement target has changed.

  11. Manage training data using a hybrid snapshot-derivation cache

    To balance fast access with deterministic regeneration, implement a hybrid storage pattern:

    1. Source of Truth: Commit the slice.json to Git.
    2. Data Cache: Store the resulting parquet files in a cache, keyed by the hash of the slice.json. Regenerate the parquet if there is a cache miss.
    3. Immutable Archive: Store the original Label Studio exports in an archive, keyed by their own unique SHA.

  12. Implement gating logic for model retraining

    Beyond data volume, use additional gates to ensure training quality and cost-efficiency:

    1. Task Distribution: Ensure corrections span multiple tasks to avoid overfitting to a specific task distribution (e.g., using a Gini coefficient for task coverage).
    2. Eval Set Stability: Ensure the evaluation set has not drifted since the last retrain to maintain comparable performance metrics.
    3. Human-in-the-loop: Print the retraining decision to stdout rather than auto-triggering to avoid unexpected GPU costs.
    def should_retrain(state) -> tuple[bool, str]:
    delta = state.corrected_episodes_since_last_train
    corpus = state.cumulative_training_corpus
    
    # Gate 1: Volume threshold (10%)
    if delta < 0.10 * corpus:
        return False, f"need 10% delta, have {delta/corpus:.1%}"
    
    # Gate 2: Task distribution (avoid overfitting)
    if state.task_coverage_gini > 0.5:
        return False, "corrections concentrated in too few tasks"
    
    # Gate 3: Eval set stability
    if state.eval_set_changed_since_last_train:
        return False, "eval set drifted, pin it before retraining"
    
    return True, "ready"