DOCSEARCHbyAlgolia
Submit Docs

OLDP (Open Legal Data Platform)

repository·main·Indexed 41 days ago

https://github.com/openlegaldata/oldp

A Python 3.12 web application built on Django for processing legal texts. It provides a REST API and an Elasticsearch-based search engine to make court decisions and laws freely accessible. Features include a CRUD REST API powered by Django REST Framework, support for Lucene search syntax, and a citation graph for legal resources. Version 0.13.2.

Tokens
67.5K
Snippets
193
Records
309
Agent score
100%

What's inside OLDP

  1. Overview of Open Legal Data Platform (OLDP)

    main

    OLDP is a web application built with Python 3.12 and the Django web framework. It serves as a platform for processing legal text, providing a REST API, and hosting an Elasticsearch-based search engine.

    Its primary purpose is to make legal information (such as court decisions and laws) freely accessible to the public and third-party applications. It acts as the core component of a larger ecosystem that includes a German theme, a data ingestor, and a dump-preprocessing toolkit.

  2. Overview of the Case Statistics API

    main

    The Case Statistics API provides aggregated case counts with date-bucketed time series, optimized for data analysis and plotting.

    Key features:

    • Time Bucketing: Data can be grouped into year, month, or day intervals.
    • Dimensions: Statistics can be retrieved for the overall total or broken down by country, state, court, or source.
    • Default Behavior: All endpoints default to the last year with monthly buckets.
    • Visibility:
      • Anonymous users: See statistics for accepted cases only.
      • Staff users: See all cases (including pending and rejected). Use the review_status parameter to filter specific statuses.

  3. Overview of OLDP: Open Legal Data Platform

    main

    OLDP is a Python 3.12 web application built on the Django framework designed for processing legal text. It provides a REST API and an Elasticsearch-based search engine to make legal documents (court decisions and laws) freely accessible.

    Key features include:

    • Cases: Court decisions with metadata and HTML content.
    • Laws: Full-text laws, regulations, and related case-law.
    • Courts: Browsable court hierarchies (states, jurisdiction, level of appeal).
    • Search: Elasticsearch/Haystack-based engine supporting common search syntax and faceting.
    • API: CRUD REST API powered by Django REST Framework (DRF) with auto-generated Swagger clients.
    • Themes: Customizable UI for different country needs.

  4. Overview of OLDP Django applications

    main

    OLDP is organized into several focused Django applications located under oldp/apps/. Each app manages a specific domain of the legal data platform:

    AppResponsibility
    casesCourt decisions (Case, RelatedCase): metadata, HTML content, file number, ECLI, review status.
    lawsLegislation: LawBook (a collection with revisions) and Law (a section), plus RelatedLaw.
    courtsCourt entities and their geography: Court, State, City, Country.
    referencesCitation extraction and linking between content items (Reference, reference markers).
    searchFull-text search via Haystack + Elasticsearch; SearchableContent mixin, SearchQuery.
    annotationsUser annotations on content: AnnotationLabel, CaseAnnotation, CaseMarker.
    sourcesProvenance of imported content: Source and the SourceContent mixin.
    accountsUser accounts, API tokens and granular per-token permissions (extends django-allauth).
    processingThe content-processing pipeline (ContentProcessor, input handlers, timeouts).
    mcpModel Context Protocol server exposing OLDP to AI agents.
    libShared utilities, notably markers.py (reference-marker insertion with drift recovery).
    topicsTopic / category taxonomy.
    homepageLanding page and static page rendering.
    contactContact / feedback form.

  5. Overview of OLDP Production Architecture

    main

    A production deployment of the Open Legal Data Platform (OLDP) typically requires the following components:

    • OLDP app: A Django application served by an application server like Gunicorn.
    • Relational Database: MariaDB or MySQL is recommended.
    • Elasticsearch: Version 7.17.x is required for search functionality.
    • Optional Components:
      • Redis: Used as a cache backend.
      • CDN / Reverse Proxy: (e.g., Nginx) placed in front of the application.

    The provided docker-compose.yaml includes app, db (MariaDB), and search (Elasticsearch) services, which serves as a baseline for single-host deployments.

  6. Understand the OLDP ecosystem and project roles

    main

    OLDP is a core platform for storing and serving legal data, surrounded by a small ecosystem of independent, MIT-licensed Python packages. Each project serves a specific stage in the legal data lifecycle:

    • oldp: The core Django web application. It manages models, the REST API, Elasticsearch search, and provides an MCP server. It is used via manage.py.
    • oldp-de: A German-specific theme and configuration package for the core platform.
    • oldp-ingestor: A CLI tool used to scrape or fetch data from external legal sources and push it into OLDP.
    • oldp-toolkit: A CLI tool (oldpt) used to preprocess OLDP data dumps into formats like Parquet, JSONL, or HuggingFace datasets.

  8. Understand Law Book revision management and the `latest` flag

    main

    The API uses an approval-driven revision management system to ensure public data remains stable during updates.

    Key Concepts

    • review_status: Submissions via API tokens are created with review_status="pending". They only become accepted once approved (e.g., via admin).
    • latest flag: This boolean identifies the newest accepted revision. It is reassigned only upon approval, not upon submission.
    • Stability Invariant: A pending submission (latest=false) never demotes the currently-published revision. This prevents the public site from showing empty or unverified data while a new revision is under review.

    Revision Lifecycle

    1. Submit: POST a new revision $\rightarrow$ latest=false, review_status="pending". The existing latest revision remains active.
    2. Approve: Once approved, the new revision becomes latest=true and the previous latest revision is demoted to false.
    3. Reject: The published latest revision remains unchanged.

    If the latest flag becomes inconsistent due to legacy data, you can repair it using the management command: manage.py backfill_latest_books.

  10. How citation lookups work in the backend

    main

    Citation lookups in the CaseIndex rely on multi-value fields. When querying for cited laws or cases, the system uses the following fields:

    • cited_laws: A list of tokens formatted as "<book_slug>__<section_slug>".
    • cited_cases: A list of cited-case Primary Keys (PKs) as strings.

    Important: To ensure query stability across book revisions, do not manually concatenate tokens. Instead, use the helper function oldp.apps.cases.search_indexes.cited_law_token(book, section) to generate the correct token format.

  11. Understand the OLDP data dump snapshot contract

    main

    OLDP data dumps follow a specific contract to ensure data integrity and reproducibility:

    Always-accepted filter

    Records for Case, Law, LawBook, and Court are guaranteed to have review_status == "accepted". Pending or declined records are excluded.

    Stable ordering

    Records are written in ascending primary-key order. This ensures that two consecutive dump runs on the same database state produce byte-identical files, which is critical for reproducible benchmarks and dataset versions.

    Latest-only LawBooks

    By default, only LawBook rows with latest=True are exported, and laws.jsonl.gz is filtered to laws whose parent book is the latest revision. This matches the public website and most downstream consumers. To include historical revisions, use the --include-lawbook-revisions flag.

    Self-describing manifest

    A manifest.json file is included in every dump. It contains the snapshot_date, oldp_version, applied filters, and a files object containing row_count for each exported resource. Downstream consumers should pin their processes to the snapshot_date to maintain reproducibility.

  12. Understand performance profiling metrics and caveats

    main

    When profiling OLDP using django-querycount and Silk, keep the following in mind:

    • Querycount Overhead: The querycount header values for warm (cached) requests will often be greater than 0. This is due to profiling middleware/instrumentation overhead (e.g., Silk writes), not actual application SQL.
    • Warm Request SQL: In a healthy warm state, warm_sql_count in Silk should be 0, and meta_num_queries will typically be 1 (representing the profiling instrumentation write).
    • Timing Context: Performance timings reported in development (using SQLite and the dev server) are intended for relative comparisons and regression detection; they are not accurate estimates for production latency.
    • Schema/Docs Caching: While internal server-side caching is applied to /api/schema.json, /api/schema.yaml, and Swagger/ReDoc routes, the drf-yasg library still emits Cache-Control: max-age=0, no-cache, no-store, must-revalidate, private headers. This means internal server-side caching works, but downstream/browser caching remains disabled.