juriscraper

The goDLS function is used in PACER to generate a form, append it to the document, and submit it to trigger a document download (typically a PDF). When automating PACER, you may need to replicate this POST request behavior.

District Court goDLS Parameters

When calling goDLS in District courts, the following parameters are used in the POST body:

• hyperlink: The HTML 'action' attribute (the URL where the form is posted).
• caseid: The internal PACER ID for the case (de_caseid).
• de_seq_num: The internal PACER document number within the case (de_seqno). Note that this may differ from the visible document number.
• got_receipt: If set to '1', it bypasses the receipt page and downloads the PDF immediately. This is typically set after a user has viewed the receipt.
• pdf_header: Controls the blue header on generated PDFs. 1 includes the header, 2 excludes it.
• pdf_toggle_possible: (Unknown purpose).
• magic_num: (Used when an NEF hyperlink is clicked on multi-document filings).
• hdr: (Used when an NEF hyperlink is clicked on ROA/Appendix).

Bankruptcy CMECF goDLS Parameters

In Bankruptcy CMECF, the signature changes to include bankruptcy-specific parameters for claim documents, replacing the hdr parameter:

• claim_id: Bankruptcy-specific claim ID.
• claim_num: Bankruptcy-specific claim number.
• claim_doc_seq: Bankruptcy-specific claim document sequence.

// District Court signature
function goDLS(hyperlink, de_caseid, de_seqno, got_receipt, pdf_header, pdf_toggle_possible, magic_num, hdr) {
    // ... implementation ...
}

// Bankruptcy CMECF signature
function goDLS(hyperlink, de_caseid, de_seqno, got_receipt, pdf_header, pdf_toggle_possible, magic_num, claim_id, claim_num, claim_doc_seq) {
    // ... implementation ...
}

Some scrapers require an automated WebDriver (like Geckodriver for Firefox). You can use a local installation or connect to a remote WebDriver (e.g., a Selenium Docker container) using environment variables.

Local Installation

Download the appropriate Geckodriver package for your OS from the Geckodriver releases, extract it, and move it to your path:

sudo mv geckodriver /usr/local/bin

Remote WebDriver Configuration

Use the following environment variables to control WebDriver behavior:

• WEBDRIVER_CONN: The connection string for the remote driver. Defaults to local. To use a remote driver, set this to the URL (e.g., http://YOUR_DOCKER_IP:4444/wd/hub).
• SELENIUM_VISIBLE: Set this to any value to disable headless mode (if the driver supports it).

docker run \
    -p 4444:4444 \
    -p 5900:5900 \
    -v /dev/shm:/dev/shm \
    selenium/standalone-firefox-debug

The AbstractSite class is the base class for all Juriscraper implementations. To create a new scraper, you must subclass AbstractSite and implement the necessary data-gathering methods.

Key lifecycle steps in a scraper include:

1. Initialization: Define court_id, url, parameters, and the list of attributes to scrape in _all_attrs (e.g., self._all_attrs = ['case_names', 'case_dates', ...]).
2. Parsing: The parse() method (which you typically call) orchestrates the download and then calls _get_<attr>() methods for every attribute listed in _all_attrs to populate the data.
3. Data Cleaning: The class automatically calls _clean_attributes() to sanitize scraped data.
4. Post-Processing: You can override _post_parse() to perform custom logic after all attributes are populated.
5. Sanity Checking: The class runs _check_sanity() to ensure all attribute lists have matching lengths and required fields are present.

from juriscraper.AbstractSite import AbstractSite

class MyNewCourtScraper(AbstractSite):
    def __init__(self, cnt=None, **kwargs):
        super().__init__(cnt=cnt, **kwargs)
        self.court_id = "my_court"
        self.url = "https://example.com/cases"
        self._all_attrs = ["case_names", "case_dates", "case_numbers"]

    async def _get_case_names(self):
        # Implementation to extract case names from self.html
        return ["Case A", "Case B"]

    async def _get_case_dates(self):
        # Implementation to extract case dates
        return [datetime(2023, 1, 1), datetime(2023, 1, 2)]

    async def _get_case_numbers(self):
        return ["1", "2"]

# Usage:
# scraper = MyNewCourtScraper()
# await scraper.parse()
# print(scraper.to_json())

Use OpinionSiteLinear when a website's opinions are presented in a linear list format rather than requiring complex, separate HTML path parsing for every single attribute.

To use this class, you must:

1. Extend OpinionSiteLinear.
2. Implement the _process_html() method. This method is responsible for parsing the HTML and populating self.cases with a list of dictionaries.
3. Ensure the dictionaries in self.cases use valid shorthand keys (see Valid Keys for OpinionSiteLinear).
4. (Optional) If all cases on the page share the same status, define self.status in your __init__ method. Otherwise, include a status key in each case dictionary.

from juriscraper.OpinionSiteLinear import OpinionSiteLinear

class MyLinearSite(OpinionSiteLinear):
    def _process_html(self):
        # Logic to parse HTML and populate self.cases
        self.cases = [
            {
                "name": "Case Name",
                "url": "http://example.com/case1",
                "date": "2023-01-01",
                "status": "Published",
                "docket": "12345",
                # ... other keys
            }
        ]

The ClusterSite class is a specialized version of OpinionSiteLinear designed for sites that aggregate data from multiple sources into a single case. Unlike standard sites that return a list of attributes, ClusterSite uses a dictionary representation for each case.

In a ClusterSite, a single 'case' returned by the scraper represents an opinion cluster. Instead of a single set of opinion attributes (like authors or urls), the cluster contains a sub_opinions list. Each item in sub_opinions contains the specific details for an individual opinion within that cluster.

Key characteristics:

• Data Format: Returns a list of dictionaries (accessible via __iter__, __getitem__, or len()).
• Clustering Logic: Opinions are grouped based on matching name, docket, and date.
• Date Flexibility: You can control how strictly dates must match by setting cluster_by_date_max_days.

class MyClusterSite(ClusterSite):
    # Implementation would involve overriding parse() or providing data 
    # that results in a list of dictionaries where opinions are nested 
    # under the 'sub_opinions' key.

Use OralArgumentSiteLinear when a website's oral argument data is presented in a linear list format rather than requiring complex HTML path parsing for individual elements. To use this class, extend it and implement the _process_html() method.

Inside _process_html(), you must populate self.cases with a list of dictionaries. Each dictionary in self.cases should contain the following keys to support the standard getters:

• name: The name of the case.
• url: The download URL for the oral argument.
• date: The date string (which will be processed by convert_date_string).
• docket: The docket number.

Optional keys that can be included in the dictionaries to support optional getters include:

• judge: To use _get_judges().
• attorney: To use _get_attorneys().

from juriscraper.OralArgumentSiteLinear import OralArgumentSiteLinear

class MyLinearSite(OralArgumentSiteLinear):
    def _process_html(self):
        # Logic to parse HTML and populate self.cases
        self.cases = [
            {
                "name": "Case Name",
                "url": "https://example.com/audio.mp3",
                "date": "January 1, 2023",
                "docket": "123-ABC",
                "judge": "Judge Smith",
                "attorney": "Attorney Doe"
            }
        ]

# Usage
scraper = MyLinearSite(url="https://example.com")
scraper._process_html()
print(scraper._get_case_names())