homeharvest

repository·master·Indexed 20 days ago

https://github.com/zacharyhampton/homeharvest

A real estate scraping library (version 0.8.18) that extracts property data from Realtor.com and formats it to resemble MLS listings. It provides the `scrape_property` function to fetch structured data with flexible filtering by location, listing type, physical attributes (beds, baths, sqft, price), and time ranges. Results can be returned as a Pandas DataFrame, raw JSON, or type-safe Pydantic models.

Tokens
3.6K
Snippets
8
Records
18
Agent score
71%

What's inside homeharvest

  1. Filter properties by date and time

    master

    HomeHarvest provides several ways to filter properties based on when they were listed, sold, or updated.

    Listing/Sale Date Filters:

    • past_days: Get properties sold or listed in the last N days. Accepts an int or timedelta.
      • For PENDING: Filters by pending_date.
      • For SOLD: Filters by sold_date.
      • For FOR_SALE/FOR_RENT: Filters by list_date.
    • date_from & date_to: Get properties between two specific dates/times. Supports:
      • Date strings: "2025-01-20" (day precision).
      • Datetime strings: "2025-01-20T14:30:00" (hour precision).
      • date objects: date(2025, 1, 20).
      • datetime objects: datetime(2025, 1, 20, 14, 30).
    • past_hours: Get properties in the last N hours (requires client-side filtering). Accepts int or timedelta.

    Update Date Filters (Last Update):

    • updated_since: Filter by last_update_date. Accepts datetime or ISO 8601 string.
    • updated_in_past_hours: Filter by properties updated in the last N hours. Accepts int or timedelta.

    Note: Naive datetimes are treated as local time and converted to UTC. Timezone-aware datetimes are converted to UTC.

    from datetime import datetime, timedelta
    from homeharvest import scrape_property
    
    # Using timedelta for Pythonic usage
    results = scrape_property(
        location="Miami, FL",
        past_days=timedelta(days=7),
        updated_in_past_hours=24
    )
    
    # Using specific date ranges
    results = scrape_property(
        location="Austin, TX",
        date_from="2025-01-01",
        date_to=datetime.now()
    )
  2. Handle HomeHarvest exceptions

    master

    The following exceptions may be raised during execution:

    • InvalidListingType: Raised if an invalid option is provided to listing_type (valid: for_sale, for_rent, sold, pending).
    • InvalidDate: Raised if date_from or date_to is not in the YYYY-MM-DD format.
    • AuthenticationError: Raised if the Realtor.com token request fails.
  3. Use Pydantic models for type-safe property data

    master

    To get properties as Pydantic models, set return_type="pydantic". This provides full type hints and validation for all property fields, making it easier to access nested data like address details or property descriptions.

    from homeharvest import scrape_property
    
    # Get properties as Pydantic models for type safety and data validation
    properties = scrape_property(
        location="San Diego, CA",
        listing_type="for_sale",
        return_type="pydantic"  # Returns list of Property models
    )
    
    # Access model fields with full type hints and validation
    for prop in properties[:5]:
        print(f"Address: {prop.address.formatted_address}")
        print(f"Price: ${prop.list_price:,}")
        if prop.description:
            print(f"Beds: {prop.description.beds}, Baths: {prop.description.baths_full}")
  4. Filter properties by time and date

    master

    HomeHarvest provides several ways to filter results by time:

    • past_days (int): Number of past days to filter. Uses last_sold_date for 'sold' listings and list_date for others.
    • past_hours (int | timedelta): Number of past hours to filter (more precise than past_days). Uses client-side filtering.
    • date_from & date_to (str): Start and end dates. Both are required when used. Supports:
      • Day precision: "YYYY-MM-DD" or date objects.
      • Hour precision: "YYYY-MM-DDTHH:MM:SS" or datetime objects.
    • updated_since (datetime | str): Filter by last_update_date using ISO 8601 strings or datetime objects.
    • updated_in_past_hours (int | timedelta): Filter by last_update_date within the last X hours.

    Note: past_days, past_hours, and date_from/date_to cannot be used together.

  5. Scrape property data with `scrape_property()`

    master

    The primary entry point for HomeHarvest is the scrape_property() function. It allows you to search for real estate listings based on location, listing type, and various filters. By default, it returns a Pandas DataFrame, but you can request Pydantic models for type safety or raw JSON.

    from homeharvest import scrape_property
    
    # Default usage (returns Pandas DataFrame)
    properties = scrape_property(
        location="San Diego, CA",
        listing_type="for_sale"
    )
  6. Configure `scrape_property()` required parameters

    master

    The scrape_property() function requires two main arguments:

    • location (str): A flexible search string. Supported formats include:
      • ZIP code: "92104"
      • City: "San Diego"
      • City, State: "San Diego, CA" or "San Diego, California"
      • Full address: "1234 Main St, San Diego, CA 92104"
      • Neighborhood: "Downtown San Diego"
      • County: "San Diego County"
      • State: "California" (Note: State names work, but abbreviated states without city context may not be supported)
    • listing_type (str | list[str] | None): The status of the listing.
      • Options: 'for_sale', 'for_rent', 'sold', 'pending', 'off_market', 'new_community', 'other', 'ready_to_build'
      • A list of strings (e.g., ['for_sale', 'pending']) returns properties matching ANY status in the list.
      • None returns common types: for_sale, for_rent, sold, pending, off_market.
  7. Sort and paginate search results

    master

    Control the order and volume of your results:

    • Sorting:
      • sort_by (str): Options include 'list_date', 'sold_date', 'list_price', 'sqft', 'beds', 'baths', 'last_update_date'.
      • sort_direction (str): 'asc' (ascending) or 'desc' (descending, default).
    • Pagination & Limits:
      • limit (int): Max number of properties to fetch (default/max is 10000).
      • offset (int): Starting position for pagination. Use with limit to fetch results in chunks.
      • parallel (bool): Controls pagination strategy. Default is True (parallel for speed). Set to False for sequential fetching (useful for rate limiting).
  8. Filter properties by physical attributes

    master

    You can refine your search using the following attribute filters:

    • Bedrooms: beds_min, beds_max (int)
    • Bathrooms: baths_min, baths_max (float)
    • Square Footage: sqft_min, sqft_max (int)
    • Lot Size: lot_sqft_min, lot_sqft_max (int)
    • Year Built: year_built_min, year_built_max (int)
    • Price: price_min, price_max (int)
  9. Filter properties by time ranges

    master

    You can filter results based on when they were last updated using time-based parameters. You can use integers for hours/days or Python datetime and timedelta objects for more precision.

    Supported time parameters:

    • past_hours: Number of hours (int) or timedelta object.
    • updated_in_past_hours: Number of hours (int) or timedelta object.
    • date_from: A datetime object for the start of the range.
    • date_to: A datetime object for the end of the range.
    from datetime import datetime, timedelta
    
    # Filter by hours or use timedelta objects
    properties = scrape_property(
        location="Austin, TX",
        listing_type="for_sale",
        past_hours=24,  # or timedelta(hours=24)
    )
  10. Use scrape_property() to fetch real estate data

    master

    The primary entry point for HomeHarvest is the scrape_property function. It fetches property data from Realtor.com and returns a collection that can be exported to various formats like CSV or Excel.

    Key parameters include:

    • location: A string representing the search area.
    • listing_type: The status of the listing (e.g., for_sale, for_rent, sold, pending).
    • past_days: An integer to filter properties updated within a certain number of days.
    from homeharvest import scrape_property
    
    properties = scrape_property(
        location="San Diego, CA",
        listing_type="sold",  # for_sale, for_rent, pending
        past_days=30
    )
    
    properties.to_csv("results.csv", index=False)
    print(f"Found {len(properties)} properties")