DOCSEARCHbyAlgolia
Submit Docs

Cianparser

repository·main·Indexed 41 days ago

https://github.com/lenarsaitov/cianparser

A Python 3.8+ library for scraping structured real estate data from Cian.ru, including long-term rentals, sales, and new developments. It features methods for collecting apartment flats, suburban properties, and new objects, with support for advanced search filters, CSV export, and proxy integration to bypass Cloudflare and CAPTCHA challenges.

Tokens
1.6K
Snippets
7
Records
8
Agent score
3%

What's inside Cianparser

  1. Use proxies to bypass Cloudflare and CAPTCHA

    main

    The library uses cloudscraper to handle Cloudflare protection. However, long-term use may trigger CAPTCHAs. To mitigate this, provide a list of HTTPS proxy strings to the proxies argument during CianParser initialization. The parser will automatically test each proxy to find one that is available and does not trigger a CAPTCHA.

    proxies = [
    '117.250.3.58:8080', 
    '115.96.208.124:8080',
    '152.67.0.109:80',
]
moscow_parser = cianparser.CianParser(location="Москва", proxies=proxies)

  3. Configure advanced search filters via additional_settings

    main

    Pass a dictionary to the additional_settings argument in any collection method to refine your search.

    Common Keys:

    • start_page / end_page: Pagination control.
    • is_by_homeowner: True to filter for ads by owners only.
    • min_price / max_price: Price range in rubles.
    • min_balconies: Minimum number of balconies.
    • have_loggia: Boolean for presence of a loggia.
    • min_house_year / max_house_year: Year of construction range.
    • min_floor / max_floor: Target floor range.
    • min_total_floor / max_total_floor: Total floors in the building range.
    • house_material_type: Integer ID (1: brick, 2: monolith, 3: panel, etc.).
    • metro / metro_station: Metro system and specific station (use cianparser.list_metro_stations() for valid values).
    • sort_by: Sorting method (e.g., "price_from_min_to_max").
    • only_flat / only_apartment: Boolean filters.
    additional_settings = {
    "start_page": 1,
    "end_page": 10,
    "is_by_homeowner": True,
    "min_price": 1000000,
    "max_price": 10000000,
    "metro": "Московский",
    "metro_station": "ВДНХ",
    "sort_by": "price_from_min_to_max",
}

  4. Understand data collection limitations and best practices

    main

    Limitations

    • Page Limit: Cian only provides lists up to 54 pages (approx. 1512 ads). To collect more data, use more specific queries (e.g., instead of rooms=(1, 2), run separate requests for rooms=1 and rooms=2).
    • Environment: This parser will not work in Google Colaboratory.

    Best Practices

    • Avoid Parallelism: Do not run multiple collection processes simultaneously on the same machine. The library includes a built-in 4-5 second pause between pages to prevent IP blocking.
    • Extra Data Trade-off: Using with_extra_data=True is useful for getting details like phone, kitchen_meters, and heating_type, but it makes the process 5-10 times slower because it visits every individual ad page.
    • Missing Data: If certain attributes (like year_of_construction) are missing from an ad, the parser returns -1 for numeric fields or an empty string for string fields.

  5. Parse new developments with get_newobjects()

    main

    Use get_newobjects() to collect data specifically regarding new residential developments.

    Arguments:

    • with_saving_csv: If True, saves data to CSV in real-time.
    import cianparser

moscow_parser = cianparser.CianParser(location="Москва")
data = moscow_parser.get_newobjects()

  6. Parse apartment flats with get_flats()

    main

    Use get_flats() to collect data on apartments.

    Arguments:

    • deal_type: Type of deal. Use "rent_long" for long-term rent or "sale" for sales.
    • rooms: Number of rooms. Can be an integer, a tuple (e.g., (1, 2)), a string (e.g., "studio"), or "all" (default).
    • with_saving_csv: If True, saves collected data to a CSV file in real-time.
    • with_extra_data: If True, collects additional details (like kitchen area, phone numbers, etc.) but significantly increases collection time (5-10x slower).
    • additional_settings: A dictionary for advanced search filters (see Additional Search Settings).
    import cianparser

moscow_parser = cianparser.CianParser(location="Москва")
data = moscow_parser.get_flats(
    deal_type="sale", 
    rooms=(1, 2), 
    with_saving_csv=True, 
    additional_settings={"start_page": 1, "end_page": 2}
)

  7. Parse suburban properties with get_suburban()

    main

    Use get_suburban() to collect data on houses, land plots, and townhouses.

    Arguments:

    • suburban_type: The type of property. Options: "house", "house-part", "land-plot", or "townhouse".
    • deal_type: Type of deal ("rent_long" or "sale").
    • with_saving_csv: If True, saves data to CSV in real-time.
    • with_extra_data: If True, collects extra details (slower).
    • additional_settings: Advanced search filters.
    import cianparser

moscow_parser = cianparser.CianParser(location="Москва")
data = moscow_parser.get_suburban(
    suburban_type="townhouse", 
    deal_type="sale", 
    additional_settings={"start_page": 1, "end_page": 1}
)

  8. Initialize the CianParser

    main

    To start parsing, create an instance of CianParser. You must provide a location (e.g., "Москва"). You can use cianparser.list_locations() to see all supported locations. Optionally, you can provide a list of proxies to bypass Cloudflare or CAPTCHA challenges.

    import cianparser

# Initialize with a location
moscow_parser = cianparser.CianParser(location="Москва")

# Initialize with proxies (list of HTTPS proxy strings)
proxies = ['117.250.3.58:8080', '115.96.208.124:8080']
moscow_parser = cianparser.CianParser(location="Москва", proxies=proxies)