python-jobspy Documentation

repository·main·Indexed 23 days ago

A job scraping library for aggregating job postings from LinkedIn, Indeed, Glassdoor, ZipRecruiter, Bayt, and other boards into a pandas DataFrame. It features the scrape_jobs() API for filtering by search terms, location, and job type, and supports proxy rotation to mitigate 429 rate limit errors. Requires Python >= 3.10.

Tokens: 1.4K
Snippets: 2
Records: 7
Agent score: 38%

What's inside python-jobspy

Install python-jobspy
main
Install the python-jobspy library using pip. Requires Python version >= 3.10.
```
pip install -U python-jobspy
```
https://github.com/speedyapply/jobspy/blob/main/README.md
Troubleshoot 429 Rate Limit Errors
main
A 429 response code indicates you have been blocked by a job board for sending too many requests. To mitigate this:
1. Wait: Allow time between scrapes.
2. Use Proxies: Use the proxies parameter to rotate your IP address.
https://github.com/speedyapply/jobspy/blob/main/README.md

Use scrape_jobs() to aggregate job postings

main

The primary API for JobSpy is scrape_jobs(). It returns a pandas DataFrame containing aggregated job postings from multiple sources. You can specify which sites to scrape, search terms, locations, and other filters.

import csv
from jobspy import scrape_jobs

jobs = scrape_jobs(
    site_name=["indeed", "linkedin", "zip_recruiter", "google"],
    search_term="software engineer",
    google_search_term="software engineer jobs near San Francisco, CA since yesterday",
    location="San Francisco, CA",
    results_wanted=20,
    hours_old=72,
    country_indeed='USA',
)
print(f"Found {len(jobs)} jobs")
print(jobs.head())
jobs.to_csv("jobs.csv", quoting=csv.QUOTE_NONNUMERIC, escapechar="\", index=False)

Configure country for Indeed and Glassdoor

main

When searching on Indeed or Glassdoor, you must provide the country_indeed parameter. Supported countries include:


Argentina	Australia*	Austria*
Bahrain	Belgium*	Brazil*
Canada*	Chile	China
Colombia	Costa Rica	Czech Republic
Denmark	Ecuador	Egypt
Finland	France*	Germany
Greece	Hong Kong*	Hungary
India*	Indonesia	Ireland*
Israel	Italy	Japan
Kuwait	Luxembourg	Malaysia
Mexico*	Morocco	Netherlands*
New Zealand	Nigeria	Norway
Oman	Pakistan	Panama
Peru	Philippines	Poland
Portugal	Qatar	Romania
Saudi Arabia	Singapore*	South Africa
South Korea	Spain*	Sweden
Switzerland*	Taiwan	Thailand
Turkey	Ukraine	United Arab Emirates
UK*	USA*	Uruguay
Venezuela	Vietnam

Note: Asterisk () indicates support for Glassdoor as well.

Understand scrape_jobs() limitations
main
Certain parameter combinations are mutually exclusive for specific job boards:
Indeed Limitations: You can only use ONE of the following in a single search:
- hours_old
- job_type & is_remote
- easy_apply
LinkedIn Limitations: You can only use ONE of the following in a single search:
- hours_old
- easy_apply
https://github.com/speedyapply/jobspy/blob/main/README.md

Reference parameters for scrape_jobs()

main

The following optional parameters are available for scrape_jobs():

Parameter	Type	Description
`site_name`	`list` or `str`	`linkedin`, `zip_recruiter`, `indeed`, `glassdoor`, `google`, `bayt`, `bdjobs` (default is all)
`search_term`	`str`	The job title or keywords
`google_search_term`	`str`	Specific search term for Google Jobs; the only parameter used for filtering Google jobs
`location`	`str`	Geographic location
`distance`	`int`	Distance in miles (default: 50)
`job_type`	`str`	`fulltime`, `parttime`, `internship`, `contract`
`proxies`	`list`	List of proxies in format `['user:pass@host:port', 'localhost']`. Scrapers use round-robin
`is_remote`	`bool`	Filter for remote jobs
`results_wanted`	`int`	Number of results to retrieve per specified site
`easy_apply`	`bool`	Filters for jobs hosted on the job board site (Note: LinkedIn easy apply filter no longer works)
`user_agent`	`str`	Override default user agent
`description_format`	`str`	`markdown` (default) or `html`
`offset`	`int`	Starting index for the search
`hours_old`	`int`	Filter by hours since posting (ZipRecruiter/Glassdoor round up to next day)
`verbose`	`int`	Verbosity: `0` (errors only), `1` (errors+warnings), `2` (all logs, default)
`linkedin_fetch_description`	`bool`	Fetches full description and direct URL for LinkedIn (slower, increases requests by O(n))
`linkedin_company_ids`	`list[int]`	Search LinkedIn jobs by specific company IDs
`country_indeed`	`str`	Filter country for Indeed & Glassdoor
`enforce_annual_salary`	`bool`	Converts wages to annual salary
`ca_cert`	`str`	Path to CA Certificate file for proxies

JobPost data schema
main
The output DataFrame follows this schema:
- title
- company
- company_url
- job_url
- location (country, city, state)
- is_remote
- description
- job_type: fulltime, parttime, internship, contract
- job_function:
  interval: yearly, monthly, weekly, daily, hourly
  min_amount
  max_amount
  currency
  salary_source: direct_data, description (parsed from posting)
- date_posted
- emails
Platform Specific Fields:
- LinkedIn: job_level
- LinkedIn & Indeed: company_industry
- Indeed: company_country, company_addresses, company_employees_label, company_revenue_label, company_description, company_logo
- Naukri: skills, experience_range, company_rating, company_reviews_count, vacancy_count, work_from_home_type
https://github.com/speedyapply/jobspy/blob/main/README.md