Scrape Google Jobs with Python and APIs
This guide shows you how to programmatically extract job listings from Google Jobs using Python and the SerpAPI service. You’ll learn to set up your environment, make API calls, parse the JSON response, and save results to a CSV file—all without needing to reverse-engineer Google’s internal endpoints.
What You’ll Need
- Python 3.7+ installed on your machine
- A code editor (VS Code, PyCharm, or any text editor)
- An API key from SerpAPI (free tier available with limited queries)
- Basic familiarity with Python and JSON
Step-by-Step
- Install required packages. Open your terminal and run:
pip install requests pandas
This installs therequestslibrary for making HTTP calls andpandasfor data manipulation and CSV export. - Get your SerpAPI key. Go to serpapi.com, create a free account, and copy your API key from the dashboard. The free tier allows 100 searches per month—enough for testing.
- Set up your Python script. Create a new file, e.g.,
google_jobs.py, and start with the following imports:import requests import pandas as pd
- Define your search parameters. You’ll need your API key, a search query (e.g., “data analyst”), and optionally a location. Store these in variables:
API_KEY = "your_api_key_here" query = "data analyst" location = "New York"
- Construct the API endpoint. SerpAPI’s Google Jobs endpoint uses the following base URL:
url = "https://serpapi.com/search.json"
Then build the parameters dictionary:
params = { "engine": "google_jobs", "q": query, "location": location, "api_key": API_KEY, "hl": "en", "gl": "us" } - Make the API call. Use the
requestslibrary to send a GET request:response = requests.get(url, params=params) data = response.json()
Check the response status—if
response.status_codeis not 200, print the error and exit. - Extract job results. The JSON response contains a key
"jobs_results"which is a list of job objects. Each object has fields liketitle,company_name,location,description, andapply_link. Example:jobs = data.get("jobs_results", []) print(f"Found {len(jobs)} jobs") - Parse the data into a DataFrame. Create a list of dictionaries, then convert to pandas DataFrame:
job_list = [] for job in jobs: job_list.append({ "title": job.get("title"), "company": job.get("company_name"), "location": job.get("location"), "description": job.get("description"), "apply_link": job.get("apply_link") }) df = pd.DataFrame(job_list) - Handle pagination (optional). To get more than the default 10 results, use the
startparameter to request subsequent pages. Loop through pages until you have enough results or hit the API limit:for start in range(0, 50, 10): params["start"] = start response = requests.get(url, params=params) data = response.json() jobs = data.get("jobs_results", []) if not jobs: break # process jobs... - Save to CSV. Finally, export the DataFrame to a CSV file:
df.to_csv("google_jobs.csv", index=False)This creates a file in your working directory that you can open in Excel or any spreadsheet application.
Common Mistakes
- Using the wrong engine parameter. If you forget
"engine": "google_jobs", SerpAPI will return regular Google search results, not job listings. - Not handling API rate limits. The free tier is limited. If you exceed it, you’ll get a 429 error. Always check the response status and include error handling.
- Ignoring the
apply_linkfield. Some job listings may not have a direct apply link; always check if the key exists before using it. - Assuming all jobs have the same fields. Some listings might lack a description or location. Use
.get()with defaults to avoid KeyErrors. - Not respecting Google’s terms of service. While using an API is legitimate, scraping directly from Google’s HTML is against their ToS. Always use the official API.
Tips
- Use a delay between requests to avoid hitting rate limits—add
time.sleep(1)in loops. - Filter results by date using the
date_postedparameter (e.g.,"today"or"3days") to get fresh listings. - Store your API key in an environment variable rather than hardcoding it, to keep it secure.
- Explore the full JSON response—SerpAPI returns additional metadata like estimated salaries and job types; you can extract those too.
- Use the
chipsparameter to filter by job type (full-time, part-time) or experience level, similar to the filters on Google Jobs.
Troubleshooting
- Getting a 403 or 401 error: Your API key is invalid or expired. Double-check the key in your SerpAPI dashboard.
- Empty
jobs_results: This can happen if the query is too specific or the location is not recognized. Try a broader query or remove the location parameter. - Response is HTML instead of JSON: You might be hitting the wrong endpoint or missing the
engineparameter. Verify your URL is exactlyhttps://serpapi.com/search.json. - Rate limit exceeded: Wait for the next month’s quota or upgrade your plan. You can also add a delay between requests to stay under limits.
- Unicode or encoding issues in the CSV: When saving, specify
encoding='utf-8'into_csv()to handle special characters in job descriptions.
Frequently Asked Questions
Is scraping Google Jobs legal?
Using an official API like SerpAPI is generally acceptable, as it provides a licensed way to access search results. Directly scraping Google’s HTML pages violates their Terms of Service and may lead to IP blocks.
How many results can I get?
SerpAPI returns up to 10 results per page by default. You can paginate using the start parameter, but the free tier limits total queries. Paid plans allow more.
Can I get job descriptions?
Yes, the description field in each job result contains the full job description text, which you can save to your CSV.
Do I need to use a proxy?
No, because you’re using a third-party API, you don’t need proxies. The API handles the requests from their servers, so your IP is safe.
How do I filter by location?
Use the location parameter in your API request. You can specify a city, state, or even a country. For more precise filtering, use the chips parameter with location-specific values.
Can I schedule this script to run daily?
Absolutely. You can use cron (on Linux/macOS) or Task Scheduler (on Windows) to run your Python script at regular intervals. Just ensure your API quota is sufficient.
Related
Method: the standard formula for this calculation, rounded as each result states. See how the calculators work or report an error.