Build a live salary dataset from job postings across 6 boards

Which salary fields does a scraped job posting include?

Five pay fields sit on each row under the same names, whichever board produced it. salary_min and salary_max hold the numeric range as the posting states it, salary_currency holds the currency code such as USD or GBP, and salary_interval records the original pay period as yearly, monthly or hourly. The fifth field, salary_source, is the one to read first: direct_data means the board exposed the figure as structured data, while description means the number was extracted from the posting text. Structured values are the safer input for a median; parsed values catch postings that only mention pay in a paragraph, but they deserve a spot check before they enter a benchmark. Around the pay fields sit the segmentation columns that explain variance: job_type, is_remote, location, job_level on LinkedIn, and company_num_employees, company_revenue, company_industry and company_rating where the board publishes an employer profile. The sample Indeed row in the listing shows the shape: 120,000 to 185,000 USD yearly from direct_data, with a 3.9 employer rating and 21,432 reviews on the same row.

• salary_min and salary_max: numeric bounds exactly as the posting states them, null when no pay is published
• salary_currency: the currency code, so UK, Indian and Gulf rows never blend into USD averages by accident
• salary_interval: yearly, monthly or hourly as published, which tells you which rows need converting
• salary_source: direct_data from the board or description parsed from text, your first quality filter
• Segmentation on the same row: job_type, is_remote, job_level, employer size, revenue, industry and rating
• Descriptions ship as Markdown or HTML, so you can re-parse pay language yourself when a posting is ambiguous

How do you normalize hourly, monthly and yearly salaries into one column?

Postings quote pay in whatever unit the employer typed, and an hourly contract sitting next to an annual salaried range breaks any median you compute. The scraper has one input for this, enforceAnnualSalary. When it is on, hourly figures are multiplied by 2,080 (40 hours across 52 weeks) and monthly figures by 12, so a 45 per hour contract lands at 93,600 and a 7,500 per month role at 90,000, in the same column as a posted 95,000 salary. Leave it off when hourly and salaried markets should be studied separately, for example nursing or trades roles where hourly is the native unit; salary_interval then tells your own code which rows to convert. Normalization never converts currency. A run with countryIndeed set to uk returns GBP rows and a Naukri sweep returns Indian postings, so group by salary_currency before ranking anything, or apply your own exchange rates at analysis time. Keep salary_source in view as well: a description-parsed hourly number multiplied by 2,080 amplifies any parsing error by the same factor.

• enforceAnnualSalary: hourly x 2,080, monthly x 12, yearly left unchanged
• 45 per hour becomes 93,600 a year; 7,500 per month becomes 90,000
• Off by default, so hourly-native markets keep their original unit unless you ask
• Currency is never converted: group by salary_currency or apply your own rates
• Parsed hourly values are the rows to spot check, because the 2,080 multiplier scales any error

Which job boards publish salary data, and what does each one add?

Salary coverage depends on the board and on the employer, so the practical rule is to sweep every board that serves your market and let deduplication merge the overlaps; per-board detail pages are listed at /scrape. Indeed is the most reliable source and carries the richest employer profile, including employee count, revenue band and company country, with countryIndeed choosing the national site. Glassdoor uses the same countryIndeed setting for country targeting and is fetched through a real browser, so its rows arrive after Indeed's. LinkedIn adds job_level for seniority cuts but rate-limits at roughly 100 results per IP, so keep residential proxies on and enable linkedinFetchDescription when you need description-parsed pay. ZipRecruiter covers the US and Canada only. Naukri returns experience_range, skills and company_rating, which makes it the board to use for pay-by-experience curves in India, while Bayt covers the Middle East and accepts only the search term, with no hours or job type filters. Google Jobs and BDJobs stay selectable but currently return no rows.

• Indeed: most reliable board, employer size, revenue and country on the row, countryIndeed picks the national site
• Glassdoor: same countryIndeed setting as Indeed, browser-fetched, rows land in 1 to 3 minutes
• LinkedIn: job_level for seniority, about 100 results per IP, residential proxy recommended
• ZipRecruiter: US and Canada postings only
• Naukri: experience_range, skills and company_rating for India pay curves
• Bayt: Middle East coverage, search term only, no hoursOld or jobType filters

What does a salary benchmarking workflow look like with scraped postings?

A repeatable benchmark is a saved input plus a notebook. Start with searchTerms holding up to 5 related titles, so one run covers the whole job family and every row carries matched_search_term to show which query found it. Set location, pick the boards for your market, raise maxResults toward its cap of 100 per board per term, and set hoursOld to 168 so the sample reflects the current week. Deduplication across boards happens before rows are written, so a job listed on three boards contributes one observation. Export CSV or Excel from the Output tab, or pull the dataset by API into pandas. In the notebook, drop rows where salary_min is null, split by salary_currency, then compute the median, p25 and p75 of the midpoint by title, location and company_employees_label. A single run tops out at 4,000 rows (100 results x 8 boards x 5 terms), so paginate with offset or split by city when you need more. Schedule the saved input weekly and append each run keyed on id and scraped_at for a time series.

• Up to 5 searchTerms per run, each row tagged with matched_search_term
• maxResults up to 100 per board per term; hoursOld 168 keeps the sample to the current week
• One observation per job: cross-board duplicates are merged before rows land
• Export CSV, Excel or JSON, or fetch the rows over the Apify API
• Compute median, p25 and p75 of the midpoint by title, location and employer size
• Append weekly runs keyed on id and scraped_at for a trend line

What filter and coverage limits affect a salary sample?

Two board-side restrictions shape sample design. Indeed cannot combine hoursOld with jobType, isRemote or easyApply, and LinkedIn cannot combine hoursOld with easyApply, so a recency-filtered pull of remote-only Indeed roles takes two steps: pull by hours, then filter is_remote in your own code. Bayt ignores everything except the search term. Expect fewer rows than maxResults when a board has fewer matches or duplicates were removed; that is coverage working, not a failure. Speed also differs by board: Indeed and LinkedIn rows land 5 to 20 seconds after start, while Glassdoor, ZipRecruiter, Bayt and Naukri go through a real browser and take 1 to 3 minutes, streaming into the dataset as each board finishes. For the sample itself, treat a posted range as an advertised band, not a paid salary: it skews toward roles where pay transparency laws or competitive markets force disclosure. Reporting the share of rows with any salary alongside the median keeps the benchmark honest, and salary_source lets you publish a structured-only figure next to the broader one.

• Indeed: hoursOld cannot pair with jobType, isRemote or easyApply; filter those columns after the pull
• LinkedIn: hoursOld cannot pair with easyApply
• Bayt: search term only, so apply recency and type filters in your notebook
• Indeed and LinkedIn rows arrive in 5 to 20 seconds; browser-fetched boards take 1 to 3 minutes
• Report the share of rows with published pay next to every median

Questions, answered

Keep reading