TL;DR
I built the engine behind two job boards, a LATAM board on Tiene que haber algo más and an English-language one on Build With Talent, as a classic ETL pipeline: extract postings straight from company career pages, transform them into structured facts, and load only what passes a written policy.
The product spec covers what the boards are and who they are for. This is the other half: how the machine is actually built, what broke, and the rules I ended up writing down so it would not break the same way twice.
Today the pipeline reads 9,688 active sources (9,376 companies, 233 agencies and 78 job portals) through 42 connectors, keeps about 3,800 live postings on the LATAM board and 77,000 in the pool that feeds the paid matching product, and is guarded by 1,447 tests. When I wrote the spec it was roughly 3,200 sources and 7 applicant-tracking systems.
The short version of what I learned: an ETL for messy public data is not a scraping problem. It is a trust problem, and almost every hard decision was about where a piece of data came from and who gets to overwrite it.
The situation
Job postings in Latin America are spread across thousands of company career pages, a dozen applicant-tracking systems (ATS), regional portals with their own quirks, and recruiting agencies that repost the same role three times. Aggregators solve this with volume. The bet here was the opposite: fewer jobs, every one of them real, still open and actually available to someone in the region.
That bet only works if the pipeline can answer, for every single posting, three questions with evidence: where is this job, is it still open, and who posted it. Most of the engineering is in making those answers trustworthy at the scale of thousands of sources a day, run by one person on one laptop.
How the product works for a candidate
From the outside it is a simple page. You search by role or area, pick a location, and get a list of postings, newest first. Each card shows the company, the area, the country, the modality (remote, hybrid or on-site), the experience range it asks for and when it was published. Anything from the last three days carries a “New” badge, and there is a filter for just those.
The sidebar shows how many open postings exist in each area right now: engineering, sales, operations, accounting, customer success, marketing, data and about twenty more. Every “View job” button goes straight to the company’s own application page, never to a middleman.
What the candidate does not see is the rulebook behind that list:
- Who it is for. A posting enters the LATAM board if it is located in any Latin American country or city, in any modality, or if it is genuinely global remote with no country restriction. US-only, Europe-only and “remote, but you must live in Texas” stay out. When the location cannot be verified, it stays out too.
- Who posted it. Only direct employers. Agencies and portals still feed the matching product, but never the public board.
- Whether it is real. CV databases, job fairs, “future openings” and talent pools are filtered out.
- Whether it is still open. Every published posting is checked against the company’s own system every morning. If it closed, it comes down.
- Whether it is fresh. Nothing older than 30 days, using the date the company actually published it, not the date we first saw it.
The same engine feeds a second, English-language board, Build With Talent, with no geographic filter, and Radar, the paid matching product. The only thing that changes between them is the policy.
The architecture: six layers, one rule
The pipeline is split into layers with a strict contract between them:
| Layer | Job | When it fails, you see |
|---|---|---|
| 0 · Sources | Which companies to read | A blind source that never yields |
| 1 · Capture | Bring back what the ATS says, uninterpreted | Empty or partial snapshots |
| 2 · Parse | Turn text into facts: location, modality, area, seniority, skills, language, salary | Wrong location or modality |
| 3 · Classify | Decide if a job belongs on a board: READY / REVIEW / REJECTED | Off-policy jobs on the board |
| 4 · Publish | Sync to the live board | Board out of sync with the database |
| 5 · Lifecycle | Close what has closed | Dead jobs still visible |
The golden rule: each layer does only its own job. Capture never interprets. Classification never touches the network. It sounds pedantic until you are debugging at 11pm and need to know whether a wrong location came from the ATS, the parser or the policy. With hard boundaries, there is only one place it can be.
A full nightly run takes two to three hours: 10 parallel workers, one per source, because the source is the natural unit that shares no state and where network latency dominates. A separate cleanup job runs every morning to close what disappeared overnight.
Decision one: keep the raw payload, forever
Every posting’s original ATS response is stored untouched next to the parsed version. This turned out to be the single most valuable decision in the project.
Because the raw payload is there, every new rule can be applied retroactively without re-scraping. When I realized Workday had been sending application deadlines on 96% of its postings, the data had been sitting in storage for months. When the date parser learned to read “Hace 14 días” (posted 14 days ago), 4,323 postings got their real publication date recalculated in one pass. When a policy changed, I re-ran the transform over history instead of waiting weeks for new data.
Storage is cheap. Not being able to answer “what did the source actually say?” is expensive.
Decision two: every field knows where it came from
The most common source of bugs was inferring from free text something the ATS already declared as structured data. “Porto Alegre, Brasil” became Portugal. “Lima, OH” became Peru. A posting in “Valparaiso, IN” was read as Chile and auto-approved. “AR” became Arkansas when the ATS meant Argentina.
So each important field carries its provenance, and they are ranked in what I call provenance ladders:
- Modality: ATS literal value > description text > fallback
- Date: ATS published > ATS updated > relative date on the card (“3 days ago”) > first time we saw it
- Location: ATS structured > catalog match > empty
- Seniority: ATS structured > title (in Spanish, English or Portuguese) > years in the description > management signals > derived from role
A lower rung can never overwrite a higher one. And “empty” is a valid answer. If a posting does not state its modality, the field stays empty rather than defaulting to on-site. On October 12th I re-processed the pool and found 7,000 postings marked “on-site” that had never said so. The default had been quietly inventing data.
The ladders also make quality measurable. Every run report shows what share of each field came from which rung, so a parser regression or an ATS changing its format shows up as a shift in a table, not as a complaint from a user two weeks later.
Decision three: a silent empty list is a lie
Early on, a connector that hit a network error returned an empty list. Downstream, extract read that as “this company has no openings” and the lifecycle read it as “everything closed.” One bad afternoon with Workable produced zero postings across 447 sources and 140 false closures.
The contract now is explicit: an ATS error raises an exception; an empty list means the ATS answered 200 and genuinely had nothing. On top of that, the lifecycle has a brake: a source that had five or more postings and suddenly returns none is not closed without a second signal.
The same family of lessons showed up in testing detectors. One ATS endpoint returns 200 with an empty list for any made-up company slug, which generated 175 false positives when I was discovering new career sites. Every new detector is now first tested against a slug that does not exist.
Decision four: the spreadsheet is the backend
There is no application server. The board widget on WordPress reads a Google Sheet directly. Publishing means writing a row.
It is the right call for one operator with a laptop: zero hosting, zero deploys, and the board can be inspected and vetoed by hand. It also produced some of the most instructive bugs:
- Writing cell by cell hit Google’s rate limit, the marking failed, and 10,000 rows were re-inserted on every run. Click counts inflated twentyfold before anyone noticed. Always batch.
- Someone dragged “Approved” down a column, header included. The code could not find the column, “migrated” it by inserting an empty column, and every row shifted. Caught mid-run, before publishing. A sheet a human edits is read and written by column name, never by position.
Since October 12th the board publishes itself. Anything that passes the policy goes live at the end of the run, and the only manual control is a “Remove” column. Getting there required being able to trust every layer upstream, which is the whole point of the previous three decisions.
Decision five: knowing when a job is gone
A board full of closed jobs is worse than a small board. The lifecycle checks every published posting every morning, and since late September it also records a close date for everything a source stops listing for two good runs in a row, including rejected postings. That turned the system into a dataset about how long jobs actually stay open in the region.
Two lessons from making it fast and correct:
- Don’t fetch what you won’t use. The lifecycle was re-downloading every job description just to compare URLs. Workday alone made up to 400 requests per source to throw everything away. URL-only fast paths plus parallel workers took the check from 47 to 85 minutes down to 8.
- A paginated listing is not a census. A portal with 31,000 postings shown 20 per page would make the lifecycle close almost everything it never reached. For those sources, the connector deliberately refuses to answer, so the lifecycle treats them as unverifiable and lets postings expire on the date the portal itself declares.
Scaling sources without drowning
Going from 3,200 to 9,700 sources was mostly a supply chain problem, not a code problem.
New sources enter switched off and are released in batches, so a bad batch never floods the board. Sources are classified by type, company, agency, portal or manual post, and only direct employers reach the public board. Since October 13th a source whose postings say things like “for an important company” or “on behalf of our client” is reclassified as an agency automatically, every night.
A protection rule that sent any source with more than 30 jobs per run to manual review looked sensible against suspicious aggregators. When real large employers came in, it put 3,585 postings in review and re-flagged them on every run. The queue never went down. Grouping by source turned 3,632 decisions into 59, and every rule that sends something to a human now has a permanent “already reviewed” exit.
How it stays honest
- One confirmed bug, one new test. No exceptions. That is how the suite got to 1,447.
- 28 integrity checks per run, compared against the previous run, plus an email with what was published in the last 24 hours and every silent failure: an ATS that went down, sources that broke, fields that started coming back empty.
- Scheduled tasks are checked for their last result, not their existence. The morning cleanup was configured not to start on battery power. On a laptop, that meant it sometimes just did not run, without a log line. The English board had accumulated nearly 10,000 dead postings by the time I found it.
- A weekly read-only analysis goes through the run logs and reports bugs, trends and improvements.
What it produced
- Two boards fed by one engine, publishing automatically, with about 3,800 live LATAM postings at any time.
- 9,688 active sources across 42 connectors, from the big ATS APIs to regional portals, the UN careers site and company pages with no ATS at all.
- A pool of 77,000 open postings that powers Radar, the paid matching product built on top of it.
- A data asset: 670,000 historical postings with real publication and close dates, the base for the market reports we publish.
- 75 numbered migrations, 1,447 tests and a written architecture and lessons file that a new contributor can read in an hour.
Lessons learned
The ATS almost always knows more than the text. Before parsing a sentence, check whether the source already declares the value. Most of my location bugs were me being clever with text the ATS had already resolved.
Ask whose number it is. The parser read “we have been in the market for 130 years” as an internship requiring 130 years of experience. That one bad number fed the seniority field and produced 1,161 “Principal” roles, about 1,156 of them false. Any number extracted from free text needs to know who it describes.
Measure on the universe you keep. A report said 74% of postings had no location. It was counting 57,000 rejected postings that never get a full parse. The real number was 4%. Quality metrics only count what survives.
Default to deny. If a posting’s eligibility cannot be verified, it stays off the board. Losing a good job is cheaper than publishing one a candidate cannot apply to.
A policy change is also a migration. When the board moved to LATAM-or-global only, the rule was applied to the queue but not to what was already live. Days later, 976 off-policy postings were still published. Every rule change now re-validates the board.
Visibility is not validity. Rejecting on-site jobs to hide them from the board also removed them from the matching pool. “Should the public see this?” and “is this a real, open job?” are different questions and need different fields.
Volume changes the product, not just its size. Publishing large employers took the board from 4,200 to 10,500 postings and shifted it from mostly remote tech to mostly on-site operations. That was a conscious decision, but it was a product decision, not a data one.
The boring infrastructure is the product. Nobody will ever see the provenance ladders or the lifecycle brake. They are the reason a candidate can trust that the job in front of them is real, open and meant for them.