ZipRecruiter

Log In

1

Web Crawler Engineer Jobs (NOW HIRING)

Exa

Software Engineer, Web Crawling

San Francisco, CA · On-site

$180K - $350K/yr

As a Web Crawler engineer, you'd be responsible for crawling the entire web. Basically build Google-scale crawling! Who You Are * You have extensive experience building and scaling web crawlers, or ...

StackandCo

\n \n \n \n \n Senior Fullstack Developer needed for an impressive Irish tech company building a world leading bot\/web crawler detection product. \n \n \n \n \n \n \n We need engineers who are ...

Exa

Research Engineer, Generalist

San Francisco, CA · On-site

$125 - $150/hr

Generalist Research Engineers work across our search and retrieval stack: crawling, parsing, ML ... Make our web crawler 3x faster This is an in‑person opportunity in San Francisco. We're happy ...

Olostep Technologies Inc.

$200 - $250/hr

Engineered from the ground up for an agentic world. Why this role matters This is a high ... Design and scale a distributed web crawler optimized for freshness, coverage, and extracting agent ...

StackandCo

\n \n \n \n \n Senior Fullstack Developer needed for an impressive Irish tech company building a world leading bot\/web crawler detection product. \n \n \n \n \n \n \n We need engineers who are ...

NTENT

Java Developer (Remote Permanent Opportunity)

$52.25 - $67.50/hr

Ingestion Pipeline: process data that comes from our web crawler which discovers and fetches ... Solid experience with Java programming (we also use Spring, Spring Webflux, Reactor, Netty) * Multi ...

NTENT

Java Developer (Remote Permanent Opportunity)

$52.25 - $67.50/hr

Ingestion Pipeline: process data that comes from our web crawler which discovers and fetches ... Solid experience with Java programming (we also use Spring, Spring Webflux, Reactor, Netty) * Multi ...

Exa

Research Engineer, Generalist

San Francisco, CA · On-site

$180K - $350K/yr

Generalist Research Engineers work across our search and retrieval stack: crawling, parsing, ML ... Make our web crawler 3x faster Logistics * Location: This is an in-person opportunity in San ...

NTENT

Senior Java Engineer

$58.50 - $74.75/hr

Senior Java Engineer Position: Full time Location: Toronto, Ontario (Initially Remote) About Us: We ... Ingestion Pipeline: process data that comes from our web crawler which discovers and fetches ...

NTENT

Senior Java Engineer

$58.50 - $74.75/hr

Senior Java Engineer Position: Full time Location: Toronto, Ontario (Initially Remote) About Us: We ... Ingestion Pipeline: process data that comes from our web crawler which discovers and fetches ...

Thinking Machines Lab Inc.

$250/hr

About the Role We're looking for an engineer to join us and contribute to data infrastructure. You ... Have experience building a web crawler. * Have extensive experience understanding and scaling ...

Hanzo

Remote US or UK Crawl Engineer

Chicago, IL · On-site +1

We're seeking a Senior Crawl Engineer to be part of our professional services team. This role ... the modern web with a sophisticated crawler at its core. This job is at the heart of crawler ...

Hanzo

Remote US or UK Crawl Engineer

Chicago, IL · Remote

We're seeking a Senior Crawl Engineer to be part of our professional services team. This role ... the modern web with a sophisticated crawler at its core. This job is at the heart of crawler ...

Hanzo

Remote US or UK Crawl Engineer

Chicago, IL · On-site +1

We're seeking a Senior Crawl Engineer to be part of our professional services team. This role ... the modern web with a sophisticated crawler at its core. This job is at the heart of crawler ...

Hanzo

Remote US or UK Crawl Engineer

Chicago, IL · On-site +1

We're seeking a Senior Crawl Engineer to be part of our professional services team. This role ... the modern web with a sophisticated crawler at its core. This job is at the heart of crawler ...

next page

Showing results 1-20

All JobsBusiness JobsMarketing JobsWeb Crawler Engineer Jobs

Web Crawler Engineer information

See salary details

$11

$59

$86

How much do web crawler engineer jobs pay per hour?

As of Sep 9, 2026, the average hourly pay for web crawler engineer in the United States is $59.01, according to ZipRecruiter salary data. Most workers in this role earn between $51.20 and $66.83 per hour, depending on experience, location, and employer.

What is the job of a Web Crawler Engineer?

A Web Crawler Engineer designs and develops automated programs that systematically browse and index web content for search engines or data analysis. They typically work with programming languages like Python or Java, utilize web scraping tools, and ensure efficient, respectful crawling while managing large-scale data collection processes.

What is the salary of a Web Crawler Engineer?

The salary of a Web Crawler Engineer typically ranges from $70,000 to $130,000 annually, depending on experience, location, and company size. Skilled professionals with expertise in web scraping, data mining, and programming languages like Python or Java are often in higher demand and may earn higher salaries.

What are popular job titles related to Web Crawler Engineer jobs?

For Web Crawler Engineer jobs, the most frequently searched job titles are:

Infographic showing various Web Crawler Engineer job openings in the United States as of September 2026, with employment types broken down into 1% Internship, 92% Full Time, 3% Part Time, and 4% Contract. Highlights an 85% Physical, 4% Hybrid, and 11% Remote job distribution, with an average salary of $122,736 per year, or $59 per hour.

Software Engineer, Web Crawling

San Francisco, CA • On-site

Exa
501 - 1,000 employees
Learn more about Exaexternal

$180K - $350K/yr

Full-time

Medical, Dental, Vision

Re-posted 29 days ago

Job description

Exa is an applied AI lab building a search engine unlike the world has ever seen. We build massive-scale infra to crawl the entire web, train state-of-the-art embedding models to process it, and design super high performant vector databases to retrieve over it. We now power search for Cursor, Cognition, HubSpot, and over 400,000 developers and have raised $350m from Lightspeed, Benchmark, and a16z.
Our ultimate goal is to build perfect search over all the world's information, far beyond Google. If you want to build massive-scale ML systems that will define the way the new AI world consumes information, this is the place for you.
As a Web Crawler engineer, you'd be responsible for crawling the entire web. Basically build Google-scale crawling!
Who You Are
  • You have extensive experience building and scaling web crawlers, or would be excited to ramp up very quickly
  • You have experience with some high performance language (C++, Rust, etc.)
  • You are familiar with TypeScript, Playwright, modern web design, CDP (Chrome DevTools Protocol)
  • You're comfortable optimizing a system to an exceptional degree
  • You care about the problem of finding high quality knowledge and recognize how important this is for the world

What You Could Do
  • Build a distributed crawler that can handle 100M+ pages per day
  • Optimize crawl politeness and rate limiting across thousands of domains
  • Design systems to detect and handle dynamic content, JavaScript rendering, and anti-bot measures
  • Create intelligent crawl scheduling and prioritization algorithms for maximum coverage efficiency

Logistics
  • Location: This is an in-person opportunity in San Francisco.
  • Visas: We're happy to sponsor international candidates (e.g., STEM OPT, OPT, H1B, O1, E3). While we cannot guarantee your visa, we have historically been successful in sponsoring candidates from all over the world. If you receive an offer, our team will work hard to get you a visa.
  • Benefits: We offer premium healthcare benefits (medical, dental, vision), fertility benefits, 16 weeks of fully paid parental leave for all new parents, and a monthly wellness stipend to all of our employees.

Exa is an equal opportunity employer. All qualified applicants will receive consideration for employment without regard to race, creed, color, religion, sex, sexual orientation, gender identity or expression, national origin, disability, age, veteran status, marital status, pregnancy or related conditions, criminal histories consistent with applicable law, or any other basis protected by applicable law.

Apply