ZipRecruiter

Log In

1

Web Crawling Engineer Jobs (NOW HIRING)

CurieTech AI

Software Engineer, Data Acquisition

Sunnyvale, CA · On-site

$133K - $160K/yr

Responsibilities : • Lead and manage engineering projects focused on data acquisition, including web crawling, data ingestion, and API development. • Collaborate with cross-functional teams (AI ...

GetReal Security LLC

Senior Agent Engineer

Austin, TX · On-site

$103K - $142K/yr

The Role We are hiring a Senior Agent Engineer to own both the implementation and strategic ... Experience building web crawling and data acquisition pipelines -- bespoke crawlers, headless ...

Youlify

Senior Software Engineer

Palo Alto, CA · On-site

$144K - $190K/yr

They are seeking a highly skilled software engineer to help build and scale their core platform ... with web crawling and data analytics pipelines Company : Youlify offers AI solutions for medical ...

Saviance

Senior Backend AI Engineer

... resilient web crawling and automation pipelines for provider credentialing data collection • ... Required : • Strong backend engineering experience in Python, including production-grade services ...

Pear

Founding Engineer - Scend

New York, NY · On-site

Founding Engineer - Scend Who We Are Scend is on a mission to modernize M&A and bring efficiency to ... Web Crawling: Develop high-scale systems for crawling sites and docs from the web * Infrastructure:

Praxis, Inc.

Founding Engineer - Scend

Manhattan, NY · On-site

Founding Engineer - Scend Who We Are Scend is on a mission to modernize M&A and bring efficiency to ... Web Crawling: Develop high-scale systems for crawling sites and docs from the web * Infrastructure:

Visa Hunt

This role is ideal for engineers who love building robust distributed systems, but who also want to ... Build and operate large-scale data ingestion systems for pre-training, including web crawling ...

Numerator

Dev Ops Engineer

Chicago, IL

$54.50 - $74.50/hr

Numerator is looking for an experienced, talented and quick-thinking DevOps Kubernetes Engineer to ... web crawling and enhanced Deep Learning algorithms to deliver an unmatched view of the consumer ...

Numerator

Dev Ops Engineer

Chicago, IL · On-site

$54.50 - $74.50/hr

Numerator is looking for an experienced, talented and quick-thinking DevOps Kubernetes Engineer to ... web crawling and enhanced Deep Learning algorithms to deliver an unmatched view of the consumer ...

previous pagenext page

Showing results 21-40

All JobsWeb Crawling Engineer Jobs

Web Crawling Engineer information

See salary details

$11

$59

$86

How much do web crawling engineer jobs pay per hour?

As of Sep 9, 2026, the average hourly pay for web crawling engineer in the United States is $59.01, according to ZipRecruiter salary data. Most workers in this role earn between $51.20 and $66.83 per hour, depending on experience, location, and employer.

What is a web crawling engineer?

A Web Crawling Engineer is a professional who designs, develops, and maintains automated systems (web crawlers or spiders) that systematically browse the internet to collect data from websites. Their responsibilities include building efficient crawling algorithms, handling web data extraction, managing large-scale data storage, and ensuring compliance with website policies and legal guidelines. Web Crawling Engineers commonly work in fields such as search engines, market research, price comparison, and data analytics, where gathering large volumes of online information is essential.

What are the key skills and qualifications needed to thrive as a web crawling engineer?

To thrive as a Web Crawling Engineer, you need strong programming skills in languages like Python or Java, a solid understanding of web protocols, and experience with data extraction and parsing. Proficiency with technical tools such as Scrapy, Selenium, BeautifulSoup, and knowledge of APIs and databases is commonly required. Analytical thinking, problem-solving, and attention to detail are vital soft skills for overcoming crawling challenges and ensuring data integrity. These skills and qualities are important to efficiently gather accurate data from diverse web sources while navigating technical and ethical considerations.

What are some common challenges faced by web crawling engineers when dealing with large-scale data extraction projects?

Web Crawling Engineers often encounter challenges such as handling websites with dynamic content, managing rate limits and CAPTCHAs, and ensuring compliance with legal and ethical guidelines. Additionally, efficiently processing and storing vast amounts of data while maintaining crawler performance and avoiding IP bans requires robust infrastructure and careful planning. Collaboration with data analysts and software engineers is common to ensure the extracted data meets project requirements and integrates smoothly with downstream systems.

What is the difference between Web Crawling Engineer vs Data Engineer?

AspectWeb Crawling EngineerData Engineer
Required CredentialsBachelor's in Computer Science, relevant experience in web scraping and crawlingBachelor's or higher in Computer Science, Data Science, or related fields
Work EnvironmentFocus on developing and maintaining web crawlers, data extraction toolsDesigning data pipelines, managing large datasets, optimizing data flow
Employer & Industry UsageTech companies, search engines, data providersTech firms, finance, healthcare, any data-driven industry
Common Search & Comparison IntentUnderstanding roles in web data collectionUnderstanding data infrastructure roles

While both roles involve working with data, a Web Crawling Engineer specializes in developing tools to extract data from websites, focusing on crawling and scraping techniques. A Data Engineer builds and maintains data pipelines and infrastructure to process and store large datasets. The roles often overlap in data collection and processing but differ in scope and focus.

What are popular job titles related to Web Crawling Engineer jobs?

For Web Crawling Engineer jobs, the most frequently searched job titles are:

Infographic showing various Web Crawling Engineer job openings in the United States as of September 2026, with employment types broken down into 1% Internship, 92% Full Time, 3% Part Time, and 4% Contract. Highlights an 85% Physical, 4% Hybrid, and 11% Remote job distribution, with an average salary of $122,736 per year, or $59 per hour.

Software Engineer, Data Acquisition

Sunnyvale, CA • On-site

CurieTech AI
See more jobs at CurieTech AIexternal

$133K - $160K/yr

Full-time

Re-posted 5 days ago

Job description

Job Summary:
CurieTech AI is a silicon valley based startup that is building cutting edge AI software to help software development teams be more productive. The Data Acquisition team is responsible for all aspects of collecting data to train models, focusing on writing deep crawlers and developing data pipelines.
Responsibilities:
• Lead and manage engineering projects focused on data acquisition, including web crawling, data ingestion, and API development.
• Collaborate with cross-functional teams (AI/ML, Infrastructure).
• Design and implement highly scalable distributed systems to efficiently handle unstructured data.
• Develop algorithms for data indexing and advanced search functionalities.
• Build and maintain backend services for data storage, utilizing key-value databases and synchronization methods.
• Deploy solutions within a Kubernetes Infrastructure-as-Code environment, ensuring system reliability through routine checks.
• Conduct experiments on data to analyze and enhance system performance.
Qualifications:
Required:
• BS/MS/PhD in Computer Science or a related field.
• 4+ years of industry experience in software development, with a focus on data systems and APIs.
• Strong expertise in large stateful distributed systems and data processing.
• Proficiency in Kubernetes and Infrastructure-as-Code principles.
Preferred:
• Familiarity with large web crawlers is a plus.
Company:
CurieTech AI is a developer of AI tech transforming enterprise workflows, automation, and data-driven decision-making processes. Founded in 2023, the company is headquartered in California, USA, with a team of 11-50 employees. The company is currently Early Stage.

Apply