ZipRecruiter

Log In

1

Web Crawling Engineer Jobs (NOW HIRING)

Sonoma Consulting Inc.

Hadoop Developer

Hoboken, NJ · On-site

... and web crawling techniques is a plus Key Skills Apache NiFi, Hive programming, Hadoop and Spark ecosystems of open-source tools, NoSQL databases, Json and web services programing Qualifications ...

Polluxa, Inc.

WI · On-site

$150 - $200/hr

... web crawling, and the ingestion of millions of certificates. Censys was founded by security ... Role Summary The Enterprise Solutions Engineer serves as Censys's primary technical resource for ...

New

Youlify

Senior Software Engineer

Palo Alto, CA · On-site

$144K - $189K/yr

Experience with web crawling and data analytics pipelines will be a plus. Why Join Us ... Top-tier team- Work alongside world-class engineers, AI researchers, and healthcare experts.

Fortinet

Staff Software Development Engineer

Sunnyvale, CA · On-site

$150K - $215K/yr

Fortinet is calling for an experienced Staff Software Development Engineer who can think outside ... Experience with popular security detecting, penetration testing, web crawling tools is a plus, like ...

Fortinet

Staff Software Development Engineer

Sunnyvale, CA · On-site

$150K - $215K/yr

Fortinet is calling for an experienced Staff Software Development Engineer who can think outside ... Experience with popular security detecting, penetration testing, web crawling tools is a plus, like ...

ADP, Inc.

Software Engineer

San Francisco, CA · On-site

$100 - $125/hr

Software Engineer Full Time San Francisco, CA, US 2 days ago Requisition ID: 1037 Salary Range ... web crawling and playback operations. Build and improve foundational systems, services, and ...

Censys

Staff Scanning Engineer

... web crawling, and the ingestion of millions of certificates. Censys was founded by security ... This role sits at the intersection of backend engineering, distributed systems, and applied ...

Articul8

Machine Learning Engineer

Dublin, CA · On-site

Own and lead engineering projects in the area of data acquisition, including web crawling, data ingestion, and processing. * Collaborate with our Applied Research, Technology, and Architecture teams ...

Articul8

Machine Learning Engineer

Dublin, CA · On-site

$100 - $125/hr

Own and lead engineering projects in the area of data acquisition, including web crawling, data ingestion, and processing. * Collaborate with our Applied Research, Technology, and Architecture teams ...

Worky

Web Developer - Marketing

Dunwoody, GA · On-site

$100 - $125/hr

Familiarity with web QA tools, browser developer tools, crawling tools, accessibility testing tools or performance testing tools. * Experience working in regulated, highly governed or large ...

State Farm

Web Developer - Marketing

Richardson, TX · On-site

Familiarity with web QA tools, browser developer tools, crawling tools, accessibility testing tools or performance testing tools. * Experience working in regulated, highly governed or large ...

State Farm

Web Developer - Marketing

Dunwoody, GA · On-site

Familiarity with web QA tools, browser developer tools, crawling tools, accessibility testing tools or performance testing tools. * Experience working in regulated, highly governed or large ...

State Farm

Web Developer - Marketing

Tempe, AZ · On-site

Familiarity with web QA tools, browser developer tools, crawling tools, accessibility testing tools or performance testing tools. * Experience working in regulated, highly governed or large ...

previous page

Showing results 41-60

All JobsWeb Crawling Engineer Jobs

Web Crawling Engineer information

See salary details

$11

$59

$86

How much do web crawling engineer jobs pay per hour?

As of Sep 9, 2026, the average hourly pay for web crawling engineer in the United States is $59.01, according to ZipRecruiter salary data. Most workers in this role earn between $51.20 and $66.83 per hour, depending on experience, location, and employer.

What is a web crawling engineer?

A Web Crawling Engineer is a professional who designs, develops, and maintains automated systems (web crawlers or spiders) that systematically browse the internet to collect data from websites. Their responsibilities include building efficient crawling algorithms, handling web data extraction, managing large-scale data storage, and ensuring compliance with website policies and legal guidelines. Web Crawling Engineers commonly work in fields such as search engines, market research, price comparison, and data analytics, where gathering large volumes of online information is essential.

What are the key skills and qualifications needed to thrive as a web crawling engineer?

To thrive as a Web Crawling Engineer, you need strong programming skills in languages like Python or Java, a solid understanding of web protocols, and experience with data extraction and parsing. Proficiency with technical tools such as Scrapy, Selenium, BeautifulSoup, and knowledge of APIs and databases is commonly required. Analytical thinking, problem-solving, and attention to detail are vital soft skills for overcoming crawling challenges and ensuring data integrity. These skills and qualities are important to efficiently gather accurate data from diverse web sources while navigating technical and ethical considerations.

What are some common challenges faced by web crawling engineers when dealing with large-scale data extraction projects?

Web Crawling Engineers often encounter challenges such as handling websites with dynamic content, managing rate limits and CAPTCHAs, and ensuring compliance with legal and ethical guidelines. Additionally, efficiently processing and storing vast amounts of data while maintaining crawler performance and avoiding IP bans requires robust infrastructure and careful planning. Collaboration with data analysts and software engineers is common to ensure the extracted data meets project requirements and integrates smoothly with downstream systems.

What is the difference between Web Crawling Engineer vs Data Engineer?

AspectWeb Crawling EngineerData Engineer
Required CredentialsBachelor's in Computer Science, relevant experience in web scraping and crawlingBachelor's or higher in Computer Science, Data Science, or related fields
Work EnvironmentFocus on developing and maintaining web crawlers, data extraction toolsDesigning data pipelines, managing large datasets, optimizing data flow
Employer & Industry UsageTech companies, search engines, data providersTech firms, finance, healthcare, any data-driven industry
Common Search & Comparison IntentUnderstanding roles in web data collectionUnderstanding data infrastructure roles

While both roles involve working with data, a Web Crawling Engineer specializes in developing tools to extract data from websites, focusing on crawling and scraping techniques. A Data Engineer builds and maintains data pipelines and infrastructure to process and store large datasets. The roles often overlap in data collection and processing but differ in scope and focus.

What are popular job titles related to Web Crawling Engineer jobs?

For Web Crawling Engineer jobs, the most frequently searched job titles are:

Infographic showing various Web Crawling Engineer job openings in the United States as of September 2026, with employment types broken down into 1% Internship, 92% Full Time, 3% Part Time, and 4% Contract. Highlights an 85% Physical, 4% Hybrid, and 11% Remote job distribution, with an average salary of $122,736 per year, or $59 per hour.

Hadoop Developer

Hoboken, NJ • On-site

Sonoma Consulting Inc.
Business Management Consulting • 1 - 10 employees
Learn more about Sonoma Consulting Inc.external

Full-time

Re-posted 4 days ago

Job description

Company Description

Halo Group is a premier provider of IT talent. We place technology experts within
the teams of the world's leading companies to help them build innovative
businesses that keep them one step closer to their customers and one step
ahead of the competition. We offer a meaningful work environment for
employees, attractive and interesting engagements for consultants, and cutting-edge
digital innovation for our customers.
We delight in helping our customers execute their digital vision. Big projects or
small, Halo Group knows that by combining the highest quality talent with our
unwavering support, we will become an invaluable extension of the team. Halo
Group's experienced consultants in Detroit, Atlanta and Dallas specialize in all
areas of product/project governance, UX/UI, multi-platform applications, quality
assurance/testing, cloud computing, and data analytics.
Since its inception, Halo Group has been recognized for numerous awards, including: 
- INC 5000
- Future 50
- 101 Best and Brightest
- Michigan 50 Companies to Watch
- Goldline Research - "Most Dependable Companies"
- Ernst & Young - "Entrepreneur of the Year" Finalist

Job Description

Mandatory
3+ years of experience with Apache NiFi programming
3+ years of experience with Apache Hadoop and Spark ecosystems of open-source tools. Our data processing and modeling pipelines are built using Hortonworks platform (HDP)
2+ years of experience with Hive programming
Experience with NoSQL databases, such as HBase, Cassandra, MongoDB is a plus
Experience working with Amazon Web Services is a big plus
Working knowledge of Json and web services programing
Understanding of RDBMs and SQL programming skills, such as PostgreSQL, MySQL, MSSQL, Oracle SQL/PLSQL
Firm grasp of Big-Data Platforms and modeling frameworks, especially Spark and Hadoop; comfortable dealing with large data sets within a distributed computing environment (Hadoop, Hive/PIG, HBase, etc.)
Knowledge of any data visualization and reporting tools, such as D3, Qlik (View or Sense), Tableau, MS PowerBI, SAP Business Objects, or Microstrategy
Strong communication skills
Excellent organization and prioritization skills
Experience with web services such as AWS S3, DigitalOcean, Redshift and Spark; ability to connect data using SOAP API, REST API, and web crawling techniques is a plus
Key Skills
Apache NiFi, Hive programming, Hadoop and Spark ecosystems of open-source tools, NoSQL databases, Json and web services programing

Qualifications

Bachelor's degree or foreign equivalent required from an accredited institution. Will also consider three years of progressive experience in the specialty in lieu of every year of education.
At least 5+ years of experience with IT

Additional Information

** U.S. Citizens and those who are authorized to work independently in the United States are encouraged to apply. We are unable to sponsor at this time.
This is a Full-Time / Permanent job opportunity.
Only for US Citizen and Green Card Holder.

** All your information will be kept confidential according to EEO guidelines.


Key Skills
Apache NiFi, Hive programming, Hadoop and Spark ecosystems of open-source tools, NoSQL databases, Json and web services programing

Sonoma Consult logo

About Sonoma Consult

Sourced by ZipRecruiter

Sonoma Consult is a California based C corporation helping companies bring products to the patient by working closely with the engineering teams and the clinicians. Our goal, no matter what stage of product development, is to create and execute a plan of action to move the product through the appropriate clinical and regulatory steps. Sonoma Consult works cohesively with the engineers and physicians to translate technologies to the clinic. Our goal is to help you get the very best product to the clinic and to the market. This includes planning, execution and ensuring critical data is delivered in the right format to ensure the feedback loop to the design team ultimately delivers the most advanced technology to the patient.

Industry

Business management consulting

Company size

1 - 10 Employees

Headquarters location

Sonoma, CA, US

View All Sonoma Consult Jobs

Apply