Hadoop Developer
Hoboken, NJ · On-site
... and web crawling techniques is a plus Key Skills Apache NiFi, Hive programming, Hadoop and Spark ecosystems of open-source tools, NoSQL databases, Json and web services programing Qualifications ...
Hadoop Developer
Hoboken, NJ · On-site
... and web crawling techniques is a plus Key Skills Apache NiFi, Hive programming, Hadoop and Spark ecosystems of open-source tools, NoSQL databases, Json and web services programing Qualifications ...
C# Developer (Big Data / Market Intelligence)
Chicago, IL · On-site
$54.50 - $70.75/hr
Ready to get your hands dirty with artificial intelligence, high-volume web crawling, and big data ... Some examples of the types of challenges engineers tackle on a regular basis: Design an artificial ...
C# Developer (Big Data / Market Intelligence)
Chicago, IL · On-site
$54.50 - $70.75/hr
Ready to get your hands dirty with artificial intelligence, high-volume web crawling, and big data ... Some examples of the types of challenges engineers tackle on a regular basis: Design an artificial ...
Maintain web crawling & external data systems 2. AI / LLM Development * Develop LLM-powered systems ... Build RAG pipelines + prompt engineering solutions * Continuously improve models based on business ...
Maintain web crawling & external data systems 2. AI / LLM Development * Develop LLM-powered systems ... Build RAG pipelines + prompt engineering solutions * Continuously improve models based on business ...
At the moment our team is working on building a web crawling architecture that can crawl 10,000,000+ articles a day and make them fully searchable in real time. There is some serious engineering ...
At the moment our team is working on building a web crawling architecture that can crawl 10,000,000+ articles a day and make them fully searchable in real time. There is some serious engineering ...
... Web Crawling, Data Collection, Feature Engineering, MLOps, ML Systems, Scalable Systems, Parallel Computing, Model Architecture Design, Experimentation, Research Scientists, Research Engineers ...
... Web Crawling, Data Collection, Feature Engineering, MLOps, ML Systems, Scalable Systems, Parallel Computing, Model Architecture Design, Experimentation, Research Scientists, Research Engineers ...
Staff Software Development Engineer
Sunnyvale, CA · On-site
$150K - $215K/yr
Fortinet is calling for an experienced Staff Software Development Engineer who can think outside ... Experience with popular security detecting, penetration testing, web crawling tools is a plus, like ...
Staff Software Development Engineer
Sunnyvale, CA · On-site
$150K - $215K/yr
Fortinet is calling for an experienced Staff Software Development Engineer who can think outside ... Experience with popular security detecting, penetration testing, web crawling tools is a plus, like ...
... Web Crawling, Data Collection, Feature Engineering, MLOps, ML Systems, Scalable Systems, Parallel Computing, Model Architecture Design, Experimentation, Research Scientists, Research Engineers ...
... Web Crawling, Data Collection, Feature Engineering, MLOps, ML Systems, Scalable Systems, Parallel Computing, Model Architecture Design, Experimentation, Research Scientists, Research Engineers ...
... Web Crawling, Data Collection, Feature Engineering, MLOps, ML Systems, Scalable Systems, Parallel Computing, Model Architecture Design, Experimentation, Research Scientists, Research Engineers ...
... Web Crawling, Data Collection, Feature Engineering, MLOps, ML Systems, Scalable Systems, Parallel Computing, Model Architecture Design, Experimentation, Research Scientists, Research Engineers ...
... Web Crawling, Data Collection, Feature Engineering, MLOps, ML Systems, Scalable Systems, Parallel Computing, Model Architecture Design, Experimentation, Research Scientists, Research Engineers ...
... Web Crawling, Data Collection, Feature Engineering, MLOps, ML Systems, Scalable Systems, Parallel Computing, Model Architecture Design, Experimentation, Research Scientists, Research Engineers ...
... Web Crawling, Data Collection, Feature Engineering, MLOps, ML Systems, Scalable Systems, Parallel Computing, Model Architecture Design, Experimentation, Research Scientists, Research Engineers ...
... Web Crawling, Data Collection, Feature Engineering, MLOps, ML Systems, Scalable Systems, Parallel Computing, Model Architecture Design, Experimentation, Research Scientists, Research Engineers ...
Staff Software Development Engineer
Sunnyvale, CA · On-site
$150K - $215K/yr
Fortinet is calling for an experienced Staff Software Development Engineer who can think outside ... Experience with popular security detecting, penetration testing, web crawling tools is a plus, like ...
Staff Software Development Engineer
Sunnyvale, CA · On-site
$150K - $215K/yr
Fortinet is calling for an experienced Staff Software Development Engineer who can think outside ... Experience with popular security detecting, penetration testing, web crawling tools is a plus, like ...
Maintain web crawling & external data systems 2. AI / LLM Development * Develop LLM-powered systems ... Build RAG pipelines + prompt engineering solutions * Continuously improve models based on business ...
Maintain web crawling & external data systems 2. AI / LLM Development * Develop LLM-powered systems ... Build RAG pipelines + prompt engineering solutions * Continuously improve models based on business ...
... Web Crawling, Data Collection, Feature Engineering, MLOps, ML Systems, Scalable Systems, Parallel Computing, Model Architecture Design, Experimentation, Research Scientists, Research Engineers ...
... Web Crawling, Data Collection, Feature Engineering, MLOps, ML Systems, Scalable Systems, Parallel Computing, Model Architecture Design, Experimentation, Research Scientists, Research Engineers ...
... Web Crawling, Data Collection, Feature Engineering, MLOps, ML Systems, Scalable Systems, Parallel Computing, Model Architecture Design, Experimentation, Research Scientists, Research Engineers ...
... Web Crawling, Data Collection, Feature Engineering, MLOps, ML Systems, Scalable Systems, Parallel Computing, Model Architecture Design, Experimentation, Research Scientists, Research Engineers ...
Member of Engineering (Pre-training / Data Acquisition)
$117K - $140K/yr
As our first dedicated data acquisition engineer, you will spearhead and evolve systems that crawl ... Hands-on experience with web crawling or large-scale data extraction: understanding of HTTP ...
Member of Engineering (Pre-training / Data Acquisition)
$117K - $140K/yr
As our first dedicated data acquisition engineer, you will spearhead and evolve systems that crawl ... Hands-on experience with web crawling or large-scale data extraction: understanding of HTTP ...
Staff Software Development Engineer
Sunnyvale, CA · On-site
$150K - $215K/yr
Fortinet is calling for an experienced Staff Software Development Engineer who can think outside ... Experience with popular security detecting, penetration testing, web crawling tools is a plus, like ...
Staff Software Development Engineer
Sunnyvale, CA · On-site
$150K - $215K/yr
Fortinet is calling for an experienced Staff Software Development Engineer who can think outside ... Experience with popular security detecting, penetration testing, web crawling tools is a plus, like ...
Maintain web crawling & external data systems 2. AI / LLM Development * Develop LLM-powered systems ... Build RAG pipelines + prompt engineering solutions * Continuously improve models based on business ...
Quick apply
Maintain web crawling & external data systems 2. AI / LLM Development * Develop LLM-powered systems ... Build RAG pipelines + prompt engineering solutions * Continuously improve models based on business ...
Staff Software Development Engineer
Sunnyvale, CA · On-site
$150K - $215K/yr
Fortinet is calling for an experienced Staff Software Development Engineer who can think outside ... Experience with popular security detecting, penetration testing, web crawling tools is a plus, like ...
Staff Software Development Engineer
Sunnyvale, CA · On-site
$150K - $215K/yr
Fortinet is calling for an experienced Staff Software Development Engineer who can think outside ... Experience with popular security detecting, penetration testing, web crawling tools is a plus, like ...
... Web Crawling, Data Collection, Feature Engineering, MLOps, ML Systems, Scalable Systems, Parallel Computing, Model Architecture Design, Experimentation, Research Scientists, Research Engineers ...
... Web Crawling, Data Collection, Feature Engineering, MLOps, ML Systems, Scalable Systems, Parallel Computing, Model Architecture Design, Experimentation, Research Scientists, Research Engineers ...
... Web Crawling, Data Collection, Feature Engineering, MLOps, ML Systems, Scalable Systems, Parallel Computing, Model Architecture Design, Experimentation, Research Scientists, Research Engineers ...
... Web Crawling, Data Collection, Feature Engineering, MLOps, ML Systems, Scalable Systems, Parallel Computing, Model Architecture Design, Experimentation, Research Scientists, Research Engineers ...
Web Crawling Engineer information
See salary details
$11.54 - $18.36
4% of jobs
$18.36 - $25.17
0% of jobs
$25.17 - $31.99
0% of jobs
$31.99 - $38.81
6% of jobs
$38.81 - $45.63
5% of jobs
$50.89 is the 25th percentile. Wages below this are outliers.
$45.63 - $52.45
12% of jobs
The median wage is $59.11 / hr.
$52.45 - $59.27
23% of jobs
$65.74 is the 75th percentile. Wages above this are outliers.
$59.27 - $66.08
26% of jobs
$66.08 - $72.90
13% of jobs
$72.90 - $79.72
3% of jobs
$79.72 - $86.54
7% of jobs
$11
$59
$86
How much do web crawling engineer jobs pay per hour?
As of Jun 12, 2026, the average hourly pay for web crawling engineer in the United States is $59.01, according to ZipRecruiter salary data. Most workers in this role earn between $51.20 and $66.83 per hour, depending on experience, location, and employer.
What are some common challenges faced by Web Crawling Engineers when dealing with large-scale data extraction projects?
Web Crawling Engineers often encounter challenges such as handling websites with dynamic content, managing rate limits and CAPTCHAs, and ensuring compliance with legal and ethical guidelines. Additionally, efficiently processing and storing vast amounts of data while maintaining crawler performance and avoiding IP bans requires robust infrastructure and careful planning. Collaboration with data analysts and software engineers is common to ensure the extracted data meets project requirements and integrates smoothly with downstream systems.
What is the difference between Web Crawling Engineer vs Data Engineer?
| Aspect | Web Crawling Engineer | Data Engineer |
|---|---|---|
| Required Credentials | Bachelor's in Computer Science, relevant experience in web scraping and crawling | Bachelor's or higher in Computer Science, Data Science, or related fields |
| Work Environment | Focus on developing and maintaining web crawlers, data extraction tools | Designing data pipelines, managing large datasets, optimizing data flow |
| Employer & Industry Usage | Tech companies, search engines, data providers | Tech firms, finance, healthcare, any data-driven industry |
| Common Search & Comparison Intent | Understanding roles in web data collection | Understanding data infrastructure roles |
While both roles involve working with data, a Web Crawling Engineer specializes in developing tools to extract data from websites, focusing on crawling and scraping techniques. A Data Engineer builds and maintains data pipelines and infrastructure to process and store large datasets. The roles often overlap in data collection and processing but differ in scope and focus.
What are the key skills and qualifications needed to thrive as a Web Crawling Engineer, and why are they important?
To thrive as a Web Crawling Engineer, you need strong programming skills in languages like Python or Java, a solid understanding of web protocols, and experience with data extraction and parsing. Proficiency with technical tools such as Scrapy, Selenium, BeautifulSoup, and knowledge of APIs and databases is commonly required. Analytical thinking, problem-solving, and attention to detail are vital soft skills for overcoming crawling challenges and ensuring data integrity. These skills and qualities are important to efficiently gather accurate data from diverse web sources while navigating technical and ethical considerations.
What is a Web Crawling Engineer?
A Web Crawling Engineer is a professional who designs, develops, and maintains automated systems (web crawlers or spiders) that systematically browse the internet to collect data from websites. Their responsibilities include building efficient crawling algorithms, handling web data extraction, managing large-scale data storage, and ensuring compliance with website policies and legal guidelines. Web Crawling Engineers commonly work in fields such as search engines, market research, price comparison, and data analytics, where gathering large volumes of online information is essential.
Job description
Company Description
Halo Group is a premier provider of IT talent. We place technology experts within
the teams of the world's leading companies to help them build innovative
businesses that keep them one step closer to their customers and one step
ahead of the competition. We offer a meaningful work environment for
employees, attractive and interesting engagements for consultants, and cutting-edge
digital innovation for our customers.
We delight in helping our customers execute their digital vision. Big projects or
small, Halo Group knows that by combining the highest quality talent with our
unwavering support, we will become an invaluable extension of the team. Halo
Group's experienced consultants in Detroit, Atlanta and Dallas specialize in all
areas of product/project governance, UX/UI, multi-platform applications, quality
assurance/testing, cloud computing, and data analytics.
Since its inception, Halo Group has been recognized for numerous awards, including:
- INC 5000
- Future 50
- 101 Best and Brightest
- Michigan 50 Companies to Watch
- Goldline Research - "Most Dependable Companies"
- Ernst & Young - "Entrepreneur of the Year" Finalist
Job Description
Mandatory
3+ years of experience with Apache NiFi programming
3+ years of experience with Apache Hadoop and Spark ecosystems of open-source tools. Our data processing and modeling pipelines are built using Hortonworks platform (HDP)
2+ years of experience with Hive programming
Experience with NoSQL databases, such as HBase, Cassandra, MongoDB is a plus
Experience working with Amazon Web Services is a big plus
Working knowledge of Json and web services programing
Understanding of RDBMs and SQL programming skills, such as PostgreSQL, MySQL, MSSQL, Oracle SQL/PLSQL
Firm grasp of Big-Data Platforms and modeling frameworks, especially Spark and Hadoop; comfortable dealing with large data sets within a distributed computing environment (Hadoop, Hive/PIG, HBase, etc.)
Knowledge of any data visualization and reporting tools, such as D3, Qlik (View or Sense), Tableau, MS PowerBI, SAP Business Objects, or Microstrategy
Strong communication skills
Excellent organization and prioritization skills
Experience with web services such as AWS S3, DigitalOcean, Redshift and Spark; ability to connect data using SOAP API, REST API, and web crawling techniques is a plus
Key Skills
Apache NiFi, Hive programming, Hadoop and Spark ecosystems of open-source tools, NoSQL databases, Json and web services programing
Qualifications
Bachelor's degree or foreign equivalent required from an accredited institution. Will also consider three years of progressive experience in the specialty in lieu of every year of education.
At least 5+ years of experience with IT
Additional Information
** U.S. Citizens and those who are authorized to work independently in the United States are encouraged to apply. We are unable to sponsor at this time.
This is a Full-Time / Permanent job opportunity.
Only for US Citizen and Green Card Holder.
** All your information will be kept confidential according to EEO guidelines.
Key Skills
Apache NiFi, Hive programming, Hadoop and Spark ecosystems of open-source tools, NoSQL databases, Json and web services programing
About Sonoma Consult
Sourced by ZipRecruiter
Sonoma Consult is a California based C corporation helping companies bring products to the patient by working closely with the engineering teams and the clinicians. Our goal, no matter what stage of product development, is to create and execute a plan of action to move the product through the appropriate clinical and regulatory steps. Sonoma Consult works cohesively with the engineers and physicians to translate technologies to the clinic. Our goal is to help you get the very best product to the clinic and to the market. This includes planning, execution and ensuring critical data is delivered in the right format to ensure the feedback loop to the design team ultimately delivers the most advanced technology to the patient.
Industry
Business management consulting
Company size
1 - 10 Employees
Headquarters location
Sonoma, CA, US