1

Web Crawling Engineer Jobs (NOW HIRING)

Familiarity with web QA tools, browser developer tools, crawling tools, accessibility testing tools or performance testing tools. * Experience working in regulated, highly governed or large ...

Familiarity with web QA tools, browser developer tools, crawling tools, accessibility testing tools or performance testing tools. * Experience working in regulated, highly governed or large ...

... Web Crawling, Data Collection, Feature Engineering, MLOps, ML Systems, Scalable Systems, Parallel Computing, Model Architecture Design, Experimentation, Research Scientists, Research Engineers ...

... Web Crawling, Data Collection, Feature Engineering, MLOps, ML Systems, Scalable Systems, Parallel Computing, Model Architecture Design, Experimentation, Research Scientists, Research Engineers ...

Familiarity with web QA tools, browser developer tools, crawling tools, accessibility testing tools or performance testing tools. * Experience working in regulated, highly governed or large ...

Familiarity with web QA tools, browser developer tools, crawling tools, accessibility testing tools or performance testing tools. * Experience working in regulated, highly governed or large ...

Collaborate with ML engineers, other data engineers, and software developers to deliver actionable ... Experience with web scraping/crawling (e.g., Beautiful Soup, Selenium, Scrapy) * Experience with ...

Our adversarial red teaming, model evaluations, and intelligence collection enable engineering ... Experience with web scraping/crawling (e.g., Beautiful Soup, Selenium, Scrapy) * Experience with ...

OR · On-site

You build for the humans visiting today and the AI agents already crawling the web. LLM-ready ... programming, and more. Because we take our jobs seriously, we add in time for rest with Unlimited ...

... engineered for maintainability and ownership transfer • Experience designing web application ... crawling and scraping approaches, including structured content extraction, normalization, and ...

AI Specialist

Englewood Cliffs, NJ · On-site

$67K - $80K/hr

Maintain and enhance web crawling and external data collection engines. * Ensure data quality ... Design and optimize Prompt Engineering, RAG (Retrieval-Augmented Generation) pipelines.

You build for the humans visiting today and the AI agents already crawling the web. LLM-ready ... programming, and more. Because we take our jobs seriously, we add in time for rest with Unlimited ...

LLM Dataset Engineer

San Francisco, CA · On-site

$155K - $210K/yr

In this role, you will own the end-to-end data lifecycle-from raw web-scale crawling to the fine ... This position is ideal for a scientist who views data as a high-scale engineering challenge and an ...

This includes designing scalable crawling architectures, reverse-engineering web applications to identify data points, and implementing evasion techniques to bypass IP rate-limiting and bot detection.

Software Engineer III

Reston, VA · On-site +1

$59.75 - $80.25/hr

This includes designing scalable crawling architectures, reverse-engineering web applications to identify data points, and implementing evasion techniques to bypass IP rate-limiting and bot detection.

next page

Showing results 1-20

Web Crawling Engineer information

See salary details

$11

$59

$86

How much do web crawling engineer jobs pay per hour?

As of Jun 12, 2026, the average hourly pay for web crawling engineer in the United States is $59.01, according to ZipRecruiter salary data. Most workers in this role earn between $51.20 and $66.83 per hour, depending on experience, location, and employer.

What are some common challenges faced by Web Crawling Engineers when dealing with large-scale data extraction projects?

Web Crawling Engineers often encounter challenges such as handling websites with dynamic content, managing rate limits and CAPTCHAs, and ensuring compliance with legal and ethical guidelines. Additionally, efficiently processing and storing vast amounts of data while maintaining crawler performance and avoiding IP bans requires robust infrastructure and careful planning. Collaboration with data analysts and software engineers is common to ensure the extracted data meets project requirements and integrates smoothly with downstream systems.

What is the difference between Web Crawling Engineer vs Data Engineer?

AspectWeb Crawling EngineerData Engineer
Required CredentialsBachelor's in Computer Science, relevant experience in web scraping and crawlingBachelor's or higher in Computer Science, Data Science, or related fields
Work EnvironmentFocus on developing and maintaining web crawlers, data extraction toolsDesigning data pipelines, managing large datasets, optimizing data flow
Employer & Industry UsageTech companies, search engines, data providersTech firms, finance, healthcare, any data-driven industry
Common Search & Comparison IntentUnderstanding roles in web data collectionUnderstanding data infrastructure roles

While both roles involve working with data, a Web Crawling Engineer specializes in developing tools to extract data from websites, focusing on crawling and scraping techniques. A Data Engineer builds and maintains data pipelines and infrastructure to process and store large datasets. The roles often overlap in data collection and processing but differ in scope and focus.

What are the key skills and qualifications needed to thrive as a Web Crawling Engineer, and why are they important?

To thrive as a Web Crawling Engineer, you need strong programming skills in languages like Python or Java, a solid understanding of web protocols, and experience with data extraction and parsing. Proficiency with technical tools such as Scrapy, Selenium, BeautifulSoup, and knowledge of APIs and databases is commonly required. Analytical thinking, problem-solving, and attention to detail are vital soft skills for overcoming crawling challenges and ensuring data integrity. These skills and qualities are important to efficiently gather accurate data from diverse web sources while navigating technical and ethical considerations.

What is a Web Crawling Engineer?

A Web Crawling Engineer is a professional who designs, develops, and maintains automated systems (web crawlers or spiders) that systematically browse the internet to collect data from websites. Their responsibilities include building efficient crawling algorithms, handling web data extraction, managing large-scale data storage, and ensuring compliance with website policies and legal guidelines. Web Crawling Engineers commonly work in fields such as search engines, market research, price comparison, and data analytics, where gathering large volumes of online information is essential.
Web Developer

Web Developer

State Farm

Dunwoody, GA • On-site

Full-time

Medical, Dental, Vision, Life, Retirement, PTO

Posted 4 days ago


State Farm rating

7.5

Company rating: 7.5 out of 10

Based on 1,362 frontline employees who took The Breakroom Quiz

195th of 261 rated insurance


Job description

Overview

Being good neighbors – helping people, investing in our communities, and making the world a better place – is who we are at State Farm. It is at the core of how we operate and the reason for our success. Come join a #1 team and do some good!

Sponsorship: Applicants are required to be eligible to lawfully work in the U.S. immediately; employer will not sponsor applicants for U.S. work authorization (e.g. H-1B visa) for this opportunity.


Responsibilities

State Farm is hiring a Web Developer to build, maintain and improve digital experiences that support content, site performance, accessibility, measurement and organic search visibility. This role is a good fit for an experienced Web Developer who can work independently on assigned page builds, enhancements and fixes while partnering with design teams to translate approved design files into high-quality web experiences and collaborating with senior technical partners on more complex platform or architecture decisions.

In this role, you will:

  • Build, test and maintain web pages, content modules and site enhancements that support digital marketing, technical SEO, accessibility and site performance goals.
  • Implement page builds and template updates in Adobe Experience Manager or similar content management systems, following design, content and technical requirements.
  • Independently complete assigned development tasks, troubleshoot issues and validate that updates meet quality, accessibility, SEO and measurement requirements.
  • Support technical SEO implementation, including metadata, redirects, canonical tags, internal linking support, structured data markup and crawl/indexing issue resolution.
  • Improve page quality through fixes and enhancements related to site speed, accessibility, responsive behavior, cross-browser compatibility and user experience.
  • Support analytics tagging and validation, including event tracking, data layer checks and post-release measurement QA in partnership with analytics and SEO teams.
  • Troubleshoot production issues, document fixes and support release, change management and quality assurance processes.
  • Maintain clear technical documentation and contribute to reusable components, templates, standards and development patterns.
  • Partner with SEO, content strategy, copywriting and UX/design teams to ensure content is implemented accurately and displays correctly across pages, templates and components.
  • Partner with brand, UX and experience design teams to interpret design files, clarify requirements and ensure web pages and components are implemented accurately across devices, browsers and templates.
  • Collaborate with senior web development, technology and engineering partners on development standards, platform requirements, environments, deployments and technical constraints.
  • Support page launches, testing, optimization updates and ongoing improvements across digital marketing experiences.

Qualifications

Required Qualifications

  • Bachelor's degree in computer science, web development, information technology, digital media or a related field.
  • 4+ years of professional web development experience supporting websites, content management systems or digital marketing experiences.
  • Experience building and maintaining web pages using HTML, CSS and JavaScript.
  • Experience working in a content management system, such as Adobe Experience Manager, WordPress, Sitecore, Drupal or a similar platform.
  • Ability to work independently on assigned development tasks, troubleshoot issues and deliver quality work with limited day-to-day direction.
  • Understanding of responsive web development, cross-browser compatibility, accessibility and web performance best practices.
  • Familiarity with technical SEO implementation, such as metadata, redirects, canonical tags, internal linking, structured data and crawl/indexing considerations.
  • Experience supporting analytics tagging or measurement implementation, such as event tracking, data layer updates or tag validation.
  • Ability to perform QA, validate releases and document technical changes clearly.
  • Experience collaborating with cross-functional partners, such as SEO, content, UX/design, analytics, marketing, product or technology teams.

Preferred Qualifications

  • Experience with Adobe Experience Manager components, templates, content fragments, experience fragments or similar enterprise CMS capabilities.
  • Experience supporting technical SEO, Core Web Vitals, page speed improvements, structured data or accessibility remediation.
  • Familiarity with analytics and tag management tools, such as Google Tag Manager, Adobe Launch, Adobe Analytics, GA4 or similar platforms.
  • Experience working with design systems, reusable components or standardized front-end patterns.
  • Familiarity with web QA tools, browser developer tools, crawling tools, accessibility testing tools or performance testing tools.
  • Experience working in regulated, highly governed or large-enterprise digital environments.
  • Familiarity with Agile, Jira, release management or enterprise development workflows.

Items of Note

  • Job may require travel via commercial transportation and/or operating motor vehicles
  • May be subject to working outside of normal working hours, including weekends
  • May require involvement in management-approved education programs and activities as applicable for continued development
  • Exhibits thorough knowledge of department and company goals, strategies and vision, including contractual and compliance standards such as brand, intellectual property, copyright and trademark guidelines

Our Benefits

Because work-life balance is a priority at State Farm, compensation is based on our standard 38:45-hour work week!

  • Potential starting salary range:  $100,000 - $125,000
    • Starting salary will be based on skills, background, and experience
    • High end of the range limited to applicants with significant relevant experience
  • Potential yearly incentive pay up to 15% of base salary

At State Farm, we offer more than just a paycheck. Check out our suite of benefits designed to give you the flexibility you need to take care of you and your family!

  • Get Paid! On top of our competitive pay, you are eligible for an annual raise and bonus.
  • Stay Well! Focus on you and your family’s health with our robust health and wellbeing programs. State Farm pays most of your healthcare premium, and we offer multiple healthcare plan options, including a high deductible plan. All medical plans provide 100% coverage for in-network preventative care, AND you and your family have access to vision, dental, telemedicine, 24/7 mental health professionals, and much more!
  • Develop and Grow! Take advantage of educational benefits like industry leading training programs, top-notch tuition assistance programs, employee resource groups, and mentoring.
  • Plan Ahead! Plan for those big moments in life with benefits like fertility/IVF/adoption assistance, college coaching, national discount programs, interactive monthly financial workshops, free financial coaching, and more. You can also start a savings account or consider financing through our State Farm Federal Credit Union!
  • Take a Little “You” Time! You will have access to our generous time off policies designed so you can plan around holidays, family events, volunteering, or just to take a relaxing day off. With the opportunity to initially earn up to 20 days annually plus parental leave, paid holidays, celebration day, life leave (40 hours/year), bereavement leave, and community service/education support days, there will be plenty of time for you!
  • Give Back! We offer several ways to give back through our Matching Gift Program, Good Neighbor Grant Program, and the Employee Assistance Fund.
  • Finish Strong! Plan for retirement using free financial advisors and a 401(k) plan with company contributions of up to 7% of your salary.

Visit our State Farm Careers page for more information on our benefits, locations, and the hiring process of joining the State Farm team!

Remote: Qualified candidates (outside of hub locations listed below) may be considered for 100% remote work arrangements based on where a candidate currently resides or is currently located.
Hybrid: Qualified candidates (in or near hub locations listed below) should plan to spend time working from home and some time working in the office as part of our hybrid work environment.
Hub locations: Dunwoody, GA; Richardson, TX; Tempe, AZ; or Bloomington, IL
Sponsorship: Applicants are required to be eligible to lawfully work in the U.S. immediately; employer will not sponsor applicants for U.S. work authorization (e.g. H-1B visa) for this opportunity.
For Los Angeles candidates: Pursuant to the Los Angeles Fair Chance Initiative for Hiring, we will consider for employment qualified applicants with criminal histories.
For San Francisco candidates: Pursuant to the San Francisco Fair Chance Ordinance, we will consider for employment qualified applicants with arrest and conviction records.
For Colorado, Hawaii and Washington State candidates:
Salary range: $110,000 - $137,500
Application deadline is expected to close on 6/26/2026. Applicant volume and hiring needs may result in early closure or extension beyond the listed deadline. To submit an application, click "Apply" on the job listing page on the State Farm career site.
For California, New York, and CT candidates:
Potential salary range: $125,000 - $156,250
Potential yearly incentive pay: up to 15% of base salary
Competitive Benefits, including:

  • 401k Plan
  • Health Insurance
  • Dental/Vision plans
  • Life Insurance
  • Paid Time Off
  • Annual Merit Increase
  • Tuition Reimbursement
  • Health Initiatives

For more details, visit our benefits summary page.

Qualifications:

Required Qualifications

  • Bachelor's degree in computer science, web development, information technology, digital media or a related field.
  • 4+ years of professional web development experience supporting websites, content management systems or digital marketing experiences.
  • Experience building and maintaining web pages using HTML, CSS and JavaScript.
  • Experience working in a content management system, such as Adobe Experience Manager, WordPress, Sitecore, Drupal or a similar platform.
  • Ability to work independently on assigned development tasks, troubleshoot issues and deliver quality work with limited day-to-day direction.
  • Understanding of responsive web development, cross-browser compatibility, accessibility and web performance best practices.
  • Familiarity with technical SEO implementation, such as metadata, redirects, canonical tags, internal linking, structured data and crawl/indexing considerations.
  • Experience supporting analytics tagging or measurement implementation, such as event tracking, data layer updates or tag validation.
  • Ability to perform QA, validate releases and document technical changes clearly.
  • Experience collaborating with cross-functional partners, such as SEO, content, UX/design, analytics, marketing, product or technology teams.

Preferred Qualifications

  • Experience with Adobe Experience Manager components, templates, content fragments, experience fragments or similar enterprise CMS capabilities.
  • Experience supporting technical SEO, Core Web Vitals, page speed improvements, structured data or accessibility remediation.
  • Familiarity with analytics and tag management tools, such as Google Tag Manager, Adobe Launch, Adobe Analytics, GA4 or similar platforms.
  • Experience working with design systems, reusable components or standardized front-end patterns.
  • Familiarity with web QA tools, browser developer tools, crawling tools, accessibility testing tools or performance testing tools.
  • Experience working in regulated, highly governed or large-enterprise digital environments.
  • Familiarity with Agile, Jira, release management or enterprise development workflows.

Items of Note

  • Job may require travel via commercial transportation and/or operating motor vehicles
  • May be subject to working outside of normal working hours, including weekends
  • May require involvement in management-approved education programs and activities as applicable for continued development
  • Exhibits thorough knowledge of department and company goals, strategies and vision, including contractual and compliance standards such as brand, intellectual property, copyright and trademark guidelines
Education:UNAVAILABLEEmployment Type: FULL_TIME

What State Farm employees say

Pay

Benefits

Hours and flexibility

Workplace

Get the full story on Breakroom