1

Web Crawler Developer Jobs (NOW HIRING)

Working cross-functionally with product, engineering, design, and analytics teams, you will help ... Define how we measure AI, using log data, crawler activity, and emerging tooling to turn a murky ...

AI Search & SEO Manager

Pleasanton, CA · On-site

$140K - $174K/yr

... crawler access policies for LLM bots * Define and track emerging AI search metrics LLM citation ... web vitals, schema/structured data, site architecture in close collaboration with engineering

AI Search & SEO Manager

Pleasanton, CA · On-site

$140K - $174K/yr

... and crawler access policies for LLM bots * Define and track emerging AI search metrics - LLM ... web vitals, schema/structured data, site architecture - in close collaboration with engineering

NY · On-site

$102.76 - $148.43/hr

... Web Vitals, JS rendering, hreflang, and multilingual SEO. * Experience leading pre- and post ... Confident working with engineering teams -- you can scope a ticket, validate the fix, and explain ...

... URL structure, and Core Web Vitals. * Launch and migration SEO. URL mapping, redirect ... Enough HTML, CSS, and JavaScript to diagnose a problem and write a specification a developer can ...

... URL structure, and Core Web Vitals. * Launch and migration SEO. URL mapping, redirect ... Enough HTML, CSS, and JavaScript to diagnose a problem and write a specification a developer can ...

Showing results 41-47

Web Crawler Developer information

See salary details

$16

$45

$72

How much do web crawler developer jobs pay per hour?

As of Aug 22, 2026, the average hourly pay for web crawler developer in the United States is $45.12, according to ZipRecruiter salary data. Most workers in this role earn between $34.62 and $54.57 per hour, depending on experience, location, and employer.

What is a web crawler developer?

A Web Crawler Developer is a software engineer who specializes in designing, building, and maintaining web crawlers or web scraping tools. These tools automatically browse the internet and extract data from websites for various applications, such as search engines, data analytics, and market research. Web Crawler Developers often work with programming languages like Python, Java, or Node.js and must be knowledgeable about web protocols, data parsing, and ethical considerations related to web scraping.

What are the key skills and qualifications needed to thrive as a web crawler developer?

To thrive as a Web Crawler Developer, you need strong programming skills in languages like Python or Java, experience with web protocols, and a solid understanding of data extraction and parsing techniques. Familiarity with tools such as Scrapy, BeautifulSoup, Selenium, and knowledge of APIs and database management are commonly required, along with expertise in handling large-scale data. Attention to detail, problem-solving abilities, and effective communication help developers manage complex crawling tasks and collaborate with teams. These skills are crucial to efficiently gather, process, and maintain high-quality web data while navigating technical and ethical challenges.

What are some common challenges web crawler developers face when extracting data from modern websites?

Web Crawler Developers often encounter challenges such as dynamic content loaded via JavaScript, frequent changes to website structures, and anti-scraping measures like CAPTCHAs or rate limiting. Staying up-to-date with new web technologies and adapting crawlers to handle Single Page Applications (SPAs) is crucial. Additionally, collaborating closely with data analysts or product teams ensures that the extracted data meets quality and compliance standards.

What is the difference between Web Crawler Developer vs Data Engineer?

AspectWeb Crawler DeveloperData Engineer
Required CredentialsBachelor's in Computer Science, programming skillsBachelor's in CS, data management certifications
Work EnvironmentWeb development, scripting, backend systemsData pipelines, database management, cloud platforms
Industry UsageWeb scraping, search engines, data collectionData processing, analytics, big data infrastructure
Search & Comparison IntentFocus on web crawling, data extractionFocus on data storage, transformation, and analysis

The Web Crawler Developer primarily designs and maintains web crawlers to extract data from websites, focusing on web scraping and data collection. In contrast, Data Engineers build and manage data pipelines and infrastructure to process and store large datasets. While both roles require programming skills and work with data, Web Crawler Developers specialize in web-based data extraction, whereas Data Engineers focus on data integration and management across systems.

More about Web Crawler Developer jobs
Infographic showing various Web Crawler Developer job openings in the United States as of August 2026, with employment types broken down into 1% Internship, 80% Full Time, 11% Part Time, 1% Temporary, and 7% Contract. Highlights an 82% Physical, 4% Hybrid, and 14% Remote job distribution, with an average salary of $93,848 per year, or $45.1 per hour.

Product Leader, Growth & Acquisition

Restoration Hardware

Corte Madera, CA

Full-time

Posted 19 days ago


Restoration Hardware rating

7.4

Company rating: 7.4 out of 10

Based on 50 frontline employees who took The Breakroom Quiz

18th of 63 rated furniture retailers


Job description

RH is seeking a Product Leader of Growth and Acquisition to own how customers discover and choose our brand across organic search, AI-powered discovery, and the broader digital acquisition journey. As consumer behavior continues to evolve, you will lead the strategy and execution of RH's organic growth engine across a large, global ecommerce platform spanning multiple markets and languages. This role is an ideal fit for a strategic SEO leader ready to step into broader product ownership, or a growth PM with genuine, hands-on SEO depth. Combining deep SEO expertise with a product management mindset, you will drive experimentation, leverage analytics to uncover growth opportunities, and optimize the customer discovery experience to increase qualified traffic and business impact. Working cross-functionally with product, engineering, design, and analytics teams, you will help shape the future of how customers find RH while raising the analytical rigor and growth capabilities of the broader product organization.

YOUR RESPONSIBILITIES

  • Lead RH's organic growth strategy, defining the roadmap, prioritizing opportunities, and aligning initiatives to measurable business outcomes.
  • Drive enterprise SEO across a large, global ecommerce platform, optimizing site architecture, crawlability, structured data, page performance, and technical implementations that strengthen long-term organic visibility. 
  • Develop RH's AI discovery strategy, ensuring our content and products are optimized for emerging search experiences including AI Overviews, ChatGPT, Perplexity, and future conversational platforms
  • Define how we measure AI, using log data, crawler activity, and emerging tooling to turn a murky signal into decisions
  • Build a disciplined experimentation program by developing hypotheses, running A/B tests, analyzing results, and translating insights into scalable improvements across acquisition and conversion.
  • Collaborate with product, engineering, design, merchandising, content, and analytics teams to deliver growth initiatives while influencing priorities through data-driven recommendations.
  • Measure performance using analytics and experimentation, connecting organic growth initiatives to customer acquisition, revenue, and long-term business impact.
  • Champion a culture of data-informed decision making, helping elevate analytical rigor and growth thinking across the digital product organization.

OUR REQUIREMENTS

  • 4+ years of experience in Growth Product Management, SEO, or digital growth, with a demonstrated ability to drive measurable organic acquisition and business results at scale.
  • Deep expertise in enterprise SEO, including technical SEO, site architecture, structured data, Core Web Vitals, large product catalogs, faceted navigation, and complex ecommerce environments.
  • Strong analytical capabilities with experience using GA4 (or equivalent), experimentation frameworks, dashboards, and SQL or comparable data analysis tools.
  • Exceptional communication and cross-functional leadership skills, with the ability to influence teams and executives through thoughtful analysis and clear recommendations.
  • A product thinker first. You treat organic as a revenue driver, not just a traffic channel, and you don't just execute a channel checklist. You use data to find the signal, form a hypothesis, and move, without mistaking endless analysis for progress.
RH is an equal opportunity employer and does not discriminate against any applicant on the basis of race, color, religion, national origin, gender, marital status, age, disability, sexual orientation, military/veteran status, or any other status protected by federal or state law or local ordinance.

At RH, we are committed to promoting pay equity. Rate of pay is determined based on each individual's experience, qualifications, and the geographic location of the role.

What Restoration Hardware employees say

Pay

Benefits

Hours and flexibility

Workplace

Get the full story on Breakroom