Working cross-functionally with product, engineering, design, and analytics teams, you will help ... Define how we measure AI, using log data, crawler activity, and emerging tooling to turn a murky ...
Working cross-functionally with product, engineering, design, and analytics teams, you will help ... Define how we measure AI, using log data, crawler activity, and emerging tooling to turn a murky ...
Product Leader, Growth & Acquisition
Corte Madera, CA · On-site +1
$120K - $160K/yr
Working cross-functionally with product, engineering, design, and analytics teams, you will help ... Define how we measure AI, using log data, crawler activity, and emerging tooling to turn a murky ...
Product Leader, Growth & Acquisition
Corte Madera, CA · On-site +1
$120K - $160K/yr
Working cross-functionally with product, engineering, design, and analytics teams, you will help ... Define how we measure AI, using log data, crawler activity, and emerging tooling to turn a murky ...
AI Search & SEO Manager
Pleasanton, CA · On-site
$140K - $174K/yr
... crawler access policies for LLM bots * Define and track emerging AI search metrics LLM citation ... web vitals, schema/structured data, site architecture in close collaboration with engineering
Quick apply
AI Search & SEO Manager
Pleasanton, CA · On-site
$140K - $174K/yr
... crawler access policies for LLM bots * Define and track emerging AI search metrics LLM citation ... web vitals, schema/structured data, site architecture in close collaboration with engineering
AI Search & SEO Manager
Pleasanton, CA · On-site
$140K - $174K/yr
... and crawler access policies for LLM bots * Define and track emerging AI search metrics - LLM ... web vitals, schema/structured data, site architecture - in close collaboration with engineering
AI Search & SEO Manager
Pleasanton, CA · On-site
$140K - $174K/yr
... and crawler access policies for LLM bots * Define and track emerging AI search metrics - LLM ... web vitals, schema/structured data, site architecture - in close collaboration with engineering
NY · On-site
$102.76 - $148.43/hr
... Web Vitals, JS rendering, hreflang, and multilingual SEO. * Experience leading pre- and post ... Confident working with engineering teams -- you can scope a ticket, validate the fix, and explain ...
... URL structure, and Core Web Vitals. * Launch and migration SEO. URL mapping, redirect ... Enough HTML, CSS, and JavaScript to diagnose a problem and write a specification a developer can ...
... URL structure, and Core Web Vitals. * Launch and migration SEO. URL mapping, redirect ... Enough HTML, CSS, and JavaScript to diagnose a problem and write a specification a developer can ...
... URL structure, and Core Web Vitals. * Launch and migration SEO. URL mapping, redirect ... Enough HTML, CSS, and JavaScript to diagnose a problem and write a specification a developer can ...
... URL structure, and Core Web Vitals. * Launch and migration SEO. URL mapping, redirect ... Enough HTML, CSS, and JavaScript to diagnose a problem and write a specification a developer can ...
Web Crawler Developer information
See salary details
$16.83 - $21.85
3% of jobs
$21.85 - $26.88
3% of jobs
$26.88 - $31.91
12% of jobs
$34.33 is the 25th percentile. Wages below this are outliers.
$31.91 - $36.93
15% of jobs
$36.93 - $41.96
16% of jobs
The median wage is $42.54 / hr.
$41.96 - $46.98
14% of jobs
$46.98 - $52.01
11% of jobs
$53.14 is the 75th percentile. Wages above this are outliers.
$52.01 - $57.04
11% of jobs
$57.04 - $62.06
8% of jobs
$62.06 - $67.09
5% of jobs
$67.09 - $72.12
3% of jobs
$16
$45
$72
How much do web crawler developer jobs pay per hour?
What is a web crawler developer?
What are the key skills and qualifications needed to thrive as a web crawler developer?
What are some common challenges web crawler developers face when extracting data from modern websites?
What is the difference between Web Crawler Developer vs Data Engineer?
| Aspect | Web Crawler Developer | Data Engineer |
|---|---|---|
| Required Credentials | Bachelor's in Computer Science, programming skills | Bachelor's in CS, data management certifications |
| Work Environment | Web development, scripting, backend systems | Data pipelines, database management, cloud platforms |
| Industry Usage | Web scraping, search engines, data collection | Data processing, analytics, big data infrastructure |
| Search & Comparison Intent | Focus on web crawling, data extraction | Focus on data storage, transformation, and analysis |
The Web Crawler Developer primarily designs and maintains web crawlers to extract data from websites, focusing on web scraping and data collection. In contrast, Data Engineers build and manage data pipelines and infrastructure to process and store large datasets. While both roles require programming skills and work with data, Web Crawler Developers specialize in web-based data extraction, whereas Data Engineers focus on data integration and management across systems.
What job categories do people searching Web Crawler Developer jobs look for?
The top searched job categories for Web Crawler Developer jobs are:

Restoration Hardware rating
7.4
Based on 50 frontline employees who took The Breakroom Quiz
18th of 63 rated furniture retailers
Job description
RH is seeking a Product Leader of Growth and Acquisition to own how customers discover and choose our brand across organic search, AI-powered discovery, and the broader digital acquisition journey. As consumer behavior continues to evolve, you will lead the strategy and execution of RH's organic growth engine across a large, global ecommerce platform spanning multiple markets and languages. This role is an ideal fit for a strategic SEO leader ready to step into broader product ownership, or a growth PM with genuine, hands-on SEO depth. Combining deep SEO expertise with a product management mindset, you will drive experimentation, leverage analytics to uncover growth opportunities, and optimize the customer discovery experience to increase qualified traffic and business impact. Working cross-functionally with product, engineering, design, and analytics teams, you will help shape the future of how customers find RH while raising the analytical rigor and growth capabilities of the broader product organization.
YOUR RESPONSIBILITIES
- Lead RH's organic growth strategy, defining the roadmap, prioritizing opportunities, and aligning initiatives to measurable business outcomes.
- Drive enterprise SEO across a large, global ecommerce platform, optimizing site architecture, crawlability, structured data, page performance, and technical implementations that strengthen long-term organic visibility.
- Develop RH's AI discovery strategy, ensuring our content and products are optimized for emerging search experiences including AI Overviews, ChatGPT, Perplexity, and future conversational platforms
- Define how we measure AI, using log data, crawler activity, and emerging tooling to turn a murky signal into decisions
- Build a disciplined experimentation program by developing hypotheses, running A/B tests, analyzing results, and translating insights into scalable improvements across acquisition and conversion.
- Collaborate with product, engineering, design, merchandising, content, and analytics teams to deliver growth initiatives while influencing priorities through data-driven recommendations.
- Measure performance using analytics and experimentation, connecting organic growth initiatives to customer acquisition, revenue, and long-term business impact.
- Champion a culture of data-informed decision making, helping elevate analytical rigor and growth thinking across the digital product organization.
OUR REQUIREMENTS
- 4+ years of experience in Growth Product Management, SEO, or digital growth, with a demonstrated ability to drive measurable organic acquisition and business results at scale.
- Deep expertise in enterprise SEO, including technical SEO, site architecture, structured data, Core Web Vitals, large product catalogs, faceted navigation, and complex ecommerce environments.
- Strong analytical capabilities with experience using GA4 (or equivalent), experimentation frameworks, dashboards, and SQL or comparable data analysis tools.
- Exceptional communication and cross-functional leadership skills, with the ability to influence teams and executives through thoughtful analysis and clear recommendations.
- A product thinker first. You treat organic as a revenue driver, not just a traffic channel, and you don't just execute a channel checklist. You use data to find the signal, form a hypothesis, and move, without mistaking endless analysis for progress.
What Restoration Hardware employees say
Pay
Benefits
Hours and flexibility
Workplace
Get the full story on Breakroom
About Restoration Hardware
Sourced by ZipRecruiter
Industry
Specialized design services
Company size
5,001 - 10,000 Employees
Headquarters location
Corte Madera, CA , CA, US
Year founded
1979