1

Web Crawler Developer Jobs (NOW HIRING)

... and engineering standards * Apply component-based and front-end best practices to ensure internal ... Explore and implement web crawler and scraper approaches, structured extraction workflows, and ...

... and engineering standards * Apply component-based and front-end best practices to ensure internal ... Explore and implement web crawler and scraper approaches, structured extraction workflows, and ...

... and engineering standards * Apply component-based and front-end best practices to ensure internal ... Explore and implement web crawler and scraper approaches, structured extraction workflows, and ...

... and engineering standards * Apply component-based and front-end best practices to ensure internal ... Explore and implement web crawler and scraper approaches, structured extraction workflows, and ...

... and engineering standards * Apply component-based and front-end best practices to ensure internal ... Explore and implement web crawler and scraper approaches, structured extraction workflows, and ...

... and engineering standards * Apply component-based and front-end best practices to ensure internal ... Explore and implement web crawler and scraper approaches, structured extraction workflows, and ...

Research Crawling Engineer

New York, NY · Remote

$100K - $130K/yr

Additionally, the team has engineered sophisticated pipelines for the ingestion, segmentation, and ... Contribute to the development of a web-scale crawler and knowledge graph at the forefront of AI ...

Research Crawling Engineer

New York, NY · On-site +1

$100K - $130K/yr

Additionally, the team has engineered sophisticated pipelines for the ingestion, segmentation, and ... Contribute to the development of a web-scale crawler and knowledge graph at the forefront of AI ...

Senior Software Engineer, Next.JS

$125K - $165K/yr

... web - and the systems that turn readers into long-term members. What Does This Team Do? The Fool ... crawler requests resolve. Much of fool.com is server side rendered. What You'll Do * Build and ...

OR · On-site

$122K - $161K/yr

... web - and the systems that turn readers into long-term members. What Does This Team Do? The Fool ... crawler requests resolve. Much of fool.com is server side rendered. What You'll Do * Build and ...

Its product portfolio includes crawler, wheel, and mini excavators, wheel loaders, dozers ... • Translate engineering specs and product briefs into compelling content that dealers and ...

next page

Showing results 1-20

Web Crawler Developer information

See salary details

$16

$45

$72

How much do web crawler developer jobs pay per hour?

As of Jun 17, 2026, the average hourly pay for web crawler developer in the United States is $45.12, according to ZipRecruiter salary data. Most workers in this role earn between $34.62 and $54.57 per hour, depending on experience, location, and employer.

What is the difference between Web Crawler Developer vs Data Engineer?

AspectWeb Crawler DeveloperData Engineer
Required CredentialsBachelor's in Computer Science, programming skillsBachelor's in CS, data management certifications
Work EnvironmentWeb development, scripting, backend systemsData pipelines, database management, cloud platforms
Industry UsageWeb scraping, search engines, data collectionData processing, analytics, big data infrastructure
Search & Comparison IntentFocus on web crawling, data extractionFocus on data storage, transformation, and analysis

The Web Crawler Developer primarily designs and maintains web crawlers to extract data from websites, focusing on web scraping and data collection. In contrast, Data Engineers build and manage data pipelines and infrastructure to process and store large datasets. While both roles require programming skills and work with data, Web Crawler Developers specialize in web-based data extraction, whereas Data Engineers focus on data integration and management across systems.

What are the key skills and qualifications needed to thrive as a Web Crawler Developer, and why are they important?

To thrive as a Web Crawler Developer, you need strong programming skills in languages like Python or Java, experience with web protocols, and a solid understanding of data extraction and parsing techniques. Familiarity with tools such as Scrapy, BeautifulSoup, Selenium, and knowledge of APIs and database management are commonly required, along with expertise in handling large-scale data. Attention to detail, problem-solving abilities, and effective communication help developers manage complex crawling tasks and collaborate with teams. These skills are crucial to efficiently gather, process, and maintain high-quality web data while navigating technical and ethical challenges.

What are some common challenges Web Crawler Developers face when extracting data from modern websites?

Web Crawler Developers often encounter challenges such as dynamic content loaded via JavaScript, frequent changes to website structures, and anti-scraping measures like CAPTCHAs or rate limiting. Staying up-to-date with new web technologies and adapting crawlers to handle Single Page Applications (SPAs) is crucial. Additionally, collaborating closely with data analysts or product teams ensures that the extracted data meets quality and compliance standards.

What is a Web Crawler Developer?

A Web Crawler Developer is a software engineer who specializes in designing, building, and maintaining web crawlers or web scraping tools. These tools automatically browse the internet and extract data from websites for various applications, such as search engines, data analytics, and market research. Web Crawler Developers often work with programming languages like Python, Java, or Node.js and must be knowledgeable about web protocols, data parsing, and ethical considerations related to web scraping.
More about Web Crawler Developer jobs
Infographic showing various Web Crawler Developer job openings in the United States as of June 2026, with employment types broken down into 91% Full Time, and 9% Part Time. Highlights an 6% Physical, and 94% Hybrid job distribution, with an average salary of $93,848 per year, or $45.1 per hour.

Senior Product Architect

Contentful

Denver, CO • On-site

Other

PTO

Posted 8 days ago


Job description

About the Opportunity

The Customer Insights & Adoption (CIA) team is hiring a Senior Product Architect, Internal Tools (f/m/d) to design and build internal products that help Contentful teams work smarter. This role serves internal product development, turning ambiguous workflow problems into useful systems that improve how Customer Experience, Sales, and Partner-facing teams operate every day.

This is a hands-on role with approximately 60% implementation and 40% architecture and planning. You will actively contribute production code while owning technical design and direction for a focused set of internal tools that support pipeline, retention, and customer experience outcomes. You'll prototype, iterate, and operationalize internal systems that are lightweight where they should be, durable where they need to be, and designed for clear long-term ownership.

If you are excited by the idea of shipping high-leverage internal tools, bringing order to messy workflows, and translating promising prototypes into systems people actually rely on, this role offers the chance to have visible impact across the business.

Why this role matters

CIA's internal tools work is part of how we turn customer, field, and partner insights into operational advantage. The right tools help our teams move faster, reduce manual work, improve data quality, and create a more consistent way of supporting customers and partners. This role exists to make those improvements real - not as one-off experiments, but as well-scoped internal products and patterns that deliver demonstrable value for our internal teams.

  • Join a high-impact team building internal products that improve customer experience, strengthen field effectiveness, and help Contentful scale more effectively
  • Shape how CIA approaches internal product development, R&D, and technical architecture - with meaningful room to influence both the systems we build and the way we build them
  • Work on a wide range of interesting problems, from internal apps and integrations to structured data workflows, experimentation, and AI-assisted tooling
  • Partner with deeply cross-functional stakeholders across CX, Sales, Partnerships, Product, IT, and Platform teams, with visible impact on how Contentful operates internally
What to expect? 

This is a hands-on role with approximately 60% implementation and 40% architecture and planning

  • Own and evolve a focused set of internal tools, utilities, and integrations that support the Customer Experience organization (CSMs, Support, Professional Services, Learning Services) and cross-functional internal users such as Sales, Partnerships, and Pricing & Packaging
  • Lead R&D and rapid prototyping efforts to identify high-leverage solutions, then determine which ideas should remain lightweight, which should be operationalized, and which should be designed for clean handoff to other teams
  • Build production-quality front-end and back-end code, including API design, data flows, resilience, observability, and robust internal user experiences that align with our design system and engineering standards
  • Apply component-based and front-end best practices to ensure internal tools are consistent, scalable, and maintainable
  • Leverage Contentful and other internal platforms where appropriate to create reusable patterns and scalable internal solutions rather than reinventing common capabilities from scratch
  • Explore and implement web crawler and scraper approaches, structured extraction workflows, and other internal data-ingestion patterns
  • Explore and implement AI-assisted workflows where relevant to improve internal tools, accelerate development, and support pragmatic experimentation without compromising quality or maintainability
  • Collaborate with IT, InfoSec, Platform, Legal, and Privacy partners on responsible data handling, hosting, identity, compliance, and operational readiness
  • Define success metrics, instrument telemetry, report impact, and document architecture & playbooks so that stakeholders understand where they unlock value for Sales, CSMs, Professional Services, and adjacent teams, and how other teams can support or extend the systems over time
What success looks like
  • Customer Experience, Sales, and partner-facing internal teams rely on a focused set of internal tools that are easy to use, well adopted, and meaningfully improve how work gets done across the business
  • Internal tools built by this role are consistent, scalable, and reliable, reflecting strong front-end patterns, sound API and data-flow design, and user experiences that align with Contentful's design system and internal platform standards
  • Stakeholders have clear visibility into the impact of CIA-built systems through trusted success metrics such as adoption, time saved, data quality, and reliability, and use those signals to prioritize future investments
  • Cross-functional partners experience CIA as a pragmatic, high-leverage internal product and architecture function that can take ambiguous problems, define the right solution, and deliver useful systems that support pipeline, retention, and customer experience outcomes
  • Reusable patterns, technical documentation, and operating playbooks make it easier for adjacent teams to build on this work, support it over time, and extend internal tooling without accumulating unnecessary complexity or one-off solutions
What you need to be successful
  • Demonstrated ability to balance system design with hands-on implementation, including recent experience contributing to production codebases
  • 7-10 years building and operating production web applications
  • Hands-on expertise with modern TypeScript, React and Next.js (or equivalent), state management, testing, performance, and accessibility best practices
  • Back-end experience with Node.js and TypeScript, API design, eventing or queues, and serverless or containerized services, along with sound observability practices across logs, metrics, and traces
  • Experience with Contentful or similar API-first CMS and composable content platforms
  • Ability to translate ambiguous requirements into clear technical designs and scalable implementations, engineered for maintainability and ownership transfer
  • Experience designing web application architectures, data flows, and integration points across multiple systems, with judgment about when to build a full product, when to automate, and when to keep a solution intentionally small
  • Experience establishing lightweight engineering practices for a small team, including clear documentation, code review norms, testing approaches, release discipline, and operational readiness
  • Mentorship and technical leadership that raises the bar on code quality, design rigor, and documentation while helping peers make better architectural decisions
  • Clear written and verbal communication with technical and non-technical stakeholders, along with a bias for action and evidence-based decision making
Nice to Have
  • Practical experience with AI and LLM-based workflows, particularly spec-based development, evaluation, and test or guardrail processes
  • Experience working with component-based front-end architectures and design systems, including design tokens, component libraries, CSS architecture, accessibility, and Figma design and development experience
  • Familiarity with web crawling and scraping approaches, including structured content extraction, normalization, and change detection
  • Experience building internal tools, dashboards, or workflow systems for customer-facing, revenue, or operations teams
What's in it for you?
  • Join an ambitious tech company reshaping the way people build digital experiences
  • Full-time employees receive equity awards for the opportunity to share in the success of our company
  • Comprehensive healthcare package covering 100% of monthly health premiums for employees and 85% of costs for your dependents. 
  • Fertility and family building benefits, including a lifetime reimbursable wallet to support your growing family.
  • We value Work-Life balance and You Time! A generous amount of paid time off, including vacation days, sick days, compassion days for loss,  education days, and volunteer days
  • Company paid parental leave to care for and focus on your growing family 
  • Use your personal annual education budget to improve your skills and grow in your career
  • Enjoy a full range of virtual and in-person events, including workshops, guest speakers, and fun team activities, supporting learning and networking exchange beyond the usual work duties 
  • An annual wellbeing stipend to care for your physical, financial, or emotional health
  • A monthly communication stipend and phone hardware upgrade reimbursement.
  • New hire office equipment stipend for hybrid or distributed employees. Get the gear you need to work at your best.