Laurel, MD · On-site
$160K - $200K/yr
... and optimize web scraping and crawler infrastructure to support continuous data collection ... engineering environment with evolving priorities - Must be authorized to work in the United States ...
Quick apply
Laurel, MD · On-site
$160K - $200K/yr
... and optimize web scraping and crawler infrastructure to support continuous data collection ... engineering environment with evolving priorities - Must be authorized to work in the United States ...
Denver, CO · On-site
... and engineering standards * Apply component-based and front-end best practices to ensure internal ... Explore and implement web crawler and scraper approaches, structured extraction workflows, and ...
Denver, CO · On-site
... and engineering standards * Apply component-based and front-end best practices to ensure internal ... Explore and implement web crawler and scraper approaches, structured extraction workflows, and ...
New York, NY · On-site
... and engineering standards * Apply component-based and front-end best practices to ensure internal ... Explore and implement web crawler and scraper approaches, structured extraction workflows, and ...
New York, NY · On-site
... and engineering standards * Apply component-based and front-end best practices to ensure internal ... Explore and implement web crawler and scraper approaches, structured extraction workflows, and ...
... and engineering standards * Apply component-based and front-end best practices to ensure internal ... Explore and implement web crawler and scraper approaches, structured extraction workflows, and ...
... and engineering standards * Apply component-based and front-end best practices to ensure internal ... Explore and implement web crawler and scraper approaches, structured extraction workflows, and ...
... resilient web crawling and automation pipelines for provider credentialing data collection • ... • Writing Crawler experience (such as: Scrapy, Apache Nutch, Heritrix) • Familiarity with ...
... resilient web crawling and automation pipelines for provider credentialing data collection • ... • Writing Crawler experience (such as: Scrapy, Apache Nutch, Heritrix) • Familiarity with ...
$125K - $165K/yr
... crawler using Playwright and Stagehand . * Tackle the gnarly realities of modern web apps: SPA ... Collaborate with the attack-team engineers who consume your crawl output, and help shape the graph ...
$125K - $165K/yr
... crawler using Playwright and Stagehand . * Tackle the gnarly realities of modern web apps: SPA ... Collaborate with the attack-team engineers who consume your crawl output, and help shape the graph ...
OR · On-site +1
$122K - $161K/yr
... web - and the systems that turn readers into long-term members. What Does This Team Do? The Fool ... crawler requests resolve. Much of fool.com is server side rendered. What You'll Do * Build and ...
OR · On-site +1
$122K - $161K/yr
... web - and the systems that turn readers into long-term members. What Does This Team Do? The Fool ... crawler requests resolve. Much of fool.com is server side rendered. What You'll Do * Build and ...
$125K - $165K/yr
... web - and the systems that turn readers into long-term members. What Does This Team Do? The Fool ... crawler requests resolve. Much of fool.com is server side rendered. What You'll Do * Build and ...
$125K - $165K/yr
... web - and the systems that turn readers into long-term members. What Does This Team Do? The Fool ... crawler requests resolve. Much of fool.com is server side rendered. What You'll Do * Build and ...
$165 - $185/hr
... crawler behavior.* Deploy and operate services on AWS using Docker , with first-class ... the web -- and the systems that turn readers into long-term members.### **What Does This Team Do?
$165 - $185/hr
... crawler behavior.* Deploy and operate services on AWS using Docker , with first-class ... the web -- and the systems that turn readers into long-term members.### **What Does This Team Do?
Collaborate with SEO, Content, and Engineering teams. * Track KPIs and present performance insights ... Required Skills * 4+ years in Product Ownership, SEO, or Web Analytics. * Experience with AEO/GEO ...
Collaborate with SEO, Content, and Engineering teams. * Track KPIs and present performance insights ... Required Skills * 4+ years in Product Ownership, SEO, or Web Analytics. * Experience with AEO/GEO ...
... web engineering, data/structured-markup). * Own, groom, and prioritize the AEO/GEO product backlog; balance quick wins against structural improvements (structured data, llms.txt, crawler policy ...
... web engineering, data/structured-markup). * Own, groom, and prioritize the AEO/GEO product backlog; balance quick wins against structural improvements (structured data, llms.txt, crawler policy ...
Redmond, WA · On-site
$188K - $304K/yr
Overview The Crawler Team, part of Microsoft Search's Web Data organization, builds and operates ... developer productivity. * Mentor senior engineers and drive engineering excellence through ...
Redmond, WA · On-site
$188K - $304K/yr
Overview The Crawler Team, part of Microsoft Search's Web Data organization, builds and operates ... developer productivity. * Mentor senior engineers and drive engineering excellence through ...
Ashburn, GA · On-site
$125 - $150/hr
The client operates a vast distributed crawler, granting unique access to high-quality public web ... * DevOps Integration: Collaborate closely with engineering teams to integrate DevOps best ...
Ashburn, GA · On-site
$125 - $150/hr
The client operates a vast distributed crawler, granting unique access to high-quality public web ... * DevOps Integration: Collaborate closely with engineering teams to integrate DevOps best ...
Ashburn, VA · Hybrid
$125K - $150K/yr
The client operates a vast distributed crawler, granting unique access to high-quality public web ... * DevOps Integration: Collaborate closely with engineering teams to integrate DevOps best ...
Ashburn, VA · Hybrid
$125K - $150K/yr
The client operates a vast distributed crawler, granting unique access to high-quality public web ... * DevOps Integration: Collaborate closely with engineering teams to integrate DevOps best ...
Ashburn, VA · On-site
$125K - $150K/yr
The client operates a vast distributed crawler, granting unique access to high-quality public web ... * DevOps Integration: Collaborate closely with engineering teams to integrate DevOps best ...
Ashburn, VA · On-site
$125K - $150K/yr
The client operates a vast distributed crawler, granting unique access to high-quality public web ... * DevOps Integration: Collaborate closely with engineering teams to integrate DevOps best ...
Ashburn, VA · On-site
$125K - $150K/yr
The client operates a vast distributed crawler, granting unique access to high-quality public web ... * DevOps Integration: Collaborate closely with engineering teams to integrate DevOps best ...
Quick apply
Ashburn, VA · On-site
$125K - $150K/yr
The client operates a vast distributed crawler, granting unique access to high-quality public web ... * DevOps Integration: Collaborate closely with engineering teams to integrate DevOps best ...
Oregon, WI · On-site
$90 - $130/hr
This role focuses on developing scalable web crawling solutions that power Product Discovery ... Monitor crawler performance, identify failures, and resolve data extraction issues. * Collaborate ...
Oregon, WI · On-site
$90 - $130/hr
This role focuses on developing scalable web crawling solutions that power Product Discovery ... Monitor crawler performance, identify failures, and resolve data extraction issues. * Collaborate ...
The ingestion layer is the machinery that turns the open web, licensed corpora, and other large ... crawler architecture and politeness/scheduling, distributed processing (Ray/Beam/Spark ...
The ingestion layer is the machinery that turns the open web, licensed corpora, and other large ... crawler architecture and politeness/scheduling, distributed processing (Ray/Beam/Spark ...
Understanding of web technologies, web crawlers, or HTML structure. * SQL skills for data ... Tim Team, which builds and maintains the enterprise website crawler using the Chrome/Lighthouse ...
Understanding of web technologies, web crawlers, or HTML structure. * SQL skills for data ... Tim Team, which builds and maintains the enterprise website crawler using the Chrome/Lighthouse ...
Working cross-functionally with product, engineering, design, and analytics teams, you will help ... Define how we measure AI, using log data, crawler activity, and emerging tooling to turn a murky ...
Working cross-functionally with product, engineering, design, and analytics teams, you will help ... Define how we measure AI, using log data, crawler activity, and emerging tooling to turn a murky ...
$16.83 - $21.85
3% of jobs
$21.85 - $26.88
3% of jobs
$26.88 - $31.91
12% of jobs
$34.33 is the 25th percentile. Wages below this are outliers.
$31.91 - $36.93
15% of jobs
$36.93 - $41.96
16% of jobs
The median wage is $42.54 / hr.
$41.96 - $46.98
14% of jobs
$46.98 - $52.01
11% of jobs
$53.14 is the 75th percentile. Wages above this are outliers.
$52.01 - $57.04
11% of jobs
$57.04 - $62.06
8% of jobs
$62.06 - $67.09
5% of jobs
$67.09 - $72.12
3% of jobs
$16
$45
$72
| Aspect | Web Crawler Developer | Data Engineer |
|---|---|---|
| Required Credentials | Bachelor's in Computer Science, programming skills | Bachelor's in CS, data management certifications |
| Work Environment | Web development, scripting, backend systems | Data pipelines, database management, cloud platforms |
| Industry Usage | Web scraping, search engines, data collection | Data processing, analytics, big data infrastructure |
| Search & Comparison Intent | Focus on web crawling, data extraction | Focus on data storage, transformation, and analysis |
The Web Crawler Developer primarily designs and maintains web crawlers to extract data from websites, focusing on web scraping and data collection. In contrast, Data Engineers build and manage data pipelines and infrastructure to process and store large datasets. While both roles require programming skills and work with data, Web Crawler Developers specialize in web-based data extraction, whereas Data Engineers focus on data integration and management across systems.
The top searched job categories for Web Crawler Developer jobs are:
$160K - $200K/yr
Contractor
Re-posted 19 days ago
Join Hire Hangar and work with fast-growing global companies while building a long-term career.
Job Title| Software Engineer, Graph Systems & Data Visualization
Location | Remote (US Only)
Time Zone | US Time Zones (EST–PST)
Role Overview:
We are seeking a Software Engineer to help build and scale a graph-database-backed social listening platform. This role sits at the intersection of backend data infrastructure and front-end visualization, supporting a product that ingests high-volume social and web data in real time and visualizes complex network relationships. The ideal candidate is comfortable working across graph databases, streaming data pipelines, and modern front-end frameworks, and thrives in a fast-moving, early-stage engineering environment.
Key Responsibilities:
- Design, build, and maintain backend services that interface with graph databases to model and query complex relationship data
- Develop and maintain real-time data ingestion pipelines handling high-throughput social and web data
- Build and optimize web scraping and crawler infrastructure to support continuous data collection
- Develop front-end features and dashboards that visualize network and relationship data in an intuitive, performant way
- Collaborate cross-functionally to translate product requirements into scalable technical solutions
- Write clean, maintainable, well-tested code across the stack
- Troubleshoot performance bottlenecks across data ingestion, storage, and visualization layers
Required Qualifications:
- Hands-on experience with graph databases (such as Neo4j, TigerGraph, Dgraph, ArangoDB, or Amazon Neptune) and graph query languages (such as Cypher or Gremlin)
Experience with real-time/streaming data infrastructure (such as Kafka, Kinesis, Flink, or Spark)
- Strong proficiency in at least one backend language such as Python, Go, or Node.js/TypeScript
- Front-end development experience with React, Next.js, and TypeScript
- Experience building or working with data pipelines, ETL processes, or web scraping/crawler infrastructure
- Comfortable working in a high-growth, early-stage engineering environment with evolving priorities
- Must be authorized to work in the United States without sponsorship
Preferred Qualifications:
- Experience with data visualization libraries such as D3, Cytoscape, Sigma.js, vis.js, or Recharts, particularly for graph/network visualizations
- Experience with social listening, social media data, or large-scale web data platforms
- Familiarity with Elasticsearch, OpenSearch, Redis, or Airflow
- Prior experience at a seed-to-Series B startup
- Experience with AWS, GCP, Docker, or Kubernetes
Tools & Technology:
Graph database platforms (Neo4j, TigerGraph, Dgraph, ArangoDB, or similar)
Streaming/data pipeline tools (Kafka, Spark, Flink, or similar)
React, Next.js, TypeScript
Python, Go, or Node.js
Cloud infrastructure (AWS/GCP), Docker, Kubernetes
We connect top talent with vetted employers, competitive pay, and real growth opportunities.
Compensation Range: $160K - $200K
