ZipRecruiter

Log In

1

Web Crawler Jobs (NOW HIRING)

Wynd Labs

Senior Software Engineer (Backend)

New York, NY · On-site

$134K - $176.70K/yr

We are at at the forefront of developing a web-scale crawler and knowledge graph that allows ordinary people to participate in the process, and share in the benefits of AI development. * Culture. We ...

Wynd Labs

Policy & Advocacy Lead

New York, NY

We are at at the forefront of developing a web-scale crawler and knowledge graph that allows ordinary people to participate in the process, and share in the benefits, of AI development. * Culture. We ...

Wynd Labs

Full Stack Engineer

New York, NY

We are at at the forefront of developing a web-scale crawler and knowledge graph that allows ordinary people to participate in the process, and share in the benefits of AI development. * Culture. We ...

Wynd Labs

Policy & Advocacy Lead

New York, NY · On-site

We are at at the forefront of developing a web-scale crawler and knowledge graph that allows ordinary people to participate in the process, and share in the benefits, of AI development. * Culture. We ...

Wynd Labs

Smart Contract Engineer - SVM

New York, NY

We are at at the forefront of developing a web-scale crawler and knowledge graph that allows ordinary people to participate in the process, and share in the benefits of AI development. * Culture. We ...

Wynd Labs

Growth Associate

New York, NY

Opportunity. We are at the forefront of developing a web-scale crawler and knowledge graph in a way that allows ordinary people to participate in the process and share in the benefits of AI ...

Wynd Labs

Comms and PR Lead

New York, NY · On-site

$31.20K - $31.70K/yr

We are at at the forefront of developing a web-scale crawler and knowledge graph that allows ordinary people to participate in the process, and share in the benefits, of AI development. * Culture. We ...

Wynd Labs

Comms and PR Lead

New York, NY

$31.20K - $31.70K/yr

We are at at the forefront of developing a web-scale crawler and knowledge graph that allows ordinary people to participate in the process, and share in the benefits, of AI development. * Culture. We ...

Wynd Labs

Data Engineer

New York, NY

$125.30K - $150.40K/yr

We are at at the forefront of developing a web-scale crawler and knowledge graph that allows ordinary people to participate in the process, and share in the benefits of AI development. * Culture. We ...

Wynd Labs

Network Infrastructure Engineer (DevOps)

Ashburn, VA · Hybrid

$109.90K - $144.20K/yr

We are at at the forefront of developing a web-scale crawler and knowledge graph that allows ordinary people to participate in the process, and share in the benefits, of AI development. * Culture. We ...

MLabs

Network Infrastructure Engineer (DevOps)

Ashburn, VA · Hybrid

$125K - $150K/yr

The client operates a vast distributed crawler, granting unique access to high-quality public web data at a global scale. Their work involves building advanced pipelines for ingesting, segmenting ...

MLabs

Network Infrastructure Engineer (DevOps)

Ashburn, VA · On-site

$125K - $150K/yr

The client operates a vast distributed crawler, granting unique access to high-quality public web data at a global scale. Their work involves building advanced pipelines for ingesting, segmenting ...

MLabs

Network Infrastructure Engineer (DevOps)

Ashburn, VA · On-site

$125K - $150K/yr

The client operates a vast distributed crawler, granting unique access to high-quality public web data at a global scale. Their work involves building advanced pipelines for ingesting, segmenting ...

Develon

Product Marketing Specialist

Suwanee, GA · On-site

Its product portfolio includes crawler, wheel, and mini excavators, wheel loaders, dozers ... including web, email, print, and presentations • Ability to translate technical or product ...

next page

Showing results 1-20

People also search for

All JobsWeb Crawler Jobs

Web Crawler information

See salary details

$39K

$80.9K

$144K

How much do web crawler jobs pay per year?

As of Jun 1, 2026, the average yearly pay for web crawler in the United States is $80,851.00, according to ZipRecruiter salary data. Most workers in this role earn between $55,000.00 and $94,500.00 per year, depending on experience, location, and employer.

What is a Web Crawler job?

A Web Crawler job involves developing and maintaining automated scripts or programs that systematically browse the internet to extract, index, and analyze data from websites. Professionals in this role typically work with programming languages like Python and use libraries such as Scrapy or BeautifulSoup. They ensure data is collected efficiently while adhering to legal and ethical guidelines, such as respecting robots.txt rules. Web Crawlers are used for various applications, including search engine indexing, market research, and competitive analysis.

What are the key skills and qualifications needed to thrive in the Web Crawler position, and why are they important?

To thrive as a Web Crawler, you need strong programming skills (often in Python, Java, or similar languages), familiarity with data extraction techniques, and a solid understanding of web protocols and data structures. Experience with web scraping frameworks such as Scrapy, BeautifulSoup, or Selenium, and knowledge of APIs is highly valued, and some roles may prefer candidates with relevant certifications in data engineering or web technologies. Attention to detail, problem-solving ability, and adaptability are important soft skills that help you overcome changing web structures and handle data quality challenges. These combined skills ensure you can efficiently gather accurate, large-scale web data while navigating technical and legal constraints.

What are the typical daily responsibilities of a Web Crawler?

As a Web Crawler, your daily tasks may include designing and developing automated scripts or programs to extract data from a variety of websites, monitoring data collection processes for accuracy, and troubleshooting access or formatting issues. You’ll often collaborate with data analysts, data scientists, or software engineers to ensure the data fits project needs and complies with current web policies and ethical standards. Regular maintenance of existing crawlers, updating them to accommodate changes in website structures, and documenting your work are also important aspects of the role. A detail-oriented approach and strong communication skills are key to succeeding in these activities.

Is ChatGPT a web crawler?

ChatGPT is an AI language model developed by OpenAI and is not a web crawler. Web crawlers are automated programs that systematically browse the internet to index content, whereas ChatGPT generates responses based on pre-existing training data without actively browsing the web.
More about Web Crawler jobs
What are the most commonly searched types of Web Crawler jobs? The most popular types of Web Crawler jobs are:
What states have the most Web Crawler jobs? States with the most job openings for Web Crawler jobs include:
What job categories do people searching Web Crawler jobs look for? The top searched job categories for Web Crawler jobs are:
Web Crawler jobs near you
Infographic showing various Web Crawler job openings in the United States as of May 2026, with employment types broken down into 50% Full Time, and 50% Temporary. Highlights an 12% Physical, and 88% Hybrid job distribution, with an average salary of $80,851 per year, or $38.9 per hour.

Senior Software Engineer (Backend)

Wynd Labs

New York, NY • On-site

$134K - $176.70K/yr

Full-time

Posted 16 days ago

Job description

Senior Software Engineer (Backend)

$180k – $300k

Who We Are.

Wynd Labs is an early-stage startup that is on a mission to make public web data accessible for AI through contributions to Grass.

Grass is a network sharing application that allows users to share their unused bandwidth. Effectively, this is a residential proxy network that directly rewards individual residential IPs for the bandwidth they provide. Grass will route traffic equitably among its network and meter the amount of data that each node provides to fairly distribute rewards.

In non-technical terms: Grass unlocks everyone's ability to earn rewards by simply sharing their unused internet bandwidth on personal devices (laptops, smartphones).

This project is for those who lead with initiative and seek to challenge themselves and thrive on curiosity.

We operate with a lean, highly motivated team who revel in the responsibility that comes with autonomy. We have a flat organizational structure, the people making decisions are also the ones implementing them. We are driven by ambitious goals and a strong sense of urgency. Leadership is given to those who show initiative, consistently deliver excellence and bring the best out of those around them. Join us if you want to set the tone for a fair and equitable internet.

The Role.

We are looking for a Senior Software Engineer (Backend) who is experienced and brings deep expertise in backend development. You will join a small, talented team and lead efforts to architect scalable solutions, drive technical excellence, and support our vision for a future where Grass plays a critical role in shaping the internet's evolution.

Who You Are.

  • BS or MS degree in a STEM major or equivalent job experience required
  • 4+ years experience in software development and writing very high quality code that is robust and easy to maintain
  • Knowledge on how to build large scalable systems
  • Knowledge and experience in identifying and solving issues with intricate, live software systems, as well as in-depth familiarity with contemporary software development processes such as continuous integration and continuous deployment.
  • Professional or native English language proficiency

What You'll Be Doing.

  • Develop new features and software improvements
  • Test, deploy and debug backend API services
  • Write clear, tested, documented code
  • Create and review technical design, code, and documentation
  • Contribute to Wynd's infrastructure which includes mobile, desktop and server-side applications

Technologies Used.

  • Go, Node.js, Redis Clusters, Kubernetes, AWS (EKS, SQS, DynamoDB, Elasticache, S3, etc.)

Why Work With Us.

  • Opportunity. We are at at the forefront of developing a web-scale crawler and knowledge graph that allows ordinary people to participate in the process, and share in the benefits of AI development.
  • Culture. We're a lean team working together to achieve a very ambitious goal of improving access to public web data and distributing the value of AI to the people. We prioritize low ego and high output.
  • Compensation. You'll receive a competitive salary and equity package.

Apply