ZipRecruiter

Log In

1

Web Crawler Jobs (NOW HIRING)

StackandCo

$42.50 - $56.25/hr

\n \n \n \n \n Senior Fullstack Developer needed for an impressive Irish tech company building a world leading bot\/web crawler detection product. \n \n \n \n \n \n We need engineers who are ...

StackandCo

\n \n \n \n \n Senior Fullstack Developer needed for an impressive Irish tech company building a world leading bot\/web crawler detection product. \n \n \n \n \n \n We need engineers who are ...

NTENT

Java Developer (Remote Permanent Opportunity)

$52.25 - $67.50/hr

Ingestion Pipeline: process data that comes from our web crawler which discovers and fetches content from the web and other sources Skills and Qualifications: * Bachelors of equivalent years of ...

NTENT

Java Developer (Remote Permanent Opportunity)

$52.25 - $67.50/hr

Ingestion Pipeline: process data that comes from our web crawler which discovers and fetches content from the web and other sources Skills and Qualifications: * Bachelors of equivalent years of ...

Microsoft

Principal Software Engineer

Redmond, WA · On-site

$188K - $304.20K/yr

We are looking for a Principal Software Engineer to help us further scale our platform in Web Crawler team that is fundamental to all Search and Recommendation scenarios. Today we crawl tens of ...

NTENT

Senior Java Engineer

$58.50 - $74.75/hr

Ingestion Pipeline: process data that comes from our web crawler which discovers and fetches content from the web and other sources Skills and Qualifications: * BS or MS degree in Computer Science

NTENT

Senior Java Engineer

$58.50 - $74.75/hr

Ingestion Pipeline: process data that comes from our web crawler which discovers and fetches content from the web and other sources Skills and Qualifications: * BS or MS degree in Computer Science

Wynd Labs

Web Scraping Specialist

New York, NY

$102.80K - $103.40K/yr

We are at at the forefront of developing a web-scale crawler and knowledge graph that allows ordinary people to participate in the process, and share in the benefits of AI development. * Culture. We ...

Contentful

Senior Product Architect

New York, NY · On-site

Explore and implement web crawler and scraper approaches, structured extraction workflows, and other internal data-ingestion patterns * Explore and implement AI-assisted workflows where relevant to ...

Contentful

Senior Product Architect

Denver, CO

Explore and implement web crawler and scraper approaches, structured extraction workflows, and other internal data-ingestion patterns * Explore and implement AI-assisted workflows where relevant to ...

Contentful

Senior Product Architect

San Francisco, CA · On-site

Explore and implement web crawler and scraper approaches, structured extraction workflows, and other internal data-ingestion patterns * Explore and implement AI-assisted workflows where relevant to ...

Contentful

Senior Product Architect

New York, NY · On-site

Explore and implement web crawler and scraper approaches, structured extraction workflows, and other internal data-ingestion patterns * Explore and implement AI-assisted workflows where relevant to ...

Contentful

Senior Product Architect

San Francisco, CA · On-site

Explore and implement web crawler and scraper approaches, structured extraction workflows, and other internal data-ingestion patterns * Explore and implement AI-assisted workflows where relevant to ...

next page

Showing results 1-20

People also search for

All JobsWeb Crawler Jobs

Web Crawler information

See salary details

$39K

$80.9K

$144K

How much do web crawler jobs pay per year?

As of May 29, 2026, the average yearly pay for web crawler in the United States is $80,851.00, according to ZipRecruiter salary data. Most workers in this role earn between $55,000.00 and $94,500.00 per year, depending on experience, location, and employer.

What is a Web Crawler job?

A Web Crawler job involves developing and maintaining automated scripts or programs that systematically browse the internet to extract, index, and analyze data from websites. Professionals in this role typically work with programming languages like Python and use libraries such as Scrapy or BeautifulSoup. They ensure data is collected efficiently while adhering to legal and ethical guidelines, such as respecting robots.txt rules. Web Crawlers are used for various applications, including search engine indexing, market research, and competitive analysis.

What are the key skills and qualifications needed to thrive in the Web Crawler position, and why are they important?

To thrive as a Web Crawler, you need strong programming skills (often in Python, Java, or similar languages), familiarity with data extraction techniques, and a solid understanding of web protocols and data structures. Experience with web scraping frameworks such as Scrapy, BeautifulSoup, or Selenium, and knowledge of APIs is highly valued, and some roles may prefer candidates with relevant certifications in data engineering or web technologies. Attention to detail, problem-solving ability, and adaptability are important soft skills that help you overcome changing web structures and handle data quality challenges. These combined skills ensure you can efficiently gather accurate, large-scale web data while navigating technical and legal constraints.

What are the typical daily responsibilities of a Web Crawler?

As a Web Crawler, your daily tasks may include designing and developing automated scripts or programs to extract data from a variety of websites, monitoring data collection processes for accuracy, and troubleshooting access or formatting issues. You’ll often collaborate with data analysts, data scientists, or software engineers to ensure the data fits project needs and complies with current web policies and ethical standards. Regular maintenance of existing crawlers, updating them to accommodate changes in website structures, and documenting your work are also important aspects of the role. A detail-oriented approach and strong communication skills are key to succeeding in these activities.

Is ChatGPT a web crawler?

ChatGPT is an AI language model developed by OpenAI and is not a web crawler. Web crawlers are automated programs that systematically browse the internet to index content, whereas ChatGPT generates responses based on pre-existing training data without actively browsing the web.
More about Web Crawler jobs
What are the most commonly searched types of Web Crawler jobs? The most popular types of Web Crawler jobs are:
What states have the most Web Crawler jobs? States with the most job openings for Web Crawler jobs include:
What job categories do people searching Web Crawler jobs look for? The top searched job categories for Web Crawler jobs are:
Web Crawler jobs near you
Infographic showing various Web Crawler job openings in the United States as of May 2026, with employment types broken down into 50% Full Time, and 50% Temporary. Highlights an 12% Physical, and 88% Hybrid job distribution, with an average salary of $80,851 per year, or $38.9 per hour.

Senior Fullstack Developer

StackandCo

On-site

$42.50 - $56.25/hr

Full-time

Posted 26 days ago

Job description

\n <\/head>\n \n \n
\n Senior Fullstack Developer needed for an impressive Irish tech company building a world leading bot\/web crawler detection product.<\/span>
\n <\/div>\n
\n
\n <\/div>\n
\n We need engineers who are particularly strong Java developers with extensive frontend exposure.<\/span>
\n <\/div>\n
\n
\n <\/div>\n
\n You will join a tight\-knit group who love building product and have a keen interest in data driven applications(advantage if you have experience.<\/span>
\n <\/div>\n
\n
\n <\/div>\n
\n Experience
<\/u><\/b>\n <\/div>\n
    \n
  • 6+ years' Java\/Spring Development experience
    <\/li>\n
  • Spring Boot experience
    <\/li>\n
  • Strong frontend experience with any JS framework (React would be an added bonus)
    <\/li>\n
  • Fast learner and creative
    <\/li>\n
  • Strong skills in relational databases
    <\/li>\n
  • Experience working on data driven applications would be great
    <\/li>\n
  • Strong communicator, happy to voice opinions
    <\/li>\n <\/ul>\n
    \n
    \n <\/div><\/span>
    \n <\/body>\n<\/html>

Apply