ZipRecruiter

Log In

1

Web Crawler Jobs (NOW HIRING)

CTAC

Sr. DevOps Engineer

Falls Church, VA · On-site

$138K - $178K/yr

This role requires experience supporting scalable web platforms, cloud-native services, search and ... Supporting search indexing, content discovery, chatbot, crawler, and text-processing services ...

MLabs

Network Infrastructure Engineer (DevOps)

Ashburn, VA · Hybrid

$125K - $150K/yr

The client operates a vast distributed crawler, granting unique access to high-quality public web data at a global scale. Their work involves building advanced pipelines for ingesting, segmenting ...

MLabs

Network Infrastructure Engineer (DevOps)

Ashburn, VA · On-site

$125K - $150K/yr

The client operates a vast distributed crawler, granting unique access to high-quality public web data at a global scale. Their work involves building advanced pipelines for ingesting, segmenting ...

MLabs

Network Infrastructure Engineer (DevOps)

Ashburn, VA · On-site

$125K - $150K/yr

The client operates a vast distributed crawler, granting unique access to high-quality public web data at a global scale. Their work involves building advanced pipelines for ingesting, segmenting ...

Western States

Field Technician

Pasco, WA · On-site

$25 - $51/hr

Uses web based STW, SIS, and ET effectively to address technical issues. Performs all required ... Knowledge of scrapers and crawler tractors, loaders, excavators, off highway trucks, preferred.

Semrush

  • PTO

Understanding of web technologies, web crawlers, or HTML structure. * SQL skills for data ... Tim Team, which builds and maintains the enterprise website crawler using the Chrome/Lighthouse ...

RH

Product Leader, Growth & Acquisition

Corte Madera, CA · On-site +1

$120K - $160K/yr

Define how we measure AI, using log data, crawler activity, and emerging tooling to turn a murky ... Web Vitals, large product catalogs, faceted navigation, and complex ecommerce environments.

Western States CAT

Field Technician

Kalispell, MT · On-site

$19.50 - $26.75/hr

Uses web based STW, SIS, and ET effectively to address technical issues. Performs all required ... Knowledge of scrapers and crawler tractors, loaders, excavators, off highway trucks, preferred.

Western States

Field Technician

Kalispell, MT · On-site

$19.50 - $26.75/hr

Uses web based STW, SIS, and ET effectively to address technical issues. Performs all required ... Knowledge of scrapers and crawler tractors, loaders, excavators, off highway trucks, preferred.

Vagaro

AI Search & SEO Manager

Pleasanton, CA · On-site

$140K - $174K/yr

  • Medical

  • Dental

  • Vision

  • Life

  • Retirement

  • PTO

... crawler access policies for LLM bots * Define and track emerging AI search metrics LLM citation ... Lead technical SEO crawlability, core web vitals, schema/structured data, site architecture in ...

Vagaro

AI Search & SEO Manager

Pleasanton, CA · On-site

$140K - $174K/yr

  • Medical

  • Dental

  • Vision

  • Life

  • Retirement

  • PTO

... and crawler access policies for LLM bots * Define and track emerging AI search metrics - LLM ... Lead technical SEO - crawlability, core web vitals, schema/structured data, site architecture - in ...

Western States

Resident Field Technician

Butte, MT · On-site

Uses web based STW, SIS, and ET effectively to address technical issues. Performs all required ... Knowledge of scrapers and crawler tractors, loaders, excavators, off highway trucks, preferred.

Western States

Resident Field Technician

Pasco, WA · On-site

$25 - $51/hr

Uses web based STW, SIS, and ET effectively to address technical issues. Performs all required ... Knowledge of scrapers and crawler tractors, loaders, excavators, off highway trucks, preferred.

previous page

Showing results 41-60

All JobsBusiness JobsMarketing JobsWeb Crawler Jobs

Web Crawler information

See salary details

$39K

$80.9K

$144K

How much do web crawler jobs pay per year?

As of Aug 17, 2026, the average yearly pay for web crawler in the United States is $80,851.00, according to ZipRecruiter salary data. Most workers in this role earn between $55,000.00 and $94,500.00 per year, depending on experience, location, and employer.

What is a web crawler?

A Web Crawler job involves developing and maintaining automated scripts or programs that systematically browse the internet to extract, index, and analyze data from websites. Professionals in this role typically work with programming languages like Python and use libraries such as Scrapy or BeautifulSoup. They ensure data is collected efficiently while adhering to legal and ethical guidelines, such as respecting robots.txt rules. Web Crawlers are used for various applications, including search engine indexing, market research, and competitive analysis.

What does a web crawler do?

As a Web Crawler, your daily tasks may include designing and developing automated scripts or programs to extract data from a variety of websites, monitoring data collection processes for accuracy, and troubleshooting access or formatting issues. You’ll often collaborate with data analysts, data scientists, or software engineers to ensure the data fits project needs and complies with current web policies and ethical standards. Regular maintenance of existing crawlers, updating them to accommodate changes in website structures, and documenting your work are also important aspects of the role. A detail-oriented approach and strong communication skills are key to succeeding in these activities.

What are the key skills and qualifications needed to thrive in the web crawler position, and why are they important?

To thrive as a Web Crawler, you need strong programming skills (often in Python, Java, or similar languages), familiarity with data extraction techniques, and a solid understanding of web protocols and data structures. Experience with web scraping frameworks such as Scrapy, BeautifulSoup, or Selenium, and knowledge of APIs is highly valued, and some roles may prefer candidates with relevant certifications in data engineering or web technologies. Attention to detail, problem-solving ability, and adaptability are important soft skills that help you overcome changing web structures and handle data quality challenges. These combined skills ensure you can efficiently gather accurate, large-scale web data while navigating technical and legal constraints.

More about Web Crawler jobs

What are the most commonly searched types of Web Crawler jobs?

The most popular types of Web Crawler jobs are:

What states have the most Web Crawler jobs?

States with the most job openings for Web Crawler jobs include:

What job categories do people searching Web Crawler jobs look for?

The top searched job categories for Web Crawler jobs are:

Web Crawler jobs near you
Infographic showing various Web Crawler job openings in the United States as of August 2026, with employment types broken down into 67% Full Time, and 33% Contract. Highlights an 100% In-person job distribution, with an average salary of $80,851 per year, or $38.9 per hour.

Sr. DevOps Engineer

CTAC

Falls Church, VA • On-site

$138K - $178K/yr

Full-time

This job post has expired today. Applications are no longer accepted.

Job description

Job Title
Sr. DevOps Engineer
# of Hires Needed
1
Date Needed By
8/31/2026
Category
Information Technology
Education
Bachelor's Degree
Career Level
Experienced (Non-Manager)
Job Type
Full-time
Location
CTAC HQ - Falls Church, VA 22042 US (Primary)
Travel
0 - 10%
Job Description
The Full Stack Developer / Cloud Application Engineer will support the operations, maintenance (O&M), enhancement, and continuous modernization of enterprise web applications, data platforms, search services, and cloud-hosted digital solutions operating within a secure federal cloud environment.
The successful candidate will be responsible for maintaining and enhancing multiple interconnected systems built on highly available, containerized architectures running in AWS cloud environments. The applications are heavily dependent on Docker-based workloads deployed on AWS EC2 infrastructure, and the ideal candidate will have strong hands-on experience supporting, troubleshooting, and optimizing Dockerized applications in production environments. Experience with container orchestration and modernization approaches leveraging Amazon ECS or similar technologies is highly desired as part of the long-term modernization roadmap.
This role requires experience supporting scalable web platforms, cloud-native services, search and indexing solutions, content management systems, APIs, chatbot and text-processing services, and data visualization platforms. The candidate should have practical experience implementing application scaling techniques, performance optimization strategies, high-availability architectures, and operational automation to support mission-critical public-facing systems.
This position will work closely with project managers, cloud engineers, business analysts, cybersecurity personnel, content teams, and stakeholders to deliver reliable, scalable, secure, and user-focused solutions. The candidate will support both day-to-day operational activities and long-term modernization initiatives, including application enhancements, platform upgrades, automation, cloud optimization, and performance improvements.
Key responsibilities include:
  • Supporting operations, maintenance, troubleshooting, and enhancement activities across multiple enterprise web applications and cloud-hosted systems
  • Designing, developing, testing, and deploying new features and functionality for web applications, APIs, search platforms, and data-driven services
  • Supporting modernization initiatives involving containerization, automation, cloud-native architectures, and application optimization
  • Managing and maintaining applications running in Linux-based cloud environments utilizing container technologies and virtualized infrastructure
  • Developing and supporting applications built using open-source frameworks, middleware, and content management systems
  • Supporting search indexing, content discovery, chatbot, crawler, and text-processing services operating within distributed application environments
  • Working with relational databases, data integration processes, and data visualization platforms to support reporting and analytics capabilities
  • Supporting CI/CD pipelines, release management, deployment automation, and environment configuration management
  • Monitoring application performance, system health, logging, and operational metrics while proactively identifying areas for improvement
  • Coordinating with cloud infrastructure and security teams to ensure systems comply with federal security, accessibility, and operational requirements
  • Assisting with remediation of vulnerabilities, patch validation, application hardening, and continuous monitoring activities in accordance with federal compliance standards
  • Supporting content publishing workflows and integrations across internal and public-facing systems
  • Troubleshooting production issues, performing root cause analysis, and implementing corrective actions to improve reliability and resiliency
  • Creating and maintaining technical documentation including architecture diagrams, deployment procedures, operational runbooks, and system configurations
  • Collaborating with cross-functional teams in Agile or hybrid development environments to support iterative delivery and continuous improvement

The candidate must be comfortable operating in a highly collaborative environment and interfacing directly with client leadership, technical teams, and external stakeholders both virtually and in person.
This position may incur regular travel to/from Falls Church, Va. and Rockville, Md. The position will be located at our headquarters in Falls Church, Va. CTAC supports a flexible hybrid remote/in-office schedule.
Job Requirements
Required Qualifications:
  • Bachelor's degree in Computer Science, Information Technology, Software Engineering, or related technical discipline
  • 5+ years of experience supporting enterprise web applications, cloud-hosted systems, or full stack software development initiatives
  • Strong experience with modern web development technologies and frameworks, including front-end and back-end development concepts
  • Strong hands-on experience with Docker containerization technologies in AWS EC2 environments
  • Experience supporting scalable, highly available web applications and implementing application scaling and performance optimization techniques
  • Experience developing and supporting applications hosted in AWS environments
  • Experience with web servers, middleware technologies, APIs, and application integration patterns
  • Experience supporting relational databases and data-driven applications
  • Experience with scripting and automation for deployments, monitoring, and operational support
  • Understanding of CI/CD pipelines, source control, release management, and DevOps practices
  • Experience troubleshooting complex application, infrastructure, and integration issues across distributed environments
  • Understanding of cloud infrastructure concepts including compute, storage, networking, load balancing, and security controls
  • Familiarity with application monitoring, logging, and performance optimization tools
  • Strong understanding of secure coding practices, vulnerability remediation, and application security principles
  • Experience working within the Software Development Lifecycle (SDLC), including requirements analysis, testing, deployment, and production support
  • Strong written and verbal communication skills with the ability to work collaboratively across technical and non-technical teams
  • Ability to manage multiple priorities and support both operational maintenance and modernization efforts simultaneously

Desired Skills:
  • Experience with Drupal or other enterprise content management systems (CMS)
  • Familiarity with PHP, Python, JavaScript, or other modern programming languages commonly used in web application development
  • Experience with AWS services such as EC2, RDS, S3, CloudWatch, IAM, Lambda, or related cloud-native services
  • Familiarity with autoscaling strategies, load balancing, and cloud-native modernization approaches within AWS environments
  • Experience supporting search technologies, indexing platforms, chatbot services, or text extraction tools
  • Familiarity with data visualization and analytics platforms such as Tableau or similar reporting technologies
  • Experience working within federal government or highly regulated IT environments
  • Understanding of federal security and compliance frameworks including FISMA, NIST, and FedRAMP
  • Familiarity with accessibility standards such as Section 508 compliance
  • Experience with Agile development methodologies and tools such as Jira and Confluence
  • Knowledge of CDN technologies, secure remote access solutions, and network security best practices
  • Strong analytical and problem-solving skills with high attention to detail
  • Ability to quickly learn new technologies and adapt to evolving technical requirements

Clearance Required: Ability to obtain and maintain a Level 5 Public Trust Clearance
CTAC is an equal opportunity employer. Qualified applicants will receive consideration for employment without regard to race, color, religion, sex, age, sexual orientation, gender identity, national origin, disability, or protected veteran status. VEVRAA Federal Contractor

Apply