Overview Black Canyon Consulting (BCC) is searching for a Web Service Protection Engineer to ... Perform deep log analysis to identify overuse, scraping, abuse, DDoS, and attack patterns across ...
Overview Black Canyon Consulting (BCC) is searching for a Web Service Protection Engineer to ... Perform deep log analysis to identify overuse, scraping, abuse, DDoS, and attack patterns across ...
Manager of Data Science
Arlington, VA · On-site
They are comfortable working with APIs, web scraping, and SQL/no-SQL databases. They automate ... Partner with engineering, IT, and data platform teams to enable scalable, reliable, and well ...
Manager of Data Science
Arlington, VA · On-site
They are comfortable working with APIs, web scraping, and SQL/no-SQL databases. They automate ... Partner with engineering, IT, and data platform teams to enable scalable, reliable, and well ...
Experienced in using UiPath Studio, Orchestrator, Screen, Data & Web scraping techniques * Programming/coding skills in multiple areas preferred
Experienced in using UiPath Studio, Orchestrator, Screen, Data & Web scraping techniques * Programming/coding skills in multiple areas preferred
... Web scraping techniques Programming/coding skills in multiple areas preferred
... Web scraping techniques Programming/coding skills in multiple areas preferred
Data Engineer
$118K - $142K/yr
Data Engineer Location: Chantilly, VA Work Schedule: Full-Time, Onsite Clearance Required: Active ... Develop and maintain web scraping and data ingestion workflows to collect and process open-source ...
Quick apply
Data Engineer
$118K - $142K/yr
Data Engineer Location: Chantilly, VA Work Schedule: Full-Time, Onsite Clearance Required: Active ... Develop and maintain web scraping and data ingestion workflows to collect and process open-source ...
Engineering Fellowship
Washington, DC · On-site
Experience with web scraping/crawling (e.g., Beautiful Soup, Selenium, Scrapy) Machine Learning Engineering: * * Computer vision skills (OCR, image classification, deep fake detection) * Familiarity ...
Engineering Fellowship
Washington, DC · On-site
Experience with web scraping/crawling (e.g., Beautiful Soup, Selenium, Scrapy) Machine Learning Engineering: * * Computer vision skills (OCR, image classification, deep fake detection) * Familiarity ...
Data Engineer with Security Clearance
Chantilly, VA · On-site
$118K - $142K/yr
Data Engineer Location: Chantilly, VA Work Schedule: Full-Time, Onsite Clearance Required: Active ... Develop and maintain web scraping and data ingestion workflows to collect and process open-source ...
Data Engineer with Security Clearance
Chantilly, VA · On-site
$118K - $142K/yr
Data Engineer Location: Chantilly, VA Work Schedule: Full-Time, Onsite Clearance Required: Active ... Develop and maintain web scraping and data ingestion workflows to collect and process open-source ...
Python Tutor
College Park, MD · Remote
$18 - $40/hr
Emphasizes readable, maintainable code and connects Python to machine learning, web scraping, scientific computing, and DevOps applications. * Curriculum Awareness & Adaptive Instruction: Familiar ...
Python Tutor
College Park, MD · Remote
$18 - $40/hr
Emphasizes readable, maintainable code and connects Python to machine learning, web scraping, scientific computing, and DevOps applications. * Curriculum Awareness & Adaptive Instruction: Familiar ...
Python Tutor
Laurel, MD · Remote
$18 - $40/hr
Emphasizes readable, maintainable code and connects Python to machine learning, web scraping, scientific computing, and DevOps applications. * Curriculum Awareness & Adaptive Instruction: Familiar ...
Python Tutor
Laurel, MD · Remote
$18 - $40/hr
Emphasizes readable, maintainable code and connects Python to machine learning, web scraping, scientific computing, and DevOps applications. * Curriculum Awareness & Adaptive Instruction: Familiar ...
Python Tutor
Rockville, MD · Remote
$18 - $40/hr
Emphasizes readable, maintainable code and connects Python to machine learning, web scraping, scientific computing, and DevOps applications. * Curriculum Awareness & Adaptive Instruction: Familiar ...
Python Tutor
Rockville, MD · Remote
$18 - $40/hr
Emphasizes readable, maintainable code and connects Python to machine learning, web scraping, scientific computing, and DevOps applications. * Curriculum Awareness & Adaptive Instruction: Familiar ...
Experience with web scraping/crawling (e.g., Beautiful Soup, Selenium, Scrapy) Machine Learning Engineering: * * Computer vision skills (OCR, image classification, deep fake detection) * Familiarity ...
Experience with web scraping/crawling (e.g., Beautiful Soup, Selenium, Scrapy) Machine Learning Engineering: * * Computer vision skills (OCR, image classification, deep fake detection) * Familiarity ...
Python Tutor
Bowie, MD · Remote
$18 - $40/hr
Emphasizes readable, maintainable code and connects Python to machine learning, web scraping, scientific computing, and DevOps applications. * Curriculum Awareness & Adaptive Instruction: Familiar ...
Python Tutor
Bowie, MD · Remote
$18 - $40/hr
Emphasizes readable, maintainable code and connects Python to machine learning, web scraping, scientific computing, and DevOps applications. * Curriculum Awareness & Adaptive Instruction: Familiar ...
Black Canyon Consulting (BCC) is searching for a Web Service Protection Engineer to support the ... scraping, abuse, DDoS, and attack patterns across millions of daily requests. • Operate and tune ...
Black Canyon Consulting (BCC) is searching for a Web Service Protection Engineer to support the ... scraping, abuse, DDoS, and attack patterns across millions of daily requests. • Operate and tune ...
Python Tutor
Alexandria, VA · Remote
$18 - $40/hr
Emphasizes readable, maintainable code and connects Python to machine learning, web scraping, scientific computing, and DevOps applications. * Curriculum Awareness & Adaptive Instruction: Familiar ...
Python Tutor
Alexandria, VA · Remote
$18 - $40/hr
Emphasizes readable, maintainable code and connects Python to machine learning, web scraping, scientific computing, and DevOps applications. * Curriculum Awareness & Adaptive Instruction: Familiar ...
Data Engineer
Columbia, MD · On-site
$93K - $136K/yr
Our engineers, designers, and strategists cut through complexity to create intuitive products and ... web scraping, Direct Connect) and internal agency sources into the data lake landing zone and ...
Data Engineer
Columbia, MD · On-site
$93K - $136K/yr
Our engineers, designers, and strategists cut through complexity to create intuitive products and ... web scraping, Direct Connect) and internal agency sources into the data lake landing zone and ...
Data Engineer
Columbia, MD · On-site
$93K - $136K/yr
Our engineers, designers, and strategists cut through complexity to create intuitive products and ... web scraping, Direct Connect) and internal agency sources into the data lake landing zone and ...
Data Engineer
Columbia, MD · On-site
$93K - $136K/yr
Our engineers, designers, and strategists cut through complexity to create intuitive products and ... web scraping, Direct Connect) and internal agency sources into the data lake landing zone and ...
Python Tutor
Washington, DC · Remote
$18 - $40/hr
Emphasizes readable, maintainable code and connects Python to machine learning, web scraping, scientific computing, and DevOps applications. * Curriculum Awareness & Adaptive Instruction: Familiar ...
Python Tutor
Washington, DC · Remote
$18 - $40/hr
Emphasizes readable, maintainable code and connects Python to machine learning, web scraping, scientific computing, and DevOps applications. * Curriculum Awareness & Adaptive Instruction: Familiar ...
Python Tutor
Leesburg, VA · Remote
$18 - $40/hr
Emphasizes readable, maintainable code and connects Python to machine learning, web scraping, scientific computing, and DevOps applications. * Curriculum Awareness & Adaptive Instruction: Familiar ...
Python Tutor
Leesburg, VA · Remote
$18 - $40/hr
Emphasizes readable, maintainable code and connects Python to machine learning, web scraping, scientific computing, and DevOps applications. * Curriculum Awareness & Adaptive Instruction: Familiar ...
Python Tutor
Fairfax, VA · Remote
$18 - $40/hr
Emphasizes readable, maintainable code and connects Python to machine learning, web scraping, scientific computing, and DevOps applications. * Curriculum Awareness & Adaptive Instruction: Familiar ...
Python Tutor
Fairfax, VA · Remote
$18 - $40/hr
Emphasizes readable, maintainable code and connects Python to machine learning, web scraping, scientific computing, and DevOps applications. * Curriculum Awareness & Adaptive Instruction: Familiar ...
Data Engineer
Centreville, VA · On-site +1
$113K - $136K/yr
Develop and implement web scraping and data ingestion workflows to collect open-source data ... Support feature engineering, aggregations, and transformations at scale. * Assist in deploying ML ...
Data Engineer
Centreville, VA · On-site +1
$113K - $136K/yr
Develop and implement web scraping and data ingestion workflows to collect open-source data ... Support feature engineering, aggregations, and transformations at scale. * Assist in deploying ML ...
Web Scraping Engineer information
What are the key skills and qualifications needed to thrive as a web scraping engineer, and why are they important?
To thrive as a Web Scraping Engineer, you need strong programming skills (especially in Python), knowledge of data extraction techniques, and familiarity with web protocols and HTML structure. Expertise with tools like Scrapy, BeautifulSoup, Selenium, and experience with APIs or anti-bot evasion techniques is typically required. Attention to detail, problem-solving, and strong analytical thinking are essential soft skills in this field. These skills ensure efficient, ethical, and reliable extraction of data from diverse web sources while navigating technical and legal complexities.
What does a web scraping engineer do?
A Web Scraping Engineer designs and develops software tools or scripts to automatically extract data from websites. They work with programming languages like Python and use libraries such as BeautifulSoup or Scrapy to gather and process web data efficiently. In addition to building scrapers, they also handle challenges such as dealing with website restrictions, CAPTCHAs, and ensuring compliance with legal and ethical guidelines. Their work is often used for market research, data analysis, or feeding information into business applications.
What is the difference between Web Scraping Engineer vs Data Engineer?
| Aspect | Web Scraping Engineer | Data Engineer |
|---|---|---|
| Primary Focus | Developing and maintaining web scraping tools to extract data from websites | Building and managing data pipelines and infrastructure for data storage and processing |
| Skills & Certifications | Python, APIs, HTML, CSS, web scraping libraries (BeautifulSoup, Scrapy) | SQL, ETL tools, cloud platforms, programming (Python, Java) |
| Work Environment | Tech companies, data-driven startups, research projects | Large enterprises, data warehouses, cloud environments |
| Industry Usage | Data collection for analytics, research, competitive analysis | Data integration, analytics, machine learning pipelines |
While both roles involve working with data, a Web Scraping Engineer specializes in extracting data from websites using scraping tools, whereas a Data Engineer focuses on building data pipelines and infrastructure for storing and processing large datasets. The roles often overlap in skills like Python programming but serve different core functions within data ecosystems.
What are some common challenges faced by web scraping engineers when extracting data from dynamic websites?
Web Scraping Engineers often encounter challenges such as navigating websites that use JavaScript to load content dynamically, dealing with anti-bot measures like CAPTCHAs or IP blocking, and ensuring data accuracy as website structures frequently change. To address these issues, engineers typically use headless browsers, rotating proxies, and robust error handling strategies. Staying up-to-date with the latest web technologies and adapting scripts proactively are essential for long-term success in this role.
What are popular job titles related to Web Scraping Engineer jobs in Washington? For Web Scraping Engineer jobs in Washington, the most frequently searched job titles are:
What job categories do people searching Web Scraping Engineer jobs in Washington look for? The top searched job categories for Web Scraping Engineer jobs in Washington are:
What cities in Washington are hiring for Web Scraping Engineer jobs? Cities in Washington with the most Web Scraping Engineer job openings:
Full-time
Medical, Dental, Vision, Retirement, PTO
Re-posted 10 days ago
Job description
Overview
Black Canyon Consulting (BCC) is searching for a Web Service Protection Engineer to support the National Center for Biotechnology Information (NCBI). This opportunity is full time and onsite at the NCBI in Bethesda, MD.
NCBI is part of the National Library of Medicine (NLM) at the National Institutes of Health (NIH). NCBI is the world's premier biomedical center hosting over six million daily users that seek research, clinical, genetic, and other information that directly impacts biomedical research and public health - at NCBI you can literally help to accelerate cures for diseases! NCBI's wide range of applications, platforms (node, python, Django, C++, you name it) and environments (big data [petabytes], machine learning, multiple clouds) serve more users than almost any other US Government Agency according to https://analytics.usa.gov/.
We attract the best people in the business with our competitive benefits package that includes medical, dental and vision coverage, 401k plan with employer contribution, paid holidays, vacation, and tuition reimbursement. If you enjoy being a part of a high performing, professional service and technology focused organization, please apply today!
Duties & Responsibilities
We attract the best people in the business with our competitive benefits package that includes medical, dental and vision coverage, 401k plan with employer contribution, paid holidays, vacation, and tuition reimbursement.
We offer a competitive salary commensurate with experience and location. If you enjoy being a part of a high performing, professional service and technology focused organization, please apply today!
Black Canyon Consulting (BCC) is searching for a Web Service Protection Engineer to support the National Center for Biotechnology Information (NCBI). This opportunity is full time and onsite at the NCBI in Bethesda, MD.
NCBI is part of the National Library of Medicine (NLM) at the National Institutes of Health (NIH). NCBI is the world's premier biomedical center hosting over six million daily users that seek research, clinical, genetic, and other information that directly impacts biomedical research and public health - at NCBI you can literally help to accelerate cures for diseases! NCBI's wide range of applications, platforms (node, python, Django, C++, you name it) and environments (big data [petabytes], machine learning, multiple clouds) serve more users than almost any other US Government Agency according to https://analytics.usa.gov/.
We attract the best people in the business with our competitive benefits package that includes medical, dental and vision coverage, 401k plan with employer contribution, paid holidays, vacation, and tuition reimbursement. If you enjoy being a part of a high performing, professional service and technology focused organization, please apply today!
Duties & Responsibilities
- Develop and own protection metrics and alerting - build dashboards and alert pipelines that surface anomalies across a range of network and application-layer signals.
- Perform deep log analysis to identify overuse, scraping, abuse, DDoS, and attack patterns across millions of daily requests.
- Operate and tune cloud-based edge security controls - configure and update security policies, rate limiting, and adaptive protection rules in response to evolving threats.
- Enforce traffic controls - apply a range of mitigation strategies to abusive traffic while minimizing impact to legitimate users.
- Monitor continuously and respond quickly - this role requires a bias for action. You will triage incidents and either resolve them directly or escalate to development or sysadmin teams with clear, actionable information.
- Support protection across a mixed cloud and on-premises infrastructure - NCBI operates services in both environments, and protection coverage must extend consistently across both.
- Strong understanding of high-volume web service architecture - how traffic flows, where bottlenecks and abuse vectors appear, and how load balancers and edge infrastructure make routing decisions.
- Hands-on experience with a major cloud provider's security and networking stack, including:
- Global load balancing and traffic management
- Edge-layer security policy enforcement and WAF capabilities
- Large-scale log querying and metric-driven alerting
- Solid grasp of HTTP/HTTPS protocol internals - headers, TLS behavior, connection patterns, and how these relate to traffic analysis and fingerprinting techniques.
- Experience analyzing traffic using network and application-layer signals including address-based, organizational, and transport-layer fingerprinting methods.
- Familiarity with common web server platforms, their log formats, and configuration.
- Ability to read, write, and tune access control and rate-limiting rules under pressure.
- Comfort working across hybrid infrastructure environments.
- Experience with advanced abuse mitigation techniques, including traffic redirection and challenge-response mechanisms.
- Scripting or automation experience (Python, Bash, or similar) for log parsing and rule generation.
- Prior work protecting high-traffic government or research platforms.
- Knowledge of bot detection techniques beyond simple blocking - behavioral signals, fingerprinting, headless browser detection, and similar approaches.
- Familiarity with evolving attacker tradecraft and how modern scrapers and abusers adapt to countermeasures.
We attract the best people in the business with our competitive benefits package that includes medical, dental and vision coverage, 401k plan with employer contribution, paid holidays, vacation, and tuition reimbursement.
We offer a competitive salary commensurate with experience and location. If you enjoy being a part of a high performing, professional service and technology focused organization, please apply today!