Observability Engineer Jobs in Washington (NOW HIRING)

Job Description Everforth ECS is seeking a Senior ML Observability Engineer to work in the National Capital Region covering the Pentagon, Falls Church, and Fairfax . Please Note: This position is contingent upon contract award. The War Data Platform (WDP) is a key initiative within the U.S. Department of War's (DoW) AI-First strategy introduced in early 2026. The WDP focuses on operational warfighting data and aims to accelerate the deployment of artificial intelligence (AI) on the battlefield. The WDP extends to Unclassified, Secret, and Top Secret environments, and supports collaboration between Combatant Commands, Joint Staff directorates, Senior Executive Service leaders, and operational analysts. The Senior ML Observability Engineer architects and governs the instrumentation and telemetry infrastructure needed to ensure production AI and machine learning models deployed across WDP's multi-enclave environment perform reliably and securely at mission scale. This role is essential to maintaining real-time visibility into model behavior, pipeline execution, and cross-domain access interactions in direct support of Combatant Command and Joint Staff decision-making needs. • Designs, implements, and governs observability and instrumentation architectures supporting AI and machine learning model-serving operations across Unclassified, Secret, and Top Secret enclaves within the War Data Platform (WDP) Core Integration enterprise.
• Develops semantic conventions, runtime instrumentation patterns, and telemetry pipelines that generate latency metrics, error signatures, throughput indicators, model-specific performance signals, and operational readiness measurements for deployed models and serving surfaces.
• Integrates observability capabilities into existing data pipelines, model-deployment workflows, API access patterns, and serving runtime frameworks to provide mission-relevant monitoring aligned with Combatant Command and Joint Staff decision-support needs.
• Configures and validates instrumentation using platforms such as OpenTelemetry, Prometheus, Grafana, Elastic, Splunk, Amazon CloudWatch, and service mesh telemetry components to deliver real-time visibility into model behavior, cross-domain access interactions, and pipeline execution characteristics.
• Conducts observability readiness reviews, supports test and evaluation gates, and collaborates with cybersecurity personnel to embed anomaly-detection signals aligned with Zero Trust and DoW cyber standards.
• Works with serving engineers, pipeline engineers, platform teams, and external provider integration engineers to maintain observability consistency across enclaves and resolve domain-specific telemetry constraints.
• Produces observability standards, instrumentation specifications, dashboards, alerting configurations, and performance analysis reports that strengthen reliability, accelerate incident response, and reinforce mission assurance for production model access across all security networks.
• Performs other duties as assigned. Required Skills • Current Secret security clearance with the ability to obtain and maintain a Top Secret (TS) security clearance with Sensitive Compartmented Information (SCI).
• 10 or more years of progressive experience in systems engineering, platform operations, or ML/AI infrastructure roles, with a demonstrated focus on observability, telemetry, and monitoring in classified or federal government cloud environments.
• Hands-on experience designing and implementing observability pipelines using industry-standard tooling such as OpenTelemetry, Prometheus, Grafana, Elastic, Splunk, or Amazon CloudWatch, including instrumentation of AI/ML model-serving runtimes and data pipelines.
• Experience operating across multi-enclave environments, including NIPRNet, SIPRNet, and JWICS, with demonstrated ability to adapt telemetry and observability architectures to cross-domain constraints and multi-level security requirements.
• CompTIA Cloud+ certification or equivalent, demonstrating foundational knowledge of cloud infrastructure, security, and operational monitoring standards.
• Strong problem-solving and decision-making capabilities, with a proven ability to weigh the relative costs and benefits of potential actions and identify the most appropriate solution.
• Highly developed interpersonal and oral/written communication skills, with the ability to effectively and professionally interact with a diverse set of stakeholders (from peers to end-users to executive management). Desired Skills • Active Top Secret (TS) security clearance with Sensitive Compartmented Information (SCI) eligibility.
• Advanced cloud certification such as AWS Solutions Architect (Professional), AWS DevOps Engineer (Professional), or an equivalent credential demonstrating deep expertise in cloud-native observability and infrastructure-as-code practices in GovCloud or classified cloud environments.
• Practical experience with AI/ML model monitoring concepts including model drift detection, performance degradation alerting, and model validation pipelines using frameworks such as TensorFlow, PyTorch, or MLflow.
• Familiarity with Zero Trust Architecture principles and Risk Management Framework (RMF) requirements as they apply to telemetry data handling, anomaly detection, and continuous monitoring in DoW-compliant environments.
• Experience contributing to DevSecOps pipelines and CI/CD workflows in support of production AI/ML model deployment, including integration of observability gates at promotion checkpoints across development, test, and production environments. ECS Federal LLC is an equal opportunity employer and does not discriminate or allow discrimination on the basis any characteristic protected by law. All qualified applicants will receive consideration for employment without regard to disability, status as a protected veteran or any other status protected by applicable federal, state, or local jurisdiction law. is the federal segment of , a $4B global organization with over 10,000 employees. Our nearly 3,500 professionals deliver advanced technology solutions in data and AI, cybersecurity, and enterprise transformation, serving defense, intelligence, and federal civilian agencies. Our work powers mission-critical outcomes, strengthens technology partnerships, and creates meaningful opportunities for our people. We are defined by a commitment to excellence in delivery, a culture of innovation, and an environment where talent can thrive and grow. We value: * Attracting and developing top talent and high-performing teams * Fostering a culture that is engaging, accountable, and mission-driven