Remote Forward Deployed Engineer Jobs in Hemet, CA

Locations: India (Offshore) and United States (Onsite)
Compensation:

• 
  

Role Overview
We are seeking a hands-on Senior Datadog Architect & Implementation Engineer to lead the Datadog implementation for a multi-account AWS observability rollout across Development, UAT, and Production environments.
The platform supports a portfolio of AI agent solutions built using Amazon Bedrock, Bedrock AgentCore, AWS Lambda, and Amazon ECS. This is an implementation-focused position—not an advisory or dashboard-only role.
The successful candidate will deliver a fully operational observability solution, including live dashboards, correctly configured alerts, centralized logs, metrics, distributed traces, and production-ready notification routing.
Key Responsibilities

• Design and deploy an organization-wide Datadog AWS integration across Dev, UAT, and Prod environments.
• Implement the integration using AWS CloudFormation StackSets, AWS Organizations, and secure cross-account IAM roles.
• Build and maintain the CloudWatch Logs-to-Datadog pipeline using the Datadog Forwarder Lambda and CloudWatch subscription filters.
• Configure Datadog AWS metrics polling across multiple AWS accounts.
• Enable CloudWatch Transaction Search and configure the Datadog AWS X-Ray integration.
• Ensure AgentCore trace and span data is visible and searchable within Datadog.
• Implement distributed tracing and telemetry using AWS X-Ray and OpenTelemetry.
• Create tag-driven dashboards organized by environment, AWS account, application, and AI solution.
• Build dashboards for AI token consumption, platform usage, performance, reliability, and cost monitoring.
• Define, configure, and tune Datadog monitors and alert thresholds.
• Establish notification routing, escalation paths, and production-support procedures.
• Troubleshoot issues involving missing logs, metrics, traces, permissions, tags, and cross-account integrations.
• Work with security, cloud infrastructure, platform, and application teams to complete IAM and compliance reviews.
• Provide production support and maintain the implemented Datadog platform.

Required Qualifications

• Strong hands-on experience implementing Datadog in multi-account AWS environments.
• Proven experience configuring Datadog AWS metrics polling.
• Hands-on experience deploying and supporting the Datadog Forwarder Lambda.
• Experience configuring CloudWatch Logs subscription filters and centralized log forwarding.
• Strong knowledge of AWS Organizations and organization-wide cloud deployments.
• Hands-on experience with CloudFormation and CloudFormation StackSets.
• Strong understanding of cross-account IAM roles, trust policies, permissions, and security controls.
• Experience with AWS X-Ray, OpenTelemetry, distributed tracing, metrics, logs, and spans.
• Hands-on experience supporting AWS Lambda, Amazon ECS/Fargate, CloudWatch Logs, and CloudWatch Metrics.
• Experience building production Datadog dashboards, monitors, alerts, and notification workflows.
• Ability to implement solutions against an existing enterprise architecture.
• Experience working through security, compliance, and IAM approval processes.
• Strong troubleshooting, cross-team delivery, and production-support skills.
• Excellent written and verbal communication skills.

Preferred Qualifications

• Experience monitoring AI or generative AI solutions built with Amazon Bedrock or Bedrock AgentCore.
• Experience creating token-usage and cost-monitoring dashboards for AI workloads.
• Knowledge of Datadog tagging strategies and AWS tag taxonomies.
• Datadog or AWS professional certifications.
• Experience delivering observability solutions in regulated enterprise environments.

Expected Deliverables
By the end of the engagement, the engineer will be expected to deliver:

• Working Datadog integrations across all required AWS accounts and environments.
• Operational CloudWatch log-forwarding and metrics-polling pipelines.
• Visible and searchable AgentCore traces and spans.
• Live, tag-driven operational and cost dashboards.
• Correctly firing and tuned monitors and alerts.
• Production-ready notification routing and support documentation.

Important Candidate Requirement
Priority will be given to candidates who have personally designed and implemented Datadog across multiple AWS accounts.
Candidates whose experience is limited to using existing Datadog dashboards, reviewing logs, or managing monitors without implementing the underlying AWS integration, Forwarder Lambda, StackSets, cross-account IAM roles, and tracing pipeline will not be considered a strong match.

This is a remote position.