Role Overview

Join Datadog's IDM team as a Senior AI Engineer - APM Integrations and revolutionize how engineers build and maintain cloud application monitoring integrations. This high-impact role focuses on creating AI-assisted tools that streamline day-to-day engineering workflows, from drafting code changes to automated testing and validation. Working remotely from Germany or Italy, you'll develop trustworthy AI agents that handle complex APM (Application Performance Monitoring) integrations, ensuring Datadog seamlessly connects to the tools and services customers rely on.

Datadog is the leading monitoring and security platform for cloud-scale applications, helping DevOps teams, developers, and engineering leaders optimize performance and troubleshoot issues in real-time. As a Senior AI Engineer, you'll take full ownership from early prototypes to production deployment and ongoing support. This role demands strong product thinking, ML fundamentals, and distributed systems expertise to build tools engineers can trust in production environments.

The IDM team specializes in Datadog APM integrations, maintaining high quality and reliability across the integration catalog. Your AI solutions will synthesize context from codebases, documentation, telemetry data, and historical incidents to generate code that matches Datadog conventions and customer expectations. This is a senior position with significant autonomy and cross-team collaboration.

Key Responsibilities at Datadog

• Build sophisticated agent workflows that transform integration requirements from planning through implementation, with human approval checkpoints ensuring quality control.
• Develop systems that intelligently synthesize context from diverse sources including codebases, technical documentation, specifications, telemetry data, and past incidents.
• Generate production-ready integration code and comprehensive test suites, including realistic end-to-end scenarios that mirror customer workloads and product capabilities.
• Design robust evaluation harnesses featuring golden datasets, scenario baselines, semantic validation, performance thresholds, and automated release gating to prevent regressions.
• Create portfolio-wide automation handling proactive updates for upstream breaking changes, tracer feature rollouts, schema migrations, and targeted test coverage expansion.
• Collaborate closely with Product Managers, support engineers, and integration-owning teams to ensure AI tools become trusted, adoptable parts of daily engineering workflows.
• Define success metrics for AI tools, implement measurement systems, and drive continuous improvement based on real-world usage data and feedback.
• Maintain evaluation infrastructure that catches silent quality degradation, ensuring AI outputs remain reliable as models and requirements evolve.
• Champion observability best practices within AI systems, applying Datadog APM expertise to monitor tool performance and integration health.

Qualifications & Requirements

Datadog seeks product-minded engineers who ship AI to production with proven track records in backend systems development:

• 6+ years building scalable backend systems using Go, Java, or .NET, emphasizing simplicity, correctness, and performance optimization.
• Demonstrated success delivering LLM/agent features to production environments, including advanced prompting techniques, custom tooling, comprehensive evaluations, and safety guardrails.
• Strong comfort navigating technical ambiguity, rapidly iterating from prototypes to production systems, and measuring impact through clear, actionable metrics.

Candidates must possess strong ML fundamentals:

• Deep understanding of the complete ML lifecycle: task definition, dataset curation, model development, evaluation frameworks, deployment strategies, and iterative improvement cycles.
• Expertise in both offline and online evaluation methodologies including golden set validation, automated regression testing, and evaluation harnesses preventing quality drift.

Distributed systems and observability expertise is essential:

• Hands-on experience optimizing microservices performance through distributed tracing, latency analysis, concurrency patterns, and resiliency strategies.
• Production operations mindset with practical experience in monitoring, alerting systems, and participating in on-call rotations.

Bonus qualifications include experience with distributed tracing stacks (OpenTelemetry/Datadog APM), planning/agent frameworks, RAG systems, developer tools, semantic conventions, and scenario-based testing at scale.

Salary & Benefits

Datadog offers competitive compensation for Senior AI Engineers ranging from $140,000 - $220,000 USD base salary (equivalent in local currency), plus comprehensive benefits:

• New hire stock equity (RSUs) and Employee Stock Purchase Plan (ESPP)
• Global health insurance covering employees and dependents
• Free mental health benefits for employees and dependents age 6+
• Continuous professional development and specialized product training
• Career pathing with dedicated mentor and buddy programs
• Inclusive culture through Community Guilds and resource groups
• Hybrid workplace flexibility optimizing work-life harmony
• Competitive global benefits package customized by location

Why Join Datadog?

Datadog empowers modern cloud teams with end-to-end observability and security for their entire technology stack. As a Senior AI Engineer on the APM Integrations team, you'll work on cutting-edge AI systems that directly impact thousands of engineering teams worldwide. Our hybrid culture fosters deep collaboration while providing the flexibility to work remotely from Germany or Italy.

Datadog values diverse perspectives and encourages candidates who may not meet every qualification to apply. We prioritize passion for technology, rapid learning, and the drive to build reliable AI systems at cloud scale. Join a team that's defining the future of observability through AI-powered developer tools.

How to Apply

Ready to build production-grade AI for Datadog APM integrations? Apply now for the Senior AI Engineer position. Submit your resume highlighting backend systems experience, LLM production deployments, and observability expertise. Datadog's recruitment team reviews applications on a rolling basis - don't miss this opportunity to shape AI-assisted engineering at a leading cloud monitoring platform.

Datadog is committed to creating an inclusive environment where all employees can thrive. We celebrate diverse backgrounds and perspectives that drive innovation in observability and cloud security.