Role Overview

Join Datadog's IDM team as a Senior AI Engineer - APM Integrations and revolutionize how engineers build and maintain cloud application monitoring integrations. This remote position in Portugal focuses on creating AI-powered tools that assist with code generation, bug fixes, testing, and validation for Datadog APM (Application Performance Monitoring). You'll develop trustworthy AI agents that handle complex integration workflows from planning to production deployment.

Datadog is the leading observability and security platform for cloud-scale applications, trusted by thousands of engineering teams worldwide. The IDM team owns the critical integrations connecting Datadog to customer tools and services. Your AI systems will synthesize context from codebases, documentation, telemetry data, and historical incidents to generate production-ready code that matches Datadog's quality standards and customer expectations.

This senior role offers high ownership from early prototypes through production support. You'll define success metrics, build evaluation frameworks to prevent AI quality drift, and partner with PMs, support engineers, and integration teams to embed these tools in daily workflows. Perfect for product-minded engineers passionate about shipping reliable AI to production.

Key Responsibilities at Datadog

• Build sophisticated agent workflows that transform integration requirements into complete implementations with human approval checkpoints
• Develop context-aware systems that analyze codebases, technical specifications, telemetry data, and past incidents for accurate code generation
• Generate integration code, comprehensive test suites, and end-to-end validation scenarios reflecting real customer workloads
• Design robust evaluation harnesses including golden datasets, semantic similarity checks, performance thresholds, and release gating mechanisms
• Create portfolio-wide automation handling upstream breaking changes, tracer feature rollouts, schema migrations, and targeted test coverage expansion
• Collaborate closely with product managers, support engineers, and integration-owning teams to ensure seamless tool adoption
• Establish quality benchmarks, implement continuous measurement, and drive iterative improvements based on real-world usage data
• Implement production monitoring, alerting, and on-call practices to maintain AI system reliability at scale
• Champion AI safety through comprehensive guardrails, regression testing, and quality assurance frameworks

Qualifications & Requirements

Product-minded AI Engineer: 6+ years building scalable backend systems in Go, Java, or .NET with emphasis on simplicity, correctness, and performance. Proven track record shipping LLM/agent features to production including advanced prompting techniques, evaluation frameworks, and safety mechanisms. Comfortable with ambiguity, rapid prototyping, and measuring concrete business impact.

ML Expertise: Deep understanding of complete ML lifecycle from task formulation through deployment and iteration. Expertise in offline/online evaluation methodologies including golden sets, automated regression detection, and statistical experiment design. Experience preventing silent quality degradation in production AI systems.

Systems & Observability: Strong background in distributed systems performance analysis including tracing, latency profiling, concurrency patterns, and resiliency strategies. Production operations experience with monitoring, alerting, and on-call rotations. Familiarity with OpenTelemetry, Datadog APM, profilers, and observability pipelines highly valued.

Bonus: Hands-on experience with agent frameworks, RAG systems, developer tools, semantic conventions, scenario-based testing, and large-scale code transformation pipelines.

Salary & Benefits

Datadog offers competitive compensation for Senior AI Engineers in Portugal ranging from $135,000 - $185,000 USD base salary (equivalent in EUR) plus equity. Total compensation includes:

• New hire RSU stock grants and ESPP participation
• Comprehensive global health coverage for families
• Free mental health benefits for employees and dependents 6+
• Professional development stipend and training programs
• Mentorship programs and career pathing
• Hybrid work model with office culture benefits
• Community Guilds and Inclusion Talks
• Competitive PTO and global benefits package

Why Join Datadog?

Datadog empowers modern cloud teams with end-to-end observability and security for their entire technology stack. As a Senior AI Engineer, you'll work on cutting-edge AI systems that directly impact thousands of engineering teams worldwide. Our hybrid culture balances collaboration with work-life harmony, while our growth programs accelerate your career trajectory.

Thrive in an inclusive environment with mentorship, Community Guilds, and continuous learning opportunities. Datadog values diverse backgrounds and encourages passionate technologists to apply even if you don't meet every qualification.

How to Apply

Ready to build the future of AI-assisted observability? Click 'Apply Now' to join Datadog's mission-critical IDM team. Submit your resume and a brief note about your most impactful AI production experience. We're hiring rapidly for this Portugal-remote role - don't miss your chance to shape AI engineering at scale!