Role Overview

Datadog's ML Observability team is at the forefront of building cutting-edge tools to monitor, explain, and improve AI systems in production. Specializing in Large Language Models (LLMs) and generative AI, we deliver robust, scalable observability solutions including drift detection, model evaluation, and behavior tracing. As a Staff Software Engineer - ML Observability in Boston, Massachusetts, you'll lead development of new features and foundational capabilities within Datadog's LLM Observability product.

Your deep expertise in AI systems and software engineering will shape product direction, drive experimentation, and solve complex problems in the fast-evolving AI landscape. This role directly impacts how customers monitor, troubleshoot, and optimize LLM-based applications at scale. Join us to build the foundational tools making AI systems observable, understandable, and reliable in real-world production environments.

Key Responsibilities at Datadog

• Drive design and implementation of LLM observability features for production AI workloads
• Ideate, prototype, and scale innovative product features providing actionable insights for generative AI systems
• Collaborate cross-functionally with engineering, product, UX, and applied science teams to achieve product-market fit
• Develop advanced tools for tracing, evaluating, and debugging Large Language Models in production
• Influence critical architecture decisions and mentor engineers to build resilient, high-performance systems
• Stay deeply connected to customer pain points to prioritize engineering roadmap
• Monitor industry trends in machine learning and observability to drive team innovation
• Lead experimentation initiatives including A/B testing for ML observability capabilities
• Build scalable solutions for cloud-native AI applications using modern DevOps practices

Qualifications & Requirements

• Advanced Degree: BS/MS/PhD in Computer Science, Engineering, or related scientific field (or equivalent practical experience)
• Systems Expertise: Deep understanding of distributed systems and scalable backend architectures
• AI Experience: Hands-on experience building and shipping LLM-powered or GenAI applications
• Senior Leadership: 5+ years software engineering with proven track record leading complex, high-impact projects
• Observability Knowledge: Strong experience with monitoring, tracing, and observability systems
• Technical Proficiency: Expertise in Python, Go, or similar backend languages; familiarity with Kubernetes, Docker, AWS

Salary & Benefits

Datadog offers competitive compensation for Staff Software Engineers in ML Observability, typically ranging from $220,000 - $320,000 base salary plus equity, bonuses, and comprehensive benefits. Our total compensation packages are designed to attract top talent in Boston's competitive tech market.

• Competitive Equity: Meaningful stock options package
• Health Insurance: Comprehensive coverage for family members
• Commuter Benefits: Full transportation reimbursements
• Fitness Reimbursements: Gym memberships and wellness programs
• Professional Development: Learning stipends, conference attendance, certifications
• Unlimited PTO: Flexible time off policy
• Hybrid Workplace: Balance office collaboration with remote flexibility
• 401(k) Matching: Generous retirement savings contributions

Why Join Datadog?

Datadog is the leading cloud observability platform trusted by thousands of innovative companies worldwide. Our ML Observability team is pioneering the future of AI monitoring, giving you the opportunity to work on production-scale LLM systems that power enterprise generative AI applications. Boston's vibrant tech ecosystem combined with Datadog's hybrid culture creates the perfect environment for technical excellence and career growth.

Work with cutting-edge technologies, collaborate with world-class engineers, and directly impact customer success. At Datadog, you'll gain exposure to the entire product lifecycle from ideation to production deployment, while staying ahead of AI industry trends.

How to Apply

Ready to shape the future of ML observability at Datadog? Apply now for the Staff Software Engineer position in Boston, Massachusetts. Submit your resume and tell us why you're passionate about building observability for the AI era. Our team reviews applications continuously - don't miss your chance to join a market leader in cloud monitoring and security.

Datadog is an Equal Opportunity Employer committed to diversity and inclusion.