About the Role

As a Site Reliability Engineer in Monitoring, you will focus on developing and managing monitoring solutions, with heavy emphasis on Grafana for creating dashboards that provide visibility into datacenter health. You will leverage programming skills to automate monitoring, analyze data, and scale business operations through insightful visualizations. This role requires collaboration with datacenter teams to deliver actionable insights and minimize downtime in xAI's infrastructure.

Responsibilities

• Design, build, and maintain Grafana dashboards tailored for datacenter technician organizations, providing real-time views into system health, performance metrics, and monitoring alerts.
• Develop automation scripts and tools using languages such as Java, Golang, Python, C/C++/C#, Bash, or Linux shell scripting to integrate monitoring systems and process data in JSON formats.
• Collaborate with Datacenter Operations Technicians to identify monitoring needs, troubleshoot issues, and ensure dashboards support efficient incident response and preventive maintenance.
• Evaluate and optimize existing dashboards for scalability, drawing from past experiences in creating monitoring solutions that have driven business growth.
• Manage dashboard lifecycle, including version control, updates, and performance tuning to handle large-scale datacenter environments.
• Participate in on-call rotations, incident analysis, and root cause investigations using monitoring data to improve system reliability.
• Document monitoring strategies, dashboard designs, and best practices to foster knowledge sharing within the team.

Required Qualifications

• Bachelor's degree in Computer Science, Software Engineering, or a related field (or equivalent experience).
• 5+ years of experience in site reliability engineering or monitoring roles, preferably in datacenter or cloud environments.
• Proficiency in at least two of the following programming languages: Java, Golang, Python, C/C++/C#, with strong skills in Linux and Bash scripting.
• Hands-on experience working with JSON for data parsing, integration, and API interactions.
• Expert-level knowledge of Grafana, including creating complex dashboards, queries, and integrations with data sources like Prometheus or InfluxDB.
• Proven track record of developing dashboards that provide health and monitoring views for operational teams, with examples of how they scaled business operations.
• Experience managing monitoring tools and dashboards, including optimization, alerting, and integration into CI/CD pipelines.
• Strong problem-solving skills with a focus on data-driven decision-making and collaboration in fast-paced environments.

Preferred Qualifications

• Experience in AI/ML infrastructure or high-performance computing monitoring.
• Familiarity with other monitoring tools (e.g., Grafana) and observability practices.
• Prior work in a startup or tech company like xAI, with contributions to scalable monitoring systems.