Sr/Staff Software Engineer, Observability at Crusoe: Powering the AI Revolution Sustainably

Crusoe Energy Systems is on a mission to align the future of energy with the future of computing. We are building innovative solutions that reduce both the costs and the environmental impact of the world’s expanding digital infrastructure. As a Sr/Staff Software Engineer, Observability, you will play a crucial role in ensuring the reliability, performance, and security of our rapidly growing cloud infrastructure that supports High Performance Compute (HPC) clusters with GPUs. You will be at the forefront of building and operating observability platforms that are scalable, language-agnostic, and capable of handling the demands of AI/ML and HPC workloads.

Role Overview: Architecting the Future of Observability

As a Senior/Staff Software Engineer, Observability, you will be a key member of our team, working to build and maintain the observability platform that underpins our entire cloud infrastructure. This is not just about monitoring; it's about creating a comprehensive, proactive system that enables us to understand the internal state of our distributed systems. Your work will directly impact our ability to deliver reliable, high-performance computing solutions to our customers. You will be supporting the Network Engineering team, ensuring they have the tools and insights needed to maintain a robust and efficient network.

A Day in the Life: Shaping Crusoe's Observability Landscape

Your day-to-day activities will be varied and challenging. You might:

• Design and operate scalable observability systems (metrics, logging, tracing) across multi-datacenter Kubernetes environments.
• Architect end-to-end telemetry pipelines, including ingestion, storage, querying, and visualization.
• Extend monitoring and alerting with Prometheus, Alertmanager, Thanos/Cortex, Grafana, and OpenTelemetry.
• Build scalable log collection and processing pipelines with Fluent Bit, Vector, Loki, or ELK/Opensearch stacks.
• Implement distributed tracing platforms (Tempo, Jaeger, OpenTelemetry) and integrating with service meshes, load balancers, and APIs.
• Define and drive adoption of SLOs, SLIs, and error budgets across services and teams.
• Automate provisioning and scaling of observability infrastructure with Kubernetes, Terraform, and custom tooling (Go, Python).
• Ensure reliability and cost efficiency of telemetry pipelines while supporting high-volume workloads (AI/ML, HPC clusters, GPU infrastructure).
• Embed security best practices into observability platforms, including RBAC, TLS, secret management, and multi-tenant access controls.
• Mentor engineers and shape Crusoe’s observability strategy and technical roadmap.

You will be working with cutting-edge technologies and collaborating with a team of talented engineers to solve complex problems.

Why San Francisco? Innovate in the Heart of Tech

San Francisco is the epicenter of technological innovation, and Crusoe's location here offers unparalleled opportunities for growth and learning. You'll be surrounded by a vibrant community of engineers, researchers, and entrepreneurs. The Bay Area's proximity to leading universities and research institutions ensures a constant influx of new ideas and talent. San Francisco also offers a rich cultural scene, diverse culinary experiences, and stunning natural beauty, making it an ideal place to live and work.

Career Path: Growth and Advancement at Crusoe

At Crusoe, we are committed to the growth and development of our employees. As a Sr/Staff Software Engineer, Observability, you will have opportunities to advance your career in several directions. You could become a Principal Engineer, leading the technical direction of our observability platform. You could also move into a management role, leading a team of engineers and shaping the future of our technology. We provide ongoing training and development opportunities to help you achieve your career goals.

Salary & Benefits: Competitive Compensation and Comprehensive Benefits

Crusoe offers a competitive salary and a comprehensive benefits package that includes:

• Comprehensive health insurance (medical, dental, vision)
• Generous paid time off (PTO) and holidays
• 401(k) retirement plan with company match
• Employee stock options
• Professional development opportunities
• Flexible work arrangements
• Company-sponsored events and activities
• Wellness programs and resources
• Commuter benefits
• Parental leave
• Life insurance
• Disability insurance

We believe in rewarding our employees for their hard work and dedication.

Crusoe Culture: Innovation, Collaboration, and Sustainability

Crusoe's culture is built on a foundation of innovation, collaboration, and sustainability. We are a team of passionate individuals who are committed to making a positive impact on the world. We encourage creativity and collaboration, and we provide a supportive environment where everyone can thrive. We are also committed to sustainability and are working to reduce the environmental impact of our operations.

How to Apply: Join the Crusoe Team

If you are a talented and motivated engineer who is passionate about observability and sustainability, we encourage you to apply. Please submit your resume and cover letter through our online application portal. We are looking for individuals who are eager to learn, grow, and make a difference.

Frequently Asked Questions (FAQ)

1. What is Crusoe's mission?

   Crusoe's mission is to align the future of energy with the future of computing. We build innovative solutions that reduce both the costs and the environmental impact of the world’s expanding digital infrastructure.

2. What technologies will I be working with?

   You will be working with a wide range of technologies, including Kubernetes, Prometheus, Grafana, OpenTelemetry, Fluent Bit, Vector, Loki, ELK/Opensearch stacks, Jaeger, Tempo, Go, Python, and Terraform.

3. What is the work environment like at Crusoe?

   Crusoe offers a collaborative, innovative, and supportive work environment where employees are encouraged to learn, grow, and make a difference.

4. What are the career growth opportunities at Crusoe?

   Crusoe is committed to the growth and development of its employees, with opportunities for advancement into roles such as Principal Engineer or Management positions. We provide ongoing training and development to help you achieve your career goals.

5. What kind of benefits does Crusoe offer?

   Crusoe offers a competitive salary and a comprehensive benefits package, including health insurance, paid time off, a 401(k) plan, employee stock options, and more.

6. Is prior network engineering experience required?

   No prior network engineering experience is required for this observability role. Support to the Network Engineering Team is a key aspect of the position.

7. How does Crusoe contribute to sustainability?

   Crusoe is committed to sustainability by reducing the environmental impact of digital infrastructure through innovative solutions that align the future of energy with the future of computing.

8. What skills are most important for this role?

   Deep expertise in metrics systems (Prometheus, Thanos, Mimir, Cortex), logging pipelines (Fluent Bit, Vector, Loki, ELK/Opensearch), and tracing platforms (Jaeger, Tempo, OpenTelemetry) are most important.

9. What is the typical team size for Observability at Crusoe?

   While team sizes vary, expect to be part of a focused and agile team, typically ranging from 3-7 engineers.

10. Does Crusoe offer relocation assistance?

    Relocation assistance eligibility will vary. Please inquire with the recruiting team.