Sr/Staff Software Engineer, Observability at Crusoe

Crusoe's mission is to accelerate the abundance of energy and intelligence. We’re crafting the engine that powers a world where people can create ambitiously with AI — without sacrificing scale, speed, or sustainability.

Be a part of the AI revolution with sustainable technology at Crusoe. Here, you'll drive meaningful innovation, make a tangible impact, and join a team that’s setting the pace for responsible, transformative cloud infrastructure.

About This Role

We are looking for a highly skilled engineer with deep expertise in building and operating observability platforms at scale.

As a Senior/Staff Software Engineer, Observability, you'll be supporting our team of Network Engineers (no network engineering experience required). The Crusoe Cloud Network Engineering Team is responsible for designing, building, and operating the global edge, backbone, and data center network for High Performance Compute (HPC) Clusters with GPUs.

This is a unique opportunity wherein you'll be building an observability platform specific to devices; one that is language agnostic and scalable. You will design, develop, and run Crusoe’s next-generation observability stack, enabling engineers to understand the internal state of distributed systems through metrics, logs, and traces.

A Day in the Life

• Designing and operating scalable observability systems (metrics, logging, tracing) across multi-datacenter Kubernetes environments
• Architecting end-to-end telemetry pipelines, including ingestion, storage, querying, and visualization
• Extending monitoring and alerting with Prometheus, Alertmanager, Thanos/Cortex, Grafana, and OpenTelemetry
• Building scalable log collection and processing pipelines with Fluent Bit, Vector, Loki, or ELK/Opensearch stacks
• Implementing distributed tracing platforms (Tempo, Jaeger, OpenTelemetry) and integrating with service meshes, load balancers, and APIs
• Defining and driving adoption of SLOs, SLIs, and error budgets across services and teams
• Automating provisioning and scaling of observability infrastructure with Kubernetes, Terraform, and custom tooling (Go, Python)
• Ensuring reliability and cost efficiency of telemetry pipelines while supporting high-volume workloads (AI/ML, HPC clusters, GPU infrastructure)
• Embedding security best practices into observability platforms, including RBAC, TLS, secret management, and multi-tenant access controls
• Mentoring engineers and shaping Crusoe’s observability strategy and technical roadmap

You Will Thrive In This Role If

• 6+ years of experience with distributed systems, with a focus on observability and monitoring systems
• Deep expertise with metrics systems (Prometheus, Thanos, Mimir, Cortex), logging pipelines (Fluent Bit, Vector, Loki, ELK/Opensearch), and tracing platforms (Jaeger, Tempo, OpenTelemetry)
• Strong programming skills in Go or Python for automation, operators, and custom integrations
• Experience running observability platforms on Kubernetes and operating them at scale across multi-datacenter environments
• Proven ability to design, optimize, and scale telemetry pipelines handling high cardinality and high throughput data
• Solid understanding of distributed systems, performance engineering, and debugging complex workloads
• Familiarity with service meshes, networking, and workload instrumentation (Envoy, Istio, OpenTelemetry SDKs)
• Strong collaboration skills and the ability to influence engineering teams to adopt observability best practices

Why San Francisco?

San Francisco, California, offers a unique blend of technological innovation and cultural vibrancy, making it an ideal location for a Senior/Staff Software Engineer specializing in observability. As a global hub for tech companies, San Francisco provides unparalleled opportunities for professional growth and networking. The city's proximity to Silicon Valley fosters a collaborative environment where cutting-edge ideas and technologies thrive.

Beyond the professional advantages, San Francisco boasts a high quality of life with its diverse neighborhoods, world-class dining, and stunning natural landscapes. From the iconic Golden Gate Bridge to the lush parks and vibrant arts scene, San Francisco offers a rich tapestry of experiences that cater to a wide range of interests. The city's commitment to sustainability and innovation aligns perfectly with Crusoe's mission, making it an inspiring place to live and work.

Furthermore, San Francisco's robust infrastructure and access to top-tier talent make it an advantageous location for Crusoe to build and expand its engineering team. The city's focus on technology and sustainability creates a synergistic environment that supports Crusoe's goals of accelerating the abundance of energy and intelligence.

Career Path

The Senior/Staff Software Engineer, Observability role at Crusoe offers a clear and promising career path. Starting as a Senior Engineer, you will have the opportunity to advance to a Staff Engineer position, taking on more responsibilities and leadership roles. From there, you can progress to Principal Engineer, where you will be a key technical leader, guiding the company's observability strategy and mentoring other engineers.

Beyond the technical track, there are also opportunities to move into engineering management roles, such as Engineering Manager or Director of Engineering, where you will lead and manage teams of engineers, driving the execution of critical projects and initiatives. Crusoe is committed to providing its employees with the resources and support they need to grow and advance in their careers.

Salary & Benefits

Crusoe offers a competitive salary and benefits package that is designed to attract and retain top talent. The estimated salary range for this role in San Francisco is $170,000 to $260,000 per year, depending on experience and qualifications. In addition to a competitive salary, Crusoe offers a comprehensive benefits package that includes:

• Comprehensive health insurance plans
• Generous paid time off
• Professional development opportunities
• Employee stock options
• Wellness programs
• Parental leave
• 401(k) with company match
• Company-sponsored events
• Commuter benefits
• Relocation assistance (if applicable)

Crusoe is committed to providing its employees with a supportive and rewarding work environment where they can thrive and grow in their careers.

Crusoe Culture

At Crusoe, we foster a culture of innovation, collaboration, and sustainability. We are a team of passionate individuals who are committed to making a positive impact on the world. We value diversity and inclusion, and we believe that everyone has something to contribute. We encourage our employees to be creative, take risks, and challenge the status quo.

We believe in working hard and playing hard. We offer a variety of company-sponsored events and activities, such as team outings, happy hours, and volunteer opportunities. We also provide our employees with the resources they need to stay healthy and productive, such as wellness programs and flexible work arrangements.

How to Apply

If you are a highly skilled and motivated engineer with a passion for observability and sustainability, we encourage you to apply for the Senior/Staff Software Engineer, Observability role at Crusoe. To apply, please submit your resume and cover letter through our online application portal. We look forward to hearing from you!

FAQ

1. What is Crusoe's mission?

   Crusoe's mission is to accelerate the abundance of energy and intelligence. We’re crafting the engine that powers a world where people can create ambitiously with AI — without sacrificing scale, speed, or sustainability.

2. What does the Crusoe Cloud Network Engineering Team do?

   The Crusoe Cloud Network Engineering Team is responsible for designing, building, and operating the global edge, backbone, and data center network for High Performance Compute (HPC) Clusters with GPUs.

3. What is unique about this observability role at Crusoe?

   This is a unique opportunity wherein you'll be building an observability platform specific to devices; one that is language agnostic and scalable.

4. What are the key responsibilities of this role?

   Key responsibilities include designing and operating scalable observability systems, architecting end-to-end telemetry pipelines, and implementing distributed tracing platforms.

5. What skills and experience are required for this role?

   Required skills and experience include 6+ years of experience with distributed systems, deep expertise with metrics, logging, and tracing platforms, and strong programming skills in Go or Python.

6. What are the benefits of working at Crusoe?

   Crusoe offers a competitive salary, comprehensive health insurance plans, generous paid time off, professional development opportunities, employee stock options, and more.

7. What is the career path for this role?

   The career path includes opportunities to advance to Staff Engineer, Principal Engineer, Engineering Manager, or Director of Engineering roles.

8. What is the company culture like at Crusoe?

   Crusoe fosters a culture of innovation, collaboration, and sustainability. We value diversity and inclusion, and we encourage our employees to be creative and take risks.

9. How can I apply for this role?

   To apply, please submit your resume and cover letter through our online application portal.

10. What is the estimated salary range for this role?

    The estimated salary range for this role in San Francisco is $170,000 to $260,000 per year, depending on experience and qualifications.