Consulting/Principal Site Reliability Engineer

About the Role
We're looking for a Consulting/Principal Site Reliability Engineer to guide our observability and reliability initiative for Government Systems, running on Azure Government. This is an advanced individual-contributor role for someone with a proven track record leading an organization through a transition to SLO-based reliability engineering — you've done this before, and you know what it takes to get development teams bought in. This is a hands-on SRE role, not a software development role: you'll work alongside other SREs on the initiative, guiding development teams on telemetry and instrumentation strategy rather than writing that code yourself. You'll operate as the go-to expert on Datadog and observability practice, helping define meaningful SLOs and error budgets and making data-driven reliability decisions. You'll also help train and mentor junior SREs, working within priorities set by the SRE Manager.
Eligibility: This role requires U.S. citizenship and the ability to obtain and maintain a Public Trust clearance.
What You'll Do
Guide the observability and reliability initiative, working alongside other SREs to set the technical direction for how the organization measures and improves reliability
Lead the transition to SLO-based reliability engineering, building on prior experience doing this elsewhere
Guide development teams on telemetry and instrumentation strategy for their applications, without doing the application development work yourself
Serve as the primary technical authority on Datadog, from dashboarding to alerting to SLO tracking
Recommend service level objectives in partnership with Product and Development teams
Act as a senior escalation point during incidents
Help improve and evolve the broader SRE framework
Build disaster recovery plans, including advanced fault injection scenarios
Advise on SRE training curriculum and content, and help set and follow priorities established by the SRE Manager
Tech Stack You'll Work With
Cloud: Microsoft Azure Government
Server Environment: Linux and Windows
CI/CD: Azure DevOps Pipelines, GitHub Actions
Infrastructure as Code: Terraform
Observability: Datadog (primary)
AI-assisted development: GitHub Copilot
What We're Looking For
Observability (core focus)
Significant, hands-on experience with Datadog — dashboards, alerting, SLO/error-budget tracking, and telemetry pipelines
Proven experience leading an organization's transition to SLO-based reliability engineering, not just supporting one
Track record guiding development teams on instrumentation and telemetry strategy (advisory, not hands-on application coding)
Deep technical understanding of observability techniques across the full stack; able to bring clarity to complex incidents or performance issues
Able to build templated observability dashboards and configuration as code so others can implement quickly
Able to set and influence appropriate SLOs and Error Budgets org-wide
Comfortable operating across both Linux and Windows server environments
Incident Management
Experience preparing and assessing SREs for on-call readiness, including mentoring and identifying training needs
Skilled at running post-mortems where participants feel safe to contribute; fosters a culture of learning from failure
Comfortable acting as a senior on-call escalation point and guiding restoration efforts
Experience partnering with SRE and Development leads to identify and fix root causes of excessive alerting and pager load
Design for Reliability
Advanced understanding of systems design, including high availability, deployment, and recovery techniques, plus failure modes and system behaviors
Specialized depth in one or more areas such as denial-of-service protection or containerization, with the ability to consult others on it
Able to make resilience and recoverability decisions backed by load testing data and fault injection results
Experience documenting SRE design practices and contributing to a broader SRE framework
Disaster Recovery
Experience contributing to DR plans, including recovery priorities, procedures, and guiding others through runbooks
Able to design and lead DR practice scenarios and incident response exercises, identifying knowledge gaps and improvements
Experience planning chaos engineering scenarios to simulate faults across memory, network, I/O, or distributed systems
Platforms and Automation
Able to build engineering and architecture consensus for new standard components and services on the Paved Road, including platforms and CI/CD
Experience promoting adoption and standardization, with contributions back via inner sourcing
A champion for Paved Road adoption, driving efficiency by removing rework and silos
Reliability Culture
Creates and shares SRE best practices and training material
Comfortable coaching other SREs and identifying development gaps
Able to spot and eliminate large-scale toil
AI-Driven Development (desired)
Experience using GitHub Copilot to accelerate troubleshooting, automation, or reliability tooling
Interest in helping define best practices for AI-assisted development within an SRE context
Behavioral Competencies
Collaboration and Teamwork
Customer & External Focus
Solves Problems and Analyzes Issues
Learning Agility
Builds Relationships
Develops Others
U.S. National Base Pay Range: $118,300 - $219,800. Geographic differentials may apply in some locations to better reflect local market rates.This job is eligible for an annual incentive bonus. We know your well-being and happiness are key to a long and successful career. We are delighted to offer country specific benefits. Click here to access benefits specific to your location.
We are committed to providing a fair and accessible hiring process. If you have a disability or other need that requires accommodation or adjustment, please let us know by completing our Applicant Request Support Form or please contact 1-855-833-5120.
Criminals may pose as recruiters asking for money or personal information. We never request money or banking details from job applicants. Learn more about spotting and avoiding scams here .
Please read our Candidate Privacy Policy.
We are an equal opportunity employer: qualified applicants are considered for and treated during employment without regard to race, color, creed, religion, sex, national origin, citizenship status, disability status, protected veteran status, age, marital status, sexual orientation, gender identity, genetic information, or any other characteristic protected by law.
USA Job Seekers:
EEO Know Your Rights.
Originally posted on Himalayas