DevEx @ GitHub

What is developer experience?

GitHub Blog -> Developer experience: What is it and why should you care?

DevEx refers to the systems, technology, process, and culture that influence the effectiveness of software development. It looks at all components of a developer’s ecosystem—from environment to workflows to tools—and asks how they are contributing to developer productivity, satisfaction, and operational impact.

GitHub DevEx formula

• Productivity: how quickly or simply a change can be made to a codebase
• Impact: how frictionless it is to move from idea to production
• Satisfaction: how the environment, workflows, and tools affect developer happiness
• Collaboration: the multiplier across the entire DevEx

Why is developer experience important?

A positive DevEx through efficient processes and tools drives business impact with superior product quality and employee retention.

Research

• Forrester Report Snapshot: Developer Experience (DevEx) impacts overall business performance
• McKinsey: Why your IT organization should prioritize developer experience

What makes a good developer experience?

GitHub Blog -> Developer experience: What is it and why should you care?

• Collaboration
• Speed
• Short feedback loops
• High degrees of automation and integration
• Low levels of friction or toil
• Transparent, well-documented processes

For more insights, check out Tim Cochran‘s essay, Maximizing developer effectiveness, ACM Queue’s piece, DevEx: What actually drives productivity and Measuring enterprise developer productivity from the GitHub Blog.

What are some key developer experience metrics?

Unfortunately, there are currently no standardized industry metrics to measure DevEx. Each organization has unique goals across developer productivity and happiness that will influence the signals and metrics used to measure DevEx.

DevOps Research and Assessment (DORA) Metrics are popular DevEx indicators

• Deployment frequency (DF) - how frequently an organization releases new software
• Lead time for changes (LT) - the time taken from when a change is requested or initiated to when it is deployed
• Mean time to recovery (MTTR) - the average time it takes to recover from a failure
• Change failure rate (CFR) - the percentage of changes that result in a failure

LinkedIn utilized "Goals-Signals-Metrics" (GSM) to determine metrics that indicate developer productivity and happiness

• Developer Build Time (P50 and P90) - measures the time, in seconds, developers spend waiting for their builds to finish locally during development.
• Code Reviewer Response Time (P50 and P90) - measures how long it takes, in business hours, for code reviewers to respond to each update of the code review from the author.
• Post-Commit CI Speed (P50 and P90) - measures how long it takes, in minutes, for each commit to get through the continuous integration (CI) pipeline.
• CI Determinism - the opposite of test flakiness—the chance that a test suite’s result will be valid (not a flake)
• Deployment Success Rate - measures how often deployments succeed.
• Net User Satisfaction (NSAT) - measures, on a quarterly basis, how happy developers are overall with our development systems.

How does AI impact developer experience?

GitHub Blog -> Survey reveals AI’s impact on the developer experience

Key survey findings:

• AI is here and it’s being used at scale. 92% of U.S.-based developers are already using AI coding tools both in and outside of work.
• Waiting on builds and tests is still a problem. Despite industry-wide investments in DevOps, developers still say the most time-consuming thing they’re doing at work besides writing code is waiting on builds and tests.
• Developers want more collaboration. Developers in enterprise settings work with an average of 21 other engineers on projects—and want collaboration to be a top metric in performance reviews.
• And they think AI will help. More than 4 out of 5 developers expect AI coding tools will make their team more collaborative.
• Developers also see big benefits to AI. 70% say AI coding tools will offer them an advantage at work and cite better code quality, completion time, and resolving incidents as some of the top anticipated benefits.

Tips for tracking DevEx with GitHub

GitHub SE Blog -> Measuring the impact of Developer Experience and GitHub Copilot

DORA

• Pelorus Open Source Project
• DevOpsMetrics Open Source Project
• Lead Time to Changes is a measure the time taken from when a change is requested or initiated to when it is deployed. Lead Time to Changes metric requires two important pieces of data: when the commit happened, and when the deployment happened. This means that for every deployment, you need to maintain a list of all the changes included in the deployment.

Regular DevEx Surveys

Jonathan Carter, technical advisor of the CEO at GitHub

The more we can treat developer happiness as a goal, and measure thoughtful signals to make sure we’re doing that, the better. This requires addressing culture, tooling, and policies to make sure teams have clarity and autonomy.

Eirini Kalliamvakou, staff researcher at GitHub

Measuring DevEx underscores the need to continually check in with developers and see how they’re feeling. While organizations already know how to capture system performance data to gauge how efficient processes are, most don’t collect developers’ opinions on the systems they’re using. How can we improve developers’ experiences without checking in with developers about what their experience is?

Copilot Specific Survey Resources

• Copilot Survey Guide
• Sample Copilot Insights Survey (Google Forms Template)
• Survey reveals AI’s impact on the developer experience

Collaboration

• GitHub Blog: How to measure innersource across your organization
• Organization Insights provide data about your organization's activity, contributions, and dependencies.
• Pull Request Stats is a Github Action to print relevant stats about Pull Request reviewers. The objective of this action is to:
  • Reduce the time taken to review the pull requests.
  • Encourage quality on reviews.
  • Help to decide which people to assign as reviewers.

Short feedback loops

• What is your Onboarding time? Measure time to first commit for a new developer or developer onboarding onto new project.
• How long does it take for the longest CI/CD check to complete for a pull request? (GitHub Actions Metrics, Observability, and Reporting)

High degrees of automation and integration

• GitHub Blog: A beginner’s guide to CI/CD and automation on GitHub: CI/CD, security, release management, 3rd-party integrations, community and team management
• GitHub Blog: 5 automations every developer should be running

Low levels of friction or toil

• AI pair programming tools like GitHub Copilot accelerate coding boilerplate code and writing unit tests. Measure Copilot Acceptance Rate + Lines of Code generated.

Transparent, well-documented processes

• Are you measuring Customer Response Time? A strong response time indicates that the team has what they need to move quickly, while feeling empowered and helpful.
• Are your teams leveraging GitHub Discussions, GitHub Pages, and GitHub Wikis?