Azure DevOps: Configuration Portal
Scaling a self-service configuration portal in Azure DevOps to increase Azure's stability, saving Microsoft $1B ARR
At a Glance
My Role: Solo Lead UX Designer, Full E2E Design: Scoping, Research, Interaction & Visual Design, Prototyping
Timeline: 4 Months
Team: Immediate team: 1 PM, 4 developers, Secondary team: 2 PMs, 10 Developers
Problem
Azure Build Health (ABH) is an internal dev platform at Microsoft that reduces the risk of bad code being deployed which can bring down Azure services. Release Managers, Senior-Staff level developers and TPMs are the primary users of ABH. Currently, teams must be manually white-glove onboarded onto the platform one at a time due to the bespoke nature of each team's data pathways and pipelines. This process was not scalable.
I needed to craft a portal in ABH where users can add, remove, and modify their data pipelines and pathways without assistance.
What I Did
I owned and led the end-to-end strategy and design of each iteration of the onboarding experience. I conducted research, defined visual direction, created user journey maps, user flows, and prototypes, and designed integrations between Azure Build Health and other parts of Azure DevOps.
Impacts
1mo → 4hrs
Time to onboard teams onto ABH decreased.
1mo → 4hrs
Time to onboard teams onto ABH decreased.
1mo → 4hrs
Time to onboard teams onto ABH decreased.
40 → 1500
Teams onboarded onto ABH in 6 months utilizing the configuration portal.
40 → 1500
Teams onboarded onto ABH in 6 months utilizing the configuration portal.
40 → 1500
Teams onboarded onto ABH in 6 months utilizing the configuration portal.
$1B ARR
Will be saved by Microsoft from teams utilizing ABH.
$1B ARR
Will be saved by Microsoft from teams utilizing ABH.
$1B ARR
Will be saved by Microsoft from teams utilizing ABH.
Pixel Preview
The Configuration Hub where team's can manage their build pipelines, test pipelines, and more.
Each build pipeline can have bug and incident queries to establish automatic linking between builds and their issues without needing manual input.
Teams with advanced needs can customize "Release Views" on top of build pipelines so they can limit their view to only relevant data.
Act I: Creating an MVP to allow onboarded team's to add and maintain new test data and views.
“Teams utilizing Azure Build Health (ABH) need the ability to modify their Test Pipelines and Release Views without us manually doing it”
-Ask from ABH Onboarding PM
Primary Persona: Release Managers
Release Managers: senior-staff developers and technical Product Managers.
Test Pipelines
Pipelines that contain tests which are run against a build of code to test its quality.
Test Pipelines
Pipelines that contain tests which are run against a build of code to test its quality.
Test Pipelines
Pipelines that contain tests which are run against a build of code to test its quality.
Release Views
Custom filtered views teams could save and add to let them view relevant data.
Release Views
Custom filtered views teams could save and add to let them view relevant data.
Release Views
Custom filtered views teams could save and add to let them view relevant data.
Building a Configuration Hub
With a delivery timeline in 2 weeks, this work was planned as a quick-fix as there were limited dev resources for a larger scope of work. I approached this focusing on short-term impacts.
I worked with the PM to understand their current white glove experience. This let me learn about the items that existed outside of the current planned scope of work and design for the bigger picture.
I broke down the onboarding configuration experience into two pages: Release Views & Test Linking.
Test Pipelines
Release Views
Act II: Creating a self-service onboarding hub to allow Azure's largest team's to onboard themselves onto the platform.
“Fully onboard the last 10 large (80-2000 people) critical service teams onto ABH in the next 6 weeks."
-Ask from Azure's EVP, Scott Guthrie
Adding Additional Features
1 month later, Azure's EVP wanted the last 10 of Azure's bedrock teams (80-2000 people per team) onboarded onto ABH. I needed to add more features so teams could onboard themselves:
Build Pipelines
Build pipelines contain code, and new ones need to be linked to Azure Build Health.
Build Pipelines
Build pipelines contain code, and new ones need to be linked to Azure Build Health.
Build Pipelines
Build pipelines contain code, and new ones need to be linked to Azure Build Health.
Bugs & Incidents
Issues need to be associated with the correct build pipelines without needing to manually associated each time.
Bugs & Incidents
Issues need to be associated with the correct build pipelines without needing to manually associated each time.
Bugs & Incidents
Issues need to be associated with the correct build pipelines without needing to manually associated each time.
Combining Release Views and Build Pipelines
There was another delivery timeline of 2 weeks and was planned as a quick fix from the front-end side. The more complex and bespoke needs of the team's to be onboarded were handled with backend-only work. Our existing customers were utilizing Release Views to navigate inside of ABH so I made them the first level of hierarchy which leads to their applicable test pipelines and bug queries.
Our customers were all utilizing Release Views to see relevant data inside of build pipelines, so we as a team decided we wanted to enforce the usage of Release Views. This created a singular system inside of the configuration portal and the Azure Build Health portal, making release views the first level of hierarchy.
Inside of each Release View, users most common action would be to link new test pipelines so I prioritized that action and which data users would need.
Inside of each Release View, users most common action would be to link new test pipelines so I prioritized that action and which data users would need.
Inside of each Release View, users most common action would be to link new test pipelines so I prioritized that action and which data users would need.
Act III: Suffering from Success, in which I create a configuration hub that grows the platform from 44 teams to 1500 in 6 months.
“All 1,500+ dev teams on Azure must be fully onboarded onto Azure Build Health in 6 months." [I had 6 weeks of design time]
-Ask from Azure's EVP, Scott Guthrie
On the Twelfth Day of No Downtime, Azure's EVP Gave to Me…
Every December, Microsoft implements an annual code freeze, an intentional pause on code deployments. During this freeze, Azure experienced significantly fewer outages, highlighting how most outages are caused by faulty code pushes. This insight showcased the critical value of ABH. As a result, the EVP of Azure requested that our platform expand rapidly, from supporting 44 services to onboarding all 1500+ Azure services within the next six months.
Hol' up, Wait a Minute, Something Ain't Right
My team wanted to do another accelerated sprint to start dev work ASAP. We had already re-architected the product once, so to avoid the need for that again, I proposed taking a bit of time to clarify the black boxes we didn't yet fully understand of onboarding and continual onboarding. This ensured that the configuration hub would have a solid foundation and when custom features were needed, there would be less dev and product work to do.
Drawin' with Devs
I led three workshops and diagraming sessions with my PM and Dev Lead. As well as having a two heavily technical discussion and debates with other stakeholders, and from these discussions, I came up with a unified portal architecture.
Paradigm Lost and the Interaction Journey
Conducting more user research I found that the Release Views containing the Build Pipeline introduced friction for teams with less complex release processes. I swapped the hierarchy to be pipeline-centric, and the teams who needed more complex configurations could add a release view on top of the pipeline. I explored a variety of different interaction paradigms through this work.
Going Above and Beyond the Configuration Hub
While working on this project, I created several designs and discussed them with the larger Azure Build Health team to guide the product to work better for our users.
Integrated Flows with Other Azure DevOps Areas
I integrated Release Views into the Pipelines area of Azure DevOps to create more seamless and intuitive experience for our users.
Integrating Continual Onboarding into the Main Flow
Onboarding new test pipelines was a common task for our developers. I built out a workflow where users could do so right from the Azure Build Health page without needing to go into the configuration hub, reserving it for more heavy technical tasks.
Validation for the Present and the Future
I tested the designs with several teams inside of Azure to make sure the new designs supported all of Azure team's data pathways and pipelines. I conducted solo guerilla testing with dev teams outside of the Azure org to validate if the designs worked for them to ensure further scalability as the platform grew.
Configuration Hub shows entry points for pipelines and release views. Release views are the main view for complex teams, so I put them same hierarchy as build pipelines.
Queries track issues related to each new build that passes through a pipeline, requiring only a single-time entry.
Team's with advanced needs can customize "Release Views" on top of build pipelines so they can limit their view to only relevant data.
Impacts And Learnings
Working closely with my dev and PM partners, I was able to navigate this complex world to guide the product to a solution that works for 1,500+ bespoke teams with different complex data pathways and pipelines.
1mo → 4hrs
Time to onboard teams onto ABH decreased.
1mo → 4hrs
Time to onboard teams onto ABH decreased.
1mo → 4hrs
Time to onboard teams onto ABH decreased.
1mo → 4hrs
Time to onboard teams onto ABH decreased.
40 → 1500
Teams onboarded onto ABH in 6 months utilizing the configuration portal.
40 → 1500
Teams onboarded onto ABH in 6 months utilizing the configuration portal.
40 → 1500
Teams onboarded onto ABH in 6 months utilizing the configuration portal.
40 → 1500
Teams onboarded onto ABH in 6 months utilizing the configuration portal.
$1B ARR
Will be saved by Microsoft from teams utilizing ABH.
$1B ARR
Will be saved by Microsoft from teams utilizing ABH.
$1B ARR
Will be saved by Microsoft from teams utilizing ABH.
$1B ARR
Will be saved by Microsoft from teams utilizing ABH.
Learnings
I learned how to guide highly technical discussions with multiple stakeholders by creating designs to create alignment between stakeholders to clarify ambiguous structures and requirements.
There are some times where you need to design for edge cases, other times you should focus on the golden, or simplest, path first, utilizing progressive disclosure for users who need additional complexity.