top of page

ONBOARDING FUNNEL IMPROVEMENT

CONTEXT 

Background: CircleCI support of GitLab, the platform's first decoupled experience, had been silently launched in the Spring of 2022 and went to GA in July. Teams using GitLab SaaS could now build, test, and deploy on CircleCI.

Objective: New users needed a simple funnel to get onto the platform so they could quickly get value by beginning to deliver software fast.

Problem:
Users were dropping out of the onboarding funnel, and analytics indicated sign-up completions had plateaued well below target.

A page in the CircleCI onboarding funnel where users connect to source code

TIMELINE

This research spanned approximately 3 months of continuous iteration in 2-week sprints following GA.

study criteria in UserZoom

STARTING OFF

  • Qualitative research had not been conducted on the new decoupled user onboarding experience. A baseline understanding of the experience was needed.
     

  • I communicated through Slack and meetings with designers, researchers, and PMs to align.
     

  • I originally planned to run an unmoderated usability test with UserZoom, recording participants' interactions with the onboarding flow, including reactions and expectations surrounding content and interface elements, time on task, task success rate, and other quantitative and qualitative feedback.
     

  • Ideally, I needed 6 quality participant recordings.

KEY QUESTIONS

  • What are user expectations when onboarding with a complex developer tool, whether they use it for work or personal projects? What are their goals?
     

  • Where are users experiencing confusion? What is impeding them from accomplishing their goals?
     

  • How do new-to-platform users interpret the terminology we use in the onboarding flow and within the UI? Are these intuitive, too abstract, or misaligned with their mental models?
     

  • Where are users getting stuck? How do they find help? At what point do they give up?
     

  • Where are the moments of delight? How does the experience make them feel? (say vs. think vs. do)

responses to questions and ratings
participant finds themselves in an empty state, unsure of what to do next.

Participants found themselves in a loop where they could not succeed if they had attempted to authorize using oAuth.

CHALLENGES

Due to the complexity of the tasks, as well as technical and time constraints, an unmoderated approach failed to yield the meaningful insights needed. 

 

Within a few days, I successfully pivoted to a moderated usability interview approach, using our Rolling Research interview script as a template and tailoring it to usability tests focused on the onboarding funnel.

I was able to quickly source participants through User Interviews from small companies to global enterprise corporations. Participants ranged from new Software Developers to experienced Principal Engineers. As the Activation Team PM put it, “These interviews are treasures!” They were chock full of actionable insights.

MEASURING RESULTS

The moderated usability interviews generated significant actionable insights.

Quantitative:

  • 99% could not complete authorization with GitLab through either OAuth or personal access token (PAT).
     

  • No matter their engineering experience level, only 2 participants were extremely confident they knew what YAML was, a critical element of the CircleCI setup.
     

  •  100% of participants needed to consult product documentation to get the most fundamental setup steps completed.

Qualitative: ​

  • 100% verbally or non-verbally (sighs, facial expressions, etc.) expressed that they expected interactive password validation. 
     

  • 100% verbally or non-verbally expressed dissatisfaction while waiting for the page to load after they clicked "sign up." 
     

  • While 99% indicated the sign-up process was very easy or neutral and only 1 indicated it was moderately difficult, we observed that participants struggled with authorization during project setup or simply did not realize they were unsuccessful (which would present challenges with subsequent tasks).
     

  • Several participants noted the quality of CircleCI's documentation.

loading time after selecting sign up

While not all participants vocalized concern about the loading speed, non-verbal expressions showed low-key annoyance.

Personal Access Token complexity

This validated assumptions.

ITERATING

  • What are user expectations when onboarding with a complex developer tool, whether they use it for work or personal projects?
     

  • Where are users experiencing confusion?
     

  • How do new-to-platform users interpret the terminology we use in the onboarding flow and within the UI? Are these intuitive, too abstract, or misaligned with their mental models?
     

  • Where are users getting stuck? How do they find help? At what point do they give up?
     

  • Where are the moments of delight?

  • Through fine-tuning our analytics, tracking backend events, rather than relying on frontend events which can be skewed by pop-up blockers, and hearing directly from users what the most significant pain points were, how much they loved the option to authorize with OAuth, and how cumbersome personal access tokens were, PMs were able to prioritize the work of moving to OAuth only, dramatically simplifying the experience. 
     

  • The team made continuous small changes to the funnel, gaining rapid feedback so that fluctuations in metrics could be mapped closely to each iterative change.
     

  • The product team worked together, brainstorming the problems and potential solutions, and that diversity of ideas produced better results faster.
     

  • The research also clearly validated the assumption that users expected a simpler way to run their first build. Using learnings gleaned from past research and experiments, I was asked to take a POC design and create an onboarding wizard that would meet key user expectations. 

Early-stage interactive prototype

bottom of page