ONBOARDING FUNNEL IMPROVEMENT
CONTEXT
Background: CircleCI support of GitLab, the platform's first decoupled experience, had been silently launched in the Spring of 2022 and went to GA in July. Teams using GitLab SaaS could now build, test, and deploy on CircleCI.
Objective: New users needed a simple funnel to get onto the platform so they could quickly get value by beginning to deliver software fast.
Problem: Users were dropping out of the onboarding funnel, and analytics indicated sign-up completions had plateaued well below target.
TIMELINE
This research spanned approximately 3 months of continuous iteration in 2-week sprints following GA.
STARTING OFF
-
Qualitative research had not been conducted on the new decoupled user onboarding experience. A baseline understanding of the experience was needed.
-
I communicated through Slack and meetings with designers, researchers, and PMs to align.
-
I originally planned to run an unmoderated usability test with UserZoom, recording participants' interactions with the onboarding flow, including reactions and expectations surrounding content and interface elements, time on task, task success rate, and other quantitative and qualitative feedback.
-
Ideally, I needed 6 quality participant recordings.
CHALLENGES
Due to the complexity of the tasks and technological and time constraints, unmoderated tests failed to yield the needed insights.
Within a few days, I pivoted to a moderated approach, using our Rolling Research interview field guide as a template and tailoring it to usability tests focused on the onboarding funnel.
I was able to quickly source participants through User Interviews from small companies to global enterprise corporations. Participants ranged from new Software Developers to experienced Principal Engineers. As the Growth Team Staff PM said, “These interviews are treasures!” They were chock full of actionable insights.
KEY QUESTIONS
-
What are user expectations when onboarding a complex developer tool, whether they use it for work or personal projects? What are their goals?
-
Where are users experiencing confusion? What impedes them from accomplishing their goals?
-
How do new-to-platform users interpret the terminology we use in the onboarding flow and the UI? Are these intuitive, too abstract, or misaligned with their mental models?
-
Where are users getting stuck? How do they find help? At what point do they give up?
-
Where are the moments of delight? How does the experience make them feel? (say vs. think vs. do)
Participants found themselves in a loop where they could not succeed if they had attempted to authorize using oAuth.
MEASURING RESULTS
The moderated usability interviews generated actionable insights.
Quantitative:
-
99% could not complete authorization with GitLab through either OAuth or personal access token (PAT).
-
No matter their engineering experience level, only two participants were extremely confident they knew what YAML was, a critical element of the CircleCI setup.
-
100% of participants needed to consult product documentation to get the most fundamental setup steps completed.
Qualitative:
-
100% verbally or non-verbally (sighs, facial expressions, etc.) expressed that they expected interactive password validation.
-
100% verbally or non-verbally expressed dissatisfaction while waiting for the page to load after they clicked "sign up."
-
While 99% indicated the sign-up process was very easy or neutral, we observed that participants struggled with authorization during project setup.
-
Several participants noted the quality of CircleCI's documentation.
-
What are user expectations when onboarding with a complex developer tool, whether they use it for work or personal projects?
-
Where are users experiencing confusion?
-
How do new-to-platform users interpret the terminology we use in the onboarding flow and within the UI? Are these intuitive, too abstract, or misaligned with their mental models?
-
Where are users getting stuck? How do they find help? At what point do they give up?
-
Where are the moments of delight?
While not all participants vocalized concern about the loading speed, non-verbal expressions showed low-key annoyance.
This validated assumptions.
ITERATING
-
By fine-tuning our analytics, tracking backend events, and hearing directly from users about pain points, how they loved OAuth, and how cumbersome personal access tokens were, PMs could prioritize the work of moving to OAuth only, dramatically simplifying the experience.
-
The team made continuous small changes to the funnel, gaining rapid feedback so that fluctuations in metrics could be mapped closely to each iterative change.
-
The product team worked together, brainstorming the problems and potential solutions, and that diversity of ideas produced better results faster.
-
The research validated the assumption that users expected a simpler way to run their first build. Using learnings gleaned from past research and experiments, I was asked to take a POC design and create an onboarding wizard to meet key user expectations.