M365 Copilot Analysis Study

Blue Shield’s UX team was called upon to evaluate both user sentiment and overall effectiveness of M365 Copilot usage after its initial rollout at BSC and Stellarus. We wanted to find out which roles were seeing the most value, where adoption gaps existed, and what unmet training needs remained to determine how to optimize usage.

My team launched a pulse check survey to 600+ licensed M365 Copilot users to evaluate the tool’s capabilities, identify the highest value use cases and most importantly determine what is holding people back from getting high value out of Copilot. The insights I captured from analyzing this survey ensured that licenses could be directed to the right roles and unmet training needs would be addressed, maximizing impact and effectiveness of the tool. My findings also served a basis for general AI readiness at BSC.

My Role

UX Researcher

Worked with: UX team, directors, leadership

Timeline

June 2025 - August 2025

Tasks

Led UX research
Storytelling
XFN collaboration
Mixed methods analysis

Tools Used

Figma
UserTesting
Microsoft Excel
Microsoft Copilot
Viva Insights

Context

Blue Shield’s UX team created a pulse check survey around a year after launch to evaluate the effectives of M365 Copilot usage at BSC once licenses from a $2M investment were rolled out. Beyond measuring adoption, maturity, training gaps and high value use cases to ensure limited licenses went to the right roles and teams, this survey also sought to validate the previous hypothesis that these licenses would improve productivity by saving an employee about 2 hours of work per week. I was the primary researcher working through this analysis with some minor support from my team as my manager on PTO and my teammates not having enough bandwidth.

The Main Problem

Microsoft Copilot usage, and by extension company wide AI readiness at Blue Shield of California has a lot of room for improvement.

What Did I Do?

47 Slides

This study analysis was compiled in the form of a slide deck to be presented to leadership. I created 47 of the slides in the final presentation of the research.

600+ Users

The survey I analyzed was created and sent out on UserZoom. There were over 600 licensed users who responded to this survey, giving me a lot of data to work with.

There was a significant volume of both qualitative and quantitative data to analyze. I applied a mixed methods approach, integrating survey metrics with open-ended feedback to ensure I captured both measurable trends and deeper user perspectives.

A Lot of Analysis

Research Process

Pre-Analysis Thinking

The previous survey from one year ago with ~300 participants surfaced valuable information about M365 Copilot’s strengths and weaknesses at BSC. The key findings were:

Positive Feedback

  • 87% of users reported that they liked using M365 Copilot.

  • 57% of users felt confident in using M365 Copilot, but others felt that they were only “scratching the surface” and expressed desire to learn more advance use cases for the tool.

  • Users reported an estimated 2 hours of time savings per week from using M365 Copilot. This was self reported data, it may have been inaccurate.

Negative Feedback

  • 40% of respondents reported needing to remember to use Copilot. AI tools were not currently a part of daily workflows.

  • 63% of users did not use Copilot for technical use cases like data analysis. They felt Copilot often missed the mark, highlighting a tool issues than a user issue.

  • Users reported frequent unreliable output in Excel and Powerpoint.

Research Goals

With these findings as a baseline, I began to brainstorm my goals for the pulse check analysis. After confirming with my manager and my team, these were my goals with the study analysis:

  • Understand adoption and maturity – Evaluate how widely Copilot was being used and how effectively employees were integrating it into their workflows. Beyond this, we also wanted to see if its functionality with technical tasks, Excel or Powerpoint had improved since last year.

  • Identify unmet training needs – Pinpoint where users lacked guidance, support, or confidence that limited their ability to get value from Copilot. While training did exist, there was a notable amount of users who still felt they needed more assistance to use it effectively.

  • Surface high value use cases – Highlight the roles, tasks, and scenarios where Copilot delivered the most impact, ensuring licenses were allocated where they mattered most. Understanding who benefits most, how they use it, and why was essential not only to help others maximize Copilot’s value but also to lay the groundwork for positioning BSC as more AI-forward.

Target Users

  • BSC and Stellarus employees with licenses for M365 Copilot (internal study).

Deployment

  • Surveys created and sent out on UserZoom by UserTesting.

  • Slide deck created and presented via Figma.

Previous Survey

  • Secondary Research: I referenced the previous studies ran by my UX team on M365 Copilot as a starting point, using them as a foundation to inform myself and identify key areas to explore further.

Past Survey Insights

Research Process

My Methodology

Convergent Parallel Analysis

Quantitative Data: Insights were gathered directly through UserZoom and organized into charts and graphs.

Qualitative Data: Write in responses were exported from UserZoom and organized into Excel. I used thematic analysis to work through this data, looking for trends in user’s feelings about the tool as well as common benefits and issues they experienced.

*Text altered to protect NDA.

Key Findings

M365 Copilot Performance and AI culture at Blue Shield of California Improved


93% of respondents found M365 Copilot to be a valuable tool (up 6% from previous survey).

80% of users reported that using M365 Copilot provided them with meaningful productivity gains.

70% of all respondents use M365 Copilot daily with the most common applications being in Teams summaries, writing/ editing communications and general summarization.

Licensees save around 2.6 hours of work per week.

Users noted intangible benefits such better mental health as a result of better stress management and more time to commit to more intellectually challenging tasks.

Despite training being rolled out to every user with a license, some users were completely unaware of training existing, hindering adoption.

Many users noted that while they would love to learn more about Copilot, they do not have the time to invest into the long training courses.

Adoption is limited when users are unaware of how tools fit into their daily tasks. Bringing awareness to valuable, common use cases can result in greater adoption.

My Recommendations:

Revise Existing Training

Modular Learning

Gather More User Feedback

An audit of the current training site found that the layout needs some work. The current layout is not very user friendly, has beginner tutorials located in harder to reach areas than advanced tutorials and has a non-linear training structure. Redesigning the training site and making it more visible may help newer and more advanced users unlock greater value from M365 Copilot.

The current training paths themselves have several issues, mainly the amount of time to complete a module is not explicitly stated. Structuring existing training into shorter modules with defined time frames may improve accessibility. Organizing the material into beginner, intermediate and advanced learning pathways may help support users of all skill levels in more effectively leveraging the tool.

Likely due to negativity bias, I found that many users who reported low perceived value from the tool did not provide reasons why. Conducting 1-on-1 interviews and creating a dedicated M365 Copilot feedback channel could reveal concrete barriers and boost adoption by adding richer individual context.

Project Impact

Informing AI Adoption

The M365 Copilot pulse check analysis study validated past investment hypotheses from earlier research, which had encouraged the company to purchase roughly $2 million worth of licenses. The investment was successful and resources not wasted; users reported meaningful boosts to their productivity, finding that M365 was a valuable tool to have at their disposal, lower administrative costs and some even reported improvements to their mental health because they could now manage their workload and stress better.

High Value Use Cases

Time Savings

My analysis uncovered the highest value roles and use cases currently for M365 Copilot, as well as the roles and use cases that perceived the lowest value gains. This knowledge can inform future training as well as ensure that future license investments are rolled out to the areas that experience the greatest benefits.

My team and I calculated that BSC employees save an estimated 2.6 hours per user per week with usage of M365 Copilot, approximately 0.6 hours more than a year ago. Beyond getting work done more efficiently, users stated the time savings brought other benefits such as being freed to focus more on higher impact projects rather than administrative tasks like meeting notes or writing emails.

Challenges

Bias and Learning By Doing

Avoiding Bias

One of the main challenges I experienced during this analysis was recognizing and managing my own biases. At times I leaned toward confirmation bias when results aligned with earlier hypotheses of productivity gains, and I anchored too heavily on last year’s survey as a benchmark. I even cherry-picked quotes which in hindsight was completely incorrect, elevating insights that represented only 2–5% of the data which risked overstating their importance to try and make the data fit the conclusions the previous study had reached.

To counter this, I revisited the dataset and checked both my findings and interpretations with my incredible UX team to keep the results grounded and made sure the insights were accurate and meaningful. I redid my thematic analysis with guidance from our lead UX researcher and found much richer context that allowed me to discover more impactful insights. My analysis ended up being its own, not trying to conform to the findings of previous surveys and being as bias-free as possible.

Self Teaching

The other main challenge I dealt with during this study was was my first in depth UX research project, and my manager was on PTO for most of the duration of the analysis. I was unable to ask her questions throughout most of the process, so instead I leaned on my team who were super supportive and helped me work through key parts of the analysis. I also used BSC’s resources to teach myself through courses and kept learning as I went. Eventually, my manager returned from and we finished up and presented the analysis to key stakeholders together.

Reflection

Becoming A Better Researcher

What I learned

Navigating Ambiguity

  • With my manager on PTO, I relied on courses and the support of my incredible team to quickly self-teach research techniques and apply them in real time to keep the study on track. While my first pass showed bias and some cherry-picking, revisiting the analysis with my team’s guidance helped me correct those mistakes and ultimately deliver valuable, impactful insights.

Staying Positive

  • Despite the steep learning curve and receiving varied critiques on all of my iterations of the analysis, I stayed positive and open-minded. This mindset helped me absorb feedback constructively, refine my analysis and ultimately produce far more impactful results.

This summer was most definitely one of growth. I learned many new research methods, analysis techniques, how to check my own biases, stay positive through rounds of critique, maintain a story through analysis and use feedback to create more impactful work. Most importantly, I discovered that being a good researcher is about more than analyzing data, it’s about asking the right questions, communicating findings clearly, and ensuring insights drive meaningful outcomes. I’m grateful for the opportunity to work on this project and excited to keep building my skills as a UX designer and researcher.

Manager Feedback

POV: Working With Danial

Thank you for reading :)