M365 Copilot User Research Analysis Study

A year after Blue Shield of California invested in M365 Copilot licenses, the company wanted to assess if the rollout was successful. Blue Shield’s UX team was called upon to evaluate both user sentiment and overall effectiveness of M365 Copilot usage. We wanted to find out which roles were seeing the most value, where adoption gaps existed, what unmet training needs remained and if the hypothesis of saving around 2 hours per week from Copilot usage was accurate.

Over 600 licensed users across BSC and Stellarus responded, generating 13,200+ data points. As the primary researcher (with my manager on PTO and limited team bandwidth), I led the full analysis, uncovering which roles were seeing the most value, where adoption was lagging, and what prevented employees from getting more out of Copilot. The findings were synthesized into a 48-page report presented to senior leadership.

The insights I captured from analyzing this survey ensured that licenses could be directed to the right roles and unmet training needs would be addressed, maximizing impact and effectiveness of the tool. My findings served a basis for general AI readiness at BSC, informing future training plans that are created specifically for improving usage of M365 Copilot.

My Role

UX Researcher

Worked with: UX team, directors, leadership

Timeline

June 2025 - August 2025

Roles/Responsibilities

Led UX research
Storytelling
XFN collaboration
Mixed methods analysis

Tools Used

Figma
UserTesting
Microsoft Excel
Microsoft Copilot
Viva Insights

The Problem

At the end of the M365 Pilot Program at Blue Shield of California, we did not understand if the rollout was successful.

What Did I Do?

48 Page Report

This study analysis was compiled into a 48 page report via Figma which was presented to senior leadership.

600+ Users

The pulse check survey was created and sent out on UserZoom. I imported results from UserTesting into Excel for analysis.

There was a significant volume of both qualitative and quantitative data to analyze with 20+ questions, many of them open-ended. I applied a mixed methods approach to capture both measurable trends and deeper user perspectives.

13,200+ Datapoints

Research Process

Pre-Analysis Thinking

The previous survey from one year ago with ~300 participants surfaced valuable information about M365 Copilot’s strengths and weaknesses at BSC after its initial smaller rollout as part of the pilot program. The key findings were:

Positive Feedback

  • 87% of users reported that they liked using M365 Copilot.

  • 57% of users felt confident in using M365 Copilot, but others felt that they were only “scratching the surface” and expressed desire to learn more advance use cases for the tool.

  • Users reported an estimated 2 hours of time savings per week from using M365 Copilot. This was self reported data, it may have been inaccurate.

Negative Feedback

  • 40% of respondents reported needing to remember to use Copilot. AI tools were not currently a part of daily workflows.

  • 63% of users did not use Copilot for technical use cases like data analysis. They felt Copilot often missed the mark, highlighting a tool issues than a user issue.

  • Users reported frequent unreliable output in Excel and Powerpoint.

Research Goals

Target Users

  • BSC and Stellarus employees with licenses for M365 Copilot.

Deployment

  • Surveys created and sent out on UserTesting.

  • Slide deck created and presented via Figma.

Previous Survey

  • Secondary Research: I referenced previous studies ran by my UX team on M365 Copilot as a foundation to inform myself and identify key areas to explore further.

Previous Pilot Launch Survey Results (From 1 Year Ago)

Research Process

My Methodology

To analyze more than 13,200 data points, I used a convergent mixed-methods approach that combined quantitative trends with qualitative depth.

Quantitative Data: Insights were gathered directly through UserZoom and organized into charts and graphs.

Qualitative Data: Write in responses were exported from UserZoom and organized into Excel. I used thematic analysis, looking for trends in user’s feelings about the tool as well as common benefits and issues they experienced.

*Text altered to protect NDA.

Key Findings

M365 Copilot Performance and AI culture at Blue Shield of California Has Improved

Copilot Performance Ratings Improved

  • 93% of respondents found M365 Copilot to be a valuable tool (up 6% from previous survey).

  • 89% of users rated M365 Copilot high quality.

  • 80% of users reported that using M365 Copilot provided them with meaningful productivity gains.

  • Average reported time savings were around 2.6 hours per week.

  • Users also noted intangible benefits such as reduced burnout, better stress management, and more time for higher-impact work

My Recommendations:

*Images shown above are part of the larger 48 slide report, feel free to contact me to see the entire report!

Low Tool & Training Awareness Persist

  • Despite training being rolled out to every user with a license, some users were completely unaware of training existing, hindering adoption.

  • Many users noted that while they would love to learn more about Copilot, they do not have the time to invest into the long training courses.

  • Adoption is limited when users are unaware of how tools fit into their daily tasks. Bringing awareness to valuable, common use cases can result in greater adoption.

Adoption & Value Differ By Workflow

  • 70% of all respondents use M365 Copilot daily with the most common applications being in Teams summaries, writing/ editing communications and general summarization.

  • Knowledge workers and technical users report the highest productivity gains and value from Copilot, indicating strong alignment with their workflows and task types. Administrative assistants reported the lowest value gains.

  • The 39% of respondents who mainly use Teams meeting summaries reported high tool value and significant productivity improvements.

Research Impact

Informing AI Adoption At Blue Shield

The M365 Copilot pulse check analysis study validated the investment hypothesis which had encouraged the company to make a sizable investment into new licenses. The investment was successful and resources were not wasted, with a majority of users reporting meaningful boosts to their productivity. Users found that M365 was a valuable tool to have at their disposal, administrative costs were lowered and some users even reported improvements to their mental health because they could now manage their workload and stress better.

Validated Pilot Hypothesis 

Informed Continued Training

The findings from my survey analysis proved the pilot hypothesis correct, showing that licensees save around 2.6 hours of work per week. The time savings brought other benefits such as being freed up to focus more on higher impact projects, improving communication quality, and supporting health literacy. I proved continued investment into this tool moves the needle and would be incredibly beneficial for BSC.

My analysis uncovered the highest value roles, the lowest value roles, and the specific use cases where M365 Copilot either excelled or fell short. A content audit of the current training site revealed additional areas where training could be improved. This knowledge is currently informing the development of new training plans by senior leadership to optimize Copilot usage.

Shaped Adoption Strategy

I identified where Copilot delivers maximum value and where it does not, giving BSC’s AI leadership clarity on how to prioritize future license distribution. My insights ensure that new licenses are allocated to the roles and teams where they will have the greatest impact, maximizing ROI and driving effective adoption across the company.

Reflection and Learnings

Becoming A Better Researcher

This study was a major milestone in my growth as a UX researcher. I learned how to check my own assumptions, synthesize a massive dataset with mixed methods analysis, and build a research report that clearly communicated actionable insights to leadership. I also gained experience validating a major investment hypothesis with real evidence, confirming where Copilot delivered value and guiding future investments. Most importantly, I learned that effective research goes beyond finding insights, it requires asking the right questions, grounding conclusions in evidence, and ensuring the work drives meaningful decisions.

Manager Feedback

POV: Working With Danial

Thank you for reading :)