M365 Copilot Analysis Study

Blue Shield’s UX team was called upon to evaluate both user sentiment and overall effectiveness of M365 Copilot usage after its initial rollout at BSC and Stellarus. We wanted to find out which roles were seeing the most value, where adoption gaps existed, and what unmet training needs remained to determine how to optimize usage.

My team launched a pulse check survey to over 600 licensed M365 Copilot users to evaluate sentiment, the tool’s capabilties and identify the highest value use cases. The insights I captured ensured licenses could be directed to the right roles and teams, maximizing impact and effectiveness.

My Role

UX Researcher

Worked with: UX team, directors, leadership

Timeline

June 2025 - August 2025

Tasks

Led UX research
Storytelling
XFN collaboration
Mixed methods analysis

Tools Used

Figma
UserZoom
Microsoft Excel
Microsoft Copilot

Context

As part of my internship, I along with a PM intern and engineering intern were brought together by senior leadership to create a product that would allow care managers to do their jobs more effectively. In addition to hospital work, Blue Shield’s care managers use a digital platform called Care Connect to asses and administer care for their patients. However, digital care is largely impacted by data delays, with care managers often receiving the data they need weeks or even months late, which negatively affects both manager and patient, especially if the data is no longer accurate. While my internship was at Blue Shield, this project was completed for Stellarus, which can be thought of as the technological arm of Blue Sheild.

The Main Problem

Microsoft Copilot usage, and by extension company wide AI readiness at Blue Shield of California has a lot of room for improvement.

My Deliverables

What Did I Do?

47 Slides

This study analysis was compiled in the form of a slide deck to be presented to leadership. I created 47 of the slides in the final presentation of the research.

600+ Users

This project was created under the company Stellarus, the technology arm of Blue Shield. I created a new design system combining elements of Blue Shield for the new Stellarus product.

In addition to designing the product, since we had such a short amount of time to present it I worked directly with the engineering intern to aid in the development of Stella through Roo Code OpenRouter and minor CSS.

A Lot of Analysis

The Solution

Meet Stella

A generative AI data query tool that allows care managers to instantly access vital patient data to better administer individualized care.

*Screens partially blurred to protect NDA

*No real patient health information is shown here, the display text above is pulled from a JSON document containing artificial patient data

Research

Initial Thinking

With any project, I like to start with questions:

  • How do care managers currently request, receive, and use patient data in their daily workflow?

  • What barriers currently exist that prevent care managers from accessing data? Who do they rely on for data?

  • How does delayed or inaccurate data affect care managers’ ability to support patients?

Blue Shield of California had already done research on the care manager problem space in the past for the creation of Care Connect, a care management system that combines data from 13 different systems into one platform. I worked with the UX team to synthesize the existing interview data and personas both pre and post launch for Care Connect to determine what issues that care managers faced in the past and that carry over to today. Knowing Stella would be an internal product, we tailored our synthesis to focus on internal workflows, data access barriers, and system usability challenges unique to Blue Shield’s care managers.

Findings: data access at Blue Shield is too difficult.

  • Delayed decision making: Care managers cannot act quickly, which affects timely patient support.

  • Data barrier: Care managers need to go through at least one middle man to access their data. The process of receiving data involves contacting someone in charge of data then waiting for it to be sent over, which is a lengthy process.

  • Difficulty drilling down into data: Even when data arrives, it is often not in a usable format for deeper analysis. You need to ask the person who sent you the data to drill down into it for you instead of being able to do it yourself.

Primary Goal

In a nutshell: Help Blue Shield care managers access accurate patient data faster and more easily so they can deliver timely, effective care.

What this means: Design an internal tool that can streamline data access, reduce approval bottlenecks, enable care managers to drill down into patient information on their own and integrate with BSC’s current data infrastructure. By ensuring data is both accessible and timely, Stella could potentially allow care managers to make more informed, quicker decisions at the point of care and improve overall workflow efficiency through accurate, timely information.

Target Users

  • Care managers at Blue Shield of California (internal company usage only).

Deployment

  • GenAI chatbot that pulls from all existing Blue Shield from a centralized database.

Research Methods

  • Analyzing previous care manager research.

  • Partnering with engineering to understand how data is structured, accessed and managed at BSC.

Research Outcomes

Challenges

Navigating Bumps in the Road

This was a project given to us with only about 3 weeks to complete before presenting it to senior leadership, including multiple directors and the C-suite (no pressure!). Earlier on, we knew the UX, PM and engineering interns would be collaborating on a project as a group of 3, but we never knew for what or why. Initially, due to ambiguity from upper management we had to deal with product requirements changing weekly or even multiple times during the week while in our initial product exploration phase. Our first concept we were told to work was as a software that was supposed to do PM work on Jira, not remotely similar to what Stella ended up becoming. This lead to a lot of challenges, namely:

Unclear Scope and Direction

Compressed Timeline

Stakeholder Communication

To manage unclear scope and scope creep, I prioritized features that directly supported the evolving core concept and emphasized frequent team alignment to quickly assess feasibility when leadership refined priorities. This approach saved countless hours of wasted effort and ensured that once Stella’s concept was finalized, we included only the most necessary features.

With only 3 weeks to work on Stella once our xfn teams agreed on its concept, we had to build fast. I focused on designing while prioritizing functionality, ensuring that the tool only did what it needed to do. To assist our engineer, I learned how to use OpenRouter and RooCode to help create functionality for the Stella prototype and ensure we could ship on time.

To navigate the lack of clarity, I frequently communicated changes to my UX team, trying to understand what I needed to create and what already existed to avoid redundancy. I also escalated the ambiguity problem, getting my manager on our calls with leadership, which greatly helped to clarify direction and aligned the team towards what we were working on.

Project Impact

Informing Blue Shield’s Future

As of today, Stella is a proof of concept project, meant to validate the idea of using a chatbot interface to query a centralized data base. This tool was specifically meant for care managers, helping them avoid delays in their work due to poor data access. Potential future outcomes of this project include:

Assessing Timelime

Potential Savings

Validating Feasibility

With a team of 3 interns, we were able to get this project to a working state in only 3 weeks thanks to the assistance of AI development tools. This timeline can inform future endeavors at Blue Shield and help in setting realistic expectations.

According to a 2011 study, inadequate care coordination led to $25-$45 billion in avoidable healthcare costs, a number that has likely grown with the rising cost of healthcare. By streamlining data access for care managers, the project showed potential to reduce delays in decision-making, improve early interventions and ultimately save money for both customer and Blue Shield while boosting HEDIS and STAR ratings.

The Stella proof of concept validated both feasibility and business value by showing that care managers could access patient data faster and more reliably with the tool. It confirmed the idea worked technically while directly addressing the core business problem of delayed data access, demonstrating impact before Blue Shield committed to larger-scale investment.

Reflection

Adapting to Change

Be agile

  • I had to frequently navigate unclear scope, changing requirements and projects moving in completely different directions across the development of Stella. I also had no programming experience going into the internship but ended up assisting an engineer with development through the power of GenAI tools, something I did not expect myself to do.

What I learned

Communication

  • Most of the confusion and backtracking to complete Stella was because of poor communication. Despite receiving unclear instructions from leadership as to what we were building, I made the mistake of not immediately bringing in my manager or the UX team to help provide clarity. To correct this, I made it a habit to share frequent updates with my UX and Stella team, clarify what needed to be designed versus what already existed and I escalated by involving my manager in calls with leadership, which quickly brought clarity and focus. These experiences reinforced that proactive, transparent communication is a vital skill to have.

Follow the North Star

  • Stella reminded me to always keep sight of a north star. Despite severe ambiguity, scope creep and multiple pivots, the ultimate goal remained making data access easier and faster for care managers. Holding onto this north star gave our team clarity and purpose when the details were hazy. I focused on keeping the main thing the main thing, and it all worked out eventually.

My internship at Blue Shield of California was my first real, hands-on industry design experience. If I had to sum up my summer in one word, it would be adaptability. I dealt with constantly changing requirements, learned how to say no when I didn’t have the bandwidth for extra projects, picked up programming skills, and absorbed countless new lessons about design. Most importantly, I learned that being a good designer isn’t just about creating good designs; it’s about fostering collaboration, communicating clearly, aligning stakeholders and empowering others to work more effectively. I loved working on Stella and learned so much more than I could’ve expected. I am very grateful for the opportunity I had to work there this summer and I can’t wait to keep on learning.

Manager Feedback

POV: Working With Danial

Thank you for reading :)