A step-by-step guide to piloting M365 Copilot: Limitations and learnings
You are currently viewing A step-by-step guide to piloting M365 Copilot: Limitations and learnings
A step-by-step guide to piloting M365 Copilot: Limitations and learnings

To pilot Microsoft 365 Copilot, start by creating a clear plan and building a cross-functional team that’s willing to embrace the future of AI-assisted work. The team we built includes a mix of employees from various departments, including R&D, Sales, Marketing, HR and others. It’s also important to include users with various levels of technical acumen, ranging from foundational proficiency in Microsoft 365 to more advanced users who can script or even build their own apps.

Here’s a step-by-step rundown of how to approach a pilot program for Copilot. These are the steps we took, which can serve as a blueprint for other organizations looking to explore Copilot’s capabilities.

Step 1: M365 Copilot License Activation and Account Setup

Before embracing this new technology, all team members must have access to the tool. We activated a number of 365 Copilot licenses and assigned them to the appropriate accounts.

In our case, there was an initial hiccup with licenses assigned to the wrong accounts, but it was quickly resolved. Keeping a keen eye on these details from the get-go can prevent delays in deployment.

Step 2: Tracking Microsoft 365 Copilot Usage and Accountability

When piloting 365 Copilot, everyone needs to get the most out of their subscription. We used our internal M365 management solution to keep tabs on usage, but other options are available for monitoring usage and adoption. (E.g. a different Microsoft management tool or even PowerShell) But right now, there’s no ready-made tool that can do this for you.

Ultimately, this approach helps us continuously identify who is actively using the tool and who may need a nudge or additional support.

Moreover, I recommend following the principle of use-it-or-lose-it—if there’s no usage within a week, the license can be given to someone else who is keener to try out Copilot’s possibilities.

Pilot Microsoft 365 Copilot: a step-by-step guide

Step 3: Establishing Feedback Mechanisms for Microsoft 365 Copilot

To make sure all feedback gets back to IT, teams should establish a system for gathering feedback from your beta testers.

We created a Teams chat where all Copilot users could share how they were using Copilot. Users were then required to prefix all feedback messages with “Feedback:”

That way, the team can easily sort through the conversations (or have Copilot do it for you!) and extract feedback.

Step 4: Real-Time Experimentation and Documentation for M365 Copilot

Encouraging the beta testers to use Copilot for a variety of tasks—from scheduling meetings to summarizing discussions—allows you to collect a wide range of data on its performance.

Document both positive and negative feedback in real-time,to help the other users and the IT team get a nuanced understanding of Copilot’s capabilities and limitations.

M365 Copilot use cases

One of the best ways to evaluate the effectiveness of a new technology is to look at the real-world scenarios and advantages it creates.

Through our 365 Copilot trial, these are the best use cases that surfaced:

Use case #1: Summarize meetings and list action items with M365 Copilot

One of the standout features of Copilot is its ability to digest lengthy conversations and distill them into concise summaries. This is particularly useful for meetings, where key points and decisions can be quickly reviewed.

During a meeting, one of our beta testers used the prompt”Recap the meeting so far” with Copilot. The tool provided a quick summary to the users who joined the meeting late, helping them quickly catch upon the discussion.

Moreover, the “Get action items” command,used at the end of a call, proved to be beneficial, albeit with varying degrees of accuracy.

It provides a solid starting point for defining tasks and responsibilities post-meeting. However, you should expect to manually adjust action items after a meeting since Copilot isn’t always super accurate.

Example of “Recap the meeting” functionality:

Example of Recap the meeting functionality in Copilot

Use case #2: Summarize emails and generate to-do lists with M365 Copilot

Another highly praised use case is Copilot’s ability to sift through an overflowing inbox and summarize the content, extracting a list of to-dos.

Microsoft MVP nominee and in-house M365 expert, Kas Nowicka, has shared her personal experience using Copilot:

“The best use case so far and available out-of-the-box for me is its ability to summarize and catch up on unread emails with a list of to-dos.”

Team members found this particularly valuable, as it helped them catch up on missed correspondence and prioritize tasks without having to read every single email.

Summary by Copilot

Use case #3: Use M365 Copilot for Sales and Marketing activities

Extending beyond the confines of internal operations, Copilot also demonstrated its utility in customer-facing situations. For instance,when asked to generate a bilingual presentation description for an upcoming event, Copilot delivered a succinct and professional summary that me tour expectations, saving valuable time in the process.

Here’s the Copilot prompt that Vincent Convert, one of the Regional Sales Managers responsible for French-speaking territories, used to create his presentation description in both French and English:

“Organize a presentation on the topic of governance in Microsoft 365 for an upcoming event dedicated to end customers. Provide a brief description for this presentation in French and English.”

Copilot provided a suitable description, demonstrating its ability to save time and aid in bilingual communication.

Crafting effective M365 Copilot prompts

The art of crafting Copilot (or ChatGPT or any other AI models) prompts turns out to be a critical skill when working with Copilot. A well-phrased prompt could mean the difference between a useful response and a misfire.

Our team worked on developing a set of “out-of-the-box” prompts that yielded consistently positive results, tailored to common business scenarios. Sharing these prompts within the team helped everyone get the most out of Copilot.

Here are some examples of the prompts our team used:

  • To summarize meetings, use ‘Recap the meeting so far.’
  • To generate a list of action items after a meeting, use ‘Get action items.’
  • To see a summary of to-dos from your emails, use ‘Summarize my emails and list the to-do’s.’
  • Organize a presentation dedicated to [topic of presentation] for [event] dedicated to [audience] in [language].

Here are some more prompts that may be useful:

Copilot Prompt Example 1: Drafting a Professional Email Response

This prompt is helps employees quickly and professionally respond to common questions. Use it to generate concise, polite, and informative replies, saving time and ensuring communication stays consistent.

“Draft a professional response to a client asking for an update on their project status, ensuring to convey that we are on track and will meet the deadline. Include a brief overview of the completed milestones and what is scheduled for the next phase.”

Copilot Prompt Example 2: Organizing Project Information into a Presentation

This example uses Copilot’s capabilities to prep meeting materials. Use this prompt below to organize thoughts and data into a coherent outline and build presentations faster.

“Create a presentation outline for the quarterly project review meeting. Include slides on project objectives, progress to date, challenges faced, solutions implemented, and next steps. Use bullet points for slide content and suggest relevant graphics or charts.”

Copilot Prompt Example 3: Generating a Report from Data

This prompt is intended for data analysis tasks. Use it to sift through data you provide (in this case, sales data) to extract meaningful insights, trends, and suggestions. This can significantly speed up the process of report generation and make sure key information is not overlooked.

“Generate a summary report from the attached sales data for the last quarter. Highlight top-performing products, identify sales trends, and suggest areas for improvement. Format the report with an introduction, key findings, detailed analysis, and conclusion.”

Limitations of Microsoft Copilot

Limitation #1: Scheduling Constraints in M365 Copilot

One of the more immediate limitations we met was Copilot’s inability to perform tasks that require direct interaction with calendar tools and scheduling software.

One employee highlighted this gap:

“I asked Copilot to schedule a meeting and it got the available times wrong. Plus of course, it cannot schedule an actual meeting as it is only a chat tool.”

This reflects a broader challenge with Copilot’s integration capabilities, where it can provide information but cannot take actions within third-party applications or services just yet.

Limitation #2: Accuracy and Precision in Responses in M365 Copilot

Our team found that the accuracy of Copilot’s outputs could sometimes be hit or miss, particularly when generating action items or summarizing complex discussions.

The “Get action items” feature, for instance, was less accurate than the summarization function, indicating that Copilot sometimes struggles with context and specificity. This necessitates a degree of oversight and manual correction, which can offset some of the time-saving benefits.

In fact, in a recent Copilot readiness assessment of CoreView’s customers over 83% of respondents are concerned about Copilot generating incorrect or out-of-date information.

Over 83% of survey respondents are concerned that Copilot will generate incorrect or out-of-date information.

Limitation #3: Limited Task Execution in Microsoft Copilot

As an AI chat tool, Copilot is bound by its nature to be an assistant that provides suggestions and information rather than executing tasks. This means that while it can propose a draft for a presentation or an email, it cannot carry out the actual sending or presenting. This limitation emphasizes the importance of understanding Copilot’s role as a supportive tool rather than a replacement for hands-on task management.

Comparing M365 Copilot with Other AI Tools

When comparing Copilot to other AI tools like our internal version of ChatGPT, CoreChat, the majority of the team members noticed a significant difference in productivity enhancement.

The sentiment was that “CoreChat is “10x” more efficient and helpful, which helped us conclude for the time being, that Copilot, while useful, may not always meet the high expectations set by other AI chat tools in certain applications. This points to a need for continuous development and fine-tuning of Copilot’s algorithms to better serve its user base.

The pilot also revealed that Copilot’s performance could vary in real-time scenarios. For example, when asking simple questions in Copilot Edge, the responses received were not always on par with the same queries posed to ChatGPT. This inconsistency can affect user trust and reliance on the tool for immediate, accurate information.

Microsoft’s update to 365 Copilot access to SharePoint

In a recent conversation with Microsoft 365 expert Kas Nowicka, she shared an update on Microsoft’s new feature called Restricted SharePoint Search:

“This new capability limits Copilot for Microsoft 365 to only search within 100 selected SharePoint Online sites to avoid accidental exposure of sensitive data. This feature, available from April onwards, narrows down the scope of enterprise-wide search, and focuses on a smaller number of sites and user-generated content, while still allowing users to search in their frequently visited sites, OneDrive files, and files that are shared with them. This measure aims to balance security concerns with functionality, addressing possible data disclosure issues while preserving Copilot’s usefulness.”

The Restricted SharePoint Search feature is a welcome addition to the Copilot for Microsoft 365 tool, as it enhances the security and privacy of data in large enterprises. By limiting the search scope to only relevant and trusted sites, users can avoid exposing sensitive information to unauthorized parties or external sources.

However, this feature may also come at the cost of reducing the functionality and accuracy of Copilot, especially in real-time scenarios where users need quick and reliable answers. Therefore, users may need to weigh the trade-offs between security and convenience when using Copilot for their work tasks.

Microsoft 365 Copilot – the future of work?

Microsoft 365 Copilot stands at the forefront of workplace evolution, an AI tool that promises to streamline our daily tasks. This pilot experience has revealed its potential and its current limitations, offering a balanced perspective on its role in an organization’s digital toolkit. As organizations continue to use Copilot, it’s important to balance human creativity with technological assistance.

Ultimately though, organizations that embrace Microsoft 365 Copilot in daily operations will gain immense value.

As you navigate through all of AI tools available, remember that choosing the right technology can seriously improve your business outcomes. And, keep in mind how you will adjust your M365 governance strategy to account for this new technology.

Source: https://coreview.com/blog/m365-copilot-pilot