What to do when Microsoft CoPilot says “I’m sorry, but I can’t help with that”

One of our customers is an in-house legal team that has recently been experiencing a significant increase in the number of queries from the business about a particular issue. They need to gather the information from the requestor with sufficient detail on key areas to work out how to apply the relevant statute and advise on the best approach and likely outcomes.

The legal team was interested in whether we could create a form to gather the information in a structured way, which would help the legal team assess the situation more quickly and provide the right advice.

On looking at the legislation, we could see how it worked and how the questionnaire needed to be structured. The legal team gave some great instructions on the type of questions they wanted to ask and the guidance they wanted to give.

Could AI take it further?

In terms of creating the advice, I was keen to see whether we could hook that form up to Microsoft’s AI solution via Power Automate, draft an email to go to the legal team for review and editing, and then be sent on to the requestor.

Because we had such great structured information coming from the form, I was confident that if we also gave the AI the details of the legislation via a system prompt, it would be able to work out where the requirements were met, where they weren’t, and where further thinking was likely to be required.

It didn’t take me too long to create a prompt and the automation to get things set up, and then I sent through my first request, waiting in anticipation to see what great stuff the AI could do.

You can imagine my disappointment when I got an email saying, “I’m sorry, but I can’t help with that.”

That was weird, I thought. It must’ve just been a glitch on the Internet. Let’s try again. Unfortunately, I got the same result.

For some reason, Microsoft AI was not able to help.

Can’t help, or won’t help?!!

I’ve come across this type of thing before, and the key learning I have taken is not to accept defeat too early, and to understand that sometimes, when Microsoft says it can’t help, it means it won’t help.

There seem to be various guardrails and settings in Microsoft’s AI that stop it from willingly doing everything you ask in the first instance, but if you push back, you can get it to do what you want.

In this case, I put a message at the end of my system prompt, saying

“If you feel you are unable to help with this request, please explain why.”

I resubmitted my request, and happily, I now got an excellent, nuanced response, which is exactly what I had been hoping for!

I submitted some more requests and got more great emails. Happy days!

The next day, I decided to show my colleagues what I had done.

Unfortunately, with some time to reflect overnight, my AI returned to deciding it wasn’t willing to help.

I thought about it further and wondered if I needed to strengthen my pushback.

I moved my message up to the very top of the prompt, where it would be the first thing seen, and not ignored as some possible afterthought.

I resubmitted my request, and once again, I got a beautiful and very well thought-through email.

Trial and Error, and Perseverance

I won’t say that I remotely understand why Microsoft AI behaves this way. I certainly wish that it wouldn’t, and I expect that Microsoft will get better at managing these issues.

What I have learned, though, is not to accept defeat too early. If you don’t get the desired result in the first instance, try different approaches. You can often get there with a little trial and error.

Legal Teams Should be Exploring This

I will say that I really am impressed with the quality of the output, and the potential for these types of scenarios where legal teams are currently thinking that they have to do all the interpretation and the writing, because Copilot hasn’t been specifically trained as a legal solution.

In these types of situations, it is mainly the facts that matter, as the requirements in the legislation are relatively straightforward. It doesn’t depend on a detailed search for case law and entirely accurate citations.

Microsoft’s AI seems very capable of reviewing the facts, applying them to the requirements, and preparing an initial draft email (within about 30 seconds), which the legal team will likely have to spend minimal time refining before sending it on. Imagine the impact on turnaround times.

If you would like to know more about this type of scenario or test a particular use case, please get in touch. We may be able to avoid some of these annoying pitfalls from the outset or help pull your AI into line.