Generative answers from an authors book catalog

[TimmyJadeTech]

Hi everyone 🙂

I'm struggling with what I assumed would be quite a simple use case for generative AI answers from word documents as follows:

Use case: I am managing the 365 environment for an author of 25+ published books. All manuscripts are stored in OneDrive in our 365 business environment. We own copyright on all material. We wish to leverage generative AI in order to ask questions about the characters, story line, scenes and so on from the back catalogue of books in order to check the consistence for upcoming books, research back stories and provide general ease of access to the content of the books.

Environment:  We have 365 business and one user Copilot license (for me).

Idealised solution: 

• To be able to quiz the back catalogue of books with prompts such as: "In <book title> who are the main characters?", "Describe <character> from <book>", "Summarise chapter <number> from <book title>.", "Tell me about <character / aspect> from <book title>.", "Find examples of conversations between <character> and <character> from <book title>.", "Find scenes with <character> in from <book title>.", "Find the scene where <character> does <thing> in <book title>." etc etc. 
• Internal use only is required.
• Multiple users would require access.
• No setup for end users i.e. setting of context or attaching documents etc - just type and go through an easy interface.

Current progress: I have tried the following:

• Copilot in Word with the relevant document open. So far this provides mixed results however it doesn't meet the use case as I have to open the document and quiz it. I want to be able to type direct prompts indicating the required book and character / secene etc. 
• Copilot app in 365 with document attached. This also provides mixed results but again requires that I attach the required document each time.
• Copilots created in Copilot studio with document uploaded, conversation boost on and low content moderation. Testing this Copilot provides the least answers of any above solutions. Usually it is just "I’m sorry, I’m not sure how to help with that. Can you try rephrasing?".

A few thoughts from me: A bespoke Copilot created in Copilot studio seems to me the most likely candidate for the ideal solution for the following reasons:

• I can specifically provide the required document(s) to the Copilot without having to keep attaching or setting context.
• The created Copilot can then be delivered in various means to other users in the environment by the Power Platform options or through Teams etc (I think).
• It could be that individual books or book series need to be ring fenced as a context so crossovers in story lines or character names can be avoided - this could be met by creating Copilots for specific books or series.
• I believe I could also create topics that are relevant to the content of specific books or series to further improve the quality of the interactions and generative answers.
• These aspects meet my required ideal scenario of being able to deliver generative answers regarding the back catalogue of books to users without any setup required from them.

Things that maybe relevant to an ideal solution:

• The content of these books is of an extremely erotic nature and would very likely trigger any background content moderation on such material implemented in Copilot or 365. We need this not to be so. I bring this up because I have noticed Copilot sometimes start to formulate an answer to the screen and them "change it's mind" and decide not to respond. It also alludes to erotic material but never seems keen to quote such material from the books. I wonder if this is behind the lack of relevant answers.
• We have no tenant license for Copilot hence I need to understand whether I can deliver solutions created in Copilot studio to other users through Teams or Power platform without this license.
• Currently if I publish a Copilot created in studio to a demo website the created chatbot reports an IntegratedAuthenticationNotSupportedInChannel exception. I'm guessing this is because I have Teams only auth setup (hence Team channel only) or no tenant license. 

What I need help with:

• Whether bespoke Copilots created in Copilot Studio is the likely the way to go with this or whether there is another more obvious solution I am missing.
• Why a Copilot created in Copilot studio with the manuscript uploaded seems to provide no generative answers about the content (as opposed to Copilot in Word and Copilot app in 365 giving answers).
• Whether content moderation of erotic material is likely to be getting in the way of creating a working solution with this and if so how to turn this off and be sure that it is off.
• Whether I need a tenant license or other users licenses for Copilot in order to deploy Copilots created in Copilot Studio to other users.
• What options I have in terms of effectively delivering these Copilots to other users i.e. Teams (how does this look?), Power Apps / Pages etc. Internal use only is required. 

I was hoping this would be a relatively simple use case for generative AI answers in that we have all the required data at our finger tips and it seemed like it should be a simple case of ring fencing the context accordingly and away we go. 

I realise there is a lot here and this seems to me like a situation whereby a Microsoft employee with some good domain knowledge would walk me through this in a couple or hours given all my hard earned dollars i'm handing over for these tools but I'm guessing that is a fantasy, right?

Thanks for reading.

Timmy 🙂

[Umianta · Accepted solution]

Hi @TimmyJadeTech 

Please find below answers to some of your questions -

• Whether bespoke Copilots created in Copilot Studio is the likely the way to go with this or whether there is another more obvious solution I am missing.
  • I believe your requirements can be fulfilled using Copliot created in Copliot Studio.
• Why a Copilot created in Copilot studio with the manuscript uploaded seems to provide no generative answers about the content (as opposed to Copilot in Word and Copilot app in 365 giving answers).
  
  • Refer to below documentation for supported files for Generative answers 
    • https://learn.microsoft.com/en-us/microsoft-copilot-studio/nlu-documents
    • https://learn.microsoft.com/en-us/microsoft-copilot-studio/generative-answers-sharepoint-no-response
• Whether content moderation of erotic material is likely to be getting in the way of creating a working solution with this and if so how to turn this off and be sure that it is off.
  • Generative Answers follows the MS responsible AI principle, it will block the harmful content 
  • 
• Whether I need a tenant license or other users licenses for Copilot in order to deploy Copilots created in Copilot Studio to other users.
  • For users, it is charged by messages see below 
  • 
• What options I have in terms of effectively delivering these Copilots to other users i.e. Teams (how does this look?), Power Apps / Pages etc. Internal use only is required. 
  • MS Teams would good option for internal users where Copilot Studio would internally handle authentication.

If this was helpful:
⭐Please mark it as a solution
⭐Give me a thumbs-up
⭐Connect on LinkedIn
⭐I consistently upload technical videos. If you're interested, please visit and subscribe to my YouTube Channel
⭐Always glad to help, Umesh Khandelwal!

[adilei]

Copilot Studio and other platforms that were mentioned are not useful when trying to respond to questions such as: "Find examples of conversations between character A and character B".

This is because Copilot Studio (as well as the M365 Copilot) utilizes a pattern which is called RAG (Retrieval Augmented Generation). RAG leverages a two-stage process: it will first search for your query (or an approximation of your query) in the data source, and then feed the results to an LLM so an answer or a summary can be generated.

A question like "find examples of conversations" would work only if it is explicitly stated in the data source that a specific segment of text is a conversation between two characters, but most books or texts are not so explicit about that. So typically, because the underlying search mechanism does not do interpretative work, no results will come back.

To respond to questions such as the ones you mentioned, additional work has to be done before hand, like tagging or classifying the texts, or fine tuning the LLM. Those methods are more advanced and require additional services (e.g., Azure OpenAI), but once they are implemented, the results can be used in conjunction with Copilot Studio.

[TimmyJadeTech]

Great - I understand. That is very useful to know. Thanks.

[TimmyJadeTech]

Thankyou for your answers. I will look further into some of your points.