Feat/user summary prompt

[astkwcz]

User defined system prompt for summary task

Related work:

This PR is stacked on #437 :

• Feat/custom system prompts #437 - configurable system prompts for summary and translation (admin defined)
• this PR - additional configurable system prompt for summary (user defined), it was removed from Feat/custom system prompts #437

Feel free to check #437, it describes the whole work for system prompts for different task types. It also provide a description of User defined system prompt for summary and presents UI look. Hierarchy is as follows:

• user defined system prompt (Complexity and Format options are ignored)
• admin defined system prompt (Complexity and Format options are appended)
• default system prompt

@edward-ly asked to move the user-defined system prompt part out of #437 and suggested to add user-defined system prompt part in separate PR with justification.

Justification for user defined system prompts

I agree with @edward-ly that it adds complexity for the user. The user has to know a little about how system prompts work for LLM and how to write instructions. On the other hand, this functionality is available after clicking Show advanced options so we can split users into two group:

• base options - for everyday users, nothing to configure (by the user) here
• advanced options - for a little bit more advanced users, who are able to write their own instructions for LLM

Use cases and why admin prompt is not enough

Admin defined system prompt for summary is a good option but it has to be general, because it has to work for everyone.

User-defined system prompt gives a user more control. This control might be required for different type of text to summarize. For example:

• meeting notes - someone gives a transcript as an input and asks to summarize a meeting and extract decisions, action items etc
• code - user might want to summarize some log file and call out errors/warnings
• contracts - user might tell LLM to summarize it and extract parties, duties and deadlines
• academic papers - user defined system prompt might include instruction to preserve citations, authors and key-findings

So those are field-specific summary tasks types. In general it is a summary, but based on the domain, the user might need different kind of information to be extracted.

What is more, with user defined system prompt for summary a user can set output format themselves for their needs:

• tabular data
• markdown format with ## headings
• company specific structure of a file
  • specific section names
  • specific language
  • some sections bullet points only
  • different sections with paragraph format
  • section that only includes people and their responsibilities
  • section with deadlines

Output formats can be mixed in summary if user defines that.

It all can be achieved with setting a good instruction for a model, but it is field/user specific instruction. Well written admin system prompt can be a good fit for most cases, but for specific kind of field/work user-defined system prompt might give better results if written correctly.

I would really appreciate your thoughts on these use-cases. If you prefer a different approach I am happy to rework this.

[edward-ly]

Hello, the way I see it, all of these use cases have existing workarounds or can be implemented in simpler ways.

advanced options - for a little bit more advanced users, who are able to write their own instructions for LLM

Why not just use the Assistant chat or the Generate text task instead, providing the "system prompt" and the user prompt in the same field? Separating them wouldn't provide much extra benefit and only confuses all users even more. Plus, overriding the system prompt means users could theoretically provide non-summary prompts in a summary task, which is clearly not the intention.

So those are field-specific summary tasks types. In general it is a summary, but based on the domain, the user might need different kind of information to be extracted.

An LLM might be able to extract these important parts already without the need for additional prompts. But even if it doesn't, you can still use the same workaround as before.

a user can set output format themselves for their needs

I would recommend adding these additional options to the Format input parameter instead. If users require any custom formatting, I would again point them to the same workaround as before.