Abstract
Since the launch of ChatGPT in November 2022, conversational Large Language Models (LLMs) have generated significant interest in legal circles for the purpose of generating automated expert advice on legal matters. ChatGPT and the gpt-4 model was reported to have passed the multistate part of the US bar exam. Real life cases, however, are generally more complicated than the artificial ones included in standardised tests and often demand a nuanced understanding of the client's specific circumstances. As explained later in this paper, legal advice generated by LLMs may prove unsound, erroneous, or even absurd. According to a recent study, gpt-4 and other similar public models “hallucinate at least 58% of the time, struggle to predict their own hallucinations, and often uncritically accept users’ incorrect legal assumptions”. LLMs can appear to demonstrate expert knowledge which is simply a side-effect of models trained to provide examples of the usage of natural language. From the outset, they have not been developed as knowledge elicitation tools.
In this chapter, we present the “Insolvency Bot”, a hybrid AI tool developed for providing legal advice to stakeholders involved in insolvency issues for micro and small enterprises (MSEs) with a level of competency comparable to an experienced paralegal. Our paper is structured as follows. We first discuss the problem presented by small businesses in financial distress in the UK and the technical characteristics and limitations of LLMs. We then present our solution, a hybrid tool querying LLMs with prompt engineering and using Retrieval-Augmented Generation (RAG) to match user queries with on-point legal information from our manually curated knowledge base containing legislation, legal cases and HMRC forms. According to a tailor-made mark scheme, we show that our Insolvency Bot consistently outperforms LLMs queried without our hybrid setup, and we show that newer versions of LLMs consistently outperform older ones when queried with Insolvency Bot. We present the promising results of an unmoderated user experience test before we discuss further avenues of improving and expanding the system, and its potential to ensure “access to justice” to affected businessmen.
In this chapter, we present the “Insolvency Bot”, a hybrid AI tool developed for providing legal advice to stakeholders involved in insolvency issues for micro and small enterprises (MSEs) with a level of competency comparable to an experienced paralegal. Our paper is structured as follows. We first discuss the problem presented by small businesses in financial distress in the UK and the technical characteristics and limitations of LLMs. We then present our solution, a hybrid tool querying LLMs with prompt engineering and using Retrieval-Augmented Generation (RAG) to match user queries with on-point legal information from our manually curated knowledge base containing legislation, legal cases and HMRC forms. According to a tailor-made mark scheme, we show that our Insolvency Bot consistently outperforms LLMs queried without our hybrid setup, and we show that newer versions of LLMs consistently outperform older ones when queried with Insolvency Bot. We present the promising results of an unmoderated user experience test before we discuss further avenues of improving and expanding the system, and its potential to ensure “access to justice” to affected businessmen.
| Original language | English |
|---|---|
| Title of host publication | Research Handbook on Sustainable Insolvency Systems |
| Publisher | Edward Elgar Publishing Ltd. |
| Chapter | 29 |
| Publication status | Submitted - 28 Feb 2025 |
Keywords
- insolvency
- MSMEs
- AI-generated technology
- LLMs
- RAG
