A GPT-based legal advice tool for small businesses in distress

Description

Conversational Large Language Models (LLMs), such as ChatGPT, have generated significant interest in various domains for performing tasks that range from giving medical assessments through generating computer code to providing expert advice on legal matters. However, at closer analysis, some of the advice provided has proven to be unsound or erroneous. By building a prototype system, we tested LLM performance in the procedural and technical area of English insolvency law in which our team has relevant expertise.

While LLMs can seem to replicate the response of an expert in language that is confident and compelling, it is important to keep in mind that the purpose of the text corpora used to train the models is to provide examples of natural language use. Because these items of text inevitably concern specific knowledge domains, the responses of LLMs can appear to demonstrate expert knowledge. However, this is a side-effect of the generation of the LLMs; they have not been developed as knowledge elicitation tools. The proposed paper explores and tests methods by which an LLM can be enhanced to provide a trusted knowledge source with a certain level of professional expertise. Specifically, our goal is to support the triage of potential legal cases for stakeholders involved in insolvency issues for micro, small and medium enterprises (MSMEs) with a level of competency comparable to a Level 6 or 7 English law student. This is a specific area of law where many solo practitioners and smaller law firms lack sufficient legal expertise, so our system could - if successful enough - provide a helping hand to such practitioners in expanding the scope of their services.

Specifically, in this paper we evaluate the hypothesis that query responses from an LLM will be improved if the model is enhanced with a trusted domain-specific knowledge base. We demonstrate that statistically more accurate results to evaluation questions come from a design which adds a curated knowledge base to produce quality responses when querying LLMs. We evaluated our prototype system head-to-head on an unseen test set of twelve questions about insolvency law against the unmodified versions of gpt-3.5-turbo and gpt-4 with a mark scheme similar to those used in examinations in law schools. On the “unseen test set”, our so-called Insolvency Bot based on gpt-3.5-turbo outperformed gpt-3.5-turbo (p = 1.8%), and our gpt-4 based bot outperformed unmodified gpt-4 (p = 0.05%). These promising results can be expanded to cross-jurisdictional queries and be further improved by matching on-point legal information to user queries. Overall, they demonstrate the importance of incorporating trusted knowledge sources into traditional LLMs in answering domain-specific queries.

Period	25 Apr 2024
Event title	The 5th International and Comparative Law Insolvency Symposium
Event type	Conference
Location	Egham , United KingdomShow on map
Degree of Recognition	International