A generative AI-based legal advice tool for small businesses in distress

Description

We developed and tested the performance of a retrieval augmented generation (RAG) system for answering legal queries related to corporate insolvency in England and Wales. The Insolvency Bot uses pattern matching and zero-shot classification with a vector index of sentence embeddings to identify relevant legal passages and HMRC forms, and it constructs a prompt concatenating matched legal information with a user’s query. We evaluated our bot head-to-head on an unseen test set against the unmodified versions of large language models (LLMs) gpt-3.5-turbo, gpt-4, or gpt-4o with a mark scheme similar to those used in examinations in law schools. The Insolvency Bot outperformed each unmodified LLM (p = 0.05%). An additional user experience survey suggested the need for creating two versions of the bot, one for lay people who expect practical and actionable advice and another for professionals with the relevant legal authorities. Our legal chatbot demonstrates the benefits of combining a generative AI system with a trusted knowledge base and shows future promise to cover cross-jurisdictional and insolvency-related queries and could be further improved in its technical architecture.

Period	5 Dec 2024
Event title	NLP Cafe: Research seminar series of the Centre for Translation Studies, University of Surrey
Event type	Seminar
Location	Guildford, United KingdomShow on map
Degree of Recognition	Regional