Prompt Engineering and Provision of Context in Domain Specific Use of GPT

Marton Ribary, Paul Krause, Miklos Orban, Eugenio Vaccari, Thomas Wood

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Large Language Models (LLMs) can appear to generate expert advice on legal matters. However, at closer analysis, some of the advice provided has proven unsound or erroneous. We tested LLMs’ performance in the procedural and technical area of insolvency law in which our team has relevant expertise. This paper demonstrates that statistically more accurate results to evaluation questions come from a design which adds a curated knowledge base to produce quality responses when querying LLMs. We evaluated our bot head-to-head on an unseen test set of twelve questions about insolvency law against the unmodified versions of gpt-3.5-turbo and gpt-4 with a mark scheme similar to those used in examinations in law schools. On the “unseen test set”, the Insolvency Bot based on gpt-3.5-turbo outper-formed gpt-3.5-turbo (p = 1.8%), and our gpt-4 based bot outperformed unmodified gpt-4 (p = 0.05%). These promising results can be expanded to cross-jurisdictional queries and be further improved by matching on-point legal information to user queries. Overall, they demonstrate the importance of incorporating trusted knowledge sources into traditional LLMs in answering domain-specific queries.
Original languageEnglish
Title of host publicationLegal Knowledge and Information Systems
Subtitle of host publicationJURIX 2023: The Thirty-sixth Annual Conference, Maastricht, the Netherlands, 18–20 December 2023
EditorsGiovanni Sileno, Jerry Spanakis, Gijs van Dijck
Place of PublicationAmsterdam
PublisherIOS Press
Pages305-310
Number of pages6
ISBN (Electronic)978-1-64368-473-4
ISBN (Print)978-1-64368-472-7
DOIs
Publication statusPublished - 18 Dec 2023
Event36th International Conference on Legal Knowledge and Information Systems - Maastricht University, Maastricht, Netherlands
Duration: 18 Dec 202320 Dec 2023
https://jurix23.maastrichtlawtech.eu/

Publication series

NameFrontiers in Artificial Intelligence and Applications
PublisherIOS Press
Number379

Conference

Conference36th International Conference on Legal Knowledge and Information Systems
Abbreviated titleJURIX 2023
Country/TerritoryNetherlands
CityMaastricht
Period18/12/2320/12/23
Internet address

Keywords

  • legal tech
  • large language models (LLM)
  • natural language processing
  • prompt engineering
  • insolvency law
  • Chatbot

Cite this