Prompt Engineering and Provision of Context in Domain Specific Use of GPT

Marton Ribary; Paul Krause; Miklos Orban; Eugenio Vaccari; Thomas Wood

doi:10.3233/FAIA230979

Prompt Engineering and Provision of Context in Domain Specific Use of GPT

Marton Ribary, Paul Krause, Miklos Orban, Eugenio Vaccari, Thomas Wood

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract

Large Language Models (LLMs) can appear to generate expert advice on legal matters. However, at closer analysis, some of the advice provided has proven unsound or erroneous. We tested LLMs’ performance in the procedural and technical area of insolvency law in which our team has relevant expertise. This paper demonstrates that statistically more accurate results to evaluation questions come from a design which adds a curated knowledge base to produce quality responses when querying LLMs. We evaluated our bot head-to-head on an unseen test set of twelve questions about insolvency law against the unmodified versions of gpt-3.5-turbo and gpt-4 with a mark scheme similar to those used in examinations in law schools. On the “unseen test set”, the Insolvency Bot based on gpt-3.5-turbo outper-formed gpt-3.5-turbo (p = 1.8%), and our gpt-4 based bot outperformed unmodified gpt-4 (p = 0.05%). These promising results can be expanded to cross-jurisdictional queries and be further improved by matching on-point legal information to user queries. Overall, they demonstrate the importance of incorporating trusted knowledge sources into traditional LLMs in answering domain-specific queries.

Original language	English
Title of host publication	Legal Knowledge and Information Systems
Subtitle of host publication	JURIX 2023: The Thirty-sixth Annual Conference, Maastricht, the Netherlands, 18–20 December 2023
Editors	Giovanni Sileno, Jerry Spanakis, Gijs van Dijck
Place of Publication	Amsterdam
Publisher	IOS Press
Pages	305-310
Number of pages	6
ISBN (Electronic)	978-1-64368-473-4
ISBN (Print)	978-1-64368-472-7
DOIs	https://doi.org/10.3233/FAIA230979
Publication status	Published - 18 Dec 2023
Event	36th International Conference on Legal Knowledge and Information Systems - Maastricht University, Maastricht, Netherlands Duration: 18 Dec 2023 → 20 Dec 2023 https://jurix23.maastrichtlawtech.eu/

Publication series

Name	Frontiers in Artificial Intelligence and Applications
Publisher	IOS Press
Number	379

Conference

Conference	36th International Conference on Legal Knowledge and Information Systems
Abbreviated title	JURIX 2023
Country/Territory	Netherlands
City	Maastricht
Period	18/12/23 → 20/12/23
Internet address	https://jurix23.maastrichtlawtech.eu/

Keywords

legal tech
large language models (LLM)
natural language processing
prompt engineering
insolvency law
Chatbot

Access to Document

10.3233/FAIA230979Licence: CC BY-NC

2 Invited talk
2 Oral presentation

A generative AI-based legal advice tool for small businesses in distress
Ribary, M. (Speaker) & Wood, T. (Speaker)
5 Dec 2024
Activity: Talk or presentation › Invited talk
The inner life of a legal chatbot: A TED-style talk
Ribary, M. (Speaker)
13 Jun 2024
Activity: Talk or presentation › Invited talk
A GPT-based legal advice tool for small businesses in distress
Ribary, M. (Speaker), Vaccari, E. (Speaker), Krause, P. (Speaker), Orban, M. (Speaker) & Wood, T. (Speaker)
25 Apr 2024
Activity: Talk or presentation › Oral presentation

Cite this

Ribary, M., Krause, P., Orban, M., Vaccari, E., & Wood, T. (2023). Prompt Engineering and Provision of Context in Domain Specific Use of GPT. In G. Sileno, J. Spanakis, & G. van Dijck (Eds.), Legal Knowledge and Information Systems: JURIX 2023: The Thirty-sixth Annual Conference, Maastricht, the Netherlands, 18–20 December 2023 (pp. 305-310). (Frontiers in Artificial Intelligence and Applications; No. 379). IOS Press. https://doi.org/10.3233/FAIA230979

Ribary, Marton ; Krause, Paul ; Orban, Miklos et al. / Prompt Engineering and Provision of Context in Domain Specific Use of GPT. Legal Knowledge and Information Systems: JURIX 2023: The Thirty-sixth Annual Conference, Maastricht, the Netherlands, 18–20 December 2023. editor / Giovanni Sileno ; Jerry Spanakis ; Gijs van Dijck. Amsterdam : IOS Press, 2023. pp. 305-310 (Frontiers in Artificial Intelligence and Applications; 379).

@inproceedings{b08a690247b24a69851c112555724782,

title = "Prompt Engineering and Provision of Context in Domain Specific Use of GPT",

abstract = "Large Language Models (LLMs) can appear to generate expert advice on legal matters. However, at closer analysis, some of the advice provided has proven unsound or erroneous. We tested LLMs{\textquoteright} performance in the procedural and technical area of insolvency law in which our team has relevant expertise. This paper demonstrates that statistically more accurate results to evaluation questions come from a design which adds a curated knowledge base to produce quality responses when querying LLMs. We evaluated our bot head-to-head on an unseen test set of twelve questions about insolvency law against the unmodified versions of gpt-3.5-turbo and gpt-4 with a mark scheme similar to those used in examinations in law schools. On the “unseen test set”, the Insolvency Bot based on gpt-3.5-turbo outper-formed gpt-3.5-turbo (p = 1.8%), and our gpt-4 based bot outperformed unmodified gpt-4 (p = 0.05%). These promising results can be expanded to cross-jurisdictional queries and be further improved by matching on-point legal information to user queries. Overall, they demonstrate the importance of incorporating trusted knowledge sources into traditional LLMs in answering domain-specific queries.",

keywords = "legal tech, large language models (LLM), natural language processing, prompt engineering, insolvency law, Chatbot",

author = "Marton Ribary and Paul Krause and Miklos Orban and Eugenio Vaccari and Thomas Wood",

note = "Marton Ribary is a Lecturer in Law at Royal Holloway, University of London (RHUL) where his research focuses on legal reasoning. He works with Natural Language Processing (NLP) and algorithmic rule modelling methods applied to historial (Roman and Rabbinic) as well as modern (English) texts in private law. Paul Krause is Emeritus Professor at the University of Surrey. He has been working on theoretical and practical aspects of AI for forty years. His career has included time at the National Physical Laboratory, the Imperial Cancer Research Fund, and Philips Research Laboratories. Mikl{\'o}s Orb{\'a}n is a Visiting Professor in practice at the University of Surrey. He graduated from ELTE Law School and Georgetown University where he was a Fulbright scholar. He is a legal entrepreneur with 20+ years of experience as a New York and Hungarian qualified lawyer. He is the CTO of gunnercooke LLP and the managing partner of gunnercooke CEE. He is the co-founder of OPL, EXPLICO and Lexitup. Eugenio Vaccari is a Senior Lecturer in Law at Royal Holloway, University of London. He is a qualified Italian lawyer specialized in corporate insolvency and bankruptcy law. Eugenio is a member of the Executive Committee of III NextGen Class XI, a co-chair of the Emerging Scholars Group of Insolvency Law Academy and an active member of INSOL International, INSOL Europe and the Insolvency Lawyers{\textquoteright} Association (UK). He is an Academic Fellow of the Middle Temple and a co-author of English Corporate Insolvency Law: A Primer, which was an invaluable resource for the development of this project. Thomas Wood is the founder and director of Fast Data Science. He holds an MPhil in Computer Speech, Text and Internet Technology from the University of Cambridge. He has worked in a psychology lab in London, two small companies in Germany and Spain, and in Tesco{\textquoteright}s data science team. Thomas has built machine learning models to extract information from unstructured documents such as interview transcripts, scientific papers, and other scientific and healthcare data.; 36th International Conference on Legal Knowledge and Information Systems, JURIX 2023 ; Conference date: 18-12-2023 Through 20-12-2023",

year = "2023",

month = dec,

day = "18",

doi = "10.3233/FAIA230979",

language = "English",

isbn = "978-1-64368-472-7",

series = "Frontiers in Artificial Intelligence and Applications",

publisher = "IOS Press",

number = "379",

pages = "305--310",

editor = "Giovanni Sileno and Jerry Spanakis and {van Dijck}, Gijs",

booktitle = "Legal Knowledge and Information Systems",

address = "Netherlands",

url = "https://jurix23.maastrichtlawtech.eu/",

}

Ribary, M, Krause, P, Orban, M, Vaccari, E & Wood, T 2023, Prompt Engineering and Provision of Context in Domain Specific Use of GPT. in G Sileno, J Spanakis & G van Dijck (eds), Legal Knowledge and Information Systems: JURIX 2023: The Thirty-sixth Annual Conference, Maastricht, the Netherlands, 18–20 December 2023. Frontiers in Artificial Intelligence and Applications, no. 379, IOS Press, Amsterdam, pp. 305-310, 36th International Conference on Legal Knowledge and Information Systems, Maastricht, Netherlands, 18/12/23. https://doi.org/10.3233/FAIA230979

Prompt Engineering and Provision of Context in Domain Specific Use of GPT. / Ribary, Marton; Krause, Paul; Orban, Miklos et al.
Legal Knowledge and Information Systems: JURIX 2023: The Thirty-sixth Annual Conference, Maastricht, the Netherlands, 18–20 December 2023. ed. / Giovanni Sileno; Jerry Spanakis; Gijs van Dijck. Amsterdam: IOS Press, 2023. p. 305-310 (Frontiers in Artificial Intelligence and Applications; No. 379).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

TY - GEN

T1 - Prompt Engineering and Provision of Context in Domain Specific Use of GPT

AU - Ribary, Marton

AU - Krause, Paul

AU - Orban, Miklos

AU - Vaccari, Eugenio

AU - Wood, Thomas

N1 - Marton Ribary is a Lecturer in Law at Royal Holloway, University of London (RHUL) where his research focuses on legal reasoning. He works with Natural Language Processing (NLP) and algorithmic rule modelling methods applied to historial (Roman and Rabbinic) as well as modern (English) texts in private law. Paul Krause is Emeritus Professor at the University of Surrey. He has been working on theoretical and practical aspects of AI for forty years. His career has included time at the National Physical Laboratory, the Imperial Cancer Research Fund, and Philips Research Laboratories. Miklós Orbán is a Visiting Professor in practice at the University of Surrey. He graduated from ELTE Law School and Georgetown University where he was a Fulbright scholar. He is a legal entrepreneur with 20+ years of experience as a New York and Hungarian qualified lawyer. He is the CTO of gunnercooke LLP and the managing partner of gunnercooke CEE. He is the co-founder of OPL, EXPLICO and Lexitup. Eugenio Vaccari is a Senior Lecturer in Law at Royal Holloway, University of London. He is a qualified Italian lawyer specialized in corporate insolvency and bankruptcy law. Eugenio is a member of the Executive Committee of III NextGen Class XI, a co-chair of the Emerging Scholars Group of Insolvency Law Academy and an active member of INSOL International, INSOL Europe and the Insolvency Lawyers’ Association (UK). He is an Academic Fellow of the Middle Temple and a co-author of English Corporate Insolvency Law: A Primer, which was an invaluable resource for the development of this project. Thomas Wood is the founder and director of Fast Data Science. He holds an MPhil in Computer Speech, Text and Internet Technology from the University of Cambridge. He has worked in a psychology lab in London, two small companies in Germany and Spain, and in Tesco’s data science team. Thomas has built machine learning models to extract information from unstructured documents such as interview transcripts, scientific papers, and other scientific and healthcare data.

PY - 2023/12/18

Y1 - 2023/12/18

N2 - Large Language Models (LLMs) can appear to generate expert advice on legal matters. However, at closer analysis, some of the advice provided has proven unsound or erroneous. We tested LLMs’ performance in the procedural and technical area of insolvency law in which our team has relevant expertise. This paper demonstrates that statistically more accurate results to evaluation questions come from a design which adds a curated knowledge base to produce quality responses when querying LLMs. We evaluated our bot head-to-head on an unseen test set of twelve questions about insolvency law against the unmodified versions of gpt-3.5-turbo and gpt-4 with a mark scheme similar to those used in examinations in law schools. On the “unseen test set”, the Insolvency Bot based on gpt-3.5-turbo outper-formed gpt-3.5-turbo (p = 1.8%), and our gpt-4 based bot outperformed unmodified gpt-4 (p = 0.05%). These promising results can be expanded to cross-jurisdictional queries and be further improved by matching on-point legal information to user queries. Overall, they demonstrate the importance of incorporating trusted knowledge sources into traditional LLMs in answering domain-specific queries.

AB - Large Language Models (LLMs) can appear to generate expert advice on legal matters. However, at closer analysis, some of the advice provided has proven unsound or erroneous. We tested LLMs’ performance in the procedural and technical area of insolvency law in which our team has relevant expertise. This paper demonstrates that statistically more accurate results to evaluation questions come from a design which adds a curated knowledge base to produce quality responses when querying LLMs. We evaluated our bot head-to-head on an unseen test set of twelve questions about insolvency law against the unmodified versions of gpt-3.5-turbo and gpt-4 with a mark scheme similar to those used in examinations in law schools. On the “unseen test set”, the Insolvency Bot based on gpt-3.5-turbo outper-formed gpt-3.5-turbo (p = 1.8%), and our gpt-4 based bot outperformed unmodified gpt-4 (p = 0.05%). These promising results can be expanded to cross-jurisdictional queries and be further improved by matching on-point legal information to user queries. Overall, they demonstrate the importance of incorporating trusted knowledge sources into traditional LLMs in answering domain-specific queries.

KW - legal tech

KW - large language models (LLM)

KW - natural language processing

KW - prompt engineering

KW - insolvency law

KW - Chatbot

U2 - 10.3233/FAIA230979

DO - 10.3233/FAIA230979

M3 - Conference contribution

SN - 978-1-64368-472-7

T3 - Frontiers in Artificial Intelligence and Applications

SP - 305

EP - 310

BT - Legal Knowledge and Information Systems

A2 - Sileno, Giovanni

A2 - Spanakis, Jerry

A2 - van Dijck, Gijs

PB - IOS Press

CY - Amsterdam

T2 - 36th International Conference on Legal Knowledge and Information Systems

Y2 - 18 December 2023 through 20 December 2023

ER -

Ribary M, Krause P, Orban M, Vaccari E, Wood T. Prompt Engineering and Provision of Context in Domain Specific Use of GPT. In Sileno G, Spanakis J, van Dijck G, editors, Legal Knowledge and Information Systems: JURIX 2023: The Thirty-sixth Annual Conference, Maastricht, the Netherlands, 18–20 December 2023. Amsterdam: IOS Press. 2023. p. 305-310. (Frontiers in Artificial Intelligence and Applications; 379). doi: 10.3233/FAIA230979

Prompt Engineering and Provision of Context in Domain Specific Use of GPT

Abstract

Publication series

Conference

Keywords

Access to Document

Activities

A generative AI-based legal advice tool for small businesses in distress

The inner life of a legal chatbot: A TED-style talk

A GPT-based legal advice tool for small businesses in distress

Cite this