Generative Artificial Intelligence in law

from evaluation methodology to the legal correctness of LLMS and AI agents

Abstract

This work analyzes the use of Generative Artificial Intelligence, specifically Large Language Models (LLMs), to support legal activities by systematically testing fifteen of the main AI models available on the market (ChatGPT o1, ChatGPT 4o, ChatGPT 4.5, ChatGPT o3-mini, ChatGPT o3-mini-high, Claude Sonnet 3.5, Claude Sonnet 3.7, Grok 3, Gemini Pro 2.0 Experimental, Gemini Pro 2.5 Experimental, DeepSeek R1, Qwen 2.5 Max, Perplexity, Sabiá-3 e Copilot). Grounded in the double correction channel theory, the research adopted a multidisciplinary approach for the LLM analyses, six typically legal tests were developed to evaluate the AIs’ performance in: (a) knowledge of Brazilian legislation; (b) legal application; (c) application of legal principles; (d) solving hard cases; (e) identification of the ratio decidendi in the judicial precedents; and (f) organization of case data/information. This study follows a qualitative-quantitative research design with a descriptive-exploratory method, by analyzing results from intersection between Law and Artificial Intelligence. The outcomes are: (a) the establishment of a basic framework for legally testing LLMs; (b) which generative AI models currently best fit each of the tested tasks; (c) the legal tasks for which LLM use is not recommended (or for which greater caution is advised); and (d) the tasks in which LLMs perform best in the legal field. The conclusions are: (a) it is not advisable to use generative AI (LLMs) to access systematized legal knowledge (such as normative texts); (b) it is not a good practice to use generative AI to interpret legal norms freely, without additional controls; (c) most LLMs struggle to infer all correct normative propositions from statutory texts; (d) the use of Retrieval-Augmented Generation (RAG) techniques or auxiliary AIs based on knowledge representation is necessary; (e) the best LLMs showed creativity and a claim to legal correctness when applying legal principles; (f) the best LLMs can be used to assist in generating legally correct argumentation; (g) generative AI models are not reliable repositories of pre-existing knowledge but are capable of creating new knowledge (AI’s knowledge/creation paradox); (h) the top generative AI models can be used in the argumentative exploration of judicial precedents (identifying the ratio decidendi); (i) LLMs can retrieve data and information from legal documents; (j) appropriate prompt engineering does not always prevent hallucinations; (k) there is no single generative AI model that is superior or perfect for all legal problems; and (l) the latest-generation LLM is not always the most efficient for a given legal task.