The Second Draft - Volume 39, No. 2
Assessing Research Skills in the Age of Generative AI: Asking Questions that Machines Cannot Answer DOWNLOAD PDF
October 5, 2026Large Language Models (LLMs) in Artificial Intelligence (AI) have made it easier than ever for students to outsource legal research and writing, circumventing the learning process and posing an existential threat to the concept of academic integrity and honesty. Students need only produce a prompt of no more than a few sentences describing their assignment to receive a polished, high-quality piece of work within minutes. In legal education, the challenge extends beyond ChatGPT as law students now have access to AI tools built directly into Westlaw and Lexis that are trained on legal materials. As the use of these AI tools becomes more common, legal educators face a new challenge: How can we teach students and assess their legal research skills despite AI being able to perform this work for them?
To be sure, law students must receive exposure and training with artificial intelligence tools to prepare for practicing law. However, traditional legal research principles remain highly relevant. At a minimum, understanding what a real citation looks like can help lawyers steer clear of using the fabricated citations that have appeared in court filings across the country. Understanding AI’s limitations and the errors it can produce and knowing how to verify AI’s output are essential components of modern legal research instruction. Yet AI creates a significant pedagogical challenge: if students are offered AI tools to locate authorities, analyze legal issues, and generate written responses, how can law professors accurately assess whether students themselves possess the underlying legal research skills? When it comes to teaching legal research, this assessment challenge can be addressed by asking simpler questions about specific intermediate steps, requiring students to navigate complex research paths to arrive at the ultimate answer.
Analysis
Legal research assignments can take many different forms. By posing short questions that require students to demonstrate how they navigated to a particular website or resource, professors disable students from using AI to craft and edit a narrative response. Historically, students have demonstrated legal research proficiency as the ‘solution’ to a legal research question in the form of a memo or some other short written work product. Today, however, Lexis and Westlaw AI tools allow students to ask AI to identify relevant authorities and draft an analysis, creating the risk that students may submit a successful assignment without having directly located and accessed the appropriate underlying resources.
However, Lexis and Westlaw AI often cannot provide the highly specific details that students can only uncover by conducting the research themselves. It is precisely these granular particulars that make the following research problems resistant to AI-generated answers. AI struggles (and often falls short) when the assignment requires a precise answer that can only be obtained by interacting specifically with the research platform itself. Thus, although the questions below are relatively easy for a human researcher to answer, these questions often produce incorrect responses from Lexis, Westlaw, or ChatGPT. Some of the questions below require students to locate precise numerical information, while others require students to navigate a specific resource to retrieve information that AI cannot reliably access.
Question example 1: Requiring a precise, verifiable answer
I posed the following question to Westlaw’s AI Deep Research and Lexis+ Protégé’s Ask feature:
How many reported state cases in Massachusetts cite to M.G.L. c. 140 Section 155?
Before discussing the AI responses, it is helpful to identify the correct answer and the research process required to obtain it. To successfully answer this question, students would need to locate the statute, click on Citing References (Westlaw) or Citing Decisions (Lexis), limit the results to cases in the Massachusetts jurisdiction, then filter to exclude unpublished opinions. The correct answer is simply a number. As of the date of this writing, Westlaw reports 51 cases and Lexis reports 62 cases (see Figures 1 and 2).
When presented with this question, Westlaw AI Deep Research declined to answer, instead reporting simply with the following: “Your question is outside the scope of this feature.” The Lexis+ Protégé Ask feature provided a more detailed but still inaccurate response. Instead of stating that this kind of question was not a realistic example of a legal research prompt, it identified six cases that it claimed discussed the statute’s application and implications in various contexts. (see Fig. 3)
Similarly, ChatGPT provided an approximation of six cases with an explanation of why a number doesn’t exist (see Fig. 4.) At the end of its answer, it suggested the following:
If you want, I can pull a comprehensive list of all Massachusetts appellate decisions citing § 155 (with counts by court)—that’s usually what you’d want for briefing or teaching purposes.
I responded with “Yes, do that” and ChatGPT proceeded to provide a list of eight cases and a numerical estimate of between 8 and 12 cases. (See Fig. 5)
Through this question, one can see that AI has difficulty reporting features of legal resources where further filtering may be required. Instead of calculating a number, Lexis and ChatGPT each gave an approximate response. By asking students for a precise numerical answer, professors can confirm that students completed the required research steps rather than relying on AI-generated output.
Question example 2 : Requiring a very specific resource
For this type of question, an instructor should focus on using a specific tool or resource in a way that produces a small piece of ‘proof’ that the student used a real path to get the correct answer. For this example, I’ll expect students to use the West Key Number system, and I’ll ask a set of questions pertaining to one case. The question, as posed to students, is:
Locate the case Commonwealth v. Sharma, 488 Mass. 85, 171 N.E.3d 1076 (2021) on Westlaw.
How many key numbers in the West Headnotes section pertain to juvenile offenders?
Expected answer: 9
Click on one of the links to the juvenile offender key number. How many opinions from Massachusetts state courts are associated with this key number?
Expected answer: 111
How many of those cases are from the last 12 months?
Expected answer: 4 (see Fig. 6)
Westlaw’s AI product is the only good one to ask to check this question for how it stands up to AI. Here is what I asked Westlaw’s Deep Research:
How many times is the juvenile justice key number used in the case 171 N.E.3d 1076?
The answer: “Your question is outside the scope of this feature.”
So next I asked:
How many opinions from Massachusetts state courts are associated with the key number for juvenile justice?
I received the same response.
Potential Issues
The most obvious problem with the above AI-proof questions is the fact that they don’t require analysis or issue spotting. To resolve this, an instructor should consider using these types of questions creatively. Pose a hypothetical that requires students to identify issues before asking the more pointed questions. Consider the following example:
Your law firm’s new client, Ramone Perez, is a collage artist. He is being sued in federal court in the Southern District of New York by Annie Leibowitz for copyright infringement after some of his new collages were featured in a national art magazine. The collage at issue contains portraits of Patti Smith and Whoopi Goldberg taken by the plaintiff, a famous photographer. Perez carefully traced around each woman’s body in the portraits, then pasted the images over a cartoon image of a dumpster on fire. He added the words, “this is not fine,” in cursive across the top of the collage and changed the color of both women’s skin to green.
Your supervisor would like you to conduct some preliminary research on this topic. He thinks that the facts sound a bit like the case, Cariou v. Prince, 714 F.3d 694 (2d Cir. 2013). Consider that case and the above facts to answer the following questions:
What defense does Mr. Perez have against the copyright infringement claim?
Expected answer: Fair Use
Which of the statutory factors related to the above defense would produce your strongest argument?
Expected answer: The first factor, through which courts consider the purpose and character of the use. (Any similarly phrased answer would be acceptable.)
Which West Key Number could you use to identify other cases that deal with this factor?
99k744: Purpose and Character of Use
How many cases identified by this key number, assuming all of them are valid and relevant, would be binding on your case?
Expected answer: 180 (28 from the Supreme Court, 152 from the Second Circuit) (See Fig. 7.)
Which West Key Number could you use to identify other cases that deal with the concept of “Transformative Use”?
Expected answer: 99k747: Transformative Use
How many cases identified by this key number, assuming all of them are valid and relevant, would be binding on your case?
Expected Answer: 88 (10 from the Supreme Court, 78 from the Second Circuit. (See Fig. 8.)
Another problem with the specific-resource approach described above is that it doesn’t use Legal AI, a technology with which students desperately need experience. This can be remedied by having students draft AI prompts that help them determine the best causes of action or defenses in response to legal hypotheticals. Students can then compare the answers they reached through their own research to the results provided by AI tools. This would allow them to gain experience both using and checking the work of AI, which are both valuable skills considering the limits of legal AI technology at this time. For example, a professor could ask students to copy the above hypothetical into an AI Tool on Westlaw or Lexis and provide questions, such as:
What defenses may be available to Mr. Perez?
What Second Circuit federal cases address fair use when a photograph is alleged to have been copied?
Based on the AI responses, an instructor could ask students to identify what issues may appear in AI-cited cases (such as red or yellow flags), whether the report leads to follow up questions, and whether the AI appears to have a neutral or argumentative response which could be changed to make a better argument for their client’s position.
Conclusion
As technology continues to develop, so must we. By tailoring assessments to work around the capabilities of AI, legal research instructors can design assessments which embrace this technology while continuing to test essential research skills. Asking for small pieces of proof around specific questions as to intermediate research steps in assessments prevents AI from being able to answer questions and helps students prove that they can locate and assess legal resources on their own.
Figure 1

Figure 2

Figure 3

Figure 4
Figure 5

Figure 6

Figure 7

Figure 8

Nitza Davidovitch, Ruth Dorota & Aleksandra Gerkerova, Academic Dishonesty in Higher Education, 17 J. Compar. & Int’l Higher Educ. 131, 132-33 (2025).
Id. at 133.
Lexis announced the beginning of its AI rollout to law schools in December of 2023, and Thomson Reuters announced the same in September of 2025. See Press Release, LexisNexis, LexisNexis Collaborates with U.S. Law Schools to Roll Out Lexis+ AI, Marking First Widespread Use of Legal Generative AI Solution in Law School Education (Dec. 30, 2023),
https://www.lexisnexis.com/community/pressroom/b/news/posts/lexisnexis-collaborates-with-u-s-law-schools-to-roll-out-lexis-ai-marking-first-widespread-use-of-legal-generative-ai-solution-in-law-school-education (last visited Aug. 17, 2026); Press Release, Thomson Reuters, Thomson Reuters Expands AI-Powered Legal Education with Next-Generation Tools for Law Schools (Sept. 24, 2025), https://www.thomsonreuters.com/en/press-releases/2025/september/thomson-reuters-expands-ai-powered-legal-education-with-next-generation-tools-for-law-schools (last visited Aug. 17, 2026).
Julie L. Kimbrough, Developing Lawyering Skills in the Age of Artificial Intelligence: A Framework for Legal Education, 29 J. Tech. L. & Pol'y 31, 36 (2025).
For examples of situations in which fake citations resulted in disciplinary actions, see Sarah Martinson, Judges’ AI Orders Keep Trickling in as Fake Citations Persist, Law360 (July 21, 2025), https://www.crowell.com/a/web/hATTWfj2h9N1Tf42uN3QFg/judges-ai-orders-keep-trickling-in-as-fake-citations-persist.pdf (last visited Apr 3, 2026); Noland v. Land of the Free, L.P., No. BC716737 (Cal. Ct. App. Filed Sept. 12, 2025); Kurt Dunphy, Lawyer fined for using AI-generated legal documents with fake citations, Spellbook (June 20, 2026), https://www.spellbook.legal/learn/lawyer-fined-using-ai-legal-fake-citations (last visited Aug. 17, 2026).
Alyson Drake & Amanda Watson, Legal Research Instruction in the NextGen Era, 72 Buff. L. Rev. 1309, 1324-26 (2024).