Why AI Can’t Replace Historians: The Importance of Human Insight

Why AI Can't Replace Historians: The Importance of Human Insight

Microsoft’s recent study on jobs likely to face AI augmentation sparked a debate about job security, especially for historians, whose role was notably high on the list. Many historians expressed concern about AI potentially taking over their responsibilities. However, my exploration of generative AI tools using specific historical queries suggests there’s little reason for these professionals to worry just yet.

As someone fascinated by the movies presidents have watched while in office, I decided to test these AI tools with queries centered around this topic. Since 2012, I have dedicated myself to researching which films presidents enjoyed, starting with Ronald Reagan’s movie list from the White House. This journey led me to submit a Freedom of Information Act (FOIA) request for Barack Obama’s movie screenings, only to discover presidential records remain confidential for five years post-tenure. Undeterred, I continued exploring, including Teddy Roosevelt’s first documentary screening in 1908.

AI’s Performance: A Closer Look

The purpose of my tests was to evaluate AI’s accuracy when faced with queries I knew intimately. Many users often ask about unfamiliar topics, but I sought to challenge the AI’s claims by inputting known facts. Given the hype surrounding these tools, the insights gained could be illuminating.

Initial Experiments with GPT-5

First, I assessed OpenAI’s GPT-5 by asking about films presidents viewed on specific dates spanning the terms of Woodrow Wilson to George W. Bush. Each inquiry repeatedly yielded the same answer: that there was no record of any presidential screenings on the given dates. While ChatGPT didn’t mislead outright, it failed to provide straightforward information, raising concerns about the tool’s reliability.

Problems with Copilot’s Responses

Next, I shifted my questions to Microsoft Copilot, utilizing both Quick Response and Deep Research modes. Quickly, I learned that the Quick Response provided incorrect information, claiming that President Eisenhower watched The Unconquered on August 11, 1954. However, the AI inaccurately identified Eisenhower’s brief appearance in archival footage as a basis for this response.

Deep Research yielded an even lengthier but similarly flawed response, suggesting Eisenhower probably viewed Suddenly, a film whose release date claimed it was impossible for him to have watched on August 11, 1954. My own historical knowledge, bolstered by proof from the White House projectionist’s logs, confirmed he actually viewed River of No Return.

Exploring Other Chatbots

Curiously, I explored other chatbots like Google Gemini and Perplexity, yet they also struggled with these historical queries. They often guessed Suddenly without accurately citing verified historical sources. In an interesting twist, xAI’s Grok, after a few attempts, identified River of No Return, but only by referencing my own previous posts on social media.

Why does this matter? It illustrates the necessity of human oversight in historical research. AI may deliver decent answers for popular inquiries but falters significantly when faced with nuanced or less-documented historical facts.

Sifting Through Nixon’s Movie Choices

When I inquired about what movie Richard Nixon viewed on February 12, 1971, I found conflicting results. Copilot mistakenly claimed he watched Patton that day, yet in reality, he enjoyed The Great Chase, as evidenced by reputable sources.

Understanding Woodrow Wilson’s Screenings

For my query on Woodrow Wilson’s movie choices on March 6, 1917, Copilot erroneously asserted that he watched The Birth of a Nation, although he had seen it two years earlier. This underlines how AI may perpetuate inaccuracies, reinforcing the value of expert historians.

Reagan’s Viewing Habits Debunked

Despite Copilot’s claims about Reagan watching Rambo: First Blood Part II, the actual screening was The Lion in Winter. Accessing original records proves invaluable for historians striving for accuracy.

Questions That Matter

What role does AI play in historical research today?
AI offers initial data but often lacks depth, necessitating a historian’s expert input for contextual analysis.

How accurate are AI-generated historical accounts?
While AI can provide information, it may misinterpret or misstate facts, emphasizing the need for human verification.

Can AI replace historians altogether?
For straightforward tasks, AI can assist but lacks the critical thinking and deeper comprehension that historians provide.

Using diverse queries across several AIs, I’ve confirmed a pattern. My tests may not represent a full scientific study, but they reflect the realities of how AI handles historical inquiry. While these tools are becoming prevalent, they still require continual human guidance and fact-checking to maintain accuracy.

In conclusion, AI can enhance research, but reliance solely on these tools can lead to considerable factual errors. Historical inquiry encompasses not just gathering data but rigorously analyzing and contextualizing it. For those engaged in historical research, reviewing AI outputs critically is paramount.

Continue exploring this fascinating overlap between technology and history at Moyens I/O.