An explainable AI framework for studying query complexity and hallucinations in LLM scheduling tasks

Large language models (LLMs) are AI systems that generate text and can assist with tasks such as answering questions and scheduling appointments. However, they are susceptible to “hallucinations,” where the model provides inaccurate information or misinterprets tool outputs. We hypothesized that increasing query difficulty (i.e., multi-day or ambiguous time window queries) would facilitate higher occurrences of tool-use hallucinations by LLMs in appointment scheduling tasks because models would misinterpret structured calendar data. To test our hypothesis, we built a framework combining GPT-4o with tool-use (LangChain), rule-based validation (Prolog), and explainability methods (LIME and attention visualizations) to test generated queries categorized as simple, complex, or edge case. For simple queries, there was a baseline hallucination rate of 10.2%, 14.8% for complex, and 18.5% for edge cases, with explainability analyses showing that increased error rates resulted from misplaced model attention to more difficult phrases. This framework not only helps diagnose query types that cause LLMs to hallucinate but also shows how we can mitigate these errors to make models more dependable in real-world settings.