Scoring AI for Accessibility: A One-Year Rubric-Based Comparison of Generative AI and Metadata

Scheduled at 11:30 in Matchless on Thursday, November 19.

Speaker(s)

• Stacy Snyder, Digital Collections Projects and Compliance Librarian; Accessibility Coordinator, University at Buffalo, State University of New York

Session Details

• Length of Session: 1-hr
• Format: Lecture
• Expertise Level: Beginner
• Type of session: General Conference

Summary

Driven by the need to address large-scale accessibility remediation under ADA Title II, in early 2025 the project team assessed outputs from three AI tools using a rubric measuring accuracy, relevance, and clarity. Results revealed significant limitations, including hallucinations and omissions of major elements. A year later, the team repeated the assessment. This presentation will discuss the new findings and compare them to the results of the original test.

Abstract

This presentation shares findings from a multi-year evaluation of generative artificial intelligence (AI) tools used to create alternative text and descriptive metadata for digital collections at the University at Buffalo Libraries. Digital collections in this context consist of digitized primary source archival materials, such as historical photographs, manuscripts, and postcards, that have been converted from physical formats into digital images for online access.

Driven by the urgent need to address large-scale accessibility remediation of thousands of digital images, the project team assessed outputs from three AI tools (ChatGPT, Claude, and Gemini). To do this, the team developed a rubric to measure factual accuracy, relevance and task completion, and clarity of communication in the outputs. The project team evaluated AI-generated alt text and descriptions of digitized images from a range of archival collections.

Results revealed significant limitations, including hallucinations, omission of critical visual details, and cultural insensitivity. After using ChatGPT to generate alt text and descriptions for one year, the team decided to rerun their 2025 assessment to examine whether development of the tools resulted in better outputs.

This presentation will address challenges such as evaluating rapidly evolving tools and comparing the results over time. It will also discuss best practices and workflows that have been put into place over the last year to address the accessibility needs of a large and diverse digital collection. The presenter will share lessons about the importance of human review and cross-unit collaboration with user experience and accessibility colleagues, and will provide examples of the project’s rubric, workflows, and documentation.

Keypoints

1. AI-generated alt text for archives still shows accuracy gaps, missing details, and bias.
2. A structured rubric enables systematic evaluation of AI hallucinations, relevance, and clarity over time.
3. Human review and cross-unit workflows are essential to ethical, scalable accessibility remediation.

Disability Areas

Deaf/Hard of Hearing, Vision

Topic Areas

Artificial Intelligence, Other

Speaker Bio(s)

Stacy Snyder

Ms. Snyder is the Digital Collections Projects and Compliance Librarian and Accessibility Coordinator at the University at Buffalo Libraries, where she oversees rights compliance and digital accessibility for all library digital collections. She holds an MLS from the University?at?Buffalo.