Caption Accuracy Metrics
Real-time captioned news is a lifeline service for people who are deaf or hard of hearing, providing critical information about their local communities, national events and emergencies. Captioning mandates designed to provide equal access to television have resulted in rapid growth of the caption industry, but a shortage of skilled real-time stenocaptioners, and the downward pressure on rates by program providers, has made the low quality of live captioning of news broadcasts a growing issue.
Disability organizations have filed complaints and a formal petition with the Federal Communications Commission (FCC) which reflects frustration with chronic problems related to live captioning quality, transmission errors, and lack of industry response to their concerns. However, without a common means of measuring accuracy and quality, the FCC, consumers and broadcasters have no efficient method of tracking and improving stenocaption accuracy performance.
The WGBH National Center for Accessible Media is utilizing language-processing tools to develop a prototype automated caption accuracy assessment system for real-time captions for live news programming or classroom-based communication access realtime translation (CART) captioning. We are researching whether text-based data mining and automatic speech recognition technologies can produce meaningful data about stenocaption accuracy that meets the need for caption performance metrics.
Prototypes will be reviewed by major stakeholders at Technical Review Meetings. Advisors include the National Institute of Standards and Technology, the Massachusetts Institute of Technology, Gallaudet University and the National Technical Institute for the Deaf.
Iterative tests and modifications within major stenocaption and broadcast operations facilities will provide real-world assessments of the system's ability to produce meaningful caption accuracy metrics.
A reliable performance measurement tool that can analyze the quality of real-time captioning, developed with input from industry leaders, deaf education experts, and the National Institute of Standards and Technology will provide Congress and the FCC with much-needed independently verified data to establish caption accuracy requirements. This will greatly improve the ability of the television community to monitor and maintain the quality of live captioning they offer to viewers who are deaf or hard of hearing and ease the current burden on caption viewers to document and advocate for comprehensible captions to ensure they have equal access to important national and local information.
Deliverables include:
Disability organizations have filed complaints and a formal petition with the Federal Communications Commission (FCC) which reflects frustration with chronic problems related to live captioning quality, transmission errors, and lack of industry response to their concerns. However, without a common means of measuring accuracy and quality, the FCC, consumers and broadcasters have no efficient method of tracking and improving stenocaption accuracy performance.
The WGBH National Center for Accessible Media is utilizing language-processing tools to develop a prototype automated caption accuracy assessment system for real-time captions for live news programming or classroom-based communication access realtime translation (CART) captioning. We are researching whether text-based data mining and automatic speech recognition technologies can produce meaningful data about stenocaption accuracy that meets the need for caption performance metrics.
Prototypes will be reviewed by major stakeholders at Technical Review Meetings. Advisors include the National Institute of Standards and Technology, the Massachusetts Institute of Technology, Gallaudet University and the National Technical Institute for the Deaf.
Iterative tests and modifications within major stenocaption and broadcast operations facilities will provide real-world assessments of the system's ability to produce meaningful caption accuracy metrics.
A reliable performance measurement tool that can analyze the quality of real-time captioning, developed with input from industry leaders, deaf education experts, and the National Institute of Standards and Technology will provide Congress and the FCC with much-needed independently verified data to establish caption accuracy requirements. This will greatly improve the ability of the television community to monitor and maintain the quality of live captioning they offer to viewers who are deaf or hard of hearing and ease the current burden on caption viewers to document and advocate for comprehensible captions to ensure they have equal access to important national and local information.
Deliverables include:
- Publication of experimental ontology of caption error types;
- Publication of research into error capture capabilities of text mining software agents, customized with rules and classifications derived from stenocaption ontology; and
- Prototype software application that provides a technical framework to utilize language processing tools to conduct ontology-based detection, ranking and reporting of stenocaption errors.
Project Resources
Now available:
Caption Accuracy Metrics Research into Automated Error Ranking
Caption Accuracy Metrics presentation at 2011 CSUN conference
Caption Accuracy Metrics Survey Final Report
Caption Accuracy Metrics Survey
Press Release, 11/9/10: WGBH Works with Nuance Communications, Inc. on Effort to Improve the Quality of Live News Captioning
FCC Closed Captioning of Video Programming
Project Staff
Marcia Brooks, Project Directormarcia_brooks@wgbh.org
617 300-3431
Tom Apone, Project Technical Manager
tom_apone@wgbh.org
617 300-3717