Files in this item



application/pdfResearch_Showca ... g&JodiSchneider_ Final.pdf (2MB)
(no description provided)PDF


Title:Evaluating an automatic data extraction tool for evidence synthesis through real-life case studies
Author(s):Hoang, Linh K.; Cao, Linh T.; Guan, Yingjun; Cheng, Yi-Yun; Schneider, Jodi
Subject(s):Systematic Review
Data Extraction
Evidence Synthesis
Abstract:Computer support tools are increasingly designed to help reduce the time required for evidence synthesis tasks such as manually extracting information from scientific papers. We present two case studies evaluating RobotReviewer, an automatic data extraction tool used to help synthesize evidence from primary research papers for use in review papers. We use primary research papers related to oral health and dental medicine for both our case studies. The first case study uses the same published review we presented at last year's Research Showcase, with a new evaluation metric and 3 novice annotators. Through manual annotation and a content analysis of the six studies synthesized in the review paper, we compare how well (1) novices and (2) the RobotReviewer data extraction compare to the Cochrane Review paper (seen here as a expert gold standard). (Feedback at last year's Research Showcase was instrumental, and we have improved the methodology we presented last year in several ways; for instance those preliminary results had just one novice annotator.) The second case study is based on a real systematic review project being conducted in part at the School of Dental Medicine at the University of Buffalo. Real reviewers collaborate by comparing RobotReviewer's results with their own manual extraction results and describing how well it meets their expectations. By evaluating existing tools such as RobotReviewer, we would be able to identify gaps between what computer support tools are available and how well these tools work. This could help propose new directions for automated support systems that would help to reduce the time and human labor required for evidence synthesis tasks such as data extraction.
Issue Date:2017
Publisher:iSchool UIUC Research Showcase 2017
Genre:Conference Poster
Date Available in IDEALS:2018-01-16

This item appears in the following Collection(s)

Item Statistics