Identifying retraction information within DOI datasets using crowd-sourced annotations

Adhikesaven, Sanjay

Permalink

https://hdl.handle.net/2142/118215 Copy

Description

Title

Identifying retraction information within DOI datasets using crowd-sourced annotations

Author(s)

Adhikesaven, Sanjay

Contributor(s)

• Schneider, Jodi
• Subin, Karen Ann

Issue Date

2023-03-17

Keyword(s)

• crowdsourcing
• retracted publications
• retraction notices
• DOIs
• Crossref
• Cochrane Crowd
• annotation
• guideline development

Abstract

Crowd-sourced data annotation tasks are important in labeling large amounts of data. In this work, we design a crowdsourcing system to identify retraction information with large DOI datasets. Specifically, we want DOIs to be classified into one of six categories: retracted paper, retraction notice, correction, no access, uninterpretable, or papers using retraction. Thus, we create a set of annotation instructions designed such that any user can perform this annotation task. In the annotation instructions, we include descriptions and definitions of each of the six categories such that a user can identify which description best matches with the paper. We also include examples of DOIs that fit each of the categories and figures to aid in annotation. For preliminary testing, we used two DOI datasets each containing around 200 DOIs which all related to retraction. After testing the instructions, we found that users without much prior knowledge before using the instructions perform with high accuracy, with the category that has the most variance being the no access category since it varies depending on the specific libraries that a given user has access to. Our future work will be on conducting further testing on our instructions and ultimately to implement the present system within Cochrane Crowd webforms to assess the feasibility on a larger scale.

Publisher

iConference 2023 Student Symposium

Type of Resource

Presentation

Language

en

Sponsor(s)/Grant Number(s)

• Alfred P. Sloan Foundation G-2020-12623
• Alfred P. Sloan Foundation G-2022-19409

Owning Collections