Credibility Factors 2020

This dataset focuses on 50 articles about climate science, which were annotated completely by 49 students, 26 Upwork workers, 3 science and 3 journalism experts.

Before participation, crowd raters filled out a demographic survey followed by committing to an Annotator Code of Conduct of performing their duties in as accurate and diligent a manner as possible provided in their informed consent. Once eligible, they received reading and rating tasks that included their credibility perception per article on a 5-point Likert scale, ranging from very low (1) to very high (5). Instructions to fill out the seven question survey across a 7-10 day period (estimated at 10 hours) were provided in a handbook with a recommended limit of 10-15 minutes per article.

Articles chosen were written in English and represented a range of liberal to conservative positions or attitudes towards climate problems. To gather articles, the team used the Buzzsumo social media research tool in late 2018 to find the most popular articles over the previous year with the keywords of “climate change,” “global warming,” “environment,” and “pollution.” Among the top results, our team selected a set of articles with varying amounts of scientific reference.