Individual Gestalt Is Unreliable for the Evaluation of Quality in Medical Education Blogs: A METRIQ Study

Autor: Brent Thoma, Stefanie S. Sebok-Syer, Keeth Krishnan, Marshall Siemens, N. Seth Trueger, Isabelle Colmers-Gray, Rob Woods, Emil Petrusa, Teresa Chan, Charlotte Alexander, Mohammed Alkhalifah, Saeed Alqahtani, Scott Anderson, Shelaina Anderson, Colin Andrews, Jocelyn Andruko, Felix Ankel, Nikytha Antony, Diptesh Aryal, Barbra Backus, Jennifer Baird, Andrew Baker, Sarah Batty, Jared Baylis, Braeden Beaumont, Chris Belcher, Brent Benavides, Michael Benham, Elyse Berger Pelletier, Julian Botta, Nicholas Bouchard, Victoria Brazil, Emily Brumfield, Anthony Bryson, Wisarut Bunchit, Kat Butler, Lindy Buzikievich, David Calcara, Rob Carey, Stephen Carroll, Casey Lyons, Louise Cassidy, Kirsty Challen, Tim Chaplin, Natasha Chatham-Zvelebil, Eric Chen, Lucy Chen, Sushant Chhabra, Alvin Chin, Eric Chochi, Tina Choudhri, Jeremy Christensen, Kimberly Connors, Veronica Coppersmith, Abby Cosgrove, Gregory Costello, Kevin Cullison, Andrew D'Alessandro, Kerstin de Wit, Marie Decock, Rayan Delbani, William Denq, Julianna Deutscher, Brendan Devine, Maia Dorsett, Taylor Duda, Justin Dueweke, Teresa Dunphy, Sean Dyer, Kathryn T. Eastley, Marcia Edmonds, Ken Edwards, Robert Ehrman, Youness Elkhalidy, Preston Fedor, Brian Ficiur, Caley Flynn, Bill Fraser, Meagan Fu, James Fukakusa, Eric Funk, Damjan Gaco, Viktor Gawlik, Kenn Ghaffarian, Laleh Gharahbaghian, Phil Griffith, Andrew Griffith, Andrew Grock, Tanner Gronowski, Cathy Grossman, Jaroslaw Gucwa, Pawan Gupta, Alexandra Gustafson, Andrew Guy, Mary Haas, Stanislaw Haciski, Emina Hajdinjak, Andrew K. Hall, Regina Hammock, Jan Hansel, Alexander Hart, Larissa Hattin, Brandon Herb, SueLin Hilbert, Jesse Hill, Jeff Hill, Amy Ho, Emily House, Nina House, James Huffman, Charlie Inboriboon, Alex Ireland, Mohammed Ali Jamal, Victor Jansen, Zach Jarou, Vivian Jia, Levi Johnston, Drew Kalnow, Puneet Kapur, Seth Kelly, Kyle Kelson, William Kent, Rishi Khakhkhar, Jaasmit Khurana, Ashley Kilp, Scott Knapp, Sebastian Kohler, Ivanna Kruhlak, Nadim Lalani, Samantha Lam, Patrick Lank, Zander Laurie, Kristina Lea, Ernest Leber, Ching-Hsing Lee, Haakon Lenes, Nilantha Lenora, Jesse Leontowicz, Kelly Lien, Yingchun Lin, Michelle Lin, Andrew Little, Ivy Liu, Harry Liu, Steve Liu, Stephanie Louka, Elise Lovell, David Lowe, Ashley Lubberdink, Jessica Luc, Sheng-Hsiang Ma, Hugh MacLeod, Nick Mancuso, Anali Maneshi, Dra. Maria Rosa Carrillo, Jesse May, John Mayo, Mike McDonnell, Susan McLellan, Carolyn McQuarrie, Julia Nood, Therese Mead, Cory Meeuwisse, Patrick Meloy, Perry Menzies, Anne Messman, Stephen Miazga, Logan Mills, Ken Milne, Allan Mix, Steve Montag, Brendon Moore, Justin Morgenstern, Sarah Mott, P. Mukherj, Ali Mulla, Sheena Nandalal, Taylor Nikel, Sean Nugent, Morgan Oakland, Werner Oberholzer, Onyeka Otugo, Taofiq Segun Oyedokun, Mike Paddock, Alim Pardhan, Kinjal Patel, Quinten Paterson, Catherine Patocka, Christine Patterson, James Pearlman, Alexis Pelletier-Bui, Marc Phan, Zafrina Poonja, Aubrey Powell, Kamini Premkumar, Gregor Prosen, Vishal Puri, Tanis Quaife, Ryan Raffel, Ali Raja, Randi Ramunno, Louise Rang, Suzanne Rannazzisi, Shauna Regan, Milan Ridderikhof, Vanessa Rogers, Christine Roh, Keith Rosenberg, Marina Roure, Sherri Rudinsky, Joshua Rudner, Adeeb Saleh, Will Sanderson, Owen Scheirer, Paul Schofield, Paul Schunk, Evan Schwarz, Parisa Shahrabadi, Eric Shappell, Julia Sheffield, Jonathan Sherbino, Manpreet Singh, Hector C. Singson, Dave Slessor, Sam Smith, Paula Sneath, Robert Sobehart, Kerry Spearing, James Stempien, Britni Sternard, Tara Stratton, Katherine Stuart, Bob Stuntz, Michael Susalla, Colleen Sweeney, Loice Swisher, Henry Swoboda, Shahbaz Syed, Taku Taira, Nikhil Tambe, Richard Tang, Elisha Targonsky, Rachel Taylor, Alan Taylor, Todd Taylor, Paxton Ting, Gerhard Tiwald, Kelvin Tran, Evelyn Tran, Jason Trickovic, Paul Trinquero, Seth Trueger, Aaron Tyagi, Manrique Umana, Patrick Vallance, Patricia Van den Berg, Luis Vargas, Rene Verbeek, Sandra Viggers, Zlata Vlodaver, Matthew Wagner, Noorin Walji, Joe Walter, Miranda Wan, Rachel Wang, Gregory Wanner, Wyatt Warawa, Mike Ward, Jennifer Weekes, Kristen Weersink, Cara Weessies, Anna Whalen-Browne, Brian Whiteside, Matthew Willis, Jonas Wilmer, Nelson Wong, Mark Woodcroft, Lawrence Yau, Jessica Yee, Calvin Yeh, Simon York Ming Huang, Katherine Yurkiw, Fareen Zaver, Alexander Zozula
Přispěvatelé: Graduate School, Emergency Department, ACS - Heart failure & arrhythmias, ACS - Diabetes & metabolism
Rok vydání: 2016
Předmět:
Zdroj: Annals of emergency medicine, 70(3), 394-401. Mosby Inc.
ISSN: 1097-6760
0196-0644
Popis: Study objective Open educational resources such as blogs are increasingly used for medical education. Gestalt is generally the evaluation method used for these resources; however, little information has been published on it. We aim to evaluate the reliability of gestalt in the assessment of emergency medicine blogs. Methods We identified 60 English-language emergency medicine Web sites that posted clinically oriented blogs between January 1, 2016, and February 24, 2016. Ten Web sites were selected with a random-number generator. Medical students, emergency medicine residents, and emergency medicine attending physicians evaluated the 2 most recent clinical blog posts from each site for quality, using a 7-point Likert scale. The mean gestalt scores of each blog post were compared between groups with Pearson's correlations. Single and average measure intraclass correlation coefficients were calculated within groups. A generalizability study evaluated variance within gestalt and a decision study calculated the number of raters required to reliably (>0.8) estimate quality. Results One hundred twenty-one medical students, 88 residents, and 100 attending physicians (93.6% of enrolled participants) evaluated all 20 blog posts. Single-measure intraclass correlation coefficients within groups were fair to poor (0.36 to 0.40). Average-measure intraclass correlation coefficients were more reliable (0.811 to 0.840). Mean gestalt ratings by attending physicians correlated strongly with those by medical students (r=0.92) and residents (r=0.99). The generalizability coefficient was 0.91 for the complete data set. The decision study found that 42 gestalt ratings were required to reliably evaluate quality (>0.8). Conclusion The mean gestalt quality ratings of blog posts between medical students, residents, and attending physicians correlate strongly, but individual ratings are unreliable. With sufficient raters, mean gestalt ratings provide a community standard for assessment.
Databáze: OpenAIRE