Honesty may not be the best policy for hospital safety grades, study suggests

Basing grade on hospitals’ self-reports about hospital safety protocols skews results on Leapfrog grades; new methodology and grades due soon

ANN ARBOR, MI – When a child brings home a report card from school, part of their grade comes from how often they made it to class or turned in homework. But the larger part comes from how they did on tests, in class and on take-home assignments. In other words, how much they’ve learned, or how hard they’re trying.

These days, hospitals also get report cards, from several organizations that combine different measurements of how well hospitals serve their patients and keep them safe. Some only use objective patient outcomes, like infection rates, while others allow hospitals to report on their efforts to improve safety.

But a new study finds that a well-known hospital grading system may put too much weight on the wrong things.

The study, published in the journal Medical Care by a University of Michigan team, takes an in-depth look at the Hospital Safety Grade system, run by The Leapfrog Group, a nonprofit organization that focuses on patient safety.

The researchers find that most hospitals that participate in Leapfrog’s Hospital Survey self-report that they’re in perfect compliance with most, if not all, of the “Safe Practices” that the survey asks about. But the researchers show that how a hospital did on these measures had little in common with how that same hospital did on independent measurements of hospital-acquired infections -- or with whether the government had penalized it for high infection or readmission rates.

“The Safe Practices part of the Hospital Safety Grade is based on whether a hospital self-reports that it has adopted certain protocols, and not whether those protocols are actually followed,” says Jennifer Meddings, M.D., M.Sc., the study’s senior author and an assistant professor of internal medicine at the U-M Medical School.

For example, she says, one Safe Practices measure involves hand hygiene. “It may be news to consumers that this measure involves hospital self reports of having specific hand washing protocols in place, not how often healthcare workers are washing their hands.”

She also notes that any one hospital’s Safe Practices scores counts toward its overall Hospital Safety Grade based on a comparison with the average Safe Practices measure score reported across all participating hospitals. So, hospitals receive little grade advantage by reporting high scores -- but are at risk of much lower grades if they report less-than-perfect scores.  

The researchers note that the Leapfrog organization will issue a new grading formula next month – but not in time to calculate the new grades that will come out soon after.

To self-report or not to self-report? 

Hospital Safety Grades combine data from a number of national sources, including public data from the Centers for Medicare and Medicaid Services and the Centers for Disease Control and Prevention -- and for hospitals that choose to participate, the voluntary Leapfrog Hospital Survey.

Just over half of hospitals in the study completed the Leapfrog survey, which includes self-reporting of eight National Quality Forum Safe Practices measures. If a hospital reports on them, it accounts for nearly a quarter of the hospital’s total grade. Hospitals that do not complete the voluntary Leapfrog Hospital Survey also receive a Leapfrog Hospital Safety Grade, calculated entirely using publicly-available measures.

The researchers ran simulations to see how these non-participating hospitals’ grades might have changed if they had self-reported low, average, or perfect Safe Practices measures, while their infection rates remained the same.

They find that -- even if their actual performance on hospital infections is good -- such hospitals would likely end up with lower Leapfrog Hospital Safety Grades if their performance on the self-reported Safe Practices was anything less than perfect.

In other words, hospitals that self-report less-than-perfect Safe Practices scores are likely better off not completing the Leapfrog Hospital Survey.

Shawna Smith, Ph.D., the study’s lead author, explains, “Hospitals that have less-than-perfect performance on Leapfrog’s list of protocols get a big hit to their grade, but those with most or all protocols in place are rewarded very little. Our simulations show that some hospitals would have had better grades when they didn’t report than when they reported imperfect compliance.” Smith is a U-M Internal Medicine statistician and Institute for Social Research postdoctoral fellow.

This disincentive also creates a problem for moving hospitals forward on safe practices. “One of the principles of promoting safe practices is to ensure that hospitals are comfortable reporting that they still have work to do,” says Meddings, a member of U-M’s Institute for Healthcare Policy and Innovation who also holds a position at the VA Center for Clinical Management Research.

A path forward for a pioneering effort

The authors note that Leapfrog has encouraged public reporting on patient safety by hospitals for 16 years – dating back to a time when the public had little or no access to information about how safe their hospitals were. Their paper gives a timeline of public reporting by Leapfrog and others.

Now, more than half of hospitals that complete the Leapfrog survey report perfect performance on all of the Safe Practice measures. The researchers say it’s time to put more weight on publicly reported safety measures from objective sources rather than self-reporting of protocols.

While the voluntary Leapfrog hospital survey is free, hospitals are encouraged to use their grades to promote their hospitals to patients and potential patients – but must pay licensing fees to use the grades as a marketing tool showing their grade.

“The Leapfrog approach was really cutting edge when it first came out, and started an important movement,” says Meddings. “But Leapfrog relies in part on process measures which can ‘top out’ as a new process becomes widely adopted. It’s time to have a new discussion about how to measure new metrics and use them in innovative ways to evaluate hospitals.”

Adds Smith, “Leapfrog has a number of well-regarded experts involved, and their use of the Safe Practices measures does add something different to the universe of metrics out there. But they need to think how they are used in calculating the grade, because the current methodology penalizes hospitals for reporting scores that are anything less than perfect. If we want to encourage hospitals to continually improve safety, we need to incentivize acknowledging they have work to do.”

Disclosure: The University of Michigan’s academic medical center, Michigan Medicine, participates in the Leapfrog survey

The research was funded by the Agency for Healthcare Research and Quality (HS19767, HS018334). Co-authors are Heidi A. Reichert, MA and Jessica M. Ameling, MPH.

Reference: Medical Care, doi: 10.1097/MLR.0000000000000716