In an article published recently in the Annals of Internal Medicine, Ashwini Sehgal analyzed U.S. News's “Best Hospitals” rankings. The Algorithm U.S. News uses to create this ranking, relies heavily on repetitional ratings generated by professional peers.
Sehgal wanted to know how well that “Best Hospitals” ranking correlated to actual medical outcomes based on objective performance criteria. The answer, he discovered, was that they didn't. It turns out that having good medical outcomes, doesn't translate easily into “reputation” as judged by other doctors.
Results: On average, rankings based on reputation score alone agreed with U.S. News & World Report's overall rankings 100% of the time for the top hospital in each specialty, 97% for the top 5 hospitals, 91% for the top 10 hospitals, and 89% for the top 20 hospitals. Hospital reputation was minimally associated with objective quality measures (mean Spearman ρ2.
Related: Who comes out on top, in any ranking system, is really about who is doing the ranking.