My recent realisation has been these shops fall in a category of what I call "rating agency" all of which share common characteristics. For example; Moody's, Gartner, Michelin Stars.
1. Stripped of all the paraphernalia these are yellow pages ranked by popularity in their domains.
2. They are in the business of selling customers to the said businesses. So their primary customers are businesses; restaurants, mutual funds etc.,
3. They need to maintain authenticity of impartiality so it's not easy to find out how they make money.
4. Their secondary audience is decision makers; CIOs, money managers etc., Think of -- "no one got fired for picking at the 1st ranked software from Gartner's list", "But Moody's rated it AAA, how was I supposed to know that fund would tank". It's a terrific CYA tool.
5. I'd say about 80% of the times they do a decent job of stack ranking. But there is an element of virtuous cycle there. The top rated "things" are popular because they are ranked to be so by these rating companies. Once a critical mass adopts then that tech/restaurant/etc indeed becomes numerically popular.
I think it's worth adding: the domains that these organisations rank are chosen by those organisations. How the ranking works is chosen by the ranking org. A major source of revenue for the ranking orgs is conference sponsorships, and one of the things included in those sponsorships is access to the analysts writing the reports.
It's also worth noting that ranking companies don't rank products, they rank vendors. So a company that does everything kind of ok over multiple products can get an advantage over a company that solves one problem very well with one product, if the ranking companies choose to define the category in an overly broad way.
Have a source on this? I researched this and found that the only benefit of paying them as a client is access to the analysts that make the determination.
1. Stripped of all the paraphernalia these are yellow pages ranked by popularity in their domains.
2. They are in the business of selling customers to the said businesses. So their primary customers are businesses; restaurants, mutual funds etc.,
3. They need to maintain authenticity of impartiality so it's not easy to find out how they make money.
4. Their secondary audience is decision makers; CIOs, money managers etc., Think of -- "no one got fired for picking at the 1st ranked software from Gartner's list", "But Moody's rated it AAA, how was I supposed to know that fund would tank". It's a terrific CYA tool.
5. I'd say about 80% of the times they do a decent job of stack ranking. But there is an element of virtuous cycle there. The top rated "things" are popular because they are ranked to be so by these rating companies. Once a critical mass adopts then that tech/restaurant/etc indeed becomes numerically popular.