Why you can’t trust App World ratings

If you’ve visited the BlackBerry App World storefront lately you’ll have noticed a few new free apps. It’s part of RIM’s compensation package for users affected by the recent outage. Under Top Free you’ll see two apps tagged with “Compliments of BlackBerry.” You might also notice that they carry very low star ratings: 2 stars for Bubble Bash 2, and 2.5 stars for DriveSafe.ly. Those ratings seemingly carry heavy weight, since they result from thousands of reviews. But I’m having a hard time buying the ratings on these and other apps.

Strong disagree on these ratings Really, it’s tough to trust any aggregate rating. They can be influenced by a number of factors, many of which are irrelevant to your experience. Take the Facebook app for example. Any time there’s a bug in one of the releases App World gets flooded with negative reviews. There is also a set of negative reviews from previous, less feature-rich versions that help weigh down the overall ranking. This one stood out to me, because I think the Facebook for BlackBerry app is a wonderfully useful app. But because of issues that have nothing to do with my own experience, it receives a poor rating in App World. If you look through the reviews for DriveSafe.ly Pro and Bubble Bash 2, you’ll notice a rash of reviews in recent days. This makes enough sense, because RIM gave away these apps for free. That opens up the user base from people who researched and then bought the app, to literally every BlackBerry user. The reviews, then, are completely unreliable. They come from types of users who wouldn’t normally download these apps — or perhaps any apps. You’ll notice many one-line reviews that do not elaborate at all on the rating. Those are useless. For all we know, these users messed up on their end, and blame the app for their own incompetence. This isn’t to say that there can’t be bad apps. There certainly can. It is to say that just because an app, particularly a free app, receives a low aggregate rating in App World doesn’t mean that it’s necessarily a bad app. This goes especially for the new apps, which now have users that they never would have previously. One glance through the reviews makes that clear. If someone rates an app zero stars and writes just “Horrible!” in the review, well, that’s of no use to me. Yet it affects the app’s rating as much as any other review. (Also, the number of exclamation points and capital letters in a review is inversely correlated to its usefulness.) This brings up something about App World that I feel is sorely lacking. What it really needs in order for its ratings system to be useful in any way is an Amazon-esque breakdown. Seeing that Bubble Bash 2 has a 2-star aggregate rating through 1,914 reviews does nothing for me. What I want to see is how many 5-star reviews it got. I especially want to see the number of zero- and 1-star reviews. And then I want to read those reviews. That’s how I work on Amazon. If the 1-star reviews are well-written and thought out, I give them more weight. If they’re asinine complaints from people who clearly missed the point, I can discard them. App World absolutely needs a system like this, or else the ratings are useless. I’ve played Bubble Bash 2, and it’s really fun. Had I listened to the App World rating, I wouldn’t have known that. But if I saw that the zero- and 1-star reviews were pointless, I would have downloaded it despite the poor aggregate rating. In a marketplace such as App World, reviews and ratings are necessary. They’re supposed to give users a sense of what they might like. But in its current implementation, the App World ratings are essentially useless. There’s no way to filter out the low ratings that provide no elaboration. Those might as well not exist. There’s no way to quickly filter the ratings and see which low ratings are caused by user error, rather than a flaw in the app. Until RIM can address these issues and provide ratings that we can actually use, I’ll safely disregard any star ratings and reviews in App World. It’s sad, but unfortunately there’s little, if anything, to gain from them.