Repo rating system: 5 stars or binary ratings?

[Hồng Thất Công]

Yeah like ptitSeb said and just show the numbers of downloads.  The more downloads the more popular the app is ... 

And if you don't like an app/game, speak up in the comment section and say that it sucks without hiding in the stars so devs can kick your ass

 

[milkshake]

Why not make ratings an optional thing so people who find them useful can still make / use ratings and those that dont can switch them off?

 

[rygD]

Because then those who put stuff on the repo (and have it disabled) will get curious and check and see that it only has 1 rating, and it is 1 star, and they feel like no one likes what they are giving us, so then they stop putting stuff on the repo.

milkshake, stop trying to kill the repo! 

 

[ekianjo]

I think most people in this thread who complain about the ratings are obviously creators/porters and while they are not happy for good reasons about the feedback they get they are completely missing the point from the user side.

The point of having ratings is for discovery. If tomorrow you remove ratings how will a user ever find anything thats remotely good versus pnds that are just not working anymore or just not very good out there or completely outdated ? And no, "downloads" is just a stupid idea because something that has been longer out there will be higher in number of downloads regardless of quality.

There's a reason NO REPO SYSTEM OUT THERE (steam, appstore, google play, and more) dropped ratings. They are useful for the user, and even if you think they are not representative they are still helpful as no one is going to download 1000+ PNDs or read 3000 comments before deciding what is worth it or not.

So instead of dropping ratings just because you dont like it, id like to see guys who complain come up with a BETTER solution for content discovery (and better as in, does not suck balls and does not take 10 times longer to find something worth checking out).

I stand by my previous recommendations:

1) mandatory comments for each rating

2) encouraging users to rate more often (there are many things to try before giving up, and noone has tried them out)

3) potentially two ratings instead of one (one for content, one for technical/porting aspects)

 

[ekianjo]

As a side note notaz says the ratings have no meaning yet I struggle to find any shite software in the top 50 based on ratings. There may be a couple which are questionable but as a whole this clearly shows the ratings work: shite does not move to the top.

 

[ptitSeb]

@ekianjo: I don't think I forgot the end-user. The current rating is not efficiant because of the lack of rates. Current TOP Rated is not relevant. For example, "Kiki the nano bot" is a nice game, but I don't think it deserve to be top rated, in front of many classic or truly must-have PND.

I would be interrested if @milkshake could run a simulation of my thumbup/double thumbup proposal on the current repo stat and give us what would be the Top 20 using thoses rules, just to see.

 

[rygD]

If tomorrow you remove ratings how will a user ever find anything thats remotely good versus pnds that are just not working anymore or just not very good out there or completely outdated ?

Comments.  In fact, by checking the date of comments you can see when they were working.  I often do this ("oh, someone said it was good last week, I will check it out" or "it was working 2 years ago, but I wonder if the recent updates killed it...I don't have time to mess around much, I will skip it for now").  By looking at ratings you just see that most of the 8 people that rated it at some point since it was originally put on the repo rated it positively.  You don't know when, how many, or why they rated it the way they did.

So instead of dropping ratings just because you dont like it, id like to see guys who complain come up with a BETTER solution for content discovery (and better as in, does not suck balls and does not take 10 times longer to find something worth checking out).

I don't think I am a complainer, I just don't use ratings, and I feel if some who populate the repo feel less motivated to continue due to ratings it needs to be addressed, but I think several options that have been suggested would be an improvement.  The problem is that some think that requiring a comment with a rating is a bad choice.  Comments are valuable in the eyes of several in this discussion, so that should be kept in mind, regardless of what, if anything, is done.  Guidelines on how to rate, or even changing what options are presented to raters, might help, but will have the same results if they are disregarded, or you only have a handful of people rating. 

Another of my stupid ideas:  Limit rating to three choices, "Positive", "Neutral", and "Negative".  You can still present the results with five stars, or a comment that says something like "This is regarded as very bad based on 1 rater" or "Neutral, according to 2 ratings".  This really changes very little, and requires more effort than it is worth, but maybe it can give others ideas.

I stand by my previous recommendations:
1) mandatory comments for each rating

2) encouraging users to rate more often (there are many things to try before giving up, and noone has tried them out)

3) potentially two ratings instead of one (one for content, one for technical/porting aspects)

I am pretty much fully with you on all of these.  I just wish I could rate/comment when I am updating or searching the repo (probably user error).

As a side note notaz says the ratings have no meaning yet I struggle to find any shite software in the top 50 based on ratings. There may be a couple which are questionable but as a whole this clearly shows the ratings work: shite does not move to the top.

Software that isn't going to be used by most people will also not rise to the top.  If only 2% of users will use a particular program, but it is the best of this type of thing on the planet, and made specifically for the Pandora, yet has no ratings, the ratings are a bit useless for people that might benefit from it.  I suspect there are some hidden gems with no, or negative, ratings.  If we judge things strictly based on popularity why are we using Pandoras?  If this way of making a decision is best for everyone then we obviously we need to ditch it in favor of something thinner, with only a touchscreen, or we need Windows on it.  When you only have a few raters it is really kind of difficult to accurately sort the quality of things based on ratings.  Even you said there are questionable things in the top 50.  Popularity is not quality, especially when judged by few.

 

[ekianjo]

Rygd you cant rely on comments for discovery since there is no way to filter pnd by comments. So its not helping to find your way in 1000+ pnds at all. Comments cannot be averaged, cannot be summarized and there do not help for discoery.

 

[ekianjo]

Your last point is moot as well. You are telling me ratings dont work because its not perfect, while i merely said it was working and had a use. Because its not perfect does not mean it has no use.

 

[_jr_]

Your logic is flawed. From 'there is no crap in the top 50' simply does not follow 'our existing repo ratings are useful to explore the 1000+ PNDs'. And actually they aren't as explained at length in this thread. That they kind of work for the top 50 actually makes it more of a problem, because this gives newcomers the wrong impression that the repo ratings are useful even beyond the top 50.


As proposed above it might work to just not display any rating for PNDs with fewer than 50 (or something) ratings. This would keep the rating for the top rated applications intact and faithfully show that most PNDs don't have enough ratings to give a useful indication (because individual ratings have much too much weight then). Even that is fishy, because - as notaz succinctly put - the ratings have very little meaning beyond 'I like or dislike author, effort, general idea, or whatever'. Only comments make it more clear and that's why a long time ago it was already proposed somewhere to make comments more prominent on the repo. E.g. show a list of recently commented on PNDs instead of the list ordered by rating. This would foster useful feedback and make it easier to police trolls.


In the end, ignoring the ratings is currently the right thing to do beyond the most rated PNDs and the user is served much better with the already existing (both on the repo and in PNDManager) search by category and then looking at comments and download numbers (missing from PNDManager and currently not available by version). We all know the categories have been poorly designed and are therefore broken, too, but much less so than the repo rating system. They easily reduce the search space to manageable proportions (at least as long as we don't have significantly larger numbers of PNDs).


One last thing this discussion made obvious to me: having the repo is essential. Even if the rating system may not work out as intended, the typical package repositories are much less accessible. Thanks milkshake.

 

[ekianjo]

Another of my stupid ideas: Limit rating to three choices, "Positive", "Neutral", and "Negative". You can still present the results with five stars, or a comment that says something like "This is regarded as very bad based on 1 rater" or "Neutral, according to 2 ratings". This really changes very little, and requires more effort than it is worth, but maybe it can give others ideas.

The problem is not about the scale of ratings whether you make it 3 scale or binary. That won't change the issue that RATINGS are QUANTITATIVE by nature and therefore need a certain number of answers to be meaningful (which is the opposite of comments which are qualitative by nature).

The point that notaz was making earlier that "ratings do not mean anything because we don't know what people mean by it" is also moot, because even if people rate things differently, as long as enough people vote the sample of ratings becomes comparable. That's how statistics work, and I feel very strange that I have to explain this kind of things here to people who should probably know about stats in the first place. That's why asking 200 people in the street about their opinion can more or less represent the national opinion, as long as the sample is relevant.

Now, coming back on potential solutions, since ratings are qualitative and only meaningful when a certain threshold is reached, I would propose to make ratings "invisible" to the public (except the developer) until the threshold is reached. And a message such as "more votes are needed" should be displayed to encourage voting for those PNDs which do not have votes, instead of showing blank stars (which makes it look that it has ZERO stars, even though that's not its intention) or light blue stars based on a single rating, which is by nature meaningless because it's below the threshold.

Only comments make it more clear and that's why a long time ago it was already proposed somewhere to make comments more prominent on the repo. E.g. show a list of recently commented on PNDs instead of the list ordered by rating. This would foster useful feedback and make it easier to police trolls.

As mentioned above, again and again, comments do not help for discovery, even if you make the last 10 comments visible. because you can't filter them on any criteria.

 

[ekianjo]

ignoring the ratings is currently the right thing to do beyond the most rated PNDs

I disagree. The top rated list shows a different list than the top downloaded one (which almost never changes) and therefore helps the discovery of new PNDs. Look at NPRquake which is currently first, it could never have been showcased before if you were sorting by number of downloads、and none of the comments would have helped people to have a look at it since there's not much contents in them either. So, explain me how one would have found about NPRQuake (which is an excellent port of ptitSeb, that I want to thank here again by the way, even though I disagree with his opinion on removing ratings), without the opportunity presented by the top ratings.

And by the way, you are free to ignore the top ratings anyway, Nobody is forcing you to look at them or use them.

 

[ptitSeb]

still, I'll be interested in a simulation of the "thumbup" rating to see what it will give (it may be too close to most downloaded, I agree with that).

I think, for the "discovery" part, that rating is not the best choice. Something based on update/download/comment activity would be better.

(Ignore rating is not always easy)

 

[_jr_]

Ekianjo, I'm not sure why you insist of arguing the same points again. There have been proposals how to navigate the repo that work for all PNDs as good as the rating (e.g. search by category and look at comments and download numbers) and you alread agree that rating mostly works for the top 50. To discover new releases I'd say the list of recent uploads is already more than adequate. And because new releases by definition don't have enough ratings to show in the right list I don't see how this should be an argument forkeeping the rating system. If you are counting on the first three (or is it five?) ratings to be 5 star, which would put the PND temporarily at the top of the rated list, then I agree that that is a nice side effect of the brokenness of the rating system and one of the reasons why I previously argued for the current system. But if you look at the PNDs that are now responsible for the current episode of this discussion it becomes clear that while these PNDs very much deserve to be highlighted as noteworthy, they can never move to the top of the rating list, just because the first rating was negative. That's exactly why the rating system is broken.


This 'repo spotlight' feature is actually worthwile to have, but independent of the rating discussion. Actually Pandora Live - thanks to your hard work! - is already doing quite a good job of highlighting noteworthy PNDs. How about adding a spotlight section to the repo front page with recently reviewed PNDs and a link to the review (e.g. on Pandora Live)? Adding reviews probably must be restricted to avoid spam, but it might still be possible to have this relatively cheap in terms of administrative overhead and might even increase the number of reviews.


edit: made it clear, I'm talking to ekianjo

 

[rygD]

Rygd you cant rely on comments for discovery since there is no way to filter pnd by comments. So its not helping to find your way in 1000+ pnds at all. Comments cannot be averaged, cannot be summarized and there do not help for discoery.

You check a category related to what you are interested in, or do a specific search, then read the comments.  I don't understand how useless ratings are going to be more helpful on their own than looking at a category or doing a search.  If all you are looking for is something that is highly rated, then you are correct, but if you are looking for something less general then a rating alone will not help with discovery.

Edit:  It seems _jr_ pretty much said the same thing earlier, but more clearly.

 

[rygD]

Another of my stupid ideas: Limit rating to three choices, "Positive", "Neutral", and "Negative". You can still present the results with five stars, or a comment that says something like "This is regarded as very bad based on 1 rater" or "Neutral, according to 2 ratings". This really changes very little, and requires more effort than it is worth, but maybe it can give others ideas.

 The problem is not about the scale of ratings whether you make it 3 scale or binary. That won't change the issue that RATINGS are QUANTITATIVE by nature and therefore need a certain number of answers to be meaningful (which is the opposite of comments which are qualitative by nature).

The point that notaz was making earlier that "ratings do not mean anything because we don't know what people mean by it" is also moot, because even if people rate things differently, as long as enough people vote the sample of ratings becomes comparable. That's how statistics work, and I feel very strange that I have to explain this kind of things here to people who should probably know about stats in the first place. That's why asking 200 people in the street about their opinion can more or less represent the national opinion, as long as the sample is relevant.

Now, coming back on potential solutions, since ratings are qualitative and only meaningful when a certain threshold is reached, I would propose to make ratings "invisible" to the public (except the developer) until the threshold is reached. And a message such as "more votes are needed" should be displayed to encourage voting for those PNDs which do not have votes, instead of showing blank stars (which makes it look that it has ZERO stars, even though that's not its intention) or light blue stars based on a single rating, which is by nature meaningless because it's below the threshold.

I think you missed my last sentence that you quoted.  My idea is pretty useless, and I only shared it in case it might give someone a better idea.
I don't think what notaz and others were getting at is moot at all.  When you have very few ratings the method used for ratings is VERY important.  If, as you state, given a large enough data pool, it won't matter as much if there are variations in interpretation of the rating scale, this further shows the lack of value of the ratings, because we don't have enough people rating.  I really don't see how your suggestion to hide ratings until there are enough is any different than getting rid of them in general.  They will still not be used for the less rated items, and "Most Downloaded" is probably a better measurement of popularity and quality in many cases than "Top Rated".

Edit:  I think I am going to walk away (because I can't get caught up), as it seems this was addressed as well, but I will respond to that here in a minute.

 

[rygD]

ignoring the ratings is currently the right thing to do beyond the most rated PNDs

 
I disagree. The top rated list shows a different list than the top downloaded one (which almost never changes) and therefore helps the discovery of new PNDs. Look at NPRquake which is currently first, it could never have been showcased before if you were sorting by number of downloads、and none of the comments would have helped people to have a look at it since there's not much contents in them either. So, explain me how one would have found about NPRQuake (which is an excellent port of ptitSeb, that I want to thank here again by the way, even though I disagree with his opinion on removing ratings), without the opportunity presented by the top ratings.


And by the way, you are free to ignore the top ratings anyway, Nobody is forcing you to look at them or use them.

The fact that a single vote can get something to top rated is actually a problem.  It shows that a single rater can influence what is listed there.  I can go find something that has no ratings and give it five stars and it shows up there.  Someone else can just as easily keep it from showing up there by rating it one star.  Check out NPRquake as an example again.  I changed my rating.  If you look through the "Latest" list you can see what is new or recently updated, which still gets "showcased".  Ratings are meaningless without more raters.

Edit:  Screw it, _jr_ said most of the same things I wanted to.

 

[milkshake]

The fact that a single vote can get something to top rated is actually a problem. It shows that a single rater can influence what is listed there. I can go find something that has no ratings and give it five stars and it shows up there. Someone else can just as easily keep it from showing up there by rating it one star.

this is not true, an app with 1 rating will not affect the top ratings they need at least 4, it's true it will have a light blue star rating but all light blue ratings appear below dark blue ratings.

 

[rygD]

The fact that a single vote can get something to top rated is actually a problem. It shows that a single rater can influence what is listed there. I can go find something that has no ratings and give it five stars and it shows up there. Someone else can just as easily keep it from showing up there by rating it one star.

this is not true, an app with 1 rating will not affect the top ratings they need at least 4, it's true it will have a light blue star rating but all light blue ratings appear below dark blue ratings.

Ok, that is cool, and a good choice.  It doesn't change the fact that in my test I was able to knock #1 on the "Top Rated" off by changing my rate.

I thought dark blue was showing the ones I rated for.

 

[b_o_b]

Double thumbs up for PtitSeb's thumbs up method! His suggestion seems to be abiding by the KISS principle, good news for Milkshake

One thumb up : it works I appreciate the effort

Two thumbs up: great  

Comment if you like to extend the appreciation or have technical issues.