Repo rating system: 5 stars or binary ratings?

[ptitSeb]

But trust me, these 1 stars votes cut the motivation of any coder around rather low...

I agree with that.

 

[ingoreis]

But trust me, these 1 stars votes cut the motivation of any coder around rather low...

I agree with that.

I understand what you mean,happend me with Dungeon Keeper GOG PND too.

Edit: oh o see i have 7 Votes on it?

Thank you,thought that one CCF voted Star was there only :rolleyes:

 

[WizardStan]

The answer for me is simple.

Require that a comment be left if you rate it. More than likely, a 1-star rating will be paired with a negative comment. Then on the PNDs main page, show only the 4/5 star comments with an option to show the 1-star ratings. That way, the negative "can't be arsed" comments from CCF will be hidden unless the user wants to see them.

D.

That doesn't take away the fact that they're still there and causing developers problems.The way I see it, your proposal is just another form of censorship (yes, it is. The comment may still be there but it is mostly invisible; any form of suppression is censorship, hiding things (even without actually deleting them) definitely falls under the definition of censorship) and if people are generally okay with one form of censorship at an arbitrarily drawn line then why don't we consider just moving that line?

Y'all are thinking like programmers, there's a specific problem and you're trying to find a general solution to that problem, but unless misused 1 star ratings becomes rampant I think you're over-engineering.

The solution is extremely simple: we can all agree that CCFs comments are pure noise, can't we? Even the good ratings he's leaving are basically just "I didn't have to use my brain to make it work so that makes me happy 5/5". Useless, yeah? He is the problem. "Hampering free speech!" but that's what you're trying to do with all this debate, find a general solution to hampering useless comments. "Slippery slope!" but we're basically all agreeing that in this one case the rating system has been abused, and I have 100% faith in Milkshake that he won't suddenly go on a deletion spree just because we've agreed that this one case needs to go.

Just delete his comments, his ratings, and ban him from making any more. Problem solved. If it happens again with a different user we debate it again. If it happens a third time then we can figure something else out.

 

[Rockthesmurf]

I don't know if my post about outliers didn't pick up any traction because I didn't explain it well, or whether it was because it wasn't seen as useful. If it is the latter that is fair enough, if it is the former I thought I should explain a little more.

So with a data set like:

5* - 5 votes

1* - 1 vote

A basic average would come back as:

( 5 + 5 + 5 + 5 + 5 + 1 ) / 6 = 4.33

If we identify the outliers, using an algorithm like this: http://www.miniwebtool.com/outlier-calculator/ it identifies that the outlier is 1, so instead we'd do:

( 5 + 5 + 5 + 5 + 5 ) / 5 = 5.00

So that eliminates the 1 which seems to not be in line with all other ratings. But what happens if another user votes 1, so we have:

5* - 5 votes

1* - 2 votes

Well, the algorithm above then indicates there are no outliers, as it is deemed that 1 is no longer an outlier as it has some company, so the rating:

( 5 + 5 + 5 + 5 + 5 + 1 + 1 ) / 7 = 3.86

So at this point the rating has been pulled down, but the idea is that the 1* is no longer a nonsense review, and instead should taken into consideration (like there is possibly a genuine issue with the software at this point).

You can plug other data into the algorithm to see what it thinks are outliers, so for example:

5* - 1 vote

2* - 2 votes

1* - 3 votes

In this case it identifies 5* as the outlier; perhaps a bad application has been given a positive rating by the developer. If we wanted to be 'nice' we could allow outliers which are higher than the average, but we could also eliminate them as above.

With the algorithm linked to, there is of course going to be limits to what can be achieved, if there is just a single rating, from a 'problematic' user, and the rating is 1*, no algorithm is going to be able to fix that (unless the algorithm takes into account the username of each rating..!). Of course, if there is a single 1* and a single 5* there is no way an algorithm can work out if any of them is invalid.

Another example:

5* - 10 votes

4* - 6 votes

3* - 3 votes

1* - 2 votes

Current average : 4.05

With outliers removed (1*'d identifier as outliers) : 4.37

So, that hopefully explains it a little better, my initial suggestion was this could be the default rating displayed, and an option could be there to see the average with outliers (obviously worded/labelled in a nicer way). The same outliers approach could be used to by default hide comments, and again have an option to also include these comments.

If people don't like this idea that is totally cool; it was just an idea - I only made this post to try and make sure it is obvious what I was going on about.

 

[Rockthesmurf]

I also agree about blacklisting CCF, however I assume if he goes to the effort to leave comments, with no real purpose, he'll probably just create another account and do the same, so it becomes a never ending problem which gets more complicate if you start wanting to verify users and becomes a maintenance hassle etc.

I quite like the 'was this review helpful' option some sites have, so if we made it so you can leave 3/4/5 star ratings without a comment, but 1/2 star requires a comment (comment why you are leaving a bad review). People can then say was a review helpful (yes/no) if they want, and then if a review has a bunch of 'not helpful' votes, it can be removed/hidden, and it's associated rating could also be removed/hidden. This free thinking is not really taking into account implementation and what it means for Milkshake, just throwing ideas around.

 

[ptitSeb]

I am against blacklisting CCF. I appreciate the fact he leaves comment, even if I do not agree to some of them, he takes time to discribes his feeling, he deserves credits for that.

My issue is that the rating system permit "negative rating". And that negative rating doesn't help the end user (like on a big repo with paid apps), but only hurts the dev./porter.

That is why I proposed a simple "thumb up" kind of rating (you like it/ recommand it, or you don't care). Your proposal Steven Craft is also probably acceptable, but it doesn't solve the issue with a single 1-star comment (because it doesn't adress the basic fact than everything not 5 stars can be seen as negative).

 

[WizardStan]

My issue is that the rating system permit "negative rating"

Sometimes you need a negative rating, though."Instead of chair, package contained bobcat"

If there's a PND that is genuinely bad for reasons it may not be consistently bad for everyone, there may be people that are like "ok, it crashed hard and made my nose bleed, but the 2 minutes before that happened was pretty fun. Thumbs up". No way to downvote. You'd expect such a bad app to have fewer votes, but if something has fewer votes is there a way to tell whether that is because it is bad or because it just isn't well known?

Or more real world example, suppose flashy lights gives someone a seizure? "1/5, gave me a seizure" and that may not be helpful to you but someone reading through the ratings might be interested in the 1 stars, recognize that they are also prone to seizures, and pass over this potential hazard.

Or less hypothetical, suppose it's just a bad and uninteresting game: again, a few people will probably vote for it, but how do you know the difference between a bad game with a few pity likes vs a good game that hasn't been advertised so well?

Negative ratings are important.

I appreciate the fact he leaves comment, even if I do not agree to some of them, he takes time to discribes his feeling, he deserves credits for that.

I understand why you would say that, but his comments are basically noise, it is literally interfering with other comments and ratings. If he were sitting in a movie theatre describing his feelings in detail about the movie he's watching in a way that was detrimental to the other viewers he would be asked (and eventually forced) to leave; I don't see much difference here.

 

[Granitehead]

Idea/suggestion/idle thoughts on the subject:

I would make multiple categories, at least "port quality" and "app quality", app quality could be further divided. Have some exact definitions for each number of stars provided and place them prominently both when voting and when browsing apps (like 1 star port quality "Doesn't launch after doing my best to make sure it wasn't my fault, i.e. reading all the instructions, redownloading the app, etc.").

Don't require a user to vote in all categories.

Have some weighting algorithm which considers both outliers and age of the ratings. Age of ratings would ideally not consider linear time, but numbers of app and os updates since the rating.

Have a dialog pop up when a .pnd is quit which has been mounted for a certain amount of time in total (store time in appdata). The dialog should have an easy way to rate the .pnd (even without internet access, auto-submit when there is access again) and a way to disable this functionality permanently, not just the dialog but also the logging of mounted time.

Have an entry in first boot wizard for this, making it possible to disable it before it first triggers. Another entry in system settings to (re)enable/disable it.

 

[rygD]

outliers

Isn't this most Pandora owners?

I dislike the idea of banning people because others don't like their method of rating.  I won't go into detail about why I feel this way, and all the problems I see with it that haven't already been mentioned, but it seems very wrong to me.  It seems worse than the method of censorship WizardStan mentioned.

I quite like the 'was this review helpful' option some sites have, so if we made it so you can leave 3/4/5 star ratings without a comment, but 1/2 star requires a comment (comment why you are leaving a bad review). People can then say was a review helpful (yes/no) if they want, and then if a review has a bunch of 'not helpful' votes, it can be removed/hidden, and it's associated rating could also be removed/hidden. This free thinking is not really taking into account implementation and what it means for Milkshake, just throwing ideas around.

This sounds like a much better idea.  So do some of the others.  This one stands out because everything is still available and the only things that aren't counted in the rating, or shown by default, is that which others (unanimously?) agree is not relevant.

If he were sitting in a movie theatre describing his feelings in detail about the movie he's watching in a way that was detrimental to the other viewers he would be asked (and eventually forced) to leave; I don't see much difference here.

There are major differences.  His ratings/comments are more like him standing outside a theatre that lets people watch films for free and yelling "don't watch that one, it sucks" to everyone who tries to get a ticket for something he didn't like.

 

[klapse]

Despite the volumes written on the subject I don't have any problem with 1-5 star ratings. They are equivalent to -2-+2 as far as I can tell. Current system has few ratings but that is OK. I download applications based on forum threads anyway, and not their rating.

 

[Granitehead]

They are equivalent to -2-+2 as far as I can tell.

That they are definitely not. I usually treat them more or less like this. It makes sense. But that's not how they are usually used by most people. See that xkcd strip linked earlier in this thread.
Edit: I'm strictly not saying anything else, just to make this clear. NO implicit meaning. I'm not pro or contra 1-5 stars.

 

[ZXDunny]

The answer for me is simple.

Require that a comment be left if you rate it. More than likely, a 1-star rating will be paired with a negative comment. Then on the PNDs main page, show only the 4/5 star comments with an option to show the 1-star ratings. That way, the negative "can't be arsed" comments from CCF will be hidden unless the user wants to see them.

D.

That doesn't take away the fact that they're still there and causing developers problems.The way I see it, your proposal is just another form of censorship (yes, it is. The comment may still be there but it is mostly invisible; any form of suppression is censorship, hiding things (even without actually deleting them) definitely falls under the definition of censorship) and if people are generally okay with one form of censorship at an arbitrarily drawn line then why don't we consider just moving that line?

Ok, then why not just list the number or 1s, 2s, 3s, 4s an 5 Star ratings, and allow the user to click those ratings to see the comments?

D.

 

[ekianjo]

The answer for me is simple.


Require that a comment be left if you rate it. More than likely, a 1-star rating will be paired with a negative comment. Then on the PNDs main page, show only the 4/5 star comments with an option to show the 1-star ratings. That way, the negative "can't be arsed" comments from CCF will be hidden unless the user wants to see them.


D.

That doesn't take away the fact that they're still there and causing developers problems.
The way I see it, your proposal is just another form of censorship (yes, it is. The comment may still be there but it is mostly invisible; any form of suppression is censorship, hiding things (even without actually deleting them) definitely falls under the definition of censorship) and if people are generally okay with one form of censorship at an arbitrarily drawn line then why don't we consider just moving that line?

Ok, then why not just list the number or 1s, 2s, 3s, 4s an 5 Star ratings, and allow the user to click those ratings to see the comments?


D.

Just do like on Amazon and show the distribution of 1 to 5 ratings so one can see if the overall ratings are positive or not, and if the only negative one comes from an ass who puts a 1 for no reason. (paired with comment, of course - again just like on Amazon). You can also have an option "I found this comment helpful" or not, so stupid comments would get relegated to the bottom of the page instead of the top. 

 

[rygD]

They are equivalent to -2-+2 as far as I can tell.

 
I guess, in some sense, this is how I look at it too.

I download applications based on forum threads anyway, and not their rating.

I think most involved here do this.  That is why having the ratings is a problem.  For those involved with the community the ratings have very little positive use.  Since there aren't very many ratings it looks pretty bleak to the rest of the world if there are few ratings and 1 or 2 out of the 5 for some software is 1 star.  If you look at it that way you can really understand why those who are populating the repo feel strongly about it.  It tells the rest of the world that either they can't make a quality product.

It seems like several of us are in favor of something similar to what Amazon has, so maybe that is what should be considered.  I still think that splitting up the rating between multiple categories, and providing guidelines on how to vote, will be helpful (it doesn't mean a particular rater won't ignore these efforts).  I would like to hear what the uploaders have to say about Amazon style ratings.

 

[WizardStan]

His ratings/comments are more like him standing outside a theatre that lets people watch films for free and yelling "don't watch that one, it sucks" to everyone who tries to get a ticket for something he didn't like

Close, except that it's not just him standing there, there are many other people also giving tips on movies to see, he just happens to be very loud about his opinion, sometimes louder than anyone else: his opinions, frequently based on false information, are literally drowning out those with true information trying to give positive reviews. If we further the analogy that the entire theatre is on private property (as is the case for a private web server) then my point would still stand anyway: in a real world scenario this loud and obnoxious guy would inevitably be asked and eventually forced to leave.

 

[rygD]

If we further the analogy that the entire theatre is on private property (as is the case for a private web server) then my point would still stand anyway

You are correct, and I forgot about this.  Still, ptitSeb is opposed to it for valid reasons, and he is the one who brought this discussion back up.

 

[milkshake]

Just do like on Amazon and show the distribution of 1 to 5 ratings so one can see if the overall ratings are positive or not.

I do like this idea.

 

[Granitehead]

His ratings/comments are more like him standing outside a theatre that lets people watch films for free and yelling "don't watch that one, it sucks" to everyone who tries to get a ticket for something he didn't like

Close, except that it's not just him standing there, there are many other people also giving tips on movies to see, he just happens to be very loud about his opinion, sometimes louder than anyone else: his opinions, frequently based on false information, are literally drowning out those with true information trying to give positive reviews. If we further the analogy that the entire theatre is on private property (as is the case for a private web server) then my point would still stand anyway: in a real world scenario this loud and obnoxious guy would inevitably be asked and eventually forced to leave.

I think it's more like there is a board in front of the cinema where you can put little notes about how much you like a movie (you are /supposed/ to put your real opinion there). You need to give the movie a rating from 1-5, but you can decide freely how to give those points, there are no official guidelines or anything. Most people ignore this board and just go discuss the movie with their friends in the restaurant next to the cinema and also ask there when they wonder how good a movie is.Now there is one person who dislikes lots of movies and puts lots of notes on that board. Because no one else is really using it, when someone /does/ once in a while look at that board, it seems there are lots of bad movies.

Solution: Remove board, or add clear instructions how to vote, so a vote can actually be /wrong/ and thus removed (right now it really /can't/ be wrong in that sense) and serial wrong-voters banned, or get more people to vote, or ...

 

[thatgui]

Just do like on Amazon and show the distribution of 1 to 5 ratings so one can see if the overall ratings are positive or not.

I do like this idea.

But it does not solve the problem that led to ptitseb posting his rant here, if there is only one bad rating (from ccf) its still shown. Which doesn not need to be a bad thing after all, there may be cases where a one star rating is justified (but that should be explained in a comment).How about showing the rating only if more than one rating has been made ? This way one single rating can't pull the entry down, but it still adds to the overall rating. Of course he could just create a second account and rate it again, but maybe he doesn't bother that much about it, as his comment is still shown ?

@Milkshake

While its certainly makes no sense to change the rating system completly "on call", as this may worsen the current situation even more - I guess people will be even less motivated to rate something, if they can't be sure they would have to rate something again as soon as the next "iteration" of the rating system gets live.

Maybe you could be persuaded to change how download counts are handled ?

To me, it looks like the download counter is of no real value as you can't tell wether the download counter is high because its very popular or because the entry gets updated rather often, or both.

Maybe you could either:

- show only the number of downloads for the latest version

- store the download count per update and only show and average of this

Another column could be added to the the archive view that shows the number of downloads per version, and maybe the sum at the bottom.

Maybe a more "cleaner" download figure could even be an additional input parameter in the next iteration of the rating system ?

 

[Neelix]

Just do like on Amazon and show the distribution of 1 to 5 ratings so one can see if the overall ratings are positive or not.

I do like this idea.

But it does not solve the problem that led to ptitseb posting his rant here, if there is only one bad rating (from ccf) its still shown. Which doesn not need to be a bad thing after all, there may be cases where a one star rating is justified (but that should be explained in a comment).How about showing the rating only if more than one rating has been made ? This way one single rating can't pull the entry down, but it still adds to the overall rating. Of course he could just create a second account and rate it again, but maybe he doesn't bother that much about it, as his comment is still shown ?

The problem isn't that bad ratings are shown, it's that there is no context for bad ratings. The Amazon histograms show exactly how many people have voted at each level, so the end user can easily see the difference between a program with a genuinely bad rating and one with an insignificant rating which happens to be be bad but may not be representative. Of course to be really effective the histogram would need to be viewable in PNDManager too.

That said It's not the histograms alone that make the Amazon system so useful. It also ties each rating to a comment and links to those comments from the histogram. If you see that someone has given it a one star rating, being able to quickly review the one star ratings to see the justification can give you a good idea if the reason for the rating is something you actually care about.

- Neelix