Repo rating system: 5 stars or binary ratings?

[thatgui]

The problem isn't that bad ratings are shown, it's that there is no context for bad ratings. The Amazon histograms show exactly how many people have voted at each level, so the end user can easily see the difference between a program with a genuinely bad rating and one with an insignificant rating which happens to be be bad but may not be representative. Of course to be really effective the histogram would need to be viewable in PNDManager too.

That said It's not the histograms alone that make the Amazon system so useful. It also ties each rating to a comment and links to those comments from the histogram. If you see that someone has given it a one star rating, being able to quickly review the one star ratings to see the justification can give you a good idea if the reason for the rating is something you actually care about.

- Neelix

I know that, but as said it doesn't not help anything if there is just one rating at all

 

[ekianjo]

The Amazon histograms show exactly how many people have voted at each level, so the end user can easily see the difference between a program with a genuinely bad rating and one with an insignificant rating which happens to be be bad but may not be representative

and when you click on the Amazon histograms you get access to the rating+the message associating to the rating. You can clearly see what was the reason for the rating.

IMHO Amazon ratings work extremely well in this way. i've never bought a product that had more than 4 stars and that was obviously a shitty item. As long as enough people review a certain item after purchase/usage, the rating system is very, very fair.  

 

[_jr_]

Nobody denies that with enough ratings and comments everything would be much better, but we still have no valid reason to believe we will ever get even 5 ratings for a lot of PNDs.


Having long discussions that lead nowhere is just giving the impression that having long useless discussions is more important than maintainers. I know, this is the internet, but I think the very few individuals in the community that actually do more than talking deserve better. Fearing change because it may demotivate comments and ratings is ridiculous, since there are about two accounts on the repo regularly commenting and rating anyway and one of them is part the problem. Any change would be better than this discussion.

 

[ptitSeb]

@_jr_ :I do agree, this thread is becomming more irritating than the 1 star voting itself!

Senless post like I have seen here should go in the "Absurdism corner"...

That beeing said, I was imaginating the change 5 Strars -> Thumb up voting, with the following rules:

1. You can vote for 1 thumb or double thumb (just 2 thumbs instead of 1) ,or you don't vote (0 thumb) => Only positive rating here.

2. For migration, use the following rules: 1-2 stars => nothing, 3-4 Stars: 1 thumb, 5 stars: 2 thumbs.

Nothing more, no link to comment, mandatory non sense or heavy censorship.

Still, the main main issue is the very small active base.

Right now, I estimates the user base (individual people that do download things) to be around 200~250 peoples, and the active base (people that do rate / comment on the repo) around 20.

Not much can be done here with thoses figures (the active base used to be much bigger, so my migration will lead to have "old" packages ont the top rated, not that it's wrong, has PCSX or Firefox are essentials packages).

So, because of the very small user base, I think doing nothing is fine (and I'll continue to do my best to ignore rating).

 

[Hồng Thất Công]

Or just take out the rating system altogether. After all, the stuff on the repo are the hard work of devs given to the community for free. If someone doesn't like the app/game, they can say so in the comment so devs can have a chance to respond. Who gives a shit about 1star, 2star, 3star, or whatever. I don't.

 

[WizardStan]

Even if you remove the stars, CCF's bogus comments are still there, him complaining about things that "don't work" and calling the developers out on releasing garbage or whatever wording he happens to use.

Now I could be completely misreading this, but it seems to me that ptitSeb is just as bothered by the negative comments as he is by the one stars. Maybe not just as, but still very bothered by them.

If you remove the ratings and only allow comments you still have the same problem of a single person leaving a negative comment to dissuade people from downloading and discourage the developer from continuing, it's just one click removed. And now you don't even have a way of sorting the comments, positive from negative. You can't just get rid of the star system and trust that comments will work out better.

 

[ptitSeb]

Even if you remove the stars, CCF's bogus comments are still there, him complaining about things that "don't work" and calling the developers out on releasing garbage or whatever wording he happens to use.


Now I could be completely misreading this, but it seems to me that ptitSeb is just as bothered by the negative comments as he is by the one stars. Maybe not just as, but still very bothered by them.


If you remove the ratings and only allow comments you still have the same problem of a single person leaving a negative comment to dissuade people from downloading and discourage the developer from continuing, it's just one click removed. And now you don't even have a way of sorting the comments, positive from negative. You can't just get rid of the star system and trust that comments will work out better.

I am less bothered by a comment than by a 1-star rating because I can answer to a comment. I can do nothing about a rating.

 

[Hồng Thất Công]

Even if you remove the stars, CCF's bogus comments are still there, him complaining about things that "don't work" and calling the developers out on releasing garbage or whatever wording he happens to use.


Now I could be completely misreading this, but it seems to me that ptitSeb is just as bothered by the negative comments as he is by the one stars. Maybe not just as, but still very bothered by them.


If you remove the ratings and only allow comments you still have the same problem of a single person leaving a negative comment to dissuade people from downloading and discourage the developer from continuing, it's just one click removed. And now you don't even have a way of sorting the comments, positive from negative. You can't just get rid of the star system and trust that comments will work out better.

I am less bothered by a comment than by a 1-star rating because I can answer to a comment. I can do nothing about a rating.

Exactly!  This is my point.  With a comment, devs can respond and defend for themselves because devs know exactly who give them bad comments.  With the rating, goofy people can just give an app 1 star and doesn't say a thing, and devs have no idea who that goof is and the reason why he does not like the app.  Look at the Android app store, I can't believe people gave DraStic less than 4.9 star because they didn't know where to put the bios file...And app store involves money...

 

[ekianjo]

Even if you remove the stars, CCF's bogus comments are still there, him complaining about things that "don't work" and calling the developers out on releasing garbage or whatever wording he happens to use.


Now I could be completely misreading this, but it seems to me that ptitSeb is just as bothered by the negative comments as he is by the one stars. Maybe not just as, but still very bothered by them.


If you remove the ratings and only allow comments you still have the same problem of a single person leaving a negative comment to dissuade people from downloading and discourage the developer from continuing, it's just one click removed. And now you don't even have a way of sorting the comments, positive from negative. You can't just get rid of the star system and trust that comments will work out better.

I am less bothered by a comment than by a 1-star rating because I can answer to a comment. I can do nothing about a rating.

Exactly!  This is my point.  With a comment, devs can respond and defend for themselves because devs know exactly who give them bad comments.  With the rating, goofy people can just give an app 1 star and doesn't say a thing, and devs have no idea who that goof is and the reason why he does not like the app.  Look at the Android app store, I can't believe people gave DraStic less than 4.9 star because they didn't know where to put the bios file...And app store involves money...

The rating helps to sort out apps if you have not noticed.

 

[WizardStan]

The rating helps to sort out apps if you have not noticed

Exactly. You can't get a "what's good, what's bad" scale without both positive and negative ratings of some kind.

 

[_jr_]

But we are still waiting for users that actually use the ratings as such a criterion. Until now everybody stated that due to the very few ratings available they learnd to just ignore the rating.


I regularly look at the what's new list on the repo frontpage, I sometimes glance at the most downloaded list (just to notice that it hasn't changed) and almost never look at the third column. The same with PNDManager, I do all the work with the left listing (ordered by upload date) and never (except in my very early days as a Pandora owner) use the right list (ordered by rating). Though I admit to once or twice having downvoted a PND by Commander Beef when he was particularly annoying on the forum so I didn't have to see it at the top of the right list in PNDManager.


Effectively the rating display is just misleading newcomers, while it might be ok for the most rated PNDs on the repo it isn't a usable criterion for exploring it. Perhaps just increasing the 'valid rating' threshold to 50 votes or something and showing the app as unrated before that time in the repo (just like in PNDManager) would already do the trick? This should be a trivial change and single one star ratings are automatically irrelevant (i.e. no downvote by single individuals possible anymore).

 

[WizardStan]

But we are still waiting for users that actually use the ratings as such a criterion. Until now everybody stated that due to the very few ratings available they learnd to just ignore the rating

Ok, I am stating that I use the ratings to decide what to try next. There may be very few ratings but with limited time I can't just play everything and even minimal ratings are at least as good as any other arbitrary decision maker and typically better.

 

[ekianjo]

I am usibg ratings too.

 

[Rockthesmurf]

This is not really a proper suggestion, but just a musing, maybe it will be helpful in the long run. But I wonder if it would be interested in the rating someone makes isn't what actually gets displayed. What I mean by this, is if someone only ever ranks these 1* or 2*, then 2* can almost be seen as a 'perfect' rating, so a 2* rating from that person gets displayed (and used to average as if) it is a 5* rating. I don't really know exactly how the algorithm would work, but we have an issue where different people use the ratings differently (there is no real notion of what makes something 1* and what makes something 5*) however if we can at least detect the range of ratings a user is giving, perhaps we can determine how that interpret the rating system (some of the time at least). It would also means if some users (kindly) rate everything 5*, the system could detect that essentially that person isn't really effecting the overall vote either way. As I say, this isn't exactly a real suggestion, I don't really see it being implemented, but I thought some of the ideas are kind of interesting to think about. If you could combine that with a multiplier for how 'important' each rating is (where importance is worked out by how many people have marked a reviews comments as useful/not useful) you could potentially start to build up a single number 'rating' like we currently have, which begins to remove 'bad votes' and starts to account for that fact people vote in different ways.

 

[Granitehead]

This is not really a proper suggestion, but just a musing, maybe it will be helpful in the long run. But I wonder if it would be interested in the rating someone makes isn't what actually gets displayed. What I mean by this, is if someone only ever ranks these 1* or 2*, then 2* can almost be seen as a 'perfect' rating, so a 2* rating from that person gets displayed (and used to average as if) it is a 5* rating. I don't really know exactly how the algorithm would work, but we have an issue where different people use the ratings differently (there is no real notion of what makes something 1* and what makes something 5*) however if we can at least detect the range of ratings a user is giving, perhaps we can determine how that interpret the rating system (some of the time at least). It would also means if some users (kindly) rate everything 5*, the system could detect that essentially that person isn't really effecting the overall vote either way. As I say, this isn't exactly a real suggestion, I don't really see it being implemented, but I thought some of the ideas are kind of interesting to think about. If you could combine that with a multiplier for how 'important' each rating is (where importance is worked out by how many people have marked a reviews comments as useful/not useful) you could potentially start to build up a single number 'rating' like we currently have, which begins to remove 'bad votes' and starts to account for that fact people vote in different ways.

This could be done.I could probably come up with the maths to do this.

I don't really like that idea, we've discussed that some time ago when discussing about voting stuff in general, maybe for Pyra features or something along the lines.

 

[notaz]

FWIW I'd just kill that star rating, it does more harm than good. Nobody knows what it means, why have it at all?

In many cases this star rating ends up showing ridiculous things, like:

- do I (the user) have enough computer skills to copy a file?

  - No - 1 star

  - Yes - 5 stars (or maybe 4 or 3)

- do I like the author who produced the PND?

  - No - 1 star

  - Yes - 5 stars

(yes we have a member who openly admitted to vote like this)

- Is this game from a game genre I like?

  - No - 1-3 stars

  - Yes - 3-5 stars

Even with CCF lurking around the corner, the world (ok, ok, the repo) would be a better place without this undefined and counter-productive star rating system.

 

[ptitSeb]

Yeah!

 

[milkshake]

So no rating system at all?

 

[ptitSeb]

We can open a new thread to discuss about what to put instead of that column (something like "popular", but not based on rating), but yeah, no rating sounds good to me.

 

[b_o_b]

I like the thumbs up idea. Suggestion for converting: >2* = thumbs up