Rating software on repo.openpandora.org

[mcobit]

Use PndManager

 

[diligentcircle]

What's the point of having a like / dislike - System ?


Maybe I don't get it (or I refuse to), but what does this kind of rating say about an application ? Except the obvious of course, that there are x people that like it and that there are y people that don't like it. There is no in between, just the two extremes. What about the people that though: it was well made/well thought/a nice idea/looks promsing but is just not my cup of tea ?

If there isn't some sort of simple anonymous rating system, the only appreciation a developer will be able to see is that of people who actually care enough to comment. Having a likes/dislikes system also helps notify people that something is excellent or crap fairly accurately. Not perfect, but I think it's better than 5-star ratings, as I explained in my blog post.

I also didn't understand how the manipulation of a star rating system should work. They got one vote to give, this may influence the overall vote if only a small number of people voted, but the bigger the count of voters gets the more a single vote gets averaged out - maybe someone could explan it ?

That's a huge problem, actually. A really popular app is going to get a lot more votes than a really unpopular app, so unpopular apps are going to be more successfully manipulated by those who do manipulate. This puts less well-known apps at a severe disadvantage, and it's actually what I'm most concerned about. Even with popular apps, though, it can present a problem (or at least eliminate the 5-star system's strengths) if, as I think, rating manipulation is a common practice (e.g. 20% or more people do it); not as big of a problem, but something that users are split on whether it should be 4 or 5 stars would likely end up closer to 4 stars rather than the more fair 4.5 stars (since the manipulators at the 4-star side can do it more effectively by voting 1 star, where as it's impossible to manipulate it closer to 5 stars).

 

[thatgui]

If there isn't some sort of simple anonymous rating system, the only appreciation a developer will be able to see is that of people who actually care enough to comment. Having a likes/dislikes system also helps notify people that something is excellent or crap fairly accurately.

Thats exactly what I'm getting there, its either excellent or crap nothing in between - which makes this system inferior for me.

That's a huge problem, actually. A really popular app is going to get a lot more votes than a really unpopular app, so unpopular apps are going to be more successfully manipulated by those who do manipulate. This puts less well-known apps at a severe disadvantage, and it's actually what I'm most concerned about. Even with popular apps, though, it can present a problem (or at least eliminate the 5-star system's strengths) if, as I think, rating manipulation is a common practice (e.g. 20% or more people do it); not as big of a problem, but something that users are split on whether it should be 4 or 5 stars would likely end up closer to 4 stars rather than the more fair 4.5 stars (since the manipulators at the 4-star side can do it more effectively by voting 1 star, where as it's impossible to manipulate it closer to 5 stars).

Sorry, but I still don't get it. What you wrote is true (especially the fact that its harder to vote something up than something down), but this is only applicable if the votecount is relativly low. The one vote problem (regarding manipulation) still exists wether the application is popular or not.


Is still think the afformentioned expansion of the five star system would be sufficient

 

[diligentcircle]

Thats exactly what I'm getting there, its either excellent or crap nothing in between - which makes this system inferior for me.

I think you're missing my point. People, as a whole, do not use the 5-star rating system properly. Google saw this very clearly in YouTube videos back when YouTube had 5-star ratings; people pretty much either rated things 1 star or 5 stars, and only a small minority of people actually used the middle ratings. My thought is further: I think a significant number of people use the 5-star rating system not to rate what they think it should be rated, but what will bring the average rating closer to what they think it should be rated.


Did you read my blog post that I mentioned earlier? It's here: http://al.ly/Q-C

Is still think the afformentioned expansion of the five star system would be sufficient

Changing the anonymous ratings to reviews would be a much more drastic change than changing them to likes/dislikes. My conjecture is that it would simply result in less people voting, which wouldn't be a problem if there were a ton of people on the Repo, but the fact is there are very few of us, so all this can result in (if my conjecture is correct) is a situation where tons of apps just aren't rated at all, because not enough people are comfortable spending time writing a review.


I honestly have a problem with requiring people to attach a name to simple ratings, too, because all I can see that doing is skewing the ratings to ones that are higher than they would otherwise be. Anonymous "likes only" would hardly be worse.


Making votes non-anonymous attempts to solve the problem of a couple of jerks and bad voters by deterring them, but the problem I see in this is it also deters negative feedback when it's deserved. Changing it to a likes/dislikes system, on the other hand, attempts to solve the problem by deterring manipulation of ratings while at the same time allowing viewers to easily see how many people have voted. Seeing a new app that has one dislike is not like seeing a new app that has an average rating of 1 star; in the former case, you know that the one dislike isn't necessarily particularly meaningful, but in the latter case, it looks like the overall consensus is that the app sucks, so you're going to be more likely to take it seriously, not knowing that only one person voted at all.

 

[Stephane Hockenhull]

+1


I think the youtube style like/dislike idea is a good one.

 

[Binky]

Maybe have a 'neither like nor dislike' too


Given the small number of votes stuff on the repo gets, you could end up with people not having a strong opinion and so not voting - when It would be worth knowing that someone downloaded it and thought it was OK.

 

[thatgui]

I think you're missing my point. People, as a whole, do not use the 5-star rating system properly. Google saw this very clearly in YouTube videos back when YouTube had 5-star ratings; people pretty much either rated things 1 star or 5 stars, and only a small minority of people actually used the middle ratings. My thought is further: I think a significant number of people use the 5-star rating system not to rate what they think it should be rated, but what will bring the average rating closer to what they think it should be rated.


Did you read my blog post that I mentioned earlier? It's here: http://al.ly/Q-C

I did read your post and I did got your point, and I still think this is the inferior system. A lot of information -> the "inbetween" is just lost, for gaining a benefit which significance is still in doubt. Is the youtube audience really comparable to the Pandora Community ? How high is the percentage that tries to manipulate the rating in our case - a thing that still does not make any common sense to me. Maybe there is a percentage of people who tries to manipulate, but how long will they do this ? Human brains usually don't base minor decisions on long term perspectives, but on late memories. So even if these people try to manipulate, their manipulation won't have any real effect - there may be an all over effect on the rating, but there is none that they can really experience - except the cases where only 5 to 10 people have voted, but who cares about them anyway (in a youtube dimension). So the likely run is, that they will do this maybe 10 - 100 time, experiencing that this won't have any tangible effects on them, and then change their "strategy" (by commenting, etc.). This is a problem if your running a site with maybe hundreds or thousand of new users every day, but not here.


Or take Amazon for example, they are running their five star system (more or less) successfull for a long time now - I read your blog post, so I know that you are differentiating between rating and review, but in a case like Amazon , reviews can be boiled down to ratings as a lot of people just put in a few words, usually you do not see them as they are voted down, and are placed at the end of the list. And I can think of a long list with local websites on it, that are using a similiar system but without the necessity of a review, which are up and running successfully for years now.

Changing the anonymous ratings to reviews would be a much more drastic change than changing them to likes/dislikes. My conjecture is that it would simply result in less people voting, which wouldn't be a problem if there were a ton of people on the Repo, but the fact is there are very few of us, so all this can result in (if my conjecture is correct) is a situation where tons of apps just aren't rated at all, because not enough people are comfortable spending time writing a review.


I honestly have a problem with requiring people to attach a name to simple ratings, too, because all I can see that doing is skewing the ratings to ones that are higher than they would otherwise be. Anonymous "likes only" would hardly be worse.


Making votes non-anonymous attempts to solve the problem of a couple of jerks and bad voters by deterring them, but the problem I see in this is it also deters negative feedback when it's deserved. Changing it to a likes/dislikes system, on the other hand, attempts to solve the problem by deterring manipulation of ratings while at the same time allowing viewers to easily see how many people have voted. Seeing a new app that has one dislike is not like seeing a new app that has an average rating of 1 star; in the former case, you know that the one dislike isn't necessarily particularly meaningful, but in the latter case, it looks like the overall consensus is that the app sucks, so you're going to be more likely to take it seriously, not knowing that only one person voted at all.

Maybe I should have made a clearer statement regarding this in my last post: I also don't think that forcing reviews or that attaching a name to a rating will make things better (on the contrary), I would simply break the overall rating down to the individual values (and figures).


Having values besides the stars won't give the impression that the app from your example sucks, the viewer would see that there are a also people that gave four or five stars. So he can evaluate on a broader basis wether he wants to try it or not.

 

[diligentcircle]

Is the youtube audience really comparable to the Pandora Community ? How high is the percentage that tries to manipulate the rating in our case - a thing that still does not make any common sense to me. Maybe there is a percentage of people who tries to manipulate, but how long will they do this ? Human brains usually don't base minor decisions on long term perspectives, but on late memories. So even if these people try to manipulate, their manipulation won't have any real effect - there may be an all over effect on the rating, but there is none that they can really experience - except the cases where only 5 to 10 people have voted, but who cares about them anyway (in a youtube dimension). So the likely run is, that they will do this maybe 10 - 100 time, experiencing that this won't have any tangible effects on them, and then change their "strategy" (by commenting, etc.). This is a problem if your running a site with maybe hundreds or thousand of new users every day, but not here.


Or take Amazon for example, they are running their five star system (more or less) successfull for a long time now - I read your blog post, so I know that you are differentiating between rating and review, but in a case like Amazon , reviews can be boiled down to ratings as a lot of people just put in a few words, usually you do not see them as they are voted down, and are placed at the end of the list. And I can think of a long list with local websites on it, that are using a similiar system but without the necessity of a review, which are up and running successfully for years now.

I'm honestly not quite sure what you're trying to say here. You seem to be talking about YouTube and saying it's not comparable to the Repo, but you're not saying what's different about the Repo. I don't see the difference, really; what matters is how many people vote on a particular video or app. In the grand scheme of things, YouTube is a lot bigger than the Repo, sure, and the videos that get thousands or millions of views probably have no problem dealing with manipulation of the 5-star rating system I describe, but a much less viewed video, such as most of mine, is a bit more comparable to the many apps on the Repo that only get a few (less than 10) votes.


As for Amazon, I haven't seen a whole lot of terrible reviews on Amazon. Most of them that I read tend to be perfectly good. This can be chalked down to the review rating system, as you mention; crappy ones get filtered to the bottom of the list. This says nothing about anonymous ratings, and more importantly, Amazon is a much bigger website than the Repo (quality reviews would be much harder to come by on a smaller site like the Repo, if this type of feature was implemented).

 

[ZXDunny]

FWIW, I'd prefer the "like" or "dislike" system as opposed to the current system. As things stand, stuff I like gets 5/5, and the stuff I don't like gets 1/5. I don't like to give ratings in the middle, because I really, really dislike the stuff I rate 1/5. I really love Hurrican, it's a great game so I'm rating it 1/5 because it runs at 640x480. Like/Dislike would be great. Anonymous or not would not deter me.


D.

 

[thatgui]

Is the youtube audience really comparable to the Pandora Community ? How high is the percentage that tries to manipulate the rating in our case - a thing that still does not make any common sense to me. Maybe there is a percentage of people who tries to manipulate, but how long will they do this ? Human brains usually don't base minor decisions on long term perspectives, but on late memories. So even if these people try to manipulate, their manipulation won't have any real effect - there may be an all over effect on the rating, but there is none that they can really experience - except the cases where only 5 to 10 people have voted, but who cares about them anyway (in a youtube dimension). So the likely run is, that they will do this maybe 10 - 100 time, experiencing that this won't have any tangible effects on them, and then change their "strategy" (by commenting, etc.). This is a problem if your running a site with maybe hundreds or thousand of new users every day, but not here.


Or take Amazon for example, they are running their five star system (more or less) successfull for a long time now - I read your blog post, so I know that you are differentiating between rating and review, but in a case like Amazon , reviews can be boiled down to ratings as a lot of people just put in a few words, usually you do not see them as they are voted down, and are placed at the end of the list. And I can think of a long list with local websites on it, that are using a similiar system but without the necessity of a review, which are up and running successfully for years now.

I'm honestly not quite sure what you're trying to say here. You seem to be talking about YouTube and saying it's not comparable to the Repo, but you're not saying what's different about the Repo. I don't see the difference, really; what matters is how many people vote on a particular video or app. In the grand scheme of things, YouTube is a lot bigger than the Repo, sure, and the videos that get thousands or millions of views probably have no problem dealing with manipulation of the 5-star rating system I describe, but a much less viewed video, such as most of mine, is a bit more comparable to the many apps on the Repo that only get a few (less than 10) votes.


As for Amazon, I haven't seen a whole lot of terrible reviews on Amazon. Most of them that I read tend to be perfectly good. This can be chalked down to the review rating system, as you mention; crappy ones get filtered to the bottom of the list. This says nothing about anonymous ratings, and more importantly, Amazon is a much bigger website than the Repo (quality reviews would be much harder to come by on a smaller site like the Repo, if this type of feature was implemented).

I wrote a long post, but it got lost because I accidentially hit the back button on my keyboard. Long story short:


a ) Is the youtube user base comparable to the repo userbase => No, so there is no reason to believe that the youtube like/dislike system automatically is applicable to your "needs".


b ) Amazon was just another try to cement my argument, that star rating systems are superior to like / dislike systems in many (but not all) cases. And that the amazon review system is basically a rating system with "text topping"


Having a System like this:

Overall Rating SSS

SSSSS   by 1 User

SSSS	 by 2 Users

SSS	   by 3 Users

SS		 by 0 User

S		   by 1 User

S should represent a star.


In this example you can clearly see that altough the overall rating is average there was just one user that voted the application down and at least three users that gave it an average or above rating.

 

[relliker]

Use PndManager

"Dang!" How did I miss this? Thanks! I guess I was too preoccupied with my lame rebirth entry to try everything. Still, for people who won't receive your suggestion or those who haven't yet discovered pndmanager, I think adding the description to the built-in one would still be worthwhile


EDIT: Better still, just replace the built-in with this!


EDIT2: "Jeebuzz!" How did I miss this (again). Managing PNDs is now an ultra-cool breeze


More on topic.....I agree with a Like/Neutral/Dislike rating (i.e. just 3 choices) system. A lot of likes = Definitely worth it, a lot of dislikes = Don't install unless you're a sadist, a lot of neutrals = try it and judge for yourself dude!

 

[mcobit]

And there is even a feature in the works, that may allow rating and commenting from clientapplications

 

[diligentcircle]

Having a System like this:

Overall Rating SSS

SSSSS   by 1 User

SSSS	 by 2 Users

SSS	   by 3 Users

SS		 by 0 User

S		   by 1 User

S should represent a star.

In this example you can clearly see that altough the overall rating is average there was just one user that voted the application down and at least three users that gave it an average or above rating.

The problem I have with this system is twofold: first off, it's a bit complicated, so therefore easier to misinterpret. Consider the following:

SSSSS   by 600 Users

SSSS	by 122 Users

SSS	 by 213 Users

SS	  by 389 Users

S	   by 298 Users

This could seem at a glance like an app that's really well-liked, when the actual reaction is extremely mixed (average rating about 3.08/5). Of course, this usually won't happen, mostly because people who think it should be 2/3/4 stars won't vote as much as people who think it should be 1 or 5 stars, but then, what's the point in the middle ratings?


The other problem I have with it is there's no good way to put this on e.g. the front page of the Repo. There's just too much information.

 

[Christoph.Krn]

Changing the anonymous ratings to reviews would [...] simply result in less people voting, which wouldn't be a problem if there were a ton of people on the Repo, but the fact is there are very few of us, [...]

In the very excellent book "The Design of Everyday Things", Donald A. Norman talks, amongst many other things, about human error. He argues that knowledge is in both the head and the world. Some pages of the book can be read for free on Amazon: http://www.amazon.com/exec/obidos/ASIN/0465067107/donnormanA


--


Why is this relevant? Because it offers a new view on the problem of there appearing to be few people on repo.openpandora.org who care to vote. How come that there's a group of people that does not vote? Are they all lazy? I don't think so. I think that in a software repository, people will only begin to vote on a large scale once it becomes more than a plain file vault, i.e. once it not only is a place that accumulates all software for easy access, but actually helps people to orient in all this information. In other words, the way I see it, people will not begin to rate software on repo.openpandora.org on a large scale as long as the rating system did not succeed in helping them orient when they needed to.


How could this chicken-or-egg dilemma be solved? Well, how does it work outside of software repositories, where there's information abound? There are many 'solutions', and one of them is 'social glue'. Currently, many people find interesting PNDs by communicating on boards.openpandora.org, on IRC, or somewhere else. If these communication channels were closed down, it appears likely that people would begin to comment and rate on repo.openpandora.org a lot more. So, this forum's function as a PND suggestion system appears to be very significant -- even though in this function, the forum is a giant, chaotic, overly complex system. Or maybe *because* this forum is is a chaotic system? If there is no way that information can be systematically sorted because there is no useful meta-information to sort by, then can a slightly less accurate sorting that is sufficiently systematic for its particular purpose be used to create 'order from chaos from order', and could this let arise useful meta-information that could then lead to 'fully' systematic sorting?


Here is quite a small suggestion: Every software has a 'Show on frontpage tomorrow' button which can be clicked by any logged-in user. Clicks on these buttons are counted, weighted, and somewhat randomized in order to come up with a number of potentially-interesting/obscure/little-known software entries to be featured on the front page the next day. (No need to tell me that this idea sounds insane. I already know it does!)

 

[thatgui]

The problem I have with this system is twofold: first off, it's a bit complicated, so therefore easier to misinterpret. Consider the following:

SSSSS   by 600 Users

SSSS	by 122 Users

SSS	 by 213 Users

SS	  by 389 Users

S	   by 298 Users

This could seem at a glance like an app that's really well-liked, when the actual reaction is extremely mixed (average rating about 3.08/5). Of course, this usually won't happen, mostly because people who think it should be 2/3/4 stars won't vote as much as people who think it should be 1 or 5 stars, but then, what's the point in the middle ratings?

Why should it be misleading ? You still got the overall/average rating as a first orientation. The point of the middle rating is,that you can deduce a trend: ignoring the five star ratings you have a decreasing number of votes each "star-step" down, indicating that the app isn't very good, but not crap.


Could this example happen in real live ? the apportionment seems a bit weird to me.

The other problem I have with it is there's no good way to put this on e.g. the front page of the Repo. There's just too much information.

- you could still show the average rating, and show the details on mouse over


- Show five stars and put the number of votes in the star (would probably lead to unreadable numbers)


- show five stars and instead of printing numbers,"fill" the stars with the color according to the percentage of the appropiate vote (imagine a fluid level) simple example: the app get 5 1star ratings and 5 3 star rating, this would lead to five stars where the leftmost star is half filled from bottom to top, the next star would be empty, the third star would also be half filled, and the following to stars would also be empty

 

[diligentcircle]

If you show the average rating, that hardly solves the problem, now does it? You can already see how many people voted if you really want to right now. The problem is not everyone is going to do that with every app, which means people are going to randomly not bother with apps with low average ratings, regardless of whether it is an actual consensus or just one guy. This puts apps that aren't extremely popular at a disadvantage.


Plus, you still have the voter manipulation as long as there is a single bar (or number) that increases or decreases based on what you vote. At least, that's my hypothesis.

- Show five stars and put the number of votes in the star (would probably lead to unreadable numbers)


- show five stars and instead of printing numbers,"fill" the stars with the color according to the percentage of the appropiate vote (imagine a fluid level) simple example: the app get 5 1star ratings and 5 3 star rating, this would lead to five stars where the leftmost star is half filled from bottom to top, the next star would be empty, the third star would also be half filled, and the following to stars would also be empty

I think both of these would most likely just confuse new viewers. Plus, the second one fails to account for how many people actually voted.

 

[ZXDunny]

[tl;dr]

I think I get what you're saying. We should have a number of eggs displayed per PND file, and using social glue we should stick feathers from the chickens to the eggs. Would more eggs be detrimental? or more feathers a positive thing?


D.

 

[kuru]

^ It's a great thing having you contribute.

 

[Christoph.Krn]

[...] using [...] glue we should stick feathers from the chickens to the eggs [...]

No, we're not producing chicken nuggets here.


Why the hostility?

 

[ZXDunny]

[...] using [...] glue we should stick feathers from the chickens to the eggs [...]

No, we're not producing chicken nuggets here.

I'm not sure I do understand you then. Chicken nuggets would require a small piece of chicken meat per PND file, with breadcrumbs attached as a rating. I was talking about chickens and eggs.

Why the hostility?

What hostility?


D.