Feedback on Repo.OpenPandora.org

[Neelix]

To get into the top 300 rated programs, you need a score between 4 and 5 stars. 

after that point voting something as anything less than 5, brings the rating down,  not up.

That this is true at such a large portion of the top end of the scale has effects on how people rate things in general.

Put simply, with the system we have now,  there is nothing to indicate what the rating meant to the person who submitted it other than by numerical comparison with other ratings that are just as lacking in point of reference.

I didn't say the Amazon system was perfect, I said it had some good points.  I agree that it has some of the same failings as our current system, but that doesn't mean that the good points that it does have can't be adapted. 

On amazon I ignore the overall rating and go straight to the histogram and reviews.  Coupling the rating and comment systems means that you have a collection of justified ratings,  and the justification aspect makes  those ratings useful, even without a particularly large sample size.

- Neelix

 

[ekianjo]

To get into the top 300 rated programs, you need a score between 4 and 5 stars.  after that point voting something as anything less than 5, brings the rating down,  not up.

So what ? Anyway only the highly rated applications will make it to the top and it's not specific to that system. I don't see what a 4 would be negative vs a 5 - you are talking in terms of "relatively negative", but it's not negative in absolute.  

 

[ekianjo]

Put simply, with the system we have now,  there is nothing to indicate what the rating meant to the person who submitted it other than by numerical comparison with other ratings that are just as lacking in point of reference.

It does not matter what reference people use. Amazon does not explain either why something should be rated 4 or 5. But by compounding sufficiently large amounts of data, the overall score reflects actual user perception. My 4 may be a 5 for you, but it does not matter anymore when there are 30 other people voting -> these imperfections will be rounded up, and they are representative of the fact that people are not machines. 

 

[ekianjo]

On amazon I ignore the overall rating and go straight to the histogram and reviews.  Coupling the rating and comment systems means that you have a collection of justified ratings,  and the justification aspect makes  those ratings useful, even without a particularly large sample size.

I look at both average rating and distribution on Amazon, and then the comments too. This is very useful, and a binary system would completely destroy that level of information you have in the distribution charts.

There's no reason why we cannot replicate this system for the Pandora repo, in order to make it more like Amazon -> and benefit from its advantages.  

 

[Neelix]

I never said that a binary system would be my preference.

You were arguing however that the current system is superior, just by having multiple levels.  The point I was trying to make is that I don't see the existence of multiple levels in the current system on it's own as being more useful than the binary system described.  

I agree we should have a system with multiple levels,  but those levels need to be meaningful in some way.  Showing the distribution and requiring a comment to go with the rating would help a lot,    but I also think that points of reference need to be supplied, to ensure that that everyone is rating to the same scale.

- Neelix

 

[ekianjo]

but I also think that points of reference need to be supplied, to ensure that that everyone is rating to the same scale.

Again, not needed. I have done years of market researches with 5-points scales, and there are always differences, as I have explained, in terms of how you, me, and someone else will perceive what a 5 is worth or not. It's OK to have these differences, and it does not matter in the end as long as you have a large amount of data. You can't expect people to understand the scale in the same way, and it's perfectly OK for someone to vote 3 even if I vote 5, because that person will probably vote even lower for something I vote 3 myself. So, overall, it will still make sense in terms of score/average/distribution.

The biggest flaw we have in the current system is not that the scale system is not working, it's that we don't have enough people voting. If we had 20 or 30 votes for each entry in the repo, I can tell you you'd have a significantly better rating for each application out there.

Of course, this is not something you can do much about, because we are a small community - you can't "make up data" when you don't have it, but what you can do is build a better system to encourage people to vote more, or make voting easier/painless/faster, or reward voting in any way by gamifying the system. Give badges to those who vote or shit like that. Maybe that would work. I don't know. But tweaking the system is not going to make up for insufficient data. It's statistics 101.

 

[Neelix]

The biggest flaw we have in the current system is not that the scale system is not working, it's that we don't have enough people voting. If we had 20 or 30 votes for each entry in the repo, I can tell you you'd have a significantly better rating for each application out there.

Of course, this is not something you can do much about, because we are a small community - you can't "make up data" when you don't have it, but what you can do is build a better system to encourage people to vote more, or make voting easier/painless/faster, or reward voting in any way by gamifying the system. Give badges to those who vote or shit like that. Maybe that would work. I don't know. But tweaking the system is not going to make up for insufficient data. It's statistics 101.

Which is exactly why we need a system that relies on more than just statistics.  We need a system that will give meaningful feedback even without a large enough sample size for meaningful statistics because we will never have that, because the community is just too small for it.

- Neelix

 

[sebt3]

May be these boards can give us a hint where to go. If I mark a post as liked here, everyone would be able to tell I liked it : it's notified at the bottom of that post.

Why not having a system of public likes/dislikes. This way, if a PNDs is liked by someone I trust (say notaz) I'll be able to notice. On the other hand, if someone around have the same gaming taste as me and dislike a PND I might not want to test it out.

You know because all the votes are not equal. Beside it fixes the problem posed by the few raking troll we have around. Having their dislike identified might stop them, and even if not, we'll all be able to know they are trolling thuss allowing us to disregard that vote.

While I'm at that, what do you think about having your PND install list public ?

 

[ekianjo]

Which is exactly why we need a system that relies on more than just statistics. We need a system that will give meaningful feedback even without a large enough sample size for meaningful statistics because we will never have that, because the community is just too small for it.

Then if that's what you want, I suggest to remove all together the star systems, and just rely on comments. But, uh h, there are very few comments and even less of them than ratings. I'm not sure what you expect, short of a miracle.

Why not having a system of public likes/dislikes. This way, if a PNDs is liked by someone I trust (say notaz) I'll be able to notice. On the other hand, if someone around have the same gaming taste as me and dislike a PND I might not want to test it out.

What you want is a recommendation system. That's different from a rating discussion. But that could be interesting as well.

While I'm at that, what do you think about having your PND install list public ?

That could be easily done if there was a "link" between the PND you download and the profile on the repo. But that does not mean a PND you have is a PND you use.

You know because all the votes are not equal. Beside it fixes the problem posed by the few raking troll we have around. Having their dislike identified might stop them, and even if not, we'll all be able to know they are trolling thuss allowing us to disregard that vote.

One way around that is the make comments mandatory when rating. Trolls would be pissed off.

 

[_wb_]

Public likes/dislikes is a nice idea. But anonymity can be important too -- people may not want to make enemies by giving a honest rating.

So what about a -2, -1, 0, +1, +2 system, where you can choose whether your rating is public or not, and the extreme positions (-2 and +2) can only be given if you do so publicly. To sort stuff from good to bad automatically, the sum of all ratings suffices -- that also solves the problem of finding a good threshold for a minimum number of ratings, since no such threshold would be needed. Humans who want to get more information can see the names of the public ratings and the histogram for all ratings, and of course the comments (it's also a good idea to associate comments with ratings).

Changing it from 1, 2, 3, 4, 5 to -2, -1, 0, +1, +2 makes a big difference: it is much clearer what the meaning is of the rating, and giving something a 1 instead of a 2 does no harm (giving a 4 instead of a 5 can pull the average down, while giving a +1 instead of a +2 still brings the sum up, just not as much as a +2).

 

[ekianjo]

Changing it from 1, 2, 3, 4, 5 to -2, -1, 0, +1, +2 makes a big difference: it is much clearer what the meaning is of the rating, and giving something a 1 instead of a 2 does no harm (giving a 4 instead of a 5 can pull the average down, while giving a +1 instead of a +2 still brings the sum up, just not as much as a +2).

That's exactly what the star system is supposed to mean anyway (3 is neutral and 1 means not good, everyone AT LEAST understands that), but if you think that it will make a difference, i dont have any issue with it. 

 

[_wb_]

Changing it from 1, 2, 3, 4, 5 to -2, -1, 0, +1, +2 makes a big difference: it is much clearer what the meaning is of the rating, and giving something a 1 instead of a 2 does no harm (giving a 4 instead of a 5 can pull the average down, while giving a +1 instead of a +2 still brings the sum up, just not as much as a +2).

That's exactly what the star system is supposed to mean anyway (3 is neutral and 1 means not good, everyone AT LEAST understands that), but if you think that it will make a difference, i dont have any issue with it. 

3 being neutral and 1 being not good is not necessarily the only interpretation of a star system. E.g. hotel stars or Michelin stars work in a rather different way

 

[ekianjo]

E.g. hotel stars or Michelin stars work in a rather different way

yeah, but usually people are aware than more stars means better than less, so the understanding is still there. And if that really IS the key problem at hand here, just add a smal script to display "very good" when you hover 5 stars and "very bad" on one star, to make it more obvious. But I seriously doubt it would change anything. 

We need more votes, and if we could vote directly FROM inside the application the friction to vote would be much lower. we need an api to do that.

 

[comradekingu]

3 being average vote, in turn being ok, is not apparent, and is a cognitive bias present in every 1-5 star system I have ever seen.  http://xkcd.com/325/ https://xkcd.com/937/ http://xkcd.com/1098/

Instead of -2 and +2 only being face value marks (for which you could plausibly fakereg an account anyway) I propose they are only available if you add a comment along with it to explain matters.

That gives other cause for concern, and it directs attention towards whats good or bad, rather than "this is apparant to me = *stars"

 

[sebt3]

Public likes/dislikes is a nice idea. But anonymity can be important too -- people may not want to make enemies by giving a honest rating.

Want anonymity ? just dont vote, simple

 

[hedwards]

Public likes/dislikes is a nice idea. But anonymity can be important too -- people may not want to make enemies by giving a honest rating.

Want anonymity ? just dont vote, simple

That's a terrible way of handling the problem of too few votes. Unless and until there's some evidence to support the supposition that anonymity is causing the ratings to deteriorate, there's not reason to even consider doing away with anonymous ratings.

It's deeply disturbing how many people are willing to chuck anonymity under the bus because they don't personally understand the implications of having all their information combined from various parts of the internet into a profile. What's more, how do you enforce this anyways? What's to stop trolls from creating new profiles with pseudonyms to troll with? And how do you know that sbt3 here is the same one as on the repo web site?

 

[sebt3]

That's a terrible way of handling the problem of too few votes. Unless and until there's some evidence to support the supposition that anonymity is causing the ratings to deteriorate, there's not reason to even consider doing away with anonymous ratings.

? No vote mean no care, that's it. If you dont care sharing that *you* liked to the 5000 out there that can also play it, then you dont like that PND much.

It's deeply disturbing how many people are willing to chuck anonymity under the bus because they don't personally understand the implications of having all their information combined from various parts of the internet into a profile. What's more, how do you enforce this anyways? What's to stop trolls from creating new profiles with pseudonyms to troll with? And how do you know that sbt3 here is the same one as on the repo web site?

You reply your own question here. If you cannot tell if sebt3 on boards is the same as the one on the repo, how the fuck can *you* link that sebt3 nick I have with my RL identity ? You cant. Hence I'm anonymous here. Only a selected few knows that link.

 

[comradekingu]

So why force any kind of non-anonymity then?

 

[hedwards]

That's a terrible way of handling the problem of too few votes. Unless and until there's some evidence to support the supposition that anonymity is causing the ratings to deteriorate, there's not reason to even consider doing away with anonymous ratings.

? No vote mean no care, that's it. If you dont care sharing that *you* liked to the 5000 out there that can also play it, then you dont like that PND much.

We already have a problem of people not caring enough to vote and you want to erect additional hassles? Seriously? Are you even reading what you're posting?

It's deeply disturbing how many people are willing to chuck anonymity under the bus because they don't personally understand the implications of having all their information combined from various parts of the internet into a profile. What's more, how do you enforce this anyways? What's to stop trolls from creating new profiles with pseudonyms to troll with? And how do you know that sbt3 here is the same one as on the repo web site?

You reply your own question here. If you cannot tell if sebt3 on boards is the same as the one on the repo, how the fuck can *you* link that sebt3 nick I have with my RL identity ? You cant. Hence I'm anonymous here. Only a selected few knows that link.

You must be new to the internet. The technology is there by anybody that really cares to figure that sort of thing out. It's rather naive of you to think that anything you do or say online isn't traceable back to you buy anybody that's interested in doing so.

You're at best pseudo-anonymous here. Post some chld porn here and see how long it takes the authorities to subpoena the boards and track you down. I'm guessing it would take less than a week. There's literally huge server farms dedicated to tracking everybody so that they can serve ads to them that are properly targeted. Not to mention that your way of saying things is hardly unique.

In other words, just because you choose to be willfully ignorant of how easy it is to track people online with the right access to the databases, does not mean that you're in any way anonymous.

 

[moxie]

Yesyes, but that is irrelevant. As long as you yourself is neither Google nor NSA (nor privy to their databases), the fact that they'd track Sebt3 down in a week has no bearing on whether he is anonymous visavi the rest of the forum dwellers, which is the relevant thing here. Anonymity is not a binary thing, it is something you'd have in degrees and in relation.

For instance: To the general forum dweller, the larger identity of sebt3 is a mystery, but we can at least get an inkling of the persona that sebt3s handler wants to present, since we can assume that all posts from the same forum identity comes from the same source. That is fairly good anonymity. The admins can see stuff like IP numbers and so on, which means slightly less anonymity, and in particular those who also have admin rights to the repo can use that to ascertain with a fair degree of confidence whether forum-sebt3 is equal to repo-sebt3.  So, visavi the admins, you have a slightly lesser degree of anonymity. 

And so on. Saying "Nuh-huh, N00b, you're never anonymous on the net cuz GOOGLE PWNS Yr 4SS!!!" is just about as relevant as saying "Nuh-huh, but you didn't win this fight, cuz Mike Tyson'd beat you to a PULP!"