Feedback on Repo.OpenPandora.org

[hedwards]

_wb_: using median instead of average will take care of that.

 
Nope, the median of 5 scores of 5 will still be 5. You can't trump the lack of data.

Well, yes. But there will always be differences in the sample size between applications. That means that with averaging one vote has more weight in some ratings than others. This is what the median is supposed to fix, not the small sample size.

This isn't a particular problem. Once you have a sufficient number of samples to have a statistically significant set of data, it doesn't really matter how many more data points you have. Sure, you will continue to see improvements in the accuracy, but the amount of accuracy will still be limited by the accuracy of the incoming data. For a system of ratings that is as inherently imprecise as the ones for apps, expecting even a 90% confidence is probably not possible.

As far as the median goes, that's not really a good way of doing it. If I've got 5 votes or a million votes for 5 stars and nothing else, the median is still 5 stars. If I add a 1 star and 3 star to the mix, the median will be 3 stars in both cases.

I suppose we could create a synthetic score that involves the standard deviation and the mean, but we run into the problem of a lack of data anyways. Even if we do solve the problem of the lack of data, we're left with the problem of biased sampling as you're likely to only get people rating apps that really love them or have them, people who feel indifferent are unlikely to actually rate an app.

We could probably weight the ratings that come with comments somewhat more than ones without, but that has its own issues as well.

 

[comradekingu]

The median is more susceptible to vandalism with fewer votes. Also it doesn't work how you described it, its 5 in both cases.

http://en.wikipedia.org/wiki/Median

If each vote has a bugreport, and is actually somewhat telling, it doesnt have to be an accumulated approach.

Then the barrier to vote is much higher and you will get a few votes as you have comments.

There is an if, because it isn't mandatory. Some people are constructive, and some people aren't, but both varieties can contribute to ability.

 

[Gausen]

Maybe some mixed thing like thumbs up and a somehow qualitative scoring. If it could be possible to ensure that one user registers one download only (despite the updates to the program and subsequent downloads of the same user) something like score = thumbs up / downloads could help a little. A perfect score would be 1, so the closest to one the better. It has problems though:

• This requires to login in order to download something (this is a big issue).
• It does nothing about a perfect score with one download and one thumbs up (this can be partially handled by showing both numbers together [ratio score and total downloads or total thumbs up]).
• It is not that different than what we have (except the voting system would be binary and the negative voting is expressed by the omission of thumbs up, so it can be a little easier on the devs).

 

[Binky]

I suggest a ternary system:

* Bad

* OK

* Good

....and it should be impossible to vote with out leaving a comment.

What about putting a bar chart on each app's page - of votes against rating.

then people can read all they need to know from that:

Rating troll:


bad: #
ok:
good:#####

Yawn


bad: ##
ok: #####
good:###
Brilliant


bad: #
ok: ####
good:##################
Rating is meaningless


bad: #
ok:
good:
Malware? Doesn't work?

bad: #########
ok: 
good:

 

[Neelix]

The main reason I almost never rate anything is that to me the scale is meaningless.  What we have now is essentially a binary system anyway...  you either rate it a 5 or lower than 5...   anything lower than 5 is a bad rating.

I can't help thinking that if you want a better sample size, you need to first implement a meaningful rating system that encourages an honest evaluation.

- Neelix

EDIT:  I do think Binky's idea above has merit.  It reminds me of the system Amazon use on the kindle store.

 

[Hồng Thất Công]

Heh, just take out the rating system and just leave the comment box. Problem solved

You don't like something, say something in the comment box.  If you like something, say you like it in the comment box.  If you don't like something and you don't say something, it's OK - no bad comments .  If you like something and put down something nasty in the comment box to make dev mad or revenge or whatever, dev will respond to your comments with his clarification and will ask you to give more details as to why you say such a nasty thing.  If you don't like something but say nice thing in the comment box, dev is happy and no one's feeling is hurt

Take out rating system and let users use their mouth to describe whether they like it or not

 

[_wb_]

Heh, just take out the rating system and just leave the comment box. Problem solved

You don't like something, say something in the comment box.  If you like something, say you like it in the comment box.  If you don't like something and you don't say something, it's OK - no bad comments .  If you like something and put down something nasty in the comment box to make dev mad or revenge or whatever, dev will respond to your comments with his clarification and will ask you to give more details as to why you say such a nasty thing.  If you don't like something but say nice thing in the comment box, dev is happy and no one's feeling is hurt

Take out rating system and let users use their mouth to describe whether they like it or not

How do you sort that?

Maybe some mixed thing like thumbs up and a somehow qualitative scoring. If it could be possible to ensure that one user registers one download only (despite the updates to the program and subsequent downloads of the same user) something like score = thumbs up / downloads could help a little. A perfect score would be 1, so the closest to one the better. It has problems though:

• This requires to login in order to download something (this is a big issue).
• It does nothing about a perfect score with one download and one thumbs up (this can be partially handled by showing both numbers together [ratio score and total downloads or total thumbs up]).
• It is not that different than what we have (except the voting system would be binary and the negative voting is expressed by the omission of thumbs up, so it can be a little easier on the devs).

I don't want to sacrifice anonymous downloading just for the sake of better scores.

But maybe it is not needed.

It should already be possible to record downloads per version number instead of one total number. Also, with a small modification to PNDManager, it should be possible to distinguish a new download from an update (as long as the update was done through PNDManager and not by manually anonymously downloading and then replacing). The combined data should be enough to get a pretty good estimate of the total number of installs, while just looking at the number of downloads of the most recent version should be a good indication of the number of people who still have it installed.

 

[Gausen]

^ Yes, it's true. I feel the same about anonymous downloads. I haven't thought of PNDmanager. That will be a better way of tracking downloads. But then how can you track downloads (one per user [or PNDmanager install] and per app) without login after a full reflash? (I guess people also download the same stuff over and over, I know I do!). I don't know maybe keeping separate statistics for people who used the repo key in PNDmanager can help? Then again, this will be pretty similar to logging in. Also this will give different rights to registered users and anonymous ones. Now that I think of it we already have different rights between registered and anonymous repo users.

 

[Hồng Thất Công]

@_@b_:  why do you need to sort? 

 

[_wb_]

@_@b_:  why do you need to sort? 

If you want to be able to show a list of "top rated" PNDs, then you need to be able to order/sort the ratings.

 

[Hồng Thất Công]

^ That's the whole issue right there.  Devs don't trust the rating system and make them feel bad if there are unjust 1-star rating.  To me, users should just read the comments and judge it for themselves whether the PND is worth dowloading...

 

[Neelix]

^ That's the whole issue right there.  Devs don't trust the rating system and make them feel bad if there are unjust 1-star rating.  To me, users should just read the comments and judge it for themselves whether the PND is worth dowloading...

You missed the point.    That would work for someone who has already selected a PND that they are interested in,  but it doesn't help someone who wants to use the ratings system to select a PND in the first place.

- Neelix 

 

[bzar]

Yes. PND browsing/discovery by reading ALL THE COMMENTS doesn't really work

 

[Hồng Thất Công]

^ That's the whole issue right there.  Devs don't trust the rating system and make them feel bad if there are unjust 1-star rating.  To me, users should just read the comments and judge it for themselves whether the PND is worth dowloading...

 
You missed the point.    That would work for someone who has already selected a PND that they are interested in,  but it doesn't help someone who wants to use the ratings system to select a PND in the first place.


- Neelix

READ THE COMMENTS! Perhaps it's just me but I always read the comments, not every single comments but kind of skim through them quickly to get an idea, even though it has five stars. This goes for all my online shoppings or looking for good restaurants because I don't trust the rating system when it comes to these.

 

[comradekingu]

You may read the comments, but thats beside the point. The ratings are broken and are in need of fixing.

 

[Hồng Thất Công]

Heh, it ain't broken.  It's just one-user-and-one-dev problem  

 

[foxblock]

Here is a wacky idea: How about a points-based rating system for apps?

Every app would start with some arbitrary number (say 1600 - this should be somewhat based on the size of the community or active voters and number of apps) - the point value of the app.

The user has  the option to "like" the app or "dislike" it. (*)

"Liking" the app will add 25 points (again, this number should be somewhat calculated), likewise "disliking" it will subtract 25 points - voting points. (**)

Therefore each app will start out as "mediocre" and good apps will start to show over time, while bad apps will slowly but surely sink to the bottom.

This will also factor in number of votes very heavily.

I don't really know if it is applicable, it's just to throw another idea in the mix.

Some problems I see:

The rating will take a very long time to become meaningful, since it needs a decent number of votes to push the point value of the app out of the "mediocre" area. One could adjust the voting points to get a s-shaped curve of the point value, so votes on very low or very high rated apps will count less (this could deflect troll votes, but also means once an app is in this crap or excellent region it will hardly get out of there for good or for bad).

Display of the point value is non-trivial. You could limit it at [0;3200] and display it using a "progress bar", could display the raw point value and since that's not really meaningful to most people a red/yellow/green light alongside.

(*) I personally like binary rating systems, since star-based ones ultimately boil down to the same thing - many people will either vote 5 or 1 and anything below 4.8 is not considered good, since it does not show up in a meaningful position on the ranking. Also expressing whether you "liked" something or not is much easier than expressing "how much you liked it", possibly getting more people to vote.

(**) You could even make these numbers variable based on how active the user is (factor in number of votes this month or even better: number of comments/reviews he gave this month). You could also go with a 5-star rating system and a point value of +25,+12,0,-12,-25

---

In any case I personally dislike the star-based rating and this sums up why: http://davidcel.is/blog/2012/02/01/why-i-hate-five-star-ratings/ (though I disagree with some of his remarks)

But there are other and in my opinion more important changes to be made

• It needs to give new apps more (especially good ones) more publicity (currently 10 new apps are shown on the front page, this should give equal spotlight to all new apps, but apps uploads are not uniformly distributed over time and not every new app is equally great)
• It needs to give forgotten or overlooked gems more publicity (this would be a hand picked list, maybe accompanied by a pandoralive blog post)
• Top XYZ lists should be rebooted every now or then
• Commercial apps should get more of a spotlight
• Users should be encouraged to rate and comment (e.g. display a text "don't forget to rate and comment" after downloading or display a "you recently downloaded, but have not rated yet" message when a user logs in. Move the comment box somewhere more visible, currently it is hidden at the bottom of the page. Get an incentive for commenting, like a badge on this forum or more weight to their ratings).
• "Users who downloaded this also liked XYZ" on the app page
• Statistics for app developers would be nice (how often was my app downloaded over what period of time, etc.)
• "Name your price" for commercial apps
• Encourage donations on free apps (see encourage ratings above)
• A black theme would be nice

On the encourage ratings/comments note: Newgrounds does this pretty nicely. Each user account has a level which can be increased by gaining XP. You get XP for rating entries (it's like 20 XP for 5 votes limited to once per day) and the top-1000 raters each month are entitled to select the top-of-the-month list from a bigger list of entries. Additionally the higher level you are the more weight your rating gets (so it is counted as 3.4 ratings for example).

There you have an incentive for rating, which is shallow, but really seems to work (granted this is for a much higher number of entries each month).

Similar things could be applied to comments: Have a "top-commenter" badge on this forum (reset this every month or so). Give top-commentors more weight in the calculation of the apps rating. Give each rating more weight if it is accompanied by a comment (random comments luckily can be reported and deleted, so hopefully this will not lead to many spam comments).

Anyway, thanks to milkshake for the repo. I hope the next version will be even greater and you will give a peak in the development process every now and then.

 

[hedwards]

SNIP

This is pretty much the same problem that we have. There aren't a sufficient number of votes to be meaningful. It's just that we would have a situation where nearly all the apps are within 100 points of the starting position. Adding a layer that requires people to understand isn't going to improve that. It reminds me of the year I spent under the tyranny of the metric system, it took a solid 6 months before the temperature system made any sense to me at all, because it's not based upon human experience and wasn't based on a system I was used to either.

Binary systems generally work even worse than the star system. There's no way of differentiating from a slightly below average app and one that's truly horrible. Plus, if it's broken that's reflected the same as the person didn't like it. Rather than getting mostly 1 and 2 stars for down and 4 and 5 for good, you just get 100% 1s and 5s, which is even less informative. Plus, some things are generally mediocre, and people should have the option of saying that they neither love nor hate it.

 

[comradekingu]

The metric temperature system is based on water and it freezes at 0 and boils at 100 given normal atmospheric pressure.

Foxblock: Wouldnt that mean once it gets "meaningful" it also gets stagnant.

 

[_wb_]

It reminds me of the year I spent under the tyranny of the metric system, it took a solid 6 months before the temperature system made any sense to me at all, because it's not based upon human experience and wasn't based on a system I was used to either.

What's wrong with the metric system? Phase transitions (freezing and boiling) of pure water make much more sense to me, than the defining points of Fahrenheit, which are based on the freezing temperature of brine (salty water) and some estimate of average human body temperature.

Anyway, back on topic: the main problem is that not enough people give ratings. One explanation could be that many people use PNDManager instead of downloading from the repo website directly, and there are three big obstacles to giving ratings in PNDManager:

1) it is not clear to most people that it is even possible to give ratings from within PNDManager

2) it is relatively difficult to enable your repo account in PNDManager (would be easier if there was a GUI for that)

3) AFAIK you cannot easily get a list of "recently installed PNDs", which is what you would need if you want to give ratings to stuff you tried out recently.