Repo rating system: 5 stars or binary ratings?

[ekianjo]

With 1-5 stars you cannot do that because even 1 star ratings are "positive".

Have you ever used Amazon ? They have 1-5 stars system as well, and it works perfectly for me. I clearly understand that 1 means "crap".

I was talking about adding ratings together to get an overall picture. With a 1-5 system you have to average the ratings, not add them, to get something meaningful. And then you get the problem of how to sort, because 4.9 star average with 500 votes is clearly better than 5 star average with 2 votes, so just the average is not enough.

You always get problems between different sample sizes. There is no easy way around it, even if you sum things up without average : does a app with 1000 people saying +2 is better than a new app with 10 people saying +2 ? You don't know, since the sample is not the same size.


The best way to measure it when you have such data is a combination of average and standard deviation, because the standard deviation tells you how consistent the scores are, on top of the average.

 

[_wb_]

With 1-5 stars you cannot do that because even 1 star ratings are "positive".

Have you ever used Amazon ? They have 1-5 stars system as well, and it works perfectly for me. I clearly understand that 1 means "crap".

I was talking about adding ratings together to get an overall picture. With a 1-5 system you have to average the ratings, not add them, to get something meaningful. And then you get the problem of how to sort, because 4.9 star average with 500 votes is clearly better than 5 star average with 2 votes, so just the average is not enough.

You always get problems between different sample sizes. There is no easy way around it, even if you sum things up without average : does a app with 1000 people saying +2 is better than a new app with 10 people saying +2 ? You don't know, since the sample is not the same size.


The best way to measure it when you have such data is a combination of average and standard deviation, because the standard deviation tells you how consistent the scores are, on top of the average.

Yes, I agree. We could use boxplots instead of averages.


But my original point was that the star system is not a very good one, because it's unclear what the meaning of 2-4 stars really is, and because most people vote in a more or less binary (or maybe ternary) way anyway: 1 star if it sucks, 4 stars if it's good but can be improved, 5 stars if it's really good.

 

[PowerGod]

There must be taken in consideration even the updates of an app, for example in the comments on the repo it's present the reference to the version at the time of writing.


If an app v0.1 gets a bad score, even if the v2.0 is totally great it suffers from the previous score.


Anyway I actually don't know how this can be implemented...

 

[Znirk]

I think you're just more likely to get ratings from people who strongly like or strongly dislike something. A points-out-of-five rating scheme is fine for professional reviewers whose job it is to express an opinion on whatever they're assigned, even if it's kind of meh. With volunteer raters, the middle ground won't be used much: not because nobody has an ambivalent, kinda-ok, not-really-my-thing opinion, but because those people are far less likely to express their opinion through comments or ratings than the people who feel strongly about the matter.


Hmm ... how about this: Every user who downloaded something counts as 3 stars (or call it zero out of -2..+2) unless they go in and change that rating. That way the average for any widely-used but rarely-rated product will automatically drift towards mediocrity


Seriously though, it would be preferable to encourage more people to rate stuff. I don't have any brilliant ideas on how to do this though -- does pndstore have a function which helps a user go through their unrated apps? Some kind of list ("Drag to rank! And while you're at it, rate and comment!"), or a way to step through a random selection of titles? I'm guessing the repo itself doesn't necessarily know what everyone has installed.

 

[ZXDunny]

I always rate software on the repo as a 5, because I believe that everyone should feel good about their creations.


D.

 

[_wb_]

Hmm ... how about this: Every user who downloaded something counts as 3 stars (or call it zero out of -2..+2) unless they go in and change that rating. That way the average for any widely-used but rarely-rated product will automatically drift towards mediocrity

This is actually not a bad idea. Of course it should only apply to non-anonymous downloads. It would encourage people to rate more things, which would help to make the ratings more useful.

I always rate software on the repo as a 5, because I believe that everyone should feel good about their creations.

I always rate home-brew things 4 or 5 (for the same reason); if I give a 4 I leave a comment on how to improve it and I revise my rating to 5 if it gets improved.


Bad quick ports I give lower ratings.

 

[bzar]

I agree that averaging votes can throw the result off with small sample sizes. The obvious solution would be to use median to sort the apps and display variance present how unanimous the rating is. Is an app with a 4.82 rating really better than one with 4.73? Or are they both just great pieces of software that get 5 stars most of the time? Of course this breaks top ten lists and instead divides all apps to five slots, but I'd argue that any order within the slots is arbitrary.


EDIT: I'd also attach literal interpretations to the star-ratings. Something like:

• 5 stars: Everyone should have this on their pandoras
  
• 4 stars: Very polished application, prefer this if you have any need for something like this
  
• 3 stars: Good application. Does what it says.
  
• 2 stars: Usable, but needs work
  
• 1 star: Unusable
  

 

[foxblock]

I have always been in favour of a YouTube-style +1/-1 system over the 5-star ratings.


5-Star ratings tend to be biased, too, since people don't vote based on their opinion, but they try to push the average rating towards their opinion (so if you think something is "okay" you would vote 3, since the current average rating is 5 you would vote 1 to bring it closer to 3, although 1-star-rating does not reflect your actual opinion). That's also part of the reason 5 and 1 star ratings tend to be used over the middle 2,3,4 ratings. (There was a study about this, can't find the link any more sadly).


Then there is the already mentioned problem of comparing ratings (the repo does this somewhat, by displaying apps with 5 votes or fewer in a different colour).


Also it's easier to give a +/- rating as you either like an app or dislike it, you don't have to think about "how much" you like it, which puts people off voting.

Have you ever used Amazon ? They have 1-5 stars system as well, and it works perfectly for me. I clearly understand that 1 means "crap".

The star rating works for Amazon because you have to enter a review for your rating and justify it (which helps making up your mind and putting in more thought - leading to less 1 or 5 ratings). Personally I also find the literal reviews much more helpful than the ratings alone. Sometimes I buy a 3-4 star item, just because the 1-star reviews complain about something I don't care about or which apparently has already been fixed (it certainly does not always mean "crap" since some people are just stupid and love to complain).


This is not possible with the repo as comments and ratings are not linked (just like on YouTube)!


Though I would suggest a +1/0/-1 system. (which also isn't biased like a +2/+1/-1 system).

 

[Dimacus]

I'm also in favor of a +1/-1 system, unless a system like what Znirk suggested is implemented, then i'd probably like to stick with the 5 star system, even tough we all have different opinions on what said stars means.


For me, they mean 1 = bad, 2 = ok, 3 = good, 4 = Excellent, 5 = This is _THE_ best application ever.


But if a +1/-1 system is implemented, there also needs to be a "Did not work for me" option that dosn't requre (but encourages) a comment on what failed, since the -1 that application would otherwise receive will be read as _bad application content_ rather than anything else.

 

[_wb_]

Sorry for the gravedig, but we had some discussion about this on IRC and here's a new proposal:

To keep the amount of changes small, we keep the 5-star system, but we change the meaning of the stars:

1 star = -2 (really bad)

2 stars = -1 (bad)

3 stars = 0 (meh)

4 stars = +1 (good)

5 stars = +2 (very good)

(another option would be to make the scale go from -1 to +3, so 2 stars would be "meh")

When making a ranking of "top rated" PNDs on the repo, the current procedure is as follows: first all PNDs that got at least 4 ratings, sorted by average rating; then all PNDs that have 3 or less ratings, sorted by average rating; then all unrated PNDs.

The alternative procedure I propose is to simply sort all PNDs based on the SUM of all ratings (where the rating ranges between -2 and +2).

This is in my opinion a better and more robust way to make the ranking. Let me illustrate with an example:

Application A has 100 ratings: 90 times 5*, 9 times 4*, 1 time 1*

Application B has 50 ratings: 45 times 5*, 5 times 4*

Application C has 4 ratings: 4 times 5*

Application D has 3 ratings: 3 times 5*

Application E has no ratings

Application F has 3 ratings: 3 times 1*

Application G has 4 ratings: 4 times 1*

Application H has 30 ratings: 1 time 2*, 29 times 1*

How would you order these applications from best to worst? I would order them in the order I just gave: A, B, C, D, E, F, G, H. My reasoning would be as follows: clearly A is extremely good, because a lot of people took the effort to rate it and rated it very high. B also gets very high ratings, but only half as many people voted so it may be somewhat less relevant. C and D get the highest possible average rate, but only a few people voted so that is less meaningful than the rating of A and B. Those four are the only with a positive score; E is unknown (maybe new or so boring nobody tried it?), F and G both get the worst possible rating unanimously but only a few raters, and H certainly is the worst because lots of people went through the effort of giving it a very bad rating.

In the current system, the order would be C, B, A, H, G, then D, F, then E. The worst one gets ranked 4th out of 8, and the best one only gets ranked 3rd.

In my proposed system, the order would be A (187), B (95), C (8), D (6), E (0), F (-6), G (-8), H (-59).

The disadvantage of my proposal is that it gives a big bonus to stuff that gets lots of ratings (as long as those ratings are positive, i.e. 4 or 5 stars), e.g. something with 100 four-star ratings will be considered better than something with 10 five-star ratings. So it may become more of a popularity contest than an actual rating. In my opinion, that's not really a bad thing, because most people will use the list of top-rated PNDs to find things that are relevant to them, and things that have very few ratings but with a very good average score (e.g. C or D in the example above) are probably not that relevant (they are probably of high quality but with a specific purpose that only a small number of people need), so it's fine if those don't end up at the top of the list.

 

[milkshake]

What do other people think of this?

 

[ptitSeb]

I don't like rating. Beeing to current method or the new proposed by _wb_.

As an uploader, I find the rating to be a factor of demotivation, never a positive factor.

As a downloader, I mostly trust donwload number (which is confusing, because it's the the sum of all download for all version), and comments. I can choose without star system.

But I may not be representative here, it's just my opinion.

I would prefer some kind of label "Great on the Pandora" (for the 1st time users), when the Soft works perfectly on the pandora, rather than a rating (again, most soft here are free, what do you loose by downloading something not great?)

 

[ZXDunny]

As I'm sure I pointed out earlier (it was a while ago though), I vote 5 stars if I like the author. Regardless of the system implemented, I'll continue to do so

D.

 

[foxblock]

I don't think the proposed change really addresses the main issue (being not enough people voting).

Looking at the top lists, the overall star-rating seems fine to me (as far as a star-based rating can be fine). Just shifting around the zero-point of the scale will hardly have any effect.

Also having a good algorithm for sorting is not critical in our case, since most people will only look at the very top of the list anyway and else look for specific software, which, given our small community, will be unique, so ratings don't matter too much in that case (besides telling whether the app appears to be shit or not). The current algorithm could be tweaked to take a trend of ratings and volume into account (like many 1-star ratings), but I think it is fine for most cases.

To address the key problem (in my eyes) the rating system needs to be changed to a binary system (+1,-1, see YouTube) and, more importantly, it should be possible to rate applications from software like PNDManager or even directly from the app itself (same with leaving a comment).

Additionally the repo site should suggest leaving a comment to every rating given, since those matter more than some (rather arbitrary) value.

After all a rating can only be so useful, people need to experience the apps for themselves and should be encouraged to do so, since everyone has different standards.

EDIT: Another idea to help people find the right software: Expand the search to something apps.open-pandora did (just better implemented) and allow people to search for "GB emulator", display all emulators, give an optional explanation of "emulation" or whatever, in short make the search engine more "intelligent".

 

[PowerGod]

Because one must be registered in order to be able to vote, many people will not do it... I think that it should be better to permit something like "anonymous" ratings and comments, maybe giving them less importance/visibility than the registered ones... (something like the last compo voting system)

I like the _wb_ proposal, I just add a consideration for anonymous votes

 

[Ziz]

I see no problem in anonymous comments + internal IP saving for security purposes.

I don't like the idea of application-internal voting. That nerves me already in Android: "You started this game the second time, VOTE FOR ME!!!". Furthermore not every software would use this feature: The programs, which would, would get more (probably 5 star) votes. So if you beg for votes you get rewarded...

I like the idea of a login possibility in the repo program on the pandora much more (if it isn't even possible yet, can't test it, have no pandora). E.g. if a program is updated, the repo-manager could say "Hey, the following programs got updates. Do you like them? Vote now!"

That would reward software with updates. Or even better: "You gave 4 stars the last time, the software got a new version, would you like the change your vote?" Because I think often especially for hand made games only the final touch is missing and that is the reason to get "only" 4 stars (yes, the amazon-star-system isn't that good).

Just my 2 cents.

 

[Rockthesmurf]

A rating system that is based on both downloads and the actual rating given to applications will have a negative affect on paid applications because paid applications will most certainly have far less downloads than free offerings. A lot of stores to have separate paid/free charts which obviously fixes this, however I don't feel the Pandora store warrants a separate paid section.

I haven't found any major problems with the store as it is, I would be happy to see App Store style 'extra's like rotating banner at the top of the site that has 5-6 applications (Editors Choice, Game of the Week, Retro Classic) obviously this requires both work from someone to decide the titles and also means ultimately there will be a person/group selecting what titles are shown.

 

[milkshake]

more importantly, it should be possible to rate applications from software like PNDManager or even directly from the app itself (same with leaving a comment).

PND Manager can already allow you to vote for apps as long as you set up your configuration file in the app data folder with your repo api key.

 

[freamon]

I don't like the 5 star system either.

I'd prefer it if there's was just a simple 'recommended' tag, so that you could sort it as "Most recommended apps", and see "Most recommended SNES emulator" (for example). Also, it could conceivably be possible to see something like "Apps recommended by freamon" (so I could put the link in my sig, along with the "My PNDs" one).

I think this best reflects how most people currently use the repo (4/5 stars if the app works, but not bothering if it doesn't).

 

[pmprog]

TBH, I never actually look at any ratings on the repo. I'll download any PND that I think I can/will use, then I'll keep it, or remove it based on if it fits my needs.