Repo rating system: 5 stars or binary ratings?

[Kumaki]

It is a good idea, because often things come up after the fact of release, that aren't known, and the developer fixes them...or upgrades the capabilities of it to play more games.  I am thinking in terms of emulators here.

 

[zLouD]

Let us use Facebook Likes instead!

@_wb_: I'm not shouting!!

 

[_wb_]

I'd rather have :

:angry:   <_<   :mellow:   ^_^  

Not usable - Needs improvements - Okay - Good - Everybody must have this

-2 | -1 | 0 | +1 | +2

Each rating should be linked to the version of the app used, that way, we can easily follow the improvements.

Let's change the graphics to something like this instead of the stars. Let's change the interpretation to [-2, +2] instead of [1, 5]. Let's assign ratings to versions and ignore the old BAD ratings (as in negative) but count the old GOOD ratings (at least until the rater updates his rating to the newer version): the assumption here is that updates tend to be improvements.

And let's somehow take the number of ratings into account in a better way than the current threshold at 4 ratings: my proposal was to do SUM() instead of AVG() but that could give a bit too much weight to the mere quantity of votes. A compromise would be something like SUM / sqrt(COUNT): this would mean that situation X := 50 times +2 and 50 times +1 ((50*2+50*1)/sqrt(100) = 15) would be better than situation Y := 41 times +2 and 40 times +1 ((41*2+40*1)/sqrt(81) = ~13.55) even though AVG(X) is lower than AVG(Y), but it would not be better than situation Z := 60 times +2 (60*2/sqrt(60) = ~15.49) even though SUM(X) is higher than SUM(Y).

 

[F_Slim]

I hesitate to give a rating worse than 4, not sure I have ever done so. Almost everything in the repo is given to us for free and was done by someone in his/her spare time, and usually it takes up quite a lot of time!! I guess it becomes easier for more experienced people, also because they set up a system which allows to port things more easily, but at the same time they start to do more advanced and complicated stuff.

Anyway, I'd hate to discourage someone by giving a bad rating. Either my rating is 5, or it's 4 and I try to give constructive criticism ... or I just don't vote at all. I'm happy if someone actually bothers to give a vote for my stuff at all. If I'd get a lot of bad votes, I'd probably just stop trying to put anything up, because to me the community would look like a bunch of spoiled brats

Since there are not many votings, and giving a vote is a bit of an effort (you have to log in, find the .PND again, press another button) I would skip bad votes entirely. Just allow 2 votes: "good" and "great". If you don't like it, just don't bother to vote, but be thankful there are people devoting their time to bringing software to the Pandora for free.

 

[b_o_b]

I hesitate to give a rating worse than 4, not sure I have ever done so. Almost everything in the repo is given to us for free and was done by someone in his/her spare time, and usually it takes up quite a lot of time!! I guess it becomes easier for more experienced people, also because they set up a system which allows to port things more easily, but at the same time they start to do more advanced and complicated stuff.

Anyway, I'd hate to discourage someone by giving a bad rating. Either my rating is 5, or it's 4 and I try to give constructive criticism ... or I just don't vote at all. I'm happy if someone actually bothers to give a vote for my stuff at all. If I'd get a lot of bad votes, I'd probably just stop trying to put anything up, because to me the community would look like a bunch of spoiled brats

Since there are not many votings, and giving a vote is a bit of an effort (you have to log in, find the .PND again, press another button) I would skip bad votes entirely. Just allow 2 votes: "good" and "great". If you don't like it, just don't bother to vote, but be thankful there are people devoting their time to bringing software to the Pandora for free.

That is my method of rating too

I hate it when people rate 1 star and don't comment. You could add a 'need improvements' rating that is visible but doesn't count for the average rating. Selecting 'need improvements' should then force people to add a comment what they think needs to be improved. 

That way a great app without fancy gui that is fine for some users can still have a good rating, but there is the possibility to add a comment and request improvements. 

'need improvements' no star (add a separate counter) 

3 stars : useful / ok

4 stars : good

5 stars : great

Count also latest ratings higher then older ratings. So e.g. after a year the 5 stars weighs 50% of a new rating. After 2 years 25%. After 3 years a new rating is needed until the Pandora is 'dead' system like the GP2X then remove the algorithm. 

 

 

[Asiyura]

I'm the kind of guy that start working seriously only if my work get bad ratings. I know it can be hard to get them, but their needed to have a more accurte feedback.

That's why I did not use negative words like crap or bad in my proposal.

The worst rating is that the software isn't working for the aimed usage, the devs need to rethink the complete thing, that would "weight" -2 in the rating system.

If the app crashes frequently, then it needs improvements, it's not really negative, it proves you took the time to used the soft but it didn't met - quality-wise - your requirements. And that's a -1.

If the app runs smoothly (even with a bug now and then that will get fixed in the next version) and mostly do the job correctly but you won't use it, that's a neutral review of 0.

Now if it's working well and have a nice interface adapted to pandora, that's +1

And finally if the app has numerous features a really nice interface and you think everyone should have it on their pandora (like the PNDmanager), that's a +2

EDIT: "bad" reviews, that way, wouldn't mean that "the soft is crap" but "the soft needs

more work on it before being useful"

(can't get used to pandora's keyboard lol)

 

[Neelix]

The Platinum, Gold, Silver, Bronze was serious.

So instead of 1,2,3,4 or 5 stars, you want platinum, gold, silver and bronze? In fact the same system, just with 4 different choices aaaand with fancy names? Or was this meant for this thread?

Actually I see some sense to that, arbitrary numerical ratings are just that, arbitrary. Giving the levels names give one more of a frame of reference to go by for assigning a meaningful rating. That being said, I think more descriptive names describing what each level means like what KodeIn describes above would be better.

- Neelix

 

[ptitSeb]

Sorry to grave digging this, but after some debate, nothing happens on the repo and we still have the 5 stars system. And I'm getting more and more annoyed by it.

Getting a "1 star vote", with no explanation (like freespace 2 just got, but that just one example), make no sense. Well, the only logical explanation for this behavour would be someone watching the repo and put a 1star to lower the score of PND that are to high in the "Most rated" column of the repo.

Again, can we just have a binary system, with just "thumb up" or nothing, and you count thumb up (or golden pandora)?

 

[ekianjo]

Sorry to grave digging this, but after some debate, nothing happens on the repo and we still have the 5 stars system. And I'm getting more and more annoyed by it.

Getting a "1 star vote", with no explanation (like freespace 2 just got, but that just one example), make no sense. Well, the only logical explanation for this behavour would be someone watching the repo and put a 1star to lower the score of PND that are to high in the "Most rated" column of the repo.

Again, can we just have a binary system, with just "thumb up" or nothing, and you count thumb up (or golden pandora)?

One (intermediate) way would be to remove the anonymity of the votes. If we see there is a person always rating 1 for no reason, that would be ignored by the rating system.

I don't really like the thumbs-up system, it does not seem very fair either : so either like it or nothing? There should be degrees to rate "how good" is something, if not it just turns out like Facebook which is pretty meaningless.

 

[Neelix]

I don't really like the thumbs-up system, it does not seem very fair either : so either like it or nothing? There should be degrees to rate "how good" is something, if not it just turns out like Facebook which is pretty meaningless.

Adding degrees to a scale with doesn't add any meaning, as the scale itself is still completely subjective.     At least with the binary system (as seen on you tube) you get a clear indication of popularity.  

- Neelix 

 

[ZXDunny]

Don't worry Seb, I just voted you a 5. It's worth a 3 (4 at a push) but I wanted to make you feel better.

The problem is - what if someone really does think you did a bad job with Freespace? What if they really dislike it? They're entitled to vote it as a 1. I really enjoy your cannonball port, but if I were to comment on the quality it would get a 1 or 2 - it's horrendously sub-optimal and really, really slow for no reason other than sloppy coding (it's translated from the original disassembly, so isn't particularly optimised). 

All of that is not your fault - you just ported it and added some cool stuff. However, I can either vote for it based on the quality of the port (good) or the quality of the actual product (bad).

Which is it to be?

D.

 

[ptitSeb]

Don't worry Seb, I just voted you a 5. It's worth a 3 (4 at a push) but I wanted to make you feel better.

The problem is - what if someone really does think you did a bad job with Freespace? What if they really dislike it? They're entitled to vote it as a 1. I really enjoy your cannonball port, but if I were to comment on the quality it would get a 1 or 2 - it's horrendously sub-optimal and really, really slow for no reason other than sloppy coding (it's translated from the original disassembly, so isn't particularly optimised). 

All of that is not your fault - you just ported it and added some cool stuff. However, I can either vote for it based on the quality of the port (good) or the quality of the actual product (bad).

Which is it to be?

D.

I get your point. In fact, the think that really bother me is a low vote with no reason given. You put a low vote to signify some is not good. Not optimized enough like you said, are bad control, or crappy gameplay... But with no explanation, how to improve something?

For the vote, I take the vote as a mix port/product. For cannonball for example, I work on it periodically to optimize what I can and make it faster, because if it's too slow, it wont be interesting to play.

That beeing said, vote/quality is subjective, and not every one as the same values for rating, the same meanings for a 3 stars or 5 stars. That why I would prefer a binary system, but then, only quantity of good votes will make the difference, so maybe be it's not more fair than the 5 stars.

Depends on the main use of the rating system I guess.

 

[CocoCreekFisherman]

The Repo is totally democratic and subjective and should remain this way. The fact that there is such a low response rate at the Repo is upsetting to me. I believe that if someone is going to rate an app say between a 1<>3/5 then an explanation would be appropriate and maybe helpful to the developer..depending on the depth of the comment or grammar of the rater for that matter . There is no prerequisite to comment. Unless the format is changed so one must comment in so many characters if they elect to rate numerically. Personally I would be fine with that.

I do not own a solar powered Pandora.  B)

 

[ZXDunny]

As I have said previously, I always rate a 5. That way the guy who ported it is happy that someone, somewhere, thought his port was awesome. It's just a nice thing I do.

D.

 

[WizardStan]

Perhaps a solution is that ratings left by people who have left a comment on the app have more weight than anonymous or non-commented ratings.

Arbitrarily choosing a 2x weight scale, that means if someone gives a 5 star rating and leaves a comment about how good it was and someone else leaves a 1 star without saying what was wrong, that makes an average score of (2*5)+1/3 = 3.67. If a lot of people are enjoying it and commenting as such, the 1 star uncommented ratings will be rapidly drowned out.

More importantly, I think you shouldn't take it so seriously. It's "the internet", there are enough people out there that you can't help but get a few jerks, or people that just react without thinking. That means you're going to get the occasional 1 star review, but if it's in a field of 4 and 5 stars it shouldn't bother you, it's just one person who has a stick up their butt and there's probably nothing you can do about it.

 

[PowerGod]

Maybe can be added an "obligatory" comment system ?

1 STARS: (Makes obligatory the reason selection, else no rating is saved)

O Doesn't work

O It's not for me

O Too Slow

O Other (with the comment textbox)

.

.

.

5 STARS: (Maybe comments are optional here)

O Perfect

O Other (with textbox)

 

[b_o_b]

I agree with ptitSeb, anonymous downvoting should be stopped. 

It makes me cringe to see bad ratings without further explanation. Also from a users perspective. If a program has bad ratings I will probably be less motivated to try it. So ratings should be taken seriously. 

Maybe it could be more sophisticated, something what PowerGod also suggests. 

Votes for the technical achievement (works ok yes/no) - I think that is nice to know for a dev. Even if people don't like the program at least you know it works. In this category I think a comment should be mandatory for bad ratings. 

Votes for the content (useful - fun) 

 

[ekianjo]

As I have said previously, I always rate a 5. That way the guy who ported it is happy that someone, somewhere, thought his port was awesome. It's just a nice thing I do.

D.

If everyone did like you we'll end up with all applications (crappy ones and good ones) having the same rating. It's as pointless as the guy who puts 1 to everything.

 

[ekianjo]

I don't really like the thumbs-up system, it does not seem very fair either : so either like it or nothing? There should be degrees to rate "how good" is something, if not it just turns out like Facebook which is pretty meaningless.

Adding degrees to a scale with doesn't add any meaning, as the scale itself is still completely subjective.     At least with the binary system (as seen on you tube) you get a clear indication of popularity.  

- Neelix 

We don't care about popularity, especially on the repo where you get so few votes. Its more important to get an indication of how good and usable the application is.

Thumbs up is subjective as well, then -

The point of averaging the 5 point scale is that you smooth the subjectivity away when you have sufficient votes. Thumbs up does not mean ANYTHING at all.

I don't think I have thing any app in the top 10 best rated ones which didn't deserve to be there. So, for me, this system is working, we just need to ensure people can't rate 1 for no reason whatsoever.

Maybe can be added an "obligatory" comment system ?

1 STARS: (Makes obligatory the reason selection, else no rating is saved)

O Doesn't work

O It's not for me

O Too Slow

O Other (with the comment textbox)

.

.

.

5 STARS: (Maybe comments are optional here)

O Perfect

O Other (with textbox)

Yeah, I can agree with that. That's a better place to be.

 

[ZXDunny]

As I have said previously, I always rate a 5. That way the guy who ported it is happy that someone, somewhere, thought his port was awesome. It's just a nice thing I do.

D.

If everyone did like you we'll end up with all applications (crappy ones and good ones) having the same rating. It's as pointless as the guy who puts 1 to everything.

Not true - my strategy makes people feel good.

D.