Repo rating system: 5 stars or binary ratings?

[b_o_b]

The rating system is indeed flawed, only 'likes' and comments would be a lot better. I also prefer to have one account/login for the forum + download section - would solve the CCF 'problem' too

there are only 1337 PNDs at the time of this writing 

Really? 1337? I think that should be celebrated. 

 

[Rockthesmurf]

I think we should be able to rate on peoples ratings, so the stupid 1 star ratings we can all rate 1 star so they end up worthless!


More seriously, incredible amounts of thanks to you ptitSeb, so many people are constantly astounded by not only the amount of release you make, but the care and attention each release gets to make them as good as possible.


It sucks your PNDs don't get enough 5* ratings, but most people on this forum would do one better than that and vote you a 5* person. I hope that makes you feel a little better

 

[alxm]

there are only 1337 PNDs at the time of this writing 

Really? 1337? I think that should be celebrated. 

It should, I have about 130 on my card and always find cool new things when I spend a few minutes looking around the repo. What I meant is that there are 1337 PNDs spread over many specialized categories, so if you're looking for a certain kind of game or application you don't really need ratings to make sense of what's available.

 

[M-HT]

Recently, I read this article about game ratings.

Did someone consider the Rotten Tomatoes rating system for repo?

 

[ekianjo]

It is a free world and that is why CFF can give 1 star if he wishes.


Deleting his posts or removing his ratings would make the system open to abuse.

Nope its not a free world here thats why we have moderators.

 

[levi]

Hmm, how about a system where the pnd_run script asks you for a rating/to update your rating when you exit a program? I don't have a problem with CCF's ratings - he doesn't blanket rate everything a 1 star, but more ratings from people who actually invest a little time in getting things running properly would help.

I should rate more of my apps, but as I run most things from the menu via pnd_run, I don't immediately know what I've rated and what I haven't. Perhaps the system should not ask you if you've already made a rating, and only ask you if you've not. I guess you'd need to enter your api key into your system configuration somehow, unless it pulls it from your pnd_manager appdata, and the former could be a good way to opt in to my crazy rating system idea.

 

[milkshake]

The problem is not that some one voted 1star is that people that do like the app don't vote at all which skews the rating in favour of those that do vote. (in this case the low rating)

 

[Tenka]

Hmm, how about a system where the pnd_run script asks you for a rating/to update your rating when you exit a program?

Noooo!  Like the forced rating idea before, this approach relies on getting in the way, inconveniencing someone so that it's just as easy for them to rate as not to rate, yet causing not rating to be less convenient than before.  This is never nice!  Forced participation increases the percentage of participants who don't care, participants who don't care generally don't have anything helpful to say, and ratings only left because the user was slapped with the rating window again and again aren't going to be fair ratings.  Ignoring that though, it's still a popupy spamy dick move.

 

[FBnil]

For the repo I would like to search on unrated apps, this way we can at least rate those unrated apps, it is sad if you make a pnd (lots of effort, but nobody knows how hard it has been to get it compiled, fast enough, get it to working dimensions) and then nobody rates it... The other would be to make a pnd that looks at your usage of PND's and compiles that info to see usage (for example, in panorama, hint hint wink wink).

 

[milkshake]

U can easily sort the pnds by rating already and all the unrated apps will be at the bottom

 

[levi]

Forced participation increases the percentage of participants who don't care, participants who don't care generally don't have anything helpful to say, and ratings only left because the user was slapped with the rating window again and again aren't going to be fair ratings.  Ignoring that though, it's still a popupy spamy dick move.

That's why I suggested it ought to be an opt-in - if you don't set your repo API key on the NAND somewhere, pnd_run won't ask you for a rating. If everyone still thinks it's a popup spammy dick move, then nobody apart from me would enable it, so that's still a concern, but I hope it's not just me that would rate more if it was as easy as not rating (and it would only ask you once per PND).

 

[ekianjo]

U can easily sort the pnds by rating already and all the unrated apps will be at the bottom

Yeah but can't you do an inverted filter to put the unrated ones at the top ?

Forced participation increases the percentage of participants who don't care, participants who don't care generally don't have anything helpful to say, and ratings only left because the user was slapped with the rating window again and again aren't going to be fair ratings.  Ignoring that though, it's still a popupy spamy dick move.

That's why I suggested it ought to be an opt-in - if you don't set your repo API key on the NAND somewhere, pnd_run won't ask you for a rating. If everyone still thinks it's a popup spammy dick move, then nobody apart from me would enable it, so that's still a concern, but I hope it's not just me that would rate more if it was as easy as not rating (and it would only ask you once per PND).

Problem is, you don't rate something after trying it just once. Or at least, I don't.

 

[levi]

Yeah, that's true. I was thinking that on the first few play throughs I'd dismiss the dialogue and only settle on a score once I'd had another few plays and a think. The dialogue should have a cancel button to dismiss it for now, but not change anything so it reminds you again after subsequent playthroughs.

 

[_jr_]

I have been arguing in favor of the rating system and putting the blame on nonvoting users, but I now think that just removing it is really better than hoping for increased activity from the minuscule user base or trying to design a better system. It would be great, if Milkshake could find the time to implement number of downloads per version or even configurable time span, but since most apps are free, there is already more than enough information for an informed decision even without ratings.

 

[rygD]

The other would be to make a pnd that looks at your usage of

As long as it is something you have to choose to use.  I am a quite uncomfortable with the idea of something watching users.

Forced participation increases the percentage of participants who don't care, participants who don't care generally don't have anything helpful to say, and ratings only left because the user was slapped with the rating window again and again aren't going to be fair ratings.  Ignoring that though, it's still a popupy spamy dick move.

That's why I suggested it ought to be an opt-in - if you don't set your repo API key on the NAND somewhere, pnd_run won't ask you for a rating. If everyone still thinks it's a popup spammy dick move, then nobody apart from me would enable it, so that's still a concern, but I hope it's not just me that would rate more if it was as easy as not rating (and it would only ask you once per PND).

If there was a way to make rating things easier I would probably use it.  Maybe something (that you opt in for) that asks you to rate stuff you haven't rated when you go to update or look for new stuff on the repo.  Otherwise just thumbs up or something that doesn't give a single rater enough power to scare off potential users/buyers.

 

[elwing]

Just kill the rating dead horse... better to rely on reviews like Ekianjo's for newcomers then just look at the new/updated release in PNDManager...

 

[ekianjo]

Another way, which is not Pandora-centric, is to send emails to people who download stuff from the repo while already logged in. Send them a reminder one week after they download some stuff to make sure they have rated x or y application that they downloaded the week before, with a direct link to do so. 

Anyway, It's not a matter of five-stars rating vs something else, it's a matter of not enough participation anyway. We need more folks to vote, and it's good to think how to optimize it now for a small base size in the Pandora community, before we move to a hopefully higher base size with the Pyra next year. The repo is outdated in many respects (not pointing fingers at anyone but there's a ton more stuff that could be done to make it more modern and I hope someone will tackle it sooner or later). 

By the way, I don't believe that "small user base" is an all-encompassing excuse for the low amount of feedback received on the repo. I got more than 170 answers on the Pandora survey (results coming soon on Pandoralive) and that was WAY more time consuming than posting a comment on the repo. 

 

[Neelix]

By the way, I don't believe that "small user base" is an all-encompassing excuse for the low amount of feedback received on the repo. I got more than 170 answers on the Pandora survey (results coming soon on Pandoralive) and that was WAY more time consuming than posting a comment on the repo.

Your survey asked meaningful questions. A star rating on the other hand is completely arbitrary, so is essentially meaningless. I rarely rate anything in the repo because I generally find that I don't have any meaningful basis to assess how I should rate it. What this means in the end is that If I really like a program I'll rate it a 5, and otherwise I generally won't bother.

I still believe what we need is a system where the system itself inherently collects meaningful feedback. The structure of such a system would provide a basis for assessment. I believe such a system would actually encourage participation, but even if it didn't it would still provide more useful results because it would not be reliant on a large number of respondents to provide meaningful feedback.

- Neelix

 

[comradekingu]

I think something has to change about the system of voting on the repo then. Also i think the email voting should be opt-in.

Would be very nice if the first assumption pans out. The survey was nice because it was nicely laid out, and i feel my opinion matters, whereas the repo system seems somewhat broken.

This is how it is for me.

I give 1-3 stars.  I feel bad because im not about to re-visit and give something higher

I give 4-5 stars.  I am still taking away from a perfect score, which is the norm.

I give 5 stars.     Sometimes i do this, even if it should be 5.

For me the  -2  -1   0   +1   +2  system would solve this because it seems neutral

Maybe   Green Lightgreen Orange  Yellow   Red  In this order, because even if you get a green, its more noob friendly, it inspires growth rather than not contributing.  I think if you want to rate green or lightgreen it should come with required comment, because then those are most important. It makes trolling a bit harder too.

I think also the listings make it so. Maybe every app should be rated based on 5 ratings of the last version? If its less then 5, it gets put in a todo category. Maybe even the last 5 anyhow.

Fun to vote, cast many votes quick, system is very vibrant instead of stagnant.

No system is perfect, but i think something that works inherently does because people generally are nice and want to contribute.

 

[thatgui]

I don't think having a "like" rating is a good idea, as there is no real relation one could base an evaulation on: There is not necessarily a relation between the number of likes and the actual quality of an app, but rather the popularity, e.g. a flawed port of an emulator many people will like (like dreamcast) will probably have more likes than a "perfect" one for a less popular system (like SchneiderCPC).

With the long standing problem of minimal rating participation, I'm inclined, to join those that recommenond to remove them completly.

ptitsebs post remineded me of the biggest problem I have with ratings in the repo independent of how it actually is executed: What exactly should I rate ?

Only the technical part or also the "experience/satisfaction" I get from using it ? And there is also the problem that if I reduce my evaluation to only technical aspects, there is still the possibilty that one dev seems "offended" because I am not able to "see" what effort he put hin > Imagine port A gets a five star rating (or whatever else rating there may be) because the software just works as the it was a rather easy thing to port it to the Pandora. Port B only gets four stars, because it lacks certain aspects of convenience (like no fullscreen possible) to the end user, but the dev already spent several days getting this on the Pandora (reasons could be messy code, x86 based optimizations, etc).

A spontaneous idea was to implement some kind of "curation" system (that not necessarily needs to replace a rating system):

Every repo user has the ability to tag a repo entry with the tag "recommended", but needs to enter a small text to complete the process. Other users are able to see which users have recommended a particular entry and are also able to see some kind of user profile that contains a list of all entries that were recommended by the user. In an ideal case the user could also outline his gaming/pandora usage interest in an additional box, so people can get a rough idea what they can expect from a certain curator. This could of course be combined with other mechanics, to create more attention, like "following" a curator and getting notified if he recommends an entry or creating a list of "top" curators (but the term "top" should need some clarification beforehand).

This does not alleviate all problems (like the one I described in the first paragraph) but at least it gives the user another parameter to base its decisions on.