Repo rating system: 5 stars or binary ratings?

[Christoph.Krn]

My issue is that the rating system permit "negative rating"

The rating system does not permit negative ratings. It permits positive ratings between 1 and 5, but of course people will generally interpret low positive ratings as being negative.
That's kind of the entire problem, the quality of something as complex or even artistic as some software is cannot accurately be represented by a single number (well, technically it can, but I think you understand what I mean). Even switching to a likes-only system would still leave the rating data open to interpretation.

[...] I was imaginating the change 5 Strars -> Thumb up voting, with the following rules:

1. You can vote for 1 thumb or double thumb (just 2 thumbs instead of 1) ,or you don't vote (0 thumb) => Only positive rating here.

[...]

See? That's basically a 2-star system then. It's prone to the same interpretation problems.

That being said, a like/dislike system would make votes more easily interpretable than a stars system.

So instead of dropping ratings just because you dont like it, id like to see guys who complain come up with a BETTER solution for content discovery (and better as in, does not suck balls and does not take 10 times longer to find something worth checking out).

First of all, this: http://www.goproblems.com/test/wilson/wilson.php?v1=0&v2=0&v3=3&v4=0&v5=0
Then, the other thing is that many people come to the repository with the mind set on getting software. The process of going to the software they downloaded previously and rating it is not part of the workflow. Potential solutions to this could be:

• Celebrating a "Rating Day", say, once every two weeks.
• To users who are logged in to the repository, embed a small list of software that was downloaded >= 7 or so days ago, with 1-click voting options which make the respective software disappear from the list.
• Displaying "You have not rated this software yet. Really uninstall?" in client software such as PNDManager, and extending the upgrading with direct voting options.

...does anyone have any other suggestions that could go on this list?
Lastly, I would like for the detailed rating information (how many votes per star-level for each item on the repository) to be publicly accessible, so that the process of finding a better sorting method can be done on actual data.

 

[Christoph.Krn]

Sorry, this was missing from my previous post:

[...] an app with 1 rating will not affect the top ratings they need at least 4, it's true it will have a light blue star rating but all light blue ratings appear below dark blue ratings.

So THAT'S the difference between light blue and dark blue ratings. This is not very obvious, how are people supposed to know this? I would strongly advise to change this so that any ratings below the treshold are not shown at all.

 

[Granitehead]

• Celebrating a "Rating Day", say, once every two weeks.
• To users who are logged in to the repository, embed a small list of software that was downloaded >= 7 or so days ago, with 1-click voting options which make the respective software disappear from the list.
• Displaying "You have not rated this software yet. Really uninstall?" in client software such as PNDManager, and extending the upgrading with direct voting options.

...does anyone have any other suggestions that could go on this list

Have a dialog pop up when a .pnd is quit which has been mounted for a certain amount of time in total (store time in appdata). The dialog should have an easy way to rate the .pnd (even without internet access, auto-submit when there is access again) and a way to disable this functionality permanently, not just the dialog but also the logging of mounted time.

Have an entry in first boot wizard for this, making it possible to disable it before it first triggers. Another entry in system settings to (re)enable/disable it.

Substitute mount time for times mounted, if you want. Not sure what is better. The latter is easier to implement.

 

[Christoph.Krn]

T4b, the difference between these methods is that from a usibility point of view, modal dialogs are very disruptive (annoying, if you will), and should generally only be used for actions which are irreversible (or very expensive to reverse).

You're not going to get anyone to give meaningful votings by nagging people about it.

 

[ptitSeb]

@Christoph.Krn: the double thumb up is litterally 2 thumb up (in one vote), meaning you cannot (really) differentiente a thumb and double-thumbup (ok, odd/even can give you an hint), the point here is just to sum up the thumb up, not to calculate a mean value based on number of voters. But that's details, it doesn't really mater as we will not change anything anyway.

 

[Granitehead]

T4b, the difference between these methods is that from a usibility point of view, modal dialogs are very disruptive (annoying, if you will), and should generally only be used for actions which are irreversible (or very expensive to reverse).

You're not going to get anyone to give meaningful votings by nagging people about it.

I'm pretty sure you can. I know it works for me.Not by "nagging" users, as you say, but by reminding them exactly one time per .pnd that they could rate it if they want, after they quit it - so not disturbing them while they want to do something, but after they are finished.

Could look like this:

+----------------------------------------------------------+
| Publicly rate <xxx>? |
| Rate: <numeric rating> |
| Comment: |
| +-----------------------------------------------+ |
| | | |
| | | |
| | | |
| +-----------------------------------------------+ |
| |
| +-----------------+ +----------------+ +---------------+ |
| | Never ask again | | Rate next time | | <submit/quit> | |
| +-----------------+ +----------------+ +---------------+ |
+----------------------------------------------------------+
while no comment or rating is given <submit/quit> would be"Don't rate <xxx>" and would make the logging and asking stop for this .pnd

otherwise it would be

"Submit" and would additionally put the comment and rating in a queue which is regularly emptied/posted by a system service as soon as there is internet.

Have an entry in the settings where you can specify how soon it asks you and connect with a repo account (and of course disable it).

Thought behind this is that most users aren't unwilling to vote, but too lazy. This would make it much easier. I know I would appreciate something like this as a user, I like to vote stuff and give my opinions, but I'm also notoriously lazy and forgetful.

 

[b_o_b]

 Potential solutions to this could be:

• Celebrating a "Rating Day", say, once every two weeks.
• To users who are logged in to the repository, embed a small list of software that was downloaded >= 7 or so days ago, with 1-click voting options which make the respective software disappear from the list.
• Displaying "You have not rated this software yet. Really uninstall?" in client software such as PNDManager, and extending the upgrading with direct voting options.

I don't use PNDmanager for ratings but use the repo through the boards. I often don't even use a pnd manager at all and directly download the pnds. 

All additional requests just complicate matters and discussion about it will only lead to not changing anything. The basic complaint was we don't like to see bad ratings from folks who are just a PITA or newbies. Thumbs up seems very easy to implement and still gives you a pretty good impression what is popular. Any rating is subjective, you will only know if the software you are downloading is what you need if you try it.  Comments and number of downloads are much more valuable and give you a lot more information. And the thumbs up will give the user and dev a warm and fuzzy feeling

 

[Christoph.Krn]

T4b, are you using PNDManager, PNDstore, or repo.openpandora.org? If it is a combination, then I'd be interested in hearing which of these things you are generally using for which tasks (you can also send me PM if you want). If the same things which could be achieved with the modal dialog in your case could be achieved in other ways, this would be preferable.

b_o_b, there is more than one type of problem in the current voting system. It doesn't really make sense to look at one but not the other ones, if you want an effective and easy-to-implement solution which actually works, that is.

 

[Granitehead]

T4b, are you using PNDManager, PNDstore, or repo.openpandora.org? If it is a combination, then I'd be interested in hearing which of these things you are generally using for which tasks (you can also send me PM if you want). If the same things which could be achieved with the modal dialog in your case could be achieved in other ways, this would be preferable.

b_o_b, there is more than one type of problem in the current voting system. It doesn't really make sense to look at one but not the other ones, if you want an effective and easy-to-implement solution which actually works, that is.

I prefer milky, actually. Never had the need to try anything else (use the install everything option and just go through the apps in the menu. Can also be built for x86, so I generally use the better connection of my desktop for installing/upgrading). Although I have to admit I haven't been using my Pandora a lot recently.
I agree if something can be built in to a tool someone already uses then that might be better depending on preference, although I would still prefer being reminded just after using a .pnd - that's when I'll know best how to rate it and I'll take more time to write a comment when I'm just rating .pnds one at a time than if I have a whole bunch of "to rate" .pnds in a package manager.

Another idea: After closing a .pnd, add an icon to the panel (combination of semi-transparent voting icon and app icon) which stays there for an amount of time or until there are <max value> other such icons there. When you click on it you get a dialog similar to the one I described above, which allows to vote/comment, make the icon disappear and disable the whole behavior.

I think I like this one better than my first one.

Additionally: Make it configurable whether the voting reminders should reappear after an app/os update or not (also displaying your previous comments and rating because you probably don't know anymore).

 

[Rockthesmurf]

On Android, Play Store, it recommends software to you, on the front page, this is based on a few things, including: what software you have downloaded to date, what ratings you have given software, what software your friends are installing. As you can imagine, Google can see you have downloaded software X, Y and Z, and see that the majority of other users with the same software also downloaded software W and gave it a high rating.


The same page also asks you to rate existing software, once you do that, the request for rating changes into a recommendation. I have found it drastically increases the amount of ratings I do, as I want to see what the recommendation will be, and have to vote first.


I thought this is all with mentioning as not everyone will have used this store.

 

[Neelix]

Now, coming back on potential solutions, since ratings are qualitative and only meaningful when a certain threshold is reached, I would propose to make ratings "invisible" to the public (except the developer) until the threshold is reached. And a message such as "more votes are needed" should be displayed to encourage voting for those PNDs which do not have votes, instead of showing blank stars (which makes it look that it has ZERO stars, even though that's not its intention) or light blue stars based on a single rating, which is by nature meaningless because it's below the threshold.

This! I think this is an excellent idea. This should be implemented and combined with an incentive scheme like Steven just suggested. Though the threshold would need to be set at a statistically meaningful level. (4 ratings is not statistically meaningful)

Do this and I see one of two things happening. Either the incentive scheme would lead to enough voters to make a working statistically relevant ratings system, or it will quickly become apparent that a quantitative system is not suitable for this repo.

Personally I suspect that the latter is the more likely outcome but either way I see it as a step forward.

- Neelix

 

[KickAss]

Steven's and Neelix' suggestions might be a tiny bit of work to implement (but great ideas nonetheless).


As a first step, why dont we try to make the system a little more foolproof, stepping down from 5 to 3 stars:


3 stars = works without major issues and is totally usable on the pandora -> this marks a stable release and a comment on that rating is optional.


2 stars = may have issues, but does work as expected most of the time. may need some more polishing or pandora specific customization (like window size etc.) -> mandatory comment stating issues and possible quirks


1 star = at least I can't get it to work and i am facing issues which make the software not usable at all on the pandora -> mandatory comment stating issues and possible quirks


Also counting download numbers per version is a great idea.


Reaching a threshold before showing any rating to the public is just brilliant.

 

[ekianjo]

...does anyone have any other suggestions that could go on this list?

Give an incentive to rating as well.

-> the more you rate PNDs that you download, the more we can better understand what kind of applications you like and therefore provide automated suggestions -> provide recommended PNDs when logged in the repo.

On top of that:

-> rating and commenting through PNDManager is not easy for the beginner since they need to first get their API code through the repo then copy that key on their Pandora. Too many extra steps needed. Why not have an authentification integrated right into PNDManager to remove all that complexity and make the Pandora an autonomous system in itself for providing ratings and comments?

3 stars = works without major issues and is totally usable on the pandora -> this marks a stable release and a comment on that rating is optional. 2 stars = may have issues, but does work as expected most of the time. may need some more polishing or pandora specific customization (like window size etc.) -> mandatory comment stating issues and possible quirks 1 star = at least I can't get it to work and i am facing issues which make the software not usable at all on the pandora -> mandatory comment stating issues and possible quirks

You don't have to provide meaning to stars. Again, it's a quantitative system, not a qualitative one. It will make sense once you have enough feedback, no matter what people think 1 star or 5 stars mean. You end up comparing series of results, not whatever 2 or 3 means.

 

[KickAss]

...does anyone have any other suggestions that could go on this list?

Give an incentive to rating as well.


-> the more you rate PNDs that you download, the more we can better understand what kind of applications you like and therefore provide automated suggestions -> provide recommended PNDs when logged in the repo.


On top of that:


-> rating and commenting through PNDManager is not easy for the beginner since they need to first get their API code through the repo then copy that key on their Pandora. Too many extra steps needed. Why not have an authentification integrated right into PNDManager to remove all that complexity and make the Pandora an autonomous system in itself for providing ratings and comments?

3 stars = works without major issues and is totally usable on the pandora -> this marks a stable release and a comment on that rating is optional. 2 stars = may have issues, but does work as expected most of the time. may need some more polishing or pandora specific customization (like window size etc.) -> mandatory comment stating issues and possible quirks 1 star = at least I can't get it to work and i am facing issues which make the software not usable at all on the pandora -> mandatory comment stating issues and possible quirks

You don't have to provide meaning to stars. Again, it's a quantitative system, not a qualitative one. It will make sense once you have enough feedback, no matter what people think 1 star or 5 stars mean. You end up comparing series of results, not whatever 2 or 3 means.

I'd have to disagree.
Ppl should understand what they are voting for and why. Ppl browsing the repo should have a chance to understand why others voted the way they did. If we try to make the system a little less subjective (for example by providing a 3 options voting system with somewhat defined categories) we could get quantity and quality.


The problem with ratings like the ones by ccf is that they dont respect the prerequisites needed for the pnd to work (regardless if they were properly listed in the info section). Even tho he/she was given detailed instructions he/she did not manage to get it to work. Clearly that is not the fault of the rating system and can't be changed by redesigning the system.


If I download a 3 star rated pnd I can be sure it will work and/or its fun as long a i make sure all the prerequisites are met (because it's been tested by users). 3 stars mark a stable release (maybe asking the user for his os version by the time of voting may be a good idea) and may work as a tag for the best working pnds of similar content (for example different firefox or chrome pnds).


A 2 star rated pnd should work as well, but i might need to invest some extra time. Luckily I can read about the issues i'm about to face directly in the comments.


1 star rated pnds are not ready to be properly used, yet. From a dev point of view I can see that they need work or help (which i may be able to provide). The comments will serve me as a kind of development log and I can see what ppl are working on.


Rating with stars only, without somewhat defined circumstances per star and additional comments, will be useless. No matter how often you try to turn it.

 

[ekianjo]

The problem with ratings like the ones by ccf is that they dont respect the prerequisites needed for the pnd to work (regardless if they were properly listed in the info section). Even tho he/she was given detailed instructions he/she did not manage to get it to work. Clearly that is not the fault of the rating system and can't be changed by redesigning the system.

Does not matter at all. If we manage to have 20 or 30 votes, there will always be a percentage of CCF-like people who can't make it work and will complain and rate lowly. But that does not prevent the other opinions from adding up and being representative on the whole. That's the beauty of five scale ratings. It works no matter what you apply it to. Amazon does not tell anyone what 5 stars or 1 star mean or barely provide a vague guidance for it. Same for every store out there. Yet you can clearly see differences between products once you have enough votes. This is really just statistics, I don't know how else to explain it to you.

Rating with stars only, without somewhat defined circumstances per star and additional comments, will be useless. No matter how often you try to turn it.

I'm not a proponent of star-only ratings, by the way, I have been advocating for star + mandatory comments when rating.

 

[KickAss]

The problem with ratings like the ones by ccf is that they dont respect the prerequisites needed for the pnd to work (regardless if they were properly listed in the info section). Even tho he/she was given detailed instructions he/she did not manage to get it to work. Clearly that is not the fault of the rating system and can't be changed by redesigning the system.

Does not matter at all. If we manage to have 20 or 30 votes, there will always be a percentage of CCF-like people who can't make it work and will complain and rate lowly. But that does not prevent the other opinions from adding up and being representative on the whole. That's the beauty of five scale ratings. It works no matter what you apply it to. Amazon does not tell anyone what 5 stars or 1 star mean or barely provide a vague guidance for it. Same for every store out there. Yet you can clearly see differences between products once you have enough votes. This is really just statistics, I don't know how else to explain it to you.

Rating with stars only, without somewhat defined circumstances per star and additional comments, will be useless. No matter how often you try to turn it.

I'm not a proponent of star-only ratings, by the way, I have been advocating for star + mandatory comments when rating.

Lol dude...


You seem to take this personal and i have no idea why.


Quite clearly amazon does not have problems with the numbers of it's actively rating users. Well, obviously, we do. Strictly using a rating system based on mere statistics does not work too well for our repo. I thought we already agreed on that. I may be mistaken or may have misunderstood.


Compairing the openpandora repo with amazon seems a bit off, dont you think?


Pandorians are mostly tinkerers, we want stuff to work. I believe that quality input/ratings will be of greater value than just numbers. At least in our case.


As i said before, i do believe we could go for both without an aweful lot of work.


The repo is not google play nor the appstore. And I, for one, wouldnt want it to be.

 

[WizardStan]

who can't make it work and will complain and rate lowly.

And sometimes they will be right. Sometimes they'll follow the instructions to the T and it still just won't work for them. 30 people can do it fine and then that one it just doesn't for some reason. If you remove the 1 star or negative ratings you limit the ability for others encountering the same problem to go "me too" to such problems. Negative ratings aren't the problem, negative ratings can be just as valid and useful as positive. The problem is CCF who blames his inexperience on the developer.

 

[ekianjo]

Quite clearly amazon does not have problems with the numbers of it's actively rating users. Well, obviously, we do. Strictly using a rating system based on mere statistics does not work too well for our repo. I thought we already agreed on that. I may be mistaken or may have misunderstood.

It's not about that. Many people in this thread have been saying the 5 stars rating is meaningless, and I have answered so many times that the only problem is the number of people voting more than anything else. Instead of trying to fix something that is NOT broken (ie. the star rating), there are many things to do to encourage users to vote and comment more. As I said we should make ratings invisible until they reach a certain threshold, but keep them otherwise. When you have 20 votes on something it's clearly indicative of something, when you only have 2 it's not.

Removing the ratings is just going make things works for discovery (read my earlier posts) even if it pleases people who put PNDs on the repo. The thing is, the repo is aimed at users, it's a user centric tool. There are plenty of other ways to get feedback on your PND, such as posting in Software News on the boards, and I believe many devs actually do that and use that feedback there to improve their work.

who can't make it work and will complain and rate lowly.

And sometimes they will be right. Sometimes they'll follow the instructions to the T and it still just won't work for them. 30 people can do it fine and then that one it just doesn't for some reason. If you remove the 1 star or negative ratings you limit the ability for others encountering the same problem to go "me too" to such problems. Negative ratings aren't the problem, negative ratings can be just as valid and useful as positive. The problem is CCF who blames his inexperience on the developer.

Yeah, and I have no problem with that as long as many people are voting/commenting. CCF being often one of the only persons to vote/comment on the repo is one of the problem, we should foster more activity so that all voices get heard.

 

[sebt3]

It's not about that. Many people in this thread have been saying the 5 stars rating is meaningless, and I have answered so many times that the only problem is the number of people voting more than anything else.

Nope, you're missing the point. a 1 star vote mean for dev, your app is shitty and dont even reach the features it advertise. And this is very demotivating for many of us. When you've spend over 20h working for free on your free time on something, getting this kind of feedback cut the motivation rather hard because you know the features are all working : you tested every single one of them !

CCF being often one of the only persons to vote/comment on the repo is one of the problem, we should foster more activity so that all voices get heard.

I somehow doubt this. When CCF rate 1 star he comments on why. Not every single 1star vote have come with his comment. So there is a chance that someone else is also doing this but hidding behind CCF...
There is 2 solutions that have been proposed here that match what I do expect :

- the 1/2 thumbs up ptitSeb propose would do.

- keeping the 5 star system displaying how many votes for each level have and forcing a comment for evry vote below 3 stars would do too.

No dev wants to make this feedback system useless for the end-user. We just want it to be usefull and not demotivating so the apps keeps coming. Because that's what in the balance...

 

[ZXDunny]

Okay, we're fourteen pages in now.

What have we achieved? How is the current rating system being changed?

D.