Feedback on Repo.OpenPandora.org

[iprice]

I personally don't like the scoring system as it is, but under no circumstances should there be a thumbs down option. People could spend months working on something and just receiving thumbs down, with no explanation is not going to motivate the user, increase or improve their software output. If something is bad, tell the author/porter - don't just neg them and leave them hanging.

I would never give anyone a thumbs down, knowing how much time and effort can go into even the simplest of apps.

I'm not sure there is any one system that will make everyone happy.

 

[hedwards]

Will the new repo be open source, or just the backbone library? It would be nice if everything would be open source, so that others can add features too.

Shake, could stars-rating be non-anonymous for app maintainer in repo [i dont mind it if be globally non-anonymous]? please please.

Promoting commercial apps more [bigger banners and info], would be better as well.. 

Non-anonymous ratings are not a good idea, it's easier to be honest if it's anonymous. I know it sucks if someone leaves a bad rating and you don't know who it is or why they give that rating, but giving up anonymous ratings is not worth it, to me.

I don't think commercial apps should be promoted more than non-commercial ones. At least not on the repo itself. I certainly don't want advertisements on the repo website.

http://repo.openpandora.org/?page=detail&app=ultimate-pim-app and check rating

No single info why, just fuck'd 1-star rate with smile..

Thats why i think non-anonymous rating is a must

Thats why i proposed a non-anonymous rating on repo. If you re honest , why you would want to use anonymous rating? 

And what are you planning to do, haras the person that gave the bad feedback? Barring the case that they make a comment like Conrad does, you don't really have any way of responding directly to a star rating anyways. What's more, if there's substantial negative feedback building up, you've probably got more to worry about than a few rogues abusing the rating system.

In any case, if people like the software and they see that it has a terrible rating, it's likely that they'll leave a comment to that effect.

I'm just not sure what good could possibly come from mandatory non-anonymous ratings that would make up for squashing reviews from people that don't want to leave their name.

 

[ptitSeb]

I think the current system works well enough. The only problem is the small number of ratings, but that wouldn't be any better with a positive only system. Are two (three, ten, 100) thumbs ups an indication of a perfect PND with (very) few users or does it have serious problems? And I don't see non-anonymity increasing the number of votes either.

And would a 1,2 or 3stars vote indication of a broken PND ?

Anyway, I don't move the system, I don't care. As stated above, I ignore that star rating system so I can live with it.

 

[Hồng Thất Công]

Instead of star-rating system, how about Good, Very Good, Very Very Good, Extremely Good, Superbly Good?  This way it's good all-around GOOD 

 

[comradekingu]

The white on light gray and gray on darker gray text is low contrast. How about white on dark gray and vice versa.

Also the comment system on the wiki is nice, it allows you to leave feedback, which is basically a comment attributed by a rating, so that its constructive, and you can do something about it.

The hardest things to grasp are often the ones others think are too obvious to point out. Having a system that helps the process works wonders.

Then maybe if you later on see a new version of the thing you rated, maybe the rating will change, so its more dynamic that way.

 

[Rockthesmurf]

Something that puts me off the 'thumbs up' (and no other options) is it doesn't give much scope for comparing applications. I could perhaps see see something like applications can have 'badges' (or something similar), initially the badges could be 'Good', and 'Excellent'. End users could:

1. Report issue to maintainer, this is the preferred way to report feedback. If the feedback isn't addressed and the user feels the feedback should be read by other users, they can leave a comment.
2. Do nothing, this doesn't say anything good or bad about the application.
3. +1 the 'Good' badge, if the application gets some threshold (number of goods, or number of goods vs number of downloads) it receives the good badge, clearly visible on the applications page
4. +1 the 'Excellent' badge, which would work in the same way as the good badge

The notion of 'badge' doesn't feel quite right, but the idea is to have a system where:

• You can differentiate really good software from 'normal' good software (there is lots of good software, but I imagine most users initially want to check out the really good stuff)
• The 'default' state of an application shouldn't have a negative feel to it (a 3* rating on the current system feels bad to some contributors, where as the 'default' state with badges has nothing negative, the software just doesn't yet have a badge)
• Problems can be reported to the author, it should be encouraged for issues to be addressed this way where possible, rather than the authoring checking the comments to find out their hard work is getting a public bashing

There could be other badges too, although they would require more work, so probably aren't worth while, but it would be cool if application has 'badges' for things like:

• C4A support
• Native control support
• Full screen support

I don't think these solutions are quite right, but just throwing something into the thread for others to think about.

 

[comradekingu]

Lets not hide behind general meaningless variations of a interpreted scale. Differentiate voting on stability, gameplay, and aesthetics with points attributed to each in relation to the _actual perceived value_.

Then do some logic and break down voting on an average of less than what would constitute a huge shift in a few votes, which also prevents much of the leverage to cheat.

First 5 votes to establish a rating. Cut the best and the worst, and then do average of the remaining 3. Which means you cant sway something significantly on your own.

If there are more versions, also take into consideration prior releases. This makes for a system where you are encouraged to fix things, and its implied that it is, instead of having to change ones vote. A current underrepresentation of a historic issue means its plausibly solved rather than anything else, thats what we can take from it.

On the other hand, if it was good, and is now worse, why rest on your laurels. Make older versions available so that you can get the one that works for your intended use, rather than the latest broken one. Source helps, bugreports implemented as close to metal as possible helps.

Lets say anything beyond 5 votes on a new version is a new average, mixed with averages of prior versions in descending order, maybe the older one should be half, for each revision, with a maximum of 5 concurrent versions. Release often, release well.

I can understand fully that people are discouraged from working on something if the first few 5 votes on a project are 3 star ones due to the fact its alpha or something.

If you aggregate that to a system that benefits the best rated/downloaded and only shines light on others if they are new to the table, it is a system that does the opposite of what it should do:

Help development and interaction amongst developers and audience.

The big projects are successful in their own accord, its about the smaller things. Today i would say the model is, deliver something good and polished, generate positive coefficient feedback and rise above the noise, in turn making the propagated picture worse in its inherent aid to the already able.

Some level of this is good, but i think there could be more credit to go around where credit is due.

 

[bzar]

Just a thumbs up system means no differentation between an awesome application used by few and bad application used by many, liked by few. IMHO any rating system needs to take this into consideration somehow.

EDIT: Indicating that a piece of software is bad only through comments doesn't work either, as it doesn't affect the software's total rating. Requiring comments for at least bad ratings would be good though.

 

[ekianjo]

Just a thumbs up system means no differentation between an awesome application used by few and bad application used by many, liked by few. IMHO any rating system needs to take this into consideration somehow.

Yeah binary systems are crap for evaluating stuff. Seriously people sometimes here are clueless about what they recommend.

 

[ekianjo]

I can understand fully that people are discouraged from working on something if the first few 5 votes on a project are 3 star ones due to the fact its alpha or something.

The main problem with the current repo rating is not the rating system itself, it's the fact there are not enough votes for each application. So no matter what trick you want to implement, you can't get good quality results with close to zero data. That's statistics 101, you need a reasonnable sample size to compare stuff. Most applications on the repo have less than 10 votes, it's borderline meaningless.

You need to have at least something like 30 votes per application in order to do any kind of statistical work with it.

 

[_wb_]

The amount of votes is one problem, but another is what it means to give 1,2,3,4 or 5 stars. When you are talking about Michelin stars for restaurants, one star is already quite an achievement -- but there the default is 0 stars while on the repo, you cannot give less than 1 star. Is 3 stars a good rating, or is it a bad one? If you want to get ranked high, anything other than 5 stars will pull you down. In fact it is better to get 5 5-star ratings than 999 5-star ratings and one 4-star rating.

 

[ekianjo]

In fact it is better to get 5 5-star ratings than 999 5-star ratings and one 4-star rating.

That's a sampling issue. You should not do comparisons when sample sizes are so different. It's insignificant, statistically wise. The threshold for comparison should be much higher than the current one.

 

[bzar]

_wb_: using median instead of average will take care of that.

 

[ekianjo]

_wb_: using median instead of average will take care of that.

Nope, the median of 5 scores of 5 will still be 5. You can't trump the lack of data.

 

[_wb_]

In fact it is better to get 5 5-star ratings than 999 5-star ratings and one 4-star rating.

That's a sampling issue. You should not do comparisons when sample sizes are so different. It's insignificant, statistically wise. The threshold for comparison should be much higher than the current one.

I agree that you cannot compare things with so different sample sizes, but people also want to see a list of "top rated" PNDs, which means you somehow need to compare anyway. The main advantage of a "thumbs up count" instead of a "rating average" (or median or whatever) is that you rank by the number of people who like it, not by the average rating left by those who cared to leave a rating. That means it's more about popularity than about quality, but that's not necessarily a bad thing. You can still compute a derived ranking based on thumbs up per download if you want something that does not give a disadvantage to niche applications that most people don't use.

 

[ekianjo]

In fact it is better to get 5 5-star ratings than 999 5-star ratings and one 4-star rating.

That's a sampling issue. You should not do comparisons when sample sizes are so different. It's insignificant, statistically wise. The threshold for comparison should be much higher than the current one.

I agree that you cannot compare things with so different sample sizes, but people also want to see a list of "top rated" PNDs, which means you somehow need to compare anyway. The main advantage of a "thumbs up count" instead of a "rating average" (or median or whatever) is that you rank by the number of people who like it, not by the average rating left by those who cared to leave a rating. That means it's more about popularity than about quality, but that's not necessarily a bad thing. You can still compute a derived ranking based on thumbs up per download if you want something that does not give a disadvantage to niche applications that most people don't use.

So please explain how I'm supposed to rate something which is OK, but needs improvements to become better? For example, custom controls, additional features, etc?

Thumbs up is really a retarded system because you can not say anything else that "Great" or "nothing". Thanks god we at least have a scale. I frequently revise the score I gave to an application once it's improved, and thumbs up is never going to work in that way.

 

[comradekingu]

If each vote has a bugreport, or is, in being somewhat telling in the three scales (gameplay, aesthetics, stability) with a means and a variation of two on each side (5 stars), it doesnt have to be an accumulated approach, in where you need masses of data to reconstruct a trend opinion.

ekianjo
You vote less on the first, then add a connected comment explaining things. (if you want to complain about something specific, make it specific, not a rating value)

Then if there is development on the  matter, you can be prompted to update your vote too.

Making the values -2 on one (-1) side 0 average (1) and +2  on the other to take care of the number bias, people are more likely to vote middle then.

Stability could be 0 by default with the option to subtract 2.

It makes for more valuable categories, "most aesthetic" for example, in whatever other boundary you are interested in.

If its a sortable meta-table it makes it easy to find info like what is the most positive puzzle game after date .... or which game is the most downloaded in a compo.

More things to compete for and nerd out about. We all enjoy statistics.

For programs gameplay means function and aesthetics means intitivness/use with stability being the same.

I often find myself deciding against voting just because its so discouraging to the dev. My input to change to for example a new version of a library, be lost in the comment section pages, largely unconnected to my vote. And thats best case, which is i state my view, rather than just vote.

(We already have forum threads serving this purpose, connect the two in a manner that makes more sense.)

Voting 5 now on something thats already 5 does nothing but solidify that 5, which is better generally, but also has ups and downs as i mentioned before.

Voting now means either you generate 5s or you are discouraged from trying to acheive it, because its not possible once you accumulate anything less.

In turn making it a binary system with no negative metric, either its 5, or its silence.

 

[ekianjo]

If each vote has a bugreport, and is actually somewhat telling, it doesnt have to be an accumulated approach.

Then the barrier to vote is much higher and you will get a few votes as you have comments.

 

[milkshake]

Each app could also encourage ratings like in android apps, they could even vote from inside the apps if the app author adds support for the repo api.

 

[bzar]

_wb_: using median instead of average will take care of that.

 Nope, the median of 5 scores of 5 will still be 5. You can't trump the lack of data.

Well, yes. But there will always be differences in the sample size between applications. That means that with averaging one vote has more weight in some ratings than others. This is what the median is supposed to fix, not the small sample size.