Community discussion

Competition Scoring

Started by Queex on 18 August 2010 • 10,831 views

27 posts

#5478218 August 2010

So, the statistician in me yearns for a more rigorous approach to scoring. I'd thought I'd throw some ideas out for discussion for next year.

The problem with total rating is that it penalises games that, for whatever reason, are played by fewer people- being at the bottom of the list, being released later on in the year, and so on.

An average score is too susceptible to fluctuations when a game is rated by a handful of people- also if you assume authors will rate their own and rate it highly the fluctuation will tend to be upwards.

So, what I propose is this scoring system:

(Total rating) / (Number of ratings + k)

where k is some tuning parameter chosen to get the system you like best. This is similar to an average, but penalises games with fewer ratings.

If k=0, this is the average.
If k is large, the ordering is the same as for just looking at the total.

k=5 or k=10 look like they might be the sweet spot. You can see how this year's competition might have turned out with different scoring rules in this sheet: Scoring analysis.

#7139818 August 2010

*sits on the sidelines munching biscuits, waiting for the fun to start*

#7139918 August 2010

Can I have a biscuit?

#7140018 August 2010

they've all gone now. I've moved on to the crisps *throws you a packet*

#7140118 August 2010

[quote data-userid="23" data-postid="71398"]

*sits on the sidelines munching biscuits, waiting for the fun to start*

[/quote]

LOL i think youre right this could be fun and probably i should butt out seeing as ive only just started playing this years games and therefore didnt vote anyway.
But i also played most of last years games because my mother downloaded them and we played them together quite often and i know she did vote and neither of us felt the end result was unexpected.
As far as i can see your way hasnt made any difference at all to the top 2 and very little difference to the bottom 4 although its interesting to see that your own game did a bit better so maybe thats why you would like to see a change.
But at the end of the day everyone says the quality of ALL the games is marvellous.
If the prizes for being placed were on a scale of britains got talent i would agree a change might be in order but didnt everyone get something with prizes being donated by the very people who were also competing.
The games are given free and created by volunteers which obviously every1 is aware of before they enter so surely the fact that people play and talk about the games and are willing to come back and vote for them shows they are appreciated and thats the main thing really.
If the game creators were unhappy at the way the comp was run they simply wouldnt enter again but that hasnt been the case and in my humble and totally insignificant opinion when you take into account all the facts, experienced programmers, novice dizzyage user, long or short game play, style of game, to save or not to save etc the game that came 12th was every bit as good as the game that came 1st.
People vote on personal preference anyway for so many reasons so the end result isnt really a true gauge of whether 1 game is technically or visually better than another.
As for some games being played less than others again how a person chooses a game to play depends on so many things.
My mum always goes for Deltas games first and my sister by whatever title grabs her attention.
Me - i just download so many then play them alphabetically :teethy:

PS do you have any biscuits left i'll make the coffee!!

#7140218 August 2010

I think the competition scoring system is pretty good! :teethy: {yolkfolk}:animwink:

#7140318 August 2010

Well, this is supposed to be a competition - so I guess it wouldn't hurt to want to competitive?? :tongue:

Fascinating, that the different systems would tend to have most impact on:
- COTM a lot higher than it finished originally (i felt it deserved a top 3 spot)
- SD & TTD would fair significantly worse
- DGS games would be lifted from last place (which I felt they didnt deserve to be last anyway).

Good work Queex! :tup: Now we need to work on convincing the powers that be that a change is necesary - maybe we should have a vote on it....(assuming we can agree on a suitable system first...sigh...) :x_x:

#7140418 August 2010

Didn't understand a word of that, and the spread sheet made even less sense.

I don't see whats wrong with a the simple and easy voting system we have... score the games out of ten and thats it.. simple.
Sure it would be better of each game had the same amount of voters but then no eveyeone gets the chance to play them all.. part of the comp i guess to encourage players to play yours.. depends what you find appealing.

Plus we do this for fun...

Maybe if a new soring system would land me the number one spot id be all for it.. {eggs2}:denzil:

#7140518 August 2010

Personally I think prince has a good point when he says that it's not exactly Britain's Got Talent, with a huge prize available. At the end of the day, as Meph says, it's just a bit of fun.

The good thing about the current system is that it's simple to understand. No offense, but the 'weighted' system proposed is just too complicated!

Personally I think people play the game types they like. Maybe they didn't like Daisy having her own games? I certainly think that anything that departs from normal 'Dizzy' doesn't get played as much. look at the download numbers for BTD and CoTM compared with some of my other games to see what I mean. They are way below most of my other games. I think this may also apply to the Daisy games. There are people who simply want to play Dizzy. not Daisy or top-down or arcade games, just Dizzy.

This also shows in the fact that the game that came second - Dizzy Oops - was probably the closest in style to the original Dizzy games. I thought there were quite a few games that deserved to score higher than it, but that's just me. I've had the feeling for a while that a lot of the people who play the games are in Eastern Europe, and they seem to prefer games that deviate as little from the original 'Dizzy' style as possible.

*shrugs* I'm just hypothesizing. I could be completely wrong. Sure a change of voting system would have helped CoTM, but it would also have hindered ETD. I still don't see any compelling reason to change it. As far as I can see, people are suggesting change because they don't agree with the final outcome for whatever reason. But on that basis, you're not going to get a system that absolutely everyone agrees on.

Finally, let's not forget, it's not up to us at all. :wink:

#7140618 August 2010

Sorry Queex old boy, but I'm going to pull apart some of your arguments, because the statistician in you should really have exposed those arguments to the same scrutiny as you exposed the results to! :tongue:

[quote data-userid="291" data-postid="54782"]

The problem with total rating is that it penalises games that, for whatever reason, are played by fewer people- being at the bottom of the list, being released later on in the year, and so on.

[/quote]

That's a good argument in theory, however it falls down when you look at the numbers:

All the games were scored between 20 and 26 times, with two exceptions: The highest scorer (GID - 28), and the lowest (DGS2 - 18). Neither were at the bottom of the list on the comp page, indeed they were both right next to each other. In fact the two at the bottom of the list got the joint 2nd highest number of scores. So that can't be why.

As for being released later than others, you can see by the games list that 6 games were released after DGS2, including GID - the winner. So that argument doesn't hold up either. There are also (currently) 3 other games that were in the comp that currently have less downloads than DGS2.

I would refer you to my post above as to some reasons why I think that people choose to play one game over another. I will however add another thought. It's like books. Everyone has a favourite author. Colin's games always seem to get more downloads than other games released around the same time, as do Huckleberry's. And how else do you explain ETD - which had the lowest average vote, and let's face it, wasn't brilliant - currently having the joint second highest number of downloads of the 8 games released in the run up to the comp? Even allowing for multiple downloads (due to updates), it's still got quite a few more than other games.

[quote data-userid="291" data-postid="54782"]

An average score is too susceptible to fluctuations when a game is rated by a handful of people- also if you assume authors will rate their own and rate it highly the fluctuation will tend to be upwards.

[/quote]

firstly, yes authors will rate their own games, but usually either honestly, or with a 10 (!). if they all rate then as a 10, then everyone is equal. If not (and I hope not!) then the fluctuation won't trend upwards.

secondly, you say a 'handful of people', but as I said, most games were rated by between 20 and 26 people. If you call 20 a handful, then what is 21? or 22? or 26? It could be argued that they all only got a handful of votes. It is up to the game designer to design a game that people feel worthy of spending their time playing. If they don't like it (for whatever reason, see my points above), then they will be less likely to rate it highly, or at all.
[quote data-userid="291" data-postid="54782"]

So, what I propose is this scoring system:

(Total rating) / (Number of ratings + k)

where k is some tuning parameter chosen to get the system you like best. This is similar to an average, but penalises games with fewer ratings.

[/quote]

I hate to point it out, but the current system already penalises games with fewer ratings...!

Anyway, that's my 2c worth. Nothing personal, just the statistician in me! :tongue:

#7140718 August 2010

I must agree with most of the comments above, and that games voted for the least amount of times are penalised, but as Delta says this could be fair by virtue of the fact that someone doesnt like that style of game. In my opinion, the biggest problem is the lack of voters. It seems a little disappointing to me the number of votes compared to the number of downloaders.
At the end of the day, the only way you could make the voting fairer would, I think, to have a panel of voters who all vote on ALL the games, so they all get the same number of votes, and then if its a style of game you dont like you can give it a low score, but at least you score it. And who knows, you may actually enjoy that game? COTM is not really what we expect a Dizzy game to be, but i personally am really enjoying the saga and think that it is a very good game, so my initial impression may well have been wrong. (Shame they are so complex that Jamie cant produce them more quickly!). And I do not think that the Daisy games deserved to come bottom - they were too good a quality in my opinion.
At the end of the day however, as stated by people already, it is just a bit of fun, all the games were very good in my opinion, and whatever the scoring system used, there will be possible problems / disparities, and certainly disagreements as to what should have won!

Lets not forget, the real winners are all of us who love playing the games :tup:

#7140818 August 2010

A point you made.
"...in Eastern Europe, and they seem to prefer games that deviate as little from the original 'Dizzy' style as possible..."

Not sure I agree with this considering for the past 3 years the winners have been games that have been very different to the orignal dizzy games.. SMB and GID were far more colorfully and graphically pleasing than anything offical..as was VID.

Anyway.
As i said once before a game like COTM is either a love it or hate it.. due to its' format regrades of how well executed it. (personally it was my fav of the bunch) i liked DGS it was a nice game and i enjoyed the novelty of using Daisy but i didn't like DGS2. I didn't like the format of it.
Im sure voters have there own preference, Games like Dream home and YTTR or even Same with starship or Knightmare Dizzy. (if you didn't like tv show then you might have not enjoyed this)

Theres no patten or rules really...im still surprised Swampland did better then Feaverbound... this could be an example of the older game getting more players (due to its release) i guess but who knows....

At the end of the day it's only a bit of fun..keeping Dizzy alive and keeping this community alive. Building, playing them, voting them.. none of the games have been duds. This is also why i thought it would be better if members were only allowed to enter one game (any one from that year but just one, you summit your best) makes it more accessible and easier.
Plus if you have more than one game someone has to come top and someone has to come bottom.. lol And its great your all experimenting, and making the games you want to make.. and not making the games that might pull in votes.. i just think its a lot fun.

It seems the Dizzyage easter comp has so much more debate than any other world wide competition.. LOL.. Argh.. lets just get on with it and enjoy it.. Other wise we may as well scrap the thing. :sad:

Next year...next years the one i'll be remembered for. {yolkfolk}:animwink:

#7141018 August 2010

[quote data-userid="55" data-postid="71408"]

A point you made.
“…in Eastern Europe, and they seem to prefer games that deviate as little from the original ‘Dizzy’ style as possible…”

[/quote]

oh I realise that, but how else do you explain Dizzy Oops coming second? (other than people just preferring it of course!). I read a translated forum once where they were discussing TLoD, and they basically said that anything that wasn't classic style Dizzy, wasn't Dizzy! see this thread: https://www.yolkfolk.com/bb/showthread.php?t=868 for further evidence of that.

and I wouldn't say the winners have been very different from the originals, just more colourful! Graphically a lot better, but still 'in-keeping' as it were.

#7141118 August 2010

Dizzy Oops made a nice change i think.
it wasn't my number 2, however it was the only game from Eastern Europe so maybe theres some truth in what you say.. but they still voted GID better.
Im sure if was terrible and it got the same votes we would have to look into it.. lol

Oh it's taking 3 differnet web browers and almost 20 minutes to get anything posted.. i keep getting a timed out message...on every one i use, in fact this is been a good spell of posting for me then it nornally lets me.. Been like this since easter and it still hasn't sorted itself out.. am i the only one?

#7141218 August 2010

[quote data-userid="55" data-postid="71411"]

am i the only one?

[/quote]

yes. :tongue:

have you tried ticking 'remember me' when you log in?

otherwise, check the usual stuff. do you have cookies enabled (I would imagine so)?

but three different browsers makes that seem like a computer problem. Does your connection keep dropping? do you have a firewall? over-zealous Anti-virus?

#7141318 August 2010

[quote data-userid="23" data-postid="71406"]

Sorry Queex old boy, but I’m going to pull apart some of your arguments, because the statistician in you should really have exposed those arguments to the same scrutiny as you exposed the results to! :tongue:

All the games were scored between 20 and 26 times, with two exceptions: The highest scorer (GID – 28), and the lowest (DGS2 – 18). Neither were at the bottom of the list on the comp page, indeed they were both right next to each other.

[/quote]

It's not a counter argument. If there are two games of a similar quality, one rated 18 times and the other 28 times, the latter will get something of the order of 75 more points. That specific games with low and high numbers of ratings placed next to one another in the competition means nothing.

[quote data-userid="23" data-postid="71406"]

As for being released later than others, you can see by the games list that 6 games were released after DGS2, including GID – the winner. So that argument doesn’t hold up either. There are also (currently) 3 other games that were in the comp that currently have less downloads than DGS2.

[/quote]

It doesn't have to be a uniform effect to still have an impact on scoring.

[quote data-userid="23" data-postid="71406"]

I hate to point it out, but the current system already penalises games with fewer ratings…!

[/quote]

Yes, but by more. The idea was to have middle ground between an average score and a total points.

In this year's competition, we had Hide and Seek Dizzy losing out to Swampland Dizzy by only 4 points with one fewer rating- and the average rating for both was around 7.5. It doesn't sit right with me- and it's not the only placing where that happens.

Rating things out of 10 is fine, but using the total score is pretty much the worst possibly way you could combine the scores.

I didn't think the approach was too complicated; but using the average would still be an improvement. The issue is no big deal, but if I wasn't a perfectionist about this sort of thing I wouldn't be working in stats. {yolkfolk}:bigwink:

#7141519 August 2010

Okay im sticking my neb in even further {yolkfolk}:tongue:

Only vote once for the game you liked best. Assumes all games have been played.
Not everyone has the time or the concentration level to play and complete everything in time especially if there are a lot of entries and some are released late in the game.

Allocate points to your Top 5 - Eurovision Song Contest style - 5 points to your favourite, 4 the next and so on down to 1 point.
Again assumes all games have been played and that there are more than 5 entries.

Vote for games on content then average the games total score by the number of players who voted for it.

This method gives credit where its due and allows for the fact that not all games are played.
Weve all played games that looked beautiful but the storyline sucked and vice versa.
You would be voting for what was important to you in a game and those who were not particularly advanced programmers could still score highly if their other content was good.

Maybe competition entries could all be released on the same day
They all have an equal opportunity to be played in the given time period.
No concerns about releasing a game early on and it possibly being forgotten or releasing a game late and people not having a chance to play it at all.

Allow games to be released and voted for all year with the final result being announced on a set date
For example release game between 1st April 2010 and 30th March 2011 and results finalised on 31st March 2011.
Players can vote for the game as soon as theyve completed it and its still fresh in their mind before they move on to the next on their list.
It stands to reason that if a creator wants their game to be given a decent chance he wont release it 2 days before the comp ends but he wont be at a disadvantage by releasing early on either.
This also allows newer members to get involved earlier which in turn encourages them to stick around if theres something going on all the time {yolkfolk}:clap:

I was thinking too that even the donation of prize money could become a competition in itself.
"Member 1 gave £10 last year so im going to give £20 this year" kind of thing.
Why not have a paypal account purely for prize money so everyone can give as much or as little as they can afford each year (anonymously even) and the total fund given to the winner/s or distributed how the organisers see fit.

Im also thinking ive got too much time on my hands and taking this far too seriously now.
There is no such thing as a perfect system and whatever method is used someone somewhere will find a flaw in it.
So maybe if its not being done for the fun of it all the answer is not to do it!
And 'if it aint broke - dont fix it'

#7141919 August 2010

Some good ideas, fun debate, but I think a touch of realism is required.

I wonder if the reach of the dizzyage comp may have peaked this year? {yolkfolk}:puzzled:

The comp struggled to attract up to 30 voters per game and the number of developers submitting entries can be counted in single figures.

So the issues around voting systems etc, may have some good ideas behind them but even if implemented they will make very little impact.
I like the idea of a centralised prize fund - a virtual treasurer. It would make things more secure, but may only be necessary for larger sums or many more sponsors. Given that as far as I know all entrants received their prizes without trouble, very quickly, what would be the advantage of a change?

What may be the biggest driver of change is if this community grew from its present small closeknit family of egg-loving nerds to a global phenom.
It may happen. Perhaps.

The real question is what do we want this community/comp to change into?

On balance, its nice to dream but maybe I like being a small time nerd in a closeknit group...
{eggs2}:denzil:

btw, Meph,
sounds like your working on something special....

#7142019 August 2010

Given that as far as I know all entrants received their prizes without trouble, very quickly, what would be the advantage of a change? Ive not received anything yet but hoping they be on there way on both fronts at some point...

As regards the comp, your not gonna get a perfect scoring system because not all players vote (as you mentioned struggled to get up to 30) yet my games have many downloads...and those that do won't necessarily have played all games anyway...so the idea of scoring down in order they would be forced to rate games theyve not played, then as mentioned you get games that score similar as an average but may have more voters....the points can go on and on...

I agree with Delta & if im reading correctly yourself, I would just leave as is, is just a bit of fun, anyone one of us could find a particular reason where we think it may not be fair, me personally im not particular too fussed with the comp, as ive always mentioned giving fans a game they can enjoy and having the feedback they have is enough for me, if it causes too much angst I would suggest not to have the comp and just release games for the enjoyment of others...but I stress I appreciated all those that voted for mine and yes of course your pleased if it gets recognised with the votes and does well...it's a pleasing bonus..

Plus I bet I could come up with a system (just needs a clever formula) that could get 1st down to the bottom and 12th up to the top...would be so many different ways... :tongue:

#7142319 August 2010

Right, I think the general consensus is that there is unlikely to be a perfect voting system that would please everybody, and that to change the system from that used this year is likely to be impractical and gain nothing.

However, what do people think about having categories to vote in? For example, for each game you could have votes out of 10 for originality, quality of puzzles, graphical design, technical achievments etc with the total of those scores then becoming the overall score for that game. I for one had a bit of trouble scoring some games this year, eg Mephs game i really enjoyed and loved the cleverness of the programming to build the scrolly bit, arcade bit and upside down bit into a normal Dizzy game, so would score highly in a 'technical achievement / originalty' type of category, but at the same time (when I played it) there were still some rough edges (spelling errors / bugs) which, whilst they didnt really detract from the gameplay, did take the gloss of the whole thing. Whereas with Oops, again whilst a nicely weighted game especially for a first game, I thought that the similarity to the original games in a way detracted from the overall experience, even though it was technically very competent with no errors (that I noticed)

I appreciate that this would make the voting slightly more complicated again, but nothing a simple spreadsheet couldnt sort. And if only 30 people are voting, wouldnt really make totting up the scores too much harder.