|
|
| Improving the Level Rating System Thursday, July 29, 2010 12:36 am — Posted by Livio |
ShareThis« Livio's Blog
A growing problem lately is the arbitrary way in which we rate levels. Many levels seem to be getting overrated or underrated. This is actually quite a complex issue and there are several factors that play into it.
People have a strong tendency to give out their rates in an arbitrary fashion. A perfect example of this would be to look at your own ratings. As you look through them, you'll find that they aren't necessarily ranked from your most favorite levels to your least favorite ones, and you'll probably wonder why you rated certain levels lower than others when it should be the reverse. This seems to be an unavoidable issue, as it is impractical to compare every new rate you give out with all the ones you've made in the past. The problem would be more manageable, however, if we weren't such a small community.
In fact, this is one of the most obvious factors of the overall problem with our level rating system. The community is so small that everyone's rates have a large impact. Even the way we rate levels and videos is adapted to this issue: none of the rates are anonymous and you have to attach your rating to a post, in which people expect to include a valid explanation for that rate. In much larger sites, rating isn't given such a level of importance, and in some sites you're allowed to rate even if you don't have an account.
While working on the level database reform, I wondered if we should convert our rating system into something more conventional, where rates are anonymous and definitely not attached to posts but still require that you are logged into an account. I imagine that such a move would affect the level database in the following ways:
- Increased amount of ratings per level
- Decreased sense of importance given to each rate
- Ratings would become even more arbitrary than they are now, thus hurting the credibility of level ratings in general
Or rather, let's try thinking outside the box for a bit. The purpose behind the level rating system is so that people may find the best levels easily. So now let's ask ourselves: how can we better achieve that goal? There is a flaw with the current level-rating system in that the ratings are inherently inaccurate—no, not "inaccurate" but inconsistent. People rate levels on different standards, and therefore, they will want to play levels based on different standards as well. Instead of only offering one standard (overall quality), what we should do is provide multiple and more diverse ways to promote the best levels on the site. For example, offering a way to view levels that have won awards, or maybe ones that have been featured on the homepage.
And what if we set up more kinds of ratings? We already have quality and difficulty; perhaps we could add a Fun-Factor rating. Some might consider fun as being the most important factor taken into account when deciding on a level's quality rating, so wouldn't the addition of such a rating be redundant? Not really. The beauty behind a fun-factor rating is that it is indeed a second way to measure quality, but the list of "Most Fun" levels will [hopefully] be significantly different than the "Top-Rated" list. EDIT: Like I said in the discussion below: "fun rates may end up acting like a popularity meter, where fun levels that aren't necessarily well designed can still get acknowledgement." This thus creates another unique collection of the database's "best" levels, so that people will have a better chance of finding levels that match their taste.
Having three ratings could be bothersome, but lately I've been wondering about the usefulness of the difficulty rating. The idea behind it would be so that players could have an idea of how hard the level is before they tried it. However, it doesn't really promote the playing of more levels. Rather, it seems to just be a mechanism that stops people from playing levels that may be too hard or too easy. Furthermore, the "Hardest Levels" list may be useless in the sense that not many people would be thrilled to play evil levels. However, one redeeming aspect of it is that it presents the most intricate and crazily-designed levels of the database. But then again, so does the "Biggest Levels" list, which runs automatically without the need of user-input. So in the end, it seems like a good idea to get rid of the difficulty rating.
Finally, another major problem with our rating system is that everyone interprets the level rate numbers differently. Some consider a level rated as 7/10 as good or slightly average, while others interpret it as bad or mediocre. To fix this, the quality rating should be more like the difficulty rating, in which each range is given a name such as "Normal", "Challenging", and "Hard". Except that for the quality rating, it could be something like this:
- 10 - Super-Awesome!
- 9.0-9.9 - Amazing
- 8.0-8.9 - Great
- 7.0-7.9 - Good
- 6.0-6.9 - Average
- 5.0-5.9 - Mediocre
- 4.0-4.9 - Bleh
- 3.0-3.9 - Bad
- 2.0-2.9 - Terrible
- 1.0-1.9 - Fail
- 0.0-0.9 - Spam
The Conclusion
So here is the course of action that I think should be taken:
- Improve the quality rating to have named ranges, as stated in the list above.
- Remove the difficulty rating.
- Add a fun-factor rating.
- Add more options to help people find other kinds of "best levels", such as a list of Award-Winning Levels and Hompage-Featured Levels, and maybe think up of a few more similar ideas.
Edited Conclusion
After some discussion below, here's a constantly-updated list of the course of action, while leaving the old one in tact to protect the context of posts.
- Improve the quality rating to have named ranges, as stated in the list above.
- Remove the difficulty rating.
- Add a balanced-difficulty rating, where the higher the rate, the more balanced the difficulty.
- Add a fun-factor rating.
- Add more options to help people find other kinds of "best levels", such as a list of Award-Winning Levels and Hompage-Featured Levels, and maybe think up of a few more similar ideas.
User Comments (119)
| nebnebben | ||
| Swim for your life! Age: 104 Karma: 18 Posts: 257 Gender: Male Location: u.k pm | email |
Krotomo I like your addition of a fun factor; but I disagree with your quality system; since 7 isn't a bad rate, its a good one. And 6 is better than O.K its means an average level. And I don't think 5 is pretty bad; its just few weaknesss. http://www.youtube.com/watch?v=9VDvgL58h_Y Oh no he's after me! http://www.interguild.org/greatlakes.gif | |
| Isa | ||
| No. I'm an octopus. Age: 31 Karma: 686 Posts: 7833 Gender: Male Location: Uppsala, Sweden - GMT +1 pm | email |
Enjoyment ratings and the like have already been discussed a gazillion times, and there's no debating over the current standard ratings, so I don't see why you're calling out Kro on a suggestion he made two and a half months ago. | |
| rainalonto | ||
Age: 26 Karma: 8 Posts: 77 Gender: Male Location: UK pm | email |
6 is not an average level. I think an average level is about 7.5 | |
| Isa | ||
| No. I'm an octopus. Age: 31 Karma: 686 Posts: 7833 Gender: Male Location: Uppsala, Sweden - GMT +1 pm | email |
With our standards, yes. We don't make 5.0>-levels any more, hence the lack of those ratings. | |
| nebnebben | ||
| Swim for your life! Age: 104 Karma: 18 Posts: 257 Gender: Male Location: u.k pm | email |
Yeah but new people do. http://www.youtube.com/watch?v=9VDvgL58h_Y Oh no he's after me! http://www.interguild.org/greatlakes.gif | |
| Silver | ||
Karma: 121 Posts: 3581 Gender: Female pm | email |
Newer people make a lot of 5+ ratings because they don't fully understand the concept of criticism yet. | |
| Isa | ||
| No. I'm an octopus. Age: 31 Karma: 686 Posts: 7833 Gender: Male Location: Uppsala, Sweden - GMT +1 pm | email |
'nebnebben' said: Yeah but new people do. Indeed they do, and in those cases, it's good that we have the lower rates to describe their levels. | |
| Livio | ||
Age: 31 Karma: 470 Posts: 9620 Gender: Male Location: Arizona, USA pm | email |
so back to this discussion on how to improve the level difficulty rating system, we're going to make more simplified, like the karma system except you have three options to vote for: Easy, Medium, Hard. CSD thinks that this is too simplified, but I think it's good. Difficulty is an especially subjective thing, much more so than quality. So we don't need to place so much accuracy (and therefore, importance) into it. | |
| jellsprout | ||
| Lord of Sprout Tower Karma: -2147482799 Posts: 6445 Gender: Male pm | email |
Although I still think Challenging is a better name than Medium, I support this system either way. Spoiler: | |
| Livio | ||
Age: 31 Karma: 470 Posts: 9620 Gender: Male Location: Arizona, USA pm | email |
why do you think it's better? I kinda get the impression that it would be 1 easy option and 2 hard options. | |
| Isa | ||
| No. I'm an octopus. Age: 31 Karma: 686 Posts: 7833 Gender: Male Location: Uppsala, Sweden - GMT +1 pm | email |
"Too Easy", "Just Right", "Too Hard". That's my suggested names. | |
| DeathBunni X | ||
| Eww, school. Age: 26 Karma: 87 Posts: 690 Gender: Female Location: Midwest pm | email |
We're not making porridge.  | |
| Livio | ||
Age: 31 Karma: 470 Posts: 9620 Gender: Male Location: Arizona, USA pm | email |
But we need to use difficulty as a method for browsing levels, and no one would want to play levels that are too easy or too hard. And what if a level is meant to be hard? | |
| jellsprout | ||
| Lord of Sprout Tower Karma: -2147482799 Posts: 6445 Gender: Male pm | email |
Easy and hard are clear indications of difficulty. An easy level won't take any effort, while only the best or most determined are able to beat the hard levels. However, medium is pretty meaningless in the context of difficulty. It only says how the difficulty is compared to other levels. Challenging would imply that the level takes some effort to beat, but that it should still be very possible to beat by the average Joe. Something like Easy, Challenging, Impossible could also work. Spoiler: | |
| Livio | ||
Age: 31 Karma: 470 Posts: 9620 Gender: Male Location: Arizona, USA pm | email |
Or just Easy, Challenging, Evil, since impossible may not always be true. | |
| Quirvy | ||
| Â Â Karma: 655 Posts: 7753 Gender: Male pm | email |
Or just Easy, Challenging, Hard, since evil may not always be true. It could be a fair but hard level.  spooky secret | |
| Harumbai | ||
| [|]-X-[|] Age: 30 Karma: 260 Posts: 1743 Location: New Zealand pm | email |
Why does 3 options seem best? Could we not have Easy, Moderate, Challenging and Difficult? Upcoming HatPC level: Sanctuary, coming soon to an internet browser near you... | |
| Livio | ||
Age: 31 Karma: 470 Posts: 9620 Gender: Male Location: Arizona, USA pm | email |
we're going for a simplistic approach, and under that argument, it would be best for just two votes: easy and hard, but that may not always be true, hence the third option for medium/challenging. Adding a fourth option would really get away from the simplistic approach. | |
| canadianstickdeath | ||
Age: 35 Karma: 350 Posts: 2990 Gender: Male pm | email |
I still say we just change it from a /5 rating to a drop-down consisting of the current difficulty names. It'd get rid of whatever problems people have with an /5 rating with minimal effort on Livio's part. | |
« Forum Index < The Interguild Board
In order to post in the forums, you must be logged into your account.
Click here to login.
All games copyrighted to their respective owners.