|
|
| Improving the Level Rating System Thursday, July 29, 2010 12:36 am — Posted by Livio |
ShareThis« Livio's Blog
A growing problem lately is the arbitrary way in which we rate levels. Many levels seem to be getting overrated or underrated. This is actually quite a complex issue and there are several factors that play into it.
People have a strong tendency to give out their rates in an arbitrary fashion. A perfect example of this would be to look at your own ratings. As you look through them, you'll find that they aren't necessarily ranked from your most favorite levels to your least favorite ones, and you'll probably wonder why you rated certain levels lower than others when it should be the reverse. This seems to be an unavoidable issue, as it is impractical to compare every new rate you give out with all the ones you've made in the past. The problem would be more manageable, however, if we weren't such a small community.
In fact, this is one of the most obvious factors of the overall problem with our level rating system. The community is so small that everyone's rates have a large impact. Even the way we rate levels and videos is adapted to this issue: none of the rates are anonymous and you have to attach your rating to a post, in which people expect to include a valid explanation for that rate. In much larger sites, rating isn't given such a level of importance, and in some sites you're allowed to rate even if you don't have an account.
While working on the level database reform, I wondered if we should convert our rating system into something more conventional, where rates are anonymous and definitely not attached to posts but still require that you are logged into an account. I imagine that such a move would affect the level database in the following ways:
- Increased amount of ratings per level
- Decreased sense of importance given to each rate
- Ratings would become even more arbitrary than they are now, thus hurting the credibility of level ratings in general
Or rather, let's try thinking outside the box for a bit. The purpose behind the level rating system is so that people may find the best levels easily. So now let's ask ourselves: how can we better achieve that goal? There is a flaw with the current level-rating system in that the ratings are inherently inaccurate—no, not "inaccurate" but inconsistent. People rate levels on different standards, and therefore, they will want to play levels based on different standards as well. Instead of only offering one standard (overall quality), what we should do is provide multiple and more diverse ways to promote the best levels on the site. For example, offering a way to view levels that have won awards, or maybe ones that have been featured on the homepage.
And what if we set up more kinds of ratings? We already have quality and difficulty; perhaps we could add a Fun-Factor rating. Some might consider fun as being the most important factor taken into account when deciding on a level's quality rating, so wouldn't the addition of such a rating be redundant? Not really. The beauty behind a fun-factor rating is that it is indeed a second way to measure quality, but the list of "Most Fun" levels will [hopefully] be significantly different than the "Top-Rated" list. EDIT: Like I said in the discussion below: "fun rates may end up acting like a popularity meter, where fun levels that aren't necessarily well designed can still get acknowledgement." This thus creates another unique collection of the database's "best" levels, so that people will have a better chance of finding levels that match their taste.
Having three ratings could be bothersome, but lately I've been wondering about the usefulness of the difficulty rating. The idea behind it would be so that players could have an idea of how hard the level is before they tried it. However, it doesn't really promote the playing of more levels. Rather, it seems to just be a mechanism that stops people from playing levels that may be too hard or too easy. Furthermore, the "Hardest Levels" list may be useless in the sense that not many people would be thrilled to play evil levels. However, one redeeming aspect of it is that it presents the most intricate and crazily-designed levels of the database. But then again, so does the "Biggest Levels" list, which runs automatically without the need of user-input. So in the end, it seems like a good idea to get rid of the difficulty rating.
Finally, another major problem with our rating system is that everyone interprets the level rate numbers differently. Some consider a level rated as 7/10 as good or slightly average, while others interpret it as bad or mediocre. To fix this, the quality rating should be more like the difficulty rating, in which each range is given a name such as "Normal", "Challenging", and "Hard". Except that for the quality rating, it could be something like this:
- 10 - Super-Awesome!
- 9.0-9.9 - Amazing
- 8.0-8.9 - Great
- 7.0-7.9 - Good
- 6.0-6.9 - Average
- 5.0-5.9 - Mediocre
- 4.0-4.9 - Bleh
- 3.0-3.9 - Bad
- 2.0-2.9 - Terrible
- 1.0-1.9 - Fail
- 0.0-0.9 - Spam
The Conclusion
So here is the course of action that I think should be taken:
- Improve the quality rating to have named ranges, as stated in the list above.
- Remove the difficulty rating.
- Add a fun-factor rating.
- Add more options to help people find other kinds of "best levels", such as a list of Award-Winning Levels and Hompage-Featured Levels, and maybe think up of a few more similar ideas.
Edited Conclusion
After some discussion below, here's a constantly-updated list of the course of action, while leaving the old one in tact to protect the context of posts.
- Improve the quality rating to have named ranges, as stated in the list above.
- Remove the difficulty rating.
- Add a balanced-difficulty rating, where the higher the rate, the more balanced the difficulty.
- Add a fun-factor rating.
- Add more options to help people find other kinds of "best levels", such as a list of Award-Winning Levels and Hompage-Featured Levels, and maybe think up of a few more similar ideas.
User Comments (119)
| GF4 | ||
| -X- Karma: 117 Posts: 850 Gender: Male pm | email |
'Livio' said: Quote: - Well people, as we are now, are already putting the difficulty into their rating. Quote: - People would have to enter 3 numbers with your system too? And I think that setting up a system that streamlines the way we measure quality is a mistake. People should be free to decide that for themselves. Even if it is flawed according to other perspectives, giving the user that freedom is ultimately more important than the accuracy of the rates. and laziness is only a problem when a task is mandatory. The idea behind making something called "reviews" is so that when people see it, they might be interested or inspired enough to try it--but they don't have to. Once inside the feature, they may feel obligated to write a decent description of how they rated the level. I dunno, it's not very practical, but I'm mainly just messing with people's psychology here. I thought we were fixing the main rate. I'm just suggesting a system that would make the rate more fair for the level makers as they'd be able to see where they are coming from. The system I'm suggesting still shows the fun-factor rate and difficulty rate, but combines them all into one rate so the level maker can get an idea of how their level averages out. Check my description for my accounts. | |
| krotomo | ||
| The Shepherd Age: 23 Karma: 249 Posts: 4066 Gender: Male Location: My chair pm | email |
'GF4' said: The system I'm suggesting still shows the fun-factor rate and difficulty rate, but combines them all into one rate so the level maker can get an idea of what how their level averages out. | |
| GF4 | ||
| -X- Karma: 117 Posts: 850 Gender: Male pm | email |
'krotomo' said: 'GF4' said: The system I'm suggesting still shows the fun-factor rate and difficulty rate, but combines them all into one rate so the level maker can get an idea of what how their level averages out. Uhm.. High difficulty doesn't always mean bad. Some people actually strive for difficulty. Everyone has different views on what makes a good level, that's why this is fair. Check my description for my accounts. | |
| krotomo | ||
| The Shepherd Age: 23 Karma: 249 Posts: 4066 Gender: Male Location: My chair pm | email |
that's why this isn't fair. the people who don't agree with high difficulty get this average difficulty they can't change and don't agree with. It should be fair for everyone not only people who like low difficulty or people who like high difficulty. | |
| Livio | ||
Age: 31 Karma: 470 Posts: 9620 Gender: Male Location: Arizona, USA pm | email |
Actually I made a point in this blog post saying that the main rate isn't all-important. What's important is that the level database has some mechanisms in it that will make it easy for people to find great levels. The main rating cannot be trusted to show all of those great levels, so we should have other methods for finding them: fun-factor rates, biggest levels list (b/c they tend to be very intricate), award winners list, etc. Difficulty ratings don't really help in this, and they mainly just stop people from trying out certain levels if a level is too hard or too easy, and that's why I initially wanted to remove them. But I guess it could be an interesting feature of the database to find levels of a certain difficulty, especially if you're a beginner and you think all the levels are too hard. I had a feeling, though, that this feature just isn't used a lot, but then I thought that maybe it's because of the way it's presented (in a crowded form, rather than text links like everything else), so now I plan to keep difficulty ratings in order to try them out the right way. But I feel like the most it'll come to is just a feature in the database that makes you think it's cool when you first see it, but may not get used a lot. | |
| GF4 | ||
| -X- Karma: 117 Posts: 850 Gender: Male pm | email |
'krotomo' said: that's why this isn't fair. the people who don't agree with high difficulty get this average difficulty they can't change and don't agree with. It should be fair for everyone not only people who like low difficulty or people who like high difficulty. Well, how bout if we have the option to have a rate not count towards the average. Check my description for my accounts. | |
| Livio | ||
Age: 31 Karma: 470 Posts: 9620 Gender: Male Location: Arizona, USA pm | email |
yeah, krotomo just said what I was trying to say all this time. so the current idea is: -you submit three ratings in three categories -the rates get averaged together by the site to form the overall rating -you can choose which of the ratings contribute to the average, which means that if you just want the quality rating to make up the average then you can do that. however, there is beauty in simplicity, and this does not come off as simple. Especially for something most users would expect to be super-simple, like they see on most sites. It's already quite un-simple for us to force ratings to be attached to posts, so this idea would make that even worse. | |
| krotomo | ||
| The Shepherd Age: 23 Karma: 249 Posts: 4066 Gender: Male Location: My chair pm | email |
what you're doing is way more complicated then it has to be, because then there will be people that do and won't and there will be tons of confusion. EDIT: darn, posted on top of. | |
| GF4 | ||
| -X- Karma: 117 Posts: 850 Gender: Male pm | email |
Alright. Whatever I give up. Check my description for my accounts. | |
| Livio | ||
Age: 31 Karma: 470 Posts: 9620 Gender: Male Location: Arizona, USA pm | email |
VICTORY Btw, I added some of the names to the level ratings, including the ones in posts. check them out and comment on them. | |
| Isa | ||
| No. I'm an octopus. Age: 31 Karma: 686 Posts: 7833 Gender: Male Location: Uppsala, Sweden - GMT +1 pm | email |
Solution to the difficulty level problem: Change the meaning of the numbers, where a 5/5 means "This is the perfect level of difficulty!", a 3/5 means "While the difficulty of this level was OK, you can change it to make your level slightly harder/easier", and a 0/5 would be given to levels such as Imperio...! or caves that are easier than CotW winning levels, because they're either so hard it's impossible to beat them, or so easy it's impossible to NOT beat them. I hope I made sense. Our old difficulty ratings would have to go because with this system, suddenly everyone likes Imperio. That's a small problem though. | |
| Livio | ||
Age: 31 Karma: 470 Posts: 9620 Gender: Male Location: Arizona, USA pm | email |
Sounds like we should rename it to something like: Balanced Difficulty Rating This is much more useful than having five levels of difficulty, because people just care about good difficulties. I don't know why I didn't like this idea when GF4 suggested it. Maybe because it was attached to an idea I didn't like. And because it didn't have the word "balanced" in it to make me think it was cooler. | |
| GF4 | ||
| -X- Karma: 117 Posts: 850 Gender: Male pm | email |
'Livio' said: Sounds like we should rename it to something like: Balanced Difficulty Rating This is much more useful than having five levels of difficulty, because people just care about good difficulties. I don't know why I didn't like this idea when GF4 suggested it. Maybe because it was attached to an idea I didn't like. And because it didn't have the word "balanced" in it to make me think it was cooler. Isa has that effect. Check my description for my accounts. | |
| Isa | ||
| No. I'm an octopus. Age: 31 Karma: 686 Posts: 7833 Gender: Male Location: Uppsala, Sweden - GMT +1 pm | email |
Admit it, it's because I have tentacles. | |
| Livio | ||
Age: 31 Karma: 470 Posts: 9620 Gender: Male Location: Arizona, USA pm | email |
alright so now here's the idea: -one main level rating out of 10 -two sub ratings that are less important and out of 5 -one sub rating is the Balanced Difficulty Rating -the other is the Fun-Factor Rating Maybe shorten it to "Balance Rating" when necessary, and when speaking of the top rated of this category, call it the Most Balanced Levels. | |
| Sefro | ||
Karma: 313 Posts: 1136 Gender: Male Location: Canada pm | email |
The numbers of ratings given should be factored into the cave's overall rating somehow. For example, if someone makes a mediocre level and some random member plays it for seven seconds and then says "pretty fun 10/10", it probably shouldn't then be listed as the best level on the Interguild. A cave that has 8.5/10 with ten votes deserves a higher ranking than a cave that has 9/10 with two votes. I'm not quite sure how you'd implement that, though. Also, 10/10 deserves a better caption than "Super-Awesome!!!", because that's pretty lame, and not something I'd associate with 10/10 (which is, of course, not lame). "Super Special Awesome" would be an improvement, but few people would get the reference, so I'd say "Perfect" or "Epic" or "Incredible" would do. Unless someone can think of something more creative, like "The Beard of Zeus". Anyway, how to improve the rating itself, I don't know. I'll return to this thread if I think of anything. I'm not a fan of the one you guys are talking about, though. It kind of imposes a certain style of rating upon the player. I, for example, am content incorporating the fun and difficulty and overall design into a single rating. I don't want the system to divide it for me. The problem is how influential each individual rate is, but this Balanced Rating System won't fix that. These captions do a better job of that, to be honest... Youtube also had this problem, or something similar, and they wound up relinquishing star ratings altogether and just having a Like system. I don't think we'd want that, though. | |
| Livio | ||
Age: 31 Karma: 470 Posts: 9620 Gender: Male Location: Arizona, USA pm | email |
Hmm, maybe we can make it so that a level needs at least three rates before the rating gets "official". Until then, the site treats it like it hasn't got any rates at all? I think nmap.net does it like that. Plus it's easier to code. One of the main reasons why the top-rated list allows levels with only one or two votes is because I didn't know how to make some kind of algorithm that'll rank the levels accordingly but still not let the non-fully-rated levels fall into obscurity by not showing them at all. | |
| Livio | ||
Age: 31 Karma: 470 Posts: 9620 Gender: Male Location: Arizona, USA pm | email |
actually, I can't but help think that having more than one rating system is ugly. Whenever a user posts a rating to a level, it would look many times better if it wasn't accompanied by a difficulty rating. hmmm..... | |
| Livio | ||
Age: 31 Karma: 470 Posts: 9620 Gender: Male Location: Arizona, USA pm | email |
no wait! I think I just made them look infinitely better! | |
| Isa | ||
| No. I'm an octopus. Age: 31 Karma: 686 Posts: 7833 Gender: Male Location: Uppsala, Sweden - GMT +1 pm | email |
General Rate: 8/10 Difficulty Rate: 3/5 Fun Factor Rate: 3/5 This is a review of your level Lorem ipsum Blablabla --- Does it look neat enough? We should probably try different colors and try a shortening of the names. | |
| Livio | ||
Age: 31 Karma: 470 Posts: 9620 Gender: Male Location: Arizona, USA pm | email |
Maybe make the sub-rates smaller and some shade of gray... and something like this: Difficulty: 4.2/5 | |
| Livio | ||
Age: 31 Karma: 470 Posts: 9620 Gender: Male Location: Arizona, USA pm | email |
Okay so possible suggestions for the 10/10 title: Super-Awesome!!! (apparently deemed too silly) Epic (don't like it, since the term is probably overused) Perfect (probably the best one, but it just isn't exciting enough! When you see a 9/10 rating, the title is "Amazing!", so this should be more excited-sounding. Or maybe just adding exclamation points is enough...) Incredible (too synonymous with Amazing) Wow! (wait for it...) Wow! Incredible! (Smash Bros reference, anyone?...) The Beard of Zeus (too hairy) Or maybe just make the 10/10 rating have a funky background behind it. | |
| jazz | ||
Karma: 108 Posts: 3050 pm | email |
How about 'So epic that we couldn't think of an appropriate term to use'? | |
| Isa | ||
| No. I'm an octopus. Age: 31 Karma: 686 Posts: 7833 Gender: Male Location: Uppsala, Sweden - GMT +1 pm | email |
I don't think we need to add titles at all. If you do, keep them simple and to the point, with no extra fuzz. | |
| Livio | ||
Age: 31 Karma: 470 Posts: 9620 Gender: Male Location: Arizona, USA pm | email |
I'll use "Masterpiece" for now EDIT: this is my 6969th post! to celebrate I'll post plenty of 69 smilies   | |
« Forum Index < The Interguild Board
In order to post in the forums, you must be logged into your account.
Click here to login.
All games copyrighted to their respective owners.