|
|
| Improving the Level Rating System Thursday, July 29, 2010 12:36 am — Posted by Livio |
ShareThis« Livio's Blog
A growing problem lately is the arbitrary way in which we rate levels. Many levels seem to be getting overrated or underrated. This is actually quite a complex issue and there are several factors that play into it.
People have a strong tendency to give out their rates in an arbitrary fashion. A perfect example of this would be to look at your own ratings. As you look through them, you'll find that they aren't necessarily ranked from your most favorite levels to your least favorite ones, and you'll probably wonder why you rated certain levels lower than others when it should be the reverse. This seems to be an unavoidable issue, as it is impractical to compare every new rate you give out with all the ones you've made in the past. The problem would be more manageable, however, if we weren't such a small community.
In fact, this is one of the most obvious factors of the overall problem with our level rating system. The community is so small that everyone's rates have a large impact. Even the way we rate levels and videos is adapted to this issue: none of the rates are anonymous and you have to attach your rating to a post, in which people expect to include a valid explanation for that rate. In much larger sites, rating isn't given such a level of importance, and in some sites you're allowed to rate even if you don't have an account.
While working on the level database reform, I wondered if we should convert our rating system into something more conventional, where rates are anonymous and definitely not attached to posts but still require that you are logged into an account. I imagine that such a move would affect the level database in the following ways:
- Increased amount of ratings per level
- Decreased sense of importance given to each rate
- Ratings would become even more arbitrary than they are now, thus hurting the credibility of level ratings in general
Or rather, let's try thinking outside the box for a bit. The purpose behind the level rating system is so that people may find the best levels easily. So now let's ask ourselves: how can we better achieve that goal? There is a flaw with the current level-rating system in that the ratings are inherently inaccurate—no, not "inaccurate" but inconsistent. People rate levels on different standards, and therefore, they will want to play levels based on different standards as well. Instead of only offering one standard (overall quality), what we should do is provide multiple and more diverse ways to promote the best levels on the site. For example, offering a way to view levels that have won awards, or maybe ones that have been featured on the homepage.
And what if we set up more kinds of ratings? We already have quality and difficulty; perhaps we could add a Fun-Factor rating. Some might consider fun as being the most important factor taken into account when deciding on a level's quality rating, so wouldn't the addition of such a rating be redundant? Not really. The beauty behind a fun-factor rating is that it is indeed a second way to measure quality, but the list of "Most Fun" levels will [hopefully] be significantly different than the "Top-Rated" list. EDIT: Like I said in the discussion below: "fun rates may end up acting like a popularity meter, where fun levels that aren't necessarily well designed can still get acknowledgement." This thus creates another unique collection of the database's "best" levels, so that people will have a better chance of finding levels that match their taste.
Having three ratings could be bothersome, but lately I've been wondering about the usefulness of the difficulty rating. The idea behind it would be so that players could have an idea of how hard the level is before they tried it. However, it doesn't really promote the playing of more levels. Rather, it seems to just be a mechanism that stops people from playing levels that may be too hard or too easy. Furthermore, the "Hardest Levels" list may be useless in the sense that not many people would be thrilled to play evil levels. However, one redeeming aspect of it is that it presents the most intricate and crazily-designed levels of the database. But then again, so does the "Biggest Levels" list, which runs automatically without the need of user-input. So in the end, it seems like a good idea to get rid of the difficulty rating.
Finally, another major problem with our rating system is that everyone interprets the level rate numbers differently. Some consider a level rated as 7/10 as good or slightly average, while others interpret it as bad or mediocre. To fix this, the quality rating should be more like the difficulty rating, in which each range is given a name such as "Normal", "Challenging", and "Hard". Except that for the quality rating, it could be something like this:
- 10 - Super-Awesome!
- 9.0-9.9 - Amazing
- 8.0-8.9 - Great
- 7.0-7.9 - Good
- 6.0-6.9 - Average
- 5.0-5.9 - Mediocre
- 4.0-4.9 - Bleh
- 3.0-3.9 - Bad
- 2.0-2.9 - Terrible
- 1.0-1.9 - Fail
- 0.0-0.9 - Spam
The Conclusion
So here is the course of action that I think should be taken:
- Improve the quality rating to have named ranges, as stated in the list above.
- Remove the difficulty rating.
- Add a fun-factor rating.
- Add more options to help people find other kinds of "best levels", such as a list of Award-Winning Levels and Hompage-Featured Levels, and maybe think up of a few more similar ideas.
Edited Conclusion
After some discussion below, here's a constantly-updated list of the course of action, while leaving the old one in tact to protect the context of posts.
- Improve the quality rating to have named ranges, as stated in the list above.
- Remove the difficulty rating.
- Add a balanced-difficulty rating, where the higher the rate, the more balanced the difficulty.
- Add a fun-factor rating.
- Add more options to help people find other kinds of "best levels", such as a list of Award-Winning Levels and Hompage-Featured Levels, and maybe think up of a few more similar ideas.
User Comments (119)
| GF4 | ||
| -X- Karma: 117 Posts: 850 Gender: Male pm | email |
Or maybe levels can be rated in a few different categories by each person on a scale of ten and averaged out to by the rate. EDIT: I should have read the middle chunks of texts first.. Check my description for my accounts. | |
| Quirvy | ||
| Â Â Karma: 655 Posts: 7753 Gender: Male pm | email |
I have no problem with this system, although, I have a feeling that now, I will probably be VERY VERY rarely handing out rates above 9.0, and by rare, I mean probably exclusive to Dando levels or something. I'll try to rate more caves based on this system, maybe I'll end up altering previous rates, as well.  spooky secret | |
| Livio | ||
Age: 31 Karma: 470 Posts: 9620 Gender: Male Location: Arizona, USA pm | email |
I'm hoping to make it so that whenever you see a level rate, it's title will come up next to it, so that it'll be like: 9.8/10 - Amazing But before I add it, I'm wondering if the names I picked for each one are okay. Edit: maybe change the top two to: "Super-Awesome!!!" and "Amazing!" | |
| GF4 | ||
| -X- Karma: 117 Posts: 850 Gender: Male pm | email |
The names fit fine. But I still think you should add the fun-factor rate and increase the difficulty rate to a 10 scale. And then average out the three rates out of 10 (fun, difficulty, quality) to make your actual rate. Check my description for my accounts. | |
| krotomo | ||
| The Shepherd Age: 23 Karma: 249 Posts: 4066 Gender: Male Location: My chair pm | email |
I think the difficulty rating should stay. Difficulty is a major thing in level-making and should definitely stay. And the fun-factor rating, we don't need that. Fun easily goes with the overall rating and it is not needed to add a separate system for that. I agree that the quality ratings should have names, though. | |
| krotomo | ||
| The Shepherd Age: 23 Karma: 249 Posts: 4066 Gender: Male Location: My chair pm | email |
also I think I have a better rating which I always rate on. 10=Epic 9.5-9.9= Really Amazing 9.0-9.4= Great 8.5- 8.9= Good 7.0- 7.4= Average 6.0-6.9= O.K. 5.0- 5.9= Pretty Bad 4.0- 4.9= Bad 3.0- 3.9= Horrible 2.0- 2.9= Worthless 1.9 and less=  | |
| Livio | ||
Age: 31 Karma: 470 Posts: 9620 Gender: Male Location: Arizona, USA pm | email |
GF, I remember I had a similar idea ages ago. But now that I think of it, its not a really fair way to rate because one may weigh each one differently when trying to decide upon an overall rate. Oh and wait a minute, the difficulty rate would then turn into a quality-difficult rate, and not a rating of how high the difficulty is. But I still don't like that since it could seem like a hassle just to rate a level. I'd rather keep all the rates independent. Hmm, I was just thinking about the difficulty ratings and how it seems underused whenever you browse through levels. But that's probably because it's hidden inside all those messy search options. It would become much more useful if there were links linking to "Easy Levels", "Hard Levels", "Evil Levels", etc. Whoa, suddenly I don't want to remove difficulty rates anymore, just so I could try that idea out.... | |
| Livio | ||
Age: 31 Karma: 470 Posts: 9620 Gender: Male Location: Arizona, USA pm | email |
@ krotomo, I think you're keeping all the good rates limited to 8 and up. I prefer my system better so that more numbers would be used more often. | |
| GF4 | ||
| -X- Karma: 117 Posts: 850 Gender: Male pm | email |
a big reason for that is because the difficulty system came out a while after regular rates. Check my description for my accounts. | |
| krotomo | ||
| The Shepherd Age: 23 Karma: 249 Posts: 4066 Gender: Male Location: My chair pm | email |
'Livio' said: @ krotomo, I think you're keeping all the good rates limited to 8 and up. I prefer my system better so that more numbers would be used more often. or maybe 10=  9.5-9.9=  9.0-9.4=  8.5-8.9=  8.0-8.4=  7.0-7.9=  6.0-6.9=  5.0-5.9=  4.0-4.9=  3.0-3.9=  2.0-2.9=  1.9 and less= | |
| Quirvy | ||
| Â Â Karma: 655 Posts: 7753 Gender: Male pm | email |
Am I the only one who thinks that while adding words after the rate would encourage this new system, it would seem out of place? I like having only a simple X/10. I might support something more subtle, like how fake rate shows up when you put your cursor over the fake rate. I also support limiting difficulty ratings to 0-5. I don't see any major reason to switch difficulty from 5 to 10 spooky secret | |
| krotomo | ||
| The Shepherd Age: 23 Karma: 249 Posts: 4066 Gender: Male Location: My chair pm | email |
'Quirvy' said: I also support limiting difficulty ratings to 0-5. I don't see any major reason to switch difficulty from 5 to 10 | |
| Livio | ||
Age: 31 Karma: 470 Posts: 9620 Gender: Male Location: Arizona, USA pm | email |
I guess the 0-10 difficulty rating suggestion was to make it easier to average into another overall rate. And yeah, you're right about the appearance, quirvy. Maybe something like this: 9.0/10 Amazing! | |
| GF4 | ||
| -X- Karma: 117 Posts: 850 Gender: Male pm | email |
'Quirvy' said: I also support limiting difficulty ratings to 0-5. I don't see any major reason to switch difficulty from 5 to 10 I only suggested 5 to 10 so if we had 3 rates it could all average out into one. Check my description for my accounts. | |
| Quirvy | ||
| Â Â Karma: 655 Posts: 7753 Gender: Male pm | email |
That looks better, Livio, but it still sticks out because the black area in the rate and amazing don't touch at the ends. If you can fix that, it will be acceptable, I suppose. As for goldfun, I'm confused as to what you're talking about by three rates averaging into one. Maybe it's just too late, but I have no clue what you're talking about. As a result, I'm going to bed. spooky secret | |
| krotomo | ||
| The Shepherd Age: 23 Karma: 249 Posts: 4066 Gender: Male Location: My chair pm | email |
same, but since too much is going on i'm not going to bed | |
| GF4 | ||
| -X- Karma: 117 Posts: 850 Gender: Male pm | email |
'Quirvy' said: That looks better, Livio, but it still sticks out because the black area in the rate and amazing don't touch at the ends. If you can fix that, it will be acceptable, I suppose. As for goldfun, I'm confused as to what you're talking about by three rates averaging into one. Maybe it's just too late, but I have no clue what you're talking about. As a result, I'm going to bed. Well for you whenever you get back on: Fun factor rate: 9/10 Difficulty rate: 7.5/10 Quality rate: 8/10 Average rate: About an 8.2 Check my description for my accounts. | |
| krotomo | ||
| The Shepherd Age: 23 Karma: 249 Posts: 4066 Gender: Male Location: My chair pm | email |
why can't you just multiply the X/5 rate by 2? | |
| GF4 | ||
| -X- Karma: 117 Posts: 850 Gender: Male pm | email |
'krotomo' said: why can't you just multiply the X/5 rate by 2? That'd just put more work on the system Check my description for my accounts. | |
| Livio | ||
Age: 31 Karma: 470 Posts: 9620 Gender: Male Location: Arizona, USA pm | email |
'Quirvy' said: That looks better, Livio, but it still sticks out because the black area in the rate and amazing don't touch at the ends. If you can fix that, it will be acceptable, I suppose. 'krotomo' said: why can't you just multiply the X/5 rate by 2? EDIT: man, posted on top of with a two-minute difference | |
| shos | ||
| ~Jack of all trades~ Age: 31 Karma: 389 Posts: 8273 Gender: Male Location: Israel pm | email |
'Quirvy' said: Am I the only one who thinks that while adding words after the rate would encourage this new system, it would seem out of place? I like having only a simple X/10. I might support something more subtle, like how fake rate shows up when you put your cursor over the fake rate. I also support limiting difficulty ratings to 0-5. I don't see any major reason to switch difficulty from 5 to 10  | |
| krotomo | ||
| The Shepherd Age: 23 Karma: 249 Posts: 4066 Gender: Male Location: My chair pm | email |
Here's my opinion You have The quality, difficulty and fun rates. quality=X/10 difficulty=X/5 fun=X/5 That's what I think is right, because it's nice and simple. The quality ratings, which are more important and serious than the other two, so they get X/10 And the other two which aren't as major as the overall quality, but are still there, so they're only X/5. | |
| GF4 | ||
| -X- Karma: 117 Posts: 850 Gender: Male pm | email |
'shos' said: 'Quirvy' said: Am I the only one who thinks that while adding words after the rate would encourage this new system, it would seem out of place? I like having only a simple X/10. I might support something more subtle, like how fake rate shows up when you put your cursor over the fake rate. I also support limiting difficulty ratings to 0-5. I don't see any major reason to switch difficulty from 5 to 10 People will get more of an idea of what others think of their level. Check my description for my accounts. | |
| Livio | ||
Age: 31 Karma: 470 Posts: 9620 Gender: Male Location: Arizona, USA pm | email |
'shos' said: I think that too. I can't think of a reason to change what exists now. and you're right krotomo. That sounds like a good way to do it. | |
| GF4 | ||
| -X- Karma: 117 Posts: 850 Gender: Male pm | email |
'krotomo' said: You're making this way more complicated then it has to be. You have The quality, difficulty and fun rates. quality=X/10 difficulty=X/5 fun=X/5 That's what I think is right, because it's nice and simple. The quality ratings, which are more important and serious than the other two, so they get X/10 And the other two which aren't as major as the overall quality, but are still there, so they're only X/5. There's no difference. I'm not making it anymore complicated.. Check my description for my accounts. | |
« Forum Index < The Interguild Board
In order to post in the forums, you must be logged into your account.
Click here to login.
All games copyrighted to their respective owners.