Log In
Name:
Pass:
Online Members (0)
No members are currently online.
Current Interguild Time:
Mon Apr 29 2024 5:46 am
Member Chat Box  [click here to enlarge]
Recent Posts and Comments
ShareThis
« Blogs Index < Interguild-Related Posts < ''State of the Interguild'' Blog
« Livio's Blog

A growing problem lately is the arbitrary way in which we rate levels. Many levels seem to be getting overrated or underrated. This is actually quite a complex issue and there are several factors that play into it.

People have a strong tendency to give out their rates in an arbitrary fashion. A perfect example of this would be to look at your own ratings. As you look through them, you'll find that they aren't necessarily ranked from your most favorite levels to your least favorite ones, and you'll probably wonder why you rated certain levels lower than others when it should be the reverse. This seems to be an unavoidable issue, as it is impractical to compare every new rate you give out with all the ones you've made in the past. The problem would be more manageable, however, if we weren't such a small community.

In fact, this is one of the most obvious factors of the overall problem with our level rating system. The community is so small that everyone's rates have a large impact. Even the way we rate levels and videos is adapted to this issue: none of the rates are anonymous and you have to attach your rating to a post, in which people expect to include a valid explanation for that rate. In much larger sites, rating isn't given such a level of importance, and in some sites you're allowed to rate even if you don't have an account.

While working on the level database reform, I wondered if we should convert our rating system into something more conventional, where rates are anonymous and definitely not attached to posts but still require that you are logged into an account. I imagine that such a move would affect the level database in the following ways:
  • Increased amount of ratings per level
  • Decreased sense of importance given to each rate
  • Ratings would become even more arbitrary than they are now, thus hurting the credibility of level ratings in general
It could be said that one of the problems with our current system is that people take rates too seriously, so some of the faults of a more conventional system could turn out to be for the best. And yet, I'm not too convinced that this is what we should do.

Or rather, let's try thinking outside the box for a bit. The purpose behind the level rating system is so that people may find the best levels easily. So now let's ask ourselves: how can we better achieve that goal? There is a flaw with the current level-rating system in that the ratings are inherently inaccurate—no, not "inaccurate" but inconsistent. People rate levels on different standards, and therefore, they will want to play levels based on different standards as well. Instead of only offering one standard (overall quality), what we should do is provide multiple and more diverse ways to promote the best levels on the site. For example, offering a way to view levels that have won awards, or maybe ones that have been featured on the homepage.

And what if we set up more kinds of ratings? We already have quality and difficulty; perhaps we could add a Fun-Factor rating. Some might consider fun as being the most important factor taken into account when deciding on a level's quality rating, so wouldn't the addition of such a rating be redundant? Not really. The beauty behind a fun-factor rating is that it is indeed a second way to measure quality, but the list of "Most Fun" levels will [hopefully] be significantly different than the "Top-Rated" list. EDIT: Like I said in the discussion below: "fun rates may end up acting like a popularity meter, where fun levels that aren't necessarily well designed can still get acknowledgement." This thus creates another unique collection of the database's "best" levels, so that people will have a better chance of finding levels that match their taste.

Having three ratings could be bothersome, but lately I've been wondering about the usefulness of the difficulty rating. The idea behind it would be so that players could have an idea of how hard the level is before they tried it. However, it doesn't really promote the playing of more levels. Rather, it seems to just be a mechanism that stops people from playing levels that may be too hard or too easy. Furthermore, the "Hardest Levels" list may be useless in the sense that not many people would be thrilled to play evil levels. However, one redeeming aspect of it is that it presents the most intricate and crazily-designed levels of the database. But then again, so does the "Biggest Levels" list, which runs automatically without the need of user-input. So in the end, it seems like a good idea to get rid of the difficulty rating.

Finally, another major problem with our rating system is that everyone interprets the level rate numbers differently. Some consider a level rated as 7/10 as good or slightly average, while others interpret it as bad or mediocre. To fix this, the quality rating should be more like the difficulty rating, in which each range is given a name such as "Normal", "Challenging", and "Hard". Except that for the quality rating, it could be something like this:
  • 10 - Super-Awesome!
  • 9.0-9.9 - Amazing
  • 8.0-8.9 - Great
  • 7.0-7.9 - Good
  • 6.0-6.9 - Average
  • 5.0-5.9 - Mediocre
  • 4.0-4.9 - Bleh
  • 3.0-3.9 - Bad
  • 2.0-2.9 - Terrible
  • 1.0-1.9 - Fail
  • 0.0-0.9 - Spam
The names could use some work, but you get the idea. It would definitely improve the consistency of ratings.

The Conclusion

So here is the course of action that I think should be taken:
  1. Improve the quality rating to have named ranges, as stated in the list above.
  2. Remove the difficulty rating.
  3. Add a fun-factor rating.
  4. Add more options to help people find other kinds of "best levels", such as a list of Award-Winning Levels and Hompage-Featured Levels, and maybe think up of a few more similar ideas.

Edited Conclusion

After some discussion below, here's a constantly-updated list of the course of action, while leaving the old one in tact to protect the context of posts.
  1. Improve the quality rating to have named ranges, as stated in the list above.
  2. Remove the difficulty rating.
  3. Add a balanced-difficulty rating, where the higher the rate, the more balanced the difficulty.
  4. Add a fun-factor rating.
  5. Add more options to help people find other kinds of "best levels", such as a list of Award-Winning Levels and Hompage-Featured Levels, and maybe think up of a few more similar ideas.
[?] Karma: 0
User Comments (119)
« Forum Index < The Interguild Board
«Previous | 1, 2, 3, 4, 5 | Next»

Isa
[?] Karma: 0 | Quote - Link
Thursday, July 29 2010, 4:18 am EST
No. I'm an octopus.

Age: 31
Karma: 686
Posts: 7833
Gender: Male
Location: Uppsala, Sweden - GMT +1
pm | email
While we're at it, why does the difficulty rating bar even exist? Let's remove it, it looks out of place.
jazz
[?] Karma: 0 | Quote - Link
Thursday, July 29 2010, 4:20 am EST

Karma: 108
Posts: 3050
pm | email
To be honest, I agree with Isa. It's not like any of us can't read numbers. (This is your chance!)
Livio
[?] Karma: 0 | Quote - Link
Thursday, July 29 2010, 4:23 am EST

Age: 31
Karma: 470
Posts: 9620
Gender: Male
Location: Arizona, USA
pm | email
Nah, I'll get rid of the bar when I take down the difficulty ratings and replace it with the new ratings.

Btw guys, I made a very important edit to my post at the end of the last page, so you should go back and read it.
jazz
[?] Karma: +1 | Quote - Link
Thursday, July 29 2010, 4:26 am EST

Karma: 108
Posts: 3050
pm | email
I don't trust you, Livio. All because of what you posted in the chatbox. Knowing my luck, I'll probably see a bunch of 69 smilies and that's all.
EDIT: Well, I'd be darned...
Isa
[?] Karma: 0 | Quote - Link
Thursday, July 29 2010, 4:26 am EST
No. I'm an octopus.

Age: 31
Karma: 686
Posts: 7833
Gender: Male
Location: Uppsala, Sweden - GMT +1
pm | email
I hope that you mean that my proposed system of difficulty levels shall be implanted then.
Livio
[?] Karma: 0 | Quote - Link
Thursday, July 29 2010, 4:34 am EST

Age: 31
Karma: 470
Posts: 9620
Gender: Male
Location: Arizona, USA
pm | email
yeah, I even updated the Edited Conclusion
shos
[?] Karma: 0 | Quote - Link
Thursday, July 29 2010, 5:13 am EST
~Jack of all trades~

Age: 31
Karma: 389
Posts: 8273
Gender: Male
Location: Israel
pm | email
livio, seriously why change something that works well. leave the system as it is....i'd say even without the words. if you see a 7.5/10, you don't need to also read 'good' to know that...


jellsprout
[?] Karma: +2 | Quote - Link
Thursday, July 29 2010, 6:01 am EST
Lord of Sprout Tower

Karma: -2147482799
Posts: 6445
Gender: Male
pm | email
I oppose some averaging of different numbers system. There are far more factors than fun, difficulty and "quality" (whatever that may mean). For me originality has always been one of the most important factors, design being another.
And these factors don't weigh equally either. Fun will always be the most important factor for any game and should weigh far more heavily than the other factors.
There is also the problem that the different factors aren't independent. A level that is too difficult won't be fun either. A simple /10 rating is far better to show how much a person enjoyed playing the level.

I oppose the balanced difficulty rating too. The point of the difficulty rating wasn't to show the quality of the difficulty, but actually show the difficulty itself. If you wanted a difficult challenge you could look at the difficulty ratings and pick a level with a rating of around 4. Likewise, if you just wanted to relax and pick a level you could breeze through, you would pick a level with a rating of 1.
Not to mention the balanced difficulty rating is completely pointless. People are able to determine themselves if they like a level with a difficulty of 3 better than a level with a difficulty of 5. With the added description the difficulty rating already had it isn't people would think that 5/5 - Impossible would be a good difficulty.
And there is also the problem that people prefer different difficulties. Shos would give a 4/5 difficulty level a balanced difficulty of 5/5 and a 3/5 difficulty level 4/5, while Dando would do the exact opposite. The current difficulty rating gives a far better view of the quality of the difficulty than a balanced difficulty rating would.


Spoiler:
Quirvy
[?] Karma: 0 | Quote - Link
Thursday, July 29 2010, 11:44 am EST
  

Karma: 655
Posts: 7753
Gender: Male
pm | email
Whoa, so you're just going to conclude that the system needs these additions overnight, before some people can even come online?

I've been reading a good portion of the arguments for expanding the system, and I thought I'd respond to them.

Quote:
I'm just suggesting a system that would make the rate more fair for the level makers as they'd be able to see where they are coming from.
I disagree with this entirely. "6 out of 10 for fun rating" doesn't let me know where they're coming from, their actual review does. As a cave maker, I can assure you that I don't care about how they form their rate, I just care about specific feedback. Giving us extra things like whatever you were suggesting doesn't help me understand what I need to improve. The most important part of a rate is the review, in my opinion, and there's no reason to take emphasis away from that. The rate itself should just be a basic summary of what you think of the level.

If someone wants to show us how they came upon our level's rating, they can do that in their review. Why do we need some other special feature to convey that?

It basically sounds like, "Lets throw out the difficulty rating, and split up the overall rating into difficulty balance, and fun"

I see no reason to get rid of the difficulty rating to make way for this new system, where the rating part around difficulty doesn't tell you if it's hard or easy, just if its balanced or not. You're trying to revolutionize something that is perfectly fine as it is. You have better things to do, like working on the Aeon Demo.

I think what angers me most though is that you guys pretty much decided that you're going to change this, while I was asleep. If you had 3 pages of arguing in like 6 hours, maybe the decision won't be unanimous, and you should wait an entire day before coming to a definite conclusion



spooky secret
Yaya
[?] Karma: 0 | Quote - Link
Thursday, July 29 2010, 12:11 pm EST

Age: 29
Karma: 747
Posts: 5367
Location: Ohio (US)
pm | email
Well, I read the 1st page, 2nd page, skipped the 3rd page, and read this page. Like Quirvy said, let those who were dreaming about a gunfight in K-mart while you guys were arguing have a chance to respond to this before you make a decison.

You guys are right for the most part, the levels system needs a bit of reform. Here are my stances.

I see no need to get rid of the difficulty ratings, some players do actually pay attention to them. If you guys really are that strung out about the difficulty concept, I say this: make the system start with easy and end with hard. Easy is easy, hard is hard. I don't see why we need anything beyond that. Some players like to play slightly hard levels, or pretty easy levels, but can be decieved into thinking a level is not worth playing if they see a "evil" or a "very easy" on it. By making the rates more simplified, this'd almost get rid of the evil  so called evil difficulty. A cave isn't evil unless it displays actual evil traps. None of that has really been brought up at all, so it may not be revelant.

For the problem with too many high rates, I'd hate to say it, but some people just don't rate levels well, they need to learn that just because a level is good they do not have to automaticly give it a 9.0. I don't really see any forseeable way to fix this, unless either we restrict certain peoples rates which'd be unfair, or we do the 3-rates-to-make-legit system. That also has problems, I'll use the example of a moderatly hard cave, which not many people beat. Lets say though, that Shos and Ckjr beat it and both gave it high 9.0s. Nobody else wants to try the level, but Shos and Ckjr deserved to rate it, for they beat it, so I worry, that not enough people will feel they got far enough to contribute to the offical rate, or people will blind-rate, or look-at-the preview-image-rate (a major problem) and bring down/bring up the official rating very inaccuratly.

I say we probably should start to change some our more ****** up rates, and try to make them more reflect todays ideas. We need to teach people to utilize the numbers 3-7 in the rating system. Usual good/bad rating end up on the polar opposites of the level.

Multiple different ratings, (fun, quality, effort into making etc.) there are too many factors. Perhaps make and optional system in which you can either rate a level normally or, you can fill out all these rating boxes and they'll average out for you (The Kittikiyana-cave-rater-o-matic, lol).

As for making this decision overnight without more opinions of active Interguildians
Spoiler:

Think this decison over, this idea is about as radical as the internet-wide database idea, so we need time for several more members to offer input. Atleast edit in a non-outcome-deciding-just-opinion-reflecting poll.



COMING SOON: A giant meteor. Please.
Give me +karma. Give me +karma.
Livio
[?] Karma: 0 | Quote - Link
Thursday, July 29 2010, 3:47 pm EST

Age: 31
Karma: 470
Posts: 9620
Gender: Male
Location: Arizona, USA
pm | email
hmm, nothing was really final. The edited conclusion was more like a log of what the current idea was, not a final decision. And I changed the way level rates look more because I was experimenting with how they looked.

And I like Yaya's idea of simplifying the difficulties into Easy and Hard. But I can't but help feel that we should split them three ways, for Easy, Normal, and Hard, because there are just some levels that you would neither call Easy or Hard. The Easy levels would be difficulties of 0-1.9, Normal would be 2-3.4 and Hard would be 3.5-5.

And there may have been some confusion in the conversation. The current idea was to have one main rating, and two side ratings for difficulty and fun. These ratings have nothing to do with each other at all, or I mean they don't go together to make one big rating or anything like that.
Quirvy
[?] Karma: 0 | Quote - Link
Thursday, July 29 2010, 3:50 pm EST
  

Karma: 655
Posts: 7753
Gender: Male
pm | email
Easy 0-1.9
Normal 2-3
Hard 3.1-5

That seem good enough?



spooky secret
Livio
[?] Karma: 0 | Quote - Link
Thursday, July 29 2010, 3:52 pm EST

Age: 31
Karma: 470
Posts: 9620
Gender: Male
Location: Arizona, USA
pm | email
hmm, I guess that's better. I don't know what I was thinking. i think I was trying to build that idea around the way the current difficulty ratings are broken up into. Because many levels that are a low "Challenging" probably deserve to be under Normal and not Hard.
Livio
[?] Karma: 0 | Quote - Link
Thursday, July 29 2010, 4:30 pm EST

Age: 31
Karma: 470
Posts: 9620
Gender: Male
Location: Arizona, USA
pm | email
Also, people have been saying that they see absolutely nothing wrong with the current system, and I thought I outlined it pretty well in the blog post:
Quote:
Finally, another major problem with our rating system is that everyone interprets the level rate numbers differently. Some consider a level rated as 7/10 as good or slightly average, while others interpret it as bad or mediocre.
that's why I added the titles like Good, Great, and Amazing. I added them last night mainly b/c I was experimenting with the way the design would look, but since I've added it, I've noticed myself thinking of rates under a different standard. I remember I used to use 7 and 8 ratings interchangeably like there wasn't much of a difference, but now because there are titles attached to them, I have a much clearly definition in my head of what makes a 7 cave or an 8 cave.

And the main reason why I started writing the post was because of the recent conversations and arguments that I linked to at the start of the blog. They got me thinking about the way that we manage our rates. The problem is that they aren't reliable many times. And about half way through the blog, I said that any system we come up with will be inherently flawed. So maybe instead of relying so much on one system to display good levels, we could come up with multiple systems to show good levels, like having a "Most Fun Levels" list, and an "Award Winners" list, and "Homepage-Featured Levels" list.
Livio
[?] Karma: 0 | Quote - Link
Tuesday, October 12 2010, 2:55 pm EST

Age: 31
Karma: 470
Posts: 9620
Gender: Male
Location: Arizona, USA
pm | email
we need to improve the way difficulty ratings work. I was trying to come up with a new design for how they look when I figured that I really didn't like the way it was set up.

I have a couple of ideas:

(1) We could change rename each of the different ratings you can give out. Right now it's 'Very Easy', 'Easy', 'Normal', 'Challenging', 'Hard', 'Evil'. I was thinking Very Easy and Easy are redundant, and perhaps we could merge them. Then move everything down? I guess the rating would be more of an indicator of how hard (as opposed to how easy) a level is.

(2) Or we could just simplify the whole thing. Just let people say if a level is Easy or Hard. Maybe it could be like the karma rate system, where you only vote plus one or minus one for difficulty.

(3) Going off of the simplified idea, perhaps we could let people vote three ways: Easy, Medium, Hard. I guess we could make it a vote out of three as opposed to out of five, and we could also let people vote with decimal places like we do now.

You know, I'm liking that last idea the most and I bet you are too. These ideas have been mentioned somewhere before, but I can't remember where or who initially brought them out, so sorry for not giving people credit. Of course we're gonna have to figure out how we're going to convert all the old rates into the new standard.
jellsprout
[?] Karma: 0 | Quote - Link
Tuesday, October 12 2010, 3:21 pm EST
Lord of Sprout Tower

Karma: -2147482799
Posts: 6445
Gender: Male
pm | email
That seems like a surprisingly efficient system. Especially as the difficulty rating doesn't need to be as exact as the quality rating. A basic easy/medium/difficult rating would probably work quite well. Option 3 gets my vote.


Spoiler:
Isa
[?] Karma: 0 | Quote - Link
Tuesday, October 12 2010, 3:53 pm EST
No. I'm an octopus.

Age: 31
Karma: 686
Posts: 7833
Gender: Male
Location: Uppsala, Sweden - GMT +1
pm | email
Easy, Medium and Hard sounds fair to me. How do we convert the current rates? 0-2.4 for Easy, 2.5-3.9 for Medium and 4.0-5.0 for Hard is my suggestion.
Livio
[?] Karma: 0 | Quote - Link
Tuesday, October 12 2010, 3:57 pm EST

Age: 31
Karma: 470
Posts: 9620
Gender: Male
Location: Arizona, USA
pm | email
I think just calculate the rate's percentage out of five, apply to to three, so that you get a corresponding number between 0-3.

so:

([rate]/5)*3 = new rate out of three.
jellsprout
[?] Karma: 0 | Quote - Link
Tuesday, October 12 2010, 4:00 pm EST
Lord of Sprout Tower

Karma: -2147482799
Posts: 6445
Gender: Male
pm | email
On second thought, easy/challenging/hard might be better names, with the cut-offs being at 2.4 and 3.9.


Spoiler:
Livio
[?] Karma: 0 | Quote - Link
Tuesday, October 12 2010, 4:05 pm EST

Age: 31
Karma: 470
Posts: 9620
Gender: Male
Location: Arizona, USA
pm | email
0.0-2.4 = Easy
2.5-3.8 = Challenging
3.9-4.0 = Hard

doesn't look so elegant. And besides, if it's supposed to be out of three:

0.0-1.4 = Easy
1.5-2.8 = Challenging
2.9-3.0 = Hard

I'm still not liking it much. Although the decision to rename Medium to Challenging.... maybe.
Isa
[?] Karma: 0 | Quote - Link
Tuesday, October 12 2010, 4:07 pm EST
No. I'm an octopus.

Age: 31
Karma: 686
Posts: 7833
Gender: Male
Location: Uppsala, Sweden - GMT +1
pm | email
2.5-3.9 and 4.0-5.0. Who cares if it doesn't look elegant?

Also do mind that was how to convert the current rates into a three-way system.
jellsprout
[?] Karma: 0 | Quote - Link
Tuesday, October 12 2010, 4:09 pm EST
Lord of Sprout Tower

Karma: -2147482799
Posts: 6445
Gender: Male
pm | email
I didn't see the decimal thing. I thought you meant that you simply pick from Easy, Medium and Hard as ratings instead of a numerical rating. Like the karma system, only with 3 options instead of two. I was merely suggesting how to convert the current ratings.


Spoiler:
Livio
[?] Karma: 0 | Quote - Link
Tuesday, October 12 2010, 4:10 pm EST

Age: 31
Karma: 470
Posts: 9620
Gender: Male
Location: Arizona, USA
pm | email
oh then I misread his post. I thought he was saying that the 3-point system was to be organized into the three names by those numbers.

but you know, you're right. That probably is a better way to convert the rates
Livio
[?] Karma: 0 | Quote - Link
Tuesday, October 12 2010, 4:12 pm EST

Age: 31
Karma: 470
Posts: 9620
Gender: Male
Location: Arizona, USA
pm | email
'jellsprout' said:
I thought you meant that you simply pick from Easy, Medium and Hard as ratings instead of a numerical rating.
Yeah we should probably do that, since like you said, we don't need much accuracy for difficulty
canadianstickdeath
[?] Karma: 0 | Quote - Link
Tuesday, October 12 2010, 10:44 pm EST

Age: 35
Karma: 350
Posts: 2990
Gender: Male
pm | email
If you don't like the numerical difficulty ratings, the easiest way to remove it is to replace it with a difficulty drop-down consisting of the current difficulty names.

Easy/Medium/Hard seems like too few qualifiers to me?

« Forum Index < The Interguild Board
«Previous | 1, 2, 3, 4, 5 | Next»

In order to post in the forums, you must be logged into your account.
Click here to login.

© 2024 The Interguild | About & Links | Contact: livio@interguild.org
All games copyrighted to their respective owners.