January 6, 2007 at 3:50 am #250245
And there are 24 days left. And these new positions were only announced three days ago.
CometgreenJanuary 6, 2007 at 6:04 am #250291
To me there is too much uncertainty in the current ratings system. Every person has decided what is average for themselves, which is creating unfair ratings. Yes, the rubric does restrict what you can rate a film, but it is for the better. With a rubric, you have a description of what you actually felt about the film. No, it isn’t telling you what to feel, it’s helping you to better understand and appreciate the quality of the film you just watched. Perhaps there was fine work that you overlooked, like crisp and exciting sound design. Simply putting in a number doesn’t requite thought, because suddenly you become aware that other people may have given it a different number. Suddenly you’re trying to boost the rating to give the film what you think it deserves, therefore ruining the balance of the ratings across the directory.
I see more approval for the Staff Rating than the rubric, which is understandable, because not many of you have had a chance to use a rubric before to help you understand what marks you are giving. I credit the genesis of the idea of the Staff Rating to Cometgreen, who mentioned it while I was having a discussion with him in chat. I saw it was a good idea, and I decided to do some more research into it.
The Staff Rating would be a new feature. It wouldn’t be expected for staff members to go back and re-rate all of the films in the directory, just as it has never been expected of reviewers to go back and eliminate the poorer films when the standards are raised. It will simply be something that is included from this point onwards. If they please they can go back and give it a rating, if they think it’s an older film that gets enough attention or if they just enjoy it, or for whatever reason they come up with.
What I want to change about the ratings is the feeling put into it. It shouldn’t be about emotionless numbers, but being able to properly scale the score you’re giving a film without thinking about the final result. This is why I chose the rubric, as this is exactly what it allows. For those who do not support the use of a rubric, perhaps I can change your mind. If not, I really don’t mind.
To those who say that I’m not caring enough about the ratings system, I’m not sure what you base this on, but from this point on I will interpret it as mudslinging, as the evidence of my dedication is still clearly presented on the first page of this thread.
My views have come under question as well, which I consider to be an attempt to distract users from what I actually want to accomplish. I will state them once more for the people who insist on picking apart my words in order to better their own egos. I do not condone the abuse of the ratings system. This is exactly why I am pushing to make these changes, so instead of a few numbers, the raters will feel more responsible when they’re asked for their opinion. I will not condemn my fellow brickfilmers for letting their emotions get the best of them. You are forgetting that these people are just like everyone else. In fact, they most likely are everyone else. If I were to begin accusing people of foul play, and insisting that they be hunted down and banned, who would be left? You obviously have no idea how many people have been tempted to punish others for their forum behavior by tampering with the ratings.
I realize that people have abused the system. From my very first post I have indicated that I wish to change the system to be less open to that sort of abuse. How is it that you have overlooked that in order to create an argument?January 6, 2007 at 6:08 am #250293
“brianfast” wrote: I don’t like the idea of a rubric. I wouldn’t have paid 3 dollars to follow your guidelines. I will rate as I please. I do think the staff ratings being displayed is a good idea, however.
Sir, rating ‘as you please’ is the exact problem here. Because ‘as you please’ isn’t the same as ‘as someone else pleases’, which means that the films aren’t getting the rating that they deserve. If everyone decides for themselves what an average film is, then the ratings are never going to be accurate, and an accurate rating is what the director deserves.January 6, 2007 at 7:04 am #250296
I’ve realised my error now, and I apologise to all involved.
I do recognize that tampering with ratings is wrong, I just don’t wish to use the word ‘condemn’, because of the negative connotations that are too easily placed with it. Because of my lack of explanation on that subject, there was ensuing confusion. I apologise for not being more clear.
I hope we can now resume discussion about the changes I wish to make if I’m elected as Minister of Film Rating.
-LadonJanuary 6, 2007 at 8:01 am #250307
Just something to think about…
The trouble with ratings scales, as has been noted, is that those with extreme opinions, dishonest or not, have a greater affect on the final number than those with more measured opinions, and there is no way to coax everyone to the same level of zeal or moderation in rating. The proposed rubric system is one solution to this, but it demands a certain time commitment from the film reviewer that may be ok for a dedicated ratings group but is, I fear, too large of a barrier for casual reviwers.
The ratings scale problem is similar to that encountered in ratings voting systems: if you want your candidate to win, you’re best off lying and rating that candidate a ten and everyone else a zero. One solution to this is Approval Voting, which does away with the scale and has only “yes” or “no”, eliminating the advantage of those with extreme opinions by forcing everyone to use the extreme. I don’t see how this system would be of much use to brickfilm ratings however.
Another interesting voting system is Ranked Pairs. The voter ranks the candidates from most favored to least favored. This also does away with explicit ratings, reducing the problem of voters gaming the system by rating in the extreme in contradiction to their true ratings. To find the best candidate, Ranked Pairs takes each pair of candidates in turn, A and B, and looks to see if more ballots ranked A above B than B above A; if so, A is better than B. Since any reviewer is unlikely to have seen all films, rather than think in terms of a complete ballot with all candidates or films ranked, it is better to think in terms of preferences such as A above B. A preference of A above B is only added to the ballot box if a reviewer has watched and ranked both A and B.
From an interface usability perspective, the “review a film” part of the film page need ask but a single question to add to the ratings information: it could choose a film from those the reviewer has already watched and simply ask “is this new film better or worse than this old film you’ve already seen?. A more complex version might list some or all of the reviewer’s past rankings and ask for a click setting position the new film in the list. The “rubric” is merely comparing films to one another and requires, I believe, less effort than trying to follow a scoring sheet. It’s certainly not as accurate as a careful and honestly done ratings system; the advantages are simplicity and greater robustness to dishonesty. Of course, implementation would be quite different from what is in place now and possibly not worth the effort.
Then again, this only works for an “overall” or “enjoyment” type category; trying to remember the quality of sound effects in some film you saw 15 months ago to compare to those in the new film might be a bit of a stretch. Of course, relying on memory is also a problem with a fine grained ratings system. “These sound effects deserve a 6 or 7, but they are a little better than that other film which I gave a 6.2 … or was it 6.5?”January 6, 2007 at 9:02 am #250316
Eventide, that’s quite useful insight you’ve given me. I’ll take note of it as I continue to develop my newer method of rating. Some of the ideas you suggested, as you mentioned, are extreme. I also want to make sure the changes are implementable. What I want to do is possible, but if we get too detailed about fine-tuning the ratings, the work required will be too difficult to justify.January 6, 2007 at 9:59 am #250321
I would like to talk a bit more about freedom of rating.
Right now you have a choice of 99 different ratings for every category. I know that sounds freeing, but for the most part it is confusing. Above the rating page it says “A score of 5.0 is average and should be fairly common. A score in the 9’s would go to a truly exceptional film and will be very rare.”, but how many times have you given a 5.0 to an average film, instead choosing to go higher to be ‘even’ with the other films in the directory?
I want to eliminate the confusion caused by those 99 different ratings. I want there to be approximately 6. I say approximate because I am still working on the rubric itself.
In case you aren’t quite understanding what I mean, here is an example:
A quiz site runs a poll to see what movies people like more. They put over 100 films on the poll! Unable to decide with so many choices, the people end up voting on whichever they think of first. The poll is a failure, as it hasn’t determined which film people like better out of the ones they listed, they have simply determined which film was near the top of the list.
The next week, they try again. This time, they have 10 films on the poll. People are able to see all of the choices at the same time, and they vote according to their personal preference. The poll is a success, as the resulting statistics are clear and precisely indicate which film is favored by the voters.
With the rubric, you will quickly understand what a film should get. I am going to try and modify it to suit all genres of film, even the more quirky. In the case of the Anne Frank films, for example, the ‘Enjoyment’ rating is really what should be looked at. If the others are low in that case, that’s exactly where they should be. That was what the directors intended, is it not?
The rubric will also help directors understand what ratings they received. I personally have looked at ratings of my films and wondered what they were based on. But if the rating is based within the rubric, I will quickly and easily understand what it is that I need to work on, and what I did well.
Freedom of choice is limited by understanding. No one person should have their own determining factors for ratings, because how does the director know what those are?
I hope I’ve cleared up a few more questions, if there is any confusion about what I want to do, please let it be known so I can clear it up for you
-LadonJanuary 6, 2007 at 1:45 pm #250351
So, if I understand correctly, you want to replace the ratings page by something like
“How would you rate this film’s animation?”
1 – No animation. Still frames or live action.
2 – Very jumpy and inconsistent
3 – The framerate is low, but otherwise acceptable
10 – I thought it was real!
and so on for the other categories, where the user can choose for one option or, alternatively, put in his number between 0 and 10 manually. If this is indeed your plan, then I’m all for it.
Note that requesting something from the site programmers is not, a priori, excluded. If the idea genuinely improves the functioning of this site, then of course it’s worth spending some time on it! For the record, changing the current system so that it keeps track of individual votes would require about 15 minutes’ work, I think.
Personally I am not against keeping a record of the ratings. Apart from discouraging “corrective” ratings (the primary object of discussion in this thread, it seems), it would go some way to do justice to the way ratings are often used: as surrogate reviews. I am against systematic policing of ratings, and I believe it is undesirable to remove ratings, regardless of the motives behind them. Especially with the small number of ratings per film, extreme votes do not seem to be evened out easily, so some discouragement is warranted.
Another point: are the current categories adequate? Especially the “Special Effects” category irks me. There used to be a time when everything we do here was considered a special effect. Now we wonder what to do with films “without” special effects. We see that contest judging has moved from these categories; why not improve the ratings system likewise? Of course, this would void the many hours of work several of our members have put in the current ratings, and as such a change like this should not be implemented lightly.
Stefan.January 6, 2007 at 1:58 pm #250354
I agree with you on the problem of Effects. I am considering other terms for it, perhaps something that would encompass both Effects and Visual Clarity (such as camera focus and quality).
I have also noticed that the first four categories are technical ratings, whereas the last two are more to do with the overall film itself. I am considering separating these and moving them around, so people can look at those two ‘enjoyment’ ratings (enjoyment and story) and decide if they want to watch the film. Then they can look at the technical ratings to further decide if they will appreciate it depending on those.
I realise that adding another rating or removing one would have an impact on films already in the directory, so I’m focusing on clarifying what each category should mean and how it would affect the pre-existing ratings. I’m confident that I can find a balance between the two.January 6, 2007 at 9:01 pm #250392
Suddenly it’s not about the end result, but how much ruckus you can stir up along the way.
You opened a thread about your ideas for changing the ratings system, you should expect some response. I’ve made two posts in this thread, both of which voiced genuine concerns about your plan (which were largely alleviated by yourself and Stefan). Furthermore, with you being the only candidate so far to actually lay out a plan of any kind (and a detailed one at that), you certain have my vote as we speak. I believe Cometgreen said something similar.
So don’t complain.