Thoughts on game journalism

Posted on by

Been busy this week – lots of Dragon Commander presentations going on, so I’m happy that I found a bit of time to write this entry.

An interview Sean Ridgeley from Neoseeker did with me went online today. The topic is games journalism and we agreed that once the interview was live I’d post some extra thoughts here, basically the notes I made for myself prior to the interview.

Games journalism obviously is a sensitive topic for a game developer because it’s like a girlfriend asking – do you think there’s something wrong with how I cook? If you’re honest you’re doomed, if not, you can look forward to things like oversalted steak for the rest of your life :)

Before delving into this, I ‘d like to say that the best thing players can to do when it comes to judging which game they should buy, is to find reviewers that like the games they like, and stay aways as far as they can from sites like Metacritic. I say this because the list of factors that can affect a review is enormous.

One of the investments we made in trying to sway journalists to write well about our games

First off, you should be extremely wary about day 1 reviews – the probability that something stinks about those reviews is rather high.  Just go to metacritic if you’re into the scoring game, look at the critic scores and then look at user scores. Also look at the quantity of reviews on metacritic and check out the dates of the review. For bad games with initially high metacritic ratings, you’ll see a pattern emerging.

There’s a RPG that was released not that long ago and which I haven’t played yet, but the claims that were made about that game in certain reviews are so unbelievable, that I don’t believe a word of what  the early day reviewers have been writing. There were even previews of the review, with some of the wording completely inconsistent with previous reviews from the same magazine when it came to RPGs.

As I said I still need to play it, so I’ll find out for myself – but despite having a high meta critic ratings, it has a 6 as a user rating, so that’s probably an indication I won’t like it. If this were booking.com actually, I wouldn’t book it, based on the user reviews.

It was the PR management around that game that got me thinking again about what affects a review. Here’s a few observations:

Big studios, big publishers: As a reviewer, do you dare go against big publisher X who might happen to be the company indirectly paying a large part of your paycheck through ads? A lot of reviewers say they do, but when you look at their scoring behavior, you can see that in a lot of cases – they don’t. There’s often a little scoring bonus I think.

Not that that should surprise you, the recording industry has been doing this for ages. Money just matters, as does size. I got reminded of this the other day when after doing an interview, we got called by the the advertising manager with the question– so how much pages will you book ?

Another problem is that a lot of journalists are working in a stressed environment, typically understaffed or underpaid, with too many games to review and not enough time to review a game. I don’t think that helps them to write a reasonable review, especially if their part of a publication that wants to cover everything.

Then there’s the feeling that’s created around a game. If a big publisher takes you to 5 star hotel , organizes insanely cool things for you to do, shows you the game for a day and then lets you spend their money as you enjoy whatever exotic location for the next 3 days  – it’s hard not to feel benevolent.

Personally, I think that if you participated in this it should be mentioned at the start of your review. Environmental conditions always affect how you think about something, just like first impressions.

Another problem is that because a lot of publications are often under financial stress, even the metacritic ones, a lot of reviews for lesser-known games are written by the interns or volunteers. These obviously aren’t used to being bombarded with great proposals, and it’s easy to see that they might be easily impressed.

They also might not be aware of their responsibility. If you’re the unlucky developer who crosses a game reviewer who has opinions that don’t necessarily correspond with the target audience of your game, nor your own for that matter, you might be  doomed.  I’ve heard of one RPG reviewer who gives a -15% penalty to a RPG that doesn’t have a trade system that fits his taste..

Trends are also a dangerous thing. Early reviews are sometimes completely desynchronized with how the players perceive a game, because a big publication set the tone. This can have a disasterous impact on the developers.  If reviewers are stressed for time, it’s imaginable that they might have a look at what others think, and this could lead to an unfair review. Or if the early reviews are part of a well-organized campaign with limited access given to the code, the trend might lead to inflated scores.

The region in which the game was developed and where it’s being reviewed also matters. Games that are overhyped in the US sometimes get smashed to pieces in the Germany,  and vice versa. Yet, the taste in games is not that different when it comes to RPGs for instance. Why is that?

I also think that reviewers that really want to be developers shouldn’t review a game – at least not if they’re on metacritic. But that’s just me ;)

I also get very frustrated by the lack of consistency in some of these publications. We actually used to listen to reviewers to adjust our designs, but we gave up, because there’s no consistency. Even not within a specific publication.– they are measuring creativity on a scale, used by everybody in the magazine. That doesn’t mingle well with the subjectivity of a reviewer

I can rant on, but I’ll stop now and perhaps continue in a follow up post.

Here’s one last thought though. Perhaps somebody should do a metacritic of the reviewers individually. After all, a review is always going to be subjective, nothing to be done about that. But people should know of each reviewer how different his opinion is from the rest, and preferably according to target audience.

Anyway, that’s part of what I think on the subject. I’m not against game journalism, on the contrary. I discovered many games because of positive reviews, but I do have a dislike for how the playing field is organized nowadays, and would prefer that particular part of my industry to be extremely objective. It has an enormous influence on the types of games that get played and made. I realize that that’s a futile dream, but I’m an idealist ;)

  • http://twitter.com/chautemoc Sean R.

    “Another problem is that a lot of journalists are working in a stressed environment,
    typically understaffed or underpaid, with too many games to review and
    not enough time to review a game. I don’t think that helps them to write
    a reasonable review, especially if their part of a publication that
    wants to cover everything.” Spot on.

  • d p

    I agree with everything you said. That’s why when deciding whether or not to buy a game, I pay no attention to reviews. I just look at gameplay, features, company, competition and then decide whether or not to buy a game. If I like the gameplay and features, enjoyed companies previous games and there’s no other games which might be better then this one- I’ll buy. Sometimes I also look at reviews for the bugs, but that’s it. 

    • http://twitter.com/SeanRidgeley Sean Ridgeley

      I think the point was games journalism should be improved, not ignored. There are many reviewers that offer a great and very valuable service to gamers – it’s just a matter of seeking them out.

  • js

    I am reading Metacritic’s user reviews for DA2. Funny as hell!

  • http://www.facebook.com/Fodbaddun Thijs Morlion

    As a developer to be and a game journalist myself, I do recognize some things you say in some friends of mine. I disagree on the fact that people who want to be a developer shouldn’t be a game reviewer. Developers play games in another way other people do. And that can be interesting for a review. Ofcourse, it’s an opinion. ;)

    Personally, I have to be careful not to get too technical in my reviews. My very first review was a review of Divinity 2: Ego Draconis. And that was pretty much the one that got me into the journalist world. Since then, I try to combine my dream to be a gamedev with my freelance job as a game journalist. And it works, for now. :)

    And I always try to be honest in my articles. I really don’t mind if a game from a big company is bad or not. If it’s bad, it’s bad and my score has to reflect it. That’s my duty as a reviewer. Unfortunately, there are some big players that threaten some sites with not giving games in the future if that one game gets a bad score. So it comes from two sides, the press depends on those companies to get their games and have visitors to read their site and buyers to read their magazines. And I think it’s not good that these things happen….

  • http://rudolphthesnowdeer.myopenid.com/ Rudolf

    One thing that’s funny, though, is how Dragon Age 2 is consistently brought up to illustrate the gap between professional reviewers and users, with the underlying assumption that the users are more correct. Only… they aren’t. The game was subject to a major smear campaign by the internet hate machine, and as a result the user scores are way off the mark. The reviewers are the ones to trust on that one. Maybe not the day 1 guys, but in general.

    I also don’t think a “metacritic for reviewers” is a good idea. Some of the best reviewers around are people who are, almost consistently, not in line with the “general consensus”. See Kieron Gillen, Tom Chick, et.c.

    Metacritic is way too powerful, I agree with that. And I agree that many reviewers on Metacritic-sites don’t take their jobs seriously enough. But … it’s not really their fault that publishers are using Metacritic as the holy grail of everything. And they shouldn’t care about that. They should only care about judging the games properly and fairly, for the benefit of their readers.

    • Farflame

      I though that first of all DAO2 was subject of big PR campaign. If the game (and reviewers who clearly keep silence about the flaws) disappoints so many players is it smear campaign? Do you really think there was smear campaign and at the same time no overhyped PR campaign? Ehm ehm…
      Do you think reviewers who cant take their jobs seriously (as you said) got – in this particucal regard – some godly conscience and focus so they wrote the truth about DAO2 while thousands of players are just products of hate machine? That would mean that million(s) of dollars for PR campaign had no effect while small sort of haters influenced millions with zero dollar support … That is opinion I call FUNNY and Im sure thats not just me.. :-)

      • http://rudolphthesnowdeer.myopenid.com/ Rudolf

         I know there was a smear campaign. There was a lot of talk about it at the time, it was organized through boards like 4chan. People went to Metacritic in droves to downvote it because BioWare did something they didn’t like, and most of the user scores are completely ridiculous.

        • Tuco Benedicto

          Except user scores aren’t usually reliable as single votes, more like statistical average.
          And actually I think that DA2 shouldn’t really get anything above a 5 or 6. It’s not just a poorly executed game and a betrayal to the spirit behind the first, in some case it becomes downright insulting (see shamelessly repeated content) so to some extent a 4,2 holds more ground than a 8,5.

          • http://www.lar.net/ Swen Vincke

            >>Except user scores aren’t usually reliable as single votes, more like statistical average.<<

            That would've been the point I wanted to make – we use metacritic as an average for what reviewers think, we use tripadvisor or booking.com averages to figure out if we're going to book a hotel or not (which worked so far for me), so why wouldn't we use user scores averages, averaged over for instance and metacritic, and gamespot, and amazon, and etc…

          • Tuco Benedicto

            Well, actually I do. Sort of.
            I don’t really care about metacritic in the first place, to be honest, but I’m usually more incline to trust (and agree with) the average user score than the one from the “specialized press”.
            I could literally list dozens of examples where this is true, and just a few exception where user score is way more off than the one for the press.

            But of course ,that’s jsut me, and I get what you’re saying.

        • M[u]ddy

          There was no smear campaign. I was monitoring 4chan at that time and can confirm that while there was a very negative attitude about Dragon Age 2, nobody was suggesting to go to Metacritic and give the game a bad score. To the contrary, “raids” are frowned upon on 4chans Video Games forum.

          What really happened is, that a lot of gamers were disappointed with the game and they got angry at the high scores the game got from those early reviews.
          Bioware, trying to save face, claimed that the low scores were a smear campaign by 4chan and encouraged fans to counter the raid by giving the game good scores. Ironically they were the ones who started a campaign, even if it failed.

  • LightningLockey

    I gave up on reviewers ages ago, now it is simple word of mouth that will get me to shell out money.  It is how I ended up buying your first Divinity game.  At least now I know why they pretty much all suck, thanks!

  • http://www.lar.net/ Swen Vincke

    I actually didn’t know about the smear campaign, but I think the observation I made doesn’t only apply to Dragon Age 2. Most recently we’ve had Kingdoms of Amalur – if I could believe the initial reviews, e.g. the one on IGN, it was one of the best RPG’s ever, but that’s toned down heavily in subsequent reviews, and if you look at X360 user scores vs critic reviews you can see that they match. On PC, the user scores are still tremendously out of synch with the critic reviews.  

    • http://rudolphthesnowdeer.myopenid.com/ Rudolf

      I agree Amalur is interesting. Possibly a case of “us” having different expectations than the reviewers, though. I imagine it’s much more enjoyable for someone who loves the model in WoW.

      But again, there are examples of the opposite too. Most “users” seem to dislike Driver: San Francisco, while the reviewers loved it… and in my experience it really is an awesome game, that deserved much more success than it got.

      • http://www.lar.net/ Swen Vincke

        Which is why players need to make up their minds about which reviewers are in synch with them – which is where the metacritic for reviewers idea comes from. I see this as something like:

        John reviewed Skyrim 9/10, Dragon Age 2 6/10,  Gothic 7/10 and Fable 2 6/10. On average he is 5% above user ratings.

        That tells me John has a well-formed opinion on which type of RPGs he likes, that his scores are not that far from what the audience thinks, and more importantly, does he like the RPGs I like or not. If you want to be a bit more cynical you could add:

        If the company making the game is company X, then Jon in general scores 10% higher above average user ratings ;)
         

        • Farflame

          I agree, some sort of scores for reviewers would help. But it will also need to add more opinions/scores to reviews from more editors (by default). For example if all games in the list (Skyrim…. Fable 2) are rewieved by different person with completely different viewpoint then it will be harder to find the person who has similar view as you.

        • http://rudolphthesnowdeer.myopenid.com/ Rudolf

          Perhaps useful, yes, but don’t you think it would have an effect on the way reviewers score games? We already know how websites are constantly being pressured by the publishers, how some are blacklisted because of low review scores et.c. Don’t you think a system like this would lead to even more pressure, but on the actual reviewers instead of the websites? Is it fair to subject them to that?

          And don’t you worry that it would reinforce the lemming-type behaviour where reviewers would be afraid to stand out with a lower or higher score than average?

          As I said, it’s not the reviewers fault that Metacritic has the position it has. The problem is Metacritic being too powerful. That problem is not solved by making another Metacritic for reviewers.

          • http://www.lar.net/ Swen Vincke

            Honestly, given the responsibility they have (as a limited group) on sales and thus on whatever future games get made, I think having pressure on reviewers to offer full disclosure on how a review comes to be is the least of things. Additionally, putting a bit of pressure on them to ensure that their opinion is well-formed and based on solid comparisons to the state of the art, can’t be a bad thing. 

            In the other reply, you mentioned you can’t trust player  scores. What makes a reviewer for a magazine so different from a typical player, other than the fact that he managed to land a job with a publication ? We’ve seen scores of 2/10 for games that really don’t deserve a 2/10 from “professional” reviewers too.  

            I really think that if there’s accountability for what they write, you’ll get fairer reviews, which at the very least will be more consistent.

          • Kholdlink

            I can at least trust a reviewer with a publication to not give a game 10 or 0 just because he disagrees with the average score.

          • http://www.lar.net/ Swen Vincke

            No you can’t ;) But yes, I’ll admit that what’s going on with Mass Effect is over the top. But nobody says there can’t be a counter-movement. Maybe digital services (e.g.  Steam) should allow you to rate the game while you are playing it in order to get a real broad base for the user score, and that should be the basis of the meta-critic user score.

          • Arne

             That’s a fine idea when it comes down to ‘good games’. I can’t imagine they’ll allow ‘bad’ game scores, who’d buy it anymore? Steam is basically a shop, you can’t expect it to admit “yes this game sucks according to our customers but it’s still for sale here!”

          • Anonymous

            You mean like Newegg, or Amazon?

          • Arne

            That’s a good point

    • Johnny

      Amalur had a demo that, a handful of bugs aside, was a fair representation of the full game as a whole which is why I made the extremely rare decision to buy it on release. I also do think it was one of the best RPGs ever, at least in modern times. The world has a unique flavor and the combat is second to none compared to virtually every other similar RPG.

  • Haba

    In the old days I could read the review for… say Fallout from a magazine and then walk straight to the store and buy a copy.

    Nowadays I skim through the same magazine and when I actually want to buy a game, I go to a web forum to read the peer impressions. Usually the games I buy are not even covered in the magazines at all.

    Dragon Age, Dragon Age 2, Mass Effect, Fallout 3… So many games that have received absurdly high scores, while other games get penalized heavily for the same faults that were ignored in those titles. Especially painful when those faults are the kind that make the games unplayable for me.

    Gaming media has become completely worthless to me. At best they can offer comedy as the obviously paid off reviewers try to praise the game as much as they can without losing face (10/10 for the 2012 “Syndicate”? Come on…).

    • http://www.lar.net/ Swen Vincke

      Really, that got 10 out of 10 ? 

  • http://www.facebook.com/people/Masek-Masłowski/100002556366353 Masek Masłowski

    “The region in which the game was developed and where it’s being reviewed
    also matters. Games that are overhyped in the US sometimes get smashed
    to pieces in the Germany,  and vice versa. Yet, the taste in games is
    not that different when it comes to RPGs for instance. Why is that?”

    I would not be so sure about it. Planescape Tournament got great reviews everywhere but sold badly in USA, in Poland it was a best seller. Gothic, and Gothic 2 got good reviews in Poland, and was a best seller, but didn’t sell at all in USA.

    • http://www.lar.net/ Swen Vincke

      I’d be interested in seeing somebody really study this topic – my evidence is anecdotical but it’s a pattern I’ve seen quite a lot. 

  • Arne

    These problems could be avoided if a player was able to varify the game for himself: i’m talking about demos. That’s why I chose to buy DKS for example, (or recently Anno 2070), without ever reading any review. Thanks a lot guys!
    And yes, I do agree every gamer needs a trustful reviewer. If the current one proves not, it should be obvious they’ll go and find a more suitable next time. I do read reviews and I never did regret it.
    Like last year I read pretty exciting stuff about both DKS and DC, so why should I quit?
    Giving scores to reviewers though, perhaps it’s enough to assign some ‘favorite labels’ to them? Like “loves RPG, RTS, top 10 games: …”
    ps: finally some new DC videos coming online!

  • Fabrice Lété

    I still remember, a decade ago, before I joined Larian I was an avid reader of a video games magazine. 

    Then came the “review” of Divine Divinity: an “humorous rant” that told little about the game, but concluded it was extremely bad and gave a terrible score. One month later, in a very small erratum they basically said “sorry we tested the wrong version, the game is actually fine”.

    A few years later, Beyond Divinity came out. I only worked a very little bit on it (some quest scripting) but was feeling immensely proud nonetheless. I still vividly remember the shock when I read the dreadful review of that same magazine. They stated once again DD was awful (apparently overlooking their own erratum), and added some downright offending and condescending remarks about Belgium and Larian, before giving BD one of the lowest scores ever.

    Without mentioning my involvement in the game, I carefully managed to ask “why the hate?” to people working at this magazine, the answer I got was: “2D RPG are niche games nobody cares about anymore, good scores are out of the question”.

    Now I always keep this whole story in the back of my mind every time I am reading a review of a game or anything else for that matter. I also remember the countless hours I spent playing and enjoying DD, and the many people I met who told me how much they enjoyed DD and BD (I haven’t really played BD myself, working on it completely spoiled the story). Finally I remember how much fun I had in my gamer’s life with titles that got slaughtered by the critics.

  • js
  • Daan

    I don’t trust all reviewers anyway.. I know one site (won’t give the name) that gave one game a bad score lately. Later they said to the devs of that game, if you pay 20k, we can change our bad score to a 8. (but the publisher of that game had like 0 euro budget for marketing, so the bad score still stands)
    + I see a lot of reviewers who seem to be overexcited about just any AAA title, while there are bad or too short games between them.
    Metacritic user reviews do help a bit, but still, most people only do go there and write a review when they think the game sucks.. most people who like the game just don’t write one, like me. It can give a complete wrong view of a game, but ofc when there are a lot of bad reviews, there must be something wrong with the game.

  • http://twitter.com/JerreyRough Chris Dempsey

    This is why I like TotalHalibut. He makes his living off youtube, shoutcasts, and doing games commentaries at various conventions, but has “WTF Is…” instead of reviews. Being a cynical person, he doesn’t mind going up against the man. He gives an honest opinion about the games he covers, and he covers a nice variety of games (from AAA to random indies). I also tend to trust what he says because he’s got a law degree and tends to be sensible with his opinions.

    You know, these “Let’s Play” videos are starting to gain more and more momentum as the “tipping point” for whether gamers buy a game or not. It certainly is for me. The number of people that watch these for this reason is still small, but I think that number will only increase in the years to come.

    • http://www.lar.net/ Swen Vincke

      Quite cool & interesting take – has a higher following than a lot of magazines I know too. 

    • Peter

      Let’s not get carried away here. TotalHalibut has his biases as any other. He gushes on most games and don’t forget that he enters every beta and gets most games before they are out to the public. His privileged content (and thus his fanbase) come from the publishers and he mostly is very positive about each game. A “cynical” look at his “WTF is….” videos woulds say that they are glorified previews.

      In the end, I don’t swear by anything. I really have nothing to recommend except to be a smart consumer. You know, I think that somewhere the type of consumer who buys 12 games a year is the type of consumer who gets carried away by the hype and deep down doesn’t really care about reviews. He’s just a mindless consumers who enjoys the thrill more than the game itself. Those that only buy very few games do their homework and keep their wits about them, for example I hardly believe they give a shit about day 1 reviews.

  • Dtweak

    Game reviews used to go over everything about the game. Even if they weren’t high scored you could figure out if you should buy it. Nowadays, they are too opinionated and are mostly  ”I didn’t like this” or “I hated that ” instead of telling people who these games will appeal to.