XNA Peer Review System
July 26, 2011 9 Comments
Normally at this time I would be posting my latest Indie review. But today I have something different to talk about. Edward Di Geronimo of Saturnine Games sent me a request to review his latest game, Antipole. And although I really had a good time with it (expect a full review later today) I had some pretty major technical issues while playing it.
I had gotten about forty-five minutes into the game and was really enjoying it. I did notice an occasional hiccup in the frame rate when I would destroy an enemy, but thought nothing of it. And then, later in the game, the hiccup suddenly became a major headache that caused the game to skip like a scratched DVD. Combined with acid pits that zap all your life instantly, I had a legitimate problem on hand.
Since Ed at Saturnine Games requested I review Antipole, I figured I would just ask him if he was aware of the problem. He told me that he had heard of it from two other people. After comparing notes with his previous reports, I tried the game again and it briefly seemed better, but right before the final boss fight the skips returned. I did manage to beat the game, but did so only by learning to work within the skipping issue. I suppose Ed could just roll with it and call it a feature. It works for Microsoft.
Having been told previously with The Cannon that some games are compatible on some Xboxes and not on others, I once again compared notes with Ed. I learned that both previous reports he had came from guys who were using completely different hard drives and systems than I was. I eliminated my hard drive as the cause and switched it over to my Xbox Elite circa 2007. I was easily able to recreate the same problems. Ed reported to me that others had brought up slowdown but didn’t find it significant. And perhaps it’s not, as I was still able to finish the game and despite being frustrated by the cheap deaths the slowdown caused, I still really, really enjoyed Antipole. But suddenly what I was told had been reported by only two people now sounded like more than two.
Ed was more than gracious when I brought my concerns to him, and we are both working together to figure out what’s going on. Shortly before this piece was posted, Ed contacted me to let me know that he himself found the slowdown issue. This was disappointing for him, but at least we have established something is wrong. Which is odd because Ed assured me that Antipole passed Peer Review, and that the problems with it were isolated to myself and two other reviewers. Oh, and all those other people who mentioned it to him as well.
I’m going to chalk this whole episode up to the peer review system being fundamentally broken. Some developers have confided to me that they believe the system doesn’t work because it creates an atmosphere of “you scratch my back and I’ll scratch yours.” I’m not a developer of XNA, but as an outsider looking it I’m going to say that people who practice this are just screwing themselves.
When you do your peer reviews, holding back on any criticisms related to functionality of games is not going to help you, the developer you’re reviewing, or the XBLIG platform. Developers have told me they feel pressure to give positive reviews out to help their own games pass peer review with minimal resistance. Some have told me that before this became the norm, games could end up waiting in line to pass review for months.
I’m certainly not blaming developers for the system being pretty sucky. That’s on Microsoft and nobody else. But for now it’s all you guys have, so you should make the best of it. And that doesn’t include letting glitches or other issues slide by because you’re afraid that telling someone will render them butthurt and they will retaliate against your game. You guys are a community and this means helping one another. I seriously doubt anyone’s feelings will be hurt if they are told there is a major flaw in their game that hampers it playability. And yet seven separate developers have told me that they feel that making too much noise puts their own game in danger.
And I want to say that very few of those who I’ve spoken with have said that it’s a hostile atmosphere. Instead, they say that developers create pressure on themselves upon reaching Peer Review. At this point I’m sure seeing the light at the end of the tunnel is an exciting time. You’ve worked hard for months or maybe even years and finally the game is near completion. Developers I’ve spoken with have said that Peer Reviewers have told them that they would give a game good marks even if there was a design flaw and that they could just patch it at a later date. Others have said they feel XNA developers sympathize with being in the same situation and give a game passing marks whether it merits it or not. Or, I’m told that Indie games are just held to a lower standard and glitches are expected by people who do the Peer Reviews. Some believe a lot of the Peer Reviewers don’t even play the games at all.
I’m told this has led to alliances within the XNA community. The “back scratching” scenario I mentioned above, which helps assure that games pass peer review in a more expedient manner and reach the marketplace in all likelihood well before they’ve been subjected to true play testing. Developers make deals to give each-other good marks and pass the review. This isn’t Survivor, guys. The only thing this is going to do is get XBLIG voted off the Microsoft island. Permanently.
I would also like to say that developers should not depend on Peer Review alone. Play through the game yourself. Get your friends to do it. Take notes. Ask questions. Try to break it yourself. Not every little glitch will get discovered, but letting the big ones slip by reflects poorly on the XBLIG market as a whole.
This isn’t me trying to publicly call anyone out. This is me issuing a challenge. Despite all it’s flaws and it’s status as the red-headed step child of the Xbox 360, the Indie platform is a wonderful opportunity for all of you. But abusing the system Microsoft set up for you is not going to be the foot in the door of the game industry that you want it to be. The odds that your game is going to make enough profit to propel your entry into the industry are slim. Developers should treat XNA as a hobby and a chance to build a resume and nothing more. Don’t operate under the assumption you’re going to make a career out of this. The vast majority of you will not. In that spirit, look out for each other. Don’t be afraid to hurt feelings, and don’t worry about someone taking exception to finding a glitch in your game. If someone does threaten retaliation for a poor Peer Review (and I’ve been told there have been some thinly veiled hints of such things), the community at large will deal with them. At least I hope so.
In my month of doing Indie Gamer Chick, the amount of talent I’ve seen floors me. There’s some really amazingly gifted game developers in your community. So in closing, help each other. Nit pick. Offer advice. But do not blow smoke up each others asses. You’re all adults. Albeit adults who make games about rampaging squirrels, flatulence on dates, or God knows what else. Actually I wouldn’t be shocked if someone reads this and green lights a game where you literally blow smoke up something’s ass. It’s just so Xbox Live Indie Game.
Before the publication of this piece, copies of it were sent out to many developers who had not previously voiced concerns to me about issues with XNA. Of the ten people who received a copy of this, eight signed off on it as being completely accurate, with two not responding at all. So don’t shoot the messenger.