How reliable are VGChartz?

[avshaman]

How reliable are VGChartz?
LazySAGamer - November 12th, 2008 at 11:00 am - 118 views - 8 Comments

VGChartz vs NPD Recently I have noticed a strange phenomena going around, whenever I publish anything that relates to or mentions VGChartz there is a small but vocal group of people who then refuse to believe anything else that the article contains.

I am not sure why this has started happening but I can only presume that it is based on the fact that VGChartz use estimated figures whilst NPD use solid figures from the suppliers.

At first glance that makes sense but in fact it is not actually correct. Yes VGChartz use estimates but they also reconcile their figures when the factual figures are released and adjust their future estimates accordingly. This basically means that they can get things horribly wrong at times but when you look at the figures over time you will start to see that their estimates are very accurate and VGChartz is actually a perfectly reasonable place to gather statistics from.

Don’t believe me? Well don’t take my word for it but rather head on over to O’Reilly Radar where Robert Passarella has done a scientific and factual comparison between the two and has come to the conclusion that NPD is a perfectly valid place to gather statistical data from.

http://www.lazygamer.co.za/console-s...e-are-vgchartz
In that last sentence he obviously meant to say VGChartz.

Also read:
http://radar.oreilly.com/2008/06/dat...rtz-vs-np.html

I just thought this was interesting because we are always discussing sales data in this forum and there seems to be at least some disagreement about just how reliable or unreliable VGChartz numbers are. What do you guys think?

[bomba]

Basically they're saying VGChartz is reliable because they always edit their data after official info is released.

Thats so pathetic that people rationalise that way.

'Its OK to make up bullshit as long as you try and hide it later!"

[avshaman]

No, that isn't what they are saying. Please read again.

[Gbeav]

They are only reliable after they fix their numbers. Which makes them reliable 12 days of the year and full of shit the rest.

[bomba]

Thats exactly what they're saying, unless you interpret it as their future predictions are getting more accurate and not merely they're revising their past, in which case I'll say thats a flat-out lie.

[Dennis M.]

Even a broken clock is right twice a day.

[avshaman]

I haven't made up my mind, but I find the whole argument interesting.

Here is what a certain JesM wrote in the comment section of the Oreilly article:

1. Remember that vgchartz isn't trying to match NPD per se, they are trying to offer their own take on what the best selling games and hardware are. We are trying to assert accuracy to reality by comparing to another estimate which isn't really a smart move since you don't know where the errors lie.

2. Having said that, numbers for May from vgchartz were spot on with NPD and closer than both Pachter from Wedbush Morgan and Divnich of Eedar - both professional analysts with large pay packets for what they do. They were also closer than Simexchange, the prediction market - many users will of course use vgchartz to assist in the predictions. And still vgchartz, which releases data first, is closest. This is true virtually every month.

Saying that numbers play into your favour now and again is insanity. Why should anyone be able to get close to predicting or estimating something as multi-faceted and complex as videogame market sales numbers? To be even in the right ballpark month on month is an immense achievement, and for literally thousands of skus. All for free.

I think you need to give some credit where credit is due here and recognise just what is being done.

Eric Aston wrote:

Disruption is just it, Sri Lumpa. I started looking into disruption as I watched Wii's success, and VGChartz fits with it pretty well, including the lashing out from NPD.

To make a long story short, VGC has different processes, different motivations and different values than NPD. NPD and their surrogates are unable to respond in any way except flailing, screaming, and running upmarket to their corporate customers.

Timliness and openness are the values which helped VGC first play to the fringe market of forum dwellers who used to hang on NPD numbers, and move up towards journalists and small investors, most of whom were previously non-customers because NPD's accuracy focus overshot them. NPD values timliness so little they frequently delayed their data in 2007, and they value openness so little they attempted to stop all public data releases in 2007 as well. They are increasingly only concerned about the most upmarket, profitable customers: game publishers and other large corporations.

I've been using VGC as my primary source for playing theSimExchange game for over a year, and even without being particularly active, I'm ranked #16 currently.
http://www.thesimexchange.com/rankin...php?pageview=1

I post on the VGC forums as well, as Erik Aston.

http://radar.oreilly.com/2008/06/dat...rtz-vs-np.html

I also want to point out that VGChartz also correctly reported the number of copies of Gears of War 2 that were sold worldwide on the first day. They reported 2.1 million and that number was later confirmed by Microsoft. And I bet the NPD numbers will confirm it as well.

Anyway, read the Oreilly article and especially take the trouble to read through the comments--that is where things get more interesting and even more to the crux of the matter.

[bomba]

No, thats when VGC shills swoop in and start defending the site.

Disruption, blah blah blah, no. Its a site founded by the biggest Nintard ever, after he conned another Nintard out of the 1996-2004 complete NPD report and then acted like a great git towards said victim. He then went through and done very, very, very minor adjustments so he wouldn't get sued and put it online. Now he makes up gibberish prediction that are no more or less accurate that any specific other person, and tweaks the numbers when he routinely misses NPD by large margins. If he didn't want his estimates (read: he gets NO data from stores, so they're more predictions, but he pretends to have methodology) compared against NPD as the first comment you posted suggests, then he should stop revising them after NPD shows a contrasting view.

His site gets 95% of its traffic when Kotaku, Gamasutra, Joystiq and etc. link to it, and they're moving very fast towards TSE, a site which whilst the analyst services are questionable, the prediction market is based off the sound principles employed by HSX (movie exchange).

[Aodh]

The problem is the perception of accuracy. VGCharts really had no reason why it should be accurate as there's no visibility as to how they collect their data. As my former job was a data analyst, this is a pretty big red flag. But that doesn't mean they're inaccurate either. It's just a big warning sign that it may not be right. But for the purposes of bitching on an internet forum, some place where you have no financial stake, they have proven themselves to be accurate enough to paint broad strokes. Just have to be careful that you're not arguing small differences. Then with the historical corrections it makes the long term trend data relevant even it you stop looking at the last month.

[maley]

http://www.lazygamer.co.za/console-s...e-are-vgchartz
In that last sentence he obviously meant to say VGChartz.

Also read:
http://radar.oreilly.com/2008/06/dat...rtz-vs-np.html

I just thought this was interesting because we are always discussing sales data in this forum and there seems to be at least some disagreement about just how reliable or unreliable VGChartz numbers are. What do you guys think?

They're a good indicator if you need to know for w/e reason (usually bragging on forums) what the numbers were a little faster than NPD or Media Crate.

[jpimgt]

They are accurate when their "data" favors the console of your choice, and full of shit when it doesn't.

[RedRedSuit]

Gnome has explained in depth why VGChartz is shit and should not be used. It's a foregone conclusion for me, at this point, and I don't like to revisit the same discussions. It's not trustworthy in the U.S., and it's especially not trustworthy in Europe.

[Badger3920]

"VGChartz is shit and should never be used"

[WallyWest]

Even a broken clock is right twice a day.

I was about to post the same thing

[avshaman]

No, thats when VGC shills swoop in and start defending the site.

Disruption, blah blah blah, no. Its a site founded by the biggest Nintard ever, after he conned another Nintard out of the 1996-2004 complete NPD report and then acted like a great git towards said victim. He then went through and done very, very, very minor adjustments so he wouldn't get sued and put it online. Now he makes up gibberish prediction that are no more or less accurate that any specific other person, and tweaks the numbers when he routinely misses NPD by large margins. If he didn't want his estimates (read: he gets NO data from stores, so they're more predictions, but he pretends to have methodology) compared against NPD as the first comment you posted suggests, then he should stop revising them after NPD shows a contrasting view.

His site gets 95% of its traffic when Kotaku, Gamasutra, Joystiq and etc. link to it, and they're moving very fast towards TSE, a site which whilst the analyst services are questionable, the prediction market is based off the sound principles employed by HSX (movie exchange).

Gnome has explained in depth why VGChartz is shit and should not be used. It's a foregone conclusion for me, at this point, and I don't like to revisit the same discussions. It's not trustworthy in the U.S., and it's especially not trustworthy in Europe.

Telling me "VGChartz is shit" (which is the mantra we hear repeated over and over by some people) means nothing to me. I've heard that claim plenty of times. What I want is for someone to explain to me WHY that is true. Bare assertions and hearsay is not what I am asking for. I am asking for independently verifiable facts and reasoning that will show me why "VGChartz is shit". To be honest, as someone sitting on the fence on this issue I found the arguments of the VGChartz defenders in the comment section of that article a lot more convincing than the arguments put forward by the detractors. Bomba calls the former people "VGC Shills", but if that's the case I hope you guys realize that you come off as NPD shills. Both of you basically repeated the same bare assertions that the detractors on that site did. I don't personally know you guys in real life. For all I know both of you might work for the NPD Group or be associated professionally with them in some way. I'm sorry, but I can't just take your word on this.

From my perspective there are two very different aspects to this issue.

1. The value, if any, provided by the information put out by VGChartz.

2. The ethical question of how exactly VGChartz arrive at their numbers.

I think the first question has been settled fairly conclusively: VGChartz provide numbers that have been shown to be in the right ball park more often than not. I have seen this to be true over and over. The fact that VGChartz can give us something like the Gears of War 2 first day sales estimates and end up being not only in the right ball park but nearly right in line with the numbers Microsoft later reported proves its usefulness and value beyond a shadow of a doubt. If I, as an individual, had tried to estimate what the Gears 2 sales numbers would be, I likely wouldn't have been anywhere close.

But then again the accuracy of VGChartz is not really what is being debated. Everybody knows that their numbers are likely not as accurate as the NPD numbers. Everybody also knows that their numbers are roughly in the right ball park and are provided earlier than other sources. This is why even the most vehement detractors commenting on that O'Reilly article such as "Meister" kept repeating things like "what I said earlier is more about Brett Walton's integrity than his site's usefulness". They know that the argument against the site's usefulness is very weak, so they focus on attacking the integrity of the site's founder. The problem, however, is that they utterly failed to support their claims that VGChartz is acting in an unethical way. The detractors' bare assertions and unsupported claims regarding VGChartz' methods are shameful and next to worthless. Either support your claims with cold hard facts or keep quiet. What's more, even if the things being claimed about VGChartz having no hard data and just using publicly available data were true, it's like.. so what? That would make their relatively successful predictions even more impressive. I know I could never use such publicly available data and come anywhere close to being as accurate as they have been. Even more to the point, though, is that they wouldn't be breaking any laws or doing anything unethical.

This is how I see things at the moment. I am still trying to get to the bottom of this issue and would gladly change my tentative conclusion if you could provide me with facts and reasoning to show me how I am wrong. Repeating the mantra "VGChartz is shit", however, is not going to cut it.