A defense of vgchartz - High-Def Digest Forums
Reply
 
Thread Tools Search this Thread Display Modes
  #1  
Old 11-08-2007, 02:19 PM
BStroms's Avatar
Senior Member
Thread Starter
 
Join Date: Jun 2007
Posts: 1,088
Default A defense of vgchartz

Many people on these forums have a dislike for vgchartz and some seem to downright hate it. It's not ever going to be 100% accurate but it's far from being the completely useless guesses many people make it out to be.

Remember NPD numbers are not 100% accurate. They don't have all the numbers, they have some of them and use their formulas to estimate the total. The same method that vgchartz uses. Granted the NPD has more data, leading to more accurate numbers, they don't have a crystal ball that tells them the exact sell through figures like some people seem to assume.

So how accurate are they? It's impossible to tell, as they have no real competitor on the same level as them. But there is a country that does have two more professional services. Japan has both Famitsu and Media Create releasing numbers. And they even release for the exact same time frame making comparison all the easier. So let's take a look at the most recent data I could find released by both. 10-22-07 through 10-28-07.

Famistsu
Wii 35000
NDS 83000
PS3 16000
PS2 11000
PSP 62600
360 3500

Media Create
Wii 27502
NDS 76243
PS3 18785
PS2 11698
PSP 59792
360 3718

Deviation
Wii 27%
NDS 9%
PS3 11%
PS2 6%
PSP 5%
360 4%

This gives an average deviation of 10.33%

For the most recent vgchartz to NPD comparison you have to go back to the month of August
http://www.vgchartz.com/news/news.php?id=564
Here you get the following deviations
Wii 10%
NDS 13%
PS3 20%
PS2 20%
PSP 12%
360 11%

This gives an average deviation of 14.33%. If vgchartz is worthless with an average deviation of 14.33% then at least one of the 'official' numbers from japan must be highly unreliable at 10.33%. But looking at it reasonably, this is a field of estimating, where no one knows exactly what the right number is. And only 14.33% off on average is quite good. (And some of that will be due to NPD not being completely accurate.)


Of course it's always possible that vgchartz is lying outright about their methods and have just gotten lucky lately in the fact that their hardware numbers have been getting closer and closer to NPD numbers all year long. Even their software numbers (which they admit themselves are less accurate) have been getting better. But the NPD's recent behavior has be thinking that's unlikely. Their rant against online sites that estimate the numbers and thier cutting back on publicly released data tells me something has them spooked. They don't care if you and I visit vgchartz and believe their numbers or not. We don't pay NPD any money.

The fact that they're upset tells me that some of their clients are starting to wonder why they should pay NPD when vgchartz gives out numbers for free which are getting more accurate. And reducing the amount of data that is public will make it harder for vgchartz to benchmark their numbers and update their formulas to give more accurate estimates.



Conclusion: Yeah, don't hold vgchartz numbers as gospel, but then again don't do that with NPD either. On the other hand, their numbers seem pretty good, and will likely continue to get more accurate as they talk more businesses into giving them data, and refine their formulas. The hardware numbers especially are certainly good enough now, at least in the US, to treat as if those are the correct numbers until the NPD data is released. Especially since we'll be seeing less from the NPD in the future.
Reply With Quote
  #2  
Old 11-08-2007, 02:27 PM
Godstar's Avatar
Senior Member
 
Join Date: Jan 2007
Posts: 23,629
Default

VGChartz is deceptive. They use NPD numbers to go back and correct their guesses, so most people have no idea how innacurate their predictions really are. They also don't site sources, namely NPD. They are largely wrong, and sure it's fun to use their numbers to guess, but to make it part of any valid argument doesn't work. I'd never use them as part of any dialog with anyone, and will certainly ignore any argument that attempts to be credible which uses it.
Reply With Quote
  #3  
Old 11-08-2007, 02:31 PM
BStroms's Avatar
Senior Member
Thread Starter
 
Join Date: Jun 2007
Posts: 1,088
Default

Quote:
Originally Posted by Godstar View Post
VGChartz is deceptive. They use NPD numbers to go back and correct their guesses, so most people have no idea how innacurate their predictions really are. They also don't site sources, namely NPD. They are largely wrong, and sure it's fun to use their numbers to guess, but to make it part of any valid argument doesn't work. I'd never use them as part of any dialog with anyone, and will certainly ignore any argument that attempts to be credible which uses it.
They do go back and update totals, but they also do a comparison every single month of the numbers they came up with, and the NPD numbers. With all of their numbers officially released before the NPD numbers come out, and the number of people who hate that site, don't you think someone would have, you know, taken a screenshot of their numbers as they came out and caught them cheating when they did the comparison?

No one has, because they don't. So it doesn't invalidate any of the comparisons I've made.
Reply With Quote
  #4  
Old 11-08-2007, 03:23 PM
Godstar's Avatar
Senior Member
 
Join Date: Jan 2007
Posts: 23,629
Default

Quote:
Originally Posted by BStroms View Post
They do go back and update totals, but they also do a comparison every single month of the numbers they came up with, and the NPD numbers.
So if they go back and update their numbers, how would someone who doesn't know that and is looking at the data tell if their guesses are legitimate? They can't. They see the updated numbers and assume they're good guesses, which they're not.

And mind you a 40% margin of error betwen two pieces of data is enormous, and for the most part unusable. Most poles wouldn't even be considered if they were above +/-5% margin of error. Most are around +/-2-3. I don't see why you're so insistant on defending a data source that provides bad data. There's just no reason for it.
Reply With Quote
  #5  
Old 11-08-2007, 03:44 PM
BStroms's Avatar
Senior Member
Thread Starter
 
Join Date: Jun 2007
Posts: 1,088
Default

Quote:
Originally Posted by Godstar View Post
So if they go back and update their numbers, how would someone who doesn't know that and is looking at the data tell if their guesses are legitimate? They can't. They see the updated numbers and assume they're good guesses, which they're not.
I'm not certain on this fact, but I think they only update their totals, not the weekly estimates. I'll have to look into that though.

Quote:
And mind you a 40% margin of error betwen two pieces of data is enormous, and for the most part unusable. Most poles wouldn't even be considered if they were above +/-5% margin of error. Most are around +/-2-3. I don't see why you're so insistant on defending a data source that provides bad data. There's just no reason for it.
Where are you getting the 40% margin of error from? They most they were off on the hardware from the NPD was 20%. Even the 2 Japanese numbers went up to 29% off from each other. And given that was just one week I'm sure they're much further off than that on other occasions. And even assuming the split the difference with one going over the actual number and the other under, I highly suspect both have been off by more than 40% on some numbers on occasion.

I don't think NPD numbers are much better and even claiming +/-40% would be hard for them to make. Because while they're usually much better than that, there will be rare cases they're off by a lot.


This is very different from polling, where you can take a random sampling and everyone has one vote. It's far more imprecise in nature given the vast differences on companies, their advertisements, sales, prices, and customer demographics.

But we use NPD anyways because we don't have the actual numbers, and NPD is the best we have. So when we don't have NPD numbers why not use the best we have available, which is vgchartz. Despite what some people think I'm quite certain it's become far too accurate to just be good guesses.


Still if you want stuff within +/- 5% you might as well just refute all sell through data we get as too inaccurate and worthless.
Reply With Quote
  #6  
Old 11-08-2007, 04:53 PM
Godstar's Avatar
Senior Member
 
Join Date: Jan 2007
Posts: 23,629
Default

If the most they've been at most off on a PS3 by 20% and on the 360 by 20%, that gives you the possibility by using both consoles in the same comparison you're comparison could be off as much as 40%.

-20% and +20% = 40% difference.

And regarding the difference between using NPD numbers and VGChartz numbers is NPD numbers are not predictions, they're recording sales, whether it's all or most of available sources. VGChartz is making a guess, and that's all it is...nothing which is numerically valid until after NPD comes out.

I don't see why you insist on using VGChartz as the equivalent of NPD, when one is a guess based on NPD numbers, and the actual NPD numbers are based off sales data. There is no reason for it.
Reply With Quote
  #7  
Old 11-08-2007, 05:22 PM
BStroms's Avatar
Senior Member
Thread Starter
 
Join Date: Jun 2007
Posts: 1,088
Default

Quote:
Originally Posted by Godstar View Post
I don't see why you insist on using VGChartz as the equivalent of NPD, when one is a guess based on NPD numbers, and the actual NPD numbers are based off sales data. There is no reason for it.
You keep making this claim, but unless they're outright lying about what they do, VGChartz does the exact same thing as NPD. They get as many actual sales numbers as they can, and analyze those numbers, put them in formulas that are constantly being refined, and use it to estimate the total sales numbers. The difference is a matter of scale. The NPD obviously gets a larger data set to work with.

You can keep assuming that VGChartz lies over and over again and just randomly guesses numbers to much greater accuracy than they should be able to do, but I don't see why it's so difficult to believe that determined people couldn't get some people to give them actual sales numbers every week. Especially when they're high enough profile to get the ire of the NPD (they didn't mention vgchartz by name but do you really think they're worried about nexgenwars or one of the other places enough to go on a rant about them)


It's not that hard. I work at a small consulting firm of < 100 people, and the owner of our company was able to get all sorts of information from big companies for a client by promising everyone who participated would get some of the data they collected back to them for free.

VGChartz probably does the same thing. They get some additional data from their sources they don't make public and only give to those who provide data for them. Companies love free data on there business area and the only way I could see them unable to get a good amount of actual sales data is if they're lazy or incompetent.


The fact that their monthly numbers steadily becoming closer to the NPD data all year long being exactly what would happen if they did get actual sales numbers and were increasing their sources leading to more accurate numbers. This seems more probable than believing they're liars too lazy to do what they claim they're doing and just happen to be getting better and better numbers out of pure dumb luck when they make educated guesses with no data.
Reply With Quote
  #8  
Old 11-08-2007, 05:49 PM
Godstar's Avatar
Senior Member
 
Join Date: Jan 2007
Posts: 23,629
Default

NPD Group:

Quote:
At the same time, the numbers leaked before and will inevitably leak again, though this time without the verification we've come to expect every month. Riley anticipates the industry policing itself in these situations and even blasts some of the 3rd party groups. "Regarding rogue sites that actively use existing data from market research firms, combine that data with estimates and then claim the data as their own -- it's true I'm no fan of sociopathic behavior and, as such, have real issues with sites like these but this has nothing to do with the change, either," he said. "The best way to approach this issue is to let the industry police it, or to simply wait and let the truth come out in the end. It always does."

VGChartz founder:

Quote:
1. VG Chartz data is released BEFORE NPD so the raw data IS OUR DATA based on OUR SOURCES, calculated via our PROPRIETARY METHODS.

2. We do use historical information in a number of forms to help in extrapolating and analysing raw data - just as NPD and every tracking firm under the sun does.

3. There is no truth to come out here, it is abundantly clear what vgchartz is aiming to do and how we do it, our data is far more transparent than NPDs, ALMOST as accurate and we CHARGE NOTHING for it. Not bad if you ask me."
The one thing he doesn't say is that his proprietary method uses NPD numbers, they won't say where they get their numbers, they just mysteriously adjust them when NPD releases theirs. They even go so far as to lie about being transparant, they're not. Most everyone knows they're basing their information on NPD. Lastly, I wasn't the one making the claim, NPD was the one making this claim.
Reply With Quote
  #9  
Old 11-08-2007, 06:25 PM
Banned
 
Join Date: Feb 2007
Posts: 1,055
Default

At best, VGA chartz is an educated guess.

Even then, its more of a prediction than actual numbers.
Reply With Quote
  #10  
Old 11-08-2007, 06:35 PM
BStroms's Avatar
Senior Member
Thread Starter
 
Join Date: Jun 2007
Posts: 1,088
Default

Quote:
Originally Posted by Godstar View Post
The one thing he doesn't say is that his proprietary method uses NPD numbers, they won't say where they get their numbers, they just mysteriously adjust them when NPD releases theirs. They even go so far as to lie about being transparant, they're not. Most everyone knows they're basing their information on NPD. Lastly, I wasn't the one making the claim, NPD was the one making this claim.
Like they said in the quote, their numbers come out before the NPDs. That means they cannot be based off the NPD unless they've somehow stolen the NPD's internal data (and if they have it would be even closer to the NPD's.)

It's extremely simple, X comes out before Y is available, therefor X cannot be based on Y. So the weekly sales numbers, when they're released, are not based off of the NPD numbers.

What they're saying they're doing is that once the NPD numbers are released, they do look at them, determine how their numbers are off, and analyze them. For example, they might conclude that they weighted a certain sales source too highly because it's customers are more informed than the average consumer and is more affected by the release of big name games than other sources. So they adjust their formula to reduce its weight whenever there's a major release.


It's a good thing that they do this. It increases there accuracy, but it doesn't mean their numbers are based on the NPD's, just benchmarked against them.
Reply With Quote
Reply

Related Topics
Thread Thread Starter Forum Replies Last Post
Defense Grid: The Awakening Master X Game Room 9 09-06-2009 02:58 PM
Final Fantasy Crystal Defenders ( AKA Tower Defense ) SupraDoom Game Room 5 08-07-2009 02:11 PM
In defense of Darkfalz! TheLastGenius Gaming Smackdown 5 07-07-2008 08:49 PM
Sony Defense Force MGS 4 review ! Leon Gaming Smackdown 9 06-04-2008 12:26 PM
Take-Two Directors Building Defense Against EA Gutter21 Gaming Smackdown 3 03-23-2008 06:58 PM


Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are Off
Pingbacks are Off
Refbacks are Off