The silliest graph I’ve ever seen presented in public looked something like this:
It was an after-dinner talk at a DAMOP meeting a few years back, and the speaker was somebody associated with the Hubble Space Telescope. I don’t recall what was being plotted, but he talked for a while about ho proud they were of this data, and how well it fit the theory, and then he put up this plot. The blueish circle is the data point, and the dotted line is a theoretical fit to the data.
The physicists in the audience all guffawed. He asked “What’s so funny?” and somebody near the front asked “What’s the chi-squared on that fit?” He blinked for a second, and said “Oh, right. Well, the error bars will get smaller once we analyze the rest of the data.”
OK, maybe you had to be there. The point is, he had no idea why we were busting up laughing. I told this story to an astronomer, who recognized what it must’ve been, and said that there’s actually a valid reason for plotting the data like that– the only point they can measure is the inflection point, or something like that. To a room full of atomic physicists, though, that looks utterly ridiculous– if you only have one data point, you can’t draw a four-parameter fit through it, and claim that it means something. It doesn’t matter how small the error bars are.
I was reminded of this yesterday by a graph in an article plugged in the header bar (reproduced below the fold):
My first reaction to that is “You have to be kidding me.” Shelley reports that the correlation represented by the line “remained significant after multivariate adjustment,” but I’m sorry, I have a really hard time taking that seriously.
This is, of course, why I would never make it as a medical or social scientist. On an intellectual level, I know that statistical techniques are very powerful, and can pull small correlations out of large data sets, but my gut reaction to a graph like that remains “What a bunch of crap.”
That graph is terrible. As you say you may be able to pull out some correlations with statistics, but if they are really there it should be possible to display the data in such a way that the correlation is visible.
I would slice the data in bins of BMI and calculate the mean and error on the mean. This might be able to show some trend toward lower words at higher BMI.
That graph makes me recall a psychology teacher who curve fit a class to a gaussian distribution. The class size was 15. All attempts to discuss the applicibility of that method fell on deaf ears.
Yeah, there are lies, damn lies, and statistics.
The one data point curve is from cosmology. They are just a bit past “the cosmological constant is 1 +/- 1”. If you’re in AMO and used to atomic clock/QED accuracy, it can be quite jarring.
Why don’t we just nail a page with a couple blasts of bird shot and draw a line through it. Maybe we could get published, too! Maybe VP Cheney is available to take the shot so we can say with certainty the data was “random”…
“The correlation remained significant,” I am so sure, but what was the value of r2; let me guess, about 0.02 maybe? Less?
Social scientists always look at r rather than r2, or they do most of the time anyway. I don’t want to accuse them of doing this only to inflate the appearance of the strength of their correlations. So could anyone in the audience explain what good reason there is for this?
eh, both r and r^2 are kinda crappy anyway
In speculative defense of the astronomer who showed the plot—
From what you say, I have no idea what this is, or what the data is. However, I can see a situation where, for reasons other than just this data, you believe that something is following a certain model that is already fairly well specified. Say, how steep the droppoff is, and the top and bottom levels of the two plateaus. Given that, and given that your measurement just measures the inflection point, then it might actually be reasonable to show a plot like the one you show above. I wouldn’t call it a fit, becuase it’s not, but it could be a measurement that pins down the X-axis position of where this curve happens.
An example of something in astronomy that really does have a curve shaped something like that — if you have two different atomic transitions with different lifetimes that come from the same excited state, you can use that as a density diagnostic. In the low-density limit, the line ratio will be just the inverse ratio of the lifetimes. In the limit of higher density, only the shorter-lived line will be visible, with the longer lived line having a much higer probability of being collisionally de-excited before it radiatively de-excites. If you do it all out right, you get a sigmoidal curve.
Now, you go out and measure the line ratio. You might show a plot like this to show how you looked up the density given the line ratio. Again, this isn’t a fit… it’s a measurement, with your model being used to convert that measurement into a derived qunatity that is more what you care about.
Back to making fun of the plots:
If the presenter did call the top thing a fit, then it either has a reduced chisquare of 0/0 (which can be anything) — if there’s one parameter, then the data point will fall perfectly on the curve, but the number of degrees of freedom is also zero — or of negative zero — with more than one parameter, your point will still fall perfectly on the curve, but now you have a negative number of degrees of freedom.
In the latter case, if you overinterpret the idea that “lower chisquare means better fit”, you could then say that because of the negative chisquare, you’ve got an amazingly good fit!
As for the second plot : I’ve sometimes see people using sophisticated statistical techniques on data that clearly doesn’t warrant it. If the simple statistical techniques don’t tell you anything and it’s bloody obvious that you don’t have a correlation, when more sophisticated statistical techniques produce a result, it’s likely that you’ve produced something from nothing…..
-Rob
Any correlation is significant with enough data points.
Opiwan wrote:
Why don’t we just nail a page with a couple blasts of bird shot and draw a line through it. Maybe we could get published, too! Maybe VP Cheney is available to take the shot so we can say with certainty the data was “random”…
In the first volume of his vast autobiography, Isaac Asimov told a story about that. He was working on his chemistry Ph.D. and one day took a set of measurements which scattered randomly across his graph paper. Not one to be stymied by his own laboratory ineptitude, with which he was very familiar, he went ahead and drew a curve of the expected shape through the cloud of points.
Whereupon his thesis advisor promptly dubbed it “Asimov’s shotgun curve” and showed it to the entire Columbia chemistry department.
http://anonymous.coward.free.fr/temp/fat-vote.png
Nice rip-off of a graph on this blog
http://scientific-misconduct.blogspot.com
at
http://scientific-misconduct.blogspot.com/2006/08/procter-research-shenanigans-part-2-on.html
Link
nice fit!
yeh, i’ve been in biophysics for a few years now and every once in awhile someone will put a plot like that out of their ass and i have to stifle a laugh everytime ….
I like Blake Stacey’s summary of the fine on-topic Asimov anecdote.
I actually spent quite a while discussing mathematical enzymology with Asimov, since his dissertation was in what we today would call Classical Experimental Enzymology. He tells a screamingly funny tale in his multivolume autobiography about his PhD Dissertation Oral Defense.
Since I was the second-known SFWA Member who had Enzymology in his PhD dissertation, he made me promise to cite his dissertation — which had never been previously cited in the Biochemical Literature! I’ve done so.
He was abashed at having dropped out of Calculus after being able to do homework problems in Integral Calculus (I believe it was integration by parts) — without intuitively understanding why his answers were right. It really shook him up, epistemologically. So he was ashamed of how primitive his PhD dissertation was by modern standards. I did my best to restore his usual sunny self-esteem.
Asimov, by the way, is the human breidge between the Science Fiction Literature and the Biochemical Literature when you plot the Coauthorship Social Network Graph. Asimov Number is the equivalent of Erdos Number, as I’ve worked out at enormous length a few years ago, but not yet submitted to publication.
I get accused of massive egotism. I don’t hold a candle to heroes of mine such as Asimov, Feynman, Gell-Mann, and perhaps the ultimate example: Newton.
Feynman also warns about bad graphs. He suggests always throwing out the right-most plotted point, on the grounds that the next one right of that was probably so wacky as to not having been plotted in the publication.
Words to live by…
Fortunately, MY worst published graphs predate the digital era, so until they get digitized with the rest of everything, the laughter will stay at the normal level.