Blog

This post is a follow up to my previous on why Climategate is the real war on science.

My earlier post commented on Willis Eschenbach’s excellent column on Anthony Watts’s blog (WUWT). Here I want to reproduce a comment Eschenbach posted in the back and forth on WUWT. In it, he explains why hiding computer code is lethal in a field, like climate science, where results hinge on statistical methods and mathematical techniques.  Eschenbach’s comment follows.

Willis Eschenbach (17:15:51):

Part of the difficulty with climate science is that, unlike all other physical sciences, it does not study things — instead it studies averages.

This is because climate by definition is the average of weather over a suitably long period of time (typically taken as a minimum of 30 years).

As a result, much of the study that goes on, and the papers that are written, deal almost exclusively with mathematics and statistics. This is the reason that access to the computer codes is so critical.

It’s simple in the physical sciences to describe an experiment, e.g. “I took three grams of carbon and subjected them to a pressure of 50,000KPa and a temperature of 500C. Unfortunately, the experiment did not succeed, I could not replace the diamond I had lost from my wife’s wedding ring.” Anyone can reproduce that experiment (and get the same results).

But when you say “I took the raw temperature data, variance-adjusted it, averaged it, gridded it, area-adjusted it, extrapolated results to data-free areas within 250 km, and made a global temperature record”, that’s far from enough information. In order to determine what was done, we need far more detailed information in climate science, because in general we are describing intricate mathematical operations. These are often very hard to describe clearly in spoken or written language.

And even a crystal-clear description is not enough. Despite what he says he has done, if the scientist has inadvertently used an improper procedure (e.g. the uncentered principal components analysis used in Mann’s Hockeystick), we’ll never be able to determine that the answer is demonstrably wrong unless we have the actual code that he used. Otherwise, we could spend years trying to guess where he went wrong, but we would never be able to show that he went wrong as science demands.

This is why the insistence of scientists that their computer codes are sacrosanct private secret documents best kept under Hermetic seal in a clandestine vault is lethal to good science. Without the codes, we can’t tell if what has been done is correct and free from hidden mathematical error. Of course, this may be unconnected with the reason that Jones et. al are hiding their codes …

Given that climate science is not the study of things but of the averages of things, and that as a result math and statistics are central to climate science, the findings of the Wegman Report are now seen to be even more insightful, trenchant, and valid. They said:

It is important to note the isolation of the paleoclimate community; even though they rely heavily on statistical methods they do not seem to be interacting with the statistical community. Additionally, we judge that the sharing of research materials, data and results was haphazardly and grudgingly done. In this case we judge that there was too much reliance on peer review, which was not necessarily independent. Moreover, the work has been sufficiently politicized that this community can hardly reassess their public positions without losing credibility. Overall, our committee believes that Mann’s assessments that the decade of the 1990s was the hottest decade of the millennium and that 1998 was the hottest year of the millennium cannot be supported by his analysis.

And presciently, that was written thee years ago, well before we got the CRU emails …

The huge pile of emails purloined or leaked from the Climate Research Unit (CRU) last week does indeed “give every appearance of testifying to concerted and coordinated efforts by leading climatologists to fit the data to their conclusions while attempting to silence and discredit their critics,” as the Wall Street Journal stated yesterday. However, the main issue brought to light by these emails is even more serious.

In a column posted yesterday on Anthony Watts’s blog, amateur scientist Willis Eschenbach documents the many ruses and excuses CRU director Phil Jones and his allies employed over several years to deny outsiders access to the CRU gang’s temperature data and computer codes.

Skeptics have been accused of waging a “war on science“ because they frequently question the Intergovernmental Panel on Climate Change’s (IPCC’s) interpretation of the rapidly expanding field of climate change research.

 But science is not a set of dogmas certified by government-funded bodies. Rather, as Mr. Eschenbach points out, science is fundamentally an ”adversarial process” whereby competing scientists attempt to reproduce — that is, invalidate — each other’s results. This process absolutely depends on each combatant allowing the others to examine his data and methods. Tactics designed to hide data and methods are anti-science even if — nay, especially if – those resorting to such tricks are big-name scientists.   

“Science,” writes Eschenbach, “works by one person making a claim, and backing it up with data and methods they used to make the claim. Other scientists then attack the claim by (among other things) trying to replicate the first scientist’s work. If they can’t replicate it, it doesn’t stand.”

This means, says Eschenbach, that researchers who hide their data and computer codes to prevent others from replicating/invalidating their results “attack . . . the heart of science.” Such behavior is unethical and, as Eschenbach notes, likely illegal as well.

If you read only one commentary on Climategate, read this one. It is an eye-opener.

You can download the entire zip file of the uncovered emails and documents here but it is 61mb of text. The searchable files are here. A good list of the worst emails discovered so far is here. Here’s a very funny video on the whole mess. But here is the best article, by Marc Sheppard, that shows that all the data and wherever it has been used–Kyoto and IPCC–has to be thrown out. The data has been massaged, corrupted, manipulated, and abused beyond recognition:

One can only imagine the angst suffered daily by the co-conspirators, who knew full well that the “Documents” sub-folder of the CRU FOI2009 file contained more than enough probative program source code to unmask CRU’s phantom methodology.

In fact, there are hundreds of IDL and FORTRAN source files buried in dozens of subordinate sub-folders. And many do properly analyze and chart maximum latewood density (MXD), the growth parameter commonly utilized by CRU scientists as a temperature proxy, from raw or legitimately normalized data. Ah, but many do so much more.

Skimming through the often spaghetti-like code, the number of programs which subject the data to a mixed-bag of transformative and filtering routines is simply staggering. Granted, many of these “alterations” run from benign smoothing algorithms (e.g. omitting rogue outliers) to moderate infilling mechanisms (e.g. estimating missing station data from that of those closely surrounding). But many others fall into the precarious range between highly questionable (removing MXD data which demonstrate poor correlations with local temperature) to downright fraudulent (replacing MXD data entirely with measured data to reverse a disorderly trend-line).

In fact, workarounds for the post-1960 “divergence problem”, as described by both RealClimate and Climate Audit, can be found throughout the source code. So much so that perhaps the most ubiquitous programmer’s comment (REM) I ran across warns that the particular module “Uses ‘corrected’ MXD – but shouldn’t usually plot past 1960 because these will be artificially adjusted to look closer to the real temperatures.”

What exactly is meant by “corrected” MXD, you ask? Outstanding question — and the answer appears amorphous from program to program. Indeed, while some employ one or two of the aforementioned “corrections,” others throw everything but the kitchen sink at the raw data prior to output.

For instance, in subfolder “osborn-tree6mannoldprog” there’s a program (Calibrate_mxd.pro) that calibrates the MXD data against available local instrumental summer (growing season) temperatures between 1911-1990, then merges that data into a new file. That file is then digested and further modified by another program (Pl_calibmxd1.pro) which creates calibration statistics for the MXD against the stored temperature and “estimates” (infills) figures where such temperature readings were not available. The file created by that program is modified once again by Pl_Decline.pro, which “corrects it” – as described by the author — by “identifying and “artificially” removing “the decline.”

But oddly enough – the series doesn’t begin its “decline adjustment” in 1960 — the supposed year of the enigmatic “divergence.” In fact, all data between 1930 and 1994 are subject to “correction.”

And such games are by no means unique to the folder attributed to Michael Mann.

A Clear and Present Rearranger

In 2 other programs, briffa_Sep98_d.pro and briffa_Sep98_e.pro, the “correction” is bolder by far. The programmer (Keith Briffa?) entitled the “adjustment” routine “Apply a VERY ARTIFICAL correction for decline!!” And he/she wasn’t kidding. Now, IDL is not a native language of mine, but its syntax is similar enough to others I’m familiar with, so please bear with me while I get a tad techie on you.

Here’s the “fudge factor” (notice the brash SOB actually called it that in his REM statement):

yrloc=[1400,findgen(19)*5.+1904]

valadj=[0.,0.,0.,0.,0.,-0.1,-0.25,-0.3,0.,-0.1,0.3,0.8,1.2,1.7,2.5,2.6,2.6,2.6,2.6,2.6]*0.75 ; fudge factor

These 2 lines of code establish a 20 element array (yrloc) comprised of the year 1400 (base year but not sure why needed here) and 19 years between 1904 and 1994 in half-decade increments. Then the corresponding “fudge factor” (from the valadj matrix) is applied to each interval. As you can see, not only are temperatures biased to the upside later in the century (though certainly prior to 1960) but a few mid-century intervals are being biased slightly lower. That, coupled with the post-1930 restatement we encountered earlier, would imply that in addition to an embarrassing false decline experienced with their MXD after 1960 (or earlier), CRU’s “divergence problem” also includes a minor false incline after 1930.

And the former apparently wasn’t a particularly well-guarded secret, although the actual adjustment period remained buried beneath the surface.

Plotting programs such as data4alps.pro print this reminder to the user prior to rendering the chart:

“IMPORTANT NOTE: The data after 1960 should not be used. The tree-ring density records tend to show a decline after 1960 relative to the summer temperature in many high-latitude locations. In this data set this ‘decline’ has been artificially removed in an ad-hoc way, and this means that data after 1960 no longer represent tree-ring density variations, but have been modified to look more like the observed temperatures.”

Others, such as mxdgrid2ascii.pro, issue this warning:

“NOTE: recent decline in tree-ring density has been ARTIFICIALLY REMOVED to facilitate calibration. THEREFORE, post-1960 values will be much closer to observed temperatures then (sic) they should be which will incorrectly imply the reconstruction is more skilful than it actually is. See Osborn et al. (2004).’

Care to offer another explanation, Dr. Jones?

Gotcha

Clamoring alarmists can and will spin this until they’re dizzy. The ever-clueless mainstream media can and will ignore this until it’s forced upon them as front-page news, and then most will join the alarmists on the denial merry-go-round.

But here’s what’s undeniable: If a divergence exists between measured temperatures and those derived from dendrochronological data after (circa) 1960 then discarding only the post-1960 figures is disingenuous to say the least. The very existence of a divergence betrays a potential serious flaw in the process by which temperatures are reconstructed from tree-ring density. If it’s bogus beyond a set threshold, then any honest men of science would instinctively question its integrity prior to that boundary. And only the lowliest would apply a hack in order to produce a desired result.

And to do so without declaring as such in a footnote on every chart in every report in every study in every book in every classroom on every website that such a corrupt process is relied upon is not just a crime against science, it’s a crime against mankind.

Indeed, miners of the CRU folder have unearthed dozens of email threads and supporting documents revealing much to loathe about this cadre of hucksters and their vile intentions. This veritable goldmine has given us tales ranging from evidence destruction to spitting on the Freedom of Information Act on both sides of the Atlantic. But the now irrefutable evidence that alarmists have indeed been cooking the data for at least a decade may just be the most important strike in human history.

Advocates of the global governance/financial redistribution sought by the United Nations at Copenhagen in two weeks and the expanded domestic governance/financial redistribution sought by Liberal politicians both substantiate their drastic proposals with the pending climate emergency predicted in the reports of the Intergovernmental Panel on Climate Change (IPCC). Kyoto, Waxman-Markey, Kerry-Boxer, EPA regulation of the very substances of life – all bad policy concepts enabled solely by IPCC reports. And the IPCC, in turn, bases those reports largely on the data and charts provided by the research scientists at CRU – largely from tree ring data — who just happen to be editors and lead authors of that same U.N. panel.

Bottom line: CRU’s evidence is now irrevocably tainted. As such — all assumptions based on that evidence must now be reevaluated and readjudicated. And all policy based on those counterfeit assumptions must also be re-examined.

Gotcha. We’ve known they’ve been lying all along, and now we can prove it. It’s time to bring sanity back to this debate.

It’s time for the First IPCC Reassessment Report.

Real Climate.Org is chief defender of ”consensus” climatology on the Internet. One of its enduring missions has been to defend the dubious, indeed discredited “Hockey Stick” reconstruction of Northern hemisphere temperature history. The Hockey Stick was the basis for the IPCC’s claim in its 2001 report that the 1990s were the warmest decade and 1998 the warmest year of the past millennium.

That Real Climate (RC) should feel special solicitude for the Hockey Stick is no accident, comrade. Two of the five principals at RC — Michael Mann and Raymond Bradley — were among the three researchers (Mann, Bradley, and Malcolm Hughes) who authored the Hockey Stick.

All of the RC principals (Gavin Schmidt, Caspar Ammann, Rasmus Benestad, Mann, and Bradley) are frequent senders and recipients of the thousands of emails and other documents, now posted on many Web sites, that were hacked or leaked last week from the University of East Anglia’s Climate Research Unit (CRU).

The Wall Street Journal today published a selection of the leaked emails and an editorial concluding that the emails ”give every appearance of testifying to concerted and coordinated efforts by leading climatologists to fit the data to their conclusions while attempting to silence and discredit their critics.”

Even eco-radical George Monbiot says he is “dismayed and deeply shaken” by the emails, because, “There appears to be evidence here of attempts to prevent scientific data from being released, and even to destroy material that was subject to a freedom of information request.”

So far, the only email on which RC has seen fit to comment is one from CRU director Phil Jones dated Nov. 16, 1999, 13:31. It’s gotten a lot of buzz on the Internet, because it appears to advocate the use of a “trick” to “hide” a “decline” in global temperatures.

In a post titled “The CRU Hack” (November 20), RC writes:

No doubt, instances of cherry-picked and poorly-worded “gotcha” phrases will be pulled out of context. One example is worth mentioning quickly. Phil Jones in discussing the presentation of temperature reconstructions stated that “I’ve just completed Mike’s Nature trick of adding in the real temps to each series for the last 20 years (ie from 1981 onwards) and from 1961 for Keith’s to hide the decline.” The paper in question is the Mann, Bradley and Hughes (1998) Nature paper on the original multiproxy temperature reconstruction, and the ‘trick’ is just to plot the instrumental records along with reconstruction so that the context of the recent warming is clear. Scientists often use the term “trick” to refer to a “a good way to deal with a problem”, rather than something that is “secret”, and so there is nothing problematic in this at all. As for the ‘decline’, it is well known that Keith Briffa’s maximum latewood tree ring density proxy diverges from the temperature records after 1960 (this is more commonly known as the “divergence problem”–see e.g. the recent discussion in this paper) and has been discussed in the literature since Briffa et al in Nature in 1998 (Nature, 391, 678-682). Those authors have always recommend not using the post 1960 part of their reconstruction, and so while ‘hiding’ is probably a poor choice of words (since it is ‘hidden’ in plain sight), not using the data in the plot is completely appropriate, as is further research to understand why this happens.

So a “trick” is just scientific shorthand for a “good way to deal with a problem,” not something “secret.” But RC ducks the real issue. Is the ”trick” Phil Jones learned from Hockey Stick author Michael Mann a form of trickery? Does it create a false impression, as an illusionist does on stage, right out in the open, in front of an audience?

The trick, according to RC, is to splice onto the end of a temperature reconstruction, built on proxy data going back several centuries, the data from instrumental records starting in 1960 and 1981.

Now this is quite a trick, because it involves comparing apples (proxy data) to oranges (instrumental data) and pretending that the composite forms a continuous record.

As the Center for the Study of Carbon Dioxide and Climate Change observed years ago, researchers attempting to construct long-term (centuries to millennia) temperature records should ”finish the dance” with the (proxy) data they started with.

Grafting instrumental data onto proxy data to produce a seemingly continuous record is trickery, because instrumental data, unlike proxy data, are massively influenced by land-use changes and site-specific quality control issues.

Urban heat islands and irrigated agriculture can inject false warming biases into instrumental data that are absent from proxy data taken from remote forests or sediment cores at the bottom of lakes, for example. Improper placement of temperature sensing equipment near local heat sources (e.g. air conditioning vents, asphalt parking lots, waste water treatment plants) also generates significant false warming signals, as retired meteorologist Anthony Watts documents in gory detail.

So RC’s “nothing to see here” argument based on the alleged insider meaning of “trick” raises rather than allays suspicion that CRU is attempting to fit data to a predetermined conclusion.

Note also that RC says nothing about Phil Jones’s advice to backdate correspondence (Sept. 12, 2007, 11:30 a.m.), to delete emails related to the 2007 IPCC report (May 29, 2008, 11:04), and to evade FOIA requests, if necessary by deleting files (Feb. 2, 2005, 9:41 a.m.). RC also says nothing about Mann’s call to delegitimize the Journal of Climate for publishing papers critical of his work (March 11, 2003, 8:14).

The Wall Street Journal editorial’s concluding comment is spot on:  ”In the department of inconvenient truths, this one surely deserves a closer look by the media, the U.S. Congress and other investigative bodies.”

“Climatologists baffled by Global Warming Time-Out” declares the headline in Germany’s Spiegel online. “Global warming appears to have stalled. Climatologists are puzzled as to why average global temperatures have stopped rising over the last 10 years. Some attribute the trend to a lack of sunspots, while others explain it through ocean currents.”

Whatever the cause, it remains that there’s been no time out in the production of so-called “greenhouse gases.” More are being pumped out worldwide and the atmospheric concentration is higher than ever. Yet even while “The planet’s temperature curve rose sharply for almost 30 years, as global temperatures increased by an average of 0.7 degrees Celsius (1.25 degrees Fahrenheit) from the 1970s to the late 1990s,” it’s gone nowhere since then.

It what may a poor choice of words, Jochem Marotzke, director of the Max Planck Institute for Meteorology in Hamburg, told Spiegel, “”It cannot be denied that this is one of the hottest issues in the scientific community.”

Thomas L. Friedman’s op-ed in the NYT today could have been written by Paul Krugman.  And that’s not a compliment.

Friedman, like Krugman, waxes hysterical about those who are opposing  the cap-and-trade energy bill – those “deniers.” And, also like Krugman, he sets up those opponents as straw men that he can readily knock down.  In today’s article, Friedman worries about U.S. dependence on foreign oil supplied by  ”petro-dictators” and he fears ever-rising prices for increasingly scarce fossil fuels.

So either the opponents of a serious energy/climate bill with a price on carbon don’t care about our being addicted to oil and dependent on petro-dictators forever or they really believe that we will not be adding 2.5 billion more people who want to live like us, so the price of oil won’t go up very far and, therefore, we shouldn’t raise taxes to stimulate clean, renewable alternatives and energy efficiency.

Friedman’s terror about world population growth, especially growth in developing countries, is Malthusian.  (See Julian Simon on population and natural resources in “The Ultimate Resource II.”) . And Friedman  doesn’t seem to want those people to use energy to improve their standard of living.  Here’s what he says about that dream for a better life:

The world keeps getting flatter – more and more people can now see how we live, aspire to our lifestyle and even take our jobs so they can live how we live. So not only are we adding 2.5 billion people by 2050, but many more will live like “Americans” – with American-size homes, American-size cars, eating American-size Big Macs.

Such horror one can’t imagine for a person living at a subsistence level in India or China.

In his article, Friedman says that “clean energy” is the answer to the world’s energy problems.  He embraces  “E.T.” (no, not that visitor from another planet), but “energy technology”  that is carbon-less and efficient.

And we believe the best way to launch E.T. is to set a fixed, long-term price on carbon – combine it with the Obama team’s impressive stimulus for green-tech – and then let the free market and innovation do the rest.

His solution then is to tax conventional energy and subsidize alternative energy sources. Right.  That’s clearly an innovative solution that nobody has thought of.  And how would this affect the population bomb he fears?  Undoubtedly, raising the price of fossil fuels could indeed have an effect on developing countries’ populations.  While waiting for those alternative energy sources to develop, they’ll  continue to face poverty and resultant devastating diseases.  Not surprisingly, Friedman doesn’t address that problem.

Willie Soon and David Legates, both respected members of the American Geophysical Union, tell the story of how their planned session to discuss scientific papers that consider the many contributing factors to climate variability was a “go,” until suddenly it wasn’t:

We developed this session to honor the great tradition of science and scientific inquiry, as exemplified by Galileo when, 400 years ago this year, he first pointed his telescope at the Earth’s moon and at the moons of Jupiter, analyzed his findings, and subsequently challenged the orthodoxy of a geocentric universe. Our proposed session was accepted by the AGU.

In response to its acceptance, we were joined by a highly distinguished list of scientists – which included members of the National Academy of Sciences of the USA, France and China, as well as recipients of the AGU’s William Bowie, Charles Whitten and James MacElwane medals. Our participants faithfully submitted abstracts for the session.

But by late September, several puzzling events left us wondering whether the AGU truly serves science and environmental scientists – or simply reflects, protects and advances the political agendas of those who espouse belief in manmade CO2-induced catastrophic global warming.

Could this AGU position have anything to do with it?

The scientific consensus on climate change was expressed in an open letter sent to the US Senate on last Wednesday, 21 October….

While the signatories represent a wide variety of scientific disciplines, they all came together to express their concern over anthropogenic climate change. The letter states: “Observations throughout the world make it clear that climate change is occurring, and rigorous scientific research demonstrates that the greenhouse gases emitted by human activities are the primary driver. These conclusions are based on multiple independent lines of evidence, and contrary assertions are inconsistent with an objective assessment of the vast body of peer-reviewed science.”

What about the independent lines of evidence of no global warming the last ten years, which the vast body could not see below their extended gut?