Merging Duplicate Station Records

Haven't got much time to write this. I've gone for something simple wrt merging duplicate records. I think this is pretty much Tamino's original method.

I Take two duplicates, eg (I have shifted one of them up 10 or so degrees to make it easier to see):



I subtract one duplicate from the other. That produces a list of monthly differences:


I then get the average of those differences to use as an offset. I use that offset to adjust one of the duplicates to meet the other. I then average the two duplicates to produce a single record (in blue):



When there are more than two duplicates I continue the process, taking the merged result from the first step and trying to merge a third duplicate with that. During merging if a record contains less than 30 months overlap I abandon the merge and move onto trying to merge the next duplicate record.

I didn't want to lose information by abandoning merges, afterall even though duplicate 0 and 1 might not overlap, they might both overlap well with duplicate 2. So I after the first step above, I go back afterwards to try and merge in any records that failed to be merged in the first time. For example the process to merge 5 records could go like this:

Merge 0+1 pass
Merge (0+1)+2 pass
Merge (0+1+2)+3 fail
Merge (0+1+2)+4 fail
Merge (0+1+2)+5 pass

If I left it at that I would lose both duplicates 3 and 4. But the second go around could now find that duplicate 3 matches:

Merge (0+1+2+5)+3 pass
Merge (0+1+2+5+3)+4 fail
result= (0+1+2+5+3), duplicate 4 discarded.

As a result I only lose one duplicate. I go round and round until no more merges succeed.

Todo

I've only tested this on a handful of stations and by eye it looks okay, but I want to write some validation tests to spot any weird things happening. As it currently stands I calculate an offset when merging two duplicates, but I don't check that the difference between them is roughly a constant offset to begin with. It should be, but it might be possible that a station has two duplicates that totally contradict each other. It would be nice to at least raise a warning for weird stuff like that and review it.

It should be possible to automatically detect a number of other weird things too. For example here's a blog finding a possible problem with a raw GHCN record:
http://kenskingdom.wordpress.com/2010/02/23/cherries-anyone-another-data-“trick”-in-australia/

I haven't looked at it much but it looks like the GHCN raw record duplicate 0 for this station incorrectly contains the tail end of a record that should really be a duplicate in it's own right. As a result the early part of duplicate 0 is offset downward from the latter part - which is separated by a gap.

Here it is in GISTEMP:
http://data.giss.nasa.gov/cgi-bin/gistemp/gistemp_station.py?id=501943800000&data_set=0&num_neighbors=1

If this is an error then my merge method would end up merging the records incorrectly (or rather it wouldn't spot the problem and correct for it). It should be possible to at least identify such a large gap and that the record preceding the gap has a very different average to the record just after it. So that could be flagged as "possible problem".

In this way I could either review the possible problems or run an analysis "with possible problems" and without to see what difference it makes.

Willis Eschenbach Deconstructed

See here for what Willis Eschenbach has to say

Willis Eschenbach gets mad at Judith Curry. I present some extracts, out of order. But they are in order of relevant points.

Willis writes:
"And you wonder why we don’t trust you? Here’s a clue. Because a whole bunch of you are guilty of egregious and repeated scientific malfeasance, and the rest of you are complicit in the crime by your silence. Your response is to stick your fingers in your ears and cover your eyes."

Willis Eschenbach also argues:
"The problem is that 71.3% of what passes as peer reviewed climate science is simply junk science, as false as the percentage cited in this sentence. The lack of trust is not a problem of perception or communication. It is a problem of lack of substance. Results are routinely exaggerated. “Scientific papers” are larded with “may” and “might” and “could possibly”. Advocacy is a common thread in climate science papers. Codes are routinely concealed, data is not archived. A concerted effort is made to marginalize and censor opposing views."

Evidentially Willis is wrong. The problem is not a lack of substance. This is clear by looking at an area of climate science which is pretty much rock solid in substance. The fact of recent co2 rise and it's human cause for example. The evidence in that case is overwhelming, the science is settled. So by Willis's logic there should be no "problem" there right?

Yet we have had the Great Global Warming Swindle and Ian Plimer telling people that volcanoes emit more co2 than man. We have Ernst Beck accusing scientists of fraud when it comes to the instrumental co2 record. We have Jaworkowsi doing what amounts to the same to ice core researchers. And we have all those denier blog posts that have the side effect of making readers lose trust in the fact co2 rise is human caused.

So this is not an issue of substance as Willis claims. Otherwise the deniers would not be spamming the entire field of climate science, including very solid areas, with their lies and distortions and accusations of fraud.

As far as I am aware, the likes of Willis Eschenbach do not actively go out of their way to correct those who make absurd co2 claims.

So to Willis I say:

And you wonder why we don’t trust you? Here’s a clue. Because a whole bunch of you are guilty of egregious and repeated scientific malfeasance, and the rest of you are complicit in the crime by your silence. Your response is to stick your fingers in your ears and cover your eyes.

and with some rewording I get:

"The problem is that 71.3% of what passes as denialist talking points is simply junk, as false as the percentage cited in this sentence. The reason we call them deniers is not a problem of perception or communication. It is a problem of their lack of substance. Minor problems and errors are routinely exaggerated in denialist blog posts. “Scientific papers” are misinterpreted, or spun so that caveats such as “may” and “might” and “could possibly” are omitted. Political advocacy is a common thread in denialist blog posts. Context is routinely concealed, the big picture is routinely ignored. A concerted effort is made to marginalize and discredit science."

I was amused to see that Willis also made a reference to "policing your own backyard" too. It's utter hypocrisy:

The solution is for you to stop trying to pass off garbage as science. The solution is for you establishment climate scientists to police your own back yard.

As I have mentioned before, they don't get a free pass because they are "just blogs".

Willis also makes two statements that leaves scientists in a lose-lose situation:

"We’re not interested in scientists who don’t mention their doubts."

but also
"“Scientific papers” are larded with “may” and “might” and “could possibly”"

That's right. No sooner as scientists mention their doubts which Willis demands of them, Willis is ready to jump on them for using caveats. Uncertainty cuts both ways, it can make known risks uncertain, but can also raise the specter of risk in the unknown. Willis seems to think only one of these is valid.

Judith isn't necessarily wasting her time, but I doubt these people will be satisfied by the truth, which is that manmade global warming is a fact despite climategate and errors in the IPCC report. They want the whole thing discredited and so muddying science is their goal.

WTF

http://legis.state.sd.us/sessions/2010/Bills/HCR1009P.htm

Gridding

A customizable sized grid can now be drawn over the world map and the grid cells can be colored or contain a text item. Below I have plotted a grid of 30x30 cells. The number inside each is the number of stations in that grid cell. The stations are also plotted as blue dots. The image is quite blurry due to the jpeg compression.



In the image below a grid of 10x10 cells is drawn with cells containing more than 10 stations highlighted.



Finally 30x30 cells again with the proportion of area of the globe each cell contains:

How Skeptics Distorted the CRU Emails In the Name of "Climategate" (Part 2)

I continue my critical analysis of the SPPI "Climategate Analysis" report, email by email. I will try to stick to the following formatting to make things clear:
Parts in this color and in bold are the reports own words. Parts in this color and in italics are the report's quotes from the CRU emails. All other parts are my own analysis.

email 0843161829

The report starts:

Two days after the previous exchange, Gary Funkhouser reports on his attempts to obtain anything from the data that could be used to sell the message of climate change:

I really wish I could be more positive about the ... material, but I swear I pulled every trick out of my sleeve trying to milk something out of that. ... I don’t think it’d be productive to try and juggle the chronology statistics any more than I already have—they just are what they are ... I think I’ll have to look for an option where I can let this little story go as it is.

Now where in that text is there anything about "selling the message of climate change"? That interpretation cannot have been drawn from the email, which simply doesn't support it. So where does it come from? Where is the hidden place that the Analysis frequently dips into to draw these wild accusations?

What are Briffa and Funkhouser even discussing here? The self-styled "Climategate Analysis" doesn't say. That's right the skeptics can't even be bothered to figure out what aspect of the science is being discussed. In fact the report goes as far as to omit parts of it with ellipses as if it's all just getting in the way of the allegations they want to make.

Here's a thought - what if Gary Funkhouser is not complaining that he can't obtain data to sell the message of climate change, but is actually complaining that he can't get data to show anything? Ie that the data is too noisy to yield a signal? Wouldn't that be consistent with the email? Why yes it would. In fact one of the ellipsed out sentences "The data's tempting but there's too much variation even within stands." suggests that no?

Me thinks you "skeptics" have the idea of email analysis backwards! You are interpreting the emails to fit your conclusion. But what you should be doing is interpreting the emails to determine the conclusion. In other words the "hidden place" mentioned earlier which you are drawing to interpret individual emails is in fact your conclusion.

So I see a vicious cycle, a kind of snake eating it's own tail. Your conclusion comes from your interpretation, but at the same time your interpretation is drawn from your conclusion. So which came first, the chicken or the egg? I suspect the paradox is resolved by observing the baggage skeptics arrived with before they began the analysis. They wanted climategate so badly that they formed their conclusion before looking at the emails, which due to their bad analysis technique tainted their interpretation of the emails, which then looped back to reinforce their conclusion. There is surely not enough ridicule in the whole world that would be sufficient to heap upon these "skeptics".

The report continues:

His reluctance to report a “null result” (namely, that the data do not show anything significant) is extremely disturbing as it flies in the face of standard scientific practice, which requires that all results be reported.

Bullshit. If you can't extract a signal from some noisy data and therefore you have nothing to use for any analysis, you don't go and publish a paper saying you couldn't do any analysis because the data was too noisy. Where are these "we didn't do shit" papers published? In the International Journal of Having a Laugh?

The fundamental problem is that any censoring of results that do not lead to a predetermined conclusion will always—by design—bias the corpus of reported results towards that conclusion

Wow that describes the SPPI report. Censoring of the straight-forward interpretations of emails that do not lead to the predetermined conclusion, "bias the corpus of reported results towards that conclusion". However in case of this email, there is no hint of a predetermined conclusion being discussed. Indeed the email seems to be saying the problem is that no conclusion can be drawn.

in the same way that a gambler who always brags about his wins (but stays silent about his losses) will appear to be hugely successful, even if his losses have, in reality, far outweighed his winnings (as is generally the case, in the long run, except for the extremely skillful).

They were clever to add the "except for the extremely skillful" caveat at the end, if they hadn't done it someone might have criticized them over such an error!

We will, sadly, see that this fundamental scientific flaw—which, in and of itself, is sufficient to render the evidence for climate change completely unreliable and scientifically worthless—is one that runs throughout the entire Climategate saga.

They make the word "sadly" sound so...insincere. It's like they are trying to play on my emotions...but utterly failing. The hyperbole here based on the flawed analysis pointed out above needs no comment. We will, sadly, see a lot more of this in the rest of the analysis/saga I am sure.

Note, also, the immense power wielded—albeit ever so subtly—by Briffa: he influenced the analysis that Funkhouser performed, simply by telling him that the results would need to politically “saleable”. Scientists are not naive: they know that securing funding, publication of their papers, and interest from other institutions are the key factors determining their future.

This is talking about the last email I discussed and it showed no such thing. This is a new phenomenon we have discovered though - one email's misinterpretation being relied on as a crutch to misinterpret another. I imagine that quite a lot of the analysis works like this, a rotten construction.

Reforming the IPCC

My idea for reforming the IPCC is to set up a funded group whose sole aim is to defend science by addressing the output of the deniosphere, whether on blogs or in the media.

Given that deniers claim their smears and shit analyses are simply "questions" and that they just want good science to prevail, they can hardly complain about the construction of such a group.

How Skeptics Distorted the CRU Emails In the Name of "Climategate" (Part 1)

Global warming skeptics have published and propagated an analysis of the CRU emails which is shockingly biased.
http://scienceandpublicpolicy.org/reprint/climategate_analysis.html

This report was is based on an online analysis that was spotted and packaged up by the Science Public Policy Institute (SPPI), a global warming skeptic organization. It was then propagated uncritically through various skeptic blogs and political think-tanks of the world.

It is very revealing given how poor the analysis of the emails is, that it was accepted uncritically even by the bigger skeptic names. This makes it a good read for fence-sitters who might appreciate an insight into how global warming skeptics are prone to accepting, and indeed spreading, any garbage that supports their position. I therefore recommend that fence-sitters are referred to the report to "decide for themselves" whether the interpretations reflect the emails fairly.

I will continue my critical analysis of this report, email by email. I will try to stick to the following formatting to make things clear:
Parts in this color and in bold are the reports own words. Parts in this color and in italics are quotes from the CRU emails. All other parts are my own analysis.

email 0842992948

The report describes this email as such:

"We now turn to Keith Briffa, one of the more curious University of East Anglia characters in the Climategate saga. Gary Funkhouser of the University of Arizona writes to Briffa about some data that was collected in the late 1980s. Briffa makes it clear that he is only interested in the data if it can be used to “sell” the climate change message to the general public:"

To back this allegation up the report quotes this part of Briffa's email, which as you see does not back up their allegation at all:

Briffa: "The data is of course interesting but I would have to see it and the board would want the larger implications of the statistics clearly phrased in general and widely understandable (by the ignorant masses) terms before they would consider it not too specialised."

My interpretation of the email, which I think reflects what any unbiased person would see, is that there is a report for public consumption being produced. Briffa is stating that whatever goes into that report cannot be too technical. If it's aimed at the public it must be understandable by the public. It must be summarized and what goes in there must be relevant. This is all standard stuff that any technical group will do if producing a report for a lay audience.

The skeptic interpretation is nonsense. There's no hint that what is being discussed can be used to “sell” the climate change message to the public, let alone that Briffa is advocating any such bias. Not only have the skeptics interpreted this in a way that is unsupported by the text, but they have the gall to claim their interpretation is "clear" from what Briffa says!

If the skeptics have a genuine case, why must they resort to misinterpreting emails such as this? If they aren't just out to smear scientists and science, why do they seem they so willing to do so in the case of this email? I assure you this isn't an isolated example. The SPPI report is full of these kind of misinterpretations and I plan to continue demonstrating this email by email.

Accountablity

The problem with the current "debate", which I have seen others mention elsewhere (I can't remember where or I would link), is the asymmetry.

Scientific research labs and institutions are held accountable for what they publish and the media will actually criticize them for any mistakes. They simply couldn't get away producing and publishing anything near as low quality as the SPPI analysis of the CRU emails. Even the investigative team that is reviewing the CRU emails is under scrutiny.

In contrast, none of the usual denialist outlets have any accountability. They can't be discredited even if they deserve it. Noone writes headlines when Watt's screws up. Same with the Heartland Institute and SPPI. Noone writes headlines when an analysis like the one the SPPI published gets spread like wildfire. The deniosphere have a lack of oversight. Anyone can say what they want of course - even really ridiculous things - but they should be justly held to account and widely discredited if they do say ridiculous things or behave badly (it's overdue).

It's ironic of course that a lot of denialist's appeal to immense levels of accountability from the likes of the IPCC and the CRU, without demanding any of themselves.

Therein lies a bit of a skeptic delusion about the difference, I am sure you've heard a variant. It's that scientific organizations, etc should be held accountable because they are funded by tax payers.

This is wrong of course. Scientific organizations should be held accountable even if they are privately funded and that goes for non-scientific organizations too. If you mislead people, intentionally or not, you should be held accountable. Which in many cases can just mean some bad press. Being discredited is the risk you run and so hopefully stops people running it. That's the checking role the media and journalists play and any publishing medium that can possibly influence decisions that affect many people, should be held accountable. Yes even blogs. Even think-tanks. When they get to a certain level of influence it's the job of the media to hold them accountable.

But lets go back to the talking point, and for the moment drop the fundamental error in it's premise about taxpayer funding determining accountablity, because there is a further error in the argument even when we accept it's premise. If you are misleading people over aspects of science and that science has been taxpayer funded, then by extension you are wasting people's tax money. Therego by the same argument you should be held just as accountable for falsely discrediting the science and wasting taxpayers money.

A little sarcasm there - but there is also a point. We can't have a world where a denialosphere and psuedo-journalists can run rampant tearing down the public perception of good science to the extent that they try - and get away with it just because they "aren't funded by tax payers".

Critically the deniers don't get much bad press, even despite all the gaffes and libelous smears they make. In part it is because they aren't as visible and influencial as we might imagine. They don't afterall get much good press either if you think about it.

Of course recently they've been getting more mentions in the mainstream papers. And here a pendulum could have already been swung. The deniers should hope it hasn't been swung too far in their favor, because the back-swing has a journalist investigating a newly discovered "wattsupwiththat" blog and it's ties to certain think-tanks. There is a story there, but it's only newsworthy if the "skeptics" and their message gets so loud that the public would be interested to read it.

This happened recently in the case of the misquote of John Houghton, former IPCC head. The deniers had been relaying this quote for so long, and Monckton recently so loudly, that the a leading UK newspaper took it to print. In part it was thanks to John Houghton taking it to task, but if not for the loudness of skeptics recently I don't think it would have been newsworthy.


Now accountability might not occur. The deniers might not ever get relevant enough to get sufficient media attention for it to happen. Especially given that time is running out for them - as much as they will deny it they have gambled on warming not continuing and so when it does they will find their foundations crumbling.

Example of Pseudogate

Someone called John P. Costella has picked out the "key" CRU emails and analyzed them:
http://assassinationscience.com/climategate/

John, your analysis of the emails is appalling. Yet I believe you have created a work of art. It's like a painting of what's going on inside "skeptic" minds when they imagine Climategate.

I will take one of the emails you analyzed and show how you are trying too hard. You take a perfectly innocent email and extract the most amazing conclusion out of it.

No wonder skeptics have made it clear they plan to refuse any investigation result they don't like as a "whitewash". It's because perhaps you guys know your entire case is an illusion bourne of wishful misinterpretation of bog standard emails.

email 0837094033

First here is my interpretation of the email.

This is an email exchange between Phil Jone's and a US colleague. Phil Jones discusses a scientific issue and then sticks a big P.S at the end in which he goes into non-work related aside. It's a kind of "hey in other news you know you have those skeptics? We've got one too.". He ends with a joke. I think this kind of email is run of the mill communication between US and UK colleagues. Work stuff, followed by some trivia of "whats happening in the UK right now".

Now lets look at your interpretation...

"Note that Jones is immediately reporting the existence of this first British skeptic to climate scientists on the other side of the Atlantic, taking special note of the “air time” (exposure on television or radio) that the skeptic is apparently receiving.

This is nothing but a cheap spy story woven around a perfectly run of the mill email. You shouldn't be adding such a story to the email as part of an analysis. The email does not warrant your total lack of regard for my far more ordinary and believable interpretation. Occam's Razor my friend.

You continue:
"Already, we can start to appreciate that the politics and “spin doctoring” in this field outweighs the scientific issues. Continuing from Jones’s email:"

We can't appreciate something that is only there if you presume it. Unless of course by "appreciate" you mean "justify our preconceived conclusions". How else could you have come to such a quick broad conclusion about an entire field based on what is consistent with a run of the mill email?

And just a minor point: Phil Jones started off the email with a scientific issue. That's the main content of the email. In fact many of the emails concern scientific issues, but those tend to be the emails that skeptics don't cite. Slagging off Piers Corbyn is an aside - marked under P.S.

You continue to weave a cheap story around the email:
Jones’s report is as efficient as that of an intelligence agent: the skeptic is dangerous because he is the British equivalent of a college professor—in the “hard sciences” of physics and astronomy, no less.

Now we have talk of "intelligence agents" and "dangerous" skeptics...do you see yourself as the next Dan Brown?

And you continue:
However, he softens his attitude to the skeptic slightly: "He’s not all bad as he doesn’t have much confidence in nuclear-power safety."
We here see clearly that Jones’s assessment of a scientist’s worth is influenced strongly by his assessment of their ideology—in scientific terms, nuclear power safety is completely unrelated to the science of climate change.

Your interpretation leaves a lot to be desired. You also missed the key context in Jone's next sentence:

"He's not all bad as he doesn't have much confidence in nuclear-power safety. Always says that at the begining of his interviews to show he's not all bad!"

Sounds like a joke to me, and the first part now sounds like sarcasm. Ie Phil Jones hasn't softened his attitude at all, but is jibing at Corbyn for raising his nuclear-power safety view at the start of his interviews as if it showed he wasn't all bad.

You conclude:
"This dangerous prejudice will prove to be one of the most persistent threads throughout the Climategate scandal."

Which speaks volumes of "Climategate" that you can reach such conclusions based on such an unnoteworthy email. Dare I say it's almost as if the "climate gate" scandal is perhaps the conclusion you want and your rubbish analysis is simply a means to achieve it?

I believe that at least 90% of Climategate is just the bullshit imagination of skeptics who exaggerate it for politics. Yes ironically they go mental when the IPCC report is wrong and has exaggerated something, but they are happy to exagerate most of the stuff in the CRU emails. In fact they demand the media report it.

Just in case none of this was clear I will resort to an analogy. I am not comparing Corbyn with a creationist. It's sad that I have to make that clear, but I do. I am just showing that the analysis technique is clearly Fail.

From: UK Biologist
To: US Biologist

Some science stuff here.

PS. Britain seems to have found it's Kent Hovind (US Creationist). Our population is only 25 % of yours so we only get 1 for every 4 you have. His name in case you should come across him is Bob Creationist. He is nowhere near as good as a couple of yours and he's an utter prat but he's getting a lot of air time at the moment. For his day job he teaches medicine at a University and he predicts flu outbreaks from solar phenomena. He bets on his predictions months ahead for what will happen in Britain. He now believes he knows all there is to know about the evolution issue. He's not all bad as he doesn't have much confidence in biblical literalism. Always says that at the begining of his interviews to show he's not all bad !

Now if I applied your analysis technique to this email - what would I find?

Already, we can start to appreciate that the politics and “spin doctoring” in this field outweighs the scientific issues.

Well there goes the entire field of biology.

I'll continue using your own analysis on the biology email to show how shoddy it is.

Note that this UK biologist is immediately reporting the existence of this first British evolution skeptic to biologists on the other side of the Atlantic, taking special note of the “air time” (exposure on television or radio) that the skeptic is apparently receiving.

The UK Biologist report is as efficient as that of an intelligence agent: the skeptic is dangerous because he is the British equivalent of a college professor—in the science of medicine, no less.

However, he softens his attitude to the skeptic slightly:

"He’s not all bad as he doesn’t have much confidence in biblical literalism."

We here see clearly that the UK Biologist's assessment of a scientist’s worth is influenced strongly by his assessment of their ideology — in scientific terms, biblical literalism is completely unrelated to the science of evolution. This dangerous prejudice will prove to be one of the most persistent threads throughout the Evolutiongate scandal."

Yeah it's Evolutiongate. Throw in a few more emails with such analysis, parrot it over the INTERNET, and you generate a false rumor. Then the idiots who fall for it will go all over news sites going "why aren't you reporting on the emails??? OMFG email 0837094033 shows us that "spin doctoring" is rampant in biological sciences.

Fabricated quote used to discredit climate scientist

Fabricated quote used to discredit climate scientist

The most eye opening thing is the comments by skeptics under the article.

There was nearly an Iron-Sun-Gate

On the same BBC blog I had raised the point that even Lindzen and Choi propose about 0.5C warming from doubling co2 and that as 0.5C represents more than half the warming of the 20th century then even by Lindzen's estimate co2 will probably be the primary driver of the temperature trend over the 21st century.

This was not to counter all skeptics as a good number of skeptics accept the above, Lindzen for one. It was to counter a certain fundamentalist kind of psuedoskepticsm that I was observing on the BBC blog and others that seeks to claim co2 has no effect on climate whatsoever. This fundamentalist group are typically the worse of conspiracy theorists too, for co2 having no effect on climate whatsoever is a necessary condition for their conspiracies that it is just an excuse for taxes or a new world order, etc. If there was a genuine basis for the focus on co2 having an important role in climate change these conspiracies would flounder.

Shortly after a skeptic told me that the "science is NOT settled". Science in general is not settled, I don't think there is a theory in science that is settled. I decided to rephrase "settled" so it was clear I was talking about the current state of the science. I replied that in my view the science had settled down on the matter of doubling co2 causing at least 0.5C warming - although never say never.

I also haphazardly threw a passing grenade at psuedoskepticism by pointing out that, in my opinion anyway, the psuedoskeptic movement have opened their minds so much that nothing can be considered be settled. Any idea becomes credible (ernst beck, g&t, even recently "it has not warmed"). As an example I mentioned a global warming skeptic paper that had been published on the internet had argued the core of the sun was made of iron.

This provoked a remarkable response which chastised me for being dishonest:

"And as far as I recall, I disabused you personally of this intellectually dishonest claim some time ago. If you cannot direct me to where a genuine sceptic said this, you should be ashamed of yourself, because it's clear you make things up for political effect

I demand that you direct us -- NOW."


I don't remember being disabused, although perhaps I had posted about the iron sun paper before and not read the replies. I guess this is a little like climategate. With insufficient evidence, conclusions are drawn (meanwhile another skeptic is repeatedly demanding EVIDENCE WHERE IS THE EVIDENCE? from me)

Anyway here we have a skeptic who considers that the publishing of an iron sun paper by a global warming skeptic would reduce the credibility of skeptics sufficiently that it would be a slur to claim such a thing if it wasn't true.

So there should be outrage that such a paper does indeed exist, let alone that, contrary to what I had thought, it was actually published in Energy and Environment not simply on the internet. If we took just 1% of the outrage of climategate and applied it to this travesty, what should we get? Surely this should become Energy-and-Environment-Iron-Sun-Gate should it not?

The paper is "Earth's Heat Source - The Sun" by Oliver K. Manuel

I am no scientist and my technical knowledge on stats, math, climate science, etc is rudimentary. I also know nothing about solar models. So I have no personal technical perspective on Oliver Manuel's "iron sun" idea whatsoever. It could be right. But I have noticed a lot of experts who think the idea is a little on the fringe. And I am willing to go with that.

Although really it matters not what I think, the skeptic in this case has made it clear that they find the idea of an iron sun to be preposterous. I should be ashamed with myself for even claiming a climate skeptic has published such a paper.

But if you thought this would damage the credibility of Energy and Environment in some way, let alone become a fully fledged "Gate", well no.

"I stand corrected -- there is someone crazy enough! I guess I should have known.

But is this a typical "sceptic paper on the web" that infinity just happened to be stumble upon as part of his open-minded research into the case for scepticism, or is it something he (and you) looked up on Google specifically to show that such a paper actually existed?"

So I am still on trial, not for an "intellectually dishonest claim" this time, but for possible cherrypicking from google. For some reason the paper that the skeptic thought was a slur on skeptics if it existed matters no more now that it does - even though it was not in some dusty part of the internet as I originally remembered, but actually published in Energy and Environment.

What now matters is whether I have cherrypicked the paper from a google search. I think that matters not, if I had cherrypicked something crazy - that should be all the more reason to expect a journal wouldn't have published it.

But it gets worse. How I found this paper was not by cherrypicking from a google search, but:

"I wouldn't have heard of the paper but for a certain list of "450 Peer-Reviewed Papers Supporting Skepticism of Man-Made Global Warming" that was paraded around the internet by skeptics a year or so ago. Featured an article on WUWT etc and was posted on various blogs, etc. If you google the exact phrase "450 Peer-Reviewed Papers Supporting Skepticism of" you get 97,500 hits.

I bothered to trawl through some of the list to see what the papers were like. So I didn't so much as look for this paper up on google as had shoved in my face at the time.

Notice also the presence of strange contradiction in the title of the list itself. Skeptics tell us that climate scientists act as "gatekeepers" stopping anything skeptical of manmade global warming being published. But here they are claiming nigh on 500 papers have been published. Yet again they want to have it both ways - both claiming being persecuted and supressed, but they also want to claim a large number of skeptical publications exist. As the old adage goes heads we win, tails you lose."

So now the entire deniosphere are implicated. Will there now be an Energy-And-Envrionment-Published-An-Iron-Sun-Gate? Will there be an investigation?

I sure don't care much, this isn't outrageous, it's just part of the usual psuedoskepticism. I wasn't the one uptight about reality though, I was just pointing it out.

posted comment to BBC Richard Black's blog

I shouldn't bother reading the comments on BBC blogs anymore, they are mostly filled with stuff that winds me up rather than anything interesting. I can almost guarantee that if I read anything in such places I will lose at least the next hour as I cannot go without ranting out a reply. For some reason I can't let absolute garbage accusations go unchallenged. If someone else challenges them for me my blood pressure goes down. If not...

Is it even constructive to reply though? Will I just harden the people I reply to? Possibly. What about fence sitters who read what I say? In fact I don't really care, it's just something I have to do. I feel better if I reply than if I don't. If there is such a thing as karma though this behavior cannot be good for it. Part of me thinks just to sell my computer, pretend the Internet doesn't exist and live a relaxed life in blissful ignorance of denialist nonsense.

Anyway below is a comment I just posted expressing utter exasperation.

"So little of what "skeptics" are arguing makes sense. I don't know where to begin. There are layers upon layers of utter rubbish being said.

I have to somehow put into words the errors and tangle of contradictions I am reading from skeptics. A whole bundle of stuff that doesn't make sense. Internal contradictions. How can I do this concisely? I'll just make a davblo style list I think. We'll see how it goes.

1) My overriding point here is that there is something rotten in the "skeptic" community. There are growling contradictions in the background and glaring mistakes on top.

2) What Joe says about (20-1) co2 is a common fallacy. But nevermind, lets just write off an entire field of science over our own misunderstanding of the carbon cycle.

3) It is claimed above that AGW is merely an extrapolation of data. This is outlandishly false. AGW is based on explanation of how the climate works, not data extrapolation.

4) The claim about AGW being a hypothesis of Callendar and Keeling is just historically false.

5) The claim that the IPCC have ignored 150 years of data is bizarre, slightly comical and at the same time slightly unnerving to imagine the poster might actually believe it.

6) I see comments by skeptics claiming that the early 20th century warming is an argument against AGW. Yet think about it - "skeptics" are keen to accuse the climate scientists that produce the temperature records of willingness to commit fraud to further AGW. Why then would such scientists leave an early 20th century warming in the temperature records? That's right, "skeptics" want to both claim scientists willingly modify the surface record to support AGW and that also they've put in something that falsifies in it. The skeptics want to have their cake and eat it too. It just doesn't make sense. I submit that the "skeptics" often don't even realize their own contradictions, for their position is born of little more than downright contrarianism that requires little thought than to simply oppose ideas and generate conspiracy theories.

7) The climate models that are widely known to show AGW also show much of the early 20th century warming due to natural causes. How then can "skeptics" even begin to claim that warming in the early 20th century contradicts AGW when the very models that show AGW also show early 20th century warming?

3) "Skeptics" are trying to pour doubt on the early 20th century temperature record. That's right - they are trying to claim the early 20th century temperature record is inaccurate, but at the same time they want to use it for an argument! It's sheer ridiculous. Yet stranger-than-fiction - it's happening all over the internet.

The problem here is that "skeptics" only engage one single simple concept at a time and they seemingly lose track of the big picture, of how the different concepts tie together and how things make sense overall.

This is what science does - it makes sense of the world by trying to explain it. This is not what the skeptics are doing, they are playing a contrarian game wherein they try to knock down single concepts, one at a time. That's why they so often knock down concepts they later try to rely on to knock down others. Internal inconsistency.

If "skeptics" are making this many mistakes can we really trust they have their conclusions in order? How about instead of doubting the science because some skeptics make wild claims (20-1 co2! extrapolations!), we start doubting the "skeptics" have anything useful to bring to the subject?

I went back in hindsight and put the word skeptics in double quotes. I may have missed some."

http://www.bbc.co.uk/blogs/thereporters/richardblack/2010/02/rejigging_the_climate_panel.html#P92377321

Smoky Fools

The Anecdote to the Smoking Gun at Darwin Zero

Smoky Falls, Canada



A station with a temperature record spanning from November 1933 to December 1989 with a few years and months of data missing in between.

The raw data shows a warming trend of 0.38C/decade. But not content with sticking to the raw data, those bastard climate scientists came along and "adjusted" it in their figure fiddling ways.

They adjusted the data from 0.38C/decade warming to 0.32C/decade cooling.

Oh wait.



The adjustments are shown in blue. There are two large downward adjustments made. Both miss the large gap in 1963-1964.

Here's the two downward adjustments up close:



I can't understand why these adjustments have been made, so obviously this must be evidence that global warming skeptics have hacked into the science databases and fraudulently altered the data to hide the warming.

They've even altered the NASA GISTEMP records of this station, adjusting past temperatures higher to turn strong warming into cooling. James Hansen is going to go mental when he hears about this:





Note that the piddling 1.9C/century adjustment made at Darwin Zero is small fry compared to the massive 7C/century adjustment made here. If that matters.

Now that we have evidence "skeptics" have been manipulating the temperatures downward we should conclude that warming over the 20th century is probably larger than records indicate. Also send plenty of letters to your political representatives asking them to hold an inquiry into these spurious cooling adjustments. This is how Blog Science works isn't it?

Gatwick Zero

Gatwick Airport, London.

There are two raw duplicates. Duplicate zero spans 1961 to 1992 and duplicate one from 1987 to mid 1998, so there is some overlap. Both records show a warming trend:



There is one adjusted record from 1961 to 1992:



Compare the duplicate 0 raw and duplicate 0 adjusted data:



The adjustment has halved the warming trend over the period 1962 to 1991. From 0.38C/decade to 0.19C/decade.

The difference between the adjusted and raw record shows the pattern of the adjustments made:



I don't know why these adjustments have been made, but this doesn't mean scientists must be fabricating the data to hide global warming. Gatwick was just something local I was looking at. It might be interesting to try and find the largest and weirdest downward adjustment in the GHCN record just to give the conspiracy theorists something to think about.

I haven't been working a lot on this in the past week. Added some stuff, but still haven't done anything insightful wrt analysis. What I have added is:

-Added the ability to calculate and plot a line of best fit over an entire record, or a subset of it.
-Added the ability to query for a list of all stations within a specified distance of another station.
-Added the ability to zoom in on the map, so that very close stations don't appear as a single dot.
-Added automatic colorization and key to the graph

I used all this below to graph the first duplicate raw record of all stations within 100km of the Gatwick station as well as mapping the station locations. There are a number of graphical problems that make it unreadable, but this isn't too disimilar to the GISTEMP station selector if you pick numneighbours too high. I will just try to avoid plotting too much data at once in future.

Derivatives & Differences

Slow going in the last few days as I had little time to work on the GHCN data. All I have done is add the ability to calculate the difference between two datasets. Eg Darwin, Australia raw record duplicates 0 and 1:



And their difference:



Also the derivative of each record:

Interesting

Two imminently interesting events are due.

First will be the release of the UAH satellite anomaly for January 2010 which should happen within days. I don't know for sure, but I suspect it will show a very high anomaly and as a result there will be a little pandemonium among leaders of the denialosphere as they have to scuttle around trying to align their treasured UAH satellite record with the "ice age" January they've all been reporting.

Of course the cold January was regional, not global and the UAH satellite record is not in lockstep with the surface. That's the answer. But these complexities wash over the heads of typical commenters on psuedoskeptic blogs. I also suspect this will act as a block against certain stories appearing in the denialosphere when GISTEMP et al publish not particularly cool global January anomalies. They can hardly complain about GISTEMP when UAH shows a higher anomaly. Who wants to bet they will complain anyway?

The second interesting and more important event is the launch of Cryosat-2, which according to wikipedia is scheduled for February 25th, 3 weeks time. The first Cryosat, if you recall, failed on launch and crashed in 2005. Pity it would have captured the 2007 arctic sea ice minimum and all if it had made it. Cryosat-2 has taken this long to build as a replacement. Anyway fingers crossed.

Frequent reporting stations of past 30 years

Map of all stations that have reported at least 7 months of each year during the period 1980 to 2009 inclusive. A nice round number, 600 stations:



I also tested with the period 1980 to 2008 in case a lot of stations had simply not reported months yet for 2009, but I found that yielded 606 stations, just 6 more.

I did the same test for 2005 to 2009. There is a large gap in Africa and in Canada.



Defining years as having less than 7 months of data as invalid years, the below image shows stations with no invalid years in the period 1980 to 2009 in red. Stations with 1 invalid year in that period are shown in blue and stations with 2 invalid years are shown in orange.

GHCN Stations Still Reporting

The below map shows all GHCN stations reporting at least one raw monthly mean temperature past 1997 (ie Jan 1998 or later). 3154 stations in total.



3091 stations report past 2000.

2773 report past 2004.

Here are the stations that have reported data past 2008 (at least one month of 2009 reported). 1596 in total:



Here is a map showing all the above mentioned years.

GHCN Color by Number

Below is a not particularly useful graph showing the number of raw duplicates each GHCN station has using the color red. The color for each station is a gradient from the darkest dots with 1 raw temperature record to the brightest red dots with 10.



The color difference is so slight and my jpgs get distorted so much it's not a particularly viewable graph. Also the dots overlap each other, especially in the US. When that happens the brighter dots win as they are drawn over the top of the darker dots. If this mattered for anything other than curiosity I would perhaps plot separate maps for each level.

It does appear even from this bad graphic that some regions of the world have stations with a lot of duplicates while other regions have stations with far fewer duplicates. Not surprising I suppose.

The numbers I got for raw records are:

1 record = 4574 stations
2 records = 1109 stations
3 records = 601 stations
4 records = 502 stations
5 records = 271 stations
6 records = 114 stations
7 records = 57 stations
8 records = 39 stations
9 records = 11 stations
10 records = 2 stations (Irkutsk and Los Angeles for what it's worth)

It's not possible to have 11 duplicates in the file as the duplicates column is a single digit..

The numbers I got for adjusted records are:

0 records = 2509 stations (that's right, a lot of stations appear to only have raw data)
1 record = 3556 stations
2 records = 752 stations
3 records = 275 stations
4 records = 110 stations
5 records = 58 stations
6 records = 18 stations
7 records = 2 stations (Irkutsk again! and Sverdlovsk..wherever that is, I only know irkutsk from playing risk)

The graph this time is a bit more viewable. I plotted all stations with at least one adjusted record in blue and then over the top I plotted all stations with no adjusted records in red.



Today I compiled and store away station meta-data like number of raw and adjusted duplicates (which I used in this post) as well as the start and end dates of each record. I also have scope to add more such meta-data later. By "storing it away" I mean I have separated this info from the temperature data itself so I can load that information without loading all the actual temperature data. Will make queries on those things faster. Of course if I download an updated copy of the GHCN files I need to refresh the metadata files, but that's an infrequent task.

Next I need to add the ability to calculate linear trends over arbitrary periods. I also need to figure out how to handle duplicates in any analysis. Do I average them? Take a random duplicate? Exclude outliers? My thoughts are to try them all and see what difference it makes as I have no knowledge or basis to pick a particular strategy.

The Pits They Dwell In: The Register

The Register is one of those entertainment news site, targeted primarily at IT, although it does have a science section. For some unfathomable reason this science section has been plugged directly into the denialosphere and pushes out perhaps a dozen pseudoskeptic friendly articles a month under the category "Environment".

Infamous authors include Steve Goddard, who if you remember caused Cryosphere Today to put up a rebuttal to one of his posts about arctic ice trends as it misled so many people.

There must be some kind of psuedoskeptic edict from on high at The Register however, as there are a number of authors who publish such articles.

The most recent of which is the following by Andrew Orlowski who posts some emails he has received:
http://www.theregister.co.uk/2010/01/31/climategate_modeller_mailbag/

The first email he posts is a fair reflection:
"that useless half wit Orlowski has again been allowed to post his climate denial bullshit, without any comments allowed as a right of reply. I am sick of ElReg supporting fruit loop deniers and not allowing comment on their irrelevant rantings. Fuck you El Reg." Register reader Dave 14

Wasn't me. I wouldn't have bothered asking for comments to be allowed.

Orlowski then posts two emails for us to compare. One he notes is from a "Climate Scientist from a top university. A PhD". Climate scientist or not I found his arguments rather ineffectual and kind of lame. Perhaps why his email was used as part of the comparison. I can imagine the authors don't really get many emails trying to correct them - it's a waste of time trying to teach fools in my opinion unless you are being paid to do so.

The other email posted was from "an experienced computer modeler".

I feel these two emails are a good comparison of the "debate". On one side we have scientists who are often, but not always, ineffective at the "debate" and on the other we have psuedoskeptics who are often, but not always, idiots.

Look at the start of the 2nd email:

"I am a statistical modeller by profession and an engineer by education. I find the climate scientists argument about positive feedback the most disingenuous of them all."

So might we expect him to lay into an argument for why climate scientists are being "disingenuous" to think climate sensitivity is high?

Well no. When he says "argument about positive feedback" being "disingenuous", he really means the whole field of climatology are liars for even using the term positive feedback.

"If you want a square wave generator, or a triangle wave generator, then you use positive feedback. Otherwise the system has to be controlled with negative feedback. The reliance on positive feedback is always going to create a runaway system."

His error is to think there's only one definition of positive feedback and to think the definition in his field is the same definition used in climate science. It isn't. Climate feedback being positive means amplification, but not runaway.

Now that mistake doesn't make him an idiot. It's not obvious that Climate Science uses a different definition of feedback than most. The idiocy is connected to his gullibility. The idiocy is a twofold compounded type of idiocy:

1) Imagining that an entire field of scientists are lying to themselves, each other and scientists in other fields. It's so preposterous that for decades climate scientists would use the term "positive feedback" in a disingenuous way that you have to wonder what he was thinking. A definite WTF.

2) Ignoring the fact that scientists don't claim doubling co2 causes runaway warming. 3C warming per doubling of co2, for example, is not runaway warming. So what they call it is irrelevant. Positive feedback, negative feedback, the great and merciful feedback of mighty Dagon, it's still 3C warming per doubling.

"And what is most non-intuitive about the whole thing is that temperatures on the earth have remained remarkably stable. Yes, the equilibrium point has changed, but the stability hasn't.

It's as if by describing any change as a change from one equilibrium point to another, it means no change has actually occurred! This is a kind of definition-psuedoskeptic. One who plays with words to try and redefine reality.

I have no comments to make on the rest, well I do but they aren't interesting. I don't know enough about models and his claims are rather specific. He may be bullshitting, but I won't.

"I will be interested to see what happens when the greenhouse gas hypothesis is shown to be the charade that it is."

Hell will freeze over, and then Earth.