Postdoc studying combustion modeling, science contributor for Ars Technica, husband. Not necessarily in that order.
196 stories

A letter that I did not send to my dear uncle, who sent me a climate change denial article from a right-wing copypasta content farm

2 Comments and 11 Shares
1523-4 woodblock of a skeleton playing a drum next to a well dressed man and woman

The Lady by Hans Holbein

Dear Uncle,

I guess you sent me this article because you see how worried I am about climate change on twitter and you don’t believe it’s happening. I remember you talking about the emails hacked from the Climate Research Unit at UEA several years ago, but I didn’t realize you were skeptical about the effects of climate change.

I am very sorry to say that climate change is happening. I don’t want to believe it’s true and I don’t want to believe it’s as bad as it is. My whole PhD is premised on the idea that maybe [a really important species] would be able to adapt better than our projections said because of [cool feature of their biology]. (I don’t have the answer yet, but it’s not looking promising.)

I’ve spent the last 15 years learning about biology and ecology and the natural world and participating in the scientific process and working with other scientists. Obviously, I don’t know everything and science isn’t perfect. But most of us are trying our best and aren’t out to trick anyone. I believe climate change is happening and so does nearly every scientist I’ve ever known. We have disagreements about how fast it’s happening and what its exact effects will be and if we can survive it, but we agree that it’s going to be very, very bad.

I’m so convinced climate change is happening that I spend a great deal of time that I should be working on my PhD or that I could be doing things I love like reading novels and dancing instead writing letters to politicians and advocating for policies that would help slow climate change or at least help us adapt. I’m so convinced that I cry about it and am terrified about the world I’ll grow old in. I’m so convinced that I’ve tried to convince my mother to move (and I think you all probably should as well, especially your kids) because the southeast is going to get so much hotter and flood and fire prone during my lifetime that it’s going to severely disrupt the economy and make people very sick.

And I’m not alone. The news doesn’t spend a lot of time talking to scientists about how they feel personally about climate change, but it’s bleak. We give dry presentations and cry together over dinner at how many plants died at our study sites. We talk about fears for our children or choosing not to have them. I’m afraid of what the world is going to be like in 20 years and I’m grieving the ecosystems dying right in front of us.

The article you sent me says that the recent National Climate Assessment is based on cherry picked data and bad models and bad science and that fossil fuels have done a lot of good. It claims that all this noise about climate change is just a ploy to control politics. It isn’t, though an emergency of this scale really should affect politics.

The climate models were right before the internal combustion engine was invented

I know it can seem like everything about climate change is based on these overly complex computer models run by scientists who only care about getting their next grant, but those computer models are just fiddly details.

We knew that using fossil fuels and such could cause global warming from before the US Civil War – more than 150 years before we were able to build the complex global models of climate that we’re now using to figure out exactly how global warming will change regional and local climates.

The earliest calculations of how much carbon dioxide warmed up the planet were done by a Swedish scientist in the 1890s. His predictions weren’t perfect, but they’re not that far off from our current models. And the simple computer models we built in the 70s and 80s predict basically the same global temperature increase as the highly complex ones we have today.

(We keep building more and more complex models with more and more things because we are trying to understand more and more local effects and also feedback cycles – what does it mean for the world to warm 2 or 6 or 10 degrees? When will the ice melt? How will that affect ocean currents? How will the ocean currents full of meltwater affect the speed and sinuousness of the jetstream? Will the southwest get wetter or drier? Will there be more hurricanes or fewer? What could the economic impacts be?)

Perhaps that snippet of scientific history hasn’t convinced you to take climate change seriously. After all, even if we know that carbon dioxide and other greenhouse gases heat up the planet, maybe we’ve done our math wrong and it’s actually much slower than we think.

But what if the models are wrong and climate change isn’t a big deal?

So, what if the models are wrong about how much and how fast climate change is happening? We need to compare the risk of inaction vs action if we’re right and if we’re wrong.

What is the risk of inaction if we are right about the magnitude and speed of climate change impacts? Well, you could read the fourth NCA to find out! It does not, as the article you sent me put it “sound like something kicked around in a Hollywood brainstorming session for a science fiction thriller.” It is sober and measured and accessible – and ultimately very conservative in its discussion of potential impacts. Here’s a representative snippet from the chapter on climate change impacts occurring and expected in the Southeast

Embedded in these land- and seascapes is a rich cultural history developed over generations by the many communities that call this region home. However, these beaches and bayous, fields and forests, and cities and small towns are all at risk from a changing climate. These risks vary in type and magnitude from place to place, and while some climate change impacts, such as sea level rise and extreme downpours, are being acutely felt now, others, like increasing exposure to dangerously high temperatures—often accompanied by high humidity—and new local diseases, are expected to become more significant in the coming decades.

If our models are right and we don’t do anything, the impacts from climate change will be big and bad this century and downright apocalyptic in the next. The only reason not to act would be if you believe the impacts of trying to slow or stop climate change are worse than the risks from climate change itself.

The biggest things we can to do as a society to slow climate change are:

  • educate girls and make sure women have access to contraceptives and reproductive healthcare, including abortion
  • change the kind of chemicals we use as refrigerants (like in air conditioners) (and switch as many of them as we can to things like ground source heat pumps).
  • switch electricity generation from coal and natural gas to solar and wind as fast as we can, reducing air pollution in the process and creating a bunch of jobs
  • Eat more beans and less red meat while wasting less food – cheaper and healthier!
  • Stop burning tropical forests – most are being burnt to grow soybeans to feed cows, so this is almost a natural outcome of the previous goal
  • Bring back silvopasture farming techniques – supporting small farmers and rural areas and loosening the exploitation by companies like Smithfield.

None of these have costs higher than the ones our models predict climate change will exact, in dollars, in lives, in social and cultural disruption. Most are actually things we’d want to do regardless of climate change. Many negative effects are things that are happening anyway.

Consider what happens

If we don’t act and the models are right, we face a serious existential threat. Over the next centuries, billions of people will die in extreme weather events, wars, and famines, the economy will collapse worldwide, we could lose a great deal of technology and civilization, and could even go extinct. Large parts of the tropics and subtropics will become uninhabitable and billions will be forced to migrate by rising sea levels and increasing temperatures. Immigration by climate refugees will completely overwhelm many countries. Many, many species will go extinct.

If we don’t act and the models are wrong, we gradually decarbonize the economy anyway while demographic, habitat destruction, and agricultural problems continue unabated. The transition to solar and wind will continue, relatively slowly, because technology has advanced already to the point that it’s cheaper than at least coal already. As fossil fuels are gradually depleted and renewable tech continues to improve, the transition will speed up. This will gradually reduce the millions of deaths every year due to air pollution. Depending on the speed of transition, we will have to retrain or support people and communities who used to be dependent on fossil fuel extraction – or consign them to lives of poverty and us all to political unrest. Hundreds of millions more women will have children they weren’t ready to have and lack education and opportunities, holding back their countries’ political and economic development. Industrial agriculture will continue to destroy habitat and small farms and suck resources out of rural areas. The loss of rainforests will cause the loss of many incredible species and result in changes to global weather patterns that could devastate some agricultural regions. Obesity and metabolic diseases will continue to increase, along with associated healthcare costs.

If we act and the models are right, we’re still going to continue to see a lot of climate change impacts because we’ve just waited too long to act, but the effects won’t be so catastrophic. Fewer people will die in extreme weather events or of starvation, and immigration will be less overwhelming because fewer people will have to flee sea level rise and increasingly inhospitable climates. We will save millions of lives every year just from the reduction in air pollution from transitioning to solar and wind. Many jobs will be created in order to rapidly transition to solar and wind and improve energy efficiency in buildings. We will have to retrain or support people and communities who used to be dependent on fossil fuel extraction – or consign them to lives of poverty and us all to political unrest. More people will use affordable heat pumps or safe refrigerants to cool (and heat) their homes than before. Dietary improvements in the west will extend lives and reduce healthcare costs from metabolic disease. Population growth will slow, helping countries in the global south reduce emigration and stabilize their political systems. We will lose rare desert habitat to solar farms and some birds to wind farms, but much less than unmitigated climate change would have caused. Switching so rapidly to renewable energy with today’s technology will mean a lot of mining for rare earth minerals, which will likely cause large areas of environmental destruction in parts of the American west and China – much as uranium mining and coal mining did in the previous century.

If we act and the models are wrong, we will make improvements in agriculture, public health, political stability in tropical and subtropical countries, gender equality, and environmental protection, at the expense of some resource extraction communities. We will deal with the necessary transition away from fossil fuels earlier than we needed to to avoid climate change impacts, possibly with technologies that aren’t as advanced as they would have been had we waited. However, we will save millions of lives every year just from the reduction in air pollution from transitioning to solar and wind. Many jobs will be created in order to rapidly transition to solar and wind and improve energy efficiency in buildings. We will have to retrain or support people and communities who used to be dependent on fossil fuel extraction – or consign them to lives of poverty and us all to political unrest. More people will use affordable heat pumps or safe refrigerants to cool (and heat) their homes than before. Dietary improvements in the west will extend lives and reduce healthcare costs from metabolic disease. Population growth will slow, helping countries in the global south reduce emigration and stabilize their political systems. We will lose rare desert habitat to solar farms and lots of birds to wind farms. Switching so rapidly to renewable energy probably will mean a lot of mining for rare earth minerals, which will cause environmental destruction in parts of the American west and China – much as uranium mining and coal mining did in the previous century. Many species are saved in the rainforests.

You believe that the models are wrong and we we shouldn’t act. You’re afraid that the models are wrong and we will act.

But your fear is actually the very best outcome: the very best situation is if we act to stop climate change and we are wrong about climate change. Acting to stop climate change makes the world better even if climate change doesn’t happen.

(The above assumes, of course, that the models are overpredicting climate change impacts. We are very likely under-predicting the impacts of climate change. Choosing what to do, what to prioritize, if climate change is going to be much worse than we imagine is another discussion.)

I don’t think any of these futures are easy, even the ones where everyone does exactly what I think is politically right and things go perfectly according to plan. We have waited so long to act to stop climate change that we now have to act very fast and no matter what we’ll do, bad things will happen. Fast change is very hard and disruptive. We’re going to face fast change no matter what, but we have a choice, now about what that change looks like. We can choose the change and we can get our collective butts in gear, or we’ll be swept away by changes we didn’t see coming, alone.

What if there were no models?

But perhaps you believe any model of climate change is just wrong. (They are, of course. The only perfect model is the thing itself, but you’re not going to throw out all your maps because they don’t have every pothole in the road on them.)

So then – imagine that we don’t have any of these climate models.

We still know that some gases like carbon dioxide and methane hold more heat than others because anyone can figure that out with some sunshine and bottles filled with different gases on a sunny day and a couple thermometers.

But if no one ever built the kinds of models predicting what adding lots and lots of greenhouse gases to the atmosphere would do, then what should we do if we just don’t know what the result of dumping greenhouse gases in the atmosphere will be?

I believe we must act very cautiously. We only have one planet. It is foolish and selfish to take more than the smallest of risks with it.

If we don’t know how much greenhouse gases will warm up our planet or what warming up a planet will do to climate, we just shouldn’t do it until we do understand. Experimenting with the only planet we can currently survive on does not seem sensible.

In our climate-model-less world, we also still know that climate is a complex dynamical system, like the human brain or the power grid, where small changes in one part of the system can cause rather larger changes in another. Bigger changes are quite likely to cause bigger changes. In a world without global climate models, we would want to be very wary of changing levels of temperature-changing gases in our atmosphere much at all.

If we know that greenhouse gases can cause global warming and we know that the global climate system can behave unpredictably, then even if we don’t know how or why or how much change those gases cause, it is irresponsible and dangerous to continue increasing greenhouse gas concentrations.

I want us to fix climate change, but I don’t want you to feel like this

Academics get accused of elitism all the time. And some of that is completely warranted. This next bit is elitist.

I want everyone to take climate change seriously because we have waited so long and so must do so much, so fast to stave off the most unthinkable effects. But part of me doesn’t want to convince you, my dear uncle.

Changing your mind, individually, probably won’t make much difference to whether we get a Green New Deal or not.

But understanding what climate change impacts will look like is horrifying and painful. You’re old enough and well-off enough that you and Aunt ____ will probably be fine as long as your AC keeps going and you don’t move to the coast. I don’t want you to spend the rest of your life terrified for my cousins and your new grandbaby, mourning the changes to the land you love as trees and animals migrate and change and die.

So I’m not going to send you this letter, I’m not going to try to convince you that climate change is real. Facing it is like facing death, and of course I would spare you that if I could.

So live your life, blissfully ignorant and cheerful on the phone about the strange weather lately. Live free from the fear that everything you’ve contributed to and cared about in your life will be gone so soon and live free from the guilt that you supported the political and economic choices that may kill us all.

Read the whole story
371 days ago
This. So much this.
Corvallis, OR
165 days ago
371 days ago
I wrote basically this exact email to my uncle. The rest of my family were not amused and felt like I should go easy on him.
New York
165 days ago
370 days ago
165 days ago
Share this story

Seven Years

9 Comments and 34 Shares
Read the whole story
804 days ago
Corvallis, OR
802 days ago
Share this story
8 public comments
802 days ago
San Francisco, CA
803 days ago
Touching and beautiful! One of your best.
803 days ago
804 days ago
Melbourne, Australia
804 days ago
Greater Bostonia
804 days ago
That's why I still watch it..Time to time they deliver...
793 days ago
j'ai toujours autant de mal à comprendre la trame...
804 days ago
Awesome. I'm speechless.
804 days ago
Louisville, Kentucky
804 days ago
God damnit, Randal.
804 days ago
For those that don't know the whole story: Approximately 7 years ago (imagine that) Randall posted this on the blog and made some vague references to tough times in the comics. On in to 2011, he posted this on the blog, and things seemed to be scary but hopeful. . He's made mention several times about it over the years inside the comics, and I really believe that "Time" was made for some express purpose as to get his emotions out. But this update seriously is making a grown 32 year old man weep openly at his desk (thankfully I have a door that closes), as I always wondered how things were. Things look good, and this makes my heart happy.

Jet engines are a major part of aviation today, and this great...

1 Comment

Jet engines are a major part of aviation today, and this great video from the new LIB LAB project breaks down how jet engines operate. It focuses especially on the subject of combustion, in which fuel-air mixtures are burned to generate power and thrust. By breaking fuels down into simpler compounds, jet engines are able to accelerate exhaust gases, which creates thrust. They even provide instructions for an effervescence-driven bubble rocket so that kids can (safely!) experiment with propulsion at home. (Video credit: LIB LAB/Corvallis-Benton County Public Library)

Read the whole story
1087 days ago
Check this out—science communication video on jet engines made by my PhD student!
Corvallis, OR
Share this story

Open letter to my family in Ohio: I am still coming home for Thanksgiving this year

2 Comments and 12 Shares

Dear Mom and Pop,

This was going to be an awkward year for us from the start. Thanksgiving is normally a time when we, two dozen immigrant Jews and first-generation Americans, come together to eat a turkey stuffed with oranges, served with a side of smoked fish and a dozen competing salads.

Before the election, I thought the greatest thing that would divide our dinner table would be my recently adopted vegetarianism. But us Jews are all about dietary restrictions. Eight breadless days of weight loss every year during Passover. Fasting from food and water every year for the day of atonement. Two sets of plates for Shabbat and for everyday. I was certain that refusing meat wouldn’t stop me from feeling embraced and loved at the table.

A russian salad feast courtesy of my friend Andrey Petrov

But then last week you voted for Trump. Jewish refugees from the Ukraine now living in Ohio voted for Trump. You have told me several times that Obama is the worst president in our lifetime. And so you voted for Trump.

This letter would not have been written if this were just about politics. Both of you have always voted conservatively. And I wouldn’t be this livid if it were just about privatization of health care, or import tariffs, or the unchecked negative externalities of infrastructure spending on energy extraction. The political tide goes in and out. How we run the country makes for a spirited debate over the Thanksgiving dinner table and I welcome our family’s contrary viewpoints. But this is not about politics.

Voting for Trump is clearly a vote for hate. I know you voted against Hillary, and not “in favor” of Trump, but the action is the same. Your vote signals to me that you see our fellow citizens as less than human. History shows where this leads and history is not just words in a textbook for us. You are choosing to ignore our past.

Donald Trump Says He’d ‘Absolutely’ Require Muslims to Register

We would have seven times as many cousins if it weren’t for the Nazis. You helped elect a fascist and are now complicit in what’s going to happen to families both in and outside of our country.

It is not unreasonable for people to think that fascists, if given power and an attempt to govern, will moderate their views. But history repeats itself.

“Several reliable, well-informed sources confirmed the idea that Hitler’s anti-Semitism was not so genuine or violent as it sounded, and that he was merely using anti-Semitic propaganda as a bait to catch masses of followers and keep them aroused, enthusiastic, and in line for the time when his organization is perfected and sufficiently powerful to be employed effectively for political purposes.”
“You can’t expect the masses to understand or appreciate your finer real aims. You must feed the masses with cruder morsels and ideas like anti-Semitism. It would be politically all wrong to tell them the truth about where you really are leading them.”

I take Trump at his hate-filled words. Last time we spoke, you did not. You said he probably wouldn’t be fulfilling any of his promises, like the ones to register Muslims or deport Latinos. However, even if you do believe that his campaign was only hype, you still ignored despicable behavior and rhetoric. Normalizing racism is itself an act of racism.

Some positions are binary. You either care about the protection of minorities or you do not. I have friends now who will stand in the same shoes that my grandparents stood in before. And you knew, regardless of your enthusiasm for Trump’s economic policies, that these people would suffer. The increasing number of graffitied swastikas, along with other vile behaviors now edging into the mainstream, are the kind of atrocities your vote was supposed to suppress.

You taught me that my highest responsibility as a Jew is to never forget, to fight against discrimination. This was a common refrain told to me during seven years of Hebrew school. I take that responsibility seriously and that is what I am trying to do now. Elie Wiesel said “We must take sides. Neutrality helps the oppressor, never the victim. Silence encourages the tormentor, never the tormented.”

I understand why you voted to put Trump in office. It’s just hard for me to forgive you for it. And now is not the time to give bigotry a pass. I’m trying to live by the values you taught me, which is why I am only asking you to take our President at his word. And when he tells you of his bigoted promises to force members of a religion to register, you should be outraged.

I am coming home this Thanksgiving and it’s going to be painful, most of all for me. I try to be empathetic with how you feel about the state of our country, but what I do not understand is how you think that voting for a racist and hate-filled demogogue with no political experience is going to promote growth and peace. At best, it will alienate over half of this country, and at worst, history will repeat itself.

I am coming home because I want to believe the love that brings us together at Thanksgiving is strong enough to not only include a vegetarian, but also strong enough to allow you to understand my perspective and possibly even change your minds. I need my family with me in this fight.

Your son,

Read the whole story
1192 days ago
I just wrote this letter to my family. Thanks to those of you who helped convince me to do the hard thing and go back and tell them in person how I feel.
Cambridge, Massachusetts
1192 days ago
Good luck with the trip home. I have no idea how I'm going to deal with this when the time comes. (I'm living overseas and won't be moving back until right at the holidays, so I get a pass this year). (And this was beautifully written.)
1191 days ago
Good luck, Samuel, and good on you for believing there's even a possibility of understanding. It makes me think of the profound book I Am Asher Lev, which I read recently, and how communities work with their differences. It takes great courage. Thank you for sharing this.
1191 days ago
This will be me in a few days too (with a family of trump supporters) although I won't be around Jews (I will be the only Jew though) but still celebrating pilgrims who were escaping for their religious freedom. I am sure they may just discount me for my background and we will avoid the subject but who knows. I am bringing some strong beer.. let us know how yours go as I would like to compare notes. Oh and I have just been through Brexit so to a degree I am dealing with deja vu of sitting with a family with a huge elephant in the room.
1189 days ago
My love to you and yours, Sam.
1181 days ago
Samuel, I want to thank you from the bottom of my heart for writing that letter to your family, and most of all, maybe, for sharing it with all of us. It's a beautifully written letter. I am so proud of you for being who you are, for starting and continuing to show up and nurture the incredible NewsBlur. You are such a good person, Samuel, you have a depth to you that eludes understanding. I wish you love, and peace too, and tenacity to persevere in these terrible days. Thank you so very much.
1191 days ago
1191 days ago
Corvallis, OR
Share this story
1 public comment
1191 days ago
It hurts that this had to be written
Washington, DC

Pistons coach goes on tirade about 'racist and misogynistic' Donald Trump

4 Comments and 15 Shares


Detroit Pistons coach Stan Van Gundy has never been one to mince his opinions, so it was only a matter of time before he went on a rant after Donald Trump was elected president of the United States. 

Speaking to reporters prior to the Pistons’ NBA matchup against the Phoenix Suns on Wednesday, Van Gundy went on a nearly six-minute, unprompted tirade on the unconventional president-elect, who beat Hillary Clinton to the White House, saying he was ashamed of the public for voting in the 70-year-old.

Here is the full transcript of Van Gundy’s comments to the media:

“I didn’t vote for [George W.] Bush, but he was a good, honourable man with whom I had political differences, so I didn’t vote for him. But for our country to be where we are now, who took a guy who - I don’t care what anyone says, I’m sure they have other reasons and maybe good reasons for voting for Donald Trump - but I don’t think anybody can deny this guy is openly and brazenly racist and misogynistic and ethnic-centric, and say, 'That’s OK with us, we’re going to vote for him anyway’.

“We have just thrown a good part of our population under the bus, and I have problems with thinking that this is where we are as a country. It’s tough on [the team], we noticed it coming in. Everybody was a little quiet, and I thought, ‘Well, maybe the game the other night’. And so we talked about that, but then Aron Baynes said, ‘I don’t think that’s why everybody’s quiet. It’s last night’.

“It’s just, we have said - and my daughters, the three of them - our society has said, ‘No, we think you should be second-class citizens. We want you to be second-class citizens. And we embrace a guy who is openly misogynistic as our leader’. I don’t know how we get past that.

“Martin Luther King said, 'The arc of the moral universe is long, but bends toward justice’. I would have believed in that for a long time, but not today. What we have done to minorities… in this election is despicable. I’m having a hard time dealing with it. This isn’t your normal candidate. I don’t even know if I have political differences with him. I don’t even know what are his politics. I don’t know, other than to build a wall and ‘I hate people of colour, and women are to be treated as sex objects and as servants to men’. I don’t know how you get past that. I don’t know how you walk into the booth and vote for that.

“I understand problems with the economy. I understand all the problems with Hillary Clinton, I do. But certain things in our country should disqualify you. And the fact that millions and millions of Americans don’t think that racism and sexism disqualifies you to be our leader, in our country. We presume to tell other countries about human-rights abuses and everything else. We better never do that again, when our leaders talk to China or anybody else about human-rights abuses.

“We just elected an openly, brazen misogynist leader and we should keep our mouths shut and realize that we need to be learning maybe from the rest of the world, because we don’t got anything to teach anybody.

“It’s embarrassing. I have been ashamed of a lot of things that have happened in this country, but I can’t say I’ve ever been ashamed of our country until today. Until today. We all have to find our way to move forward, but that was - and I’m not even trying to make a political statement. To me, that’s beyond politics.

“You don’t get to come out and talk about people like that, and then lead our country and have millions of Americans embrace you. I’m having a hard time being with people. I’m going to walk into this arena tonight and realize that - especially in this state - most of these people voted for the guy. Like, [expletive], I don’t have any respect for that. I don’t.

“And then you read how he was embraced by conservative Christians. Evangelical Christians. I’m not a religious guy, but what the hell Bible are they reading? I’m dead serious. What Bible are you reading? And you’re supposed to be - it’s different. There are a lot of different groups we can be upset at. But you’re Christians. You’re supposed to be - at least you pride yourself on being the moral compass of our society. And you said, ‘Yeah, the guy can talk about women like that. I’m fine with that’. He can disparage every ethnic group, and I’m fine with that.

“Look, I don’t get it. And I’m having a hard time taking it. I’m just glad that the people I’m with here - and I’ll include you guys, too - that I like. Because I’m going to have a hard time. I will say, one point of pride, I live in Oakland County, Michigan, and I was surprised, but Oakland County voted for Clinton. At least I can look around say, ‘We weren’t the ones putting that guy in office’.

“It’s incredible. I don’t know how you go about it, if you’re a person of colour today or a Latino. Because white society just said to you, again - not like we haven’t forever - but again, and emphatically, that I don’t think you deserve equality. We don’t think you deserve respect. And the same with women. That’s what we say today, as a country. We should be ashamed for what we stand for as the United States today.

“That’s it for me. I don’t have anything to say about the game tonight.”

All white people should model ourselves after Stan Van Gundy.

Read the whole story

Barbagroup reproducibility syllabus

1 Comment
Lockheed P-80A airplane (1946). Credit: NASA Commons. —A reminder to test your code.

After my short piece, “A hard road to reproducibility,” appeared in Science, I received several emails and Twitter mentions asking for more specific tips—both about tools and documents we use in the group to train the team about reproducibility.

In answer to popular demand, then, I have collected here what we could call the “Barba-group Reproducibility Syllabus.”

Top-10 Readings in Reproducibility

Early this year, my student Olivier and I were getting started writing a book chapter and later a full-length journal article; the first was about our reproducible-research workflow and the second on our CFD replication study. These represented about three years of work, not exclusively on this project, but taking most of the graduate student’s time. As part of our “pre-writing” tasks, we decided to build—collectively as a group—our list of Top 10 papers discussing reproducible research in computational science. Here’s our current reading list (modified from our first version of Feb. 2016):

  1. Schwab, M., Karrenbach, N., Claerbout, J. (2000) Making scientific computations reproducible, Comp. Sci. Eng. 2(6):61–67, doi: 10.1109/5992.881708
  2. Donoho, D. et al. (2009), Reproducible research in computational harmonic analysis, Comp. Sci. Eng. 11(1):8–18, doi: 10.1109/MCSE.2009.15
  3. Reproducible Research, by the Yale Law School Roundtable on Data and Code Sharing, Comp. Sci. Eng. 12(5): 8–13 (Sept.-Oct. 2010), doi:10.1109/mcse.2010.113
  4. Peng, R. D. (2011), Reproducible research in computational science, Science 334(6060): 1226–1227, doi: 10.1126/science.1213847
  5. Diethelm, Kai (2012) The limits of reproducibility in numerical simulation, Comp. Sci. Eng. 14(1): 64-72, doi: 10.1109/MCSE.2011.21
  6. Setting the default to reproducible (2013), ICERM report of the Workshop on Reproducibility in Computational and Experimental Mathematics (Providence, Dec. 10-14, 2012), Stodden et al. (eds.), // report PDF
  7. Sandve, G. K. et al. (2013), Ten simple rules for reproducible computational research, PLOS Comp. Bio. (editorial), Vol. 9(10):1–4, doi: 10.1371/journal.pcbi.1003285
  8. Leek, J. and Peng, R (2015), Opinion: Reproducible research can still be wrong: Adopting a prevention approach, PNAS 112(6):1645–1646, doi: 10.1073/pnas.1421412111
  9. M. Liberman, “Replicability vs. reproducibility — or is it the other way around?,” Oct. 2015,
  10. Goodman, S. N., Fanelli, D., & Ioannidis, J. P. (2016). What does research reproducibility mean? Science Translational Medicine 8(341), 341ps12-341ps12, doi: 10.1126/scitranslmed.aaf5027

Schwab et al. (2000) report on the pioneering example of reproducible research in the Claerbout lab (Exploration Geophysics, Stanford University). The first public communication of this group’s approach that we could find goes back to 1992 [1]. That paper describes tools and processes in more detail, but for the same reason it is quite dated. So, we start with the summary account in CiSE. The Claerbout group developed an automatic build system for their published papers, including all the analyses and figures plus the typeset document. They used GNU make, certain standardized commands (burn, build, view, clean), and a notion of the file set or research compendium associated with the paper (data sets, programs, scripts, parameter files, makefiles). They report having used the system to-date for 14 papers involving 15 authors and hundreds of files. It’s remarkable to read about their careful methods for reproducible documents, given that more than two decades later we’re still struggling to adopt similar standards more widely.

Jump to Donoho et al. (2009). This could be the first group to explicitly associate reproducible research with open code and data:

Reproducible computational research, in which all details of computations—code and data—are made conveniently available to others, is a necessary response to [the credibility] crisis.

Donoho et al. admonish that computation cannot claim to be the third branch of science because most computational results cannot be verified. In the two traditional branches, standards of practice already exist for managing the ubiquity of error: deductive science uses formal logic and the mathematical proof, while empirical science uses statistical hypothesis testing and detailed methods reporting. “Many users of scientific computing aren’t even trying to follow a systematic, rigorous discipline that would in principle allow others to verify the claims they make.” Ouch!

Donoho et al. cite strong influences from Claerbout’s methods, and lament that these are still not widely practiced. This paper also repeats the classic paraphrase of Claerbout: “an article about computational science … is not the scholarship itself, it’s merely scholarship advertisement. The actual scholarship is the complete software development environment and the complete set of instructions which generated the figures.” (First appearing in a 1995 paper from this group [2].) My favorite quote from Donoho et al. (2009) is: “… if everyone on a research team knows that everything they do is going to someday be published for reproducibility, they’ll behave differently from day one.” The middle sections of the paper describe the various computational libraries developed to date in the Donoho group; those sections can be skimmed according to the reader’s interest. Towards the end, an interesting passage—written in the format of a Q&A—addresses the typical objections of researchers to working reproducibly. Many such objections are still hot topics today: it takes time and effort, we get no credit for it, competition, and so on. Notably, the final hypothetical objection is that “true reproducibility” should mean starting from scratch to re-create the results (rather than from the author-provided code and data). The rebuttal: “…it proves nothing if your implementation fails to give my results because we won’t know why it fails. The only way we’d ever get to the bottom of such a discrepancy is if we both worked reproducibly…"

The jointly authored paper of the Yale Law Roundtable participants (CiSE, 2010) expanded on the theme of transparency via open code and data. They defined reproducible computational research unambiguously as that making available all details (code and data) of the computations. Their additional recommendations include: assigning a unique identifier to every version of the data and code, describing within each publication the computing environment used, using open licenses and non-proprietary formats, and publishing under open-access conditions (or posting pre-prints). The rationale behind linking open access with reproducibility was absent, and some have criticized this aspect of the Roundtable recommendations. It may have grown out of the idea in Donoho et al. (2009) that “reproducibility means publication over the Internet,” and that authors should maintain a Web presence to facilitate discovery and access to their research. The connection between reproducible research and open-access publishing is, however, questionable. On the other hand, open code and data are valid components of reproducible computational research. Among future goals, the Yale Roundtable recognized the importance of enabling citation of code and data, of developing tools to facilitate versioning, testing and tracking, and of standardizing various aspects like terminology, ownership, policy.

Peng (2011) introduced the idea of a reproducibility spectrum. He says that reproducible research is a “minimum standard for judging scientific claims when full independent replication of a study is not possible.” Here we find an explicit distinction in terminology—something that continues to muddle the field—where full replication of a study involves collecting new data, with a different method (and code), and arriving at the same or equivalent final findings. (The distinction previously appeared in [3]) Peng mentions the Sloan Digital Sky Survey as an example of a project that would require formidable resources to fully replicate, and therefore proposes that reproducibility is a lesser standard that is more attainable. Other domains exist where full replication is unrealistic or extremely expensive. Reproducibility, says Peng, “falls short of full replication because the same data are analyzed again.” Nevertheless, it is a desirable minimum standard to assess the quality of the scientific claims. It requires that “the data and the computer code used to analyze the data be made available to others.” But, Peng laments, “the biggest barrier to reproducible research is the lack of a deeply ingrained culture that simply requires reproducibility for all scientific claims.”

Number 5 on our list (ordered chronologically) shifts to a different concern: numerical reproducibility in computations that involve parallel processing. In the discussion up until now, the concept of reproducible research assumed that running the same code twice with identical input will produce the same output. If the computation is done in serial, this assumption is good; but with parallel computing, it is not always the case. Diethelm (2012) ran an experiment using an application of finite-element analysis in computational mechanics. Executing the same simulation (same code, same input data) with varying number of processors gave different results! Investigating the differences and the source code pinpointed the cause of non-deterministic behavior: a direct solver for sparse linear systems (an external library). Diethelm goes through an example that illustrates how this can happen: a vector dot-product, computed in parallel over several partial sums. On each execution, individual processors may complete their portion of the sum in different order. In finite precision, addition is not associative and the final sum depends on the order of the partial sums. Under these conditions, ensuring numerical reproducibility involves introducing artificial synchronization points in the program, at the cost of additional run time. More elaborate techniques are available, but the conclusion is that in high-performance computing “lack of reproducibility is typically a price that must be paid for speeding up the algorithm.”

The ICERM Workshop Report (2013) builds on the contributions of the Yale Roundtable by placing particular focus on: (1) changing the culture and reward structure; (2) role of funding agencies, journals and employers; (3) teaching the skills for reproducible research. The required culture change includes valuing openness and transparency. But the academic reward structure sets critical barriers: “The current system, which places a great deal of emphasis on the number of journal publications and virtually none on reproducibility …penalizes authors who spend extra time on a publication.” Moreover, software development and data management are not valued scientific activities. The report addresses several ways to introduce incentives, requiring leadership from funders, journals and employers. Many of those discussions continue to this day in different venues (workshops, journal editorials, blogs, etc.).

I should add that I participated in the ICERM Workshop, giving a short talk titled “Reproducibility PI Manifesto.” The slides of this talk have been widely shared and commented [4].

Sandve et al. (2013) give us ten concrete actions we can take to make our research reproducible:

  1. For every result, keep track of how it was produced
  2. Avoid manual data-manipulation steps
  3. Archive the exact versions of all external programs used
  4. Version-control all custom scripts
  5. Record all intermediate results, when possible in standard formats
  6. For analyses that include randomness, note underlying random seeds
  7. Always store raw data behind plots
  8. Generate hierarchical analysis output, allowing layers of increasing detail to be inspected
  9. Connect textual statements to underlying results
  10. Provide public access to scripts, runs, and results

Some common threads run through most of these recommendations. First, recognizing that a final result is the product of a sequence of intermediate steps (the analysis workflow), a key device for reproducibility is automation. Second, the central technology for dealing with software as a living, changing thing, is version control. And finally, archive and document everything with the best tools at hand. The one, inescapable corollary for the purposes of training researchers is that command-line skills are essential.

Item 8 of our reading list (Leek and Peng, 2015) expands on the purpose of reproducible research: to protect the integrity of science and build the public’s trust on scientific results. Although a reproducible study can still suffer from poor study design, missing data, or confounding factors, reproducibility increases the rate at which we can uncover these flaws. Even so, the key is prevention via the training of more people on techniques for data analysis. Leek and Peng contribute to this goal via their massive online courses, and they also recognize the value of crowd-sourced workshops like Software Carpentry and Data Carpentry.

Next on the list is an essay by Mark Liberman, Christopher H. Browne Distinguished Professor of Linguistics at the University of Pennsylvania. He teaches introductory linguistics, as well as big data in linguistics, and computational analysis and modeling of biological signals and systems (among other topics). The subject of his essay is the big confusion of terminology that has spread on the reproducibility literature. He traces the confusion to a machine-learning workshop contribution, where the terms reproducible and replicable are swapped completely, compared to previous papers. Liberman concludes: Since the technical term ‘reproducible research’ has been in use since 1990, and the technical distinction between reproducible and replicable at least since 2006, we should reject [the] attempt to re-coin technical terms reproducible and replicable in senses that assign the terms to concepts nearly opposite to those used in the definitions by Claerbout, Peng and others.”

Our final item on the reading list is from earlier this year. Goodman et al. (2016) note that the various terms used in the field (e.g., reproducible vs. replicable) are not standardized. The importance of corroborating a previous study’s results is widely recognized. But, the authors note, “ … the modern use of ‘reproducible research’ was originally applied not to corroboration, but to transparency, with application in the computational sciences. Computer scientist [mistake: geophysicist] Jon Claerbout coined the term and associated it with a software platform and set of procedures that permit the reader of a paper to see the entire processing trail from the raw data and code to figures and tables. …This concept has been [used in] epidemiology, computational biology, economics and clinical trials…” [references provided]. Goodman et al. uphold the Claerbout/Donoho/Peng terminology, but propose a new lexicon as a way out of the confusion reigning the literature: methods reproducibility (original meaning of reproducibility), results reproducibility (previously called replication), and inferential reproducibility. Who knows if this new lexicon will stick, but what I like of this paper is its skillful discussion of differences among scientific domains that affect how each addresses reproducibility. In computational research, we’re used to a degree of determinism, for example, so methods reproducibility and results reproducibility are linked. Other fields have to deal with major stochastic variability. For most computational scientists, the second half of this paper will be alien, because it focuses on issues of statistical significance testing, clinical and pre-clinical research, and so on. It is good, however, to get a glimpse into this other world of science, where p-hacking and HARKing (hypothesis after results are known) are a thing.

Additional References

[1] Claerbout, Jon and Martin Karrenbach (1992). Electronic documents give reproducible research a new meaning, Proc. 62nd Ann. Int. Meeting of the Soc. of Exploration Geophysics, pp. 601-604, doi: 10.1190/1.1822162

[2] Buckheit, J. B. and D. Donoho (1995), WaveLab and reproducible research, In Wavelets and Statistics, edited by A. Antoniadis and G. Oppenheim, Lecture Notes in Statistics 103: pp. 55–81, doi: 10.1007/978-1-4612-2544-7_5

[3] Roger D. Peng, Francesca Dominici and Scott L. Zeger (2006), Reproducible epidemiologic research, Am. J. Epidemiol. 163 (9): 783-789. doi: 10.1093/aje/kwj093,

[4] Barba, Lorena A. (2012): Reproducibility PI Manifesto. figshare, doi: 10.6084/m9.figshare.104539.v1

Read the whole story
1212 days ago
Oh this is great... I'm totally going to "borrow" this.
Corvallis, OR
Share this story
Next Page of Stories