Uncertainty is a fact of life in science. Understanding and communicating uncertainty is an essential element of the scientific pursuit. Uncertainty in research comes from the fact that, in most situations, we don’t have perfect information and can’t be absolutely sure of the accuracy and precision of all values used as inputs in our models. Uncertainty can also be the result of research subjects being unwilling, or unable, to share critical details or information. This can leave researchers with the need to estimate missing input values.
The scientific community has developed tools to communicate uncertainty. Every serious scientific study, where uncertainty is a consideration, will include an evaluation of the uncertainty and how that uncertainty affects the certainty of any consequent conclusions. The problem is that outside of the scientific community this information is often lost. Users unfamiliar with how uncertainty is reported will often look at the results only, and in doing so, come to incorrect conclusions. One such example has to do with the relative greenhouse gas (GHG) intensity of various crude oil blends.
I can’t count the number of times I have heard policy experts and activists decrying the relative GHG emissions of Canadian crude oils. Most recently, I encountered a letter/blog post by a Professor of Law at the University of Calgary. He wrote the following in a submission to the Public Inquiry into Anti-Alberta Energy Campaigns:
What appears to be conveniently missing from the Nemeth Report is the fact that, objectively and in absolute terms, Alberta’s oil sands reserves have been – and continue to be – amongst the most GHG intensive in the world….See M. Masnadi et al., “Global carbon intensity of crude oil production” Science (31 Aug 2018), Vol. 361, Issue 6405, pp. 851-853
The paper he cites as justification for his claim is Global carbon intensity of crude oil production by Masnadi et al., and the figure from that paper he presents in his submission is reproduced below. In the blog post he doesn’t include the title block which I reproduce above the figure.
A non-scientist, looking at this graph, would indeed come to the same conclusion as Professor Olszynski. Looking at the dark blue columns, Canada does indeed appear fourth from the left. Surely that puts our average crude oil upstream GHG intensity “amongst the most GHG intensive in the world“? But that is not what the graph actually shows. Why you ask? Because the graph also includes error [uncertainty] bars.
The error bars are the sticks at the top of the blue columns [that form the letter I]. They provide an indication of the variability (or uncertainty) in the graphed results. As described in the figure title block, the uncertainty bars in this paper represent the 5th to 95th percentiles of the modelling used “to explore the uncertainty associated with missing input data“.
What do this mean? In this paper, the authors did not have access to really important inputs for many of the producer countries and so had to estimate input values in their models. The authors explain this in the Supplemental Materials for the paper which includes the graphic below. It includes the figure above but with colour-coding to distinguish countries where less information is available to derive accurate estimates of oil field GHG intensities.
Canada is one of only 9 countries presented in green, indicating that the data available for Canada is of sufficient quality to come up with relatively accurate estimates. Most of the countries are various shades of orange, indicating that little is known about critical input values necessary to develop an accurate estimate of the GHG intensity of their oil fields. Because less is understood about these countries they have a wider uncertainty band (a larger error bar).
The thing readers need to understand is that when error bars overlap it indicates that the range of statistically possible results also overlap. In most cases this means that in a practical sense the research cannot say, with confidence, that the absolute values are different. Looking carefully, the error bars from 31 of the countries overlap the Canadian error bars in some manner. To make this clear, I have drawn a line from bottom of the Canadian error bar across the graph (see below). What this means is that even though Canada is fourth from the left in the figure, thanks to the error bars Canada’s value could reasonably fit anywhere in the top two-thirds of the countries.
Given that the Canadian error bar overlaps with 31 of the 50 countries studied, what it means is that the research does not find that “objectively and in absolute terms, Alberta’s oil sands reserves have been – and continue to be – amongst the most GHG intensive in the world“. Rather, the research used to justify that statement finds that Canadian crude oil GHG intensities cannot be distinguished, with any statistical certainty, from the crudes produced in 31 of the top 50 oil producing countries. The researchers simply don’t have the data to make sufficiently reliable estimates for most major producers and according to this research Canadian crude GHG intensities could be right in the middle of the pack.
Now to be clear, I do not, for a minute, believe that heavy Canadian oil sands crude is among the least GHG intensive in the world, but the data available does not rule this hypothesis out…and it gets worse. The Masnadi report did not evaluate the effect of venting of methane in their analyses and only incorporated estimates of flaring in the development of their emission estimates due to the lack of hard data on flaring. Given methane venting and flaring significantly impacts the GHG intensity of oil fields, this means that the GHG intensity estimates for many of these countries could be significantly worse.
Canada, strictly regulates their venting and flaring and the Canadian estimates for these components are considered very accurate. As for other producers? Well consider this story (Satellites reveal major new gas industry methane leaks) which describes what happened when satellite studies were used to estimate venting and incomplete flaring in Russian oil fields:
Look at all that venting in their conventional oil. Were this venting included in the calculus the GHG intensity for Russia would be far higher. Be assured, Russia is not the only country to vent as the figures in that report showed. Massive methane spikes were observed all over Africa, the Middle East, and Asia showing that the GHG intensities generated for many oil producing countries could be much higher than the values reported by Masnadi et al. More problematically, as researchers discovered in the unintended consequences of antiflaring policies—and measures for mitigation, it appears that some nations are simply venting methane from their oil fields to avoid satellite detection of their flaring. As the authors put it:
viewed through the lens of the multitask problem, these facts suggest that state-owned Türkmengaz, the field’s operator, had been systematically venting natural gas rather than flaring it to evade detection.
Recognize that major Canadian crude competitors like Venezuela and Iran are under international sanctions that limit their oil industries. Venting to avoid detection by international monitoring mechanisms would be a realistic concern with these countries. In other locations, like Libya, Iraq and North Africa, political instability has left many oil fields in the control of non-state actors who may choose to vent to avoid detection (or simply because they do not care about the consequences).
So while it is clear that Canada has a ways to go to reduce the GHG intensity of our crude, the truth of the matter is that Canada may really be in the middle of the pack when it comes to the GHG intensity of our crude. Not only that but we now know that improvements in recent years have had the effect of lowering our GHG intensities. As reported recently:
A new report concludes that when Alberta specific data is used, upstream GHG intensity numbers of oil sands production pathways are 14 to 35 per cent lower than previously published in Masnadi et al.
If the Canadian column dropped by 35% then the lower error bar would overlap pretty much every other country on the graph (except perhaps Ghana, Saudi Arabia and Denmark). Canada would be not be statistically worse than 94% of world producers.
To conclude, I am not making excuses for the Canadian oil industry and do not deny that Canada needs to continue to work to reduce the GHG intensity of the Canadian oil industry if we want to retain a market share as we wean the planet off fossil fuels. But I do believe that if activists are going to attack the Canadian oil industry they should understand what the science really says about the relative GHG intensity of Canadian crudes.
The data is not out there to support the claim that Canadian oil is uniquely bad and deserving of particular opprobrium as has been suggested by activists. Rather, the most recent peer-reviewed research that attempts to quantify global GHG intensities of crude oil producers indicates that our GHG intensities are comparable to our peers and likely a lot lower than some of our most important competitors (once methane emissions are considered in the analysis). Put simply, if you are going to make an argument against Canadian oil, don’t misrepresent what the scientific research says about this topic.
I have reworded the paragraph on venting and flaring to correct an earlier version which did not correctly note that venting estimates are incorporated in the development of the confidence intervals. I thank Dr. Yeh (who contacted me on Twitter) who pointed out my earlier incorrect wording.