Documents with broad relevance to climate change are retrieved from citation databases using a query adapted from an overview of climate change research (Grieneisen and Zhang 2011). A typology of climate policies is developed in order to categorise research on climate policy. A random sample of potentially relevant documents is screened by hand and labelled according to whether the documents discuss climate policy instruments or not, the type of policy instrument discussed, the sector in which the policy intervenes, the country in which the policy is adopted or is proposed, and the type of evidence provided. A machine learning classifier is trained to predict inclusion/exclusion, and multilabel machine learning classifiers are trained to predict policy type, evidence type, and sector. Named entity recognition is used to extract place names from study titles and abstracts. Active learning is used to identify additional relevant documents to screen and code. Machine learning classifiers are validated on the hand-coded data using a nested cross-validation approach and used to make predictions, with uncertainty, about the inclusion, policy types, evidence types, and sectors of the remaining uncoded documents.
Searching for Articles
A broad search query focusing on the whole of climate change literature was adapted from a bibliometric study of climate change research (Grieneisen and Zhang 2011). Additional changes were made to better capture research on adaptation, as well as on specific climate policies, leading to the query included in Annex B. We expect the query to capture the vast majority of literature which explicitly deals with climate change in general, and the mitigation and adaptation of climate change in particular. Documents covering policies with climate-relevant outcomes (e.g. a congestion charge) may not be returned by our query; these studies are not included in our map unless they make an explicit reference to climate change or greenhouse gases, or unless the sole policy goal is mitigating climate change (e.g. carbon taxation, which is included as a search term).
We search the following citation indices from the Web of Science Core Collection:
- Science Citation Index Expanded (SCI-EXPANDED) --1900-present
- Social Sciences Citation Index (SSCI) --1900-present
- Arts & Humanities Citation Index (A&HCI) --1975-present
- Conference Proceedings Citation Index- Science (CPCI-S) --1990-present
- Conference Proceedings Citation Index- Social Science & Humanities (CPCI-SSH) --1990-present
- Emerging Sources Citation Index (ESCI) --2015-present
We include all documents that discuss a policy instrument, as defined in our typology, that is either presented in the context of mitigating climate change or reducing greenhouse gas emissions, or has the unambiguous objective of reducing greenhouse gas emissions. Thus studies discussing carbon taxation would be included regardless of whether the document mentions climate change or greenhouse gas emissions, as carbon taxes are only used to pursue climate policy objectives. Schemes to support renewable energy are therefore not included unless climate change or greenhouse gas emissions are mentioned in the abstract, as such schemes may be deployed for other objectives such as energy security, or local air pollution. This strategy is pursued in order to prevent ambiguity where the goals of a policy may not be clear from the abstract, although we recognise that we therefore include only a subset of all climate-relevant policies and selection of studies depends on wording in the article rather than the substance of the policy. All studies with abstracts available in English will be considered.
Sample selection and machine learning assistance
Our search queries return over 600,000 studies, which cannot all be screened manually. We therefore screen a sample of all potentially relevant documents by hand and use a machine-learning classifier to predict (with quantified uncertainty) which of the remaining documents would be included (see below for further details). Because relevant studies will constitute a small minority of the total, we will employ an active learning strategy to mitigate the class imbalance problem. This means that in addition to samples drawn at random from the population of documents, some samples to be screened will be selected on the basis of having a higher probability of being relevant. Only representative samples will be used for validation.
Manual screening process
Manual screening will be conducted using NACSOS coding platform (M. Callaghan et al. 2020). In the initial rounds of coding, documents will be screened by all coders and conflicts will be discussed. Documents will be screened by hand at the title and abstract level by pairs of coders, using the coding scheme defined below. All inconsistent codes will be discussed by the coder pair, and if agreement cannot be found, in regular plenary meetings.
Included documents will be double coded, based on the titles and abstracts by the same pairs of coders, using the same process for finding and resolving conflicting codes. The following categories will be coded. In all category types, coders can apply one or more labels:
This category is used to identify if a document relates to Mitigation or Adaptation.
- Mitigation refers to any human activity that has the effect of reducing concentrations of greenhouse gases (GHG) including:
- Emissions reductions, through the promotion of renewable energy, green technology, energy reductions or behavioural change, or the restriction of GHG polluting technologies, activities or behaviours
- GHG sink enhancements
- GHG removal
- Adaptation refers to any human activity designed to reduce the impacts of climate change on human or natural systems, including solar radiation management and other geoengineering approaches which do not target GHGs
Governance and Policy Categories
This category is used to identify the kind of policy or government response that the article aims to address.
- Not Policy Related refers to those articles that do not contain policies responding to climate change. This includes documents presenting results which are relevant for policymakers, or documents discussing technologies or industrial processes that may lead to emissions reductions, as long as they are discussed absent of government policy.
- Analysis of a Concrete Policy refers to those articles that discuss either specific enacted policies or legislative proposals.
- Analysis of Generic Policy refers to those articles that discuss generic proposals that do not exist outside of the paper.
- Broader Governance and Policy refers to those articles that do not describe climate policy instruments but are generally relevant to climate governance and policy. In practice, any documents mentioning policy or governance without detailing instruments within the typology are included here.
Documents are tagged with a hierarchical typology of policy instruments made up of 3 levels provided in Annex A. We code starting from the highest level instrument type, and specify further where possible.
Documents are labelled with the jurisdictional level at which a particular policy is implemented or suggested.
- Sub-national regional
Documents are labelled with mitigation sectors the policy/ study applies to, using the sectors defined in IPCC WGIII AR5 (IPCC 2014) (in AR6 waste and industry are merged).
Documents are labelled according to whether they analyse the effects or process of an already existing policy, or look towards future effects of actual or potential policies.
- Ex-post includes all studies looking back at the effects of a policy
- Ex-ante includes all studies which anticipate the effects of a policy, unless
- Ex-ante (IAM) includes studies using integrated assessment models to look at the effects of policies
Documents are labelled according to whether they provide qualitative or quantitative evidence on the policies discussed
Where a specific location is mentioned, we label the country the location is in. For example, a study looking at climate policy in the city of Berlin is coded as Germany.
Where a specific policy is mentioned, this must be noted under this category.
Where documents have the response type Mitigation and the governance and policy categories Analysis of a concrete policy or Analysis of a generic policy they will be considered INCLUDED. All other studies will be considered EXCLUDED.
Machine-learning-assisted screening and coding
Motivated by advances in the field of Natural Language Processing (NLP) and the rapid expansion of scientific literature, machine-learning has begun to be used to automate study identification in systematic review and systematic mapping pipelines(O’Mara-Eves et al. 2015), as well as to automate the coding of studies(M. Callaghan et al. 2021; Porciello et al. 2020). In this project we will employ supervised learning for both study identification (using a binary classifier) and coding (using a multilabel classifier for each coding level), using the studies coded by hand as training and validation sets. We will use a nested cross-validation procedure to optimize hyperparameters (settings which must be provided to machine learning models and which can affect performance) and measure the accuracy of our classifiers, using this information - in combination with the predictions made, to identify and classify, with quantified uncertainty, the relevant studies in our database. We will test a variety of models, including simpler support vector machines(Chang and Lin 2011) as well as deep learning approaches based on BERT(Devlin et al. 2019), including a BERT model that has undergone additional pre-training for a corpus of documents related to climate change(Webersinke et al. 2021). For geographical coding, we will employ pre-trained geocoding models(Halterman 2017), and validate these against the human-coded documents.
The extent to which machine learning classifiers can accurately reproduce all parts of the coding scheme is unknown a priori. We will therefore report all validation results and the implications of these for uncertainty bands. Foregrounded results will be limited to the category types which can be well reproduced using machine-learning.
Mapping and analysis
We will map the number of studies on each type of climate policy, with each type of evidence, for each country, and regional regional group of countries. For context, we will contrast this information with the number and type of climate policies enacted in each country, using data from climate policy databases (Grantham Research Institute on Climate Change and the Environment and Sabin Center for Climate Change Law 2022; IEA 2022; New Climate Institute 2020), as well as the data on the emissions, and emissions trends in those countries (Lamb et al. 2021).
Annex A: A typology of climate policy instrument types
Here we present the hierarchical typology of policy instrument types for climate mitigation used in our mapping project. This typology was developed iteratively by the protocol authors during an initial review of studies, and builds on and extends the typologies presented by the IEA, the Climate Laws of the World Database, and the New Climate Institute (Grantham Research Institute on Climate Change and the Environment and Sabin Center for Climate Change Law 2022; IEA 2022; New Climate Institute 2020). The typology is a functional one, in that it describes how a policy instrument type does what it intends. In the top level of the hierarchy we distinguish between
- Economic Instruments, where governments spend money to achieve mitigation goals directly, or sets incentives to promote green behaviour or penalise greenhouse-gas-intensive behaviour.
- Regulatory Instruments, where governments make rules, which mandate the actions which actors must or must not do, or the ways in which those actions should be performed.
- Information, Education and Training, describes policies which define whether and how governments should provide or collect information, including the provision or definition of educational or training.
- Governance, Strategies, and Targets, refers to policies in which governments design institutions or create plans in order to direct, coordinate, or guide action on mitigation.
- Agreements, refer to where governments enter into agreements with other governments or other actors that define (mutual) intentions or commitments to decarbonise.
The 2nd level of the hierarchy contains 21 further policy categories, while the 3rd level contains 48 policy categories. These are described in full in the supplementary data (sheet “MCC typology”) and shown in figure 1.
The following sections compare the typology we present to the existing published typologies, in an effort to explain where and why changes have been made, and to maintain interoperability between typology types.
New Climate Institute Climate Policy Database
The Climate Policy Database, compiled by the New Climate Institute, is a database of 5,207 climate policies categorised according to a structure based on that used by the IEA (New Climate Institute 2020). 48 individual policy instruments are grouped into 9 instrument categories; 6 sub-categories exist between these layers. Each of these sub-categories is linked to an instrument category, but not every policy instrument is linked to a sub-category. In the supplementary data (sheet “New Climate Institute”) we link each of the 48 policy instruments to one of our level III (most detailed) policy instruments where possible, or to a level II instrument. All of the remaining level III instruments from our scheme which were not linked are linked to a NCI category at the most granular level possible. Figure 2 visualises the links between the two systems as a network, where each node is a category, and nodes are linked according to hierarchical structure within each schema, and where they have been deemed equivalent as defined above.
The two typologies have some broad structural similarities, shown by the clusters of nodes of both clusters. However, the fact that the majority of the network is linked demonstrates that there are some instruments which go between the broader categories of each schema. For example, labelling requirements link the regulatory and information categories which exist in both schemas because although they deal with information, and relate to the information category in NCI, they are a form of regulatory instrument in our typology because they compel actors to behave in a certain way. The functional nature of our typology deems that it is primarily how an instrument does what it does which decides its place in the typology. This distinction also leads to differing levels of specificity between typologies. NCI includes 4 policy instruments under the “Codes and standards” sub-category: “Building codes and standards”, “Product Standards”, “Sectoral Standards”, and “Vehicle fuel-economy and emissions standards”. These instruments refer primarily to what the standard applies to, rather than how it operates. In contrast, our typology differentiates between production standards, product specification standards, intensity standards, and bans and moratoria.
Climate Laws of the World
The Climate Laws of the World project is a database of climate legislation and policies collected by the Grantham Research Institute at LSE and the Sabin Center at Columbia Law School (Grantham Research Institute on Climate Change and the Environment and Sabin Center for Climate Change Law 2022). The typology contains 24 policy instruments which are grouped into 5 broader categories. With a smaller number of policy types, the schema is in many areas less specific than other typologies (e.g. there is a single category for taxes and a single category for subsidies), although there is a greater focus on governance. We map the typology to our typology in the same way as with NCI in the supplementary data (sheet “Climate Laws or the World”) and visualise it according to the same principle as in Figure 3.
Annex B: Search query
(TS=( "climat* change*" OR "climate crisis" OR "warming climat*" OR "changing climat*" OR "climat* warming" OR "chang* in climat*" OR "climate emergency" OR "global warming" OR "greenhouse effect" OR "greenhouse gas*" OR "changes of the climate" OR "changes in the climate" ) OR TS=( "climat* variability" OR "climat* dynamic*" OR "climat* proxies" OR "climat* proxy" OR "climat* sensitivity" OR "coupled ocean-climat*" OR "early climat*" OR "future climat*" OR "historical climat*" OR "past climat*" OR "climat* shift*" OR "shift* climat*" OR "shift in climat*" OR dendroclimatolog* OR "radiative forcing" OR "transient climate response" OR ( climat* NEAR/5 scenario$ ) ) OR TS=( ( CO2 OR "carbon dioxide" OR methane OR CH4 OR N2O OR "nitrous oxide" OR ( fluorinated NEAR/2 gas* ) OR hydrofluorocarbons ) AND ( climat* ) ) OR TS=( ( "carbon cycl*" OR "carbon cycles" OR "carbon cycling" OR "carbon flux*" ) AND ( climat* OR atmospher* ) ) OR TS=( ("carbon budget*") AND (climat* OR atmospher* OR emission*) ) OR ( TS=( "global climate model" OR "regional climate model" OR "local climate model" ) NOT WC="Astronomy & Astrophysics" ) OR TS=( "sequestration of carbon" OR "sequester* carbon" OR "sequestration of CO2" OR "sequester* CO2" OR "CO2 capture" OR "CO2 storage" OR "CO2 sequester*" OR "CO2 sequestration" OR "CO2 sink*" OR "captur* of carbon dioxide" OR "captur* of CO2" OR "carbon capture" OR "carbon sequestration" OR "carbon storage" OR "carbon sink" ) OR TS=( ( ( "greenhouse gas" OR GHG OR carbon OR CO2 OR CH4 OR methane OR N2O OR "nitrous oxide" OR "fluorinated gases" OR "f-gas" OR hydrofluorocarbon ) NEAR/2 ( emission$ OR footprint$ OR atmospher* ) ) NOT nanotube* ) OR TS=( "UNFCCC" OR "Kyoto Protocol" OR "climat* policies" OR "climat* policy" OR IPCC OR "clean development mechanism" ) OR (TS="Paris Agreement" AND PY=(2014-2021) ) OR TS = ( ( emission$ OR CO2 OR carbon OR "greenhouse gas*" OR GHG ) NEAR/2 ( trade OR trading ) ) OR TS = ( offset* NEAR/2 ( carbon OR CO2 OR climate OR "greenhouse gas" OR GHG ) ) OR TS =( ( "greenhouse gas*" OR GHG OR carbon OR CO2 OR emission$ ) NEAR/2 pric* ) OR TS =( ( "greenhouse gas*" OR GHG OR carbon OR CO2 OR emission$ ) NEAR/2 ( tax OR taxes OR taxing ) ) OR TS =( "decarboni*" NEAR/2 ( energy OR power OR transport * OR electricity OR cement OR building OR industry OR industries OR sector OR aviation OR rail ) ) OR TS =( ( climat* OR "greenhouse gas" OR CO2 OR carbon OR GHG ) NEAR/2 neutral* ) OR TS =( ( climate OR mitigation OR GHG$ OR "greenhouse gas*" OR carbon OR CO2 OR methane OR CH4 OR "nitrous oxide" OR N2O OR "emission$ reduction" ) NEAR/2 ( "policy" OR "policies" ) ) OR TS = ( ( climat* OR CO2 OR carbon OR "greenhouse gas" OR GHG ) NEAR/2 legislati* ) OR TS=( "CO2 abatement" OR "CO2 mitigation" OR "GHG mitigation" OR "cap and trade" OR "carbon mitigation" OR "climat* mitigation" OR "emission* mitigation" OR "low carbon" OR "low-carbon" OR "GHG mitigation" OR "greenhouse gas mitigation" ) OR TS=( ( "solar radiation management" OR "solar geoengineering" OR "aerosol geoengineering" OR "albedo modification" OR "albedo enhancement" ) OR ( engineering NEAR/2 climat* ) OR ( geoengineering NEAR/2 climat* ) ) OR TS=( "climate impact*" ) OR TS=( "Ocean acidification" OR "sea level rise" OR "CO2 fertili?ation" OR "carbon dioxide fertli?ation" ) OR TS=( ( "snow cover" OR "extreme heat" OR "extreme heat" OR "extreme temperature*" ) AND "climate" ) OR TS=( "extreme precipitation" AND ( "climat*" OR "trend*" ) ) OR TS=( "precipitation" AND "Climat*" AND "long term trend*" ) OR TS=( ("cyclone*" OR "hurricane*") AND climate ) OR TS =( climate NEAR/2 ( adapt* OR risk$ OR resilien* ) ) )