This study uses an exploratory analysis protocol with seven major steps and a preliminary step (i.e. Step 0). These steps are depicted in Figure 1 and are described below.
Step 0. Decision Framing
This step involves defining the policy system and identifying the objectives and constraints of the decision-making process. The conceptualisation of the policy system casts the decision problem in terms of four key elements: the transitions model, the uncertainties, the levers, and the performance targets. The transitions model describes the structure and boundaries of the policy system. The uncertainties are external factors that affect the decision process. The levers are policy instruments that policymakers can use to influence the system. The performance targets are the criteria for evaluating the decision outcomes (i.e. policy objectives). This step sets the scope for the rest of the analysis.
The conceptualisation and contextualisation of the policy system and its boundaries were selected based on previous studies [2–7] and are briefly explained as follows.
Policy system structure: This study used a modified configuration of the latest version (v3.3.1) of the Energy Policy Simulator (EPS) model for Mexico, developed by Energy Innovation LLC [8]. The EPS is an open-source and peer-reviewed system dynamics (SD) computer model that quantifies the plausible effects of energy and climate policies on GHG emissions at a national or regional level. SD is a methodology that aids analysts and decision-makers in mapping the interaction of multiple variables in complex systems and explore their behaviour in time by the use of simulation models [9]. The EPS allows users to control multiple parameters that affect energy use and production, as well as GHG emissions [10]. A public web-based version of the original model may be available at https://energypolicy.solutions/.
Although the EPS model integrates a myriad of variables for the energy system (over 380,000), the modelling approach adopted to model technological progress was recently identified as a key limitation for policy analysis (e.g. technological knowledge stock was not included) [4]. The EPS model uses a combination of exogenous and endogenous learning approaches to estimating technology cost developments depending on the maturity level of each technology [8]. In the case of endogenous learning, the EPS model adopts the one-factor learning curve (1FLC) approach, being installed capacity the only driver for future cost developments in the case of wind and solar PV. Following [4], we extended the approach by explicitly including technological experience and knowledge stock as the sources of learning based on a comprehensive analysis of how technological learning is modelled for energy policy analysis [7]. Thus, modifications to the EPS-Mexico model were implemented following [4] and learning rates were estimated based on [6,7] with data from [3]. The model was implemented in Vensim DSS, and simulations were run for the period from 2020 to 2050. The modified version of the EPS-Mexico model is available on Supplementary Files as part of this protocol (epsmexico3.3.1.1EMA.zip).
Deep uncertainty characterisation: Potential uncertainties relevant to our analysis and estimated initial values were identified through literature review and model analysis. Parametric uncertainty (i.e. range of possible values for uncertain parameters) was selected based on the literature review and is presented using the deep uncertainty characterisation matrix proposed in [2]. Additionally, to the three possible locations of uncertainty initially proposed in [2] (i.e. model structure, future conditions, and outcomes of interest), we have included policy planning elements (i.e. policy objectives and instruments) for completeness in the decision framing and experiment setting steps for conducting the EMA process. A total of 33 external factors, 43 policy instruments, 19 outcomes of interest, and 8 policy objectives are included for EMA, using the modified version of the EPS model as a transition pathways generator. The deep uncertainty characterisation matrix is available on Supplementary Files as part of this protocol (EMADeepUncertaintyMatrix.xlsx).
We used the EMA Workbench [11] to conduct the exploration process. EMA Workbench is an open source Python library aimed at providing support for designing experiments, performing the experiments, and analysing the results [11].
Step 1. Policy Alternatives Generation
This step involves identifying or generating policy alternatives that have the potential to meet the policy objectives and constraints defined in the decision framing step. A policy alternative represents a mix of policy instruments with specific values and implementation schedules. There are three general approaches for generating policy alternatives and therefore conducting the exploration process: framed exploration, open exploration, and directed search. Framed exploration uses pre-specified policy instrument mixes based on policymakers’ interests. Open exploration and directed search explore alternative policy mixes not pre-defined by decision-makers [11]. This protocol focuses on open exploration and directed search.
In open exploration, sampling techniques (e.g., Latin hypercube, full factorial, partial factorial, or Monte Carlo) are used to generate policy alternatives. In this study, the decision space (i.e. policy alternatives) was sampled using Latin hypercube sampling (LHS) for the open exploration process (Step 1.1). LHS is more beneficial than other sample techniques (e.g. Monte Carlo, full factorial design) for long running models when computational resources are limited [12]. LHS forces the specified sample size to cover the whole experimental space (i.e. uncertainty space and decision space) by producing a sample that is random but relatively uniformly distributed over each dimension [12].
In directed search, optimisation techniques are used to generate a Pareto optimal set of high-performing policy alternatives (relative to a BAU scenario) [13]. This study uses a multi-objective evolutionary algorithm (MOEA) called non-dominated sorting generic algorithm II (NSGA-II) [14] to perform global optimisation and find high-performing policy alternatives for the directed search process (Step 1.1). MOEAs mimic natural evolution processes to iteratively evaluate strategies across multiple objectives until the best candidates are found. However, genetic algorithms incorporate stochasticity in their search process. As a result, the outcome of a single run may be influenced by randomness. To account for this, it is recommended to perform multiple runs with different random seeds and aggregate the results to obtain a more robust set of solutions (Step 1.2). We used the ε-NSGAII [15] to combine the results across the seeds for the optimisation runs into a single comprehensive set using an ε-nondominated sort. In addition, when utilising MOEAs, it is crucial to carefully monitor their convergence towards the optimal solutions. To achieve this, various metrics can be employed. In our study, we utilised epsilon progress, generational distance, epsilon indicator, inverted generational distance, and spacing to track convergence across different seeds used for optimisation. For details on these metrics see e.g.,[16] and [17] for distance and additive ε -indicator; [18] for spacing; and [19] for ε -progress.
Step 2. Experiment Setting
In this step, we specify the performance metrics (or outcomes of interest) in addition to the policy objectives selected in Step 0. Performance metrics differ from policy objectives (i.e. performance targets) is that the former consider behaviour over time (i.e. time series outcomes) while the latter focus on values at a particular time (i.e. single values outcomes). We then curate the policy alternatives and define the number of scenarios for the exploration.
Step 3. Performing Experiments
Using the modified EPS-Mexico model as a scenario generator, we generate a series of prospective transition pathways based on the decision framing and experiment setting discussed in the previous sections. A scenario represents a point in the uncertainty space, while a policy alternative represents a point in the decision space. The combination of a scenario and a policy alternative is called an experiment (or transition pathway).
Step 4. Policy Alternatives Analysis
This step involves examining the policy alternatives that will be evaluated using the policy system model. Our goal is to gain a deeper understanding of these policy alternatives, identify new ones, and modify or combine them as needed. We may also organise them into different timelines and select the ‘best’ options.
Step 5. Performance Metrics Analysis
This step explores the performance of the policy alternatives in the face of uncertainties. We use exploratory modelling to conduct an exploratory analysis of a wide range of assumptions and scenarios. Our goal is to understand how each policy alternative would perform in terms of the outcomes of interest under various conditions. This analysis will provide valuable insights into the robustness of the policy alternatives.
Steps 6 & 7. Medium-term (to 2030) and Long-term (to 2050) Policy
The analysis of extensive simulation results using the modified EPS-Mexico model as a transitions generator (i.e. computational experiments) was conducted using an exploratory thinking approach. By adopting an exploratory approach, more realistic insights may be obtained using statistical techniques (e.g. envelop plotting and Kernel Density Estimates (KDE)) and data-mining techniques (e.g. feature scoring and scenario discovery) over numerous experiments [20].
Based on [2], an exploratory analysis includes three analytical sub-processes: robustness analysis, vulnerability analysis, and trade-off analysis. We have included two additional analytical sub-processes (i.e. policy objectives analysis and feature scoring) to explore the effects of policy alternatives on achieving medium and long-term policy objectives set by policymakers.
Step 6.1 & 7.1 Policy Objectives Analysis
In this step, we present a visualisation of the policy alternatives in relation to the policy objectives. This provides a clear and concise overview of how each alternative aligns with the desired outcomes and help inform decision-making.
Step 6.2 & 7.2 Feature Scoring
This step focuses on understanding what policy instruments may have a stronger influence in meeting policy objectives under conditions of deep uncertainty. In this study, we implemented a random forest-based feature scoring approach [11]. Feature scoring is a family of techniques commonly used in machine learning for testing the effect that different regressors have on the target variable. This approach provides valuable insights into the relative importance of different policy instruments in achieving the desired outcomes.
Step 6.3 & 7.3 Robustness Analysis
Faced with deep uncertainty, policy decisions may be made using the principle of satisficing rather than optimising [21]. Robustness analysis is performed based on the satisficing measure adopted from [22]. Thus, robustness is defined in this study as the fraction of sampled scenarios (i.e. uncertainty space) in which a policy alternative meets policymakers’ performance requirements in one or more objectives. Policy alternatives can be ranked based on the performance thresholds established by policymakers to ‘accept’ a robust solution. When a policy alternative is ranked with 100%, the alternative satisfies the policy objective threshold in all plausible scenarios. On the other hand, a rank of 0% is assigned when a policy alternative cannot meet the established performance criteria in all plausible scenarios. The aim of using a robustness metric is to identify policy mixes that are both high-performing in terms of policy objectives and are relatively insensitive to future external changes (i.e. different scenarios) [1].
Step 6.4 & 7.4 Vulnerability Analysis
This step focuses on identifying regions of the large, multi-dimensional uncertainty space where robust policy alternatives are vulnerable to poor performance. Poor performance is defined in this study as situations when policy alternatives cannot meet the performance threshold set by policymakers (i.e. policy objectives). We used the patient induction method (PRIM) [23] as the algorithm to conduct scenario discovery [24]. PRIM is a factor mapping approach aiming at identifying sensitive ranges of uncertainties that are likely to cause a particular behaviour. It works by peeling away layers of the uncertainty space by constraining one dimension at a time, then evaluating how well those constraints capture the points of interest based on two metrics: coverage (the fraction of poor-performing scenarios that fall within the box) and density (the fraction of scenarios within the box that perform poorly) [13,22].
Step 6.5 & 7.5 Trade-off Analysis
Advanced visualisation techniques are required to communicate policy-relevant insights and to provide a clear understanding of trade-offs and potential consequences of diverse policy alternatives [2]. Thus, to present the performance of different policy alternatives, we use a three-dimensional scatter plot (including the full Pareto set in relation to the objective space with specific thresholds), a parallel axis plot (to show the robustness ranking of policy alternatives according to each policy objective), and Kernel Density Estimates (KDE) for non-constrained policy objectives (i.e. policy mix cost in this case).
Steps to reproduce the study:
a. Create a local folder with the name ema_analysis.
b. Download the compressed version of the model (i.e. zip file) named epsmexico3.3.1.1EMA.zip, available on Supplementary Files.
c. Unzip the model folder into the ema_analysis folder using a zip/tar files unpacking software (e.g. 7-Zip, WinZip). Make sure the name of the model folder is eps-mexico-3.3.1.1_EMA.
d. Download the Jupyter notebooks openexploration.ipynb and directedsearch.ipynb, available on Supplementary Files. Place these files into the ema_analysisfolder.
e. Download the python file functions.py, available on Supplementary Files. Place this file into the ema_analysis folder.
f. Create the following folders within the ema_analysis folder: archives, Figures_DS, Figures_OE.
g. Open Jupyter Notebook and navigate to the ema_analysis folder.
h. Run the openexploration.ipynbfile for an open exploration analysis and the directedsearch.ipynb for a directed search analysis.