This protocol represents Part 3 of a 5-part series outlining the phases of methods for this initiative. Part 3 outlines the methods used to extract data on adaptation from documents (coding), as well as procedures for data quality assurance. See Figure 1. Figure 2 provides a summary of the number of articles included and excluded at different phases.
1.0 Scope
This data extraction protocol describes methods used to code adaptation information from a dataset of scientific articles. The protocol describing the screening and selection of articles in the dataset can be found here: DOI ***. A total of 2032 articles were retrieved from the screening stage and deemed potentially eligible for data extraction.
The bibliographic information for articles meeting inclusion criteria during screening were imported into the platform SysRev (sysrev.com). Given that initial screening was conducted on title and abstract only, an additional screening step was undertaken during this phase (data extraction) to ensure documents contained sufficient full-text information to extract relevant data. Thus, data extraction included two initial screening questions:
1. “Is the document relevant according to inclusion/exclusion criteria?” This question was used to exclude books, conference proceedings, and other document formats missed at the initial screening phase, and to verify relevance of borderline inclusion.
2. Is there sufficient information detailed in the full text (a minimum of half a page of content documenting an adaptation-related response). This question was used to screen out documents referring to relevant adaptation responses in their title or abstract, but including no tangible detail or documentation within the article itself.
2.0 Structure of coding teams and platform
Data extraction was undertaken within the SysRev on-line systematic review application. SysRev is a freeware application designed to allow web-based data systematic extraction from documents. We created an on-line data extraction form within SysRev to enter and curate extracted data. SysRev enables management of multiple coding of documents, identifies inter-coder conflicts, and links to full-text documents for rapid and standardized review.
Bibliographic information for all documents classified as relevant to inclusion criteria during screening were imported into SysRev. Given the substantial number of documents and global scope of the review, data extraction was undertaken by small teams of researchers based on regional and sector expertise. Papers were assigned to a primary topic OR region. While each document could be coded as relevant to multiple regions and multiple sectors, an individual document was assigned to a single region or sector to facilitate coding within distinct project teams. A total of 13 ‘projects’ were created, reflecting all regions (n=7) and sectors (n=7) listed in the IPCC AR6 WGII chapter outline (Table 2). Asia and Australasia were combined since the latter had a very small volume of literature. Some coders contributed to multiple projects. Documents were independently coded by at least two individuals.
3.0 Coder recruitment
Coder recruitment focused on global researchers with expertise in climate change adaptation and one or more of the sectoral or regional topics. The majority of coders had a PhD or higher, though highly specialized researchers with lesser degrees were accepted where relevant and for under-represented topics or regions. For regional sectors, the majority of coders are based in that region, or originate from the region. Coder recruitment was based on convenience recruiting, but prioritized global diversity to seek representation by gender, region, and expertise. Recruitment was based on snowballing via team networks and through social media.
Coders were expected to code a minimum of 50 documents, and were included as a co-author team member if this was achieved, and if their codes passed quality appraisal.
4.0 Training
We developed an on-line training manual for coders. The training included both contextual information on systematic review methodologies, as well as key details to guide data extraction, including a detailed codebook. Training of coders sought to expose coders to basic concepts of systematic evidence synthesis and assessment of confidence in evidence. The training manual also served to establish a consistent baseline for the concepts, vocabulary, and definitions used within GAMI, recognizing a wide range of often conflicting definitional uses for adaptation concepts. Sections within the training manual included:
1. About the GAMI initiative
2. Why systematic review? (including an introduction to elements of systematic review)
3. Why create an adaptation database?
4. The IPCC Risk Framework
5. Scope of the review
6. What has already been done?
7. An introduction to coding
8. Working with SysRev
9. Coding documents
10. Assessment of confidence in evidence
11. Tips for excellent coding
GAMI training for coders was originally developed in on-line course format using Eliademy, an e-learning platform to share and manage courses. The Eliademy service was discontinued within 2 months of commencing coding, however, so all course materials were converted to a 26-page training manual. A copy of the training manual can be found in the Supplementary Files.
5.0 Typology for data extraction
Data extraction was guided by an adaptation typology designed to characterize who is responding, what responses are being observed, what is the extent of the adaptation-related response, and are adaptation-related responses reducing vulnerability and/or risk? Coding of regional and sectoral foci within documents allows stratified analyses for individual sectors or regions.
Questions included both closed/restricted answer questions and open-ended narrative answer questions. The former facilitate quantitative categorical analysis (e.g. descriptive statistics, summarizing studies in ordered tables) and mapping of adaptation (breadth), while the latter facilitate contextual understanding of adaptation and qualitative analysis.
The data extraction strategy is designed to create a systematic database characterizing adaptation responses that can be used for multiple types of analyses rather than a single objective. Key analytical questions are summarized in Table 3. A detailed codebook for data extraction is included in the Supplementary Files.
6.0 Missing data and outcome reporting bias
There is likely to be substantial reporting bias given that many activities that reduce vulnerability and risk are not reported or not labelled as adaptations, particularly in the case of autonomous responses to climate risks. Given the conceptual complexity of the adaptation literature, there are currently no feasible options to overcome this reporting bias at the global scale.
For individual documents, there may be insufficient information to answer a question in the data extraction form. In this case, all coders will be asked to enter ‘no data’ to distinguish absence of evidence (‘no data’) from evidence of absence. Reporting of confidence in evidence and lack of information for key adaptation needs is a key goal of this initiative.
7.0 Assessment of confidence in evidence
Quality appraisal was undertaken on all documents/studies meeting inclusion criteria, and was part of the assessment of confidence in evidence. Critical appraisal was not used for article inclusion or exclusion since this review includes literature with a range of methods. Appraisal was thus conducted to fulfill the requirements of assessment of confidence in evidence. The appraisal is guided by components of the GRADE-CerQual (https://www.cerqual.org/) approach to evaluating confidence in evidence for qualitative data. We did not appraise or extract quantitative data. The following critical appraisal questions were included in the data extraction form:
A. Are there any major methodological limitations? E.g. Are methods sufficient to answer the research question, and are findings adequately and sufficiently substantiated by empirical data (qualitative or quantitative data)? Are there any major sources of bias in the data collection, analysis, or interpretation of results? Comments on methodological limitations.
B. Assessing coherence: Did the article provide sufficient information to answer all of your coding questions? Were there particular questions for which you felt that there was: 1) limited information or unclear evidence provided, 2) divergent results or outliers that made it hard to answer or that the authors seemed to ignore, or 3) the paper/document was not really directly relevant to the questions you were asking? This question will help us assess confidence in findings. Please highlight any of your answers that may be less reliable compared to others.
C. Assessing adequacy: Please comment on the quantity and quality of data upon which the findings in this article/document are based (e.g. sample size and/or depth of research). Did the article/document contain sufficient and adequate data (quantity and/or richness) for you to feel confident answering these questions? This question will help us assess confidence in findings. We are less confident about a finding when the underlying data only come from a small number of participants, locations, or settings, or in the case of case-studies do not contain sufficient detail/richness to make a meaningful assessment.
D. Assessing relevance: Are the results of this study relevant to a particular context only (e.g. a particular region, population, or context)? If so, describe the context within which these results are valid/relevant.
8.0 Quality assurance of coding
To enable cross-article comparisons, it is important that all coders follow the coding guidelines and answer all questions. We therefore conducted a quality assessment for each coder to identify those who had missed entries or skipped significant questions within the SysRev data extraction platform. Sixteen key questions were identified that had closed-option responses and no logical conditions (i.e., were not answered only if a previous question were true). Any coder who left >10% of these key questions blank was asked to complete their codes. Response rates were calculated using R. The code is available on GitHub: doi.org/10.5281/zenodo.4010763. Any coder who was unable or unwilling to complete their codes was deemed to have unreliable codes. To be included in the database, a document must have at least one set of reliable codes. In cases where a document did not have at least one set of reliable codes, we sought a third coder.
All coders were contacted at the end of initial coding to ask them to ensure completeness of all codes and to flag key areas of potential error. This included, for example, avoiding blank entries that should instead be listed as ‘not relevant’ or ‘no information’; ensuring that multiple relevant sectors and regions are recorded, regardless of project team; and avoiding exclusion of non-English language articles. Articles assigned to coders without relevant language abilities were reassigned to another coder with appropriate language skills.
9.0 Reconciliation of double codes
Over 100 volunteers coded more than 2,500 articles. 482 articles were excluded (book chapters, not human adaptation, etc.). At least two volunteers coded 2,177 articles (the remaining 16 articles were coded by a single reliable coder). To consolidate multiple responses into a single entry for each article, we used a script in R that followed a series of if/then statements. The full code and rationale are available on GitHub (doi.org/10.5281/zenodo.4010763). For open-ended questions that asked coders to provide quotes or evidence, all responses were compiled. For True/False questions, if either coder responded True, the answer was coded as True because these questions ask about the absence or presence of certain topics in each article, and it is more likely that one coder overlooked the presence of an item (gave a false negative) than that a coder imagined the presence of something not actually present (gave a false positive). For questions with multiple responses (e.g., hazards addressed), similar logic led us to take all responses because false negatives were more likely than false positives. In all cases, decisions were made to be conservative - to overestimate the degree and amount of adaptation being documented. Reconciliation stages were systematically biased to include rather than exclude, so as to retain the most detail possible.
A final database was compiled with a single line entry for each article. Authors and title names were used to double-check for duplicates within the database (duplicate entries were merged). And articles were assigned to IPCC regions based on the countries identified during coding. In most cases, these aligned with the GAMI-assigned regions, but some island states, for example, were assigned to different regions, and a few errors in regional assignment were corrected. The final database contains 1682 articles and 70 columns (70 data points for each article).