Design and Methods
This study is a prospective, Delphi consensus-seeking exercise, mixed, iterative survey of internal and external experts, patients and other stakeholders. A Delphi consensus methodology will be used to survey a panel of experts who as individual participants will fill the online survey in (from now on panelists). Panelists fulfil the criteria of balanced representativity (age, sex, location, patients/relatives of, medical area of expertise Primary/Specialized, translational research, media/journalists, Health policy/payers, …)
We aim to sample approximately 100 participants including the Chairs, speakers and Working Group panelists of the 1st WHO webinar entitled “Expanding our understanding of Post COVID-19 condition”, held on February 9th, 2021. This webinar was attended by 42 participants and was reasonably balanced by age (no formal stats), sex (42.9% female), and location. Importantly, it also included patients.
The primary users of a clinical case definition for the Post COVID-19 condition will include: patients, relatives and caretakers, clinicians, researchers, advocacy groups, policy-makers, health and disability insurance, and media. We will therefore aim to have a diverse sample of healthcare workers with expertise in clinical care, quality improvement and/or research. Eventually, the Guideline Development Group and other WHO panels might be invited to participate, as well as individual experts and patient representatives, with an emphasis to enrich female, young and/or those in LMIC panelists. Some groups will be expanded considering broad representativeness. There are no specific exclusion criteria for panelists.
The Delphi method is a structured communication technique originally developed as a systematic, interactive forecasting method which relies on a panel of experts.[4,5]. Essentially, a Delphi process is one in which a panel of experts answers a series of questionnaires over two or more rounds in an attempt to achieve consensus. The technique can also be adapted for use in face-to-face meetings (termed “mini-Delphi” or Estimate-Talk-Estimate (ETE)). Delphi has been widely used for research and has certain advantages over another structured forecasting approaches. Delphi is based on the principle that forecasts (or decisions) from a structured group of individuals are more accurate than those from unstructured groups. The panelists answer questionnaires in two or more rounds. After each round, a facilitator or change agent provides an anonymized summary of the panelists’ forecasts from the previous round as well as the reasons they provided for their judgments. Thus, experts are encouraged to revise their earlier answers in light of the replies of other members of their panel. It is believed that during this process the range of answers will decrease and the group will converge towards the "best" answer. Finally, the process is stopped after a predefined stop criterion (e.g., number of rounds, achievement of consensus, stability of results), and the mean or median scores of the final rounds determine the results. Special attention is paid to the formulation of the Delphi theses and the definition and selection of the panelists in order to avoid methodological weaknesses that could threaten the validity and reliability of the results. For this Delphi task, the COMET software will bus used.
COMET Initiative DelphiManager is a web-based system designed to facilitate the building and management of Delphi surveys.  From a simple Excel template, COMET creates a multi-round Delphi survey using the Setup Wizard. Panelists will rate each question using a 1-9 Likert scale with an “Unable to rate” option, should they feel that item is beyond their expertise. In order to avoid a participant selecting the “Unable to rate” column accidentally, it is colored differently by default. An example of what a participant might be shown in round 2 (of a Delphi with a single panel throughout) is given in Figure 2. This survey can then be managed in terms of emailing participants regarding missing data, calculating the feedback data for individual rounds and downloading all user and scoring data in standard CSV format.
Lists held by the Principal Investigator, and the research team members, will be used to identify eligible participants. Eligible participants will be invited to participate with a recruitment letter on-line to solicit participation and engagement, along with an explanation of the study objectives, instructions and outputs. Engagement and commitment will be sought prior to the first Round of the Delphi. The survey will then be conducted electronically with a link provided to engaged participants. Returned survey responses will be on a server at the University of Liverpool (Prof Paula Williamson, personal communication). The survey will contain listed options regarding domains/variables to consider in the definition, initially as broad and comprehensive as possible. The agreed domains/variables will be followed by a series of questions relating to these variables with eventual values/thresholds related to each. There will be no questions with participant identifiers or related to personal health information. Surveys will be anonymous.
At the DelphiManager Registration Page, all participants must register to take part in the survey from round 1. It also contains a consent checkbox in order to comply with GDPR legislation. All of these fields are mandatory by default. The designated administrator will allow panelists to log in and type their responses. Once the survey has been launched, the administrator will overview the main study results during the round as well as manage the process of moving from one round to the next using the Next Round Wizard. Information will include: Total participants registered; Total completed each round; Total not completed each round; A breakdown of the completed/not completed participants for each stakeholder group (if selected); and Total of any additional outcomes added to each round.
All questions will be evaluated on a 9-point scale, from one (1) to nine (9) and participants will be asked to choose the level of importance for each variable in the definition. Some reclassifying by label in categories 1-3, 4-6, 7-9 might be needed.
The first round of the Delphi will last 5 days, and participants will be sent 2 reminders to complete the on-line survey. The survey is iterative as the Delphi methodology requires multiple rounds of questioning until consensus is achieved. During the second round, the same survey will be used. Respondents will be informed of question results from the first-round questions as a single panel. The questions for which consensus is achieved will be removed from further rounds of the survey. In questions where consensus is not achieved participants will be asked to reconsider their answers for all questions in lieu of the panel’s anonymized results. Panel answers will be provided as numbers of respondents having chosen that answer. A maximum of three rounds of questions will be performed.
Working proposal with suggested methodology based on recent experience
For this exercise, the following steps will be considered:
1. Simple survey, with wide (global, full) respondent base to rating all domains (variables) that would be important to be considered for post-COVID diagnostic criteria. Would likely just confirm existing list, but important to consult widely.
2. Three (each in a different time zone) online consensus workshops x ~25 participants each (total n=75 to 100) to discuss and establish threshold criteria (likely to be more difficult to get agreement than the domains themselves). Participants would be invited, representative of key stakeholder groups and have knowledge of the condition. May want to ask a few (selected) people to give *very short* presentations of relevant evidence.
3. If any remaining unresolved aspects of the definition after the virtual workshops, run a (short 2-round) Delphi (via the COMET software) to seek final resolution.
4. (Optional) Put final agreed case definition out for a structured, short-response time public consultation.
Primary and Secondary Endpoints
The primary outcome will be a consensus on the importance of variable inclusion into the registry. ‘Consensus’ will be obtained on a question if 70% or more of the responses fall within the same response on a 9-point Likert scale. ‘Disagreement’ will occur if 35% or more of responses fall in both of the two extreme ranges of possible options on the Likert scale. All other combinations of panel answers will be considered ‘partial agreement’. For each question, consensus proportions will be considered based on the number of respondents (excluding the category “Not my area of expertise”). Therefore, the denominator for the consensus will include only participants with knowledge and expertise for that specific question. During each round, participants will have the opportunity to add comments for each item and add variables that are not mentioned (Round 1 only).
Statistical Methods/Data Analysis
All downloaded data from DelphiManager is presented in standard CSV format, allowing it to be easily imported into most statistical analysis tools. DelphiManager does produce the feedback in rounds 2 and 3 for particular study designs, but the data extraction facility allows the study team to undertake additional analysis as required.
Where there is a value in the DelphiManager Rating column that is something other than the Likert scale values of 1 to 9, the system is coded as follows: “-9”: the value allocated when an outcome has not been rated; “10”: the value allocated to the “Unable to rate” option.
Response rate in proportion will be calculated based on the number of returned surveys. The survey results will be used descriptively. Participant responses, including baseline and demographics characteristics (survey section I) will be analyzed using basic statistics such as mean (standard deviation), median (interquartile range), and range. Responses on all other domains will be analyzed in proportions and illustrated using histograms. Reliability will be measured using test-retest in 5-10 respondents, and measures of internal consistency will be analyzed using Cronbach’s alpha, to assess how well items of the same construct are related to one another.
Registration of panelists and the actual Delphi questionnaire are accessible via this link https://delphimanager.liv.ac.uk/DefiningPostCOVID/Delphi.
Sample Size and Power
The objective will be to obtain a diverse sample of panelists from all stakeholders, including each of the following categories: clinical experts, research experts, and patient representatives. Understanding that some panelists may be experts in more than one category, we will invite at least 20 experts from each of the three categories. This should lead to a sample size of approximately 75-100 participants and solidify the generalizability of the consensus findings. Allowing for a 10-15% non-response (or non-desire to participate) rate, and 10% drop-out rate, we will invite and engage at least 100 participants (TBD).
Consent to participant in the survey will be implied by answering and returning the survey. A statement explaining implied consent is found on the title page of the COMET survey (see survey Title page). No explicit consent will be obtained.
Confidentiality of Data
Results of the anonymous questions will be entered into a database document on a secured and locked computer. Only the study team will have access to the stored survey data.
Participants will not be identified personally in the survey, however some basic demographic descriptors will be collected. These will include any characteristics such as age, place of work and geography. In the data collection form, participants will be entered as consecutive numbers based on survey receipt and completion.
JBS at [email protected] will be responsible for collecting and storing the electronic surveys into the study database. The study database will be stored for a period of 7 years after completion of the report or publication, then permanently deleted.
This study is funded internally in WHO. There will be no payments to panelists. Cost of the COMET DelphiManager licenses (however many rounds, participants, items, timescale) is £850 plus VAT, a second Delphi study would be charged at £500 plus VAT, a third and every one thereafter at £300 plus VAT; however, as this enquiry is related to the WHO’s work on COVID-19, software would be made free of charge. (Prof Paula Williamson, personal communication). There are no other expected expenses.
A list of domains (and values)
An initial list of fourteen (14) domains was considered in a definition of post COVID-19 condition. These domains were later reduced/revamped from 14 to only 10, to be asked to panelists in an online text questionnaire, available in Appendix 2. Before the actual questions there is a page with instructions on how to fill in responses and the score method, and on the full Delphi procedures. All to be uploaded to DelphiManager.
The DelphiManager checklist is fulfilled, completed as a necessary step before starting the study