Abstract
There
are a number of challenges in developing a reliable and valid instrument on health
risk behavior of adolescents because adolescents are reluctant to disclose
negative health behavior. The Delphi method, supplemented by literature review
and grounded theory, was used to reach the consensus on health risk behavior and
its determinants. Qualitative questionnaires were sent to experts in round I. In
round II, an online questionnaire with major determinants in five point Likert
scale was sent to experts. In round III, a group of experts met, revised and
developed consensus on adolescents' health risk behavior and their major
determinants.
Round
I identified 117 major determinants. Altogether, 86 determinants were selected
from round I. In round II, this was reduced to 52 determinants with more than
70 percent consensus. Altogether 12 determinants, which scored more than 50
percent and were supported by literature and grounded theory, were added after consensus
development with experts in round III. Low response rate, identification and
selection of experts, experts' time constraints were major methodological
challenges whereas logistic management and time consuming nature of the method
were major non-methodological challenges in the Delphi process. These
challenges can be overcome or minimized.
Keywords: Delphi
method, health risk, behavior, determinants, challenges
Introduction
Human behavior is one of the important factors that lead to
ill health condition and disease. A
risk behavior is defined as a
lifestyle activity that places a person at increased risk of suffering, illness or injury. Behaviors that
results in negative health outcomes are considered as health risk behavior(HRB).
HRB of individuals may
be related to a person's social, cultural, religious, moral, and legal aspects
so it is difficult to generalize HRB across the culture. (Fenton, Johnson, Mcmanus, & Erens, 2001).
HRB in adolescents has been considered
as one of the important public health issue because it shapes adult behavior
and the consequences are costly for society (RR, Campbell, MacArthur, Gunnell,
& Hickman, 2012) . HRB behavior in adolescents may be due to
cognitive process such as brain development stages or due to biological process
such as hormonal changes in transition period from childhood to adulthood(Skaar, 2009) .
The major health risk behaviors among
adolescents are substance(cigarette, alcohol, drug) abuse and unsafe sex, all
of which may lead to seriously impaired lives and even premature deaths(Sustein, 2008) . US department of
health and human Service report (2016 )showed that 9 out of 10 smokers started
smoking at the age of 18 and 99 percent started at the age of 26 so, it is
important to correct the behavior among young people.
Risk taking behavior vary in-between
adolescents and adults. Adolescents have greater willingness to accept the
situation in which the likelihood of winning or losing is unknown and they have
higher tolerance for unknown when compared with adult(Tymula et al., 2012). Moreover, the
perception of adolescents’ risk taking by adult and adolescents themselves is
different as many adult consider adolescents’ risk taking as gaining a taste of
life and necessary to life, whereas adolescents consider their own risk taking
as an adventure and thrill (Karaman, 2007).
There are many challenges for
investigating health risk behaviors; how to validly and reliably measure and analyze
risk behavior are two important aspects in research related to health risk
behavior. The measurement challenge begins from the process of instrument
development. Guidelines in the instrument development literature are variable but
thirteen major steps conducted in planning, construction, qualitative
evaluation and validation phase may provide road map in the field of
occupational therapy. State purpose of
test and target groups, review literature on construct or variable of interest,
writing objectives and selecting items format, write pool items, content
validation, develop new or revise items, prepare instrument for first pilot
testing, run item analysis, revise and second pilot testing, second pilot
administration, repeat step 9-10, begin validation and continue validation are
13 major steps in instrument development (Benson, 1982).
Most adolescent health risk
behaviors are usually measured by self administered questionnaire and the
adolescents answer retrospectively. There are number of challenges in recalling
the health risk behavior. Some risk behaviors are very sensitive that
respondents do not want to report them because they believe that engaging in
such behaviors is socially undesirable(Berner, 2003). In many instances
adolescents under report or over report deliberately. For example in a smoking
behavior related survey, adolescents may under report the incidence of smoking
because they want to hide it from parents or teachers but on the other hand
they may over report it in peer led interview to show their wo/manhood. On the
other hand asking questions to adolescents on risk behavior itself is risky
undertaking because if the adolescent who hold positive or ambiguous attitude
towards that particular behavior will lead to an increase in the performance of
that particular behavior and individual with such questions behavior effects
can be protected by altering question target, time orientation or wording or by
providing warning about the impact of
questions (Fitzsimons & Moore, 2008). So, asking
questions to adolescents about risky behavior has potential to increase that
particular behaviors which is sometimes counterproductive for adolescents.
Different methods have been used by
different researchers for developing valid and reliable instrument in
adolescents’ health risk behavior survey. The Delphi method
is one of the methods proven to be an excellent tool in establishing face and
content validity of such instruments. In the past, Delphi techniques have been overlooked
due to its labor intensive nature but now in many educational and health research,
it is considered as an effective and efficient research tool(Colton & Hatcher, 2004). In the Delphi
method, evidence for content and face validity of the instrument are due to three major characteristics:
firstly, the results are due to group opinion which is more valid than single
person decision, secondly the process is based on expert opinion in each
iteration from local context and finally the first open qualitative round allows
to item generation and successive rounds allow review and judgment as to the
appropriateness of generated items(Keeney, Hasson, & McKenna, 2011) .
The purpose of this paper is to
highlight the results of a process to develop an instrument for measuring adolescents’
HRB and identifying major determinants using the Delphi method complemented by
literature review and grounded theory. Furthermore, the paper will reiterate
the Delphi process for establishing face and content validity of measurement
instrument. The paper will also discuss both methodological and non-methodological
challenges in the instrument development process.
Methods
and Methodology
Method and methodology includes study
design, Delphi participants, consensus development process and detail of Delphi
process.
Study
Design
The instrument development process in the
study involved the Delphi process
supplemented by literature review and grounded theory. The Delphi
process is considered as time consuming process due to the time required for data
collection. In general it requires 2-5 months for data collection but it can be
varied based on data collection methods such as paper based postal or email
based online form( Okoli & Pawlowski, 2004).
Literature review is used in initial
identification of dimensions and items in most quantitative research. Sometimes,
variables and items generated only through literature review might not be
suitable to local context. So, in this process, grounded theory was also used
to overcome the challenges due to stand alone use of Delphi process or
literature review for identification of HRB and major determinants. As part of
a grounded theory component, in depth interview of students in higher secondary
school was carried out to understand the local context on the health risk
behavior where students were probed to answer open ended questions. Ten
students from two different schools were interviewed in depth during a visit to
higher secondary school in Kathmandu and Lalitpur municipality. The interview
with students started with open ended questions such as what were the major
risk behaviors of adolescents? What are the major factors that accelerate or
reduce risk behaviors in adolescents? The interviews ranged from 45-90 minutes
duration. Interviews were recorded with prior permission from students. The
recordings were listened to several times by the author and major points
related to risk behavior and issues were noted. Later on, major HRB and
determinants were quantified, compared and merged to experts responses of
Delphi round II.
The Delphi technique helped to estimate the face and content
validity of the instrument. The Delphi method is one of the most used methods
in health research design using several communication rounds to get subject
matter experts’ view and consensus on the content ( Eberman & Cleary, 2011). During the
Delphi process anonymity is maintained in order to provide equal chance for
each expert to express views and ideas unbiased by the identities of other
experts in the first two rounds. In the third round, these experts meet and
establish consensus(Keeney, Hasson, & McKenna, 2011) . Although anonymity could
theoretically lead to a lack of accountability because responses may not traced
back to individual expert( Hanafin, 2004), experts were not
blinded to investigator in this study but are only requested to keep
confidentiality among group experts.
The
Delphi method can overcome the drawback of domination of expert focus group
discussion and conference by one or two experts if Delphi rounds are not
conducted face-to-face (i.e. by correspondence). There is no universal
guideline for Delphi method but classical Delphi method is still common in
health research, sometimes refined by replacing the postal round with email or
online form in order to speed up the response from experts. However, poor
internet services in the developing world would make this refinement not as fruitful as expected. The classical Delphi
method is characterized by anonymity, iteration, controlled feedback,
statistical group response and stability in response among expertise on a
specific issue(Hanafin, 2004). It is often used
to reach consensus among experts on the content and quality of measurement
instruments.
Participants,
Panelist or Experts
The
person with informed knowledge in subject area or specialists are considered as
an expert in Delphi round(Keeney, Hasson, & McKenna, 2011) . A good selection strategy should be followed in selection of
experts, and experts can be identified on the basis of their competency based
on knowledge and experience to generate
and evaluate themes in the area of analysis(Pereira & Alvim, 2015).The major
eligibility criteria for selection of participants are experience, knowledge
and interest in the area of study, ability to contribute and ability to review
initial opinions to achieve a group consensus(Araujo, Teixeira, Yaphe, & Correia de Sousa,
2016). Therefore, experts are generally
selected on the basis of subject area knowledge and experience, capacity and
knowledge to participate, time available to participate and effective
communication skills; however, experts with all of those qualities may not be able
to fully participate voluntarily in all round of Delphi process due to their
busy schedule(Skulmoski& Hartman, 2007). In one study, selection
procedure for experts was five steps process (Okoli& Pawlowski, 2004). Step I involved
identification of relevant discipline and skills, step II involved preparing
the list of experts in each discipline or skill, organization and academic
practice. In step III, experts were nominated based on list and nominated
experts were further asked to nominate other experts, step IV involved the
process to rank experts in each discipline and skills and step V involved
inviting experts to participate. The overall result and validity of the developed instrument revolves
round the experts' views and consensus so the expert selection process is
crucial process in Delphi. In one of the studies for development and content
validation of instrument to assess the nursing care product expert panel
composed of 15 nurses who had at least ten years of professional experience,
acted as teachers, preferably as leaders of research groups in management area,
certified by Scientific and Technological Development Council and nurse
managers from hospital(Cucolo & Perroca, 2015).
Experts
selection was done in four major steps in this study. The first step is
identification of institution and organization level factors related to
adolescents’ issues. Identification of experts within organizations who have
been working with or for adolescents using telephone inquiry was done in second
step. In third step, identified experts were asked for participation and after
their agreement to participate they were requested to nominate other experts whom
they felt worthy. Finally, grouping experts for ensuring balance participation
of experts in all related areas was done. The major institutions working for or
with adolescents such as non government organization, bilateral organization,
health institution, Higher Secondary Schools were selected.
Identified
experts were contacted first by phone. A Preliminary phone call or personal
contact to all prospective experts before sending email may be better choice in
the Delphi process to increase response rate(Hsu & Sandford, 2007). The phone call
informed the identified experts in brief about the context and objectives of
the study. They were asked for voluntary participation. After getting
preliminary approval for participation, explicit cover letter, open ended
questions on health risk behavior of adolescents along with extensive detail
instructions and demographic sheet were sent to them through email in round I of Delphi method.
Consensus Achievement
among Experts
Delphi process occurs in rounds
which allows individual to change their opinions. It also allows expert to
observe group response and indicating to each expert to observe their own
response in Delphi. It allows to express judgment using summary measures of the
full group response, giving more response than just a consensus statement but
in final round, Delphi panelist compares own view with group's view sometimes
group score may influence panelists and they may change their view basis of
group opinion rather than their own judgment.
Delphi
Process
Figure 1 showed three round of Delphi
process. Identified HRB and determinants in Delphi round II were supplemented
by few more items from literature review and grounded theory.
Figure
1
Delphi
process supplemented by literature review and grounded theory
|
Literature Review
|
|
Delphi
|
|
Round
I (117 HRB determinants) ; n=40
|
|
Round
II(86 HRB determinants) n=33
|
|
Grounded Theory
In depth interview with HSS students to understand
ground reality
|
|
12 HRB determinants
|
|
Round
II(52 HRB determinants)
|
|
Round
III consensus on 64 HRB
determinants; n=15
|
Round I. Four
subjective questions related to major health risk behavior among adolescents
were sent to 50 identified experts in different organizations via email along
with cover letter, invitation and
instruction letter. The experts were mostly national and local. Their
contact details were obtained from contacted respective organizations by using
telephone directory. After their approval to participate in telephone, questionnaire
with further details were sent. The experts were from three different types of
organizations as in table 1.
Table 1
Experts
invited from different organizations to participate in round I
Types of organization
|
Percentage of expert(n=50)
|
Types of participants
|
Non Governmental Organization
|
60%
|
Program managers/officers, youth
coordinators, Behavior change officer,
research officers
|
Higher secondary School
|
30%
|
Principals, teachers
|
Hospital/Health institution
|
10%
|
Psychologist, Physicians
|
Note.
n=number of experts invited to participate in round I of Delphi
|
||
In
round I, 50 experts were identified with male to female ratio of 1:1. Some
experts were also identified by notify experts endorsement. The response rate
of such previously notify experts by endorsed experts was high.
The four major questions asked to
experts were:
1. Adolescents'
health is one of our major concerns in Nepal. Would you please mention some of
major health problems of higher secondary school students(grade 11 &12) ?
2. What are the major health risk behaviors among
adolescents to cause major health problems in urban areas ?
3. What
are the major determinants of health risk behavior of adolescents in urban
areas? Request you to elaborate your answer in personal, family, school,
community/neighborhood and societal level
4. How
can we reduce major health risk behaviors among adolescents?
Open
ended question in the first round of Delphi study reduces the chances of
excluding items or issues that the researcher may have omitted(Nworie, 2011). First,
unstructured questionnaire provides experts for an open response and allow
relatively free scope to elaborate on the topic under investigation and it
increases the richness of data(Powell, 2003). Number of
experts is often decreases in subsequent rounds and dealing with the non
response rate is important to maintain number of experts in Delphi process(Hsu & Sandford, 2007).To increase the
response rate, the experts were requested to answer within 15 days and follow ups were done twice. The first follow
up was done within 10 days by telephone call and the second follow up was done both by
telephone and by email after 13 days of
request.
Round II. An online Questionnaire was
developed in Google form on the basis of major issues identified in round I. Altogether 86 HRB and
determinants were listed and experts or
panel members were asked to rate each of item on a 5-point likert scale (from strongly
agree, agree, neutral, disagree, strongly disagree). Altogether 40 responded of
round I were requested to participate in round II but only 34 respondents were
responded. The response rate was high in round II than round I and it might be due to less time consuming
online questionnaire. The responses were compiled automatically in Google
spread sheet. Items that reached greater than 70 percent consensus (i.e.
agreement or strong agreement) were selected for round III. About 15 days time was
given for responding. Two reminders were sent in email in 10th and 14th days of
request. The response rate was high in this online version because many
responded were nongovernmental organizations where internet facilities were
easily available.
Round III. In one of the studies
conducted for developing measure of provider adherence to improve the
implementation of behavioral health services in primary care Delphi process,
few new items were added, eliminated and a high level of consensus was achieved
in remaining items among the group of experts(Beehler, Funderburk, Possemato, & Vair, 2013). Altogether 33
experts who responded in round II were invited for panel discussion through
email. Only 15 experts participated in
the panel discussion and provided their views and consensus on final
process. All 52 HRB and major determinants with more than 70 percent score consensus
from round II were selected for questionnaire development. Furthermore, HRB
determinants with more than 50 percent score in round II, supported by
literature review and grounded theory were presented and 12 such HRB
determinants were also included for questionnaire development after endorsement
by expert in round III. Therefore, 64 HRB determinants were identified for
questionnaire development (Table 6).
More
than 50 % of total items were identified
by experts with more than 70 percent score in Delphi, 31 percent items were
identified by Delphi with more than 50 percent score but supplemented by literature
and grounded theory and 19 percent items were from literatures and Grounded
theory which all were further verified in
Delphi round III by experts.
Results
Results includes outcomes of Delphi
process, literature search and grounded theorization process.
Delphi Process
The overall response rate was 80
percent in round I. Experts identified altogether 117 issues related
adolescents' health problems, adolescents' health risk behaviors and major
issues at different level or domain which affects adolescents' risk behavior (Table
2). Issues at different
levels was identified on the basis of ecological system theory.
Table 2
|
Adolescents' health problems, risk
behaviors and health issues identified by Delphi round I experts
|
Dimension
|
Frequency
|
Major issues (n=40)
|
Adolescents' Health
problem
|
15
|
Stress, anxiety,
tension, headache, unwanted pregnancy, sexually transmitted infections,
HIV/AIDS, Unsafe abortion, Road accidents, Poor sanitation and hygiene
|
Adolescents Risk
behavior
|
14
|
Smoking, Chewing
tobacco, Alcoholism, Drug abuse, Unsafe sexual activities, multiple sex
partner
|
Adolescents' health issues identified by Delphi round I experts
|
||
Personal level
|
15
|
gender, curiosity,
Current living, age, curiosity, educational attainment, educational level,
future educational plan, religion, caste, culture, personal identity
|
Household level
|
12
|
Parental
communication, Parental monitoring ,parental
relationship, parental income level, family norms value, types of
family, parental occupation, Parental marital status, violence in family,
number of siblings in family, family member involvement in risk behavior
|
Community level
|
||
School
|
17
|
School curriculum,
relationship with teacher, teacher's behavior, engagement in extracurricular
activities, school rules/norms, teaching learning methods, classmates, behave, school monitoring system, school
and classroom environment, Bullying
and victimization
|
Peer
|
6
|
Peer pressure, peer
involvement in risk behavior, Peer group affiliation, friends' behavior, Gang
and crowd behavior
|
Societal level
|
19
|
Internet, media,
television, radio, access to health clinic, skills and behavior of service
provider, societal norms, government rules, mobile phone, films, fashion,
relationship with neighbor, Youth friendly service centers, recreation
facility
|
Risk behavior reduction strategy
|
19
|
Parental coaching,
sexuality education, Counseling service, Awareness program, Peer group
education, state law and order, engage adolescents in extracurricular
activities, reduce media influence, appropriate school environment
|
Note. n=number of experts responded in Delphi round
I
|
||
The major issues identified by
experts were also very similar to the findings from literature review and
adolescents' opinion on grounded theory process. The response rate was 85
percent in round II. Altogether, 86 HRB and major determinants were identified
in round II of Delphi( Table 3).
Table
3
Items identified from Round II of Delphi process
Variables
|
Number
of HRB determinants
|
Health
risk behavior of adolescents
|
17
|
Demographic
determinants
|
14
|
Family
level determinants
|
14
|
School
level determinants
|
24
|
Community
level determinants
|
17
|
Total number of item for round II
|
86
|
In round II, Drug abuse, smoking,
unsafe sex, alcoholism and premarital sex were the major risk behavior among
adolescents, each more than 90 percent. Out of 86 HRB determinants in round II,
52 HRB determinants scored greater than 70 percent(Table 4).
Table
4
Items related to determinants of risk behavior with
more than 70 percent score
Level
|
Percentage
|
level
|
Percentage
|
|
Personal
Age
|
80.7
|
Education by visiting
experts
|
84.4
|
|
Gender
|
74.2
|
Classmate behavior
|
81.3
|
|
Occupation
|
74.2
|
School monitoring
|
81.3
|
|
Self-perception
|
74.2
|
Bullying
|
78.2
|
|
Curiosity
|
74.2
|
Awareness programs
|
78.1
|
|
Heroicness
|
71.9
|
Teacher's behavior
|
71.9
|
|
Date violence
|
71.9
|
Peer
|
||
Family
|
Peer pressure
|
96.9
|
||
Parental communication
|
96.9
|
Close involvement in
risk behavior
|
96.9
|
|
Parental monitoring
|
92.6
|
Peer involvement with
CSW
|
90.6
|
|
Parental Income
|
88
|
Affiliation with youth
gang
|
87.5
|
|
Family violence
|
87.5
|
Affiliation with youth
group
|
81.2
|
|
Relationship with
parents
|
84.4
|
Community/Society
|
||
Parents on risk
behavior
|
81.3
|
Attendance in nigh
club and pub
|
90.6
|
|
Mother's educational
level
|
75.1
|
Prone site visit
|
90.6
|
|
Pocket money for
adolescents
|
75.1
|
Gadget
possession(mobile)
|
78.1
|
|
Parents marital status
|
75
|
Youth friendly Health
Service
|
78.1
|
|
Service provider
skills
|
71.9
|
Round III. The
response rate was 45 percent in round III. With the help of 64 HRB determinants
after consensus development from experts in round III were used in
questionnaire development.
Literature Review
The literature review was carried
out with search terms "adolescents", "risk", "behavior",
"determinants", "factors" in Google scholar and Pub Med. More
than 40 articles were reviewed in the process. The major identified HRB
determinants from the literature were very similar to experts’ views but HRB
determinants in the family domain(such as the role of parents) was different
among different cultures around the world.
Grounded Theory
Adolescents pointed out that smoking
is the major health risk behavior among adolescents, in addition to alcoholism,
drug abuse, sexual activities and fighting. Some of the students mentioned that
almost 50 percent students in their class smoke. They believed caste, gender,
chosen subject stream, health education, peer pressure, parental care, family
type , media, internet facilities, gadgets possession determined the health
risk behavior of the adolescents. They also emphasized that students from
science faculty got less time to engage in risk behavior than students from
management and humanities background.
Discussion
The face and content validity of the variables and items was established through
literature review, grounded theory and Delphi process. The Delphi process
helped to find HRB and determinants based on group consensus rather than a single
view-point. The major risk behaviors, dimensions and corresponding determinants
were identified through expert consensus in three rounds of Delphi process,
with further validation through literature review and grounded theory. One Delphi
study conducted to find the most important outcomes for effectiveness studies
on migraine treatment used the flexible approach as in this study to collect data by using
series of questionnaires. The initial questionnaire collected qualitative
information through open ended questions(Wojcieszek et al., 2016)which reported
back to participants via a second quantitative round and finally, the responses
in the second round were summarized and reported to participants in third round(Smelt et al., 2014). Generally,
consensus is achieved when a predefined percentage of experts come to agreement
on issues being studied(Nworie, 2011).The consensus
among experience experts from different institutions helped to validate the content
of the instrument. Anonymity among the panelist is also very important aspect
in validation of the instrument because usually junior panelists are reluctant
to express opinions that are opposite to their seniors(Kikukawa, Stalmeijer, Emura, Roff, & Scherpbier,
2014).Moreover, the homogeneity of sample
is crucial step in Delphi process and process provide opportunity for selected
experts to express opinion freely by eliminating personal conflict (Giannarou& Zervas, 2014).
The
experts indicated smoking, alcoholism, drug abuse and unsafe sexual activities
in round I of Delphi process as a major HRB among adolescents(Table 5) which
were further validated in round II and round III. The high response rate in
round II was most likely due to the easy online objective nature of
questionnaire.
Table 5
Experts view on risk behavior of adolescents in
round II
Risk Behavior
|
Expert view in percentage (n=34)
|
Drug abuse
|
96.9
|
Smoking
|
93.8
|
Unsafe sex
|
93.7
|
Alcoholism
|
90.7
|
Premarital sex
|
90.6
|
Multiple sex partner
|
87.1
|
Sex with commercial
sex workers(CSW)
|
84.4
|
Chewing tobacco
|
84.4
|
Sex without
contraception
|
81.3
|
No casual sex partner
|
81.3
|
Note. n=number of
experts responded in Delphi round I
|
|
The major reduction in content was
made in round II where 52 determinants
scored more than 70 percent and determinants scored more than 50 percent along
with strongly supported by literature review and grounded theory were retained(
Table 6). So, out of 86 items only 52 items retained.
Table
6
|
||
Result from Round III
of Delphi process along with 12 HRB determinants from literature and grounded
theory process
|
||
Description
|
Number
of determinants
|
Percentage
|
Number
of HRB determinants which scored more than 70 percent(Items from Delphi)
|
32
|
50
|
Number
of HRB determinants which scored more than 50 percent and supported by
literature and field study
(Items
from Delphi, literature, Grounded theory)
|
20
|
31
|
Number
of items from literature review and grounded theory(Item from Literature
& Grounded Theory
|
12
|
19
|
Total item considered for tool
construction
|
64
|
100
|
12
HRB determinants which are not identified by experts in round I and
round II but the emphasized in literature and grounded theory were added for consensus
development in round III. The questions on the basis of identified determinants
were prepared. Some of important demographic items were also added so altogether
questionnaire with 127 questions were
developed.(Table 7).
Table
7
Number of
questions in each domain in an instrument
Section
|
Major
Domain
|
Number
of questions
|
A
|
Background
Information
|
18
|
B
|
Health
behavior of adolescents
|
42
|
C
|
Family
Domain
|
18
|
D
|
School
Domain
|
23
|
E
|
Community
Domain
|
17
|
F
|
Coping
Mechanism for reducing risk behavior
|
9
|
Total
|
127
|
Some studies iterated a number of
methodological issues such as quality and size of panel expertise, number of
rounds, questionnaire development, analysis and achievement of consensus arises
in Delphi process which have the capacity to threaten the credibility and validity
of the study(Hanafin, 2004). Six techniques
to minimize and eliminating the biases
such as collective unconscious, contrast effect, neglect of probability, von Restorff
effect, my side bias, recency effect, primacy effect and dominance and such
major bias can be reduced by
randomization of questions in survey, requiring individual probability and
severity ratings including reasons in controlled feedback, conducting multiple
rounds of surveys, identifying individuals that have experienced recent, relevant events and reporting results
as medians rather than means (Hallowell, 2009).
Face validity (an assessment that a
measurement instrument appears to measure what it is expected to measure)was
ascertained by an overall judgment of tool for assessing the risk behavior and
its determinants by experts. Items in the tools were also compatible with other
previously developed international tools such as IOWA Youth Survey tool 2014 (University of
IOWA, 2015) ,
Middle School Youth Risk Behavior Survey (North Dekota Department of Public
Instruction, 2015) ,
Youth Risk Behavior Surveillance system (CDC, 2016) .
Ranking of participants’ responses in different iterations helped to select the
high scored items to establish content validity. High level of agreement in
literature review, expert opinion and field survey indicate that finding has
high degree of generalizability (Gadau, Zhang, Yeung, Bian, & Lu, 2016).
Challenges in the Delphi
Process
Group pressure for consensus
development may not be true consensus, feedback mechanism may lead to
conformity rather than consensus, no accepted guidelines for determining
consensus, sample size and sampling techniques, outcomes are perceptual at
best, requires time and participants commitment, possible problem in developing
initial questionnaire to start the process, potential dangers of bias such as
selection criteria for panel and manipulation by researchers, time delay
between rounds in data collection process, dropout, response rate are the major
weakness and challenge in Delphi process(Hung, Altschuld, & Lee, 2008). There are five main
challenges in Delphi process.
Low
response rate. Low response rate is one of the major methodological
challenges in Delphi process. The response rate in Delphi process is sometimes low
due to multi round nature where experts need to participate in number of
iterations(Hsu & Sandford, 2007a). Delphi process
usually need to achieve optimum initial response rate and maintain a high
response rate in each subsequent iterations. Being unable to retain qualified
available experts and being unable to achieve an ideal response rate may affect
the validity of Delphi process(Hsu & Sandford, 2007). To overcome non
response rate, different strategies can be applied. The response rate can be
increased by using an online version of questionnaire with instructions (Araujo et al., 2016) and use of
wireless audience participation system maximize the response rates and expedite
the consensus development among experts(Aw, Loney, Elias, Ali, & Adam, 2016).The response rate
in the round II was the highest by using online Google form whereas it was the
lowest in face to race round III. Internet based rounds avoid the need for face
to face meeting of the experts(Lee et al., 2016). Internet based response system increases the
response rate due to its less time consuming nature and it has also the potential to offer more
content validity by improving ethics in research by ensuring anonymity and
confidentiality(Colton & Hatcher, 2004).
Follow up by telephone and email in each round helped to
increase response rate. At least two follow ups were carried out. Other strategy
to deal with non response rate in Delphi process are: to use assistance from
endorsed individuals who are influential or famous or renowned in the area of investigation,
initial contact in phone before requesting participation in round I, and constructing
an easy and less time consuming questionnaire.
Identification
and selection of experts. Identification and selection of experts is another
methodological challenge in Delphi process. Experts in Delphi process are knowledgeable in a field, subject matter
expertise who can provide opinion on the issue under investigation. Delphi
process relies on opinion of experts which is crucial for consensus development
and future direction. Therefore,
identification and selection of
appropriate experts in Delphi process maximizes the quality of responses
and reduces bias as well as builds credibility in the results (Nworie, 2011). However, a
recent Indian study revealed that participants having more than ten years of
experience were more inconsistent and changed their response in the successive
rounds of Delphi process(Raghav & Bhardwaj, 2016) .Two major problems
in selecting experts in Delphi are differentiation between levels of expertise
for consensus development and the relevancy of experts in the subject area
where subject matters is cultural value rather than technology(Welty, 1973). To overcome
selection bias and non response rate, as previously discussed a four step
selection procedure was used.
Experts
time constraint for participation in successive rounds. Getting experts’
time for three or more successive rounds was also methodological challenge in
Delphi process. Delphi process required multiple rounds to develop consensus
among participants which requires great deal of time and many participants drop
out during process(Gordon, 2009). Furthermore, the response rate of participants
decrease for each round of the process
and more number of reminders were needed in the process (Raghav & Bhardwaj, 2016) . Round I was completed
with 80 percent response rate, round II was completed with 85 percent response
rate but round III was completed with only 45 percent response rate. It showed
the response rate was moderate in first qualitative round, high in online
quantitative round II and low in physical presence round III.
Logistic
management. Logistic management in order to bring busy panelist together at
same time is one of the major non-methodological challenges in Delphi process.
In one of the studies conducted out of
19 people invited only six members comprised
in final panel due time pressure as a primary reason for non
participation(Courtney, O’Reilly, Edwards, & Hassall, 2010). In this
instrument development process also 34 respondents who participated in round II
were invited for round III panel discussion but only 15 experts participated. The
first reason was time constraint to attend discussion due to busy schedule on
that particular day and some of them felt venue was not appropriate for them.
Time consuming nature. One of the non
methodological challenges in Delphi process was its time consuming nature. Delphi
process was conducted in sequential and iterative manner and each step needs to
allow enough time (at least two weeks gap) to maximize the response rate. So it
required enough time between iterations. The gap between two rounds of the
Delphi process provides investigators and panelists to improve accuracy of the
results but it also increases the duration of the Delphi process (Hsu & Sandford, 2007) and lengthy process may result in participants attrition (Nworie, 2011).
This
tool development process took four months. However, many experts in final face
to face round were unable to participate in panel discussion program. It
reduced experts’ inputs in the final consensus development stage. Each participating
expert also had their own area of specialization in adolescents and youth
issues, therefore many experts 'responses in the first round were specific to a
particular issue rather than broad risk behavior issues. This process was
conducted by involving national and local experts from Nepal and the field
survey to support grounded theory was
carried out in two municipality of Nepal so generalizability of the
result might not be suitable in other part of the regions where the context are
different. Systematic literature review in different databases were not done
due to time constraint so relevant literature might have been missed during the
search process. Grounded theory was carried out by randomly selecting two
schools where total of 10 students were selected from science and management
stream of Higher Secondary School for in depth interview. It would be more
representative if students from Humanities and other stream were also selected
to understand local context.
Different instrumental development
approaches have been used across the world in sensitive issues like risk
behavior. Risk behavior tool development processes have generally been based
purely on literature review and item reduction by collaborative group members
and content experts. For example, In development of an upper extremity outcome
measure, an initial pool of 821 potential items was generated through extensive
literature search and items were reduced to 177 by expert's judgment or opinion
(Hudak, Amadio, & Bombardier, 1996). Such instruments
may not be compatible with cultural and social context. A Study conducted in
Iran used the ecological framework for the development of the questionnaire
which confirmed the cultural and social condition in order to predict
protective sexual behavior of women at risk of HIV. The process started with
qualitative study which involved in-depth interview with women at risk. The Initial
pool of items was generated on the basis of
in depth interview and literature review. In order to verify the questionnaire,
qualitative face validity, content and construct validity were assessed where
many items in appropriate content validity ratio and index were deleted before
assessing reliability of the instrument (Lotfi, Ramezani Tehrani, Yaghmaei, & Hajizadeh,
2014) as in the Delphi process of this
study.
In other study, items were generated
while developing the adolescent Exploratory and Risk Behavior Rating
Scale(AERRS) through literature review by using computerized databases such as Psycho
INFO and ERIC and existing risk behavior questionnaire. The AERRS designed to
measure participation in risk behavior and perception of risk behavior. The
participation was designed to measure in 1(never) to 4(often) likert scale. The
validation of the items selected was performed using Item Response Theory and
analysis was done by factor analysis, correlations and coefficient alpha
analyses(Skaar, 2009) .
This study used general descriptive
statistics in Delphi process to identify the HRB determinants based on
classical Delphi process and identified HRB determinants are based on Nepalese
context. Further testing identified HRB determinants would help in tool
development and to identify more generalizable findings in adolescent health
risk behavior and its determinants.
Limitation
of the study
This is a descriptive study article.
No specific hypothesis was developed and tested. The study was conducted among
experts, selected based on predefined criteria in Kathmandu valley, Nepal. The
profile of experts selected may be different in other parts of Nepal so the
study will not be generalizable across the country, especially in rural area.
Conclusion
The Delphi method supplemented by
literature review and grounded theory is designed to help to get consensus
between expert panels. The process helped to get more valid tool based on local
context and previous researches. The Delphi methodology is an effective
approach for investigating the problems with the help of judgments of expert
panelists. The view of expert panelists was further cross checked with other
research by literature review and with grounded theory by in depth interview
with adolescents.
This article described the Delphi
process during instrument development. Furthermore, HRB determinants which were
not identified in Delphi process but were relevant to local context were
identified through grounded theory. Major determinants were also cross checked
with previous research items.
Delphi techniques supplemented with
grounded theory and literature review can be useful for developing the instrument on HRB and its determinants.
The whole process helped to establish face and content validity of the
instrument. Many challenges while developing the instrument can be overcome or
can be minimized.
References
Araujo, V., Teixeira, P. M., Yaphe, J., & Correia
de Sousa, J. (2016). The respiratory research agenda in primary care in
Portugal: a Delphi study. BMC Family Practice, 17(1), 124. http://doi.org/10.1186/s12875-016-0512-1
Aw, T.-C., Loney, T., Elias, A., Ali, S., & Ádám, B.
(2016). Use of an audience response system to maximise response rates and
expedite a modified Delphi process for consensus on occupational health. Journal
of Occupational Medicine and Toxicology, 11(1), 9.
http://doi.org/10.1186/s12995-016-0098-5
Beehler, G. P., Funderburk, J. S., Possemato, K., & Vair,
C. L. (2013). Developing a measure of provider adherence to improve the
implementation of behavioral health services in primary care: a Delphi study. Implementation
Science, 8(1), 19. http://doi.org/10.1186/1748-5908-8-19
Benson, J. and C. F. (1982). A Guide for Instrument
Development and Validation. The American Journal of Occupational Therapy,
36(12), 789–800.
Colton, S., & Hatcher, T. (2004). The web-based Delphi
research technique as a method for content validatin in HRD and adult education
research. Academy of Human Resource Development International Conference,
183–189. Retrieved from http://www.eric.ed.gov/PDFS/ED492146.pdf
Courtney, M., O’Reilly, M., Edwards, H., & Hassall, S.
(2010). Content validity of the ResCareQA: An Australian residential care
quality assessment based on resident outcomes. Australian Journal of
Advanced Nursing, 28(3), 37–46.
Cucolo, D. F., & Perroca, M. G. (2015). Instrument to
assess the nursing care product: development and content validation. Revista
Latino-Americana de Enfermagem, 23(4), 642–50.
http://doi.org/10.1590/0104-1169.0448.2599
Eberman, L. E., & Cleary, M. A. (2011). Development of a
Heat-Illness Screening Instrument Using the Delphi Panel Technique. Journal
of Athletic Training, 46(2).
Fenton, K. A., Johnson, A. M., Mcmanus, S., & Erens, B.
(2001). Series editors Measuring sexual behaviour : methodological challenges
in survey research. Sex Transm Inf, 84–92.
http://doi.org/10.1136/sti.77.2.84
Fitzsimons, G. J., & Moore, S. G. (2008). Should we ask
our children about sex, drugs and rock & roll? Potentially harmful effects
of asking questions about risky behaviors. Journal of Consumer Psychology,
18(2), 82–95. http://doi.org/10.1016/j.jcps.2008.01.002
Gadau, M., Zhang, S.-P., Yeung, W.-F., Bian, Z.-X., & Lu,
A.-P. (2016). TCM Pattern Questionnaire for Lateral Elbow Pain: Development of
an Instrument via a Delphi Process. Evidence-Based Complementary and
Alternative Medicine, 2016(1), 1–12.
http://doi.org/10.1155/2016/7034759
Giannarou, L., & Zervas, E. (2014). Using Delphi
technique to build consensus in practice. International Journal of Business
Science and Applied Management, 9(2), 65–82.
Gordon, T. J. (2009). The Delphi Method. Futures Research
Methodology v3.0 [CD-ROM], 1–29.
Hallowell, M. R. (2009). Techniques to minimize bias when
using the Delphi method to quantify construction safety and health risks. In
Building a Sustainable Future-Proceedings of the 2009 Construction Research
Congress, (March), 1489–1498. http://doi.org/10.1061/41020(339)151
Hanafin, S. (2004). Review of literature on the Delphi
Technique, (March), 1–51. Retrieved from http://www.childrensdatabase.ie/documents/publications/Delphi_Technique_A_Literature_Review.pdf
Hsu, C., & Sandford, B. (2007a). The delphi technique:
making sense of consensus. Practical Assessment, Research & Evaluation(a
Peer-Reviewed Electronic Journal), 12(10), 1–8. http://doi.org/10.1016/S0169-2070(99)00018-7
Hsu, C., & Sandford, B. A. (2007b). Minimizing
Non-Response in The Delphi Process : How to Respond to Non-Response. Practical
Assessment, Research & Evaluation(a Peer-Reviewed Electronic Journal), 12(17).
Hudak, P. L., Amadio, P. C., & Bombardier, C. (1996).
Development of an upper extremity outcome measure: the DASH. Am J Ind Med,
29(JUNE), 602–608. http://doi.org/10.1002/(SICI)1097-0274(199606)29
Hung, H. L., Altschuld, J. W., & Lee, Y. F. (2008).
Methodological and conceptual issues confronting a cross-country Delphi study
of educational program evaluation. Evaluation and Program Planning, 31(2),
191–198. http://doi.org/10.1016/j.evalprogplan.2008.02.005
Karaman, N. G. (2007). Adolescent risk-taking : comparison
between adolescents ’ and adults ’ opinion Adoção de comportamentos de risco
por adolescentes : uma comparação das opiniões de adolescentes e adultos
Adopción de comportamientos de riesgo por adolescentes : una comparac. Paidéia,
17(38), 357–364.
Kikukawa, M., Stalmeijer, R. E., Emura, S., Roff, S., &
Scherpbier, A. J. (2014). An instrument for evaluating clinical teaching in
Japan: content validity and cultural sensitivity. BMC Medical Education,
14(1), 179. http://doi.org/10.1186/1472-6920-14-179
Lee, G. H. M., McGrath, C., Yiu, C. K. Y., Bolin, A., Bolin,
A., Jansson, L., … Featherstone, J. (2016). Developing clinical practice
guidelines for caries prevention and management for pre-school children through
the ADAPTE process and Delphi consensus. Health Research Policy and Systems,
14(1), 44. http://doi.org/10.1186/s12961-016-0117-0
Lotfi, R., Ramezani Tehrani, F., Yaghmaei, F., &
Hajizadeh, E. (2014). Developing a valid and reliable instrument to predict the
protective sexual behaviors in women at risk of human immunodeficiency virus. Iranian
Red Crescent Medical Journal, 16(9), e14682.
http://doi.org/10.5812/ircmj.14682
Nworie, B. J. (2011). Using the Delphi Technique in
EducationalTechnology. TechTrends, 55(5).
Okoli, C., & Pawlowski, S. D. (2004). The Delphi Method
as a Research Tool : An Example , Design Considerations and Applications 1
Introduction 2 Overview of the Delphi method. Information and Management,
42(1), 15–29.
Pereira, R. D. D. M., & Alvim, N. A. T. (2015). Delphi
technique in dialogue with nurses on acupuncture as a proposed nursing
intervention. Escola Anna Nery - Revista de Enfermagem, 19(1),
174–180. http://doi.org/10.5935/1414-8145.20150024
Powell, C. (2003). The Delphi technique: Myths and realities.
Journal of Advanced Nursing, 41(4), 376–382.
http://doi.org/10.1046/j.1365-2648.2003.02537.x
Skulmoski, G. J., & Hartman, F. T. (2007). The Delphi
Method for Graduate Research. Journal of Information Technology Education,
6.
Smelt, A. F. H., Louter, M. A., Kies, D. A., Blom, J. W.,
Terwindt, G. M., Van Der Heijden, G. J. M. G., … Assendelft, W. J. J. (2014).
What do patients consider to be the most important outcomes for effectiveness
studies on migraine treatment? Results of a Delphi study. PLoS ONE, 9(6),
1–7. http://doi.org/10.1371/journal.pone.0098933
Tymula, A., Rosenberg Belmaker, L. A., Roy, A. K., Ruderman,
L., Manson, K., Glimcher, P. W., & Levy, I. (2012). Adolescents’
risk-taking behavior is driven by tolerance to ambiguity. Proceedings of the
National Academy of Sciences of the United States of America, 109(42),
17135–40. http://doi.org/10.1073/pnas.1207144109
Welty, G. (1973). Some Problems of Selecting Delphi Experts
for Educational Planning and Forecasting Exercises. California Journal of
Educational Research, 24(3), 129–134.
Wojcieszek, A. M., Reinebrant, H. E., Leisher, S. H.,
Allanson, E., Coory, M., Erwich, J. J., … Flenady, V. (2016). Characteristics
of a global classification system for perinatal deaths : a Delphi consensus
study. BMC Pregnancy and Childbirth, 1–11.
http://doi.org/10.1186/s12884-016-0993-x
No comments:
Post a Comment