Reliability and Validity Clause Samples
Reliability and Validity. Because the concept of reliability and validity originated from the natural sciences, the values of both have been questioned with respect to determining the quality of qualitative research (▇▇▇▇▇ & ▇▇▇▇▇▇▇ 2012). Nonetheless, in an effort to help determine the strength of the study data, both will be considered against the chosen approach for the study. Reliability is concerned with ‘the consistency of measurement within a study’ (▇▇▇▇▇ 2010, p28) or replicability of the study findings if the study were to be repeated using similar methods. Early authors (▇▇▇▇▇▇ 1996, ▇▇▇▇▇▇▇ 1997) have postulated that the Delphi technique enhances the reliability of a study because of the interactive component and ability to avoid group bias by the nature of the approach. Furthermore, the membership of the expert panel for the Delphi survey, drawn from both nursing and medicine meant that data provided multiple perspectives. This ensured that there was participation within the Delphi survey from different grades of nursing staff working in emergency nursing, in addition to medical experts at consultant level. The use of participants with expertise and an interest in the phenomenon under investigation has also been postulated by ▇▇▇▇▇▇▇ (1987) as caveats to increasing the content validity of a study. Validity refers to ‘how closely what we describe, assess or measure in our research resembles what is in the world or happening in the world’ (▇▇▇▇▇▇▇▇▇ 2009, p.200). The recruitment of experts from emergency medicine, the use of three consecutive rounds and ability to reach consensus have been claimed to strengthen the validity of a study (▇▇▇▇ et al 1991, ▇▇▇▇▇▇ et al 2000). Nevertheless, ▇▇▇▇▇▇▇ (2006) suggest that the term ‘trustworthiness’ should be used in preference to validity when determining the effectiveness of a Delphi survey. However, to establish trustworthiness, four key strategies must be determined, namely; credibility, dependability, confirmability and transferability (Polit et al 2001). Credibility within this study was enhanced by providing feedback to panelists through subsequent rounds of the Delphi survey. Dependability was achieved by the representative sample of the expert panel that included both nursing and medical emergency expertise. Confirmability was demonstrated by the transparency in describing the Delphi collection and analysis process. Finally, transferability was established through confirming the applicability of the findings which were co...
Reliability and Validity. 39 Plan for Data Collection ............................................................................................ 39 Timeline..................................................................................................................... 40 Ethical Issues ............................................................................................................. 43 Summary.................................................................................................................... 44 CHAPTER 4: RESULTS ...................................................................................................... 45
Reliability and Validity. ▇▇▇▇ and ▇▇▇▇ use the term ’reliability’ and claim that it is a significant criterion for assessing the value of a piece of qualitative research: ’the analysis of qualitative data can be enhanced by organizing an independent assessment of transcripts by additional skilled qualitative researchers and comparing agreement between the raters’ (▇▇▇▇ and ▇▇▇▇, 1995). A contrary position is taken by ▇▇▇▇▇ who argues that the use of ’external raters’ is more suited to quantitative research; expecting another researcher to have the same ’insights’ from a limited data base is unrealistic: ’No-one takes a second reader to the library to check that indeed he or she is interpreting the original sources correctly, so why does anyone need a reliability checker for his or her data?’ (▇▇▇▇▇, 1994). Armstrong et al have wryly pointed out that the example used by ▇▇▇▇ and ▇▇▇▇ (1995) for inter-rater reliability was actually one of ascribing quantitative weights to pregiven ’variables’ which were then subjected to statistical analysis (Armstrong et al., 1997). They tested the degree of inter-rater reliability that might be expected by asking six researchers to identify themes in the same focus group transcript. The results showed close agreement on the basic themes but each analyst ’packaged’ the themes differently. They conclude that “Analysis is a form of interpretation and interpretation involves a dialogue between researcher and data in which the researcher’s own views have important effects.” (1997: 601) ▇▇▇▇, ▇▇▇▇▇▇▇▇ and ▇▇▇▇ respond that “Armstrong et al. conducted a tough test of inter rater agreement and one which would be unusual in a typical research study” (▇▇▇▇ et al., 2006). Nevertheless, ▇▇▇▇ et al maintain that that there may be merit in involving more than one analyst in situations where researcher bias is perceived to be a risk by others, citing examples of social scientists investigating the work of clinicians or evaluating government policy. In this study I discussed emerging themes and anonymised dialogue extracts with my supervisors. This did not represent a formal attempt at inter-rater reliability but ensured that I not only remained open the possibility of more than one possible interpretation but also that I did not ignore interesting data simply out of over-familiarity. Validity has been defined as the extent to which the account accurately represents the social phenomena to which it refers (▇▇▇▇ and ▇▇▇▇, 2006b). ▇▇▇▇▇▇▇▇ and ▇▇▇▇▇ write about int...
Reliability and Validity. 38 Data Collection .......................................................................................................... 39 Data Analysis............................................................................................................. 39 Ethical Issues ............................................................................................................. 40 Summary.................................................................................................................... 41 CHAPTER 4: FINDINGS ..................................................................................................... 42 Findings of Qualitative Research .............................................................................. 42 Qualitative Data Analysis Findings ........................................................................... 43
Reliability and Validity. ▇▇▇ (1994) points out four tests that can be used to evaluate the quality of case studies: construct validity, internal validity, external validity and reliability. Internal validity is only for causal case studies and is therefore not relevant to this study. Construct validity is to form a chain of evidence, from which conclusions can be drawn from the initial research questions (▇▇▇ et al., 1999). Such a chain of evidence derives from the feature of a case study, and is called triangulation (▇▇▇▇▇▇, 1997b). Triangulation is defined as “a validity procedure where researchers search for convergence among multiple and different sources of information to form themes or categories in a study” (▇▇▇▇▇▇▇▇ and ▇▇▇▇▇▇, 2000, p. 126). The triangulation of evidence is used to support and prove the data from other sources (▇▇▇▇▇▇, 1997a) and to enhance the validity and reliability of the study results (▇▇▇▇▇, 2003). Researchers use different sources of data to triangulate their subjective perceptions (▇▇▇▇▇▇, 1985). In this research, construct validity is guaranteed by triangulation because data is collected from interviewing different levels of managers and employees, interviewing people who work at companies and local government, attending internal seminars, and collecting materials from public and private sources. External validity is strengthened using multiple case studies because, as researchers (▇.▇. ▇▇▇▇▇▇, 1997b) have argued, it is difficult for single case studies to generate external validity. This research strengthens external validity as it adopts multiple case studies. Reliability is improved by creating a case study database (Antila, 2006). This research improves reliability through forming a case study database, which includes all information collected by this research. All in all, reliability and validity can be ensured through drawing consistent results from different methods and sources of data (▇▇▇▇▇▇▇ and ▇▇▇▇▇▇▇▇▇▇▇, 1999). Cronbach’s alpha coefficient was calculated to check the reliability of the scales that are measured in the questionnaires. Cronbach’s alpha is a coefficient of reliability, which is commonly used as a measure of the internal consistency reliability of test scores for a sample (Pallant, 2005). Management papers generally use ▇▇▇▇▇▇▇▇’s alpha to check the reliability of questions in questionnaires. Cronbach’s alpha is defined as: where α is Cronbach’s alpha coefficient, K is the number of items, is the average variance, is the...
Reliability and Validity. The means of the individual scale items ranged from 0.10 to 1.69 with standard deviations ranging from 0.46 to 1.45. The inter-item correlations ranged from 0.15 to 0.44, suggesting that there was no redundancy among items, defined as an inter-item correlation greater than 0.8544. The mean inter-item correlation was 0.24. The item-to-total correlations ranged from .05 to 0.56 with a mean item-to-total correlation of 0.41. All items except acupuncture demonstrated high correlations with the total scale score (see Table 3). Due to the fact that acupuncture was not significantly correlated with the total scale score it was not included in subsequent analyses. The scale mean was 8.68 with a standard deviation of 5.56. The internal consistency of the entire scale, excluding acupuncture, was α=0.67 and the test-retest reliability was r=.79 (p<0.01). Maximum likelihood analysis resulted in a three-factor solution with 4 items loading on factor conceptualized as “home-based CAM”, 4 items loading on a factor conceptualized as “ingested/inhaled CAM” and 4 items loading on a factor conceptualized as “body-based CAM.” Counseling/support groups loaded poorly and thus was considered an equivocal item based on the fact none of the loadings on any of the factors was 0.30 or higher, the recommended minimum criteria for a factor loading50. Internal consistency estimates for each of the subscales generated by the factor analysis were: α=0.54 for the “home-based CAM” subscale, α=0.56 for the “ingested/inhaled CAM” subscale, and α=0.59 for the “body-based CAM” subscale (see Table 4). Results from the known-groups analyses suggest that female participants did report using CAM more frequently (mean=4.22, SD=2.29) than male participants (mean=3.40, sd=2.25) though these groups did not significantly differ [F=2.32 (2,181), p=0.10].
Reliability and Validity. The quality of the 32 included articles was first assessed by examining the degree to which each assessed and/or reported any psychometric properties of the CAM use instrument(s) employed. Using an approach similar to that developed by Noar and colleagues31 each article was assigned higher numeric values if it were strong on a characteristic and lower values if it were weak on a characteristic. For example, if an article reported information about the reliability and validity of the CAM instrument, it was assigned a 2, if an article reported information about the reliability or validity of the instrument, it was assigned a 1. If this information was not reported or the assessment/reporting was unclear, it was assigned a 0 (maximum score=2).
Reliability and Validity. To enhance rigor in this study, researchers addressed reliability and validity in a qualitative context (▇▇▇▇▇▇▇, 2005; ▇▇▇▇▇ & Field, 1995). To maintain reliability in data collection, interviewers conducted interviews in the same setting, a private room on campus, for all 19 interviews (▇▇▇▇▇ & Field, 1995). Additionally, the interviewers were of the same status, age, and gender, as they were both young women employed as Graduate Research Assistants (1995). They followed the same interview protocol asking the same questions to enhance reliability. This ensured that the social context in data collection was the same during every interview (▇▇▇▇▇ & Field, 1995). To address validity, researchers remained neutral while conducting the interviews to prevent reactivity and interviewer bias (▇▇▇▇▇▇▇, 2005). The in-depth nature of the interviews enhanced validity because it resulted in rich, descriptive data. Analysis of verbatim transcripts rather than just interviewer notes provided a detailed foundation for, and test of, conclusions (▇▇▇▇▇▇▇, 2005). Also, researchers examined and analyzed all data, including both supporting and discrepant evidence, to provide a thorough and accurate account of the results (2005). This prevented bias in the interpretation. Researchers prevented threats to external validity by confirming participants’ evidence and emergent findings with outside sources, including informants and the literature (▇▇▇▇▇ & Field, 1995). This ensured that the data was credible and confirmable (▇▇▇▇▇▇▇, 2005; ▇▇▇▇▇ & Field, 1995). Thus, researchers addressed potential threats to the rigor of this study by working to enhance reliability and validity. The research assistants independently read two transcripts and created a list of codes to begin the coding process. The research assistants met to compare and review codes for agreement and developed a codebook for data analysis. Using the codebook, the two research assistants coded each transcript. The codebook includes codes based upon the social ecological model as well as other codes developed by the research assistants. If research assistants discovered new codes during coding, they discussed the additional codes and added them to the codebook. The research assistants communicated about additional codes that emerged throughout the entire coding process. The project investigator resolved any discrepancies in coding. This process was repeated until the code list was exhaustive and saturation was reached.
Reliability and Validity. ▇▇▇▇ and ▇▇▇▇ use the term ’reliability’ and claim that it is a significant criterion for assessing the value of a piece of qualitative research: ’the analysis of qualitative data can be enhanced by organizing an independent assessment of transcripts by additional skilled qualitative researchers and comparing agreement between the raters’ (▇▇▇▇ and ▇▇▇▇, 1995). A contrary position is taken by ▇▇▇▇▇ who argues that the use of ’external raters’ is more suited to quantitative research; expecting another researcher to have the same ’insights’ from a limited data base is unrealistic: ’No-one takes a second reader to the library to check that indeed he or she is interpreting the original sources correctly, so why does anyone need a reliability checker for his or her data?’ (▇▇▇▇▇, 1994). Armstrong et al have wryly pointed out that the example used by ▇▇▇▇ and ▇▇▇▇ (1995) for inter-rater reliability was actually one of ascribing quantitative weights to pregiven ’variables’ which were then subjected to statistical analysis (Armstrong et al., 1997). They tested the degree of inter-rater reliability that might be expected by asking six researchers to identify themes in the same focus group transcript. The results showed close agreement on the basic themes but each analyst ’packaged’ the themes differently. They conclude that “Analysis is a form of interpretation and interpretation involves a dialogue between researcher and data in which the researcher’s own views have important effects.” (1997: 601) ▇▇▇▇, ▇▇▇▇▇▇▇▇ and ▇▇▇▇ respond that “Armstrong et al. conducted a tough test of inter rater agreement and one which would be unusual in a typical research study” (▇▇▇▇ et al., 2006). Nevertheless, ▇▇▇▇ et al maintain that that there may be merit in involving more than one analyst in situations where researcher bias is perceived to be a risk by others, citing examples of social scientists investigating the work of clinicians or evaluating government policy. In this study I discussed emerging themes and anonymised dialogue extracts with my supervisors. This did not represent a formal attempt at inter-rater reliability but ensured that I not only remained open the possibility of more than one possible interpretation but also that I did not ignore interesting data simply out of over-familiarity. Validity has been defined as the extent to which the account accurately represents the social phenomena to which it refers (▇▇▇▇ and ▇▇▇▇, 2006b). ▇▇▇▇▇▇▇▇ and ▇▇▇▇▇ write about int...