QUALITATIVE RESEARCH (Part 2)

EXEMPLAR STUDIES




Recently, Hill and her colleagues reviewed 27 published studies utilizing the CQR method Hill et al., 2005). Interested readers can find their recommendations for using CQR in Table 11.1 and numerous exemplar studies in their article, such as several multicultural studies (e.g., Kim, Brenner, Liang, & Asay, 2003; Kasturirangan & Williams, 2003) and psychotherapy research by Hill and colleagues (e.g., Hayes et al., 1998; Knox, Hess, Petersen, & Hill, 1997; Knox, Hess, Williams, & Hill, 2003). Next, we will use the research conducted by Juntunen and colleagues (2001) as a CQR example.
The study aims at understanding the meaning and relevant concept of career or career development for a group of American Indians (Juntunen et al., 2001). The authors interviewed 18 American Indians, mostly from one Northern Plains state, with varying ages (21-59 years old), genders (11 female and 7 male), educational levels (from 10th grade to some graduate work), occupations, and tribal affiliations. The sample included college students who self-identified as American Indian recruited through a university's Native American Programs Center, and their friends or partners, as well as attendees of two local powwows. Also, the authors divided the participants into two groups according to their educational levels because later data analyses revealed differences between participants who attended college and those who did not; a detailed description of each subsample was provided in the Participants section of their article.
A semi-structured interview protocol was developed for the purpose of the study by the primary researcher and two consultants who are American Indian scholars. The protocol was piloted with two American Indians and subsequently modified based on the results of the pilot interviews. CQR researchers use pilot interviews to examine the appropriateness and relevance of the interview questions from the participant's perspective and to uncover any important concepts that are unexpected by the researchers. The research team members were trained to conduct the face-to-face interviews using the protocol. The interview procedure was presented in great detail and examples of interview questions were provided. Interviews were subsequently transcribed verbatim and each transcript was checked for accuracy.
TABLE II.1
RECOMMENDATIONS FOR USING CQR
Consideration Recommendation
Consensus process
Biases
The research team
Training team members
Sample
Interviews
Data collection
Domains
Core ideas
Cross-analysis
Auditing
Stability check
Charting the results
Writing the results and discussion section

Participant's review
1. Researchers should openly discuss their feelings and disagreements.
2. When there are disagreements among the researchers about the interviews, everyone should listen to the tape of the interview.
1. Report demographics and feelings/reactions to topic in the Methods section.
2. Discuss the influence of biases in the Limitations section.
3. Openly discuss biases among the research team throughout the process.
4. Journal reviewers need to be aware that biases are a natural part of any research, including CQR
1. Either set or rotating primary teams are acceptable.
2. All team members must become deeply immersed in all of the data.
3. At least 3 people should comprise the primary team.
4. The educational level of team members should match the abstractness of the topic.
5. Team members with more designated power should not claim "expert status."
6. Power issues should be addressed openly.
7. Rotate the order of who talks first to reduce undue influence.
1. Prior to training, read Hill et al., (1997), the present article, and exemplar studies.
2. Consult with an expert if having difficulty learning the method.
3. Describe training procedures in the Methods section.
1. Randomly select participants from a carefully identified homogeneous population.
2. Choose participants who are very knowledgeable about the phenomenon.
3. Recruit 8 to 15 participants if 1 to 2 interviews are used.
1. Review the literature and talk to experts to develop the interview protocol.
2. Include about 8-10 scripted open-ended questions per hour.
3. Allow for follow-up probes to learn more about the individual's experience.
4. Conduct several pilot interviews to aid in revising the interview protocol.
5. Train new interviewers.
6. Ideally, each interviewee should be interviewed at least twice.
1. Match the data collection format to the data desired and the needs of the study.
2. Record reactions to interviews; review tape before subsequent interviews.
1. Develop the domains from the transcripts or a "start list."
2. The entire primary team codes the data into domains in the first several cases; the remaining coding can be done by 1 researcher and reviewed by the team.
1. Use the participant's words; avoid interpretive analysis
2. The entire primary team develops the core ideas for the first several cases; the remaining core ideas can be done by 1 researcher and reviewed by the team, or the entire primary team can work together to code the domains and construct the core ideas.
1. Use frequency labels to characterize data: General applies to all or all but 1 case; typical applies to more than half up to the cutoff for general; variant applies to 2 cases up to the cutoff for typical. When more than 15 cases are included, rare applies to 2-3 cases. Findings applying to single cases are placed in a miscellaneous category and not included in results/tables.
2. When comparing subsamples, results are different if they vary by at least 2 frequency categories (e.g., general vs. variant).
3. Continually refer to the raw data in making interpretations.
4. Continue revising the cross-analyses until elegant and parsimonious.
5. If there are mostly variant or rare categories or a lot of miscellaneous items, revise the cross-analysis (e.g., combine categories, subdivide the sample, or collect more data).
6. Get feedback from others about the cross-analysis.
1. Either internal or external auditors are appropriate for the domains and core ideas, but at least 1 external auditor is desirable for the cross-analysis.
2. For inexperienced researchers, it is helpful for the auditor to examine revisions until he or she is confident that the data are characterized accurately.
3. Auditors should also be involved in reviewing the interview protocol.
The stability check (i.e., holding out 2 cases from the initial cross-analysis), as proposed by Hill et al., (1997), can be eliminated, but other evidence of trustworthiness should be presented.
Charting or other visual approaches for depicting findings (e.g., "webs" or organizational diagrams of categories) could be helpful.
1. At least the general and typical categories should be fully described in the Results section, although all categories in the cross-analysis should be included in a table.
2. Either quotes or core ideas can be used to illustrate the results.
3. Case examples are useful for illustrating results across domains.
4. In the Discussion section, pull together results in a meaningful way and develop theory.
Give transcripts of interviews and write-up of results to participants


The authors clearly portrayed the backgrounds and experiences of the researchers composing the CQR analysis team. They also took specific steps to decrease the effects of researchers' assumptions on the data analysis results and to monitor the group dynamics (e.g., documenting and discussing their assumptions about potential findings in the early phase of the analysis process, audio taping the analysis team meetings, and then reviewing these tapes to ensure that each team member was equally involved in the analysis process).
Following the CQR guidelines, researchers coded the data independently and convened to form consensus on the domains, core ideas, and categories through-cross-analyses. One auditor reviewed the work of the data analysis team and the auditor's feedback was subsequently discussed and incorporated into the final results by the team members. As mentioned earlier, the initial cross-analysis results revealed major differences in the career journey of participants with postsecondary education and those without. Therefore, the authors reexamined the domains and core ideas for each group separately. Although all of the domains remained unchanged, the categories within certain domains (i.e., supportive factors, obstacles, and living in two worlds) varied remarkably. Findings from this study were charted and summarized in Tables 11.2 and 11.3.


Overall, participants articulated their own definitions of career and success. In particular, participants with different educational backgrounds reported differences in the support for and obstacles in their career development processes and the-diverse struggles related to living in two distinct cultures. Categories, which emerged from the data, were discussed and illustrated with direct quotes from participants. Following is the description of one of the domains with categories that vary across the two subsamples (i.e., living in two worlds).
Six of the seven participants with a high school education viewed the Native and White cultures as quite distinct. In the words of one woman, "Those [White] people are way different from us . . . they don't understand a lot of our ways" (Participant 3). Another participant felt the need to adjust her own behavior to be effective outside of the reservation. She stated, "We have to live like Whites when we're out there" (Participant 6). This woman also noted that during the times she had lived off the reservation, she frequently needed "a powwow fix" as a way to stay connected to the Indian community. . . .
The experience of living in two worlds was dramatically different for the college group. Rather than seeing the two worlds as distinct and distant, the respondents who had attended college described the concept in two ways: moving between two worlds (a typical category) and finding a holistic, third world (a variant category)....
The process of moving between two worlds was often described as difficult, both emotionally and cognitively. A tribal elder remembers her experience in the college classroom, several hundred miles from her home reservation:
You always have to do twice as much ... because you have to process every¬thing you think, reprocess it back into a thinking that you know [doesn't] fit with your thinking to make it right for the school.... Back when I applied to law school . . . they said 'You got to stop being Indian and you got to get through these 3 years of school' ... anyway, you can't do that. You can't take that piece of you and set it aside. (Participant 13)
However, not all of the participants who moved between two worlds found the experience difficult.... People who sought a holistic, third world spoke of the desire to appreciate their own culture yet integrate those pieces of the majority culture that they accepted. Some respondents viewed this as a personal internal choice, which can be made whether one lived on or off the reservation.
Another woman spoke of her experience of two worlds as an evolution into a third world, which she has created from those aspects that are most important to her from each world.
I've lived in those two worlds and there's also even a middle world. I mean, you have the Native, the non-Native and you have a middle world that I have experienced and it's a very personal, personal world where you gather, it seems like you gather from both worlds, and create, you know, a happy medium for yourself. I have two children that have lived with me on the reservation and off the reservation and I like it that way. I like them to be able to know both worlds. Because it just opens so much more range of experiences. (Participant 11)
However, creating that third world does not mean that the original experience of her Native world is diminished in any way. "There's no leaving the Native American world. That comes along. When it's your core being, you can't leave it. I wouldn't want to leave it ... I'm Indian to the core" (Participant 11). (Juntunen et al., 2001, pp. 281-282)

CQR researchers often use quotes from the interviews or core ideas to illustrate each category. As Hill and colleagues (2005) suggested, visual representations of the results in tables, diagrams, or figures also help the readers understand the relationships among categories. Finally, narrative case examples may provide a holistic picture of the findings. Examples of case descriptions can be found in the articles written by Williams and colleagues (1998) and Ladany and colleagues (1997).

PRIMARY TASKS IN QUALITATIVE RESEARCH
To this point, we have discussed the paradigms that form the philosophical bases of qualitative research as well as various strategies of inquiry. Regardless of the type of research paradigms and strategies utilized, the researcher must (1) gather data, and (2) analyze and present the data, two topics we will explore in depth next.

GATHERING DATA
Although there are many ways to collect data in qualitative research, we discuss three primary sources here: observations, interviews, and existing materials. Wolcott (1992) refers to these sources in the active voice as experiencing, enquiring, and examining, respectively, which provides a glimpse of the qualitative researcher's state of mind in these three activities.


MAKING OBSERVATIONS
The Nature of Observations. Observations are obtained by a trained observer who is present and involved in the phenomenon of interest and who makes reports of his or her observations. There are several advantages of observations in qualitative research. First, because observers can experience firsthand the transactions among persons in the field, they need not rely on retrospective reports of the participants, which could be clouded by the participants' involvement in the situation. In the vernacular, observations take place "where the action is." Second, by being close to the phenomenon, observers can feel, as well as understand, the situation. The emotion present in any situation is likely to be attenuated as time passes; the oft-spoken expression "You had to be there [to understand]" aptly summarizes this advantage of observations. Moreover, informants may not be willing or able to talk about sensitive material, may not be aware of important events, or may consider important transactions to be routine. Third, deep involvement in the process over time will allow researchers to develop conceptualizations that can be examined subsequently. In contrast to quantitative research, investigators will not have a conjecture a priori that will be confirmed or disconfirmed; qualitative researchers may, over the course of the study, come to identify themes based on the observations of the participants and to relate those themes to each other, but such themes and their relations grow out of observation. This inductive process is the basis of some methods of qualitative research, such as grounded theory.
Qualitative observers traditionally have followed the dictum of nonintervention (Adler & Adler, 1994), which holds that the observer does not influence the phenomenon, but acts as a recorder of events. The observer neither asks the participants questions, nor poses problems to be solved, nor suggests solutions to dilemmas. The stream of life goes on in exactly the way that it would had the observer not been present, although qualitative researchers are aware that observers can have an effect on the observed.
Degree of Involvement of Observers. Observers can be described by their degree of involvement in the context being observed. Traditionally, the involvement of the observer has been described as ranging from complete observer, to observer-as-participant, to participant-as-observer, to complete participant (Gold, 1958).
The complete observer is entirely outside the context and would most likely be undetected by the participants. Observing public behaviors in a park by sitting on a bench would fit this category. The observer-as-participant is known to the participants, but is clearly identified as a researcher and does not cross over to membership in the group being observed or to friendship. In his observations of chemistry laboratories, Wampold and colleagues (1995) were observers-as-participants. Their presence in the laboratories during work hours was clearly recognized by the chemists, but the observers did not interact with the chemists during these periods.
Either of these two observational stances (namely, complete observer or observer-as-participant) could be used in quantitative research; the data would be transformed to numbers that indicate the degree of presence of some construct. For example, observations in a park might be used to quantify parental engagement, which could be defined as minutes spent by parent and child in face-to-face interaction. In psychotherapy research, behavior often is classified into one of many codes, with the goal of relating the degree to which the coded behaviors occurred with some other construct. As we will discuss further, even when the observer in qualitative research is not involved with the participants, the observations are not used to quantify some preconceived construct, but instead are used to gain a better understanding of the naturalistic context.
In modern conceptualizations of qualitative research, the unique contribution of participants-as-observers is their insider perspective: As participants as observers, they can experience what the other participants are experiencing, gaining an understanding of the context in ways that nonparticipant observers cannot. Participants-as-observers sometimes fill roles within the group, although "without fully committing themselves to members' values and goals" (Adler & Adler, 1994, p. 380). Many studies of schools have involved participants as observers who, in roles such as teachers or coaches (see, for example, Adler & Adler, 1991), become important people in the lives of the participants.
The final observational role is the complete participant, in which the investigator is a full-fledged member of the group before the research begins. Memoirs are examples of qualitative products created by complete participants in an activity. In an example from psychotherapy, Freud took the role of complete participant in his description of his cases. The degree to which one becomes a participant obviously depends on the phenomenon being studied. As Patton (1987) noted, one cannot become chemically addicted in order to become a participant in drug treatment programs, although one could be involved in such programs in ancillary ways (for example, as a staff member).
Whatever the role of the observer, the "challenge is to combine participation and observation so as to become capable of understanding the experience as an insider while describing the experience for outsiders" (Patton, 1987, p. 75). The tension between participant and observer is ever present and again emphasizes the necessity of research teams, which can process this tension and use it to produce an informative product.
Methods of Observations Obtaining data through observations involves several steps. In the first step, the observer must select the phenomenon and its setting. Care must be taken in making this selection. At times this selection will be made opportunistically—because the setting is available and accessible. At other times the researcher may be invited to observe because the group values or desires a consultation by the researcher, which is the case in qualitative program evaluation. It is important that the setting be appropriate given the goals of the investigation. In quantitative research, the important consideration is the representativeness of the sample, whereas in qualitative research, greater emphasis is put on selecting settings strategically so that the data are meaningful. For example, instead of choosing a representative (that is, average) therapist to study intensively, a qualitative researcher may be more interested in studying therapists identified as successful, powerful, charismatic, or even unsuccessful. Patton (1990) provides a useful typology of sampling, including the purpose of various procedures.
The second step involves training investigators to be skilled and careful observers. Observers must be trained to attend to detail, to separate the mundane from the important, to write highly descriptive field notes, to be sufficiently knowledgeable to make sense of the context, and to be open to reconciling observations with those of other research team members. Consider a study of a K-12, urban school that primarily serves African-American students and consistently produces high achievement in their graduates (Pressley, Raphael, Gallagher, & DiBella, 2004). The researchers observed classroom interactions (e.g., teachers' instructional styles, students' responses to teachers' instruction, artifacts displayed in the classrooms) and nonclassroom activities (e.g., lunch hour in the cafeteria, conversations between the teachers/principal and students/parents in the hallway). The knowledge and skills in teaching, learning, and qualitative method facilitated the observers' teamwork in recording detailed field notes, sifting through a large amount of data, and sorting out the factors that contributed to the success of the school.
The third step is to gain access to the context being studied. Gaining access, of course, varies greatly, depending on the people being studied. Observing public behavior involves no special arrangements other than finding a suitable vantage point, but studying secret societies, especially those engaged in illegal Or undesirable behavior, will be nearly impossible because these groups are unlikely to give their informed consent to participate in a study. Fortunately, most foci of qualitative research in counseling involve groups that are amenable to being observed. Often, trust is the key to gaining entree to a group. If its formal leader is trusted, then his or her support can be sufficient for acceptance and cooperation from the group. However, if the leader's personal influence is tenuous and relations strained, then the recommendation to accept the researcher may be counterproductive and the observations undermined.
The-fourth step involves deciding the time and duration of observations. At the outset, the observations are relatively unfocused and the researcher is getting the "lay of the land," and thus the duration of observation is as important as the time of the observation. Generally, observations should be taken at various times so as not to miss something particular to a certain time. For example, a study of work climate should involve observations from all shifts and during various days of the week. Clearly, the focus of the study should guide the researcher; for example, a researcher interested in how people from various disciplines negotiate their roles in multidisciplinary settings will want to observe instances in which the disciplines work together (for example, in staff meetings or multidisciplinary work groups). As the research progresses, the data will suggest ways that the observers can focus their attention. In the study of chemistry groups, Wampold and colleagues (1995) found that instances of conflict demonstrated how the array of chemists' social skills guided their actions, and consequently observations were arranged during times when conflicts were more likely (for example, during transitions from one research team to another on an expensive apparatus).
The fifth and final step is to collect the data. Most frequently observational data are the researcher's field notes (memos) taken during or immediately after the observations. Field notes are descriptions of everything relevant to understanding the phenomenon. Because relevance is not always clear in the beginning, initially field notes are likely to contain everything that happened. The novice observer will feel overwhelmed, but it should be recognized that almost anything missed will be repeated over and over again. As the observations become more focused, so will the field notes. For example, if it is observed that one group is superior in multidisciplinary work, the observations may be focused on how this is established as new members join the group; the field notes would similarly be focused on the process of transmitting power to incoming persons.
Field notes should contain basic descriptions of the setting—time, physical setting, persons present, the purpose of activity, and so forth—as well as complete descriptions of the interactions among the participants. Patton (1987) provides a good contrast between vague, generalized notes and detailed, concrete notes:
Vague: The new client was uneasy waiting for her intake interview. Detailed: At first the client sat very stiffly on the chair next to the receptionist's desk. She picked up a magazine and let the pages flutter through her fingers very quickly without really looking at any of the pages. She set the magazine down, looked at her watch, pulled her skirt clown, and picked up the magazine again. This time she didn't look at the magazine. She set it back down, took out a cigarette and began smoking. She would watch the receptionist out of the corner of her eye, and then look down at the magazine, and back up at the two or three other people waiting in the room. Her eyes moved from people to the magazine to the cigarette to the people to the magazine in rapid succession. She avoided eye contact. When her name was finally called she jumped like she was startled. (p. 93)

The latter description is more complete and involves little inference on the part of the observer, whereas the vague description both involves the inference that the client was uneasy and lacks data to enable confirmation of this inference at a later time. People's conversations should be recorded as close to verbatim as possible. Audiotapes can be used to supplement field notes if possible. Field notes should contain the observer's interpretations of events, but these interpretations should be so labeled to distinguish them from descriptions. Field notes might also contain working hypotheses, suggestions for interviewers, and so forth.
Undoubtedly, observations and field notes are influenced by the personal constructions of the observers. Multiple observers cross-checking their descriptions and their interpretations are vital for the integrity of observations, because trained observers may see a situation very differently. Acknowledging and honoring these perspectives is part and parcel of qualitative research.