Barroso 2003

7/28/2019 Barroso 2003

1/27

http://wjn.sagepub.com

Western Journal of Nursing Research

DOI: 10.1177/0193945902250034

2003; 25; 153West J Nurs ResJulie Barroso, Claudia J. Gollop, Margarete Sandelowski, Janet Meynell, Patricia F. Pearce and Linda J. Collins

The Challenges of Searching for and Retrieving Qualitative Studies

http://wjn.sagepub.com/cgi/content/abstract/25/2/153The online version of this article can be found at:

Published by:

http://www.sagepublications.com

On behalf of:Midwest Nursing Research Society

can be found at:Western Journal of Nursing ResearchAdditional services and information for

http://wjn.sagepub.com/cgi/alertsEmail Alerts:

http://wjn.sagepub.com/subscriptionsSubscriptions:

http://www.sagepub.com/journalsReprints.navReprints:

http://www.sagepub.com/journalsPermissions.navPermissions:

http://wjn.sagepub.com/cgi/content/refs/25/2/153Citations

at UNIV OF TEXAS AUSTIN on November 17, 2009http://wjn.sagepub.comDownloaded from
http://www.mnrs.org/http://wjn.sagepub.com/cgi/alertshttp://wjn.sagepub.com/cgi/alertshttp://wjn.sagepub.com/subscriptionshttp://wjn.sagepub.com/subscriptionshttp://www.sagepub.com/journalsReprints.navhttp://www.sagepub.com/journalsReprints.navhttp://www.sagepub.com/journalsPermissions.navhttp://www.sagepub.com/journalsPermissions.navhttp://wjn.sagepub.com/cgi/content/refs/25/2/153http://wjn.sagepub.com/http://wjn.sagepub.com/http://wjn.sagepub.com/http://wjn.sagepub.com/http://wjn.sagepub.com/cgi/content/refs/25/2/153http://www.sagepub.com/journalsPermissions.navhttp://www.sagepub.com/journalsReprints.navhttp://wjn.sagepub.com/subscriptionshttp://wjn.sagepub.com/cgi/alertshttp://www.mnrs.org/

7/28/2019 Barroso 2003

2/27

DOI:10.1177/0193945902250034ARTICLEWesternJournalofNursingResearchMarch2003,Vol.25,No.2

The Challenges of Searching for andRetrieving Qualitative Studies

1

Julie Barroso

Claudia J. Gollop

Margarete Sandelowski

Janet Meynell

Patricia F. Pearce

Linda J. Collins

The authorspurpose in this article is to report the results of their search and retrieval efforts to

date in an ongoing study to develop theprocedural, analytic,and interpretivetechniquesto con-

duct qualitative meta-synthesis projects, using studies on women with HIV infection as the

method case.Forresearchers conductingqualitativemeta-synthesisprojects,the idealgoal is to

retrieve all of the relevant studies in a fieldnot simply a sample of them. Batess model of

berrypickingis used asthe frameworkto describe thetechniques used to conduct these searches.

The authors discuss, in particular, the challenges of working with bibliographic databases,

including choosing which databases to search, learning about the idiosyncrasies of working

with each database, developing a list of search terms, and refining inclusion cri teria regarding

which studies to include in the meta-synthesis. Recommendations are given for searchers and

writers of qualitative research.

Keywords: qualitative meta-synthesis; qualitative research; research integration; information

retrieval; information-seeking behavior; computer literature searching

There has been an unprecedented proliferation of qualitative studies on

various aspects of health, illness, and life transitions across the disciplines

153

Western Journal of Nursing Research, 2003, 25(2), 153-178

Julie Barroso, Ph.D., A.N.P., C.S., Assistant Professor, School of Nursing, University ofNorth Carolina at Chapel Hill; Claudia J.Gollop, Ph.D., Associate Professor, School ofInformation andLibrary Science, University of North Carolina at Chapel Hill;MargareteSandelowski, Ph.D., R.N., F.A.A.N., Cary C. Boshamer Distinguished Professor ofNursing, School of Nursing, University of North Carolina at Chapel Hill; Janet Meynell,M.S.N., R.N., School of Nursing, University of North Carolina at Chapel Hill; Patricia F.Pearce, RN, MSN, CS-FNP, School of Nursing, University of North Carolina at ChapelHill;LindaJ.Collins, M.S.L.S., UserServicesLibrarian,Health SciencesLibrary, Univer-

sity of North Carolina at Chapel Hill.

DOI: 10.1177/0193945902250034

2003 Sage Publications

http://wjn.sagepub.com/http://wjn.sagepub.com/http://wjn.sagepub.com/http://wjn.sagepub.com/

7/28/2019 Barroso 2003

3/27

and health-related publication venues. These studies contain important find-

ings about a diverse range of health issues, including, most notably, personal

and cultural constructions of disease, prevention, and risk; living with and

managing the effects (including the treatment effects) of an array of chronic

conditions; and decision making on and responses to beginning- and end-of-

life technological interventions. The challenge for qualitative researchers

now is no longer finding places to publish their work but rather finding all of

the studies that are relevant to producing their work. This challenge is espe-

cially compelling for researchers conducting qualitative meta-synthesis pro-

jects in which the ideal goal is to retrieve all of the relevant studies in a field

not simply a sample of them. The most important threat to the validity of any

research integration effort is to failto conduct a sufficientlyexhaustivesearch

(Cooper, 1998).Our purpose in this article is to report the results of our own search and

retrieval efforts to date in an ongoing study to develop the procedural, ana-

lytic, and interpretive techniques to conduct qualitative meta-synthesis pro-

jects, using studies on women with HIV infection as the method case.2 We

chose this area of research because a sufficient number of qualitative studies

exist to warrant meta-synthesis, and it is a field of great significance to

womens health and nursing practice.

Qualitative meta-synthesis is a distinctive kind of research integration

study in which the findings from completed qualitative studies in a target

area are combined. Like phenomenology, ethnography, and grounded the-

ory, the term qualitative meta-synthesisrefers to both an interpretiveproduct

and the analytic processes by which the findings of studies are aggregated,

integrated, summarized, or otherwise put together (Estabrooks, Field, &Morse, 1994; Jensen & Allen, 1996; Kearney, 1998; Noblit & Hare, 1988;

Sandelowski,Docherty,& Emden,1997; Schrieber, Crooks, & Stern,1997).

The aim of qualitative meta-synthesis is to create larger interpretive render-

ings of all of the studies examined in a target domain that remain faithful to

the interpretive rendering in each particular study. Although it can be con-

sidered an analog to meta-analysis (Glass, McGraw, & Smith, 1981) in that

there is a shared interest in synthesizing empirical studies (Noblit & Hare,

1988, p. 10) and a shared desire to use a systematic, comprehensive, and

communicable approach to research integration, qualitative meta-synthesis

is not about averaging or reducing findings to a common metric (Wolf,

1986, p. 33). The aim of qualitative meta-synthesis is instead to create larger

interpretive renderings of all of the studies examined in a target domain that

remain faithful to the interpretive rendering in each particular study.

154 Western Journal of Nursing Research


7/28/2019 Barroso 2003

4/27

In the first year of our meta-synthesis project, we focused, in part, on

searching for and retrieving thepublished articles and book chapters, as well

as theunpublished theses, dissertations, and conference papers that will ulti-

mately comprise the bibliographic sample for the meta-synthesis study.

Near the end of the first year of our project, a healthy young woman died

while participating in a clinical trial at a major medical center. The New York

Times reported that the chief researcher of this clinical trial had searched the

scientific literature andfound no evidence that thedrug given to hermight be

unsafe. He missed some reports of safety problems published in the 1950s

(Death at the Hands of Science, 2001). This sadly highlights the impera-

tive, in any kind of research, to know what others have published. In a recent

report of a meta-ethnography of qualitative research on lay experiences of

diabetes and diabetes care, the authors (Campbell et al., in press) noted thedifficulties they encountered when trying to retrieve all of the literature rele-

vant to their work. They commented that they had learned of another pub-

lished meta-ethnography of diabetes only after their own study was com-

pleted and that the authors of this other report had identified only three out of

the seven papers that they themselves had included in their search. The

authors of another recent article on the challenges of retrieving qualitative

research noted that MEDLINE does not include the term qualitative as a

MeSH term (Dixon-Woods, Fitzpatrick, & Roberts, 2001), thereby hinder-

ing retrieval of qualitative research. MeSH (Medical Subject Headings)

terms are the controlled vocabulary of words and phrases used to index

much of thebiomedical literature. Evans (2002) also noted thedifficulties in

searching electronic databases for qualitative research.

We describe here some of thechallenges we encountered in searching forand retrieving qualitative studies and offer recommendations for both

searchers and writers of qualitative studies that will make this work as fruit-

ful and efficient for their purposes as their resources, expanding informa-

tion, and changing information technology will permit.

HOW WE BEGAN

A metasynthesis project typically begins by defining the substantive,

methodological, and temporal boundaries for study. We began our study in

June 2000 with the following inclusion criteria: all reports of qualitative

studies on women in the United States with HIV infection published

between 1993the year in which the first of these studies appeared as

March 2003, Vol. 25, No. 2 155


7/28/2019 Barroso 2003

5/27

indicatedby our searchat that time (wehave since found an article published

in1991)and March 1, 2003, 2 years before the anticipated end of the meta-

synthesis project in 2005. At the time of grant writing, we expected to find

approximately 35 studies, including theses and dissertations.

We defined qualitative research as encompassing a highly diverse array

of orientations to and strategies for inquiry that generally concern them-

selves with how the social world is interpreted, understood, experienced,

and produced (Mason, 1996). Data generation and analysis techniques are

flexible, reflexive, context-sensitive, and case-oriented. We specifically

excluded: (a) qualitative studies in which there are no human participants

per se (as, for example, in discourse, qualitative content, semiotic, or other

qualitative analyses of media representations of women with HIV infec-

tion); (b) qualitative studies about nonseropositive women and their experi-ences as mothers, partners, relatives, friends, and/or caregivers of seroposi-

tive persons; and (c) qualitative adjuncts (such as open-ended questions at

theendof a structured questionnaire) to largely quantitative studies. We also

excluded journalistic or other nonresearch accounts of HIV-positive

women. We created a list of method search terms that we thought would

reflect our criteria well and would therefore capture the studies we were

seeking. They included, in alphabetical order, case study, constant compari-

son, content analysis, conversation analysis, descriptive study, discourse

analysis, ethnography, exploratory study, feminist, focus group, grounded

theory, hermeneutic, interview, narrative/narrative analysis, naturalistic

study, participant observation, phenomenology, qualitative method, quali-

tative research, and thematic analysis. Where appropriate, we truncated the

terms to broaden the search.

BROWSING AND BERRYPICKING

As Bates (1989) pointed out, as more resources are brought online, the

searcher has a more complex search environment to consider, both in terms

of types of sources to use and search techniques to employ with these

resources. Browsing can be distinguished from berrypicking in that brows-

ing is less directed and may be useful to stimulate a searchers thinking, but

browsing can be an important component of several of the berrypicking

strategies we describe below.Bates (1989)arguedthat berrypickingis closer

to the actual behavior of searchers than traditional models of information

retrieval. With this model, considered a classic among information



7/28/2019 Barroso 2003

6/27

scientists, at each stage of the search, the searcher is not just modifying

search terms to get a better match for a single query. The query itself is con-

tinually shifting, in part or whole. This is called an evolving search and is

closer to the actual behavior of searchers. The query is satisfied not by a sin-

gle final retrieved set but by a series of selections of individual references

and bits of information at each stage of the ever-changing search. This is not

a straight, linear process. Just as a berrypicker would meander through the

bushes, looking for clumps of berries (Bates, 1989), the searcher wanders

through the information forest, changing directions as needed to follow up

on various leads and shifts in thinking.

Bates (1989) identified six strategies that are used in berrypicking; we

describe these now along with how we actually executed these strategies

during our search and retrieval work to obtain qualitative studies of womenwith HIV infection. We focus in particular on subject searches in bibliogra-

phies and abstracting and indexing services, as this was the most complex

strategy, and the one most commonly used by searchers.

Footnote Chasing

This technique includes following up on footnotes found in books and

articles of interest and therefore moving backward through reference lists; it

is also called backward chaining (Bates, 1989). In our project, we reviewed

more than 3,600 citations from the reference lists of research reports, and

other articles, books, and anthologies on women with HIV infection. We

searched the reference lists for anything that was remotely connected to

women and HIV infection, even if the actual document being searched was

not a research report that might be included in the meta-synthesis.

Citation Searching

Citation searching, or forward chaining, is where thesearcher beginswith

a citation, finds out who cites it by looking it up in a citation database, and

thus leaps forward (Bates, 1989). We have not yet done this type of search-

ing as we wanted to have most of what we anticipatewill be the final sample

prior to completing this step. We also surmised that this may not be a very

fruitful technique forus, given howrecently most of thearticles we areusing

were published. It usually takes some time for an article to disseminate

widely and therefore to be cited by others in more recent works.

March 2003, Vol. 25, No. 2 157


7/28/2019 Barroso 2003

7/27

Journal Runs

With this technique, once the researchers have identified a central journal

in an area, these journals are systematically hand-searched straight through

the relevant volume years (Bates, 1989). Such a technique guarantees com-

plete recall within that journal. We have done this with two journals, Quali-

tative Health Researchand theJournal of the Association of Nurses in AIDS

Care. We felt that the first is an excellent source for qualitative work, and the

second is a good source for research conducted with HIV-positive people

and is the flagship journal of the Association of Nurses in AIDS Care. These

two journals, although not precisely central to the area, were the closest to

being central. We did not find any articles in either of these journals, using

this journal-run technique, that were not also found using bibliographicdatabases.

Area Scanning

Bates (1989) described this technique as browsing the materials that are

physically colocated with materials located earlier in a search, and as a

widely used and effective technique. Once we had identified several books

that dealt with HIV-positive women, we scanned the shelf locations of these

books to find other books that might be of interest to us. We supplemented

area scanning with electronic searches as well, searching the electronic card

catalog of our library system looking for library materials that could be

found by using women and HIV as our subject search terms.

Author Searching

Based on relevant publications found previously, this strategy entails

searching by author name to see if an author has done any other work on the

same topic (Bates, 1989). We have had good results using this technique,

particularly in the iterative searching for dissertations and/or the articles

generated from those dissertations. Of the authors whose published work(s)

we hadalready retrieved, we searched by each author name fordissertations,

as many of the researchers in the area of women and HIV infection are recent

graduates of doctoral programs. We located several dissertations on women

with HIV infection using this method. We also searched for articles in every

database using each authors name to see if we had not yet retrieved a pub-lished article.



7/28/2019 Barroso 2003

8/27

Subject Searches in Bibliographies

and Abstracting and Indexing Services

This technique,which involves searching bibliographicdatabases for rel-

evant works in an area, has been the one we have focused on and will be dis-

cussed in greater detail. We chose bibliographic databases that were accessi-

ble to us, that covered a wide range of disciplines, and that we knew or

surmised would yield reports of qualitative studies on women with HIV

infection. Our access to these databases was through the library computer

network at theUniversity of North Carolina at Chapel Hill. We identified 16

bibliographic databases for our search; these are listed in Table 1, along with

the features of each of these databases. The servers we used were primarily

OVID (Ovid Technologies), SilverPlatter (Silver Platter Information), andOCLC FirstSearch (OnLine Computer Library Center, Inc.).

We have thus far located 71 published and unpublished studies (theses

and dissertations) that meet our inclusion criteria, approximately twice the

number of studies we expected to find. We have thus far been unsuccessful

in obtaining the proceedings from three national conferences on women

with HIV infection. Table 2 illustrates the yield of the bibliographic data-

base searches we have completed to date, specifying the number of citations

that met the search terms (5,829) and the number of citations that actually

met the criteria for inclusion into our bibliographic sample (213), for a total

yield of 3.66%. Most citations were found in more than one database, and

these numbers represent electronic searches only.

The yield of a search is highly related to the task complexity, the user, and

the system(s) involved (e.g., Haynes, Wilczynski, McKibbon, Walker, &Sinclair, 1994; Marchionini, 1995; Meadow, Boyce, & Kraft, 2000;

Sutcliffe, Ennis, & Watkinson, 2000). The most commonly used perfor-

mance measures in information retrieval are based on precision and recall.

The recall for a set of retrieved documents is the percentage of relevant doc-

uments in the database that have been retrieved. The precision, given a set of

retrieved documents, is the percentage of documents that have been

retrieved that are relevant. Searches may be categorized into two forms

based on precision and recall: high-recall searches, in which most or all of

thedocumentson a topic shouldbe retrieved, andhigh-precision searches, in

which the set of documents to be retrieved, often consisting of a small num-

ber of documents, is expected to be composed of predominantly relevant

documents (Losee, 2000; Marchionini, 1995). In strictly quantitative terms,

March 2003, Vol. 25, No. 2 159

(text continues on p. 165)


7/28/2019 Barroso 2003

9/27

TABLE 1: Electronic Databases Selected for Searches

Standardized Natural

Producer Thesaurus Language Language

Content/ How or With Searching for Searching

Database Kind of to Content Standardized Qualitative and Boolean Special

Name Information Access Provider Keywords? Research? Operators? Features

AcademicSearch Elite

Scholarly,multidisciplinarydatabasecovering avariety of

academic areas

EBSCOHost EBSCOPublishing

Yes Yes Yes Full text for mo1,500 journacovers majonewspapersmore than 1

peer-reviewjournals

AIDS Informa-tion Online(AIDSLINE)

Specialized indexto AIDS-relatedresearch,includingclinical andhealth policyissues

Now incorpo-rated intoOVIDMEDLINEor PubMed

National Libraryof Medicine(part of NIH)

Yes No Yes Journals plusgovernmenttechnical repmeetings abspecialpublicationsAV materialstheses

AnthropologicalIndex Online

Gateway toarticles forperiodicals inthe BritishMuseumsMankind Library

Web Department ofEthnographyat the BritishMuseum

No No NoincompleteBoolean

Specialized co

160

atUNIV

OFTEXA

SA

USTIN

onN

ov

em

ber17

,2

009

http://w

jn.s

agepub.c

om

Downlo

adedfrom
http://wjn.sagepub.com/http://wjn.sagepub.com/http://wjn.sagepub.com/http://wjn.sagepub.com/http://wjn.sagepub.com/http://wjn.sagepub.com/http://wjn.sagepub.com/http://wjn.sagepub.com/http://wjn.sagepub.com/http://wjn.sagepub.com/http://wjn.sagepub.com/http://wjn.sagepub.com/http://wjn.sagepub.com/http://wjn.sagepub.com/http://wjn.sagepub.com/http://wjn.sagepub.com/http://wjn.sagepub.com/http://wjn.sagepub.com/http://wjn.sagepub.com/http://wjn.sagepub.com/http://wjn.sagepub.com/http://wjn.sagepub.com/http://wjn.sagepub.com/http://wjn.sagepub.com/http://wjn.sagepub.com/

7/28/2019 Barroso 2003

10/27

AnthropologicalLiterature

An index toperiodicals andedited worksemphasis is onarchaeology,anthropology(biological,physical,cultural, andsocial), andlinguistics

CD-ROM HarvardUniversity(TozzerLibrary)

Yes No Yes Internationalcoverage; inabout 1,000

journals andmonographs

Black Studies Internationalindex to AfricanAmerican,African, andAfricanDiasporanstudies

CD-ROM Gale Group No No Yes Based, in partholdings of tSchomburg for ResearchBlack Cultur

CumulativeIndex toNursing andAllied HealthLiterature(CINAHL)

Bibliographicreferences andabstracts onnursing andallied healthtopics

OVID CINAHLInformationSystems

Yes Yesqualitativestudies

Yes Journals plusconferenceproceedingsmaterials,educationalsoftware, nustandards, adissertations

DissertationAbstractsIndex (DAI)

Index todissertationsaccepted ataccredited U.S.Institutionssince 1861

OCLCFirstSearch

ProQuestInformationand Learning

Yes No Yes Selectively covmasters theCanadiandissertationsBritish and oEuropeandissertations

161

atUNIV

OFTEXA

SA

USTIN

onN

ov

em

ber17

,2

009

http://w

jn.s

agepub.c

om

Downlo

adedfrom

7/28/2019 Barroso 2003

11/27

EducationalResourceInformationCenter (ERIC)

Database ofjournal andreport literaturein educationand related

disciplines

Silverplatter EducationalResourceInformationCenter

Yes Yes Yes Includes curricmaterials sucourses of sbooks, confepapers, and

MEDLINE Database ofreferences andabstracts cover-ing the fields ofmedicine, nurs-ing, dentistry,veterinary medi-cine, the healthcare system,and the preclini-cal sciences

OVID National Libraryof Medicine(part of NIH)

Yes No Yes International coage; includesapproximatelcurrent biome

journals publthe United Stand 70 foreigcountries

PsycInfo Index to theresearch andclinical literaturein psychology,psychiatry, andrelateddisciplines

Silverplatter American Psy-chologicalAssociation

Yes Nounlessauthor listsas keyphrase(not acontrolleddescriptor)

Yes International coage; includesscholarly jourdissertationsand book cha

162

TABLE 1 (continued)






atUNIV

OFTEXA

SA

USTIN

onN

ov

em

ber17

,2

009

http://w

jn.s

agepub.c

om

Downlo

adedfrom

7/28/2019 Barroso 2003

12/27

Public AffairsInformationService (PAIS)

Bibliographicindex to theliterature onpublic policy,social policy,and the socialsciences ingeneral

Silverplatter Public AffairsInformation Ser-vice, Inc.

Yes No Yes International coage; includesnals, books, gment documestatistical comtions, and coreports

PubMed Database ofreferences andabstracts cover-ing the fields ofmedicine, nurs-ing, dentistry,veterinary medi-cine, the healthcare system,and the preclini-

cal sciences

Web atPubmed.gov

National Libraryof Medicine(part of NIH)

Yes No Yes International coage; includesapproximatelcurrent biome

journals publthe United Stand 70 foreigtries; PubMeMEDLINE filesome additio

tions to life scjournals

Social ScienceAbstracts(SocSciAbstracts)

Database of refer-ences andabstracts cover-ing anthropol-ogy, economics,geography, law,political science,psychology, andsociology

OCLCFirstSearch

H. W. WilsonCompany

No No Yes More than 350international,lish-languageodicals in thesciences

Social WorkAbstracts

Index to the litera-ture in socialwork andrelateddisciplines

Silverplatter National Associa-tion of SocialWorkers, Inc.

Yes Yes Yes International coage; includes

journals, disstions, and otmaterials

163

atUNIV

OFTEXA

SA

USTIN

onN

ov

em

ber17

,2

009

http://w

jn.s

agepub.c

om

Downlo

adedfrom

7/28/2019 Barroso 2003

13/27

SociologicalAbstracts(Sociofile)

Database oftheoretical andapplied studiescovering sociol-ogy and related

disciplines

Silverplatter Cambridge Scien-tific Abstracts

Yes Yesquali-tativemethods

Yes International coage; includesthan 2,500 jomeeting abstbook reviews

dissertationsWomens

StudiesInterdisciplinary

database index-ing approxi-mately 100 Eng-lish language

journals

CD-ROM(CD editionof theWomensStudiesIndex)

Gale Group No No Yes International coage; includesarly articles, breviews, and materials

Note: NIH = National Institutes of Health.

TABLE 1 (continued)






164

atUNIV

OFTEXA

SA

USTIN

onN

ov

em

ber17

,2

009

http://w

jn.s

agepub.c

om

Downlo

adedfrom

7/28/2019 Barroso 2003

14/27

our searches were not high in precision because we intentionally traded pre-

cision to heighten recall to ensure an exhaustive search.

Working with and around bibliographic databases. Bibliographic data-bases arenot stableentities yielding fixed search results. They mayexist one

dayas independent entities andthenext dayas inaccessible, extinct,or incor-

porated into another database. For example, in our library system, access to

CD-ROM-based PsycLit was removed soon after we began our searches,

leaving us accessto anonlinedatabase, PsycINFO.Journal articles that could

be found in AIDSLINE, once an independent database, were incorporated

into PubMed as of September of 2001. This change meant that searches on

our topic, qualitative studies on women with HIV infection, resulted in inex-

act recall because PubMed now indexes other health-related journals, in

addition to those formerly found in AIDSLINE, and that are searched simul-

taneously. Databases are updated at different intervals; some are updated

daily, whereas others are updated only quarterlyor even yearly, thereby con-

tributing to a citation list that may not be current. The searcher is also not astable, unchanging entity; variations in searchers activities and fatigue or

March 2003, Vol. 25, No. 2 165

TABLE 2: Search Yields by Bibliographic Database

Bibliographic Citations Citations for %

Database Identified Inclusion Yield

Academic Search Elite 63 3 4.76

AIDS Information Online 338 27 6.96

Anthropological Index Online 98 1 1.02

Anthropological Literature 338 1 0.30

Black Studies 558 0 0.00

Cumulative Index to Nursing and

Allied Health Literature 504 38 7.54

Dissertation Abstracts Index 181 9 4.97

Educational Resource Information Center 54 0 0.00

MEDLINE 1,101 28 2.54

PsycINFO 452 42 9.29Public Affairs Information Service 5 0 0.00

PubMed 1,514 31 2.05

Social Science Abstracts 46 6 13.04

Sociofile 285 14 4.91

Social Work Abstracts 41 4 9.76

Womens Studies 201 9 4.48

Total 5,829 213 3.66


7/28/2019 Barroso 2003

15/27

frustration will influence the quality and amount of time spent conducting a

search and search results.

One of our first challenges was to improve our skills in interacting with

these database systems and to work with them to extract the information we

were seeking. Each database system offered numerous options for search-

ing, and we had to spend the time investigating which of them would have

the best yield in terms of recall. Each database access provider has an infor-

mation page with its parameters for updates and inclusion dates. We had to

become more familiar with the features of each database system, including

thesauri and the use of special options, such as Boolean and proximity oper-

ators. Almost everyone who has searched a database is familiar with

Boolean operators, in which the words and, or, and notare placed between

search terms to specify their desired relationship to each other. Proximityoperators, used to further customize a search, can request that terms appear

adjacent to each other, in a specific order, near each other, separated by a

given number of intervening words, or that the terms appear in the same

fields (e.g., title field, abstract field).

Two examples of the complexityof searching options that we had to mas-

ter were that PsycINFO has a thesaurus or controlled vocabulary, which is

further divided into detailed subheadings used to search for more specific

information. MEDLINE uses MeSH terms, as well as less specific subject

subheadings; these headings are the keys that unlock the medical literature

(Coletti & Bleich, 2001). Decisions thus had to be made regarding which

terms touse and how tobest use them. For instance, inOVID-basedsystems,

a searcher can explode a search term, which allows the searcher to broaden

the search and to further explore the subject matter and related subjects, or asearcher can focus a search term (to narrow the search) to look for citations

in which the search term is a major point of the article. We chose to explode

our searches, thus maximizing the use of search terms and MeSH headings.

A searcher canuseBoolean or other operators in these andother databasesas

well. Boolean operators allow the searcher to use set theory, which is com-

monly recognized in the form of a Venn diagram of overlapping circles that

indicate the and, or, and notfunctions, to help define the items that will be

retrievedby a search. Theuseof special operators (e.g., proximityoperators)

is generallyavailableon allof thedatabases, but some of thedatabasesallow

more flexible use than others. Expert-level searching generally entails

searches that are conducted by professional search intermediaries who

employ appropriate syntax, appropriate use of Boolean and other operators,

and inclusion of desired delimiters or expanders for the search.



7/28/2019 Barroso 2003

16/27

There were also variations within and among the databases in the map-

ping of the search terms. In mapping, the system offers additional terminol-

ogy from the controlled vocabulary of a database in response to a search

term entered by the user (Jasco, 1996). Mapping is a relatively recent addi-

tion to the usual line of searching features. Within each search, the search

terms may be combined with the options selected from the main or basic

database search screen, or terms may be entered into the advanced search

window, where more options are offered (e.g., publication year, language).

These are combined within the system to form what is known as the search

string. The system translates the information into the underlying computer

query language in operation, maps the information to the appropriate areas,

searches for citations satisfying the criteria, and returns the information to

the screen.Although there were commonalities among many of the databases, we

learned that the vocabulary that was successful in searching for studies in

one database was not necessarily successful in searching in another data-

base. Where a database resides can influence searching and subsequent

results. Thesame database provided by twodifferentvendors could conceiv-

ably return different results of the same search because searching features

vary among database vendors. Minor idiosyncrasies, such as whether a

comma should be used between an authors last name and first name (or ini-

tial) vary substantially among databases and can be irritating to the searcher,

in addition to providing an invalid search result. Few database systems

return a message telling searchers that they have made a mistake in entering

search terms that a systems language cannot map appropriately. A no

records foundmessage may not reflect the absence of records so much as asystems inability to map the search terms used. If the search term is mapped

to the best judgmentof the programming within the database, totally irrele-

vant records mayappear. Thesearcher maythus be faced with no citationsor

many irrelevant ones.

Although all the databases we used had characteristics that were support-

ive of our searches, we found OVID to be the most useful access provider/

vendor for several reasons. There are options in OVID that enable the

searcher to see the hierarchical mappings and relationships of individual

search terms and to select or deselect the terms. Any search term can be

expanded or focused in a variety of ways, and the related mappings in that

process can be viewed. Once visualized, we hadsome control in the selection/

exclusion of a particular mapping. Merging of several individual search

results to form a comprehensive searchfor thephenomenon of interest is rel-

atively friendly in OVID; this helped to reduce the number of irrelevant

March 2003, Vol. 25, No. 2 167


7/28/2019 Barroso 2003

17/27

citations that we had to sift through to find relevant works. OVID provides a

seemingly endless number of related subject headings and keywords that

represent the indexing of each citation and provide the searcher with a rich

sourceof terms to use. We found this aspect particularlyhelpful when devel-

oping an appropriate search term list. OVID provides a clear error message

when a searcher has made an error in handling search terms or applications.

OVID also permits the complex use of nested Boolean operators and allows

the user to save and e-mail the results in a form that is friendly to importing

directly into a reference manager database such as ProCite or EndNotes.

Of the 71 published articles and unpublished theses and dissertations that

met our search criteria and that we have retrieved thus far, CINAHL and

PsycINFO yielded the largest number of citations (43 each), followed by

PubMed and AIDSLine, which each had the same 38 citations. There wasconsiderable overlap among these databases. For example, one article

(Dunbar, Mueller, Medina, & Wolf, 1998) was cited in 12 different data-

bases. Of the 71 studies, 32 could also be found through the more laborious

footnote chasing discussed above. Most of the overlap or duplicate search

results may be attributed to inclusion practices of the various indexing and

abstracting database producers.

The nature of a research project will determine the nature of the search

process. Because one of the major objectives in our method project was

to explore the various processes involved in conducting qualitative meta-

syntheses, our search process was necessarily more laborious. Our primary

obligation in this project is to do the work required that would put us in a

credible position to advise other researchers of work they need not do. In the

case of search and retrieval, we deliberately overworked to test how muchwork was actually necessary to produce a valid search result.

Databases to find databases. Knowing wherepublished works likelyto be

relevant to a project are indexed is critical to finding those works. We used

two means to track down citations and to develop our list of databases to

search for the project. The publicly accessible Jointly Administered Knowl-

edgeEnvironment (jakeat http://jake.med.yale.edu/) databaseat YaleUni-

versity (hosted by the Cushing/Whitney Medical Library at the Yale Univer-

sity Schoolof Medicine) improved ourability to identify theelectronic index

or indices inwhicha particular journal couldbe found.We also utilizeddirect

links to some full text articles that are provided in jake.

An additional source of information regarding journal indexing is

Ulrichs Periodicals Directory, which is a paper-based directory available

in most libraries that can also be accessed electronically by subscription.



7/28/2019 Barroso 2003

18/27

Both have cross-referencing of journals. But Ulrichs Periodicals Directory

(2000) contains many more journal titles and is considered a more compre-

hensive compilation of journal indexing than jake (Chudnov, Crooker, &

Parker, 2000; L. Collins, personal communication, March 22, 2001).

Ulrichs also has information about the peer-reviewed status of indexed

journals.

Developing a list of searchterms for the meta-synthesis. The development

of search terms appropriate to thearea of interest is essential for theeffective

use of bibliographic databases. Some databases can only be searched by a

controlled vocabulary (or thesaurusor list of available terms), otherswork by

natural language processing, which allows common or noncontrolled terms

to be searched. Still other databases are searchable using both methods. Acontrolled vocabulary database means that the effective search terms in the

database must be on the list within the database. This list, however, is not

always readily apparent. In contrast, natural language processing allows for

the searcher to type in virtually any term or phrase, and the database will

attempt application.

Most databases allow the searcher to select the location to which the

search terms, words, or phrases in the citation should be applied: for exam-

ple, the title, author, abstract, keywords, or words anywhere (meaning every

textual piece related to a citation that is in the database). We used the words

anywhere option to ensure a comprehensive search, which is a common

technique in information retrieval (Coletti & Bleich, 2001; Marchionini,

1995; Meadow et al., 2000). By utilizing words anywhere in our searches,

we knew we would capture many more citations than would be relevant, butthis gave us more confidence that we would capture more of the relevant

literature.

We initially thought the search term list would entail a relatively small

number of terms, including women, females, mothers, HIV, AIDS, qualita-

tive research, naturalistic research, grounded theory, phenomenology, eth-

nography, and interview. After a few trial searches, however, we realized

that these search terms had neither the specificity nor sensitivity we desired;

that is, there were a large number of irrelevant citations produced, although

relevant citations that we already knew of were not captured. We had to

become more knowledgeable about the underlying mapping patterns for

each database to manipulate search terms appropriately. For example, in

some databases (e.g., PsycINFO), the search term women maps to human

female. Human female includes mothers, sisters, widows, daughters, and

wives, as well as battered females, working women, and female criminals.

March 2003, Vol. 25, No. 2 169


7/28/2019 Barroso 2003

19/27

The database may be searching for key words as provided by the author, the

indexer, or contained in the title, abstract, or in the full text of the article. In

OVID-based MEDLINE and CINAHL, these mappings can be selected or

deselected, depending on the focus of the search and retrieval activity. We

therefore manipulated our search terms according to the parameters of the

systems we searched.

Qualitative research. An especially challenging problem was the diffi-

culty locating exclusively qualitative research reports. As a search term,

qualitative researchyielded few relevant citations and many irrelevant ones,

such as qualitative laboratory assays. We accordingly generated a more

refined list of qualitative research search terms. The challenge was to use

search terms and operators that spoke the language of each of the biblio-graphic databases while still representing qualitative research. Qualitative

research in MEDLINE maps to a large numberof subject headings: research,

research design, nursing methodology research, nursing research, data inter-

pretation, health services research, nursing, and myocardial infarction. Fol-

lowing hierarchical mapping, each of these subheadings then maps to other

terms. In OVID-based CINAHL, qualitative research maps to qualitative

studies. As stated earlier, we also searched using the many terms that indi-

cated that the methodology used was qualitative in nature, and we truncated

the terms as well to cast a wider net. The most important thing we realized,

particularly because we are doing qualitative meta-synthesis work, is that

most bibliographic databases do not index articles according to research

methodology. Table 1 lists whichof the16 databaseswe used indexes accord-

ing to research methodology.

Determining relevance. Even when we refined our search terms and tech-

niques, usingthe berrypicking process, there werestill hundreds (about 20%)

of citations retrieved that could not be clearly excluded on the basis of title

and abstract alone. We therefore developed a process to track the decisions

we made about these citations. This process, which deals with both substan-

tiveandmethodological issues, is graphically shown in Figure1. Once a cita-

tion was obtained, we were sometimes able to exclude it based on the title

alone. If we were unsure, we checked the abstract. A citation could then be

excluded at this point as not meeting the criteria, or it could merit further

investigation. If further investigation was warranted, the full article was

obtained. At that point, the citation was included, excluded, or its status

remained uncertain. Uncertain citations then led to a negotiation of consen-

sus on its status and further refinement and delineation of inclusion criteria.



7/28/2019 Barroso 2003

20/27

March 2003, Vol. 25, No. 2 171

Descendency

Ancestry

Decision Point

Specific Step, w/ citation document (electronic/paper)

Study Electronic Management Database

Specific Step, w/ or w/o citation document

Initial/Ending Activity

Search Citation Database

Consult for Consensus

Search Term List

Include Unsure

Check Title

Check Abstract

Check Full Article

Include Unsure Exclude

Include Unsure Exclude

Check Author CitationsCheck Bibliography

Include Exclude

Exclusion from Bibliographic SampleInclusion for Bibliographic Sample

Exclude

Figure 1. Search, retrieval, and validation process.

NOTE: Reference for graphic symbols drawn from Harris (1999, p. 156).


7/28/2019 Barroso 2003

21/27

Once any article was finally included as part of the bibliographic sample for

themeta-synthesis, its reference list was checked for other studies that might

meet our criteria, andtheauthorwassearched in allof thedifferentdatabases

to seeif he/shehadpublished somethingelse that metthe criteria,both strate-

gies described above in the berrypicking techniques.

As shown in Figure 1, a full text copy was appraised for every study that

was included in the meta-synthesis to ensure it met all study criteria. This

often required obtainingcopiesof articles through interlibrary loan for a per-

item fee. Although we ultimately excluded many of these articles, we con-

sidered this neither a waste of time nor money as we gained articles that will

be useful to us in contextualizing the results of our meta-synthesis.

Searching for and retrieving unpublished theses and dissertations pre-

sented a special challenge to determining relevance. We located most of thetheses and dissertations we were interested in through Dissertation Abstracts

International (DAI), although some of them appeared in other databases as

well, such as PsycINFO or Sociofile. Our initial search of DAI yielded 8 the-

ses and 181 dissertations. The search for unpublished works can be a costly

venture that some searchers may not be able to afford because most of these

works would have to be purchased to determine their relevance to a project.

In most cases, these works could not be obtained on interlibrary loan. If the

researchers cannot afford to purchase these works, this may contribute to a

less complete bibliographic sample for meta-synthesis.

MANAGING INFORMATION

Projects such as this generate volumes of information that require effi-

cient management and cataloging. The foundation for ensuring valid proce-

dures and results is the establishment of a clear audit trail documenting all

procedural moves and decision-making points. For example, we had to find

a way to track citations that were removed immediately, that were ultimately

accepted, and that required more investigation. All the database systems we

used supported the transfer of our search results to a printer and/or to saving

them on a disk or hard drive. With the exception of Black Studies, Womens

Studies, Anthropology Literature, Anthropology Index, and PAIS, all the

database systems we used supported the transfer of our results through e-

mail. OVID-provided searches are also readily transferable into ProCite or

other reference manager software packages, making management of the

search results highly efficient and effective. Reference manager software,

such as ProCite and EndNote, proved invaluable to us in managing the



7/28/2019 Barroso 2003

22/27

results of our searches. We created several Access databases for managing

those citations that required more extensive tracking and handling, such as

Dissertation Abstracts. Information can be readily transferred between

EndNote, ProCite, and Access through several mechanisms. The publicly

available PubMed system is directly searchable, and our search results were

directly downloadable through our universitys ProCite license.

OTHER THREATS TO VALID SEARCHES

Despite the steps we have taken to ensure a valid search, there are other

threats to validity over which a searcher has no control. For example, most

database systems (e.g., MEDLINE ) have human indexers, and as with anywork performed by humans, they can make mistakes entering information

into an electronic citation database (Coletti & Bleich, 2001; Hjorland, 2001;

Meadow et al., 2000). Other types of errors, from authors and editors, can

affect the validity of a search as well. In a study that examined the accuracy

of entries in MEDLINE by searching for misspelled text words, a total of

200 citations were retrieved from 10 selected search terms. The authors

found that most (141/200; 71%) misspelled words occurred within the

abstract only and that 98 of the200articles with misspelled text words might

be missed if a searcher conducted a MEDLINE search using the correctly

spelled word alone without the heading (Ray & Vermeulen, 1996).

CONCLUSIONS AND RECOMMENDATIONS

We have tried here to fix in time and space a process that is in reality a

moving target. Acknowledging the situational dynamic of searching for and

retrieving qualitative studies, we offer recommendations for searchers and

writers of qualitative reports that we hope will help them manage it.We have

a suggestion for indexers, however, based on Evanss (2002) assertion that

the effectiveness of electronic database searchers would be improved

through the development of a qualitative research search filter. This type of

methodological filter is a predetermined search strategy that uses terms

related to research design to identify all those studies using the research

method of interest to the searcher. The purpose of such a filter would be to

retrieve only those studies most relevant to the review (in this case, thoseusing qualitative methods) (Evans, 2002).

March 2003, Vol. 25, No. 2 173


7/28/2019 Barroso 2003

23/27

Recommendations for Searchers

Because there is a different calculus involved with every search, search-

ers must figure out the specific calculus for their particular project or topic.

This involves configuring a numberof factors. Searchers must determine the

resources they will have available to them for their search, including the

amount of time they can expend on locating relevant works, the amount of

money they can spend to retrieve full-text copies of works, and the richness

of their search environment, including the availability of expert librarians,

servers, and citation databases. This portion of the calculus will help search-

ers define their overall search strategy: that is, how to best approach the

search process and whether to emphasize precision or recall. Because our

research project demanded that we investigate and then recommend the bestprocedures to conduct qualitative meta-synthesis studies, we deliberately

chose more laborious and time-consuming search approaches to determine

whether less laborious or time-consuming methods would produce the same

results. For example, we reviewed more than 3,600 citations appearing in

the reference lists of reports we accepted into our bibliographic sample, in

addition to those in other works about women with HIV infection. This

yielded 32 studies that were ultimately included in our sample. Yet, of these

32 studies, all could be found electronically as well. We have located 1 study

that appeared as a chapter in a book (Kass & Faden, 1996) and was not avail-

able as a citation on any of the databases we searched. We will not know

until we are further into the meta-synthesis process itself if we will find any

other studies exclusively through a hand search of a books chapters or the

reference lists in a book or in articles. Of greater importance are the conse-quences of not finding studies by doing electronic searches alone. Would the

studies that a meta-synthesist does not find change the results? We might

find that, in our case, the time we spent in doing this ancestry work was

unnecessary.

Because qualitative research spans disciplines in the sciences, humani-

ties, and arts, searchers must also determine which databases in these varied

disciplines they must have access to so as to achieve a valid search. This in

turn requires searchers to learn the intricacies and idiosyncrasies of working

with and around these databases. Searchers may need to spend more time

than they had anticipated enhancing their search and retrieval skills, which

can be done with tutorials available on most bibliographic databases. We

have seen, in hindsight, the need for an information specialist on a research

project such as ours. The irony is that with so much information available atour fingertips, retrieving relevant materials has become harder, in part,

because of advances in technology, but also because of the lack of standard-



7/28/2019 Barroso 2003

24/27

ization across database systems. Every bibliographic database has its own

unique features, and even these features can change with some frequency.

Searching in and of itself is an area of specialized knowledge, and meta-

synthesists would be advised to have an information specialist as part of the

team.

Searchers will also need to have the flexibility to match the ever-changing

situational dynamic of the search process itself. They will likely have to

rethink and refine their inclusion and exclusion criteria as the search begins

to yield studies. Because there is no fixed endpoint to searching, searchers

must decide at what point they will be satisfied with their results and, more

important, at what point they can stop searching and still meet the criterion

for having conducted an exhaustive search that all research integration pro-

jects require. Because there is a lag time between the arbitrary end of asearch and the completion of a meta-synthesis of the findings from the

reports retrieved from that search, researchers must decidewhether they will

update their search before submitting their work for publication.

Recommendations for Writers

Although we have emphasized searching and searchers here, we found

that writers play a key role in determining whether their work will be found.

Evans (2002), in discussing the importance of systematic reviews for

evidence-based practice, noted that although methodological search filters

have worked well with locating randomized controlled trials, the unique

characteristics of qualitative research limit the easy translation of these

search techniques. He noted that the lack of a clear description of the method

used in the title or abstract of a study makes it difficult to find qualitative

studies when doing electronic searches.

Although constrained by publication venues and database systems, writ-

ers could contribute to the retrievability of their work and to research dis-

semination and utilization (e.g., Barry, 1998; Coletti & Bleich, 2001) if they

attended more thoughtfully to those aspects of research reports that are most

visible in electronic systems, includingkeywords and theconstructionof the

title and abstract. For example, we recommend that, whenever possible,

researchers use the term qualitative researchas a keyword in every report of

a study using any qualitative methodologies. Although more challenging to

achieve, given the regular use of established thesauri and subheadings, we

would also recommend that indexers use the same term to index any studyconducted using any qualitative methodology. Searchers could then search

under that term rather than having to search with terms covering every kind

March 2003, Vol. 25, No. 2 175


7/28/2019 Barroso 2003

25/27

of qualitative method or technique. Because oneof thegoals of research is to

disseminate the findings to an audience, writers need to do everything they

can to ensure that they are providing appropriate terms to aid searchers.

Writers shouldalso be aware that searchershave a heightened expectancy

with bibliographic databases. Searchers become impatient while waiting

even a few seconds because of the fundamental change in temporal expecta-

tions fostered by computers. Electronic environments raise searchers

expectations for information access. But these expectations interact with

searchers physiology and psychology. Searchers must still perceive, pro-

cess, and comprehend information if they are to achieve their goals

(Marchionini, 1995). Writers need to know this to ensure that what is rele-

vant about their work will be as immediately accessible to the searcher as

possible.Searchers makecritical decisions aboutwhether to investigate cita-tions further based on what is in front of them on a computer screen. If a cita-

tion appears immediately to be irrelevant, searchers may immediately

exclude it from consideration, and thecycle of research, research dissemina-

tion, and research utilization may also, therefore, be halted.

In summary, although there is still much debateon what qualitative meta-

synthesisprojects entail andhow they shouldbe conducted, there is no argu-

ment that any meta-synthesis of qualitative findings rests on a sufficiently

exhaustive search. Qualitative research presents both searchers and writers

of qualitative reports many challenges that must be met to make the best and

the most of the qualitative findings that offer so much to the health and well-

being of the people nurses serve.

NOTES

1.Theauthorsacknowledgethe assistanceof JeanC. Blackwelland Lynne D.Morris,Health

Sciences Library, the University of North Carolina at Chapel Hill.

2. This study, titled Analytic Techniques for Qualitative Metasynthesis, is supported by

grant no. R01 NR04907 from the National Institute of Nursing Research.

REFERENCES

Barry, C. L. (1998). Document representations and clues to document relevance. Journal of the

American Society for Information Science, 49, 1293-1303.

Bates, M. J. (1989).The design of browsing andberrypicking techniquesfor online searchinter-face. Online Review, 13, 407-424.



7/28/2019 Barroso 2003

26/27

Campbell, R., Pound, P., Pope, C., Britten, N., Pill, R., Morgan, M., et al. (in press). Evaluating

meta-ethnography: A synthesis of qualitative research on lay experiences of diabetes and

diabetes care. Social Science and Medicine.

Chudnov, D.,Crooker, C.,& Parker,K. (2000).Jake: Overviewand status report.Serials Review,

26(4), 12-17.

Coletti,M. H.,& Bleich,H. L. (2001). Medical subject headings used tosearchbiomedicalliter-

ature. Journal of the American Medical Informatics Association, 8, 317-323.

Cooper, H. (1998). Synthesizing research. Thousand Oaks, CA: Sage.

Death at the hands of science. (2001, July 31). Retrieved from www.nytimes.com

Dixon-Woods, M., Fitzpatrick, R., & Roberts, K. (2001). Including qualitative research in sys-

tematic reviews:Opportunitiesand problems.Journal of Evaluation in Clinical Practice, 7,

125-133.

Dunbar,H. T., Mueller, C.W.,Medina, C.,& Wolf,T. (1998).Psychologicaland spiritual growth

in women living with HIV. Social Work: Journal of the National Association of Social

Workers, 43(2), 144-154.

Estabrooks, C. A., Field, P. A., & Morse, J. M. (1994). Aggregating qualitative findings: Anapproach to theory development. Qualitative Health Research, 4, 503-511.

Evans, D. (2002). Database searches for qualitative research.Journal of the Medical Librarians

Association, 90, 290-293.

Glass, G. V., McGraw, B., & Smith, M. L. (1981). Meta-analysis in social research. Beverly

Hills, CA: Sage.

Harris, H. (1999). Information graphics: A comprehensive illustrated reference. New York:

Oxford University Press.

Haynes, R.B.,Wilczynski,N., McKibbon, A.,Walker,C. J.,& Sinclair, J. C. (1994).Developing

optimal searchstrategies for detectingclinically sound studies in MEDLINE.Journal of the

American Medical Informatics Association, 1(6), 447-458.

Hjorland,B. (2001). Toward a theory of aboutness, subject topicality, theme,domain, field con-

tent . . . andrelevance.Journalof the American Society for Information Science and Technol-

ogy, 52, 774-778.

Jasco, P. (1996). Ovid web gateway: Nobody does it better. Online, 20(6), 24-30.

Jensen, L. A., & Allen, M. N. (1996). Meta-synthesis of qualitativefindings. QualitativeHealthResearch, 6, 553-560.

Kass, N., & Faden, R. (1996). In womens words: The values and lived experiences of HIV-

infected women. In R. R. Faden & N. E. Kass (Eds.), HIV, AIDS and childbearing: Public

policy, private lives (pp. 426-446). New York: Oxford University Press.

Kearney, M. H. (1998). Ready-to-wear: Discovering grounded formal theory. Research in

Nursing & Health, 21, 179-186.

Losee, R. M. (2000). When information retrieval measures agree about the relative quality of

documentrankings.Journalof the American Society for InformationScience, 51, 834-840.

Marchionini,G. (1995).Information seeking in electronic environments. NewYork: Cambridge

University Press.

Mason, J. (1996). Qualitative researching. London: Sage.

Meadow, C. T., Boyce, B. R., & Kraft, D. H. (2000). Text information retrieval systems. San

Diego, CA: Academic Press.

Noblit, G.W., & Hare,R. D.(1988).Meta-ethnographysynthesizing qualitative studies. Beverly

Hills, CA: Sage.Ray, J. G.,& Vermeulen,M. J. (1996).Words, words, words: Mizspellin andMEDLINE.British

Medical Journal, 313, 1658-1659.

March 2003, Vol. 25, No. 2 177


7/28/2019 Barroso 2003

27/27

Sandelowski,M., Docherty, S.,& Emden, C. (1997).Qualitativemetasynthesis: Issuesand tech-

niques. Research in Nursing and Health, 20, 365-371.

Schrieber, R.,Crooks,D.,& Stern, P. N. (1997).Qualitative meta-analysis.In J. M. Morse (Ed.),

Completing a qualitative project(pp. 311-326). Thousand Oaks, CA: Sage.

Sutcliffe, A. G., Ennis, M., & Watkinson, S. J. (2000). Empirical studies of end-user searching.

Journal of the American Society for Information Science, 51, 1211-1231.

Ulrichs Periodicals Directory. (2000). New York: Bowker.

Wolf, F. M. (1986). Meta-analysis: Quantitative methods for research synthesis. Beverly Hills,

CA: Sage.


Barroso 2003

Documents

Transcript of Barroso 2003