We describe methods as follows:

• Search to identify criteria sets
• Analysis of criteria in sets within the project’s scope
• Evaluation of criteria sets’ utility
  

Search to identify criteria sets:
In January and February 2003, we conducted World Wide Web (WWW) and literature searches to identify criteria sets that are used to evaluate the credibility of websites that offer health information. We:

• Defined the purpose and scope of the search to identify criteria of relevance
• Searched websites
  • Used various search engines (Google, MSN, Webcrawler, and, Yahoo) and search terms (health website evaluation criteria, website evaluation criteria, website evaluation)
  • For each website search, evaluated the first 100 results. Each website was assigned an initial grade based on its description; a final grade, based on a review of website contents.
  • Followed links on websites judged to be relevant
• Searched the medical literature
  • Searched Medline (the on-line reference search of the National Library of Medicine) and other sites using the same or similar terms used for website searches
  • Evaluated the relevance of retrieved articles, first based on the title of the article; then, for articles retrieved as relevant, the content of the article
  • Checked and retrieved the references of relevant articles
    • Found through searches
  • Found serendipitously
  • On hand at the start of the project
    

Through these searches we identified:

• Articles that were reports of health website evaluation criteria or relevant evaluations of health websites
• 151 criteria sets (125 websites and 26 articles) among the approximately 411 websites URLs and 60 citations identified.
  

We included a criteria set in our final data set if it was used specifically for evaluating health information on websites or if the criteria set contained elements that may be used to evaluate health information on websites. We judged 22 (20 websites and 2 articles) of the 151 criteria sets to be directly relevant to the comprehensive evaluation of health websites; 7 were a derivative of another criteria set [1]. See table 1

Analysis of criteria
In order to describe criteria in sets within the projects’ scope, we developed a set of meta-criteria. To construct this set, we:

• Developed a conceptual framework
• Reviewed the criteria in the 22 sets to be analyzed
• Developed a logical structure to array them
• Arrayed criteria in sets according to this structure
• Elaborated the structure within this context to include criteria that might reasonably be contained within criteria sets to be analyzed.
  

To describe criteria used or suggested for use in evaluating health websites, we:

• Parsed the criteria expressed in the 22 criteria sets. This step was necessary because authors often provided narratives or made conjoint statements ( for example, "this….and….that")
• Coded these criteria elements according to our meta-criteria, which encompassed every criterion in all 22 criteria sets.
• Reprinted criteria sets from our database to check
  • The accuracy of captured statements by comparing database records to the original criteria sets.
  • We had correctly classified headings, definitions, and elaboration’s of criteria that appeared in criteria sets to distinguish them from criteria.
• Reported criteria according to meta-criteria set codes in order to check the accuracy, specificity, and consistency of coding (updated the meta-criteria set and recoded records accordingly).
• Counted the number of times a specific meta-criterion appeared in criteria sets.
  

The meta-criteria had 9 domains, and a total of 115 criteria elements (appendix 1 provides a detailed description of the meta-criteria):

• Content relevance
• Content accessibility
• Content selection
• Content validity
• Content interchange
• Site transparency
• Links
• Quality assurance
• Safeguards.
  

Evaluation of criteria sets’ utility
To evaluate the utility of identified criteria sets to consumers and their consumer-friendliness, we:

• Developed a basic set of evaluation criteria, consisting of 22 criteria elements organized into 6 domains. See appendix 2 for the set of evaluation criteria.
• Asked 2 project staff members to rate independently the identified criteria sets using these evaluation criteria: each criterion was scored "met," "not met" or "partially met."
• Compared ratings.
• Resolved any differences in ratings by asking a third project staff to determine the final rating.
• Checked the logical consistency of ratings, and resolved any inconsistencies by examining individual ratings.
• Tabulated evaluation results.
  

To return to introduction