Appendix 1: Meta -Criteria

Evaluation of criteria for assessing health websites
Appendix 1: Meta -Criteria

1. Content Relevance

1.1 Purpose/Mission/Scope: Does the site state its purpose? If yes: Is it to provide information on health topics to consumers?

1.1.1 Is the mission stated? Does the mission statement convey purpose & scope of website?

1.1.1 Scope: Does the site state its scope?

1.1.2 Objective/Goals: Does the site state its objective?

1.1.3 Does the site indicate the title/subject of site?

1.2 Audience: Does the site state its intended audience?

1.3 Relevance: Can one tell easily if what’s on the site is relevant to one’s search purpose?

1.4 Breadth and depth, Coverage, Completeness, Uniqueness

1.4.1 Is the information unique or can it be found elsewhere?

1.5 Site consists only of links (i.e. no substantive information)?

1.6 Content relevance, NEC

2 Content accessibility, presentation,design:

2.1 Comprehensibility: Is the site prepared at a level that is useful to the intended audience?

2.1.1 Or, failing any statement about intended audience, is it comprehensible by the average consumer? Note: Consumers will be likely able to assess whether or not they can comprehend content (the materials on the site).

2.2 Navigability, logical organization:

2.2.1 Is the information needed to use the evaluation checklist easily accessible on the site?

2.2.1 Is site content easily accessible? E.g., doesn’t require plug-ins

2.2.3 Does it contain a search engine (when such is necessary to access easily the site’s content)?

2.2.2 Is the site easily navigable?

2.2.4 Is website designed in a user-friendly manner?

2.2.5 Is there a site index, table of contents?

2.3 Does the site organize information in a consumer-friendly way, e.g., by health topic (e.g., cures for breast cancer), rather than simply providing research reports that the consumer has to search?

2.4 Readability: Is text, graphics, video etc consumer-friendly (readable)?

2.4.1 Does site contain (useful) images?

2.4.2 Is site available in text only mode?

2.4.3 If user can’t access images, etc, is information still "complete" (useful)?

2.5 Downloadability

2.6 Are there conditions to accessing content?

2.6.1 Is there a fee to access content?

2.6.2 Does the user have to register?

2.6.3 Does accessing content require acceptance of cookies

2.7 Presentation of website: Is it appealing?

2.9 Content accessibility, NEC

3.0 Content selection/editorial policy (an important aspect of evaluating potential bias).

3.1 Editorial policy, site evaluation process, peer review process: Does the site describe how it searched for and selected the information displayed from the universe of information that exists? Note: This concept might be captured by "editorial policy."

3.1.1 Was the search strategy adequate to the site’s purpose? If no, purpose stated, was it reasonable? Does the site display or communicate search results?

3.1.2 Was the selection method adequate to the site’s purpose? If no, purpose stated, was it reasonable? Does the site display or communicate selection results?

Note: A consumer-oriented rating system would need to assess these aspects of the website. A critical issue is sponsors’ or advertisers’ participation in these processes.

3.2 Conflict of interest, bias, disclosure of sponsorship, funding sources

3.2.1 Does the site provide a conflict of interest policy?

3.2.2 Specifically, does site report any relationship between authors of content and the owner, sponsor, or operator of the website?

3.9 Content selection, NEC

4.0 Content validity/accuracy

4.1 Does the site differentiate clearly between (or it's obvious what are) essays/ commentaries/ "opinions" and reports/ information/ "facts?"

4.1.1 Is the information/content thorough and detailed?

4.1.2 Is the information accurate?

4.2 Does the site give the authors' information for "opinions" and/or reports of "facts" ("articles" presented on the website)

4.2.1 Author’s name

4.2.2 Author’s credentials (qualifications and experience)

4.3 Does the site indicate the date the "opinion" or report of "facts" was written or last updated?

4.4 For reports of "facts" (articles), does the site give the source of specific information it is communicating?
For each source referenced, does the citation to the source give all of the following information (regardless of whether or not there is a link, to promote user-friendliness):

4.4.1 Link to the source (best practice, whether on- or off-site)? Enough information to obtain it?

4.4.2 Authors of the source

4.4.3 Date the source information was generated or the date the report containing the referenced information was published.

4.4.4 Date information was posted on the site

4.4.5 Bibliography/resource list

4.4.6 Type of source e.g. government, educational institution etc

Note: Specific information refers to a statement that is the equivalent of, eg, "this treatment has been shown to cure breast cancer."

4.5 Does the site indicate whether or not it has evaluated specific information (source) before displaying it on the website?

4.5.1 Is so, does the site describe how the specific information was evaluated?

4.5.2 Are the methods adequate to their purpose?

4.6 Does the site state the evidence grade of specific information?

4.6.1 If so, does the site describe how the evidence grade was assessed?

4.6.2 Are the methods adequate to their purpose?

Note: Evidence grade refers to the likely validity of the information for its purpose based on an assessment of the type of research study that produced it, and, better yet, an assessment of the study, e.g., based on assessing the research report (the adequacy of methods in relation to reported findings)

4.7 Does the site indicate the generalizability or limitations on applicability of specific information, or provide similar assistance to consumers in its use for some spectrum of purposes?

4.8 Does the site state whether or not it carries advertising (if not obvious that it does)?

4.8.1 If the site carries advertising, does the site clearly distinguish between advertising and content?

4.8.2 Does it provide its advertising policy?

4.8.3 Is its advertising policy appropriate for consumers (eg, doesn't accept advertisement for products with dubious claims of effectiveness)?

Note: The term "advertising" includes any payment for display of (or link to) content.

4.9 Content validity, NEC

5 Content interchange, interactivity

5.1 Is there a way for consumers to request additional information? Does the site state how long it usually takes for the inquiry to be answered? Note: This is useful mostly for rating websites.

5.2 Does the site offer decision support technology (a way that consumers can access information specific to their needs or concerns), such as, eg, a consumer self-assessment tool?

5.2.1 If so, does the site describe the expert system used for this purpose? Is the system adequate for its purpose?

5.2.2 Are specific caveats given regarding the system’s use (eg, consumer must give honest information if results are to be meaningful) and limitation of relevant information?

5.2.3 Does the site describe how the DST was developed and validated? If so, was the method sufficient to validate the DST?

5.2.4 When was the DST last updated and/or validated?

5.3 If the site offers exchange of information between consumers and health professionals, does the site describe their credentials (qualifications, experience, etc). Does the site indicate whether or not it has verified their credentials? If so, were the methods appropriate?

5.3.1 Is it obvious to the consumers when they are (and are not) interacting with a specific health professional (whose credentials have been described)?

5.9 Content interchange, NEC

6 Site transparency/credibility/currency

6.1 Does the site say who owns or pays for (sponsors) or created it?

6.1.1 Does it list a physical location, telephone number, contact person, etc?

6.2 Does the site say who runs it (if not the sponsor)?

6.1.2 Does it list a physical location, telephone number, contact person, etc?

6.3 Does the site provide the email address (or other contact information) for the webmaster (or other contact at the sponsoring organization)?

6.4 Does the site describe the sponsor’s purpose or business?

6.5 Does the site disclose sources of financing of the sponsoring organization? Of the operating organization?

6.6 Currency

6.6.1 When was the site first created?

6.6.2 How frequently or often is the site/site content updated?

6.6.3 Does the site indicate when it was last updated? Better yet, when specific sections were last updated?

6.6.4 Does the site indicate date until which the content/information is valid?

6.7 Is the site essential "complete" (no or few "under-construction" banners)?

6.8 If site is moderated, is moderator’s identity disclosed?

6.8.1 Are credentials, affiliations, etc of the moderator disclosed?

6.8.2 Is method used to select moderator disclosed?

6.9 Site transparency, NEC

7 Links

7.1 Does the site link to other sites as sources of health information? If so, does it state the basis for displaying links (eg, its linkage policy)?

7.2 Does the site indicate whether or not it has evaluated the adequacy of sites to which it displays links for additional health information?

7.2.1 If so, does the site describe the basis for such evaluation? Subsidiary questions: Are the methods adequate to their purpose?

7.3 Does the site provide appropriate descriptions, disclosures, and disclaimers about links?

7.4 Are consumers alerted when they leave the site (eg, to go to a linked site)? E.g., does the site differentiate internal from external hyperlinks?

7.5 Does the site report its policy for periodically checking the integrity of links (and removing non-functioning) links?

7.6 Does the website contain email links to the authors of the content?

7.9 Links, NEC

8 QA/QI

8.1 Does the site describe any quality management or QA/QI process?

8.1.1 Does QA/QI include accessibility?

8.1.2 Relevance

8.1.3 Content selection, validity, and/or interchange?

8.1.4 Transparency/credibility?

8.1.5 Links?

8.1.6 Safeguards?

8.2 If so, are QA/QI processes adequate to their purpose?

8.3 Does the site permit easy feedback from consumers, including registration of complaints and compliments?

9 Safeguards

9.1 Does the site display terms and conditions of use of the website (including, when applicable, services offered through the website), including users’ rights? Are they appropriate?

9.2 Does the site display its privacy or confidentiality policy?

9.2.1 Is it adequate to the task of informing consumers appropriately of what information is captured about their visit or request for information and what happens to it subsequently (including, eg, its use to promote products and services)? Note: This information is useful in rating websites (or including in search engines or browser filters), but it may not be useful to consumers because by the time they land on the site it may be too late.

9.3 Disclaimers, privacy policy, transparency or sponsorship, advertising policy: Does the site display disclaimers on use of the information it contains or communicates?

9.3.1 Are they adequate to the task of informing consumers appropriately on general limitations of use of the information (content) displayed on the website?

9.9 Safeguards, NEC

To return to introduction

1.1	Purpose/Mission/Scope: Does the site state its purpose? If yes: Is it to provide information on health topics to consumers?
	1.1.1	Is the mission stated? Does the mission statement convey purpose & scope of website?
	1.1.1	Scope: Does the site state its scope?
	1.1.2	Objective/Goals: Does the site state its objective?
	1.1.3	Does the site indicate the title/subject of site?
1.2	Audience: Does the site state its intended audience?
1.3	Relevance: Can one tell easily if what’s on the site is relevant to one’s search purpose?
1.4	Breadth and depth, Coverage, Completeness, Uniqueness
	1.4.1	Is the information unique or can it be found elsewhere?
1.5	Site consists only of links (i.e. no substantive information)?
1.6	Content relevance, NEC

2.1	Comprehensibility: Is the site prepared at a level that is useful to the intended audience?
	2.1.1	Or, failing any statement about intended audience, is it comprehensible by the average consumer? Note: Consumers will be likely able to assess whether or not they can comprehend content (the materials on the site).
2.2	Navigability, logical organization:
	2.2.1	Is the information needed to use the evaluation checklist easily accessible on the site?
	2.2.1	Is site content easily accessible? E.g., doesn’t require plug-ins
	2.2.3	Does it contain a search engine (when such is necessary to access easily the site’s content)?
	2.2.2	Is the site easily navigable?
	2.2.4	Is website designed in a user-friendly manner?
	2.2.5	Is there a site index, table of contents?
2.3	Does the site organize information in a consumer-friendly way, e.g., by health topic (e.g., cures for breast cancer), rather than simply providing research reports that the consumer has to search?
2.4	Readability: Is text, graphics, video etc consumer-friendly (readable)?
	2.4.1	Does site contain (useful) images?
	2.4.2	Is site available in text only mode?
	2.4.3	If user can’t access images, etc, is information still "complete" (useful)?
2.5	Downloadability
2.6	Are there conditions to accessing content?
	2.6.1	Is there a fee to access content?
	2.6.2	Does the user have to register?
	2.6.3	Does accessing content require acceptance of cookies
2.7	Presentation of website: Is it appealing?
2.9	Content accessibility, NEC

3.1	Editorial policy, site evaluation process, peer review process: Does the site describe how it searched for and selected the information displayed from the universe of information that exists? Note: This concept might be captured by "editorial policy."
	3.1.1	Was the search strategy adequate to the site’s purpose? If no, purpose stated, was it reasonable? Does the site display or communicate search results?
	3.1.2	Was the selection method adequate to the site’s purpose? If no, purpose stated, was it reasonable? Does the site display or communicate selection results?
	Note: A consumer-oriented rating system would need to assess these aspects of the website. A critical issue is sponsors’ or advertisers’ participation in these processes.
3.2	Conflict of interest, bias, disclosure of sponsorship, funding sources
	3.2.1	Does the site provide a conflict of interest policy?
	3.2.2	Specifically, does site report any relationship between authors of content and the owner, sponsor, or operator of the website?
3.9	Content selection, NEC

4.1	Does the site differentiate clearly between (or it's obvious what are) essays/ commentaries/ "opinions" and reports/ information/ "facts?"
	4.1.1	Is the information/content thorough and detailed?
	4.1.2	Is the information accurate?
4.2	Does the site give the authors' information for "opinions" and/or reports of "facts" ("articles" presented on the website)
	4.2.1	Author’s name
	4.2.2	Author’s credentials (qualifications and experience)
4.3	Does the site indicate the date the "opinion" or report of "facts" was written or last updated?
4.4	For reports of "facts" (articles), does the site give the source of specific information it is communicating? For each source referenced, does the citation to the source give all of the following information (regardless of whether or not there is a link, to promote user-friendliness):
	4.4.1	Link to the source (best practice, whether on- or off-site)? Enough information to obtain it?
	4.4.2	Authors of the source
	4.4.3	Date the source information was generated or the date the report containing the referenced information was published.
	4.4.4	Date information was posted on the site
	4.4.5	Bibliography/resource list
	4.4.6	Type of source e.g. government, educational institution etc
		Note: Specific information refers to a statement that is the equivalent of, eg, "this treatment has been shown to cure breast cancer."
4.5	Does the site indicate whether or not it has evaluated specific information (source) before displaying it on the website?
	4.5.1	Is so, does the site describe how the specific information was evaluated?
	4.5.2	Are the methods adequate to their purpose?
4.6	Does the site state the evidence grade of specific information?
	4.6.1	If so, does the site describe how the evidence grade was assessed?
	4.6.2	Are the methods adequate to their purpose?
	Note: Evidence grade refers to the likely validity of the information for its purpose based on an assessment of the type of research study that produced it, and, better yet, an assessment of the study, e.g., based on assessing the research report (the adequacy of methods in relation to reported findings)
4.7	Does the site indicate the generalizability or limitations on applicability of specific information, or provide similar assistance to consumers in its use for some spectrum of purposes?
4.8	Does the site state whether or not it carries advertising (if not obvious that it does)?
	4.8.1	If the site carries advertising, does the site clearly distinguish between advertising and content?
	4.8.2	Does it provide its advertising policy?
	4.8.3	Is its advertising policy appropriate for consumers (eg, doesn't accept advertisement for products with dubious claims of effectiveness)?
	Note: The term "advertising" includes any payment for display of (or link to) content.
4.9	Content validity, NEC

5.1	Is there a way for consumers to request additional information? Does the site state how long it usually takes for the inquiry to be answered? Note: This is useful mostly for rating websites.
5.2	Does the site offer decision support technology (a way that consumers can access information specific to their needs or concerns), such as, eg, a consumer self-assessment tool?
	5.2.1	If so, does the site describe the expert system used for this purpose? Is the system adequate for its purpose?
	5.2.2	Are specific caveats given regarding the system’s use (eg, consumer must give honest information if results are to be meaningful) and limitation of relevant information?
	5.2.3	Does the site describe how the DST was developed and validated? If so, was the method sufficient to validate the DST?
	5.2.4	When was the DST last updated and/or validated?
5.3	If the site offers exchange of information between consumers and health professionals, does the site describe their credentials (qualifications, experience, etc). Does the site indicate whether or not it has verified their credentials? If so, were the methods appropriate?
	5.3.1	Is it obvious to the consumers when they are (and are not) interacting with a specific health professional (whose credentials have been described)?
5.9	Content interchange, NEC

6.1	Does the site say who owns or pays for (sponsors) or created it?
	6.1.1	Does it list a physical location, telephone number, contact person, etc?
6.2	Does the site say who runs it (if not the sponsor)?
	6.1.2	Does it list a physical location, telephone number, contact person, etc?
6.3	Does the site provide the email address (or other contact information) for the webmaster (or other contact at the sponsoring organization)?
6.4	Does the site describe the sponsor’s purpose or business?
6.5	Does the site disclose sources of financing of the sponsoring organization? Of the operating organization?
6.6	Currency
		6.6.1 When was the site first created?
		6.6.2 How frequently or often is the site/site content updated?
		6.6.3 Does the site indicate when it was last updated? Better yet, when specific sections were last updated?
		6.6.4 Does the site indicate date until which the content/information is valid?
6.7	Is the site essential "complete" (no or few "under-construction" banners)?
6.8	If site is moderated, is moderator’s identity disclosed?
	6.8.1	Are credentials, affiliations, etc of the moderator disclosed?
	6.8.2	Is method used to select moderator disclosed?
6.9	Site transparency, NEC

7.1	Does the site link to other sites as sources of health information? If so, does it state the basis for displaying links (eg, its linkage policy)?
7.2	Does the site indicate whether or not it has evaluated the adequacy of sites to which it displays links for additional health information?
	7.2.1	If so, does the site describe the basis for such evaluation? Subsidiary questions: Are the methods adequate to their purpose?
7.3	Does the site provide appropriate descriptions, disclosures, and disclaimers about links?
7.4	Are consumers alerted when they leave the site (eg, to go to a linked site)? E.g., does the site differentiate internal from external hyperlinks?
7.5	Does the site report its policy for periodically checking the integrity of links (and removing non-functioning) links?
7.6	Does the website contain email links to the authors of the content?
7.9	Links, NEC

8.1	Does the site describe any quality management or QA/QI process?
	8.1.1	Does QA/QI include accessibility?
	8.1.2	Relevance
	8.1.3	Content selection, validity, and/or interchange?
	8.1.4	Transparency/credibility?
	8.1.5	Links?
	8.1.6	Safeguards?
8.2	If so, are QA/QI processes adequate to their purpose?
8.3	Does the site permit easy feedback from consumers, including registration of complaints and compliments?

9.1	Does the site display terms and conditions of use of the website (including, when applicable, services offered through the website), including users’ rights? Are they appropriate?
9.2	Does the site display its privacy or confidentiality policy?
	9.2.1	Is it adequate to the task of informing consumers appropriately of what information is captured about their visit or request for information and what happens to it subsequently (including, eg, its use to promote products and services)? Note: This information is useful in rating websites (or including in search engines or browser filters), but it may not be useful to consumers because by the time they land on the site it may be too late.
9.3	Disclaimers, privacy policy, transparency or sponsorship, advertising policy: Does the site display disclaimers on use of the information it contains or communicates?
	9.3.1	Are they adequate to the task of informing consumers appropriately on general limitations of use of the information (content) displayed on the website?
9.9	Safeguards, NEC