Please note authors Eamonn Fahy and Rohan Hardikar have contributed equally to this project as co-first authors.
Copyright © Australasian Medical JournalThe popularity of the Internet has enabled unprecedented access to health information. As a largely unregulated source, there is potential for inconsistency in the quality of information that reaches the patient.
To review the literature relating to the quality indicators of health information for patients on the Internet.
A search of English language literature was conducted using PubMed, Google Scholar and EMBASE databases.
Many articles have been published which assess the quality of information relating to specific medical conditions. Indicators of quality have been defined in an attempt to predict higher quality health information on the Internet. Quality evaluation tools are scoring systems based on indicators of quality. Established tools such as the HONcode may help patients navigate to more reliable information. Google and Wikipedia are important emerging sources of patient health information.
The Internet is crucial for modern dissemination of health information, but it is clear that quality varies significantly between sources. Quality indicators for web-information have been developed but there is no agreed standard yet. We envisage that reliable rating tools, effective search engine ranking and progress in crowd-edited websites will enhance patient access to health information on the Internet.
Keywords: Internet, health, patient information, quality evaluationSince its inception, the internet has rapidly become the single largest source of information worldwide. Health information has seen an exponential increase in accessibility with the advent of this technology. In the past, patient health information had been limited to the discretion of a treating doctor. Today, patients have access to more information than ever. This changing trend has been a double-edged sword. A more involved and informed patient is unarguably a positive change, but sources of information on the internet are varied and largely unregulated. The result is inconsistency in the quality of information that reaches the patient.
In this article, we review the literature relating to the quality of health information on the internet. This is an introduction to what is an expanding field of research.
In the mid-90s, approximately 10 million people had access to the internet, while current estimates exceed two billion users worldwide. 1 A new concept to many, early articles in the medical literature focused on educating healthcare providers about the Internet. 2
Later, studies began to emerge which looked at specific medical topics, and evaluated the quality of information available to patients. The first of these was a 1997 study by Impicciatore et al, 3 which assessed the accuracy and completeness of website information relating to the management of fever in children. According to the authors, out of 41 websites, only four provided complete and accurate information on the topic. This landmark study provided a framework for articles that followed, produced by researchers in various disciplines, exploring the quality of information in their own specialty.
However, there was little consistency between these studies in the criteria used to assess quality of information. This issue became the focus for later research which considered ‘indicators of quality’ 4 - criteria which could serve as predictors of accurate information. The next section considers this literature in detail.
A patient who seeks information on a health topic may find it difficult to gauge the reliability of a webpage. Indicators of quality aim to guide patients towards information of a high calibre. ‘Core indicators’ directly assess the quality of information. They are accuracy, completeness and currency of information. 5 These criteria are the gold standard for evaluating health information on the internet. However, their use is limited because of the need for expert input - evaluating every health website using these criteria would be impossible.
And so, ‘proxy indicators’ were born out of necessity. These are indirect measurements of quality, for example readability, design and disclosures, which aim to predict the quality of information. 5 Proxy indicators are useful because they are objective and easy to assess. Their popularity was demonstrated in a study by Eysenbach, 4 which systematically reviewed 85 articles that had evaluated quality of patient health information. Proxy indicators were more commonly used than core indicators in these studies.
However, proxy indicators may not provide a reliable guide to information quality. For example, readability is based on formulae such as the Flesh-Kincaid reading level, 4 which calculates readability using average sentence length and number of syllables per word. There are clearly advantages to automatic assessment by a computer-based algorithm. However, these formulae are limited because they do not consider other important factors like use of jargon or appropriate tone, and may not accurately gauge practical readability.
Scoring systems have been developed based on the above indicators of quality to standardise evaluation. These scoring systems – or quality evaluation tools – are essentially a set of indicators of quality applied to a website in order to derive a quality score. There are many quality evaluation tools – 273 were identified in a 2004 review, 6 compared to 47 in 1998.7 We will discuss three tools that are widely used today; the HON Code, the JAMA benchmarks and the DISCERN tool.
The oldest quality evaluation tool is the HONcode. The HONcode is an 8-point code of conduct created by the Health on the Net Foundation in 1995, which allows approved websites to display an award-like badge. As of 2010 there are over 7400 sites certified, and it was accepted as the official health information certification body of France in 2007. 8 HON certification is subject to a process of annual review by the HON foundation, who also respond to any reported violations by Internet users. 8
Codes such as these do not claim to be able to evaluate accuracy of medical information, but instead relate to ethics of the information presented. 8 Nevertheless, multiple studies have found that HONcode certification was associated with an improved quality of information and increased reliability. 9 - 11 However, there have also been studies that found many websites displaying the badge to not comply fully with all eight criteria. 12
Another widely used tool, the JAMA (Journal of the American Medical Association) benchmarks, arose from a 1997 article by Silberg et al which presented a set of criteria upon which health information could be evaluated. 13 These included display of authorship, source, date of update, disclosure of ownership, sponsorship, advertising policies and conflict of interest. Despite its popularity as a rating tool, a 2010 study by Barker suggested that websites meeting all JAMA benchmarks actually contained more incorrect information than other sites. 14
| 1. | Authoritative: indicate the qualifications of the authors |
| 2. | Complementarity: information should support, not replace, the doctor-patient relationship, the mission and the audience are explicated |
| 3. | Privacy: Respect the privacy and confidentiality of personal data submitted to the site by the visitor |
| 4. | Attribution: Cite the source(s) of published information, data and medical and health pages |
| 5. | Justifiability: Site must back up claims relating to benefits and performance |
| 6. | Transparency: Accessible presentation, accurate email contact |
| 7. | Financial disclosure: Identify funding sources |
| 8. | Advertising policy: Clearly distinguish advertising from editorial content |
Reproduced with permission. From Evolution of Health Web certification through the HONcode experience (Boyer, Stud Health Technol Inform. 2011;169:53-7.)
DISCERN is a tool created by the Division of Public Health and Primary Care at Oxford University, London. It was launched in 1998, and consists of a 16-item questionnaire for patients to complete for each site they wish to evaluate. This type of system may help to educate patients in terms of quality criteria, as they apply each aspect of the questionnaire to a website. However, leaving this responsibility with the patient may be inconvenient and unreliable. For instance, a study by Rao et al found poor inter-rater reliability when using the DISCERN tool for dengue related health information. 15
Tools such as the HONcode allow compliant websites to display an award-like badge. In fact, most quality evaluation tools exist as part of a ‘seal of approval’. However, many of these seals do not publicly disclose their criteria. A 2004 study showed that participants believed seals of approval to be based on core indicators, rather than the proxy indicators upon which they more commonly are. 15
Although a number of key quality evaluation tools have been developed, it is unclear if these tools can accurately identify quality information. It may be impossible to develop a universal rating tool, given the ever-changing landscape of the Internet. These difficulties have effected a shift in focus to consider more the practical habits of the patient surfing the web. Nothing on the Internet is more habitual for us today than Google and Wikipedia.
The likelihood that a patient will view a particular web site may be influenced by its order of appearance on major search engines. It has been shown that most web users only visit the top 10 web sites listed in search results. 16 With such influence it seems appropriate that we consider the way in which search results are ordered.
Google is the most popular search engine worldwide, and the most-visited site on the internet. 17 One way in which Google ranks its search results is based on link popularity (or ‘PageRankTM’). For any given website, the number of hyperlinks pointing to it from other webpages will improve its rank in a Google search. More ‘important’ linking webpages cast a heavier vote for the rank of the website they are linking to. 18
Link popularity has been explored by a few articles 19 - 20 in terms of its association with the content and quality of websites, posing the question: is it a valid measure of website quality? Studies assessing website information on menopause 19 and post-menopausal osteoporosis 20 found that link popularity did not correlate with measures of quality. The link popularity of breast cancer related websites was found to be associated with type of information rather than quality of content. 12 For example, sites with higher link popularity were more likely to provide updates on breast cancer research and information on legislation.
But link popularity is not the full picture. Google combines PageRank with ‘sophisticated text-matching techniques’ to produce a search result which is both relevant and ‘important’. 18 However, based on the limited evidence available, it seems these efforts may not be particularly good at identifying quality health information. The overall Google rank did not significantly predict higher quality information in any of the studies mentioned above. 19 , 20
Wikipedia is a multilingual, web-based, free-content encyclopaedia project based on an openly editable model. It has generated attention from international groups aiming to harness its potential for global public health promotion. 21 Wikipedia is the sixth most accessed website on the internet 17 and its popularity may be a result of its high ranking among top search engines. 22
As an unregulated source of health information used by many Internet users, it has been suggested that medical topics on the website could be edited by experts in order to improve the quality of information. 23 Other sources have found that accuracy and completeness did not differ much between Wikipedia articles and expert-generated websites. 24 Given the potential for inconsistency in the non-peer-reviewed nature of Wikipedia, it may be useful to ensure that links are provided to more definitive sources of health information. 25 There is no doubt that Wikipedia could serve as an effective medium for patient education if properly reviewed.
Specialty Wiki pages such as SurgWiki (www.surgwiki.com) and the Cancer Guidelines Wiki produced by Cancer Council Australia (http://www.wiki.cancer.org.au), represent an innovative approach to the Wiki concept, with information edited only by experts. At this stage, however, these websites have not gained the popularity enabling them to secure high Google ranks. While the information presented is of a high calibre, it may not reach the patient who will typically access only the first 10 hits from a Google search.
Despite its limitations, Wikipedia is potentially an important medium to disseminate patient health information. As the World Wide Web continues to expand, our ability as health care providers to regulate the quality of Internet health information will continue to diminish. The openly editable content of Wikipedia has many advantages and draws on import at an enormous scale. There is a high likelihood that its popularity will continue to increase at least for the near term. It has been suggested by some 23 that as health care providers trying to regulate the quality of health information that reaches patients, our efforts would be best spent editing and updating Wikipedia pages.
There have been many studies assessing the quality of health information on the Internet. Attempts have been made at developing rating scores for Internet health information; however, it is not clear how useful these rating tools are. HONcode is the most widely used quality evaluation tool.
The Google rank system is important in determining which information will reach patients, but is a poor predictor of good quality information. Wikipedia is a frequently accessed website and ranks highly on Google searches. Despite being a potentially important source of health information for patients, there is a risk of inaccurate or commercially biased information due to its crowd-edited nature.
In the internet era, our role as doctors in guiding patients towards high-quality health information has expanded into the digital setting. It is difficult to make recommendations on an optimal approach to this complex and evolving environment. However, from this review, the potential inaccuracy of information on the internet suggests some key areas in which doctors can contribute.
In the clinical setting, more time spent in discussion with patients on aspects of diagnosis and treatment may be a beneficial, individualised alternative to internet information. For computer literate patients, guidance towards high quality, accurate web resources is a reasonable approach which may help patients to navigate the myriad of information available to them while promoting autonomy. As discussed earlier, becoming actively involved in editing and updating resources such as Wikipedia may also be an effective way to disseminate our knowledge to patients.
On a broader scale, protocols for the development of web resources, to guide the process of writing and design may be beneficial for patient understanding, and complementary to grading systems such as the HONcode.
Overall, it is likely that a combination of these approaches is required. Guidelines do not currently exist on the approach of the medical profession to health information on the Internet. Such guidelines could potentially advise on strategies from the level of the individual doctor to professional bodies and policy makers. This may be necessary in order to stay ahead of the growing body of inaccurate material on the Internet and also to harness its power as a source of health information.
PEER REVIEW
Not commissioned. Externally peer reviewed.
CONFLICTS OF INTEREST
The authors declare that they have no competing interests.
Please cite this paper as: Fahy E, Hardikar R, Fox A, Mackay S. Quality of patient health information on the Internet: Reviewing a complete and evolving landscape. AMJ 2014, 7, 1, 24-28.http//dx.doi.org/10.4066/AMJ.2014.1900.
1. International Telecommunications Union. World Telecommunication/ICT Indicators database 2013 17th Ed. [cited 2013 July 5]. Available from: http://www.itu.int/en/ITU-D/Statistics/Documents/statistics/2013/ITU_Key_2005-2013_ICT_data.xls. [Google Scholar]
2. Pallen M. Guide to the Internet: The world wide web. BMJ. 1995; 311 :1552. doi: http://dx.doi.org/10.1136/bmj.311.7019.1552. [PMC free article] [PubMed] [Google Scholar]
3. Impicciatore P, Pandolfini C, Casella N, Bonati M. Reliability of health information for the public on the World Wide Web: systematic survey of advice on managing fever in children at home. BMJ. 1997 Jun 28; 314 (7098):1875–9. doi: http://dx.doi.org/10.1136/bmj.314.7098.1875. [PMC free article] [PubMed] [Google Scholar]
4. Eysenbach G. Empirical studies assessing the quality of health information for consumers on the world wide web: a systematic review. JAMA. 2002 May 22-29; 287 (20):2691–700. doi:10.1001/jama.287.20.2691. [PubMed] [Google Scholar]
5. Burkell J. Health Information Seals of Approval: What do they Signify? Information, Communication & Society. 2004 Dec; 7 (4):491–509. doi: 10.1080/1369118042000305610. [Google Scholar]
6. Bernstam EV. Instruments to assess the quality of health information on the World Wide Web: what can our patients actually use? Int J Med Inform. 2005 Jan; 74 (1):13–9. [PubMed] [Google Scholar]
7. Jadad AR. Rating health information on the Internet: navigating to knowledge or to Babel? JAMA. 1998 Feb 25; 279 (8):611–4. doi:10.1001/jama.279.8.611. [PubMed] [Google Scholar]
8. Boyer C, Baujard V, Geissbuhler A. Evolution of health web certification through the HONcode experience. Stud Health Technol Inform. 2011; 169 :53–7. [PubMed] [Google Scholar]
9. Mousiolis A. Polycystic ovary syndrome: double click and right check. What do patients learn from the Internet about PCOS? Eur J Obstet Gynecol Reprod Biol. 2012 Jul; 163 (1):43–6. [PubMed] [Google Scholar]
10. Nason G. Scoliosis Specific Information on the Internet- has the 'Information Highway' Led to Better Information Provision? Spine (Phila Pa 1976) 2012 Jul 4; [PubMed] [Google Scholar]
11. Soobrah R. Your patient information website: How good is it? Colorectal Dis. 2012 Mar; 14 (3):e90–4. doi:10.1111/j.1463-1318.2011.02792.x. [PubMed] [Google Scholar]
12. Meric F. Breast cancer on the world wide web: cross sectional survey of quality of information and popularity of websites. BMJ. 2002 Mar 9; 324 (7337):577–81. doi: http://dx.doi.org/10.1136/bmj.324.7337.577. [PMC free article] [PubMed] [Google Scholar]
13. Silberg WM, Lundberg GD, Musacchio RA. Assessing, controlling, and assuring the quality of medical information on the Internet. JAMA. 1997; 277 :1244–1245. doi:10.1001/jama.1997.03540390074039. [PubMed] [Google Scholar]
14. Barker S. Accuracy of internet recommendations for prehospital care of venomous snake bites. Wilderness Environ Med. 2010 Dec; 21 (4):298–302. [PubMed] [Google Scholar]
15. Rao NR, Mohapatra M, Mishra S, Joshi A. Evaluation of dengue-related health information on the internet. Perspect Health Inf Manag. 2012; 9:1c Epub 2012 Apr 1. [PMC free article] [PubMed] [Google Scholar]
16. Eysenbach C. How do consumers search for and appraise health information on the world wide web? Qualitative study using focus groups, usability tests, and in-depth interviews. BMJ. 2002 Mar 9; 324 (7337):573–7. doi: http://dx.doi.org/10.1136/bmj.324.7337.573. [PMC free article] [PubMed] [Google Scholar]
17. Alexa. Top Sites – The top 500 sites on the web. [Internet]. [Cited May 2013] Available from: http://www.alexa.com/topsites. [Google Scholar]
18. Brin S, Page L. The anatomy of a large-scale hypertextual web search engine. Computer Networks and ISDN Systems. 1998; 30 :107–17. [Google Scholar]
19. Pérez-López FR. An evaluation of the contents and quality of menopause information on the World Wide Web. Maturitas. 2004 Dec 10; 49 (4):276–82. [PubMed] [Google Scholar]
20. Pérez-López FR, Pérez Roncero GR. Assessing the content and quality of information on the treatment of postmenopausal osteoporosis on the World Wide Web. Gynecol Endocrinol. 2006 Dec; 22 (12):669–75. [PubMed] [Google Scholar]
21. Heilman JM, Kemmann E, Bonert M, Chatterjee A, Ragar B, Beards GM, Iberri DJ, Harvey M, Thomas B, Stomp W, Martone MF, Lodge DJ, Vondracek A, de Wolff JF, Liber C, Grover SC, Vickers TJ, Meskó B, Laurent MR. Wikipedia: a key tool for global public health promotion. J Med Internet Res. 2011 Jan 31;(1):e14. doi: 10.2196/jmir.1589. [PMC free article] [PubMed] [Google Scholar]
22. Laurent MR, Vickers TJ. Seeking health information online: does Wikipedia matter? J Am Med Inform Assoc. 2009 Jul-Aug; 16 (4):471–9. doi: 10.1197/jamia.M3059. [PMC free article] [PubMed] [Google Scholar]
23. Kint M, Hart DP. Should clinicians edit Wikipedia to engage a wider world web? [Letter] . Brit Med J. 2012 Jul 3; 345 doi: http://dx.doi.org/10.1136/bmj.e4275. [PubMed] [Google Scholar]
24. Rajagopalan MS, Khanna VK, Leiter Y, Stott M, Showalter TN, Dicker AP, Lawrence YR. Patient-oriented cancer information on the internet: a comparison of wikipedia and a professionally maintained database. J Oncol Pract. 2011 Sep; 7 (5):319–23. doi: 10.1200/JOP.2010.000209. [PMC free article] [PubMed] [Google Scholar]
25. Leithner A, Maurer-Ertl W, Glehr M, Friesenbichler J, Leithner K, Windhager R. Wikipedia and osteosarcoma: a trustworthy patients' information? J Am Med Inform Assoc. 2010 Jul-Aug; 17 (4):373–4. doi: 10.1136/jamia.2010.004507. [PMC free article] [PubMed] [Google Scholar]
Articles from The Australasian Medical Journal are provided here courtesy of Australasian Medical Journal