How To Appraise Clinical Trials on the Internet: A Layman's Guide

The widespread use of the Internet is the greatest revolution in mass communication since the invention of the printing press. Worldwide access to a virtually limitless pool of knowledge provides enormous empowerment to individuals and offers huge opportunities for self-education and self-determination. I apologise for the hyperbole in this introduction but I do believe that in the future people will use a calendar based on BC and AI - Before Computers and After Internet. However along with personal empowerment comes personal responsibility. In healthcare, doctors are no longer the fount of all wisdom: the Internet aware patient will value their opinion and may well access websites recommended by them, but he/she will also obtain many other independent opinions on diagnosis and treatment from the Web. This approach has its downside. Anyone can set up a website and can make great claims to expertise without the end used being able to validate them. We all see examples of weird and bizarre sites whilst surfing, in medical matters as much as any other. The purveyors of snake oil and miracle cures are alive and well and living in cyberspace. 

So what can the layperson do to protect themselves from the worst excesses of the Net while continuing to make rational use of the opportunities for knowledge and empowerment that it offers? In medicine, clinicians in Europe and the US have recently adopted the term 'evidence based medicine' to describe the process of critical appraisal of research data and opinion, and in my view, this process can equally be applied to material obtained from the Internet. Interactive sites make the process easier - this gives you the chance to question results published on the Net and engage in a dialogue with the author of the site. So starting from the basics of critical appraisal I have tried to compile a list of questions to ask when reading Internet material concerning health matters. This is not comprehensive, and I'd welcome feedback and particularly other suggestions for additions to the list. 

Question one - Who wrote it?

I would place more weight on material written by a member of a recognised academic institution. It is easy to access university and medical school websites and check the qualifications of the author. Material from patient support groups is also often helpful, providing a different perspective and a breadth of information. I would be less persuaded by sites asking me for money for information or that appear overtly commercial. These are trying to sell you something rather than trying to educate you, no matter how well designed and convincing they look. 

Question two - When was the site produced?

 As the Web matures it is accumulating out of date sites rapidly. This will become a problem in time unless they are deleted. Medical knowledge and opinion changes quickly, so check when the site was updated before spending too much time studying it. 

Question three - Where's the beef?

In other words, what's the evidence for the claims made in the site? Appraisal of evidence is critically important when using the Web for health information. Evidence based medicine divides the quality of research into three grades. These can be used to decide how convinced you are by an individual piece of research. 

GRADE A - Information from meta-analysis of randomised trials or from a well conducted randomised controlled trial.

GRADE B - Information from case control studies and other non-randomised observational or experimental studies

GRADE C - Expert opinion

Some of the (so-called) experts have not taken kindly to being demoted to grade C, but today’s expert is tomorrows fool and we must accept that much of what we purvey to our patients  will be seen as misguided in a few years time.  On the other hand, randomised clinical trials should stand the test of time and, if properly conducted, will provide the basis for the medicine of tomorrow.

There are a few key phrases to look for when studying medical research papers. The first is randomisation. A worthwhile study will have randomly allocated patients volunteering for the study into different groups. This allows the investigators to compare their new treatment or idea with the current 'gold standard'. Look at how the randomisation was performed. It should be done using computer generated random numbers and should be cheat proof - in the past sealed envelopes containing a treatment allocation have been steamed open by well meaning doctors and nurses who wanted to get their patient into a particular treatment arm in a study. If a study is non-randomised it is open to all sorts of bias. One major problem for non-randomised studies is the lack of control for the placebo effect. Good doctors gain the confidence of their patients and this can have a positive effect that has nothing to do with the treatment being reported. This is particularly true for infertility treatment since many couples will conceive eventually without help and  an encouraging consultation with prescription of the latest wonder drug or treatment will have effects on frequency of intercourse, reduction of infertility related stress and even on sperm quality. Randomisation sets out to avoid this bias.

The second key word is controlled. The placebo effect can only be avoided reliably if the study contains a placebo control group. In other words, some of the volunteers were randomly allocated to receive no treatment or to be treated with an inert substance - the eponymous 'sugar pill'. Any benefit seen in the placebo group, such as an improvement in pregnancy rate, can then be attributed to factors outside the study. If the drug or treatment being assessed does better than the placebo group in producing pregnancies then this is a true effect. 

The downside to placebo control is that some volunteers do not get treated (in their eyes). Sometimes studies offer active intervention after a few months of placebo treatment, but these studies are hard to recruit and many investigators take the easy option and compare two active treatments. This is still more reliable than a uncontrolled case study but the results are harder to interpret and biases can creep in. 

As well as having to accept the possibility of being randomised to 'no active treatment', volunteers for grade A trials must also accept that the trial will be double blind. Neither the volunteer nor the investigator will know whether they are using the 'active' drug or the placebo. They will live with that uncertainty until the end of the trial, again making it difficult to persuade people to volunteer. However, there are bonuses - the so-called Hawthorne effect identifies the conundrum that participants in clinical trials seem to do better medically than their counterparts in routine care. Perhaps this is because of the level of interest that we take in our volunteers - no one seems to be sure. 

The reason that research clinicians are so committed to double blind randomised controlled trials is that they give the most reliable answers to important questions. However, there are a few more provisos. First, the study has to be adequately powered. This does not refer to the study automobile, but rather means that the study has to have enough volunteers within it to allow proper statistical analysis of the results. There's no point in testing a new drug or procedure on a handful of people, and good studies now include an assessment of the statistical power needed to produce a meaningful result. This number is known at the outset and allows the investigators to know when to stop - often a problem in the past. These power calculations will be obvious in the description of the study. 

It is also important that the study has received ethical committee approval. This not only means that the new drug or treatment has been assessed by an independent ethics committee and deemed to be safe enough for a clinical trial, but also means that the methodological quality of the study has been approved by the committee.

Question four - Do you believe what they are saying?

After assessing the quality of the study, look critically at their conclusions. Authors sometimes test one idea but state that they have tested something entirely different. This error most often appears in the abstract of the study, which is the part, other authors then quote. Always check the original paper if possible and establish whether you find the argument convincing. Be particularly critical of case series and other non-randomised trials. Enthusiasts for a particular method or drug usually produce these. This may work well in their hands but if the approach is so good, why has it not been properly tested in a randomised controlled trial? When I discover a drug that really improves sperm quality and makes babies for couples with male factor infertility then I will organise a RCT to test it. I will have plenty of volunteers because male factor infertility is common and distressing, and the available treatment of IVF with ICSI is unpleasant, expensive and does not work very well. On the other hand, if I want to make money from the gullible I'll promote my wonder drug on the Internet on the basis of 'personal experience' and 'testimony from satisfied customers', whether it works or not.

Question five - Where does this piece of information fit?

The fascination of medical practice is that we see people not machines. Research studies help us to know what works, and equally important, what does not, but they do not easily translate to the clinical care of an individual. Each patient brings their own set of problems, their own set of experiences and their own wishes for treatment. This is particularly important in fertility medicine. Infertility is seldom life threatening and it is the job of the clinician to determine what the couple are prepared to do in order to conceive, how they feel about the difficult and time consuming rigours of assisted conception and to help them to choose the right treatment pathway.  This is not necessarily the one with the highest success rate.

The internet can be particularly valuable in this context.  Couples can come to their consultation armed with a good understanding of the problems, tests and treatments allowing an informed discussion to take place.  The clinician can interpret some of the jargon, help appraise the quality of the evidence for and against different approaches and give his/her own views.  Such consultations can be rewarding for both the couple and the doctor.  In writing this paper I hope to help couples to assess the quality of research material, enabling them to draw rational conclusions regarding their medical care and to avoid the seduction of the 'quick fix' so often purveyed by the less scrupulous.