Evidence does not equal evidence

 Type “ketchup” into the Amazon search box and you will get 510 results. 510. For a tomato paste. If you’ve attempted to purchase anything online recently, you've likely experienced overchoice - a phenomenon in which people have a difficult time making a decision because they’re faced with too many options.

Do I want sugar-free ketchup? Or perhaps organic? Brand name? Or generic? The upside-down plastic bottle is certainly practical...but aren't the old-time glass bottles more classy? When experiencing overchoice, making a decision suddenly becomes overwhelming due to the many potential outcomes and risks that may result from making the wrong choice.

Now the particular brand of ketchup that adorns your dinner table may not be all that important. But there are numerous choices that involve the big life decisions as well. When buying a house, nobody’s criteria is “it should be a house”. There are different types of houses, with different qualities, and of course, different prices.

In the world of medicine, our “ketchup” is evidence. If you want to prove a point, promote a treatment, or convince your colleagues that a certain medication is more effective than another, you'll have to prove it. And, similar to ketchup, physicians recognize that evidence comes in all shapes and sizes. This may appear to be self-evident fact (the idea that there are different levels of evidence), but it seems that many people are unaware of this.

I was recently discussing the Covid-19 pandemic with a friend and he pointed out that “masks trap bacteria in our mouths and cause infections”. I found this to be an interesting point and probed. “Wow. Where did you hear this from?” I asked. “Oh, I just came up with it, and it made sense to me”. Non-expert opinion. The lowest form of evidence (i.e. no evidence at all). That is one of many corona-related non-evidence based opinions.

KINDLY SUPPORT OUR BLOG BY BROWSING THE ADS

THANKS SO MUCH, IT MEANS A LOT ESPECIALLY IN THESE DIFFICULT TIMES! 

Another example: my neighbor told me about a local woman who professes to be able to heal strep throat with unpasteurized goat’s milk, gargled in the throat. He swears that he tried it himself and it seemed to work. Now this level of “evidence” is still quite weak - but it’s actually one step better than the friend who feared that his mask was causing bacteria.

The anti-masker had no evidence at all - it was merely a person (a non-expert, in this particular case) stating his opinion. He had no proof to it.

In this second example, the person is claiming that it worked. It’s not merely an opinion but a fact (assuming he is trustworthy). In medicine, we call this “anecdotal evidence” i.e. evidence collected in a casual manner, which relies entirely on personal testimony. It is not, by any means, a well designed study - but rather a single person’s experience, where something seemed to work. Weak evidence, but evidence nonetheless.

The issue is: How does this person know that the goat's milk truly helped heal him? Maybe he was doing something else that made him better. Or maybe he would have gotten better whether he took the goat’s milk or not. His “study” lacks a control group.

The highest, most powerful form of evidence in medicine is a study performed with a control group. This means that in addition to testing the medication on a group of sick people, a second, similar, group of sick people don't get the medication and they are studied as well. This allows the experimenter to really see if it’s the drug/intervention that is truly making the participants better (i.e. if everyone that received the medicine gets better, and the control group does not get better).

Another, quite obvious factor that doctors look for when evidence is presented is the number of participants in a study. Some studies, for example, can have tens and even hundreds of thousands of people enrolled! When presented with such a study, VS the goat milk healer, who would you go with?

In the gemara shiur (Torah class) that I attend at night, the magid shiur (Talmud lecturer)noted that attendance had been slowly declining over time. One night, after the shiur, he handed everyone a piece of paper to take home. I was stunned by what I saw. It was a survey with 10 questions to fill out. Some of the questions included:

What time is realistic for you to come to the shiur?

What are your suggestions for improving the shiur?

Do you have a chavrusa to review what is learned in the shiur?

This was a magid shiur performing a scientific study! He was trying to collect data and make evidence-based changes to help improve the shiur. He could have simply gone over to one person after davening and had a discussion about the waning attendance of the shiur. But he recognized that data does not equal data. One person telling their opinion is different than collected data from the entire group of people involved. He decided to conduct a study, get evidence, and make changes based on that.

There is much more to be said on this topic, but suffice it to say that evidence does not equal evidence. It is fascinating to me, that many intelligent, thought-out people seem to rank all forms of evidence as equal. They will trust their neighbor’s advice (non-expert opinion) over a large clinical trial with a control group.

There are certainly times at which high level evidence is not available. Imagine, for example, the heart-breaking case of a patient of mine who had a rare genetic illness of which there are only 10 reported cases in the world. When dealing with such a condition, there certainly won’t be high quality studies with a large number of participants and a control group. At best there will be very small studies, or perhaps even a lower level of evidence - “expert opinion” - a recommendation (without any hard data) formulated by a group of people with clinical expertise, which is based on everything that they know about the subject.

I recognize that we can't use evidence for every life decision. When I want to know the best route to JFK airport, I will not hesitate to ask the guy sitting next to me in shul for his opinion (non-expert opinion). We all make decisions like that. Sometimes we don’t have the time to research the topic, or sometimes that is the only available “evidence” that we have available. But when it comes to big life decisions shouldn’t we give weight to how strong that evidence is?

I am not suggesting that all be rigid and calculated in every movement we make. We are certainly allowed (and encouraged) to get layman's advice from others to help guide life decisions. But let us recognize such advice for what it is --- advice. We live in a time where we have access to massive amounts of data. Don't make big life decisions based on someone telling you their opinion in a convincing or impassioned way. Do the research. Collect the data. And please please please, don't use goat's milk to heal strep throat.

Dr. Ari Grubner studied in Ner Israel Rabbinical College, and then completed medical school at the University of Maryland School of Medicine. Dr. Grubner is the host of a podcast that highlights and shares the wonders of God’s creations and the human body, “The Wonder - Niflaos Haborei Podcast” (www.anchor.fm/arigrubner). Dr. Grubner is currently a resident in pediatrics and lives in NY with his wonderful wife and their three sons. He can be reached at Niflaoshaboreipodcast@gmail.com

KINDLY SUPPORT OUR BLOG BY BROWSING THE ADS

THANKS SO MUCH, IT MEANS A LOT ESPECIALLY IN THESE DIFFICULT TIMES!