Frequently Asked Questions

Day to day, symptoms can vary widely depending on a spectrum of inputs — food, sunlight, exercise, socializing, sleep — that have nothing to do with a given treatment. Extending the duration of each treatment block, boosting the number of treatment blocks, or increasing the frequency of reporting means experimenters gather more data. This can negate the effects of non-treatment effects, and boost an experiment’s statistical validity. Every experiment, though, depends on the experimenter’s tolerance for continuing, so most experiment designers compromise on a duration that seems to the person being tested.

Some single person experiments includes “blinding” ⁠— making each treatment appear the same to reduce a psychological biases for or against each version. (Knowing that a treatment is newer or a stronger dosage can boost its perceived effectiveness by an average of 9% versus a control.)

Doctors can instruct a compounding pharmacist to create a double blind by putting different treatments or strengths (or a placebo) into identical pills or liquids. The bottles can be sequentially labelled (for example 1, 2, 3, 4, 5, 6) so that neither you or the doctor know which treatment you’re taking in a given time period.

Once the experiment is over, the pharmacist can provide a key that reveals which code corresponded to which treatment. (This might reveal, for example that you’d taken six successive weeks of A, B, B, A, A, and finally B.)

Particularly for nonpharmaceuticals, some people ask a friend to help with blinding by distributing different treatments into identical containers. This is simple for oils⁠—for $6.99, you can buy 25 plastic dropper bottles or six glass dropper bottles. Funnels are available too. The same goes for ointments and creams⁠—five jars cost $10 on Amazon.

The friend might label each as A, B, C, D, E, F (one for each treatment period using the schedule you pick) being careful to separately note which treatment went into each container, and then pass the containers over to you.

(Note that, historically, not all experiments include blinding, particularly when it’s impossible to create seemingly identical versions of each treatment, for example when a treatment involves obvious physical alterations.)

Just like everybody is different, every treatment is different. Here are some links to trials using a variety of treatment block durations.

Melatonin: 2 weeks treatment blocks (sleep in Parkinsons); 4 weeks (sleep in alcohol dependence); duration of hospital stay (delirium in elderly patients); 30 days (inflammation in athletes); 21 days (sleep). More melatonin studies.

Caffeine: 2 and 4 week treatment blocks (insulin resistance);

Cannibidiol: 6 week treatment blocks; 12 weeks; single dose x 3; 4 weeks; 4 weeks; single dose; 10 days; 8 weeks. (More CBD studies.)

Including a placebo in an experiment helps tease out the often powerful psychological effects of being treated.

But often placebos are difficult or impossible to implement, particularly if the treatment has physical manifestations (tingling, scars) that lead to unblinding or there’s no compounding pharmacist is available.

In some cases, unblinded personal experiments without placebos have yielded the same results as large, expensive gold-standard trials with blinding.

Finally, its important to note that placebo effects can be both positive or negative.

In the spring of 2019, we had been talking for nearly a year about building a software toolkit to help automate what many people already do informally⁠—track symptoms and statistically evaluate the effects of various treatments. As the idea started to jell, we started casting about for names for the toolkit.

Though there are a variety of scientific names for the methods that the WWB toolkit borrows from⁠— single person trial, n-of-1 trial⁠— none of them inspired any enthusiasm.

One day, headed to the car for a brief vacation, I realized I’d forgotten my blood thinner medications. (I had a pulmonary embolism last fall.) Walking out of my home holding two bottles of best selling blood thinners — Xarelto and Elequis — one in each hand, I was thinking about the wooziness that sometimes seemed to occur while I was on the blood thinners.

I weighed both bottles and asked myself out loud, “which works best?”

I knew, from reading and conversations, that this exact question is incredibly common for people struggling to find the right solution⁠—behavior, medicine, supplements⁠—for chronic health conditions. Luckily, the domain name was available!

WhichWorksBest is a creation of Pressflex LLC, which has been around since 1998. The name WhichWorksBest? ties into the fairly literal, “just the facts” naming conventions that Pressflex has used for other ventures.

For example, Pressflex was founded to help newspapers and magazines get online flexibly and affordably. Blogads, which we launched in 2002, was an automated service to help bloggers sell ads. Pullquote, launched in 2011, helps people store and share interesting quotes. AdBiblio, launched in 2014, helps book publishers advertise online. Racery, also launched in 2014, helps companies and charities build and host virtual races.

One remaining puzzle. Very few companies — or zero? — have a question mark in their logos. If WhichWorksBest? ever ends up in the headlines, this may confuse readers. But the question mark also conveys the confusion and questioning that is at the heart of WhichWorksBest? WhichWorksBest? is a question you ask, not an answer we give. So, for now at least, the question mark remains.

There’s lots of discussion among researchers about which scale best allows people to report on symptoms or opinions.

The consensus: whereas an even numbered scale forces people to err on the side or a positive or negative opinion (in a 1-6 scale, 3 is slightly negative and 4 is slightly positive), an odd-numbered scale allows for a “neutral” choice. So in a seven point scale 4 means neutral, neither bad or good.

Five and seven point scales are commonly used in research because too many rating options can create confusion. It’s tough enough evaluating symptoms and making comparisons about symptom severity from day to day! We opted for a seven point because it allows for 50% more nuance and granularity than a five point scale.

Treatments⁠—both pharmaceutical and nonpharmaceutical⁠—are classically evaluated in random control trials (RTCs). Large, carefully selected groups of participants and their doctors are blinded to which patients are getting which treatment or control.

In contrast, an n-of-1 trial looks only at the effects of different treatments on a single person. The trial periods and target outcomes are defined ahead of time. With the assistance of a compounding pharmacists, blinding and placebos are possible.

The two types of trials have radically different purposes⁠—the RTC is to determine broad efficacy and average treatment effects, and the n-of-1 determines the effect on a single individual.

“All RCTs do is show that what you’re dealing with is not snake oil,” according to William R. Shadish, PhD, a professor of psychological science at the University of California at Merced. “They don’t tell you the critical information you need, which is which patients are going to benefit from the treatment.”

According to Dr. Sunita Vohra and Salima Punja, BSc:

results from randomized controlled trials (RCTs), often considered the ‘gold standard’ of research evidence, are often not well suited to the realities of clinical practice given patient heterogeneity, comorbidities, and the use of multiple concurrent therapies. In fact, RCTs may exclude the majority of patients seen in routine clinical practice. In addition, evidence is lacking on the long-term effectiveness, comparative effectiveness, and additive effectiveness of many therapies for chronic conditions. This lack of relevant evidence can limit a clinician’s ability to make evidence-based decisions.

It’s easy to see why many people, even some doctors, believe n-of-1 studies are “not statistically valid.”

Many doctors weren’t exposed to the voluminous literature on N-of-1 studies in medical school or, if they were, quickly forgot the concept. Once they moved into clinical practice, they’ve been inundated, day after day, by drug reps justifying their drug’s superiority over others based on large parallel group random control trials (RCTs).

“This study of 3,500 patients shows that our drug X results in a 17% improvement in symptom Y,” says the drug rep, pointing to the study’s title. Parallel group RCTs are often perceived as the “gold standard” for proving a drug’s efficacy and safety.

In fact, statisticians have long argued that the results of RCTs are not generalizable for any given individual.

First, the RCT’s participants often have nothing in common with a given patient, whether because of age, sex, fitness levels or other variables. Historically, women and minority groups in particular have been underrepresented in studies.

Second, even if a study’s participants do appear to match the patient, research indicates that drug efficacy varies strongly from patient to patient, whether because of genetics, microbiome or behavior. Behind the headlines of all large studies reporting “average” benefits is compelling evidence that most drugs don’t work equally well for all people. This is called “heterogeneous treatment effects,” or HTC.

“All RCTs do is show that what you’re dealing with is not snake oil,” according to William R. Shadish, PhD, a professor of psychological science at the University of California at Merced. “They don’t tell you the critical information you need, which is which patients are going to benefit from the treatment.”

This means that, for an individual, a drug can’t be proven to be effective, much less optimal, unless it’s been systematically tested for that individual, ideally with crossovers, placebos and blinding. In short, an n-of-1 trial.

In fact, it’s the trial-and-error drug experiments (formally called “trials of therapy”) used by most doctors to find the best drug for an individual patient that lack scientific rigor. “These so-called trials are unblinded, have no control, and involve no formal assessment of effectiveness, making them vulnerable to invalid conclusions about treatment response,” observe Dr. Sunita Vohra and Salima Punja, BSc.

>> More on the difference between n-of-1 trials and large group RTCs.

First, it’s important to remember that what works for one person frequently does not work for another person.

Also realize that Facebook makes it possible to check on, if not prove, how legit someone is. Do you know people in common? Do they regularly post things that seem real? Do real people like those posts? How long have they been a member of Facebook groups you belong to?