Assessing Outcomes at the Alternative to Meds Center: Survey Results Prove Promising

As a professor and psychologist with a specialization in evaluation, research methods and statistical analysis, as the actor Liam Neeson has shared in his portrayal of Bryan Mills in the collection of Taken movies, I too have a very particular set of skills. And yes, I have acquired these skills over what at times feels like a very long career. Due to the new movement across so many sectors to become evidence-based and hopefully evidence-driven, I am often contacted by various organizations seeking help with documenting how their efforts make a statistically significant difference when it comes to their clients’ success. After receiving a call from the Alterative to Meds Center (ATMC), a small residential treatment center in Sedona, Arizona, and learning about their approach to holistic medication tapering and addiction treatment efforts, I was intrigued.

Most of my research typically focuses on child development, at-risk youth, and organizations just trying to help kids experience a productive and enjoyable childhood. I have consulted for more than 150 school systems and organizations, and managed the evaluation efforts for more than $19.5 million in research funded by the National Science Foundation (NSF), U.S. Department of Education and the Department of Justice. But this request from ATMC was a little different. This request was coming from a treatment center focused on helping adults with their pharmaceutical drug dependency issues. And with my interest in helping kids not be drugged for acting like kids, and recently losing one of my best friends to a long battle with insomnia and endless medications which I believe led to his suicide, I felt compelled enough to say yes.

With so many other promising holistic avenues to drug dependency treatment available across the USA, and yet still in need of documenting how their treatment approaches are successful, I thought that sharing the methodological and statistical approaches I took, and lessons learned along the way, could be of benefit to treatment centers seeking to find such statistical confirmation. But I also hope this summary can help individuals and their families get a better understanding of what an evidence-based practice should offer as assurance to their clients. So let’s take a look at some of the essential aspects that must be considered for those seeking documentation of evidence-based treatment.

Please note, the Alternative to Meds Center has been in business since 2006, and has treated more than 1,600 clients since opening its doors. Assuming that not all of their clients experiencing serious psychological and dependency challenges had success in ending such reliance on the pharmaceutical industry, I felt compelled to advise them upfront (as I do with all clients) that there was a possibility the results might not statistically show they were making a significant positive difference. I also explained that it would require a good amount of work on their part to help coordinate and implement sound data collection efforts. And I warned that, as with every study, there will always be barriers faced and change needed due to unforeseen obstacles. We did indeed encounter numerous challenges along the way. But as I will share shortly, for the Alternative to Meds Center and the vast majority of their clients, this story has a happy ending.

First Things First… Measuring the Outcomes

In the world of academia, we often give our graduate students the following advice as they begin their thesis or dissertation efforts: “Start with the end in mind.” In other words, start by visualizing what facts or outcomes you want to document when your study is complete. I quite often give this same advice to my clients. In the case of the Alternative to Meds Center, I asked them, what outcomes do you want to find that would document that your efforts are successful? The variables they wanted to explore for such evidence of their success were: 1) to what extent their clients were able to stop the use of pharmaceutical drugs; 2) how the symptoms clients were having trouble with were reduced; 3) the improvement in their clients’ sleep quality; and 4) their clients’ overall improvement in wellbeing.

After deciding upon the outcome variables, the next step was to determine reliable (consistent) and valid (accurate) ways to go about documenting such outcomes utilizing a sound methodological statistical approach. I know this last sentence might sound a little ivory tower to some, but it is paramount to understand first and foremost. Why? Because in order to perform any type of statistical analysis seeking to document statistical significance — aka showing with numbers that there is high probability that the results did not happen by chance — ethical analysts are required to use measurements (i.e. statistics, numbers) and assessment tools or scales which have established reliability and validity. In the world of psychology, these are often referred to as psychometrics.

If you cannot be sure that what you are using to measure the outcomes you wish to document are actually consistently measuring the same thing over and over again (reliability), or more importantly measuring what you think you are measuring (validity), you might as well flip a coin, and save time and money. Because without highly reliable measurements being used which truly measure what we claim to measure, the 50/50 chance of flipping a coin and calling heads or tails is possibly just as accurate as a flawed approach to statistical analysis. Sadly, in a world where not enough psychology and medical journals require the documentation of the assessment tools’ reliability and validity, there is far too much misinformation and disinformation being shared with the public.

Testing 1, 2, & 3

Additionally, in order to perform an analysis that shows a client has experienced successful outcomes, at the bare minimum we must collect a baseline assessment (pretest), and perform a posttest assessment. These tests administered at pretest and posttest must also be identical tests. You must use the same statistics, scales, test, or survey if you will, throughout the effort. If you want to show change from intake to discharge, you have to consistently measure how the client changed during that period of time.

And if we are going to pursue what some consider a golden standard in psychological or behavioral research, we not only want to document that clients made progress from intake to discharge, but also document that the improvements continued or at least were sustained for months if not years after the treatment. So the first thing organizations seeking to establish evidence-based practices must do is ensure they are consistently using the same quality psychometric assessments, at intake, discharge and follow-up periods, to document their clients’ initial progress, as well as on-going wellbeing and sustained improvement.

Gathering Enough Data… the Right Way

To begin this initial study of ATMC’s efforts, there was ample data already collected to assess the effectiveness of the program from intake to discharge on most of the outcomes where they hoped to find evidence supportive of their treatment efforts. But what they were missing were follow-up assessments on how well their clients were doing months after care.

Thus, the next major task at hand for ATMC was to do follow-up assessments on their large sample of clients who had graduated from the program. After helping them build an online survey encompassing the existing and additional assessments needed to measure their outcomes (via a website from which I could collect the responses), the staff sent this survey link to clients that had been away from the ATMC residential treatment center for more than six months but less than four years. Another important caveat at this point, however, is to make sure you have your clients use the patient code or number connected to their treatment records, so you can connect the follow-up survey efforts to each individual client. It is also important to make sure you set up the survey link using this code number to provide access to the online survey, and as a quality control procedure to make sure they only complete the follow-up survey one time.

Due to the challenge of having up-to-date addresses and contact information for this sample of clients, this equated to about 300 clients they sent either an email or a letter to asking for them to help ATMC determine its strengths, challenges, and ways it can improve. And unfortunately, due to ATMC experimenting with differing approaches to what tools and assessments they used over the past few years, there were a number of data inconsistencies that caused us to discard a portion of the responses.

Please note, this follow-up assessment process did take several months at two different times to achieve. Part of this was due to their already busy staff not being readily available to perform the time-intensive task of reaching out and requesting the clients’ participation. The other reason for adding on a second phase was due to the need for more participants to respond. In other words, if at first you don’t succeed (at getting a large enough sample to analyze), try try again!

Why? Not to get all professorial on you again, but this is a very important part of a study’s worth and success. Because in order to perform statistical analysis, an analyst will need approximately 20 to 40 respondents for each outcome variable you are assessing with the survey. In the statistical world, this is called cell size. So for a survey that has assessments looking at analyzing three variables, you would need 60 (3 x 20) to 120 (3 x 40) respondents. Such numbers are needed to ensure that you have the power to perform such analysis. And if you don’t have a large enough sample, you could end up running into what is called Type I or Type II error. This respectively refers to assuming you found significant findings when you really didn’t, or not finding significant findings when you really did. In other words, publishing a study on the results of 20 or so clients is not enough to hang your hat on, or have clients feel confident about the treatment success.

The Long and Short of It

I could play the role of Nyquil and sedate you with an endless array of additional statistical specifics essential to truly documenting a case for evidence-based treatment practices. But basically, the long and the short of it is that in order to get started and take the best path forward to collecting data worthy and strong enough to power a sound analysis, or assessing if the research is worthy of believing, a study will need to focus on the following:

• Determine what the outcome variables are that you wish to document evidence of successful treatment with.
• Only collect data using assessments with established statistical evidence of reliability and validity; and avoid use of tools developed in-house when possible.
• Make sure to (at a minimum) pretest before treatment begins, after treatment ends and do a follow-up at least six months after the discharge from treatment.
• Seek to collect data from as many clients as you can to ensure adequate power for the analysis.
• And when possible, collect the data in a convenient format to reduce the effort needed by your clients and staff to collect the feedback (e.g., online survey portal).
• Find an ethical analyst to provide the outside evaluation and analysis of your efforts; aka avoid bias by not collecting, analyzing and producing the results in-house.

As a side note, such an approach, even if it doesn’t produce positive outcomes, can be quite beneficial to organizations seeking to determine what they are doing well and what needs to be revisited or improved. Such outcomes, good or not so good, can be used as operational analytics that on a regular basis can provide you with the information you need to pursue continuous improvement efforts.

Signs of Success for ATMC

So how did the ATMC study turn out? Well, the results suggest that ATMC has an excellent success rate. Upwards of 87.5% of their clients sharing feedback had discontinued the use of pharmaceuticals and remained med free for extended periods of time — ranging from six to forty-seven months. The analysis documents that ATMC’s holistic treatment process can significantly reduce a client’s dependency on pharmaceuticals, if not eliminate use entirely.

Additionally, for the large percentage of the clients who were able to dramatically reduce or end their pharmaceutical dependency during the ATMC treatment process, the study documents that the results are long lasting. Evidence also suggests that for a majority of the clients, the more time away equates to continued reduction in pharmaceutical medication use.

This study also identified a statistically significant decrease in symptom severity scores from intake to discharge from the clinic. Furthermore, a total of 83.5% of the responding clients shared that the improved symptom levels experienced before graduating from ATMC had continued for six months to four years. Meanwhile, 71.9% of ATMC clients report experiencing more positive social relationships and interactions in life after graduating. And more than 75% of ATMC graduates report experiencing quality sleep patterns continuing six months to four years after their residential care in the center.

Now of course, with any center seeking to help clients who have wrestled with extreme emotional, traumatic or psychological challenges for various amounts of extended periods of time, thus leading to a dependency on pharmaceutical drugs so strong that they needed treatment, one cannot assume everyone who leaves the doors of a clinic is going to experience positive outcomes. From the responses and data I reviewed, I must share that this was the case for a portion of ATMC clients who responded to the study and completed the follow-up surveys. And in the follow-up survey some of these clients expressed their unhappiness with the efforts provided by ATMC.

But this is why we collect data on a large sample of clients, so that we can analyze and determine what the bigger picture is when it comes to a center’s success. Without such analyses, many are left to read online reviews, where let’s face it, similar to some reviews I have occasionally posted on Yelp after an unfortunate dining experience, it is not uncommon for the more disappointed clients to share their thoughts more often than those who experienced something more satisfying.

Of course, with online or even snail-mailed surveys, there is the possibility that one could receive feedback emanating from either the choir singing your song, or a group of unhappy customers. But this is why I recommended sending the survey to all of ATMC’s clients who met the requirements of the golden standard we were seeking with the study. The goal was to get responses from a large sample of uniquely different clients who experienced various outcomes.

A total of 103 clients responded with completed follow-up surveys, equating to a 34% response rate (more than twice that of typical industry standards). And after analyzing the data file, and documenting larger standard deviations than one might want to see (aka a limitation), it became very apparent that the sample consisted of clients with various experiences at ATMC. But as the final analysis documents, for the majority of clients who took part in this study, the outcomes were what they had hoped for: a better life.

For access to more details on the study’s outcomes, please click here to view a summary of the report.

Limitations of the ATMC Study

To be clear, it’s not always as cut and dry as the text and bullets above might suggest. For most clients I have worked with, ATMC included, too often the internal data historically or currently collected provides many limitations to quickly performing an analysis capable of producing results. The ATMC efforts ran into several challenges or limitations (e.g., adopting existing reliable and valid scales, recruiting enough participants). Also, depending on the severity of these challenges, it could equate to endless hours for an analyst to enter, clean, merge, code, recode and analyze the data file or files. And then when you are basically analyzing a sample of clients who typically seek such help due to extreme symptomology, analyzing for outliers also becomes a challenge; when basically your whole sample of clients might appear at times to all be outliers, setting up camp at both ends of the normal distribution bell curve.

Missing data is also a challenge when it comes to analysis. In order to use consistent measures across your sample, basically you need a complete data set. When medical records are riddled with missing data points, because the staff didn’t consistently collect information on specific aspects for all clients in the sample being studied or clients chose not to answer, it often forces an analyst to either remove these participants from the sample or use questionable data replacement procedures to estimate what a participant might have answered (something I would never do or recommend). And when you only have a portion of the questions being haphazardly answered, it can affect the scales’ mean scores for your outcome variables and thus the reliability and validity of the assessment tools used. In other words, you need to try your best to collect the same complete data set for every client whenever possible in order to perform the most robust analysis and produce the best results. In the case of ATMC’s study, due to either missing data not collected during treatment, or inadequate client answers on the follow-up surveys, approximately 20 clients were removed from the sample of surveys returned.

I think one of the other major limitations to be addressed that can help with future efforts is to make sure you also are documenting the treatment being provided. You need to document in more precise terms the specific type and level of “treatment” each client received. Because if you are a holistic or integrated practitioner, you most likely have a clientele which come to you with extreme but often quite different circumstances, past experiences, drug dependencies and levels of differing symptom severity. As a result, your “treatment program” for each of these clients is probably slightly different or uniquely designed for each client’s needs.

This means that if we really want to tweak out specific results six months after care, most probably need to do a better job documenting what type of specific therapy and care the client received and how much of the specific treatments they received (dosage). This is something ATMC is planning to do for future studies, so they can more precisely determine why clients who had great success, or not enough success, experienced such outcomes. It is not enough for analysis purposes to just show how many weeks they were in your care. It is not enough to just show what date they left care, and not provide a rationale as to why they left care. If you really want to know which specific efforts within your organization are leading to the best results, you need to document the protocol and specific treatment they received.

Summary

Let’s be honest, the talk about alternative drug treatment therapies not having an adequate body of research supporting their efforts is a legitimate critique. But an absence of evidence does not necessarily mean that such evidence does not exist. By all means, just as the doctor who unsuccessfully “treated” my friend for insomnia should have to show that the drugs they are prescribing and the approach they are taking are not only safe but also effective, it should be an expectation that every alternative therapy clinic has evidence supporting their specific treatment efforts. This is how legitimacy can be earned, clients can be confident in the selection they make, and the public can be aware if a treatment approach or program is or is not legitimately helping clients. With the existing collection of data at these clinics and the statistical software available today, however, there should be no reason for every center or clinic (offering highly effective services across our nation to help support clients seeking treatment for drug dependency) to not have the evidence to show they can help.

This effort was intended to illustrate what I found, and how other clinics and centers could be pursuing similar research agendas. I wanted to show how with so many currently collecting an abundance of data on their clients, finding evidence of their effectiveness is achievable. But as I have highlighted, there are numerous limitations and challenges to turning this information into data that can actually produce statistically significant results.

To some degree, it feels as if the pharmaceutical industry has developed a pill to mask the symptoms for every normal and possibly abnormal human experience or behavior a person can report. Furthermore, far too many doctors are prescribing such drugs with little concern for the side effects of addiction, suicidal ideation and so many other warnings we have become all too desensitized to hearing in the last half of those surreal 60 second commercials. And to make matters worse, many insurance companies won’t pay for therapy without a formal diagnosis code, and far too many clinicians still want to believe it is their ethical obligation to prescribe such dangerous drugs.

According to Consumer Reports, in the USA the use of prescribed pharmaceutical psychotropic drugs has never been higher than it is today. Yet as the drugs’ medication guides, research and many voices have shared for decades, psychotropic drug use can be quite dangerous and rarely holds any positive long-term benefit. And due to the many black box warnings these prescriptions come with, in countless cases even short-term advantages are questionable. As our nation wrestles with the rise of a full-blown opioid crisis, we are once again reminded that often pharmaceutical drugs (e.g., benzos, depression meds, ADHD stimulants) lead to dependency if not outright addiction. As a result, many are seeking help, and looking for more integrated physical, mental and spiritual — or might we say natural, holistic or ‘green’ mental health based — avenues for drug dependency treatment.

I hope this article has helped you conceptualize what an evidence-based practice should document. I also hope that you can now see how clients and treatment centers can benefit greatly by making such information readily available, as the Alternative to Meds Center has discovered.