Outcomes and Effectiveness Data

Consumer Satisfaction Survey

Colonial Behavioral Health prides itself on offering quality services to meet our consumers' personal goals. We regularly check in with the individuals we serve to gauge our effectiveness.

This satisfaction survey was conducted in April 2016. Individuals were asked to rank their satisfaction with our services, and the results have been combined below. A mean score of 3.5 or higher (within a 1-5 range) is considered to be satisfactory.

A sampling of additional input received follows each set of results.

Colonial Behavioral Health Adult Outpatient Services -- Williamsburg Office

Program# of Surveys ReceivedMean Score
Outpatient Counseling 35 4.3525
Intensive Outpatient Program (IOP) 4 4.375
Case Management (Geriatric, Substance Abuse) 12 4.6225
No Designation 15 4.515
TOTAL* 59 4.43

*The Total response counts for each program area reflect the total number of surveys received. Some individuals chose to complete one survey for multiple programs.

Counseling additional input:

  • "Really opened my mind and realizing sobriety is a better healthier life.”
  • “Lady at front desk always smiling and welcoming”
  • “No improvement needed you are doing a great job here I have been treated very well thank you”
  • “I am nothing but pleased w/my experiences at CBH. I have been able to continuously [get better] in my recovery and my life”

Case Management additional input:

  • "They are good people to deal with”
  • “I have excellent case management service. My CM looks out for me and always returns my calls promptly.”

Colonial Behavioral Health Medical Services

Program# of Surveys ReceivedMean Score
Williamsburg Office 50 4.34
York-Poquoson Office 19 4.7325
Capitol Landing Office (Child & Adolescent) 32 4.445
TOTAL 101 4.4475

*The Total response counts for each program area reflect the total number of surveys received. Some individuals chose to complete one survey for multiple programs.

Colonial Behavioral Health Child and Adolescent Services at Capitol Landing

Program# of Surveys ReceivedMean Score
Outpatient 14 4.555
Case Management 3 4.3325
No Designation 4 4.25
TOTAL* 21 4.4525

*The Total response counts for each program area reflect the total number of surveys received. Some individuals chose to complete one survey for multiple programs.

Child and Adolescent Services additional input:

  • "Very satisfied. Helped my son so much. My son is no longer in a fog- he is back.”
  • “ALWAYS! So nice & friendly.”
  • “Thank you for everything you do!”

Colonial Behavioral Health York/Poquoson Services

Program# of Surveys ReceivedMean Score
Individual or Family Counseling 10 4.575
Substance Abuse (SA) Group Counseling 13 4.31
Mental Health (MH) Group Counseling 7 4.64
Intensive Outpatient Program (IOP) 13 4.6725
Child and Adolescent Counseling 3 4.6675
No Designation (Child & Adolescent) 3 4.4275
No Designation (Adult) 3 5
TOTAL* 52 4.5525

*The Total response counts for each program area reflect the total number of surveys received. Some individuals chose to complete one survey for multiple programs.

Individual/Family Counseling additional input:

  • "Hillary Z. is a very good counselor. I appreciate her feedback. I feel she is very involved in helping me.”
  • “I really enjoy coming here.”
  • “Everyone is very kind. I feel I am respected and have a faith they are respectful of that.”

Mental Health (MH) Group Counseling additional input:

  • “Even after I graduated I still choose to keep coming. I have never come this far in my recovery thanks to Ron.”

Substance Abuse (SA) Group Counseling additional input:

  • “I enjoy the group. Getting to learn about new people is always nice.”
  • “They are always here to help and answer any questions.”

Intensive Outpatient Program (IOP) additional input:

  • "Hillary is the best. Very respectful. Very [stern]. Great counselor.”
  • “I think it is an exceptional program and it is helping me out.”

Child and Adolescent Outpatient Counseling additional input:

  • “The professional staff is excellent and the clerical staff is as well.”
  • “I don’t know if we would have made it through all of this without the wonderful staff at CBH.”
  • “Everyone is wonderful and doing excellent work. Dealing with mental illness is stressful the staff at CBH is so caring kind and helpful it makes it easier to deal with having such support.
Total# of Surveys ReceivedMean Score
GRAND TOTAL
233  4.465

 

Counseling Services

CBH has been obtaining systematic, ongoing data regarding treatment outcomes in counseling services since 2008. We use this data to improve treatment outcomes with CBH clients, and to evaluate our overall effectiveness and outcomes.

How do we obtain the data?
The data is typically obtained through the use of brief (1-2 minutes) rating scales that are completed each session by the client. One rating scale is completed at the start of the session to capture changes/improvements between sessions; while a second one is completed at the end of the session to provide feedback and offer suggestions about what was helpful and not helpful. The client and therapist review and discuss the rating scale scores during each session.

Why do we obtain and report this data?
The review and discussion of the rating scale scores helps guide treatment. It allows each client and therapist to identify and reinforce positive changes, and make adjustments to treatment if improvements are not reported.

These same rating scale scores also provide the data reported below. This data is used to evaluate the effectiveness and outcomes of the counseling services we provide. Reporting this data allows clients, potential clients, and stakeholders to compare and make more informed choices regarding their treatment provider.

The scores are only reported in aggregate form, and no individual clients are ever identified.

Outcome Data

Two types of effectiveness data are reported for CBH counseling clients: how many people are helped? (Diagram 1); and how much they are helped? (Diagram 2)

Diagram 1: How many people are helped?

NumberHelped2016-02


62% of CBH counseling clients achieve their "target" score compared
to 50% of counseling clients in a large national sample
Click here for more detail and statistical information regarding Diagram 1

 Diagram 2: How much improvement do clients experience?

Amount2016-02
CBH counseling clients report a 59% improvement compared to a 50%
improvement reported by counseling clients in a large national sample.
Click here for more detail and statistical information regarding Diagram 2

Summary
Based upon effectiveness data collected since 2008, CBH counseling clients benefit more than the average counseling client. The data presented above was collected through February 1, 2016.

Outcomes and Effectiveness: Details and Statistics

Diagram 1. How many people are helped:

Diagram 1 compares the percentage of counseling clients who achieve a "target" score, sometimes called a "benchmark" score or "predicted" score. The data in Diagram 1 is based on the scores reported by counseling clients on a brief rating scale completed at the beginning of each counseling session. The target score is a score that we "aim for", and it represents the expected or predicted improvement for a person based upon comparison with a large sample of counseling clients. The target score varies for each person depending on that person's level of distress when they start treatment. Reasonable or satisfactory improvement for someone who is severely distressed at the start of treatment is different than for someone reporting less severe distress at the start of treatment.

A simple example can illustrate: consider two people who both want to reduce their coffee intake due to sleeplessness and anxiety. If one person drinks 20 cups of coffee per day, and the other person drinks 5 cups of coffee per day, it might not be realistic or useful to set the same "target" score for both people. Reasonable, satisfactory improvement for the 20 cup person might be a reduction of coffee consumption to five cups per day, while reasonable, satisfactory improvement for the 5 cup person might be two cups per day.

Of course in the real world everyone with the same intake score will not achieve exactly the same final score. Some will score higher and some will score lower. However, the target score represents a reasonable estimate of expected improvement, and provides a helpful reference point to determine whether any given person is improving more than usual or less than expected. Adjustments to treatment can be made based on this information.

How a target score is calculated:
A person's target score is directly dependent on his or her initial, or intake, score. The person's intake score is matched with a large normative sample of clients (ASIST for Agencies, v. 4.08) who reported that same intake score. The "average" improvement for this large sample of clients is calculated using a linear regression equation, and this "average" improvement or relief becomes the target score for everyone with that initial or intake score.

In more technical terms, the target/benchmark score is based on severity-adjusted change scores which are calculated from the generalized linear model (GLM) of a normative client database (ASIST for Agencies, v. 4.08). The GLM is a statistical regression formula which is used to determine average final scores for each possible intake score. The GLM generalizes linear regression by allowing the linear model to be related to the response variable via a link function, and by allowing the magnitude of the variance of each measurement to be a function of its predicted value. The result is a normal distribution of "final" scores for each intake score, and the target score represents the 50th percentile of this normal distribution. Determining the target score in this way allows statistical prediction of a person's final score based only upon that person's intake score.  

While 50% of counseling clients typically achieved or exceeded their target score in a large national sample, 62% of CBH clients achieved or exceeded their target score.  

Back to Diagram 1

Diagram 2. How much improvement is experienced by CBH clients

Diagram 2 presents a comparison of the "effect sizes" between clients in a large normative sample (ASIST for Agencies, v. 4.08) and clients in treatment with CBH Outpatient services. Effect size provides information about the size, magnitude, or meaningfulness of the change people experience from counseling; in other words, how much the counseling has helped, rather than simply indicating whether it helped or not. The effect size in Diagram 2 is represented by the percentage of improvement or relief reported by clients.

A simple example can help illustrate: If two pain medications have been shown to relieve pain, but medication A provides much more pain relief than medication B, then medication A is more effective at relieving pain than medication B. In this case the effect size of medication A is larger than for medication B.

In order to provide a fair comparison, the reported effect size for CBH clients is compared to the effect size for clients who receive counseling with other therapists and agencies (i.e., clients in the normative database for ASIST for Agencies, v. 4.08). This is more meaningful in determining effectiveness than a comparison of CBH clients with individuals who are not in counseling at all, since research demonstrates that most counseling is likely to provide relief from distress (similar to the way both medications in the example provide more relief than no treatment at all). Comparing the effect sizes between different therapists can help determine which ones are relatively more effective.

How effect size is calculated:
The effect size is calculated by first subtracting each client's predicted outcome or "target" score from their actual current score. This difference between each client's actual score and target score is known statistically as a "residualized change score". The residualized change score is then divided by the standard deviation of the rating scale which the client completed. (The standard deviation provides information about the probability that a single measurement will vary from the mean.) Dividing the residual change score by the standard deviation helps account for normal variations in scores that might occur by chance, and allows comparison between different clients and different measures. The end result of these statistical calculations is called the effect size.

A positive effect size means that a client is experiencing more improvement than expected (based upon the target score) and the larger the effect size, the greater the amount of improvement. The effect size in Diagram 2 is reported as a percentage, since this is easily understood by most people. Alternatively, the effect size can be reported as the number of standard deviations above or below the mean of the normative sample (ASIST for Agencies, v. 4.08). If effect size is reported in terms of standard deviations, the amount of improvement reported by CBH clients is +0.24, or 0.24 standard deviations above the mean for the normative sample.

The normative sample is represented by a normal distribution, and the mean amount of improvement for the normative sample is 50%. In a normal distribution, 0.24 standard deviations to either side of the mean is equivalent to 9% above or below the mean. By adding 9% (since effect size for CBH clients is a positive effect size) to 50% (the mean), the percentage of improvement reported by CBH clients is 59%.  

Translating this technical information into less technical terms, the average client in the large counseling sample experiences 50% improvement or relief, and the average client of CBH Outpatient services experiences about 59% improvement or relief.

Back to Diagram 2

Intensive Outpatient Program

Outcomes Summary for July 1-December 31, 2016

Outcome Objective Williamsburg
(Evening)
York-Poquoson
(Morning)
Both Programs Percentage of Goal Achieved
Accessibility 85% of individuals seeking entrance to the IOP will begin the program within 5 business days of their initial appointment or will be offered a start date within that period.

76%
16 out of 21 individuals, started the Williamsburg IOP within 5 business days

89%
47 out of 53 individuals started the York-Poquoson IOP within 5 business days

85%
63 out of 74 individuals started IOP within 5 business days

100%
Efficiency IOP will serve a minimum of 60 individuals

40
individuals were served in the Williamsburg IOP

68
individuals were served in the York-Poquoson IOP

98
individuals were served in IOP

163%
Clinical staff will spend at least 60% of their available time in direct care with consumers.

57%
of clinical staff’s time was spent in direct care with individuals in Williamsburg

66%
of clinical staff’s time was spent in direct care with individuals in York-Poquoson

62%
of clinical staff’s time was spent in direct care with individuals in IOP

103%
Effectiveness Individuals will remain drug-free during the program and to gain the tools necessary for a sustained recovery. Substance use testing will occur weekly and aggregate data will show at least 75% normal results.

74%
of specimens sent for weekly substance abuse testing in Williamsburg were negative

66%
of specimens sent for weekly substance abuse testing in York-Poquoson were negative

69%
of specimens sent for weekly substance abuse testing in IOP were negative

92%
Individuals will maintain their gains following the program. We will follow-up with individuals 30-60 days following their discharge from IOP to ask if they have maintained their gains in remaining clean and sober, and in other areas of their recovery and show at least a 75% success rate.

100%
of individuals who were contacted following discharge from Williamsburg maintained their goals of abstinence and recovery

100%
of individuals who were contacted following discharge from York-Poquoson maintained their goals of abstinence and recovery

100%
of individuals who were contacted following discharge from IOP maintained their goals of abstinence and recovery

133%
Consumer Satisfaction IOP consumers will report an average score of 3.5 on a 1-5 scale where 5 is the highest on the CBH consumer satisfaction survey administered in November 2016. 4.14 4.55 4.39 125%

Additional Satisfaction Survey Question:
Staff treats me with dignity and respect.

4.29 4.91 4.67 133%
Discharge Follow-up Survey 85% of IOP consumers will report satisfaction with their IOP treatment on the IOP discharge follow-up survey when contacted 30-60 days following their discharge from IOP.

67%
of individuals who were contacted following discharge from Williamsburg IOP reported satisfaction

100%
of individuals who were contacted following discharge from York-Poquoson IOP reported satisfaction

89%
of individuals who were contacted following discharge from IOP reported satisfaction

105%
©2007-2017 Colonial Behavioral Health - All Rights Reserved