Australian Health Programs: Evaluating their Effectiveness in Health Insurance – Part II

Date12th January 2024

As introduced in our first blog in the “Australian Health Programs” series, Private Health Insurers (PHIs) and providers have been looking to increase the scale and effectiveness of health programs in recent years. The effectiveness of health programs can be evaluated in terms of clinical outcomes, cost-effectiveness, patient satisfaction, retention and other outcomes.

This blog is the second in a three-part series where we provide a framework to evaluate the effectiveness of the programs offered, to guide decision-making on future investments and tailor programs. This instalment outlines an approach to robust cost-effectiveness evaluations and common pitfalls to look out for, such as lack of data collection, poor selection of control groups and statistically invalid conclusions.

Control group selection

Health program evaluations need to establish causal relationships between variables to produce valid conclusions. That is, an evaluation should determine whether observed changes can be attributed to the relevant health program and not to other possible causes.

Randomised Controlled Trials (RCT) are widely regarded as the optimal approach to establishing causal relationships. The randomisation of selecting participants for treatment and control groups before the start of the intervention reduces the effects of participant characteristics such as age or health status and helps to establish cause and effect.

For Health Insurers, RCTs are often impractical or costly, as insurers may not have influence over program acceptance or inappropriateness as insurers may be reticent to deny certain policyholders’ access to care should they be eligible. In cases where there are no pre-defined control groups, we’re left with a choice, either

  1. Select appropriate control groups post-program or

  2. Do not select a control group and apply single-group evaluation methods.

Selecting an appropriate comparison group retrospectively

An appropriate comparison group can be created at the evaluation stage by creating a group with similar characteristics as the treatment group. The choice of technique will depend on the quality, granularity, and volume of available program data.

A traditional (and the simplest) method selects groups based on matching basic member characteristics such as age, sex and known program participation criteria. More advanced evaluation techniques now determine program participation propensity scores (or equivalents) to evaluate the likelihood of individuals being appropriate program participants. These techniques use a broader set of characteristics including claims history (split by diagnosis), geographic location and other attributes.

Evaluating the participant group without a comparison group

Instead of establishing a comparison group after the program, an alternative approach evaluates a program participant group against themselves; comparing their recent experience against their past experience at different time intervals. One approach is a pre-, during- and post-program approach where evaluation metrics are measured at regular intervals before, during and after the program end date. An improvement on the above takes measurements at multiple periods before the program; some time before and immediately before the program start date. This indicates the stability of experience leading up to the start of the program.

Integrating an expectation of trend is an important consideration for this type of analysis, via either simple trendline analysis, if there is low variance in historical data or more complex multi-variable analyses.

Data collection

How successful these methods will be depends on the available data, highlighting the importance of data collection for health program evaluations. Poor-quality data or a lack of data are common issues. Ideally, data requirements should be considered before the start of a health program to ensure appropriate evaluations can be performed. However, this is often not the case and data cleansing and enhancements are required at the evaluation stage.

Common data requirements include clinical codes, such as Diagnosis Related Group (DRG), demographic characteristics, provider data, and program information for each participant, including program start and end dates, and program referral source (for example self-referral or specialist referral). Data collection of relevant evaluation metrics is another key factor of successful evaluations, and so we suggest insurers invest in data quality via:

  • Implementing robust data collection protocols that include clear guidelines on data sources, formats, and frequencies

  • Close collaboration with relevant healthcare providers to establish a well-defined data collection framework.

Evaluation metrics

Appropriate evaluation metrics depend on the objectives of the program and the evaluation. Cost-effectiveness evaluations commonly use metrics such as number of hospital admissions, re-admissions and length of stay. For programs with multiple program providers, consistent measurement of evaluation metrics across providers is important for valid comparisons to be made.

Other quantitative evaluation metrics are summarised in the table below, along with the evaluation approaches discussed above.

Statistical assessment of results

Statistical significance can be used to ensure that conclusions about any differences observed between treatment and control groups are not due to chance. Confidence intervals can also be computed to assess the uncertainty of results.

The table below outlines a high-level checklist of the evaluation features you should be thinking about for your next evaluation.

Thinking through each of the components above can help avoid some of the common pitfalls in health program evaluations and lead to successful appraisals with statistically valid conclusions. Finity can assist by measuring the impacts of programs on fund performance and members outcomes – allowing you to continue to innovate and refine your offering.

In the third and final blog in this series, we will explore the implications of health programs for insurers and claims managers, and discuss future trends and opportunities.

Learn more

To learn more about our practices, reach out to the Finity team.