skip to main content

NCHAM: National Center for Hearing Assessment and Management, Utah State University

Last Modified: 05/30/2017

VII. Evaluating Tele-Intervention Outcomes

Featured Articles/ Presentations

A study of costs and outcomes of Tele-intervention Services in Utah

The telehealth literature supports claims that TI may be more effective, cost-effective and timely, with fewer cancelled appointments resulting in greater frequency and consistency of services (need citations of the literature here). Based on their collective experiences to date, NCHAM learning community members suggested a number of potential differences (shown below in Table 3) between early intervention services provided face-to-face or through tele-intervention. These claims, and similar claims in the published literature, are supported mostly by anecdotal evidence. Research to identify actual differences between services provided by tele-intervention and face-to-face home visits is sorely needed, particularly for children who are DHH. Research could provide credible evidence that tele-intervention results in similar or better outcomes for the child and the family. When this evidence is collected, and one considers the cost-effectiveness of TI, then children who are DHH and who are currently underserved due to location and the shortage of trained professionals may be able to achieve outcomes commensurate with their hearing peers. An evidence-based model of tele-intervention could make this dream a reality.

It is important to note, however, that the literature on tele-intervention evaluations is primarily based on pilot studies or pre-post designs. Recommendations for evaluating TI are provided here based on the collective planning of learning community members. Therefore, it provides a springboard for conducting future studies on TI for children with hearing loss.

On This Page

a teleintervention session in progress

Proposed Outcomes of TI

As with any type of intervention, evaluating TI is critical to ensure that it is in fact resulting in the desired outcomes:

  • Increased parent knowledge, skills/competence in fostering development, and satisfaction with services and providers, increased competence with technology
  • Increased Child developmental/communication outcomes
  • Improved parent-child interaction

Additionally, it is important to measure the impact of TI on the following additional outcomes that are associated with delivery of TI:

  • Increased frequency and intensity of intervention
  • Provider skills/competence in coaching, incorp. of natural routines, and use of technology
  • (Arlene)-- Remove this name therapist/interventionist satisfaction, use of time and decreased illness
  • Decreased costs, considering travel time, opportunity cost to providers as well as families

One of the learning communities interviewed families about what they saw as the positive outcomes of TI. The families interviewed reported the following beneficial outcomes based on their personal experience:

  • Greater knowledge of the language development process
  • Increased skills and confidence in promoting their child’s language and listening
  • Increased child responsiveness to parent
  • Improved child language development, listening skills
  • Increased skills of family members as coaches themselves

There are also factors that can influence outcomes that are important to capture:

  • How much acoustic and visual clarity needed in the technology used?
  • Can the provider offer needed emotional support to families when needed via TI?
  • Does TI work for various interventions?
  • Are some parents/children a better fit?
  • Are some “interveners” a better fit? An important place to start in evaluating the cost-effectiveness of TI is to develop a logic model.

Creating a Logic Model to Guide Evaluation

A logic model serves to guide the identification of measurement tools to answer questions such as those posed above. A logic model is a way to graphically depict the resources, actions, and expected results of an intervention. Here are some resources to guide develop of a logic model:

Download an example of a logic model developed by NCHAM [PDF]

Selecting Measurement Tools and Administration

A logic model serves to guide the identification of tools to measure outcomes. The table below corresponds with the NCHAM logic model, delineating the constructs to be measured, the tools to be used, and the frequency of data collection. Creating an outline such as the one provided below is important in creating your evaluation plan. Creating such a table helps ensure that your evaluation plan is comprehensive but also practical in regard to your capacity to employ all the measurement activities.

Measurement Tools and Administration (NCHAM, 2011)

Model of Change Component Instruments Description & Administration Frequency of Administration
Service delivery: Dosage Provider contact logs: Number & duration of sessions conducted; other providers involved; Interruptions in service delivery (technology problems, cancellations, other home visit interruptions) Provider self-report contact form completed weekly by each family in the study Each scheduled session
Service delivery: Technology satisfaction Family Technology Survey & Provider Technology Survey Self-report survey completed by both families & providers in tele-intervention group Pretest; every 6 months
Service delivery: Audiology
  1. Interventionist Hearing & Technology Management Log
  2. Audiological Services Record
  1. Provider report
  2. Provider report
  1. Each scheduled session
  2. Each scheduled session
Service delivery: Costs
  1. Provider cost form, program cost form, time diary
  2. Family Services and Cost Survey
  1. Researcher-developed measures on cost and time data completed by program and providers
  2. Family telephone survey regarding services/supports received & associated costs to the family
  1. Every month
  2. Every 6 months
Family-centered Care, Parent Knowledge & Competence
  1. Home Visit Rating Scales—A; subscales re: parent-child interaction & engagement
  2. Parental Involvement & Self-Efficacy (SPISE)
  3. Language sample: Whitehurst, et al. (1988) coding scheme applied to videotaped sessions
  1. Standardized rating scales coded by research staff
  2. Parent self report survey
  3. Standardized instructions for parent- child activity; parent Mean Length of Utterance (MLU) coded by research staff
  1. Pretest; 1 visit per month
  2. Pretest; every 6 months
  3. Pretest; 1 visit per month
Natural Environments & Provider Coaching
  1. Home Visit Rating Scales, subscales re: home visitor
  2. Coaching Practices Rating Scale
  3. Natural Environments Rating Scale
1-3. Standardized rating scales coded by research staff from videotaped sessions
  1. Every month
  2. Every month
  3. Every month
Child Communication and Development Outcomes
  1. MacArthur-Bates Communication Development Inventory/Inventarios
  2. Battelle Developmental Inventory -2
  3. Preschool Language Scale 4 (PLS-5)
  4. Language sample
  5. Auditory Skills Checklist
  1. Norm-referenced (ages 8-37 mos.) Parent self-report measure; Spanish version available; administered by providers
  2. Norm-referenced (birth – 7 yrs 11 mos.), English/ Spanish versions, administered by research staff
  3. Standardized (birth- 6 yrs 11 mos.), receptive & expressive communication; English & Spanish versions, administered via research staff
  4. MLU’s measured via coding of standardized activity by research staff
  5. Provider–scored checklist
  1. Pretest; every 6 months
  2. Pretest; every 6 months
  3. Pretest; every 6 months
  4. Pretest; monthly
  5. Pretest; monthly

Recent TI Evaluations