It can be hard to measure the real-world impacts of research. We often use proxy measures like the number of grants awarded or scientific publications produced by a program, but these measures are quite removed from what our work actually means to our stakeholders: our patients, communities and policy makers.
Over the past four years CLIC has released a number of publications that reframe the idea of research impact. Our model, which assesses, quantifies and communicates the impact of the CTSA Program, emerged from CLIC’s Common Metrics Initiative (CMI) and represents a convergence of several evaluation models and resources. Each model has strengths, but combined, they are a powerful measure of impact. We call this Stakeholder-Engaged Research Impact Evaluation (SERIE).
The Five Steps of SERIE
We can think of SERIE as five steps in a pipeline: Engagement, Definition, Data Collection, Analysis, and Communication. SERIE engages multiple stakeholders to define potential intended and unintended network impacts and the metrics that will help measure those impacts. Those impact metrics are gathered and analyzed using both qualitative and quantitative methods and are communicated from a stakeholder perspective in a variety of media.
Synthesizing Evaluation Models
SERIE is a flexible and nimble process built upon existing models and principles geared toward impact evaluation at the individual or institutional levels. By pulling from each of these models, we can begin to get a clearer picture of how networks impact health and society.
Common Metrics
The Common Metrics Initiative assesses the progress of individual hubs toward the strategic goals of the CTSA Program. Built on the Results Based Accountability™ framework, the initiative measures specific program characteristics individually (e.g. time to IRB approval, publications, trainees) and gathers Turn the Curve narratives for hub improvement. While hubs have used common metric results to create local change, this initiative is not designed to measure or demonstrate network-level change or impact.
Logic Models
Logic Models are a contextual framework used to evaluate research translation, community engagement and impact. These models engage stakeholders to define meaningful and distinguish outputs (like publications and grants) from impact (like real-world changes in community trust or research participation).
RE-AIM Framework
The RE-AIM framework was developed to assess the adoption and implementation of evidence-based public health interventions and knowledge gleaned from dissemination and implementation science. RE-AIM stands for five quantifiable elements: reach, efficacy, adoption, implementation, and maintenance. Originally a quantitative evaluation tool, it has since been adapted for planning and qualitative evaluation.
Translational Science Benefits Model
The Translational Science Benefits Model (TSBM), developed at Washington University in St. Louis, evaluates the real-world impacts of clinical and translational science at the individual or institutional level. The TSBM identifies benefits in four areas: clinical and medical benefits (procedures, guidelines); community and public health benefits (health activities, care); economic benefits (commercial products, financial savings); and policy and legislative benefits (policies, legislation).