Illustrating the potential impact of the CTSA Program: a closer look at the Common Metrics Initiative

“As a result [of the Common Metrics report], our team has had conversations about what we are doing, how we are doing it and how we can do better." - Beatrice Boateng, Ph.D., director of evaluation at the Translational Research Institute (TRI), University of Arkansas for Medical Sciences (UAMS)

 

The Clinical and Translational Science Awards (CTSA) Program is charged with accelerating and improving clinical and translational science – however, the potential impact of the program is, at times, difficult to illustrate. In an effort to help evaluate and highlight how the program can directly affect and improve public health outcomes, the CTSA Program implemented the Common Metrics Initiative.

The initiative, which is based on the principles of the Results-Based Accountability framework, employs a set of standard metrics for use in collaborative management, allowing program members to effectively measure the impact of their efforts, and shed light on potential areas for improvement. When analyzed appropriately, these metrics help focus consortium and hub activities on making significant, measurable improvements in translational science and workforce development.

In 2017, the Center for Leading Innovation and Collaborative (CLIC) assumed the role of Tufts University in coordinating and managing this initiative. Last month, CLIC sent out their first Common Metrics Initiative Hub reports based on 2016 data submitted by the hubs. These initial reports analyzed three key metrics:

  • IRB Duration Metric
    • This metric assesses the median number of calendar days from the date of receipt of the official IRB application to the official IRB approval date.
  • Pilot Funding Publications Metric
    • This metric assesses the number and the percent of research projects that expended hub pilot funding and resulted in one or more publications or in additional funding.
  • Careers in Clinical and Translational Research Metric
    • This metric assesses the number and the percent of scholars and trainees who completed the KL2 and TL1 programs and who are engaged in clinical and translational research. It also measures the extent to which female and underrepresented minority researchers are maintaining research careers.

The hub-specific reports highlighted each hub’s own data pertaining to the above metrics and outlined where they landed in comparison to the other de-identified hubs, ultimately allowing each hub to see their data in context. The National Center for Advancing Translational Sciences (NCATS), which is the funding agency for the CTSA Program, also received a fully de-identified report for a consortium-wide view.

“Our hope is that once the metrics are collected, analyzed and presented in a digestible way, the hubs may be able to see issues they were previously unaware of and take action to improve processes,” said Raquel Ruiz, associate director of Common Metrics at CLIC. “And at a broader level, we hope this initiative allows us to demonstrate the value and the collective impact of the program as a whole.”

While there’s still room for improving how the data is collected, reported and analyzed, many of the hubs are already realizing the benefit of taking time to reflect upon the critical components that help ensure translational science success.

“As a result [of the Common Metrics report], our team has had conversations about what we are doing, how we are doing it and how we can do better," said Beatrice Boateng, Ph.D., director of evaluation at the Translational Research Institute (TRI), University of Arkansas for Medical Sciences (UAMS). "We now have a plan in place to work on standardizing the institutional and organizational processes for future data reporting.”

Following the delivery of the 2016 reports, each hub received a feedback survey allowing them to share their thoughts and questions with CLIC, and encouraging them to consider potential opportunities for use of the data internally and in collaborations with other hubs.

“We want this to be a collaborative effort between the hubs and CLIC,” said Ruiz. “We can only improve future reports if we work together and gather constructive feedback from those most impacted by the presented data.”

CLIC plans to continue related conversations on October 23 at 7:30 a.m. prior to the 2018 CTSA Program Fall Meeting. The Common Metrics team is also hard at work compiling and analyzing data for the next iteration of the Hub Reports. Learn more by visiting the CLIC’s Common Metrics webpage.

 


Twitter logoFollow CLIC
Twitter logoFollow NCATS