Common Metrics Initiative Frequently Asked Questions

Scorecard

Can I use the “Import Data Values” setting in the Scorecard software for importing the Common Metrics into Scorecard from other applications?

Using this feature is helpful when handling a large amount of data on a frequently repeating schedule. Given the small amount of data required for this project, the economies of scale that usually make importing a good idea may not apply. Also, narrative text can’t be imported from Excel since it is rich formatted text. You would still need to add narrative to the software.

Can we use just one Scorecard for all the strategies in our plan for a given metric? It would be easier to just look at one Scorecard and see if our strategies are working, instead of having to switch back and forth.

If you want to use additional Scorecards, please put them in a section other than Common Metrics.

Do I create a separate scorecard for each program?

No, you can have multiple programs on a single Scorecard.

How do I include the forecast in my Scorecard?

The forecast is not required, but you can include it. If you want to include a forecast in Scorecard, place it in the Current Target Value box when you enter your current data value.

How do you edit tags?

Please do not edit tags. Information for the data uploads is based on specific tag names. If you enter you data under a different tag, your data will not be reported accruately.

How many strategies should we include in the scorecard? We understand there are many action plans that might fall under each strategy. Do we only enter those strategies that we planning to implement at this time, or all strategies we've considered?

There is no limit to the number of strategies that you can include in Scorecard. Enter the strategies that you believe you can reasonably implement - you can always enter more strategies in Scorecard at a later time.

Please review the TTC Planning tools on the CLIC website for additional suggestions. You can use these tools to keep notes of your entire TTC Plan development process.

How will the data entered into the Scorecard be shared? Will it be entirely private, or shared in some way?

Each hub is allocated two Scorecard user licenses - these are the only individuals in your hub that can see your data. The CLIC, as an administrator, can see what the hubs enter into the Scorecard software.

Hubs can purchase additional licenses from Clear Impact.

What time periods should be in Scorecard? Monthly, quarterly, or just one annual data point?

Common metric data and the Turn-the-Curve plans are entered annually. All metric data is downloaded for analysis on or near August 31st of each year. The CLIC communicates due dates to the hubs.

Who is responsible for the costs of the Scorecard software?

The CLIC provides two licenses for each hub. To request a license, contact common_metrics@clic-ctsa.org.

Implementation

Is there a specific mechanism for submitting language to revise the guidelines and/or communicate any additions (e.g., definitions of engaged in research)?

Those questions and suggestions can be emailed to the CLIC team (common_metrics@clic-ctsa.org) or submitted through the CLIC website. CLIC will forward them to NCATS.

Metrics - General

Is a Turn the Curve plan always needed [for the metric scores that require them]? We've regularly hit or exceeded our targets in for two of the common metrics.

TTC planning is important for the improvement of these metrics at the hub level, but the hope is also that we can share those best practices that are very successful in the hubs across the CTSA Consortium. We ask you to develop a TTC Plan and provide in your Story Behind the Curve some of the underlying factors that lead you to have such great success with a particular metric, as well the strategies that you used to sustain that performance and who the partners are involved to sustain that performance.  We ask you to document what you believe are the underlying factors and strategies that you use to achieve your performance. Hubs may find that there are few changes to the TTC plan year to year - this is acceptable.

Is there an expectation about how the Common Metrics will be used in annual reports?

The Scorecard data for the common metrics is downloaded and sent to the CLIC. CLIC de-identifies and aggregates the data. NCATS receives a report of the de-identified aggregated data. Hubs receive individual reports. Whether hubs choose to include these data in their RPPR is their choice.

Our CTSA first received funding last August, so currently there is very little data for what has happened since then. Should we look back at data from before we received our CTSA in order to have more of a trend, or should it be based solely on data after

Only enter data related to what happened after the award. If your hub has historical data that reflects information for earlier time periods, it should be considered in your work on developing your Story Behind the Curve.

What definition of “annually” should we be using? Calendar year, grant year, fiscal year, or academic year?

The Operational Guidelines were updated to indicate CTSA program hubs should use the calendar year for each of the Common Metrics.

What is the deadline for completing data entry for the Common Metrics?

The deadline is the August 31st of each year.

With the Turn the Curve methodology, we were asked to look for a baseline and do a forecast. Our baseline is currently just one data point. Should we be thinking about also collecting metrics for previous calendar years?

If the data are available and you have the resources to go back and calculate them for previous years, this could be helpful in determining your forecast; however, per the Operational Guidelines, this is not required.

Metrics - Careers

All of our TL1 grantees are pre-doctoral scholars, after they complete the award they have years of medical school, etc. before starting their research careers. Should they all be excluded because they are still considered "in training programs"?

If they have completed the TL1 training program and are no longer on the TL1 grant, they can be assessed for whether they are engaged in research (add them to the denominator of the metric).

If they’ve completed the TL1 training program and are participating in additional training that has a research component, they are considered “engaged in research” (add them to the numerator of the metric).

If the residency or PhD program includes dedicated time for research, they are engaged in research and added to the numerator.

Are we collecting data on the number and percentage of each of the three underrepresented categories (racial and ethnic groups, individuals with disabilities and those from disadvantaged backgrounds) or just the overall number and percentage?

You should be collecting data on the number and percentage of underrepresented persons. You do not need to break it down into the three categories. Please see NIH's Interest in Diversity statement at https://grants.nih.gov/grants/guide/notice-files/NOT-OD-18-210.html

Can a CTSA hub have a KL2 program that is only institutionally funded?

It is not possible to have a KL2 Program without a U54 CTSA Program award.

Do we combine post-doc and pre-doc trainees in the common metrics for the TL1 program?

Yes.

Do we include institutionally funded KL2 and TL1 scholars or only those whose training was paid for by the grant?

Please include only CTSA program-funded KL2 and TL1 scholars.

Do we include KL2 and TL1 scholars that are institutionally funded or just NIH/NCATS funded?

NCATS has clarified that non-CTSA grant funded scholars and trainees who participate in your KL2 or TL1 program should not be included.

For T scholars, do we count them for the year ending data in the same year they finish their program or not until the following year? Ex: T scholar finished program in June 2014. Do we count their responses in the 2014 data or wait until 2015?

Graduates should be added to the denominator of the metric starting in the calendar year that they finish their program. In your example, you would add the T scholar in the denominator for 2014. You should then also assess their eligibility for the numerator of the metric (i.e., are they involved in CTS research) starting in 2014.

How should we report KL2/TL1 data if we were in a no-cost extension from 6/2017-6/2018? Some of the Ks were active before 6/2017 so we reported on them for CY2017, but for CY 2018, there were no active K awards. Our funding was renewed 6/2018.

If your program was still "running" and you had bridge funding and none were awarded, please enter 0 (zero). If you "froze" the program then leave the field blank - and please put a note in your TTC plan.

(8/7/2019)

If a graduate is lost to follow up, should they be removed from the numerator and denominator of the metric? If a graduate does not respond to a survey, should they be included in the denominator?

Remove them from the numerator and denominator of the metric.

If a graduate is lost to follow-up (e.g., no address or email to send a survey), should they be removed from the numerator and denominator of the metric?

Yes, remove them from the numerator and denominator of the metric.

If a PhD student does not get compensated for time for research, but that research is a required part of the degree and they receive tuition assistance, should this be considered "engaged in CTR"?

Yes, this is considered to still be engaged in research.

Is there a minimum % for someone involved in research? If someone reports they are only involved in research 5% does that count or is there a minimum of for instance 20%?

See Career Metric Operational Guiedelines. “If primary role is as a clinician: some effort (e.g., 20%) in research or as a site PI for industry-sponsored clinical trials.”

Many of our participants exit the KL2 program after getting into another “K” program. We view this as a success, but it seems that the definition does not view this as a success. Please clarify.

Following completion of CTSA-funded training as a KL2 scholar, they are eligible to be counted for the metric. If they are engaged in further training by another K award, they are considered engaged in research. However, if a scholar leaves the KL2 program without completing the full training program requirements, they are excluded.

Many of our short-term TL1 trainees return to medical school and go on to residency after completing the program. Would that be considered “engaged in research”?

If medical students or residents do not have dedicated time for research, they are not considered to be “engaged in research.”

Our hub will not have any eligible graduates in the denominator as we are a relatively new hub. Therefore should we do any Turn the Curve strategizing for what we think is happening?

Yes. Include this information in your Story Behind the Curve and develop strategies that you may want to implement (e.g., adding an exit interview to your process for departing graduates) even before you are able to start collecting data for the metric.

Please provide some guidance on how to define individuals from "disadvantaged backgrounds".

The following is a Notice of NIH’s Interest in Diversity: https://grants.nih.gov/grants/guide/notice-files/NOT-OD-18-122.html
As presented in the notice: 
C. Individuals from disadvantaged backgrounds, defined as: 
1. Individuals who come from a family with an annual income below established low-income thresholds. These thresholds are based on family size, published by the U.S. Bureau of the Census; adjusted annually for changes in the Consumer Price Index; and adjusted by the Secretary for use in all health professions programs. The Secretary periodically publishes these income levels at http://aspe.hhs.gov/poverty/index.shtml.  
2. Individuals who come from an educational environment such as that found in certain rural or inner-city environments that have demonstrably and directly inhibited the individual from obtaining the knowledge, skills, and abilities necessary to develop and participate in a research career. 

The disadvantaged background category (C1 and C2) is applicable to programs focused on high school and undergraduate candidates. Thus, as the career metric is focused on TL1 trainees and KL2 scholars the disadvantaged category does not apply to this metric. 

Regarding "engaged in clincial research", if a scholar takes a break in order to start a family or care for parents, do we not include him in our Scorecard for that particular year or so, even though he will resume his research?

Graduates are to be assessed for their engagement in clinical and translational research annually.

Several of the TL1 trainees at our institution are in the MD/PhD program. Once they have completed their dual degrees, they continue with a residency for several additional years as well as subsequent fellowships. Is this considered “still in training”?

If they have completed the TL1 training program and are no longer on the TL1 grant, they can be assessed for whether they are engaged in research (add them to the denominator of the metric).

If they’ve completed the TL1 training program and are participating in additional training that has a research component, they are considered “engaged in research” (add them to the numerator of the metric).

If the residency or PhD program includes dedicated time for research, they are engaged in research and added to the numerator.

Should hubs apply the "Engaged in research" bullet points as a closed set of criteria that defines engagement or as examples that hubs should use to assess engagement in research?

The list of activities that indicate engagement in research are examples only and not criteria.

Should we include Ks and Ts who fully participate in the CTSA Career Development Program that are funded by institutional funds, not the CTSA grants?

If they are not directly funded by NCATS then they are excluded from the metric.

There are multiple definitions for underrepresented minorities. Which one are we using?

We are using the NIH definitions. The Operational Guideline has a link to more information about these definitions. The most up to date versions of the Operational Guidelines can be found on the Established Common Metrics page.

This is the link to the most current NIH definition: https://grants.nih.gov/grants/guide/notice-files/NOT-OD-18-210.html

This is a cumulative metric. If a participant is engaged in research 1 year after program completion but no longer is 3 years after completion, how should this be tracked?

The total number of program graduates over time is cumulative. It gets updated once a year by adding the new number of graduates to the previous total to make an updated denominator for the metric. The numerator, the number and percent of graduates from the denominator who are currently engaged in clinical and translational research is assessed each year.

To what extent can eRA extract help with this data collection?

eRA will be a source of information but it won’t be a complete source, because they are not all inclusive; there are activities that are outside of the system.

We are a relatively new hub and do not yet have any eligible graduates to be included in the Careers Common Metric. Should we do Turn the Curve planning for what we think will happen?

Yes. If you do some thinking about this now, you may identify some additional performance measures or strategies that you may want to implement (e.g., adding an exit interview to your process for departing graduates) even before you are able to start collecting data for the metric.

We funded short-term pre-docs (12-wk summer experience) 2006- 2014 and then discontinued it and have just supported year-long pre-docs for the last 2 years. Should these short-term awardees be excluded from the common metric counts?

Exclude the pre-docs who have only attended the 12-week summer experience from your Careers Common Metric counts. These don’t sound like they are comparable to TL1 students who you might expect to pursue research careers.

What does "current" mean in terms of research engagement and reporting on it?

NCATS expects that hubs collect this data on an annual basis from each graduate for each year after completion of the program (for graduates since January 1, 2012). If a graduate does not respond in a given year, they are not included in either the numerator or denominator for that year. However, an attempt should be made to follow up with them again the next year to determine their status. A graduate should only be permanently removed from the denominator if they are deemed “lost to follow-up”, (e.g., no address or email to send a survey to, persistent refusal to respond).

What is the expectation for the number of years that participants are tracked upon completion of a program?

There is no limit on the number of years currently specified.

When is a TL1 or KL2 student eligible to be counted for the careers metric?

If they have completed the TL1 training program and are no longer on the TL1 grant, they can be assessed for whether they are engaged in research (add them to the denominator of the metric). For KL2s, following completion of CTSA-funded training as a KL2 scholar, add them to the denominator of the metric.

Metrics - IRB

A protocol is submitted in 2014 but approved in 2015. How is it counted?

It is the approval year that determines which cohort of data the protocol is included with. If it is submitted in 2014 and approved in 2015, count it in 2015.

Are there two IRBs for a CTSA that is multi-institutional? Will we be able to report the median time to IRB approval for two institutions if there are two IRBs for two separate institutions within the hub?

For this metric, you report the median duration from the IRB(s) from your CTSA hub's primary institution. If your primary institution has three separate IRBs, then the data from all three should be combined together to compute the median. But if you have more than one institution, one of which is primary and the others which are not, you only need to report the data from the IRB(s) of the primary hub institution. 

Does it matter whether the protocols are industry or investigator initiated?

Protocols include all clinical research (including multisite studies). There is no exclusion for industry-related projects. So, if they are industry initiated or investigator initiated, they are both included in the metric.

Does the metric apply to only CTSA-associated studies or is institutional wide?

Institutional-wide, not just CTSA-related studies.

For the inclusion, do we include all submissions that occur during the time period, or all that have an end date in the timeframe?

Just include the protocols that have received IRB approval during the timeframe.

How is clinical research defined?

Please use this link (it's also available in the Operational Guidelines) - https://grants.nih.gov/grants/glossary.htm#ClinicalResearch.