Common Metrics Initiative Frequently Asked Questions

Those questions and suggestions can be emailed to the CLIC team (common_metrics@clic-ctsa.org) or submitted through the CLIC website. CLIC will forward them to NCATS.

The Scorecard data for the common metrics is downloaded and sent to the CLIC. CLIC de-identifies and aggregates the data. NCATS receives a report of the de-identified aggregated data. Hubs receive individual reports. Whether hubs choose to include these data in their RPPR is their choice.

Only enter data related to what happened after the award. If your hub has historical data that reflects information for earlier time periods, it should be considered in your work on developing your Story Behind the Curve.

The Operational Guidelines were updated to indicate CTSA program hubs should use the calendar year for each of the Common Metrics.

The deadline is the August 31st of each year.

If they have completed the TL1 training program and are no longer on the TL1 grant, they can be assessed for whether they are engaged in research (add them to the denominator of the metric).

If they’ve completed the TL1 training program and are participating in additional training that has a research component, they are considered “engaged in research” (add them to the numerator of the metric).

If the residency or PhD program includes dedicated time for research, they are engaged in research and added to the numerator.

You should be collecting data on the number and percentage of underrepresented persons. You do not need to break it down into the three categories. Please see NIH's Interest in Diversity statement at https://grants.nih.gov/grants/guide/notice-files/NOT-OD-18-210.html

It is not possible to have a KL2 Program without a U54 CTSA Program award.

Yes.

Please include only CTSA program-funded KL2 and TL1 scholars.

NCATS has clarified that non-CTSA grant funded scholars and trainees who participate in your KL2 or TL1 program should not be included.

Graduates should be added to the denominator of the metric starting in the calendar year that they finish their program. In your example, you would add the T scholar in the denominator for 2014. You should then also assess their eligibility for the numerator of the metric (i.e., are they involved in CTS research) starting in 2014.

If your program was still "running" and you had bridge funding and none were awarded, please enter 0 (zero). If you "froze" the program then leave the field blank - and please put a note in your TTC plan.

(8/7/2019)

Remove them from the numerator and denominator of the metric.

Yes, remove them from the numerator and denominator of the metric.

Yes, this is considered to still be engaged in research.

See Career Metric Operational Guiedelines. “If primary role is as a clinician: some effort (e.g., 20%) in research or as a site PI for industry-sponsored clinical trials.”

Following completion of CTSA-funded training as a KL2 scholar, they are eligible to be counted for the metric. If they are engaged in further training by another K award, they are considered engaged in research. However, if a scholar leaves the KL2 program without completing the full training program requirements, they are excluded.

If medical students or residents do not have dedicated time for research, they are not considered to be “engaged in research.”

The following is a Notice of NIH’s Interest in Diversity: https://grants.nih.gov/grants/guide/notice-files/NOT-OD-18-122.html
As presented in the notice: 
C. Individuals from disadvantaged backgrounds, defined as: 
1. Individuals who come from a family with an annual income below established low-income thresholds. These thresholds are based on family size, published by the U.S. Bureau of the Census; adjusted annually for changes in the Consumer Price Index; and adjusted by the Secretary for use in all health professions programs. The Secretary periodically publishes these income levels at http://aspe.hhs.gov/poverty/index.shtml.  
2. Individuals who come from an educational environment such as that found in certain rural or inner-city environments that have demonstrably and directly inhibited the individual from obtaining the knowledge, skills, and abilities necessary to develop and participate in a research career. 

The disadvantaged background category (C1 and C2) is applicable to programs focused on high school and undergraduate candidates. Thus, as the career metric is focused on TL1 trainees and KL2 scholars the disadvantaged category does not apply to this metric. 

Graduates are to be assessed for their engagement in clinical and translational research annually.

If they have completed the TL1 training program and are no longer on the TL1 grant, they can be assessed for whether they are engaged in research (add them to the denominator of the metric).

If they’ve completed the TL1 training program and are participating in additional training that has a research component, they are considered “engaged in research” (add them to the numerator of the metric).

If the residency or PhD program includes dedicated time for research, they are engaged in research and added to the numerator.

The list of activities that indicate engagement in research are examples only and not criteria.

If they are not directly funded by NCATS then they are excluded from the metric.

We are using the NIH definitions. The Operational Guideline has a link to more information about these definitions. The most up to date versions of the Operational Guidelines can be found on the Established Common Metrics page.

This is the link to the most current NIH definition: https://grants.nih.gov/grants/guide/notice-files/NOT-OD-18-210.html

The total number of program graduates over time is cumulative. It gets updated once a year by adding the new number of graduates to the previous total to make an updated denominator for the metric. The numerator, the number and percent of graduates from the denominator who are currently engaged in clinical and translational research is assessed each year.

eRA will be a source of information but it won’t be a complete source, because they are not all inclusive; there are activities that are outside of the system.

Exclude the pre-docs who have only attended the 12-week summer experience from your Careers Common Metric counts. These don’t sound like they are comparable to TL1 students who you might expect to pursue research careers.

NCATS expects that hubs collect this data on an annual basis from each graduate for each year after completion of the program (for graduates since January 1, 2012). If a graduate does not respond in a given year, they are not included in either the numerator or denominator for that year. However, an attempt should be made to follow up with them again the next year to determine their status. A graduate should only be permanently removed from the denominator if they are deemed “lost to follow-up”, (e.g., no address or email to send a survey to, persistent refusal to respond).

There is no limit on the number of years currently specified.

If they have completed the TL1 training program and are no longer on the TL1 grant, they can be assessed for whether they are engaged in research (add them to the denominator of the metric). For KL2s, following completion of CTSA-funded training as a KL2 scholar, add them to the denominator of the metric.

They will be updated for the 2019 reporting period.

The metric includes all patients in the research data warehouse– inpatient, outpatient, ER, etc.

Yes. If you experience any problems with the script, please contact common_metrics@clic-ctsa.org.

For the launch of the metric the script did not include min/max dates. The scripts will be updated throughout the life of the metric. Check the Github site for the most current script.

No, only notes: admission, progress, discharge, procedure, etc.

The scripts are available on Github: https://github.com/ncats/CTSA-Metrics.

Typically the TTC plan is for one year. If you are entering longer-term goals, please identify that the strategies will be addressed over the next ## years.

ICD 9/10 is used in healthcare to code both diagnoses and procedures – be sure to enter data under the appropriate performance measure.

For hubs who have achieved, 100% or near 100%, please document the strategies that have helped your hub to reach that target.
Also, you may want to begin to identify additional metrics that would be helpful consortium-wide.

Please enter this information in the Story Behind the Curve section of your TTC plan.

Hubs are welcome to use Postgres (or another data model). Hubs will need to create their own query and make sure that they match the measures of the existing script. If a hub creates a new query, please considering putting it in Github so that other hubs can use it. [SQLrender (https://github.com/OHDSI/SqlRender) allows OMOP sites to run against several other back end DBs beyond MS SQLserver and ORACLE: PostgresSql, Microsoft Parallel Data Warehouse (a.k.a. Analytics Platform System), Amazon Redshift, Apache Impala, Google BigQuery, BM Netezza.] At this time, scripts are available for these data models: OMOP, PCORnet, and i2b2/ACT. Hubs that use TriNetX, can contact TriNetX to get data.

Regarding observations, the scripts are looking for a general information on if observations are being recorded at all (present/absent) in hoped that future research data warehouse characterizations could include vital signs.

That is not an uncommon situation. Please collect the data that you can and then list this issue under the Negative: section of your Story Behind the Curve (in your TTC plan). For your TTC plan, consider partnering with your IT team and determine if there are strategies that could be implemented to improve your access to the data.

Administrative gender is a person’s gender (from HL7 and OMB), which describes their sex. It’s a standard dataset (M or F). The domain reflects if it is recorded as a count.

For the Informatics metric launch, it’s the percentage of people who had those values at the time they were being seen.

The TTC plan entered for the August 30th deadline, represents the hub’s strategies for addressing how they will turn the curve in the coming year. TTC plans are entered annually, so progress is represented by comparing progress since the previous year.

Collect the data that you have available. If you can collect data from more sites next year it will make your data look even better.

The focus of the Informatics metric is the presence or absence of data in the research data warehouse (RDW). The TTC plan should be focused on how to get absent data, or how to improve accessing data.

Include information that you think would be most helpful for investigators across the consortium.