Common Metrics Initiative Frequently Asked Questions

Metrics – Informatics

The TTC plan entered for August 30, 2019, is a plan for which year? When do we report on progress afterwards?

The TTC plan entered for the August 30th deadline, represents the hub’s strategies for addressing how they will turn the curve in the coming year. TTC plans are entered annually, so progress is represented by comparing progress since the previous year.

We are a multi-site CTSA, and we can only access one site. Is there a way to indicate that?

Collect the data that you have available. If you can collect data from more sites next year it will make your data look even better.

We don’t have direct influence over the data domains in the Informatics CM. Are you looking for something different in this CM than in the other Common Metrics?

The focus of the Informatics metric is the presence or absence of data in the research data warehouse (RDW). The TTC plan should be focused on how to get absent data, or how to improve accessing data.

We filter our patient table to only include patients that have visit data in our location because we have billing data from several hospitals where our faculty work, but no actual visit data for those patients – no other information is available.

Include information that you think would be most helpful for investigators across the consortium.

We have information for labs, procedures and medications. What do we need to report for the RxNorm domain?

Report on just the values for the RxNorm.

We noticed that there is no script for the Notes or Observations. Why is that the case and is that data is still needed?

For both the observations and notes domains, the determination is about the “presence” or “absence” of data.
Observations are a single variable or a single value – for example a test result, chest x-ray, etc.
Notes are free text documentation entered during a clinical encounter.
The scripts contain queries/sql for both the count and percent of free text notes, if the data model doesn't supports free text it
is not included.

Were the Informatics Webinars/Office Hours recorded? If so, where can I find the slides and the recordings.

All of the Informatics Webinars and Office Hours were recorded. The slides are available on the CLIC website.

The recordings are available on the Google Drive. If you want access to the Google Drive, please contact common_metrics@clic-ctsa.org.

What are observations?

Observations are a measurement of a single variable or a single value derived logically and/or algebraically from other
measured or derived values. A test result, a diastolic blood pressure, and a single chest x-ray impression are examples of
observations. Essentially, this is a yes/no variable (1 or 0) – the presence or absence of observation data.

What are the steps for loading our data into the Informatics Scorecard?

The steps for entering the Informatics data are on the CLIC website - Informatics Step-by-Step Scorecard Entry Resource.

What exactly is being recorded for the Medications/Drugs domain?

This data addresses the question, “Are you recording your patients using this as your standard vocabulary over the number of patients you have in your repository?” No medication counts as a medication.

What time interval should we include into metric?

During the launch phase, please include all of the information from the research data warehouse. For subsequent reporting, please check the CLIC website for updates.

When we run the scripts, is the worksheet automatically populated? How does it get uploaded to the Scorecard?

You will need to copy and paste the script output into the worksheet and then upload the worksheet into Scorecard.
You will also need to manually enter the data values for each of the data values into Scorecard.

Will CLIC provide hubs with feedback on their draft TTC plans?

If you want feedback, please submit a request to common_metrics@clic-ctsa.org.

Would you please provide a definition for each of the standard value data elements?

LOINC – Logical Observation Identifiers Names and Codes (https://loinc.org) – a set of codes that are used to identify test results and/or clinical observations for electronic reporting.
RxNorm ID (https://www.nlm.nih.gov/research/umls/rxnorm/) – a set of codes that provide normalized names for generic and branded clinical drugs.
ICD 9/10 – International Classification of Diseases (https://en.wikipedia.org/wiki/ICD-10) – set of codes for clinical diagnoses and/or procedures.
CPT – Current Procedural Terminology (https://www.ama-assn.org/amaone/cpt-current-procedural-terminology) – set of codes used to report outpatient clinical services and/or procedures for billing and reporting purposes.
SNOMED (www.snomed.org) – a collection of medical codes and terms used in clinical documentation.

Reports

How do I get access to my hub report if I am a PI, Admin or CM Lead and did not have a CLIC website account before 2/28/19?

Please email common_metrics@clic-ctsa.org to request access after you have created your account OR ask for access from the PI, Admin or CM Lead. Instructions for adding access can be found on the hub report page.

How do I get access to my hub report if I am not a PI, Admin or CM Lead?

To gain access, you need to be added to the group by your PI, Admin or CM Lead. Those currently with access can login to the page and follow the Instructions on the hub page to give access to another user.

I received an email I have access to my hub's CM Report, how do I access it?

Hub reports are available to the hub PI, Administrator and Common Metrics lead (given they have current CLIC website accounts.) To access your hub’s Common Metrics Report, please log in to the CLIC website. You can access the link to your hub’s report page two ways:

  1. From your Account page
    • Look for the section labeled Common Metrics Reporting
  2. From the Common Metrics Initiative page
    • In the right sidebar, look for ‘Common Metrics Reporting

Metrics - Accrual

3 hospitals – 1 has fully implemented CTMS, 1 has plans to implement, and 1 has no plans to implement. What do we do?

While ideally hubs should include all eligible clinical trials, if that is not immediately feasible, they should either randomly sample from all eligible clinical trials or, secondarily, select a nonrandom targeted sample of clinical trials that can be a focus for accrual performance improvement. Please see the Operational Guidelines Section 4 Technical Description.

Are there specific guidelines on how the power calculations should be determined?

The power calculation provided by the statistician or other investigator in the IRB approved protocol should be used. It is not the intent that the power calculation be recalculated for the purposes of the accrual metric.

Do we report on trials that only have a certain number of participants? Or just the 10 largest? Is it only clinical trials or does it include prospective cohort?

As noted in the Operational Guidelines Section 5 – General Inclusion/Exclusion Criteria for more information, exclude all trials that have an initial targeted number of less than 5 participants at your site.

Does anyone have IT modify CTMS to obtain the variables needed for this metric? We don't require entry for the amount of time recruitment is open, how should we handle that?

If the institution’s IT support staff cannot collect all the variables needed, then hubs may need to use other tools. For Example: Excel worksheet on the CLIC website and a REDCap survey contact common_metrics@clic-ctsa.org

For the variables that include number of days, should we use the actual number of calendar days, or is an average of 30 days/month sufficient?

Please calculate the actual number of days elapsed.

How can we find out if a trial is “on hold”?

The trial PI and protocol team would know if a trial has been placed “on hold”. In terms of calculating the accrual metric, trials that are “on hold” maybe included in the sample. However, a trial that has been deemed as “closed to accrual” should not be included in the sample.

How do I interpret the results of the median accrual metric?
  • When the ratio is close to or =1, the study accrual is occurring/has occurred on time. The accrual is aligned with the projected time to complete enrollment.
  • When the ratio is >1, accrual is ahead of schedule (faster than anticipated).
  • When the ratio is <1, accrual is slower than planned.
How do we deal with trials that don’t have an enrollment cap and/or no planned end date for the trial?

These trials may be excluded from the sample.

How do we report on the metric for multi-site trials?

In the case of multi-site trials, the accrual target is the portion of the total participants targeted for accrual at your own site.

How does the sampling frame apply to the Accrual metric?

During the first 2-3 years of the accrual metric adoption, hubs will be able to decide the domain of trials for which data for this metric will be collected. Ideally, hubs should include all eligible clinical trials, if that is not immediately feasible, either randomly sample from all eligible clinical trials or, secondarily, select a nonrandom targeted sample of clinical trials that can be a focus for accrual performance improvement. Please see the Operational Guidelines Section 4.

If patients consent and undergo a run-in period before randomization, are they considered accrued if they are later excluded?

A participant is considered accrued when they have signed informed consent and passed all screening requirements for the trial.

Please explain the definition for number of days elapsed since open to recruitment.

The number of calendar days elapsed since the initiation of recruitment until December 31 or the end of the actual trial recruitment, whichever comes first.

Please explain the exclusion of trials that do not require informed consent.

Exclude any clinical trials for which informed consent is not required.  Examples: chart review research studies, studies using research registry data that do not require consent beyond registry participation.

We are confused by the inclusion of studies that are on hold for enrollment. Can you provide the rationale?

The Median Accrual Ratio Metric assumes a “linear” or “constant” increase in accrual over the course of the recruitment period. In practice, clinical trials may have planned holds on recruitment or unplanned periods of recruitment is on hold. Nevertheless, over the life of a trial and, especially when estimating across multiple trials, these normal fluctuations in accrual are expected to average out. An example of an unplanned hold on recruitment might include: budgetary constraints, not adhering to the approved informed consent process, challenges with clinical site practices and oversight, etc.

What does “on hold” mean for trials?

Some institutions use the term administrative hold to indicate that a clinical trial has been placed on a temporary pause while additional information is being obtained. An “administrative hold” maybe issued by the institution’s Institutional Review Board (IRB) and is different from a clinical hold. The Food and Drug Administration (FDA) may request a clinical hold if there are participant safety concerns that need to be addressed before the trial may be allowed to continue being conducted.

What if you have multiple recruitment sites (example: from a diabetes clinic and the ER) at your institution and begin recruitment at each of those sites on a different date? How do your account for this?

If it is the same study, then the hub should use the earliest date that recruitment began regardless of site. Please see the Operational Guidelines Section 4 Technical Description.

 

What is a sampling frame?

A sampling frame is the source material from which a sample is drawn.

What is competitive enrollment?

This practice includes multiple sites competing to enroll a specific number of trial participants.

What is the definition of accrued?

The Operational Guidelines says that accrued means "…participants who both signed the consent form and passed all screening requirements for the trial. Participants who subsequently withdraw from the trial, or who are lost to follow-up, are considered accrued." By this definition, someone who screen fails would not be considered to be accrued into a trial.

To further distinguish between enrollment vs accrual, participants are enrolled when they consent to participate in the trial. A participant is then accrued when all screening activities are passed. Screen failures are participants who were enrolled but not accrued. This metric considers accruals, not enrollments.

Screen Failures: Subjects who consented to participate in research, but who were disqualified during screening procedures.

Accrual: The number of subjects that have completed or are actively in the process of completing the clinical trial. This does not include screen failures. It does include withdrawals. (Example: you enroll 100 to accrue 25.)

Enrollment: Occurs when an eligible, informed, potential participant undergoes the initial informed consent process and voluntarily agrees to participate in a research project. Example: You enroll 100 to accrue 25.

What is the definition of targeted?

The number of participants required to satisfy the sample size for power calculation in the IRB approved protocol. In the case when there are multiple estimates, the accrual target is the lowest number of participants required.

What is the operational definition of the “start date”? Often there are soft dates for start of recruitment. Should we consider the date of IRB approval as the “start date”?

The open to recruitment date should be the first date on which a trial could have accrued a participant (whether they did or not).

If your institution does not record this data, then this data may need to be gathered from each research team to ensure accuracy, as system dates such as IRB approval may not accurately reflect the accrual start date.

The metric’s operational guidelines offer flexible options for sampling frame as it is anticipated that there will be an adjustment period among some institutions as they include these key dates part of their standard requirements for conducting clinical trials.

The metric’s operational guidelines offer flexible options for sampling frame as it is anticipated that there will be an adjustment period among some institutions as they include these key dates part of their standard requirements for conducting clinical trials.

If at the end, there are trials for which the Number of Participants Targeted or the Number of days the trial will be open to recruitment are unknown and cannot be determined using any of the suggested approaches mentioned above, then the trial should be excluded.

What’s an appropriate number of studies to use for a sampling?

Each Hub would have the flexibility to determine the number of trials that reflects the sampling frame that is most feasible for their institution, when the metric is formally reported by Hubs to CLIC.