Indicator and measurement menu

The indicator and measurement menu was developed as part of CCSRI’s implementation of the results chain hierarchy to monitor and evaluate CCSRI research impacts.  The CCSRI indicator and measurement menu shows a series of performance indicators classified according to the results chain hierarchy.  The indicator and measurement menu can be used retrospectively (for evaluating) and prospectively (for planning and monitoring).  Indicators are selected in sets based on their relevance to the program being assessed or the evaluation questions being posed, and can be reworded and refined to be made program specific, as necessary. Ultimately, this tool facilitates consistent monitoring and evaluation by providing a common framework by which to monitor and assess CCSRI programs and initiatives.

The indicator and measurement menu should be read from the bottom up and from left to right. The first column identifies the levels of results (1 to 7) according to the results chain; the second provides a description of the levels of results; the third provides a set of performance indicators for each of the levels of results; and the fourth lists typical sources of data and methods of evaluation.

Chain of results Evaluation criteria Indicators Typical sources/methods

7. End Results

Measures of impact on overall problem, ultimate goals, side effects, social and economic consequences

  • Rate or incidence of cancer (incidence, mortality, morbidity)
  • Level of quality of life (index TBD)
  • Level of advances in cancer science/research
  • Specialized analyses/ evaluations
  • Statistical agency data
  • Canadian cancer statistics
  • Analytical and policy groups

6. Practice
& Behaviour Change

Measures of adoption of new practices and behaviour over time

  • Level of research used (knowledge transfer, practice adoption) by scientists/policy makers/institutions/ health care practitioners/consumers
  • Level of research used in curricula for new researchers (citation in text books and reading lists)
  • Level of research cited in ongoing health professional education material (incl. clinical guidelines)
  • Level of research cited in public policy documents
  • Level of research cited in advocacy publications
  • Level of research cited in scientific literature
  • Level of research commercialization
  • Physical observation
  • Inspections, reviews
  • Surveys
  • Evaluation studies
  • Content analysis

5. Knowledge, Attitude, Skills Aspirations

Measures of individual and group changes in knowledge, abilities, skills and aspirations

  • Level of understanding of key related science information generated through research by scientists/policy makers/institutions/ health care practitioners/consumers
  • Level of self-expressed commitment to specific areas of science/research or practice/protocol/policy change by scientists/policy makers/institutions/health care practitioners/consumers
  • Level of development of new knowledge in cancer research
  • Level of development of new methods in cancer research
  • Level of published research findings in a timely manner and in peer-reviewed journals with high “impact factors”
  • Independent review of target groups
  • Content analysis
  • Survey, group self-assessment
  • Testing/Certification
  • Bibliometrics

4. Reactions

What participants and other stakeholders say about the program; satisfaction; interest, strengths and weaknesses

  • Level of program recognition and support from key stakeholders/target groups/participants
  • Level (volume, accuracy and ‘tone’) of media coverage of research and program activities
  • Usage/participation tracking
  • Correspondence content analysis
  • Surveys
  • Media content analysis

3. Engagement
& Participation

The characteristics of program participants and other stakeholders; number, nature of involvement, and background

  • Level of engagement with other research centres, networks, academic institutions, government agencies, etc.
  • Level of engagement by stakeholders/target groups/ participants
  • Level of multidisciplinary and/or multisectorial research activities
  • Level of recruitment and retention of stakeholders/target groups/participants (e.g. junior investigators, researchers, review panellists, etc.)
  • Level of established external scientific advisory board(s)
  • Web use tracking
  • Correspondence content analysis
  • Observation of meetings/events
  • Meeting attendance records
  • Stakeholder relationship management/tracking (e.g. contracts and agreements)
  • Surveys

2. Activities
& Outputs

Implementation data on what the program actually offers

  • Level of research as per internal review guidelines
  • Extent to which plans, strategies, frameworks, etc. are delivered as per expectations (expected timelines, resource usage and quality levels)
  • Extent to which governance structure adheres to internal guidelines
  • Extent to which policy and financial decisions are made according to Board/Senior Management/Expert Advisory Committee(s) accepted  guidelines and standards
  • Extent to which internal and external communication strategies adhere to internal standards and protocols/policies (standard operating procedures)
  • Project/Initiative tracking
  • Project reports
  • Content analysis or records
  • Peer review
  • Operating reviews
  • Risk analysis

1. Inputs

Resources expended; number and types of staff involved; time expended

  • Level of human resources (staffing) at all levels
  • Level of financial resources (budgets vs. actuals) at all levels
  • Budget analysis
  • Time, reporting and budget/plan review
  • Activity-based costing

Last modified on: October 30, 2013