Objectives: 

1)  To survey the utilisation of AI-based solutions in radiation oncology settings in India and study the change in utilisation trends over three years.

2) To qualitatively and quantitatively know what factors enables or deters the utilisation of AI-based health technology in radiation oncology settings using a mixed method study.

3) To comprehensively understand the implementation process and its potential impact in radiation oncology setting of a few cancer centres using a  case study design.

4) To perform an economic evaluation of implementing AI-based health technology in the radiation oncology setting.

7. Methodology:

Include objective-wise work plan under the following sub-headings:

Objective 1: To survey the utilisation of AI-based solutions in radiation oncology settings in India and study the change in utilisation trends over three years.

a. Study design: Cross-sectional observational study at two time points: Year 1 and Year 4

b. Study area (multiple choice): Community/Hospital/Laboratory:

This cross-sectional observational study will involve radiation oncology providers working at a standalone cancer centre or a hospital providing cancer services. These providers include consultant-level radiation oncologists, doctors pursuing specialist training in radiation oncology, physicists and radiation-therapy technologists involved in the radiation therapy.

c. Sample size estimation and sampling strategy

Sample size: The total number of consultants and trainee ROs registered in the AROI member list is approximately 4000 (active members- 2500), radiation physicists (RP) in the AMPI register around 2500 (active members 1500), and radiation technologists (RTT) about 3000 (active members 1500) in the country. Considering a response rate of 20% among radiation oncologists and 10% among RP and RTT,  a total of 800 radiation oncology professionals (250 RO and trainees each, 150 RP and 150 RTT) are expected to participate in the survey.

Assuming a 30% implementation, with 95% confidence intervals with a allowable margin of error of 5%, a none response rate of 10%, the sample size was estimated to be 370. The survey will be sent to the eligible participants and we will continue to recruit till desirable sample size is met.

d. Project implementation plan

Inclusion criteria:

1) Radiation Oncology (RO) professionals- Oncologists, resident radiation oncology trainees, Medical physicists and technologists.

2) The RO professionals working in both government and private settings.

3) Willing to participate in the survey.

Exclusion criteria:

1) Radiation oncology professionals who are not practising or retired.

A questionnaire will be adopted from validated surveys performed in the UK, Canada, New Zealand and Australia with some modifications. The survey tool will have questions regarding demographics like age, gender, practice setting, years of experience, current job profile, expertise and training in AI, AI health technology implemented in RO setting, number of solutions implemented, the domains involved, tumour sites implemented, impact on the type of work, impact on workforce and future jobs and future plans of implementing AI-based solutions. We will also assess the willingness among professionals who have not yet implemented AI technology and their thoughts on the future potential of AI in radiation oncology. The modified survey will be assessed for face validity by ten other experts not part of the study.  We will ensure maximum participation in the study nationwide by engaging state and national Radiation Oncology, physicists, and RTT associations. The members’ emails will be taken from the association registers. The online survey questionnaire will be emailed to practising radiation oncologists, MP, and RTTs, both academic and private, using a web based Survey link. The email will contain the survey’s aim, voluntary nature, consent and the confidentiality clause regarding the study results. We will also email the head of radiation oncology centres for dissemination within the department and circulate via WhatsApp groups to ensure the sample size is reached. The survey will be kept open for three months. Two reminder emails will be sent at the end of 4 and 8 weeks.

The survey will be administered twice, first in year one and then at the end of year four. Second survey will look at changes from baseline and additional questions. Since AI-based technologies are rapidly being launched in the market, the second survey will enable us to see the change in the clinical implementation of AI-based health technology and its sustenance.

Outcome variable:

1) Proportion of oncologists implementing AI based solution in radiation oncology workflow.

2) The percentage change in the implementation of AI based solution in radiation oncology workflow in a 3 year period.

e. Design of statistical analysis

The data collected will be analysed using SPSS version 29. Descriptive analyses will be conducted, and the responses will be presented as proportions or point estimates with mean and standard deviation. The Pearson chi-square test will compare the responses with demographics like age, gender, and experience. One-way analysis of variance will be carried out to study the relationship between the three RO professionals and age, gender, and other demographics. Codes will be generated from responses to open-ended questions, and themes will be developed. For questions with multiple answers, the first preference will be given a higher score, and the responses across the RO professionals will be presented using radar plots.

Objective 2: To qualitatively and quantitatively know what factors enables or deters the utilisation of AI-based health technology in radiation oncology settings using a mixed method study.

a. Study design: Prospective Observational mixed method study (Quantitative and Qualitative).

b. Study area (multiple choice): Community/Hospital/Laboratory:  Hospital.

Based on the results of objective 1, the radiation oncology centres implementing at least one AI-based health technology in the clinic will be considered for inclusion in objective 2.

c. Sample size estimation and sampling strategy:

Sample size: Thirty radiation oncology centres, academic and private, from metro and tier 2 cities, representative of all regions of India, implementing AI health technology will be recruited for the mixed-method (qualitative and quantitative) study.

For the qualitative study, 25-30 radiation oncologists will be considered. 

Sampling strategy: Convenience purposive sampling.

d. Project implementation plan

Based on the survey results, radiation oncology centres deploying AI -based health technology in radiation oncology settings will be recruited. The implementation science framework will be used.

Quantitative:

Inclusion criteria:

• Radiation oncology oncologists from the radiation oncology centre with AI solutions deployed in at least one of the workflow domains.

• Radiation oncology professionals from academic or private cancer centres.

• Willing to participate in a facility visit and professionals willing to share the experience.

Exclusion criteria:

• Radiation oncology professionals who are willing to implement AI solutions in future without current implementation.

The researcher will visit the institute and record the data on the AI-based technology using a facility questionnaire after obtaining consent from the lead oncologist. The required permissions to visit the cancer centre will be taken before the visit. The questionnaire will have six parts: Details of an AI solution, Deployment requirements, Implementation workflow, Risks, safety and ethics, Benefits of the AI solutions, and Challenges faced during clinical implementation. The data captured will include AI solutions metrics, technology details, cost of the device, version of the device, pay options, cloud-based or software, clinical deployment and sustainability strategy employed by the institute/clinician, use of audits and monitoring systems for safety and quality, quality assurance, the ethical considerations, presence of AI inhouse engineers, IT support, IT-related upgradation, steps taken to mitigate the risks of AI device, the workload and patient throughput from simulation to treatment, time spent by radiation oncologist daily on the editing/verification of tasks by AI device, adverse events or errors incurred with AI device use, radiation oncology workforce recruitment strategy etc. The radiation oncologist’s satisfaction will be assessed on a Likert scale.

Qualitative:

Inclusion criteria:

• Radiation oncology professionals from the radiation oncology centre with AI solutions deployed in at least one of the workflow domains.

• Radiation oncology professionals from academic or private cancer centres.

• The key radiation personnel taking the lead in AI implementation in the centre.

Exclusion criteria:

• Radiation oncology professionals who are willing to implement AI solutions in future.

• Radiation oncology professionals who are unwilling to participate.

After obtaining consent, an in-depth interview will be conducted to study radiation oncologists’ perspectives on implementing AI-based technology. An in-depth interview guide will be developed, and it will include questions on the following domains:

The need for AI solutions in RO.

Experience and challenges in using the devices.

Barriers and facilitators of using AI-based technology.

Impact on clinical practice, research, and education.

The researcher will explain the study using the participant information sheet, and consent will be obtained with permission for the audio recording. The interviews will take 20 to 30 minutes. After collecting the demographic information, the interview will be conducted in English and recorded based on the domains of the interview guide. The researcher will return the audio recordings and save them on the designated laptop accessible by the research team, which is password protected. The interviews will be coded (filtering the identifiers), translated, and then transcribed by the researcher. The data cleaning of the interviews will be done. The confidentiality of the participants will be maintained throughout the process.

e. Design of statistical analysis:

Quantitative: The data collected will be analysed using SPSS version 29. Descriptive analyses will be conducted, and the responses will be presented as proportions or point estimates with mean and standard deviation.

Qualitative: Data collected will be in audio files, translated and transcribed by the researcher. Further interviews will be done if necessary to validate the data. The audio recordings will be destroyed after the data is validated. The transcribed data will be looked for meanings and patterns, which will be identified and coded after thorough familiarisation with the data. Further codes are grouped into themes, evaluated and revised based on the research question, analysed, and interpreted using the qualitative data analysis software. The thematic analysis will be done using N vivo software.

Objective 3:

a. Study design: Case study design

b. Study area (multiple choice): Community/Hospital/Laboratory: Hospital.

Based on the results of objective 1, the radiation oncology centres implementing at least one AI-based health technology in the clinic will be considered for inclusion in objective 3.

c. Sample size estimation and sampling strategy:

The case study will include at least five oncology centres, three of which have successfully implemented AI-based technology in one domain, contouring or planning, and two centres that have had difficulties or could not implement the AI technology or rolled it back after initial or pilot implementation.

Sampling strategy: Purposive sampling.

d. Project implementation plan

The case study design will involve an in-depth investigation of the implementation of AI-based technology in the routine workflow of a radiation oncology centre. The researcher will visit the RO centre, and centres with experience implementing the technology for over six months will be considered for the study.

Inclusion criteria:

• Radiation oncology professionals from the radiation oncology centre with AI solutions deployed in the most common domains, such as auto-contouring or AI-enabled planning either in isolation or as a part of an end-to-end solution.

• Radiation oncology professionals from the radiation oncology centre with AI solutions deployed for over six months or with the majority of the RO professionals using the solutions on a regular basis.

• Radiation oncology professionals from academic or private cancer centres.

Exclusion criteria:

• Radiation oncology professionals who are willing to implement AI solutions in future.

• Radiation oncology professionals who are unwilling to participate.

After obtaining consent, during the site visit, the researcher will examine in depth the steps taken at the institutional level: workflow organisation, day-to-day monitoring of AI output, human-in-the-loop checks, documentation involved—SOPS, audit, logs or incidents/errors, and staff involved.

Data collection will involve documentation, records, images, and videos of workflow documents, as well as researcher audio logs on direct process and researcher observation. The researcher will record an in-depth log of the observations made of the workflow. A focussed group discussion (FGD) of 4-5 oncologists, 1-2 RP, and 1-2 RTT, nurse at each of the five centres will be conducted to examine the perspectives of RO professionals on AI implementation.

The domains explored are:

Steps of implementation of AI technology or redundant steps/failures

Management of risks/errors of AI

Ethics of AI and patient consent

Impact of AI on the quality, efficiency, skills of oncologists, training needs of clinicians and residents in future

Quality assurance of AI technology

Costs involved in implementation

Patients benefits

Strategies, preparedness and critical framework for implementation of AI technology.

The required permissions to visit the cancer centre will be taken before the visit. The researcher will meet the healthcare professionals in person, explain the study using the participant information sheet, and conduct the FGD in the cancer care centre’s conference room, preferably soundproof. Consent will also be obtained for the audio recording of the discussions at the beginning of the FGD. Each discussion will last for about 30-40 minutes. The audio recordings will then be returned to the department and stored in a password-protected folder of the designated laptop, which the research team can access. Confidentiality will be maintained throughout the process. 

e. Design of statistical analysis

FGD: The data collected will be in audio files, translated and transcribed by the researcher. The transcribed data will be looked for meanings and patterns, which will be identified and coded after thorough familiarisation with the data. Further codes are grouped into themes, evaluated and revised based on the research question, analysed, and interpreted using qualitative data analysis software. The thematic analysis will be done using N vivo software.

Objective 4: To perform an economic evaluation of implementing AI-based health technology in the radiation oncology setting.

a. Study design: Prospective Observational study.  

b. Study area (multiple choice): Community/Hospital/Laboratory: Hospital.

Based on the results of objective 1, the radiation oncology centres implementing at least one AI-based health technology in the clinic will be considered for inclusion in objective 4.

c. Sample size estimation and sampling strategy: Same as objective 2.

d. Project implementation plan

The inclusion and exclusion criteria are the same as Objective 2.

Health technology assessment:

We will perform a systematic review and meta-analysis to appraise the published literature on the clinical effectiveness and economic evaluation of AI-based technologies in radiation oncology.

Population: Patients with cancer treated with radiotherapy.

Intervention: AI-based technology (autocontouring and autoplanning) implemented in radiotherapy treatment workflow.

Comparator: Standard therapy- manual interventions or use of atlas or model-based methods.

Outcomes: Accuracy and Efficiency

Cost and Risk or errors

The standard methodology of SR will be followed as per Cochrane. The search strategy will be formulated and four databases (pubmed, Embase, Cochrane, scopus) will be used. The articles will be screened against the inclusion criteria and quality will be assessed using ROB 2 and ROBINS based on type of study. The AI quality will be assessed using MI-CLAIM checklist, CHEERS AI, HTA Core Model published by the European network for Health Technology Assessment (EUnetHTA) and HTA Quality Assessment Checklist (HTA-QAC) for India.

The findings from the systematic Review will be integrated with mixed methods data to perform a comprehensive economic evaluation.  The outcomes used will be the efficiency of planning (time for contouring and time for planning), manpower hours used, and secondary analysis for accuracy. 

Costing methods:

We will collect the direct costs of implementing AI-based health technology from a provider perspective. The oncologists will be asked to provide the direct costs for the following:

1) Cost of the AI-based technology- package or monthly or per patient model.

2) Cost of the yearly upgrade, subscriptions and maintenance.

3) Cost of Internet or cloud or storage needs.

4) Cost of manpower recruitment- monthly salary.

5) Cost of hardware.

6) Cost of radiation treatment per patient- pre and post-AI implementation.

7) Cost of AI on patient.

The input costs will be based on a time-based activity capture methodology. At each participating centre, we will note the activities performed for the treatment planning and delivery for a patient. A time activity sheet will be used to capture the total person-hours of activity involved in the domain for each patient with and without the involvement of AI-based technology. The personnel involved, type (RO, MP, RTT), the number of personnel involved, the time involved for each domain of radiation planning, and the total number of patients treated monthly on average will be noted per centre. Any changes in the manpower recruitment and its cost saving will be noted. The turnaround time from simulation to treatment and the monthly throughput on the machine will be captured.

e. Design of statistical analysis

 The forest plots for systematic review will be generated using Revman. The quality of evidence will be summarised using ROB2, ROBINS CHEERS AI,  HTA core model adoption Scoring and HTA-QAC reporting and AI score using MI CLAIMS checklist. The deficiencies in assessing the AI technology using existing framework will be noted.

A decision framework will be prepared to compile the clinical and economic evidence in determining  the cost-effectiveness of AI-based health interventions.