Protocol

 

1. Title of the project:  Development and validation of multimodality Artificial Intelligence (AI) tool for prediction of short-term changes in clinical status among patients with pneumonia requiring intensive care

2. Type of Study: Prospective study- longitudinal study

3. Aims & objectives (hypotheses if applicable):

Objectives

1. Build and validate a multimodality Artificial Intelligence (AI) tool to predict clinical status over the next 24-48 hours (classification problem) of patients with pneumonia admitted to Emergency Department (ED), High Dependency Unit (HDU) and/or Intensive Care Unit (ICU) settings.
2. Build and validate a deep learning (DL) based image classification model to diagnose pneumonia on chest X-ray images.

4. Justification for study (whether of national significance with rationale):

Intensive care or high-dependency units are scarce and valuable resources which need to be used optimally and turned over rapidly to benefit a greater number of patients. Pneumonia is one of the most common causes of admission to the ICU/HDU. Currently, the decision to escalate and de-escalate the level of care between ICU/HDU and wards is clinical. It depends on the best judgement of the physician based on his/her assessment of patient data and may be subjective. However, such assessments are usually carried out only once or twice per day in the usual ICU/HDU setting during clinical rounds. This process involves a judgement/prediction on the likelihood of deterioration or improvement of a patient’s health status and is prone to errors and delays. There is scope for improvement in these predictions using data-driven approaches as evidenced by the use of scoring systems like Modified/National Early Warning Scores (MEWS/NEWS). The use of AI methods to address this issue promises to improve the accuracy of such predictions further. Moreover, an AI approach would also be more scalable and deployable in resource-limited settings with minimal involvement of doctors or healthcare providers.

The development of such an accurate AI tool can have a huge impact on the optimal utilization of ICU/HDU resources and improve patient outcomes on a large scale both nationally and internationally.

5. Departments involved: Departments of Critical Care Medicine, Emergency Medicine, Radiodiagnosis, Cardiology and Plastic Surgery.

6. Study period: Two years

7. Sample size: 3000 patients

There are no standard guidelines for calculating the sample size required for training a deep-learning model for image classification tasks or for training a multimodality AI model. Though “larger the better” is the rule in building these models, it may not be suitable for the medical domain where there are significant costs, efforts and ethical concerns involved. To minimize the sample size required to build accurate models, we plan to employ techniques like transfer learning which involves utilization of architectures trained for other related tasks to be utilized for effectively learning from the limited medical datasets. Following these principles and from the review of the literature on successful medical AI models for classification tasks, we estimate that a sample size of 3000 would be a reasonable estimate to build an accurate, clinically useful model.

8. Materials and methods:  

a) Inclusion and exclusion criteria:

Inclusion criteria:

1. All adult patients (> 18 years of age) admitted to ED, HDU and/or ICU during the study period with clinical diagnosis of pneumonia at admission.

Exclusion criteria:

1. Pregnant women.
2. Patients with diagnosed hospital-acquired pneumonia at admission.
3. Patients with advanced directives for non-escalation of care/palliative care.
4. Patients are not willing to give informed consent.

b) Biological materials required (type - blood, tissue etc and quantity): Yes ☐    No   �’   

i) Biological material: Nil

ii) Biosafety Measures: NA

c) Statistical methods:

Mean and standard deviation will be used to summarize continuous normally distributed data, and median (interquartile range) will be used to summarize non-normal data. Frequency and percentages will be used for nominal/categorical data. The steps involved in machine learning are detailed in the methods section. After models are trained using the training data, they will be evaluated on the unseen test data against the ground truths (labels) using the confusion matrix, accuracy, f1-score and ROC analysis. We will use the temporal validation method: training data will be from the first 75-80% of patients recruited in the study and test data will be from the final 20-25% of the patients recruited. Using this method, we ensure that the test data is truly unseen during the training phase. A similar testing process will be carried out for both the deep learning image classification model and the multimodality AI model.

d) Tools used: Modified Early Warning Score (MEWS), SOFA score, APACHE II score, all available for free to use.

9. Detailed description of procedure / processes:

After obtaining the ethical committee clearance, the study will be registered with the CTRI before recruiting patients. For patients admitted to hospital in the ED, HDU or ICU setting, fulfilling the inclusion criteria stated above, patients or their legal representatives will be approached for informed consent. They will be explained about the study procedure and provided with the patient information sheet. A written informed consent will be obtained before data is collected.

Clinical workflow

Baseline demographic data, clinical data including physical examination findings and laboratory data will be recorded. Clinical monitoring data for patients admitted to ED, HDU or ICU will be recorded at hourly or two-hourly intervals (based on availability and feasibility) throughout their stay. Only those parameters which are routinely monitored and those lab test data which are done as part of standard care will be recorded. No additional tests will be done for the purpose of this study. For ease of capturing data, a tool which auto-captures numbers from images of ICU monitors and ventilator screens will be used (details below). No patient photos or their identifiers will be captured. Such tools will be used only by the research team and all such data will be stored securely. Chest x-ray images of the patients will be stored after deidentification. Patient outcomes will be recorded as ‘improved’, ‘deteriorated’ or ‘status-quo’ as adjudicated by the subject expert. Details of the variables/parameters that will be recorded are provided in the proforma/data collection form.

There will be no additional treatment or tests, visits or follow-ups for the purpose of this study. All study patients will receive standard of care as per existing guidelines at the discretion of the treating physician.

Clinical status will be as determined by the treating physician as improved, deteriorated or status quo based on the status 24-48 hours ago. Clinical status will be recorded on a daily basis.

AI/Machine learning workflow

Data management software: A custom, secure data management platform will be built for this study which will have role-based access for each member of the research team. Details of the research assistants recruited for the study will be informed to the IEC as they are onboarded through amendments. All the research data will be collected in a relational database which will be part of this platform. It will have a provision for capturing data from images of the monitor screens to ease the workflow and to ensure accurate recording of serial data with time logging. The rest of the data will be entered manually by research assistants with supervision from PI and Co-Is to ensure high-quality data is captured.

Data preprocessing, model building and evaluation: All data especially the image data will be deidentified and accessed only by the research team and data scientists authorized by the research team for model building. Data will be divided into training and test data using a temporal method (as described in statistical analysis). Training data will be further divided into train and validation subsets for training the models. Appropriate data preprocessing steps will then be applied to prepare the data for ML tasks. Several candidate model architectures will be tried and the one giving the best accuracy along with good cross-validation scores (indicating robustness) will be chosen as the final model. Evaluation of the models for classification tasks will be done on unseen test data using standard methods like confusion matrix, accuracy score, f1-score and ROC analysis.

Data Sharing

Data (both clinical and imaging data) after appropriate deidentification will be shared with the Data Science team from 5C Network (India) Private Limited which is collaborating with KMC Manipal, MAHE, Manipal on this project. The data will be used to build and train deep learning image models and multimodality AI models described above. An NDA will be signed between MAHE Manipal and 5C Network (India) Private Limited for effective collaboration. Data will not be shared with any other team or organization or on public forums. 5C Network (India) Private Limited is located within the Indian national borders. Data will not be shared with any team outside the country.

No funds are being transferred from 5C Network (India) Private Limited to KMC, Manipal or MAHE, Manipal.

10. Outcome measures: Accuracy of prediction by multimodality AI tool for the change in clinical status over the next 24-48 hours. Accuracy of deep learning AI image classification model in diagnosing pneumonia.

11. Potential risks and benefits:

Potential risks: Minimal risk. This is an observational study which does not involve any additional investigations or interventions as a part of the study.

Potential benefits: Successful development of such AI tools will be of great benefit as it can be employed with minimal involvement of healthcare professionals (after further validation studies) especially in resource-limited settings to make decisions regarding escalation or de-escalation of care. Such a tool will be scalable to secondary care settings and also be suitable for tele-ICU solutions.

12. Ethical considerations and methods to address issues:  This is an observational study where only the clinical, lab and imaging data of patients are used after appropriate de-identification steps. The patients continue to receive standard of care. There are no additional tests, expenses or clinical visits for the purpose of this study. Therefore, it is minimal risk.

 

13. Budget (give details) and proposed funding source:

Sl.No	Details	Justification	Funding agency	INR
1.	Research Assistants	Needed for data collection	MAHE, Manipal	Rs. 5,00,000/-