NEWS: BSNet model was used by the two teams that won the "Covid CXR Hackathon". The code and report for the best "Explainability in AI-based COVID Prognosis" is available.
The BrixIA COVID-19 project
Worldwide, the saturation of healthcare facilities, due to the high contagiousness of Sars-Cov-2 virus and the significant rate of respiratory complications, is indeed one among the most critical aspects of the ongoing COVID-19 pandemic.
Under these conditions, it is extremely important to adopt all types of measures to improve the accuracy in monitoring the evolution of the disease and the level of coordination and communication between different clinicians for the streamlining of healthcare procedures, from facility- to single patient-level.
With this in mind we:
- propose a novel end-to-end multi-network architecture specifically designed for pneumonia severity assessment on CXRs.
- release a large fully annotated dataset of 4703 CXRs with highly expressive multi-zone COVID-19 severity score.
- provide a Consensus set annotated by 5 expert radiologist composed of 150 CXR.
Through inter-rater agreement tests and a consensus set comparison, we were able to show that our solution outperforms single human annotators in rating accuracy and consistency, thus supporting the possibility of using this tool in contexts of computer-assisted monitoring.
This project is approved by the Ethical Committee of Brescia (Italy) NP4121 (last update 08/07/2020).
What is the aim?
We designed an end-to-end deep learning architecture for predicting, on Chest X-rays images (CRX), a multi-regional score conveying the degree of lung compromise in COVID-19 patients.
Such scoring system, namely Brixia score, was applied in serial monitoring of such patients, showing significant
prognostic value, in one of the hospitals that experienced one of the highest pandemic peaks in Italy.
Moreover, we provide the full dataset with the related metadata and Brixia score annotations, and the code to reproduce our results.
Read MedIA Article Read the preprint
Brixia-score
The multi-region 6-valued Brixia-score was designed and implemented in routine reporting by the Radiology Unit 2 of ASST Spedali Civili di Brescia, and later validated for risk stratification on a large population.
According to it, lungs in anteroposterior (AP) or posteroanterior (PA) views, are subdivided into six zones, three for each lung, as shown in the above Figure:
- Upper zones (A and D): above the inferior wall of the aortic arch;
- Middle zones (B and E): below the inferior wall of the aortic arch and above the inferior wall of the right inferior pulmonary vein (i.e., the hilar structures);
- Lower zones (C and F): below the inferior wall of the right inferior pulmonary vein (i.e., the lung bases).
Whenever it is difficult to identify some anatomical landmarks, due to technical reasons (for example bedside CXR in critical patients), it is acceptable to divide each lung into three equal zones. For each zone, a score (ranging from 0 to 3) is assigned, based on the detected lung abnormalities:
- 0: no lung abnormalities;
- 1: interstitial infiltrates;
- 2: interstitial (dominant), and alveolar infiltrates;
- 3: interstitial, and alveolar (dominant) infiltrates.
The six scores may be then aggregated to obtain a Global Score in the range [0,18]. During the peak period, the Brixia-score has been systematically used at ASST Spedali Civili of Brescia to report CXR in COVID-19 patients.
BSNet
Detailed scheme of the proposed architecture. In particular, in the top-middle the CXR to be analyzed is fed to the network. The produced outputs are: the segmentation mask of the lungs (top-left); the aligned mask (middle-left); the *Brixia-score* (top-right).
GitHub
GitHub
Dataset
Annotation and CXR from Cohen’s dataset
We exploit the public repository by Cohen et al. which contains CXR images (We downloaded a copy on May 11th, 2020).
In order to contribute to such public dataset, two expert radiologists, a board-certified staff member and a trainee with 22 and 2 years of experience respectively, produced the related Brixia-score annotations for CXR in this collection, exploiting labelbox, an online solution for labelling. After discarding problematic cases (e.g., images with a significant portion missing, too small resolution, the impossibility of scoring for external reasons, etc.), the final dataset is composed of 192 CXR, completely annotated according to the Brixia-score system.
Brixia score COVID-19 dataset
We collected a large dataset of CXR images corresponding to the entire amount of images taken for both triage and patient monitoring in sub-intensive and intensive care units during one month (between March 4th and April 4th 2020) of pandemic peak at the ASST Spedali Civili di Brescia, and contains all the variability originating from a real clinical scenario. It includes 4,707 CXR images of COVID-19 subjects, acquired with both CR and DX modalities, in AP or PA projection, and retrieved from the facility RIS-PACS system.
All data we provide are anonymized DICOM files and annotation files in CSV with Brixia score and relevant metadata.
Below a list of each field in the metadata_global_v1.csv, with explanations where relevant
Scheme (click to open)
| Attribute | Description |
|---|---|
| filename | filename to link to the DICOM file |
| StudyDate | Date the CXR was taken. Format: YYYYMMdd |
| Modality | CXR modality, either DX (Digital Radiography) or CR (Computed Radiography) |
| StudyId | An anonymized RIS generated number that identifies the order for the Study |
| BrixiaScore | The 6 regions annotatated by the radiologist, (ABCDEF from 0 to 3) |
| BrixiaScoreGlobal | The sum of the 6 scores |
| ConsensusTestset | Whether it is re-annotated in the Consensus set (by 4 additional expert radiologist) |
| Subject | An anonymized id the identifies the subject |
| Sex | Sex of the patient |
| AgeAtStudyDateFiveYear | Age of the patient in groups of five (i.e. 0-4yo is 0, 5-9yo is 1, ...) |
Below a list of each field in the metadata_consensus_v1.csv annotation file
Scheme (click to open)
| Attribute | Description |
|---|---|
| filename | filename to link to the DICOM file |
| from MeanA to MeanF | The mean of 6 regions annotatated by a 5 expert radiologist, from 0 to 3 |
| S-Global | Global score from the means (sum of MeanA : MeanF), from 0 to 18 |
| from ModeA to ModeF | The mode (the score chosen most often) of 6 regions annotatated by a 5 expert radiologist, from 0 to 3 |
| J-Global | Global score from the modes (sum of MeanA : MeanF), from 0 to 18 |
Get the data
Please read the Research Use Agreement provided below. Once you register to download the BrixIA COVID-19 Dataset, you will receive a link to the download over email. It is forbidden to share the link to download the dataset with others."BrixIA COVID-19 Dataset" Research Use AgreementPremise: the BrixIA COVID-19 project involves a set of activities aiming at AI-driven interpretation of radiologic images and clinical data acquired from COVID-19 subjects at the ASST Spedali Civili (Civil Hospital) of Brescia, Italy (ASST-BS) during the pandemic period. The research activities are conducted by joint project team (JPT) belonging to the Department of Information Engineering (DII) and to the Department of Medical and Surgical Specialties, Radiological Sciences, and Public Health (DSMC) of the University of Brescia, Brescia, Italy (UniBS). The research activities and the data distribution, under the following research-only usage terms, has been approved by the Ethics Committee of Brescia (ECBS), and all the images and the associated reports and data has been de-identified by the BrixIA-Team at the 2nd Radiology Unit of ASST-BS. The "BrixIA COVID-19 Dataset" is a collection of images and related data and metadata that is made accessible for Research use only, starting from this website and after acceptance of the following terms of use. By registering for downloads, you are agreeing to this: 1. Permission is granted to view and use the "BrixIA COVID-19 Dataset" without charge for research purposes only. Its sale is prohibited. Any non-academic research use need to be evaluated case by case by the JPT in coordination with ASST-BS. If you intend to use this dataset for any non-academic research use, you need to communicate it describing the intended use and receive approval by the JPT. 2. In agreement with the mission of UniBS to promote the publication of scientific knowledge as open data, any computational model or algorithm that have used the "BrixIA COVID-19 Dataset" and is publicly referenced (e.g. in a publication etc..) is suggested to be shared including the code and model weights and any case will give appropriate credit by correctly citing the BrixIA COVID-19 project scientific papers, but not in any way that suggests that the ASST-BS and/or UniBS endorses you or your use. 3. Other than the rights granted herein, the ASST-BS retains all rights, title, and interest in the "BrixIA COVID-19 Dataset". 4. You may make a verbatim copy of the "BrixIA COVID-19 Dataset" for uses as permitted in this Research Use Agreement. If another user within your organization wishes to use the "BrixIA COVID-19 Dataset", they must comply with all the terms of this Research Use Agreement. 5. YOU MAY NOT DISTRIBUTE, PUBLISH, OR REPRODUCE A COPY of any portion or all of the "BrixIA COVID-19 Dataset" to others without specific prior written permission from the JPT that acts in coordination with ASST-BS. 6. You must not modify, reverse engineer, decompile, or create derivative works from the "BrixIA COVID-19 Dataset". You must not remove or alter any copyright or other proprietary notices in the "BrixIA COVID-19 Dataset". 7. The "BrixIA COVID-19 Dataset" has not been reviewed or approved by the Food and Drug Administration or the European Medicines Agency, and is for Research Use Only. In no event shall data or images generated through the use of the "BrixIA COVID-19 Dataset" be used or relied upon in the diagnosis or provision of patient care. 8. THE "BrixIA COVID-19 Dataset" IS PROVIDED «AS IS,» AND ASST-BS, UNIBS AND THE JPT DO NOT MAKE ANY WARRANTY, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE, NOR DO THEY ASSUME ANY LIABILITY OR RESPONSIBILITY FOR THE USE OF THIS "BrixIA COVID-19 Dataset". 9. You will not make any attempt to re-identify any of the individual data subjects. Re-identification of individuals is strictly prohibited. Any re-identification of any individual data subject shall be immediately reported to the JPT. Any violation of this Research Use Agreement or other impermissible use shall be grounds for immediate termination of use of this "BrixIA COVID-19 Dataset". In the event that the ASST-BS or UniBS determines that the recipient has violated this Research Use Agreement or other impermissible use has been made, they may direct that the undersigned data recipient immediately return all copies of the "BrixIA COVID-19 Dataset" and retain no copies thereof even if you did not cause the violation or impermissible use. 10. You agree to indemnify and hold ASST-BS and UniBS harmless from any claims, losses or damages, including legal fees, arising out of or resulting from your use of the "BrixIA COVID-19 Dataset" or your violation or role in violation of these Terms. You agree to fully cooperate in ASST-BS and UniBS defense against any such claims.
BS-Net Results
A synthesis of Results obtained with BS-Net on the BrixIA Covid-19 Dataset are repored below. More results and info can be found on the project paper.
Consistency/confusion matrices based on lung regions score values (top, 0-3), and on Global Score values (bottom, 0-18).
Results of the proposed BS-Net system on five examples from the test-set. (top) Three good predictions. (bottom) Two cases chosen between the worst prediction regarding the original clinical annotation H. For each block, the most left image is the input CXR that the network analyses, followed by the aligned and masked lungs, to analyse the quality of the segmentation and alignment block. While in the second row we show the Predicted Brixia-score with the clinical ground truth H, and the explainability maps. In those maps the relevance goes from white colour (i.e., no contribution to that prediction) to the class colour (i.e., the region had an important weight for the class decision).
Citations
For any use or reference to this project please cite the following paper.
@article{BS-Net2021,
title = {BS-Net: learning COVID-19 pneumonia severity on a large Chest X-Ray dataset},
journal = {Medical Image Analysis},
pages = {102046},
year = {2021},
issn = {1361-8415},
doi = {https://doi.org/10.1016/j.media.2021.102046},
url = {https://www.sciencedirect.com/science/article/pii/S136184152100092X},
author = {Alberto Signoroni and Mattia Savardi and Sergio Benini and Nicola Adami and Riccardo Leonardi and Paolo Gibellini and Filippo Vaccher and Marco Ravanelli and Andrea Borghesi and Roberto Maroldi and Davide Farina},
}
@article{borghesi2020covid,
title={COVID-19 outbreak in Italy: experimental chest X-ray scoring system for quantifying and monitoring disease progression},
author={Borghesi, Andrea and Maroldi, Roberto},
journal={La radiologia medica},
volume={125},
pages={509-513},
year={2020},
publisher={Springer}
}
## Acknowledgement
Part of this project has been funded with support from the Italian Ministry of University and Research (grant n. FISR2020IP_02278, “ResponsiX: Responsible and deployable AI-driven evaluation of COVID-19 disease severity on Chest X-rays”)
