Background

Recent breakthroughs in AI research, exploiting deep neural networks (DNN) have resulted in a paradigm shift within digital signal and image processing. Artificial convolutional neural networks combining many layers of convolution and pooling operations have proven to outperform handcrafted feature recognition methods that previously dominated the field. However, to untap utmost performance from the latest DNN breakthroughs, large amounts of data, computational power and efficient optimization techniques, as well as smart neural network configurations, are required in addition to having a solid understanding of the clinical problem and context at hand. When used appropriately, learning methods have been shown to enable finding complex patterns in high-dimensional data sets where DNN seems particularly powerful when large amounts of annotated data are available.

Approach

Within MARCIUS, we propose to utilize deep learning techniques to automatically analyse and classify dynamic echocardiographic images. In order to empower the DNN, a large database of annotated clinical data is already available and will be further curated during the present project (cf. WP3) to complement a huge, unique database of (validated) virtual patient data (cf. WP1). Of note is that the latter database is intrinsically annotated. In this way, the DNN’s will be trained on “big data” in order to boost their performance. In addition, virtual clinical trials are currently encouraged by regulatory offices (e.g. the Food and Drug Administration) and have been shown – in certain settings – to lead to similar conclusions as those based on true clinical trials. Within this WP, DNN’s will be developed and validated for the following tasks: i) view recognition (ESR4), ii) delineation of anatomical structures (ESR4), iii) cardiac motion tracking and strain estimation (ESR4), iv) patient classification based on strain data (ESR5) and v) patient classification based on image (dynamic) texture analysis (ESR3). In this way, a fully automatic analysis workflow will be created that can automatically extract the abnormalities of regional myocardial (mechanical) properties underlying the cardiac disease for each specific patient and suggest a diagnosis. For each of these tasks, the virtual data and their relevant annotations will be exploited as detailed in the table below. Importantly however, the virtual data will always be used in conjunction with the clinical data (cf. WP3) to avoid bias towards the simulation tools.

Of note is that synergies between the activities of different ESRs exist. However, given the availability of prior-art (synthetic and clinical) data, ESRs can make progress independently and integration of the different developments will gradually occur during the progression of MARCIUS. The organisation of regular “online MARCIUS lab meetings”, with all ESRs and supervisors participating, will ensure timely knowledge transfer between ESRs.  Initially, the DNN’s will be trained for 2D temporal sequences (i.e. 2D+t data) while they will be extended to 3D+t afterwards. In 3D, the view recognition problem still holds (i.e. parasternal versus apical echocardiographic window; respective position of the right and left ventricle in an apical view) but has less degrees of freedom and it should thus be easier to train the respective DNN. Interestingly, this approach will allow objectively evaluating the added diagnostic value of 3D over 2D strain measurements; a topic that has led to controversies in literature given that both approaches have theoretical advantages.

Tissue microstructure can have a direct impact in cardiac deformation, which will be exploited by ESR3. Fibrotic tissue (e.g. caused by tissue scarring or aortic stenosis) makes the myocardial tissue stiffer, while prolonged increased wall stress can lead to muscle thickening which again can change the local stress parameters. Other examples of abnormal tissue composition are disarray of myocardial cell structures and fatty replacement of cardiac cells. All these factors can impact both visual appearance of affected regions and the local and global deformation patterns. We will characterize tissue alternations by linking microstructural properties to ultrasound image appearance using a deep neural network based trained on in-silico and clinical data.

Quantification of myocardial motion is currently a complex semi-automated process. Our end goal is to provide full wing-to-wing automation from image selection, myocardial segmentation, motion tracking to clinical interpretation. Hereto, ESR4 will focus on developing deep learning-based models of this pipeline, ending with myocardial motion tracking. The region-of-interest for speckle tracking will be automatically defined using a combination of region proposal networks and U-net (a combination of convolutions and de-convolutions to perform image segmentation). Tracking will be done using a flow-net (combination of optical flow and a fully convolutional neural network) – from which cardiac local strain curves will be estimated.

Since understanding the interactions between these parameters and how they relate to patient prognosis is extremely challenging for the human reader, we plan to leverage DNN to fully automate echocardiographic strain measurements and a combination of recurrent neural networks (RNN) and convolution neural networks to classify disease (ESR5). The strain traces from the 18 segments of the left ventricle describing the contraction and relaxation of the patient’s heartbeat constitute a unique “finger print” of the patient’s heart. We will design an RNN trained using both clinical and simulated data sets to build a robust diagnostic support system. The system will be able to recognize patterns associated with different cardiac diseases such as asynchronous contraction, impaired relaxation, atrial fibrillation, hypertrophic heart disease, valve disease and link these to clinical diagnosis.

Outcomes & Synergies

Fully automatic image analysis, empowered by a “big data” approach taking advantage of both simulated (cf. WP1) and clinical (cf. WP3) data, will have a marked impact on the clinical workflow and therefore on the cost-effectiveness and performance of the clinical echo labs. Moreover, it will avoid operator variability in the analysis process, which combined with a computer-suggested diagnosis will help the operator to improve and objectify clinical decision-making. Both combined will make speckle tracking echocardiography an imaging modality for routine practice where it currently remains a tool used in expert centres and research only.