Unsupervised Multimodal Representation Learning across Medical Images and Reports Tzu-Ming Harry Hsu, Wei-Hung Weng, Willie Boag, Matthew McDermott, and Peter Szolovits Motivation & Contributions MIMIC Chest X-ray Dataset Results • Medical reports and images have been explored in the form of report generation, image annotation, image generation, and joint representation learning • Parallel image/report pairs are not always feasible • We investigate the effect of using semi-supervised algorithms in learning joint embedding spaces on the MIMIC-Chest X-ray * dataset • We show that, on large scale, unsupervised methods achieve comparable results on the metrics for retrieval Methods • The MIMIC Chest X-ray (MIMIC-CXR) consists of 473,057 chest X-ray images and 206,563 corresponding radiology reports from 63,478 patients admitted to critical care units at Beth Israel Deaconess Medical Center. Paired Image and Report Dataset Timeline worsening opacities in the lung bases NAME reflect worsening atelectasis … avgpool Image Text DenseNet121 W 1 (1) Supervised: EA (2) Unsupervised: Adv (3) Semi-supervised: EA + Adv Joint Embedding Space Text Feature PCA … TF-IDF Distributed Embeddings Supervision and Text Feature impression findings Report Section 0.35 0.40 0.45 0.50 Similarity tfidf word sentence paragraph impression findings Report Section 0 2 4 MRR (T ! I, ⇥10 -3 ) impression findings Report Section 0.150 0.175 0.200 0.225 nDCG@100 (T!I) 0 10 -3 10 -2 10 -1 10 0 Supervision Fraction 0.2 0.4 0.6 Similarity 0 10 -3 10 -2 10 -1 10 0 Supervision Fraction 0 5 10 MRR (⇥10 -3 ) T!I I!T 0 10 -3 10 -2 10 -1 10 0 Supervision Fraction 0.10 0.15 0.20 0.25 nDCG@k (T!I) k=1 k=100 0 10 -3 10 -2 10 -1 10 0 Supervision Fraction 0.10 0.15 0.20 0.25 nDCG@k (I!T) k=1 k=100 Highly Related Partially Related Count 0 400 800 1200 1600 CHF NOS Acute respiratry failure Pneumonia, organism NOS History of tobacco use Chr pulmon heart dis NEC Pleural effusion NOS Status-post ptca Food/vomit pneumonitis Tobacco use disorder Aortocoronary bypass Chr airway obstruct NEC Ac on chr diast hrt fail Fluid overload NEC Ac resp flr fol trma/srg Ventltr assoc pneumonia Osteoporosis NOS Pulmonary collapse Surg compl-heart Ac on chr syst hrt fail Ac/chr syst/dia hrt fail Chr diastolic hrt fail Cor ath unsp vsl ntv/gft Asthma NOS Hx-ven thrombosis/embols Acute lung edema NOS Chr systolic hrt failure Prim cardiomyopathy NEC Effusion NEC exc tb Count 0 75000 150000 PA AP Lateral Patients Reports Original Images Adjusted Images MIMIC-CXR X-ray Breakdown ICD-9 Codes Supervision Fraction Using Different Report Sections nDCG@100 0.10 0.15 0.20 0.25 bi-gram word sentence paragraph Similarity 0.00 0.25 0.50 0.75 bi-gram word sentence paragraph EA Adv Adv + Proc nDCG@k = 1 |Q| P q 2Q 1 IDCG q P k p=1 2 rel pq -1 log 2 (p+1) ing a perfect retrieval algorithm. All experime relation coefficient in [0, 1] MRR = 1 |Q| P q 2Q 1 rank q ge or text corresponding rank of the first related item • Embedding Alignment (EA) • Adversarial Domain Adaption (Adv) • Procrustes Refinement (Adv + Proc) [Grave] L EA (X, Y)= W > X - Y 2 F . L D Adv (X, Y)= E (x,y)⇠p(X,Y) ⇥ - log D ( W > x ) - log (1 - D (y)) ⇤ X, Y)= E ⇥ - log ( 1 - D ( W > x ))⇤ . L W Adv (X, Y)= E (x,y)⇠p(X,Y) ⇥ - log ( 1 - D ( W > x ))⇤ discriminator generator • Orthogonal Regularization L Proc (X, Y)= W ⊤ X − PY 2 F correspondence R ortho = β W > W ( ee > - I ) 2 F umn vector of all ones. Scanning throug off-diagonal terms • Text Features : TF-IDF, GloVe embedding, DAN [Cer] sentence/paragraph embedding (X, • Metrics for Retrieval Tasks EXAMINATION: CHEST (PORTABLE AP) INDICATION: History: 70M with intubated FINDINGS: Endotracheal tube is in standard position. [**Doctor First Name **] enteric tube courses below the left hemidiaphragm with tip off the inferior borders of the film. … IMPRESSION: 1. New extensive subcutaneous [**Doctor First Name 21**] within the neck and chest. New right fourth and fifth rib fractures anteriorly. … {stmharry,ckbjimmy,wboag,mmd,psz}@mit.edu 2001 2012 2011 2015 MIMIC-CXR MIMIC-III *sourced from MIMIC-III for patients with MIMIC-CXR records for the overlapping year only. Relevancy judged by clinician. References Grave et al. Unsupervised alignment of embeddings with Wasserstein procrustes. In arXiv, 2018. Cer et al. Universal sentence encoder. In arXiv, 2018.
Transcript
Unsupervised Multimodal Representation Learning across Medical Images and ReportsTzu-Ming Harry Hsu, Wei-Hung Weng, Willie Boag, Matthew McDermott, and Peter Szolovits
Motivation & Contributions
MIMIC Chest X-ray Dataset
Results
• Medical reports and images have been explored in the form of report generation, image annotation, image generation, and joint representation learning
• Parallel image/report pairs are not always feasible
• We investigate the effect of using semi-supervised algorithms in learning joint embedding spaces on the MIMIC-Chest X-ray* dataset
• We show that, on large scale, unsupervised methods achieve comparable results on the metrics for retrieval
Methods
• The MIMIC Chest X-ray (MIMIC-CXR) consists of 473,057 chest X-ray images and 206,563 corresponding radiology reports from 63,478 patients admitted to critical care units at Beth Israel Deaconess Medical Center.
Paired Image and Report
Dataset Timeline
worsening opacities in the lung bases NAME reflect worsening atelectasis …
both the text and image modalities with joint embedding spaces under a spectrum of supervised andunsupervised methods. In particular, we make the following contributions:
1. We establish baseline results and evaluation methods for jointly embedding radiologicalimages and reports via retrieval and distance metrics.
2. We profile the impact of supervision level on the quality of representation learning in jointembedding spaces.
3. We characterize the influence of using different sections from the report on representationlearning.
2 Methodology2.1 DataAll experiments in this work used the MIMIC-CXR dataset. MIMIC-CXR consists of 473,057 chest X-ray images and 206,563 reports from 63,478 patients. Of these images, 240,780 are of anteroposterior(AP) views, which we focus on in this work. Further, we eliminate all duplicated radiograph imageswith adjusted brightness or contrast2, leaving a total of 95,242/87,353 images/reports, which wesubdivide into a train set of 75,147/69,171 and a test set of 19,825/18,182 images/reports, with nooverlap of patients between the two. Radiological reports are parsed into sections and we use eitherthe impression or the findings sections.For evaluation, we aggregate a list of unique International Classification of Diseases (ICD-9) codesfrom all patient admissions and ask a clinician to pick out a subset of codes that are related to thoracicdiseases. Records with ICD-9 codes in the subset are then extracted, including 3,549 images from380 patients. This population serves as a disease-related evaluation for retrieval algorithms. Note thatthis disease information is never provided during training in any setting.
2.2 MethodsOur overall experimental flow follows Figure 1. Notes are featurized via (1) term frequency-inverse document frequency (TF-IDF) over bi-grams, (2) pre-trained GloVe word embeddings [21]averaged across the selected section of the report, (3) sentence embeddings, or (4) paragraphembeddings. In (3) and (4), we first perform sentence/paragraph splitting, and then fine-tune a deepaveraging network (DAN) encoder [1, 3, 13] with the corpus. Embeddings are finally averaged acrosssentences/paragraphs. The DAN encoder is pretrained on a variety of data sources and tasks andfine-tuned on the context of report sections.Images are resized to 256⇥256, then featurized to the last bottleneck layer of a pretrained DenseNet-121 model [22]. PCA is applied onto the 1024-dimension raw image features to obtain 64-dimensionfeatures.3 Text features are projected into the 64-dimension image feature space. We use severalmethods regarding different objectives.
Embedding Alignment (EA) Here, we find a linear transformation between two sets of matchedpoints X 2 RdX⇥n and Y 2 RdY ⇥n by minimizing LEA (X,Y) =
��W>X � Y��2
F.
2Commonly produced for clinical needs396.9% variance explained
2
Supervision and Text Feature
only include bi-gram in the table. Additionally, we find that the high-level sentence and paragraphembeddings approach underperformed the bi-gram text representation. Although having generaliz-ability [3], sentence and paragraph embeddings learned from the supervised multi-task pre-trainedmodel may not be able to represent the domain-specific radiological reports well due to the lack ofmedical domain tasks in the pre-training process. Unsupervised procrustes refinement is occasionally,but not universally helpful. Note that MRR is comparatively small since reports are in general highlysimilar for radiographs with the same disease types.
The Impact of Supervision Fraction We define the supervision fraction as the fraction of pairinginformation provided in the training set. Note the ICD-9 codes are not provided for training even inthe fully supervised setting. Figure 2 shows our evaluation metrics for models trained using bi-gramtext features and the semi-supervised learning objective for various supervision fractions. A minimalsupervision as low as 0.1% provided can drastically improve the alignment quality, especially interms of cosine similarity and nDCG. More annotations further improve the performance measures,but one would almost require exponentially many data points in exchange for a linear increase. Thatimplies the possibility of concatenating a well-annotated dataset and a large but unannotated datasetfor a substantial performance boost.
Figure 2: Performance measures of retrieval tasks at k retrieved items as a function of the supervisionfraction. Higher is better. Note the x-axis is in log scale. Unsupervised is on the left, increasinglysupervised to the right. Dashed lines indicate the performance by chance. Vertical bars indicate the95% confidence interval, and some are too narrow to be visible.
Using Different Sections of the Report We investigate the effectiveness of using different sectionsfor the embedding alignment task. All models in Figure 3 run with a supervision fraction of 1%. Themodels trained on the findings section outperformed the models trained on the impression sectionusing cosine similarity and MRR. This makes sense from a clinical perspective since the radiologistsusually only describe image patterns in the findings section and thus they would be aligned well. Onthe other hand, they make radiological-clinical integrated interpretations in the impression section,which means that the both the image-uncorrelated clinical history and findings were mentioned inthe impression section. Since nDCG is calculated using ICD-9 codes, which carry disease-relatedinformation, it naturally aligns with the purpose of writing an impression section. This may explainwhy the models trained on impression section worked better for nDCG.
impression findingsReport Section
0.35
0.40
0.45
0.50
Sim
ilarity
tfidf
word
sentence
paragraph
impression findingsReport Section
0
2
4
MRR
(T!
I,⇥
10�
3)
impression findingsReport Section
0.150
0.175
0.200
0.225
nDCG
@10
0(T
!I)
Figure 3: Different metrics for retrieval on either the impression or findings section using four typesof features. 95% confidence intervals are indicated on the bars.
4 ConclusionMIMIC-CXR will soon be the largest publicly available imaging dataset consisting of both medicalimages and paired radiological reports, promising myriad applications that can make use of bothmodalities together. We establish baseline results using supervised and unsupervised joint embeddingmethods along with local (direct pairs) and global (ICD-9 code groupings) retrieval evaluation metrics.Results show a possibility of incorporating more unsupervised data into training for minimal-effortperformance increase. A further study of joint embeddings between these modalities may enablesignificant applications, such as text/image generation or the incorporation of other EMR modalities.
4
only include bi-gram in the table. Additionally, we find that the high-level sentence and paragraphembeddings approach underperformed the bi-gram text representation. Although having generaliz-ability [3], sentence and paragraph embeddings learned from the supervised multi-task pre-trainedmodel may not be able to represent the domain-specific radiological reports well due to the lack ofmedical domain tasks in the pre-training process. Unsupervised procrustes refinement is occasionally,but not universally helpful. Note that MRR is comparatively small since reports are in general highlysimilar for radiographs with the same disease types.
The Impact of Supervision Fraction We define the supervision fraction as the fraction of pairinginformation provided in the training set. Note the ICD-9 codes are not provided for training even inthe fully supervised setting. Figure 2 shows our evaluation metrics for models trained using bi-gramtext features and the semi-supervised learning objective for various supervision fractions. A minimalsupervision as low as 0.1% provided can drastically improve the alignment quality, especially interms of cosine similarity and nDCG. More annotations further improve the performance measures,but one would almost require exponentially many data points in exchange for a linear increase. Thatimplies the possibility of concatenating a well-annotated dataset and a large but unannotated datasetfor a substantial performance boost.
0 10�3 10�2 10�1 100
Supervision Fraction
0.2
0.4
0.6
Sim
ilarity
0 10�3 10�2 10�1 100
Supervision Fraction
0
5
10
MRR
(⇥10
�3)
T!I I!T
0 10�3 10�2 10�1 100
Supervision Fraction
0.10
0.15
0.20
0.25
nDCG
@k
(T!
I)
k=1 k=100
0 10�3 10�2 10�1 100
Supervision Fraction
0.10
0.15
0.20
0.25
nDCG
@k
(I!
T)
k=1 k=100
Figure 2: Performance measures of retrieval tasks at k retrieved items as a function of the supervisionfraction. Higher is better. Note the x-axis is in log scale. Unsupervised is on the left, increasinglysupervised to the right. Dashed lines indicate the performance by chance. Vertical bars indicate the95% confidence interval, and some are too narrow to be visible.
Using Different Sections of the Report We investigate the effectiveness of using different sectionsfor the embedding alignment task. All models in Figure 3 run with a supervision fraction of 1%. Themodels trained on the findings section outperformed the models trained on the impression sectionusing cosine similarity and MRR. This makes sense from a clinical perspective since the radiologistsusually only describe image patterns in the findings section and thus they would be aligned well. Onthe other hand, they make radiological-clinical integrated interpretations in the impression section,which means that the both the image-uncorrelated clinical history and findings were mentioned inthe impression section. Since nDCG is calculated using ICD-9 codes, which carry disease-relatedinformation, it naturally aligns with the purpose of writing an impression section. This may explainwhy the models trained on impression section worked better for nDCG.
Figure 3: Different metrics for retrieval on either the impression or findings section using four typesof features. 95% confidence intervals are indicated on the bars.
4 ConclusionMIMIC-CXR will soon be the largest publicly available imaging dataset consisting of both medicalimages and paired radiological reports, promising myriad applications that can make use of bothmodalities together. We establish baseline results using supervised and unsupervised joint embeddingmethods along with local (direct pairs) and global (ICD-9 code groupings) retrieval evaluation metrics.Results show a possibility of incorporating more unsupervised data into training for minimal-effortperformance increase. A further study of joint embeddings between these modalities may enablesignificant applications, such as text/image generation or the incorporation of other EMR modalities.
4
Highly RelatedPartially Related
Coun
t
0400800
12001600
CHF N
OSAc
ute r
espi
ratry
failu
rePn
eum
onia,
orga
nism
NOS
Hist
ory o
f tob
acco
use
Chr p
ulm
on h
eart
dis N
ECPl
eura
l effu
sion
NOS
Stat
us-p
ost p
tca
Food
/vom
it pn
eum
oniti
sTo
bacc
o use
diso
rder
Aorto
coro
nary
byp
ass
Chr a
irway
obst
ruct
NEC
Ac on
chr d
iast h
rt fa
ilFlu
id ov
erloa
d NE
CAc
resp
flr f
ol trm
a/sr
gVe
ntltr
asso
c pne
umon
iaOs
teop
oros
is NO
SPu
lmon
ary c
ollap
seSu
rg co
mpl
-hea
rtAc
on ch
r sys
t hrt
fail
Ac/c
hr sy
st/d
ia hr
t fail
Chr d
iasto
lic h
rt fa
ilCo
r ath
uns
p vs
l ntv
/gft
Asth
ma N
OSHx
-ven
thro
mbo
sis/e
mbo
lsAc
ute l
ung
edem
a NOS
Chr s
ysto
lic h
rt fa
ilure
Prim
card
iomyo
path
y NEC
Effu
sion
NEC
exc t
b
Coun
t
0
75000
150000
PA AP Lateral
PatientsReportsOriginal ImagesAdjusted Images
MIMIC-CXR X-ray Breakdown
ICD-9 CodesSupervision Fraction
Using Different Report Sections
nDCG
@10
0
0.10
0.15
0.20
0.25
bi-gra
mword
senten
ce
paragr
aph
Sim
ilarit
y
0.00
0.25
0.50
0.75
bi-gra
mword
senten
ce
paragr
aphEAAdvAdv + Proc
Text Feature Method SimilarityMRR(⇥10�3) nDCG@1 nDCG@10 nDCG@100
Table 1: Comparison among supervised (upper) and unsupervised (lower) methods. Subscripts showthe half width of 95% confidence intervals. Bold denotes the best performance in each group. Chanceis the expected value if we randomly yield retrievals. Higher is better for all metrics.
Adversarial Domain Adaption (Adv) Adversarial training pits a discriminator, D, implementedas a 2-layer (hidden size 256) neural network using scaled exponential linear units (SELUs) [15],against a projection matrix W, as the generator. D is trained to classify points in the joint spaceaccording to source modality, and W is trained adversarially to fool D. Alternatively, D min-imizes LD
Adv (X,Y) = E(x,y)⇠p(X,Y)
⇥� log D
�W>x
�� log (1 � D (y))
⇤when W minimizes
LWAdv (X,Y) = E(x,y)⇠p(X,Y)
⇥� log
�1 � D
�W>x
��⇤.
Procrustes Refinement (Adv + Proc) On top of adversarial training, we also use an unsupervisedProcrustes induced refinement as in [6].
Semi-Supervised We also assess how much supervision is necessary to ensure strong performanceon these modalities by randomly subsampling our data into supervised and unsupervised samples.We then combine the embedding alignment objective and adversarial training objective functionsas L = LEA (X,Y) + �LAdv (X,Y) and train simultaneously as we vary the fraction trained.Preliminary experiments suggests � = 0.1.
Orthogonal Regularization [6, 28, 32] all showed that imposing orthonormality on linear projec-tions leads to better performance and stability in training . However, [2] suggested orthogonality (i.e.,not constraining the norms) can perform better as a regularization. Thus on top of the objectives, weadd Rortho = �
��W>W ��ee> � I
���2
F, where � denotes element-wise product and e denotes a
column vector of all ones. Scanning through a range shows � = 0.01 yields good performance.
2.3 EvaluationWe evaluate via cross domain retrieval in the test set Q: querying in the joint embedding space forclosest neighboring images using a report, T ! I, or vice-versa, I ! T. For direct pairings, wecompute the cosine similarity, and MRR = 1
|Q|P
q2Q1
rankqwhere rankq is the rank of the first true
pair for q (e.g., the first paired image or text corresponding to the query q) in the retrieval list. Forthoracic disease induced pairings, we first define the relevance relpq 2 [0, 1] between two entries pand q as the intersection-over-union of their respective set of ICD-9 codes. Then we calculate thenormalized discounted cumulative gain [14] nDCG@k = 1
|Q|P
q2Q1
IDCGq
Pkp=1
2relpq�1log2(p+1) , where
IDCGq denotes the ideal DCG value for q using a perfect retrieval algorithm. All experiments arerepeated with random initial seeds for at least 5 times. Means and 95% confidence intervals arereported in the following section.
3 ResultsRetrieval with/without Supervision Table 1 compares four types of text features and super-vised/unsupervised methods. We find that unsupervised methods can achieve comparable resultson disease-related retrieval tasks on a large scale (nDCG@100) without the need for labeling thechest X-ray images. Experiments show uni-, bi-, and tri-grams yield very similar results and we
3
relation coefficient in [0, 1]
Text Feature Method SimilarityMRR(⇥10�3) nDCG@1 nDCG@10 nDCG@100
Table 1: Comparison among supervised (upper) and unsupervised (lower) methods. Subscripts showthe half width of 95% confidence intervals. Bold denotes the best performance in each group. Chanceis the expected value if we randomly yield retrievals. Higher is better for all metrics.
Adversarial Domain Adaption (Adv) Adversarial training pits a discriminator, D, implementedas a 2-layer (hidden size 256) neural network using scaled exponential linear units (SELUs) [15],against a projection matrix W, as the generator. D is trained to classify points in the joint spaceaccording to source modality, and W is trained adversarially to fool D. Alternatively, D min-imizes LD
Adv (X,Y) = E(x,y)⇠p(X,Y)
⇥� log D
�W>x
�� log (1 � D (y))
⇤when W minimizes
LWAdv (X,Y) = E(x,y)⇠p(X,Y)
⇥� log
�1 � D
�W>x
��⇤.
Procrustes Refinement (Adv + Proc) On top of adversarial training, we also use an unsupervisedProcrustes induced refinement as in [6].
Semi-Supervised We also assess how much supervision is necessary to ensure strong performanceon these modalities by randomly subsampling our data into supervised and unsupervised samples.We then combine the embedding alignment objective and adversarial training objective functionsas L = LEA (X,Y) + �LAdv (X,Y) and train simultaneously as we vary the fraction trained.Preliminary experiments suggests � = 0.1.
Orthogonal Regularization [6, 28, 32] all showed that imposing orthonormality on linear projec-tions leads to better performance and stability in training . However, [2] suggested orthogonality (i.e.,not constraining the norms) can perform better as a regularization. Thus on top of the objectives, weadd Rortho = �
��W>W ��ee> � I
���2
F, where � denotes element-wise product and e denotes a
column vector of all ones. Scanning through a range shows � = 0.01 yields good performance.
2.3 EvaluationWe evaluate via cross domain retrieval in the test set Q: querying in the joint embedding space forclosest neighboring images using a report, T ! I, or vice-versa, I ! T. For direct pairings, wecompute the cosine similarity, and MRR = 1
|Q|P
q2Q1
rankqwhere rankq is the rank of the first true
pair for q (e.g., the first paired image or text corresponding to the query q) in the retrieval list. Forthoracic disease induced pairings, we first define the relevance relpq 2 [0, 1] between two entries pand q as the intersection-over-union of their respective set of ICD-9 codes. Then we calculate thenormalized discounted cumulative gain [14] nDCG@k = 1
|Q|P
q2Q1
IDCGq
Pkp=1
2relpq�1log2(p+1) , where
IDCGq denotes the ideal DCG value for q using a perfect retrieval algorithm. All experiments arerepeated with random initial seeds for at least 5 times. Means and 95% confidence intervals arereported in the following section.
3 ResultsRetrieval with/without Supervision Table 1 compares four types of text features and super-vised/unsupervised methods. We find that unsupervised methods can achieve comparable resultson disease-related retrieval tasks on a large scale (nDCG@100) without the need for labeling thechest X-ray images. Experiments show uni-, bi-, and tri-grams yield very similar results and we
3
rank of the first related item
• Embedding Alignment (EA)
• Adversarial Domain Adaption (Adv)
• Procrustes Refinement (Adv + Proc) [Grave]
worsening opacities in the lung bases NAME reflect worsening atelectasis …
both the text and image modalities with joint embedding spaces under a spectrum of supervised andunsupervised methods. In particular, we make the following contributions:
1. We establish baseline results and evaluation methods for jointly embedding radiologicalimages and reports via retrieval and distance metrics.
2. We profile the impact of supervision level on the quality of representation learning in jointembedding spaces.
3. We characterize the influence of using different sections from the report on representationlearning.
2 Methodology2.1 DataAll experiments in this work used the MIMIC-CXR dataset. MIMIC-CXR consists of 473,057 chest X-ray images and 206,563 reports from 63,478 patients. Of these images, 240,780 are of anteroposterior(AP) views, which we focus on in this work. Further, we eliminate all duplicated radiograph imageswith adjusted brightness or contrast2, leaving a total of 95,242/87,353 images/reports, which wesubdivide into a train set of 75,147/69,171 and a test set of 19,825/18,182 images/reports, with nooverlap of patients between the two. Radiological reports are parsed into sections and we use eitherthe impression or the findings sections.For evaluation, we aggregate a list of unique International Classification of Diseases (ICD-9) codesfrom all patient admissions and ask a clinician to pick out a subset of codes that are related to thoracicdiseases. Records with ICD-9 codes in the subset are then extracted, including 3,549 images from380 patients. This population serves as a disease-related evaluation for retrieval algorithms. Note thatthis disease information is never provided during training in any setting.
2.2 MethodsOur overall experimental flow follows Figure 1. Notes are featurized via (1) term frequency-inverse document frequency (TF-IDF) over bi-grams, (2) pre-trained GloVe word embeddings [21]averaged across the selected section of the report, (3) sentence embeddings, or (4) paragraphembeddings. In (3) and (4), we first perform sentence/paragraph splitting, and then fine-tune a deepaveraging network (DAN) encoder [1, 3, 13] with the corpus. Embeddings are finally averaged acrosssentences/paragraphs. The DAN encoder is pretrained on a variety of data sources and tasks andfine-tuned on the context of report sections.Images are resized to 256⇥256, then featurized to the last bottleneck layer of a pretrained DenseNet-121 model [22]. PCA is applied onto the 1024-dimension raw image features to obtain 64-dimensionfeatures.3 Text features are projected into the 64-dimension image feature space. We use severalmethods regarding different objectives.
Embedding Alignment (EA) Here, we find a linear transformation between two sets of matchedpoints X 2 RdX⇥n and Y 2 RdY ⇥n by minimizing LEA (X,Y) =
��W>X � Y��2
F.
2Commonly produced for clinical needs396.9% variance explained
2
Text Feature Method SimilarityMRR(⇥10�3) nDCG@1 nDCG@10 nDCG@100
Table 1: Comparison among supervised (upper) and unsupervised (lower) methods. Subscripts showthe half width of 95% confidence intervals. Bold denotes the best performance in each group. Chanceis the expected value if we randomly yield retrievals. Higher is better for all metrics.
Adversarial Domain Adaption (Adv) Adversarial training pits a discriminator, D, implementedas a 2-layer (hidden size 256) neural network using scaled exponential linear units (SELUs) [15],against a projection matrix W, as the generator. D is trained to classify points in the joint spaceaccording to source modality, and W is trained adversarially to fool D. Alternatively, D min-imizes LD
Adv (X,Y) = E(x,y)⇠p(X,Y)
⇥� log D
�W>x
�� log (1 � D (y))
⇤when W minimizes
LWAdv (X,Y) = E(x,y)⇠p(X,Y)
⇥� log
�1 � D
�W>x
��⇤.
Procrustes Refinement (Adv + Proc) On top of adversarial training, we also use an unsupervisedProcrustes induced refinement as in [6].
Semi-Supervised We also assess how much supervision is necessary to ensure strong performanceon these modalities by randomly subsampling our data into supervised and unsupervised samples.We then combine the embedding alignment objective and adversarial training objective functionsas L = LEA (X,Y) + �LAdv (X,Y) and train simultaneously as we vary the fraction trained.Preliminary experiments suggests � = 0.1.
Orthogonal Regularization [6, 28, 32] all showed that imposing orthonormality on linear projec-tions leads to better performance and stability in training . However, [2] suggested orthogonality (i.e.,not constraining the norms) can perform better as a regularization. Thus on top of the objectives, weadd Rortho = �
��W>W ��ee> � I
���2
F, where � denotes element-wise product and e denotes a
column vector of all ones. Scanning through a range shows � = 0.01 yields good performance.
2.3 EvaluationWe evaluate via cross domain retrieval in the test set Q: querying in the joint embedding space forclosest neighboring images using a report, T ! I, or vice-versa, I ! T. For direct pairings, wecompute the cosine similarity, and MRR = 1
|Q|P
q2Q1
rankqwhere rankq is the rank of the first true
pair for q (e.g., the first paired image or text corresponding to the query q) in the retrieval list. Forthoracic disease induced pairings, we first define the relevance relpq 2 [0, 1] between two entries pand q as the intersection-over-union of their respective set of ICD-9 codes. Then we calculate thenormalized discounted cumulative gain [14] nDCG@k = 1
|Q|P
q2Q1
IDCGq
Pkp=1
2relpq�1log2(p+1) , where
IDCGq denotes the ideal DCG value for q using a perfect retrieval algorithm. All experiments arerepeated with random initial seeds for at least 5 times. Means and 95% confidence intervals arereported in the following section.
3 ResultsRetrieval with/without Supervision Table 1 compares four types of text features and super-vised/unsupervised methods. We find that unsupervised methods can achieve comparable resultson disease-related retrieval tasks on a large scale (nDCG@100) without the need for labeling thechest X-ray images. Experiments show uni-, bi-, and tri-grams yield very similar results and we
3
Text Feature Method SimilarityMRR(⇥10�3) nDCG@1 nDCG@10 nDCG@100
Table 1: Comparison among supervised (upper) and unsupervised (lower) methods. Subscripts showthe half width of 95% confidence intervals. Bold denotes the best performance in each group. Chanceis the expected value if we randomly yield retrievals. Higher is better for all metrics.
Adversarial Domain Adaption (Adv) Adversarial training pits a discriminator, D, implementedas a 2-layer (hidden size 256) neural network using scaled exponential linear units (SELUs) [15],against a projection matrix W, as the generator. D is trained to classify points in the joint spaceaccording to source modality, and W is trained adversarially to fool D. Alternatively, D min-imizes LD
Adv (X,Y) = E(x,y)⇠p(X,Y)
⇥� log D
�W>x
�� log (1 � D (y))
⇤when W minimizes
LWAdv (X,Y) = E(x,y)⇠p(X,Y)
⇥� log
�1 � D
�W>x
��⇤.
Procrustes Refinement (Adv + Proc) On top of adversarial training, we also use an unsupervisedProcrustes induced refinement as in [6].
Semi-Supervised We also assess how much supervision is necessary to ensure strong performanceon these modalities by randomly subsampling our data into supervised and unsupervised samples.We then combine the embedding alignment objective and adversarial training objective functionsas L = LEA (X,Y) + �LAdv (X,Y) and train simultaneously as we vary the fraction trained.Preliminary experiments suggests � = 0.1.
Orthogonal Regularization [6, 28, 32] all showed that imposing orthonormality on linear projec-tions leads to better performance and stability in training . However, [2] suggested orthogonality (i.e.,not constraining the norms) can perform better as a regularization. Thus on top of the objectives, weadd Rortho = �
��W>W ��ee> � I
���2
F, where � denotes element-wise product and e denotes a
column vector of all ones. Scanning through a range shows � = 0.01 yields good performance.
2.3 EvaluationWe evaluate via cross domain retrieval in the test set Q: querying in the joint embedding space forclosest neighboring images using a report, T ! I, or vice-versa, I ! T. For direct pairings, wecompute the cosine similarity, and MRR = 1
|Q|P
q2Q1
rankqwhere rankq is the rank of the first true
pair for q (e.g., the first paired image or text corresponding to the query q) in the retrieval list. Forthoracic disease induced pairings, we first define the relevance relpq 2 [0, 1] between two entries pand q as the intersection-over-union of their respective set of ICD-9 codes. Then we calculate thenormalized discounted cumulative gain [14] nDCG@k = 1
|Q|P
q2Q1
IDCGq
Pkp=1
2relpq�1log2(p+1) , where
IDCGq denotes the ideal DCG value for q using a perfect retrieval algorithm. All experiments arerepeated with random initial seeds for at least 5 times. Means and 95% confidence intervals arereported in the following section.
3 ResultsRetrieval with/without Supervision Table 1 compares four types of text features and super-vised/unsupervised methods. We find that unsupervised methods can achieve comparable resultson disease-related retrieval tasks on a large scale (nDCG@100) without the need for labeling thechest X-ray images. Experiments show uni-, bi-, and tri-grams yield very similar results and we
3
discriminator
generator
• Orthogonal Regularization
LProc (X,Y) =!
!W⊤X−PY
!
!
2
F
correspondence
Text Feature Method SimilarityMRR(⇥10�3) nDCG@1 nDCG@10 nDCG@100
Table 1: Comparison among supervised (upper) and unsupervised (lower) methods. Subscripts showthe half width of 95% confidence intervals. Bold denotes the best performance in each group. Chanceis the expected value if we randomly yield retrievals. Higher is better for all metrics.
Adversarial Domain Adaption (Adv) Adversarial training pits a discriminator, D, implementedas a 2-layer (hidden size 256) neural network using scaled exponential linear units (SELUs) [15],against a projection matrix W, as the generator. D is trained to classify points in the joint spaceaccording to source modality, and W is trained adversarially to fool D. Alternatively, D min-imizes LD
Adv (X,Y) = E(x,y)⇠p(X,Y)
⇥� log D
�W>x
�� log (1 � D (y))
⇤when W minimizes
LWAdv (X,Y) = E(x,y)⇠p(X,Y)
⇥� log
�1 � D
�W>x
��⇤.
Procrustes Refinement (Adv + Proc) On top of adversarial training, we also use an unsupervisedProcrustes induced refinement as in [6].
Semi-Supervised We also assess how much supervision is necessary to ensure strong performanceon these modalities by randomly subsampling our data into supervised and unsupervised samples.We then combine the embedding alignment objective and adversarial training objective functionsas L = LEA (X,Y) + �LAdv (X,Y) and train simultaneously as we vary the fraction trained.Preliminary experiments suggests � = 0.1.
Orthogonal Regularization [6, 28, 32] all showed that imposing orthonormality on linear projec-tions leads to better performance and stability in training . However, [2] suggested orthogonality (i.e.,not constraining the norms) can perform better as a regularization. Thus on top of the objectives, weadd Rortho = �
��W>W ��ee> � I
���2
F, where � denotes element-wise product and e denotes a
column vector of all ones. Scanning through a range shows � = 0.01 yields good performance.
2.3 EvaluationWe evaluate via cross domain retrieval in the test set Q: querying in the joint embedding space forclosest neighboring images using a report, T ! I, or vice-versa, I ! T. For direct pairings, wecompute the cosine similarity, and MRR = 1
|Q|P
q2Q1
rankqwhere rankq is the rank of the first true
pair for q (e.g., the first paired image or text corresponding to the query q) in the retrieval list. Forthoracic disease induced pairings, we first define the relevance relpq 2 [0, 1] between two entries pand q as the intersection-over-union of their respective set of ICD-9 codes. Then we calculate thenormalized discounted cumulative gain [14] nDCG@k = 1
|Q|P
q2Q1
IDCGq
Pkp=1
2relpq�1log2(p+1) , where
IDCGq denotes the ideal DCG value for q using a perfect retrieval algorithm. All experiments arerepeated with random initial seeds for at least 5 times. Means and 95% confidence intervals arereported in the following section.
3 ResultsRetrieval with/without Supervision Table 1 compares four types of text features and super-vised/unsupervised methods. We find that unsupervised methods can achieve comparable resultson disease-related retrieval tasks on a large scale (nDCG@100) without the need for labeling thechest X-ray images. Experiments show uni-, bi-, and tri-grams yield very similar results and we
3
off-diagonal terms
• Text Features : TF-IDF, GloVe embedding, DAN [Cer] sentence/paragraph embedding
worsening opacities in the lung bases NAME reflect worsening atelectasis …
both the text and image modalities with joint embedding spaces under a spectrum of supervised andunsupervised methods. In particular, we make the following contributions:
1. We establish baseline results and evaluation methods for jointly embedding radiologicalimages and reports via retrieval and distance metrics.
2. We profile the impact of supervision level on the quality of representation learning in jointembedding spaces.
3. We characterize the influence of using different sections from the report on representationlearning.
2 Methodology2.1 DataAll experiments in this work used the MIMIC-CXR dataset. MIMIC-CXR consists of 473,057 chest X-ray images and 206,563 reports from 63,478 patients. Of these images, 240,780 are of anteroposterior(AP) views, which we focus on in this work. Further, we eliminate all duplicated radiograph imageswith adjusted brightness or contrast2, leaving a total of 95,242/87,353 images/reports, which wesubdivide into a train set of 75,147/69,171 and a test set of 19,825/18,182 images/reports, with nooverlap of patients between the two. Radiological reports are parsed into sections and we use eitherthe impression or the findings sections.For evaluation, we aggregate a list of unique International Classification of Diseases (ICD-9) codesfrom all patient admissions and ask a clinician to pick out a subset of codes that are related to thoracicdiseases. Records with ICD-9 codes in the subset are then extracted, including 3,549 images from380 patients. This population serves as a disease-related evaluation for retrieval algorithms. Note thatthis disease information is never provided during training in any setting.
2.2 MethodsOur overall experimental flow follows Figure 1. Notes are featurized via (1) term frequency-inverse document frequency (TF-IDF) over bi-grams, (2) pre-trained GloVe word embeddings [21]averaged across the selected section of the report, (3) sentence embeddings, or (4) paragraphembeddings. In (3) and (4), we first perform sentence/paragraph splitting, and then fine-tune a deepaveraging network (DAN) encoder [1, 3, 13] with the corpus. Embeddings are finally averaged acrosssentences/paragraphs. The DAN encoder is pretrained on a variety of data sources and tasks andfine-tuned on the context of report sections.Images are resized to 256⇥256, then featurized to the last bottleneck layer of a pretrained DenseNet-121 model [22]. PCA is applied onto the 1024-dimension raw image features to obtain 64-dimensionfeatures.3 Text features are projected into the 64-dimension image feature space. We use severalmethods regarding different objectives.
Embedding Alignment (EA) Here, we find a linear transformation between two sets of matchedpoints X 2 RdX⇥n and Y 2 RdY ⇥n by minimizing LEA (X,Y) =
��W>X � Y��2
F.
2Commonly produced for clinical needs396.9% variance explained
2
• Metrics for Retrieval Tasks
EXAMINATION: CHEST (PORTABLE AP) INDICATION: History: 70M with intubated FINDINGS:
Endotracheal tube is in standard position. [**Doctor First Name **] enteric tube courses below the left hemidiaphragm with tip off the inferior borders of the film. …
IMPRESSION: 1. New extensive subcutaneous [**Doctor First Name 21**] within the neck and chest. New right fourth and fifth rib fractures anteriorly. …
{stmharry,ckbjimmy,wboag,mmd,psz}@mit.edu
2001 20122011 2015MIMIC-CXRMIMIC-III
*sourced from MIMIC-III for patients with MIMIC-CXR records for the overlapping year only. Relevancy judged by clinician.
References Grave et al. Unsupervised alignment of embeddings with Wasserstein procrustes. In arXiv, 2018. Cer et al. Universal sentence encoder. In arXiv, 2018.
![A arXiv:1611.02648v2 [cs.LG] 13 Jan 2017 · Under review as a conference paper at ICLR 2017 1.1 RELATED WORK Unsupervised clustering can be considered a subset of the problem of disentangling](https://static.documents.pub/doc/80x56/5ada268c7f8b9a53618c3277/a-arxiv161102648v2-cslg-13-jan-2017-review-as-a-conference-paper-at-iclr-2017.jpg)
