Proceedings of the 59th Annual Meeting of the Association for Computational Linguisticsand the 11th International Joint Conference on Natural Language Processing, pages 3037–3049
1 The CoAI group, DCST; 1 Institute for Artificial Intelligence;1 State Key Lab of Intelligent Technology and Systems;
1 Beijing National Research Center for Information Science and Technology;1 Tsinghua University, Beijing 100084, China.2 School of Computer Science and Technology,
Unsupervised commonsense question answer-ing is appealing since it does not rely on anylabeled task data. Among existing work, apopular solution is to use pre-trained languagemodels to score candidate choices directly con-ditioned on the question or context. However,such scores from language models can be eas-ily affected by irrelevant factors, such as wordfrequencies, sentence structures, etc. Thesedistracting factors may not only mislead themodel to choose a wrong answer but also makeit oversensitive to lexical perturbations in can-didate answers.
In this paper, we present a novel SEmantic-based Question Answering method (SEQA)for unsupervised commonsense question an-swering. Instead of directly scoring each an-swer choice, our method first generates a setof plausible answers with generative models(e.g., GPT-2), and then uses these plausible an-swers to select the correct choice by consider-ing the semantic similarity between each plau-sible answer and each choice. We devise a sim-ple, yet sound formalism for this idea and ver-ify its effectiveness and robustness with exten-sive experiments. We evaluate the proposedmethod on four benchmark datasets, and ourmethod achieves the best results in unsuper-vised settings. Moreover, when attacked byTextFooler (Jin et al., 2020) with synonym re-placement, SEQA demonstrates much less per-formance drops than baselines, thereby indicat-ing stronger robustness.
1 Introduction
Pre-trained language models have been widely usedfor commonsense question answering. Finetuningpre-trained models on task-specific data producesmany state-of-the-art results (Wang et al., 2020;
Figure 1: Two examples of commonsense question an-swering, where the baseline (Pro-A) is oversensitive tolexical perturbations (SR for synonym replacement andST for sentence structure transformation). The scoresfrom Pro-A and our method for each answer choice areshown in the right columns. The underlined score indi-cates the answer choice selected by a method.
Khashabi et al., 2020; Lin et al., 2019). However,this requires amounts of labeled task data. There-fore, it is vital to study unsupervised commonsensequestion answering without relying on any labeleddownstream task data. In this paper, we investigatemultiple-choice commonsense question answeringtasks in an unsupervised setting: given a questionand a set of answer choices, a model is requiredto predict the most reasonable answer choice forthe question, but without access to any labeled taskdata.
Many existing unsupervised methods tacklethese tasks by scoring each answer choice usinga language model, e.g., estimating the generativeprobability of the answer choice conditioned onthe question (Trinh and Le, 2018; Shwartz et al.,2020; Bosselut and Choi, 2019; Tamborrino et al.,2020). Table 1 lists several typical score functions.However, these scores can be easily influenced byword frequencies, sentence structures, and other
3038
factors, which can mislead the models and makeexisting methods oversensitive to lexical perturba-tions (Abdou et al., 2020; Tamborrino et al., 2020).Figure 1 shows two examples. The correct choicesare paraphrased via synonym replacement or struc-ture transformation. In these examples, the baseline(Pro-A) produces much lower scores for the para-phrased choices and chooses the wrong choices.
Since existing methods can be easily distractedby irrelevant factors such as lexical perturbations,we argue that a commonsense question answer-ing method should focus on the answers’ se-mantics and assign similar scores to synony-mous choices. To this end, we introduce anovel SEmantic-based Question Answering model,SEQA, which aims to robustly select correct an-swers in multi-choice commonsense question an-swering in an unsupervised setting. Instead of di-rectly scoring an answer choice, we calculate theprobability of observing the choice’s semantics. Achoice’s semantic score can be obtained by sum-ming the generative probabilities of sentences thathave the same semantic meanings with the choice,where the sentences are called the choice’s support-ers. However, it is hard to obtain the supporterswhich have exactly the same semantic meaningswith the choice, so we reformulate the semanticscore into a soft version as explained in Section3.2. Each supporter is weighed by the semanticsimilarity to the answer choice, which can be com-puted with some off-the-shelf models, such as Sen-tenceBERT (Reimers and Gurevych, 2019). Sincethe supporters and their weights depend on the se-mantics rather than the surface form of the answerchoice, by this means, the effects of the distract-ing factors can be largely suppressed. Moreover,synonymous choices are likely to share the sameset of supporters, so their scores are expected tobe stably close. Our contributions in this paper aresummarized as follows:
• We propose a semantic-based question answer-ing model (SEQA) for robust commonsensequestion answering in an unsupervised setting.Instead of directly scoring the answer choices,our method first generates some plausible an-swers and then uses them to select the correctchoice by considering the semantic similaritybetween each plausible answer and each choice.
• We conduct experiments on four common-sense question answering datasets, whereSEQA achieves the best performance com-
Method Score Function
Pro-A [PLM (A|Q)]1
|A|
Pro-Q [PLM (Q|A)]1
|Q|
MI-QA[PLM (A|Q)PLM (A)
] 1|A|
SEQA (Ours)∑
S∈A ω(S|A)PLM (S|Q)
Table 1: Three existing score functions and our methodfor unsupervised commonsense question answering. Qis the question and A is the choice. A is the set of allpossible answers and ω(S|A) is a weighting functiondefined in Eq.(5). LM refers to a pre-trained languagemodel, such as GPT-2 or BERT1 (Devlin et al., 2019).
pared with strong baselines. When attackedby TextFooler (Jin et al., 2020) with synonymreplacement, our method performs remarkablymore robustly.
2 Related Work
Previous work has explored pre-trained languagemodels (LMs) for unsupervised commonsensequestion answering. In general, these approachestreat LMs as question answering modules.
Table 1 shows three representative methods,which do not use external knowledge and rely fullyon the implicit knowledge encoded in LMs for rea-soning. Probability-A (Pro-A) considers the gener-ative probability of the choice conditioned on thequestion. However, it suffers from the statisticalbias of choices, such as word frequency and sen-tence length (Abdou et al., 2020). To alleviate this,MutualInfo-QA (MI-QA) calculates the mutual in-formation between the question and the choice. An-other way to reduce the impact of statistical bias isto score each choice using the conditional proba-bility of the question rather than the choice (Trinhand Le, 2018; Tamborrino et al., 2020) , which isdenoted as Probability-Q (Pro-Q) in Table 1.
Some recent work claims that external knowl-edge can benefit commonsense reasoning. Besidesstatic knowledge bases (KBs), such as Concept-Net (Speer et al., 2017) and Atomic (Sap et al.,2019a), there are also numerous studies treatingLMs as dynamic KBs. Petroni et al. (2019) showsthat LMs can be used for KB completion. AndDavison et al. (2019) shows that BERT can dis-tinguish true and fake ConceptNet triplets. Fur-ther, the extracted knowledge can work as com-plementary information for answering a question.Rajani et al. (2019) proposes a model for Com-
1PBERT (Q|A) ,∏|Q|
i PBERT (Qi|Q/i, A).
3039
monSenseQA (Talmor et al., 2019) that generatesexplanations for questions, which are then usedas additional inputs. The shortcoming of this ap-proach is that it requires collecting human expla-nations for each new dataset to fine-tune LMs.Some following researches explore unsupervisedexplanation/knowledge generator. CGA (Bosse-lut and Choi, 2019) employs COMET (Bosselutet al., 2019) to generate intermediate inferenceswhich are then used to score the choice. However,COMET is limited by a small set of question typesso that CGA is difficult to generalize to different do-mains. Self-Talk (Shwartz et al., 2020) breaks thelimit by extracting knowledge from GPT-2 (Rad-ford et al., 2019), which has no restriction on thequery types. Thus, Self-Talk can be applied to awide range of domains. Despite the introductionof auxiliary information, these methods are essen-tially dependent on language model scores, so theyare still sensitive to lexical perturbations.
Besides directly using pre-trained LMs, somerecent efforts have been dedicated to automaticallyconstructing task-specific data to train common-sense reasoners in zero-shot settings. Wang et al.(2019) and Kocijan et al. (2019) provide some rulesto construct labeled training data from large cor-pus for pronoun disambiguation. Banerjee andBaral (2020), Moghimifar et al. (2020) and Maet al. (2020) collect training data based on knowl-edge bases, such as Atomic (Sap et al., 2019a).Though effective, they are limited by the specifictask settings or highly dependent on the task-relatedknowledge bases, which makes them difficult totransfer to other commonsense reasoning tasks.
3 Method
In this paper, we focus on unsupervised multiple-choice commonsense question answering, which isformalized as follows: given a question and a set ofchoices, models should select the correct choice:
A = argmaxA
s(A|Q),
where s refers to a score function. Note that wehave no access to any labeled task data.
3.1 Motivation
In existing unsupervised methods, the score func-tions are usually defined based on the languagemodel scores. Taking Pro-A (Table 1) as an exam-ple, it first converts the question into a statement:
• Q: I saw my breath when I exhaled. What wasthe cause of this? −→ Rewrite: I saw my breathwhen I exhaled because
And it then takes the statement as a prompt to calcu-late the generative probability of each choice. Notethat the templates for rewriting is not the focus ofthis paper, and hence we directly use the templatesof previous work (Shwartz et al., 2020; Tamborrinoet al., 2020) for our method and all the baselines inthis paper (see Appendix for details).
Though successful, language model scores canbe affected by many distracting factors, such asword frequency and sentence structure, etc. Thesefactors can disturb the score functions to a large ex-tent, as shown in Figure 1. Our goal is to alleviatethe influence of these distracting factors. Hence wepropose a new method for unsupervised common-sense question answering, which achieves betterresults and performs more robustly.
3.2 SEQASEQA is designed to predict the semantic score ofan answer choice A. Instead of directly estimat-ing the probability P (A|Q) of the single choiceA, the semantic score focuses on the probabilityP (MA|Q) where MA represents A’s semantics.Ideally, we decompose P (MA|Q) into the sum-mation of the conditional probabilities of A’s sup-porters, where the supporters indicates all possibleanswers that have exactly the same semantics MA.Formally, the semantic score is defined as
s(A|Q) , P (MA|Q) =∑S∈SA
PLM (S|Q) (1)
=∑S∈A
I(S ∈ SA)PLM (S|Q). (2)
SA is the set of supporters of choiceA, and A is theset of all possible answers. I(S ∈ SA) is an indi-cator function indicating whether S is a supporterof A. To obtain the supporter set SA, we adopt amodel to extract the sentence-level semantic fea-tures. Ideally, the indicator function is defined as
I(S ∈ SA) =
{1 if cos(hS , hA) = 1,
0 if cos(hS , hA) < 1,(3)
where hA is the semantic features of sentence A,and we assume that S andA are exactly the same insemantics if hS and hA point in the same direction.
However, Eq.(3) uses a hard constraint thatcos(hS , hA) exactly equals to 1, which can be too
3040
strict to find acceptable supporters. Therefore, wereformulate Eq.(2) into a soft version:
s(A|Q) ,∑S∈A
ω(S|A)PLM (S|Q), (4)
where the indicator function in Eq.(2) is replacedby a soft function ω(S|A). To emulate I(S ∈ SA),ω(S|A) is expected to meet three requirements: (1)ω(S|A) ∈ [0, 1] for any S and A; (2) ω(S|A) = 1if cos(hS , hA) = 1; (3) ω(S|A) increases mono-tonically with cos(hS , hA). There are several dif-ferent definitions of ω(S|A) meeting these require-ments, which are explored in Section 4.7.3. In thispaper, ω(S|A) is defined as:
ω(S|A) = 1
Z(T )exp
[cos(hS , hA)
T
]. (5)
T is the temperature, and Z(T ) = exp( 1T ) is a nor-
malization term that makes ω(A|A) = 1. If T → 0,ω(S|A) degenerates to the indicator function. IfT > 0, ω(S|A) relates to the von Mises-Fishersdistribution over the unit sphere in the feature space,where the acceptable feature vectors are distributedaround the mean direction hA
||hA|| .Since it is intractable to enumerate all possible
answers in A, we convert Eq.(4) to an expectationover PLM (S|Q):
s(A|Q) = ES∼PLM (S|Q) [ω(S|A)]
≈ 1
K
K∑i=1
ω(Si|A) (6)
=1
K · Z(T )
K∑i=1
exp
[cos(hSi , hA)
T
], (7)
where S1, · · · , SK are sentences sampled fromPLM (·|Q), and K is the sample size. hA and hSi
can be extracted from a pre-trained model, e.g.,SentenceBERT (Reimers and Gurevych, 2019).
From Eq.(7), we can see the semantic scores(A|Q) is only dependent on the semantic featurehA and regardless of A’s surface form. Therefore,our method will produce similar semantic scoresfor synonymous choices, assuming that the synony-mous choices have similar semantic features.
3.3 The Voting View of SEQAAt the beginning of Section 3.2, we define the se-mantic score as the summation of the conditionalprobabilities over the supporters. However, inEq.(7), the sampled sentences S1, · · · , SK are notA’s supporters because they may not be semanti-cally similar to A. To address the differences, we
Figure 2: Process of SEQA in the view of voting. Weuse the same templates with previous work (Shwartzet al., 2020; Tamborrino et al., 2020) to rewrite inter-rogative sentences into declarative ones. And then useGPT-2 to generate some plausible answers as voters Si,conditioned on the rewritten question. The choices andvoters are encoded via SentenceRoBERTa to obtain se-mantic features, hAj
and hSi, which are then used to
calculate the voting weights ω(Si|Aj). The choice withthe largest score s(Aj |Q) is selected as the answer.
name the sampled sentences S1, · · · , SK as vot-ers, which are plausible answers to the questionQ. In this section, we will show another view ofour method, which works like a procedure that thevoters vote out the correct choice.
Suppose there are two candidate choices A1
and A2, our method is to find the correct choiceaccording to the semantic scores, s(A1|Q) ands(A2|Q). Following Eq.(6), our method can bedecomposed into two steps: First, sample somevoters S1, · · · , SK from PLM (·|Q). This step onlyconsiders the question Q but no candidate choices.Second, each voter votes for the choices with thesemantic similarity weights. For example, Si votesfor Aj with the weight of ω(Si|Aj). The candidatechoice that receives more votes will have a highersemantic score and be selected as the final answer.
Figure 2 shows the process of SEQA in the viewof voting. Although the voting view is intuitive, theformalism in Section 3.2 provides more insights:(1) Our method approximates the probability ofsemantics, which works as the theoretical basis ofSEQA. (2) Our method can be seen as an extensionof Pro-A (see Table 1), since Pro-A only calculatesthe language model score for a single sentence,whereas our method calculates the semantic scorefor a set of supporters. (3) Eq.(4) provides guid-ance, the three requirements mention before, forthe design of the voting weight function ω(S|A).Specifically, the guidance explains the rationalityof the formulation of Eq.(5).
Table 2: Evaluation results, including the original selection accuracy before attack, the accuracy after attack, theattack success rate, the percentage of perturbed words with respect to the original sentence length in successful at-tacks, and the semantic similarity between the original and paraphrased choices. GPT-2, RoBERTa and SRoBERTarefer to GPT-2-xlarge, RoBERTa-large (Liu et al., 2019) and SentenceRoBERTa-large, respectively.
4 Experiments
4.1 Datasets
We conducted experiments on four multiple-choice commonsense question answering tasks,COPA (Roemmele et al., 2011), StoryClozeTest(SCT) (Mostafazadeh et al., 2016), SocialIQA (Sapet al., 2019b) and CosmosQA (Huang et al., 2019).For each instance, only one choice is correct. SeeAppendix for more description about datasets.
For COPA, we reported the results on its testset. As the test sets of another three datasets arehidden, for convenience of analysis, we reportedthe experiment results on their development sets.
4.2 Baselines
We employed five strong baselines. Table 1 showsthree of them, Pro-A, Pro-Q and MI-QA. There isno explicit auxiliary information used in these threemethods, while another two baselines rely on ex-plicit information supplementation. CGA (Bosse-lut and Choi, 2019) and Self-Talk (Shwartz et al.,2020) query pre-trained language models (e.g.,GPT-2, COMET (Bosselut et al., 2019)) for rele-vant knowledge, which forms part of contexts. Andthen, similar to Pro-A, they take the generativeprobabilities of choices as scores.
4.3 Experiment Settings
For each method, we tried different pre-trained lan-guage models (see Appendix for details), and thenselected the pre-trained LMs that maximized the ac-curacy on each dataset. The details of the selectionof pre-trained LMs can be found in Table 2.
For SEQA, we used GPT-2 to generate votersvia Nucleus Sampling (Holtzman et al., 2020) withp = 0.9. The sample size K of voters is set to 500.In Section 4.7.2, we show that a small sample sizecan also lead to superior performance. Self-Talkand CGA also rely on the generated answers fromGPT-2 or COMET. Different from SEQA, for thesetwo baselines, more generated answers will not al-ways lead to better performance (see Section 4.7.2).Thus, we selected the optimal sample size for themrather than the same sample size with SEQA.
When evaluating SEQA on COPA, we tuned thetemperature T on its development set, and thenreported the results on the test set with the tunedtemperature T = 0.1. Due to the absence of testsets of other datasets, we evaluated SEQA on theirdevelopment sets without tuning the temperatureand directly set T = 0.1.
4.4 Main Results
Table 2 shows the evaluation results about accuracyand robustness.
3042
4.4.1 AccuracyAmong all the methods, SEQA achieved the bestperformance on all the datasets. Especially on SCTand CosmosQA, SEQA outperformed the best base-lines by more than 10 points. It can be inferred thatthe semantic scores are beneficial for commonsensequestion answering due to the reduction of dis-tracting factors. Pro-Q performed better than otherbaselines on COPA, perhaps because it sufferedless from the statistic bias of choices (Tamborrinoet al., 2020). However, Pro-Q lost its superiorityon another three datasets, because it is unsuitablefor processing long or complex contexts.
4.4.2 RobustnessTo test the robustness under the synonym replace-ment attack, we used TextFooler (Jin et al., 2020)to attack the methods by perturbing the correctchoices of the correctly predicted examples. Thepercentage of perturbed words refers to what per-centage of words in choices are replaced in success-ful attacks. The semantic similarity is measuredbetween the paraphrased choice and the originalchoice. Considering the attack success rate and theafter-attack accuracy, SEQA is much more robustthan all baselines. To be specific, the attack successrates on SEQA are at least 39 points lower thanthose of Pro-A, CGA, and Self-Talk on all datasets.MI-QA and Pro-Q are designed to reduce the im-pact of statistic bias in choices, so that they canresist lexical perturbation to some extent. Even so,SEQA is remarkably lower than MI-QA and Pro-Qin terms of attack success rates on all datasets.
An observation is that the attack success rateon SEQA on CosmosQA is higher than those onthe other datasets. The reason is that, the contextsin CosmosQA are so complex that GPT-2 is moredifficult to generate high-quality answers. If thereis a more powerful generator, the robustness ofSEQA is expected to have a further improvement.
4.5 Consistency TestingWe have claimed that a commonsense questionanswering method should assign close scores tosynonymous choices. To verify that SEQA bettermeets this requirement, we conducted consistencytesting for all the methods on four datasets. Foreach example, the consistency testing of a methodis conducted in three steps: (1) Originally, the ex-ample has one correct and several wrong answerchoices. We randomly sample some choices fromother examples as additional wrong choices. After
Table 3: Consistency testing where the methods rank80 choices to find 4 correct ones for each example. Themetric is the standard deviation of the ranks of 4 correctsynonymous choices averaged over 500 examples.
that, the example will have one correct choice and19 wrong choices. (2) Leverage a commonly usedautomatic translation service, Baidu Translation, totranslate each choice from English into an interme-diate language, and then back-translate it into En-glish. During this process, we employ three inter-mediate languages, Chinese, Spanish, and Russian,because the translation quality of these languagesis better than others. As a result, each choice isaccompanied with three synonymous choices. (3)Use the commonsense question answering methodto calculate the scores for each choice as well as itssynonymous choices, and then sort all the choicesaccording to their scores. Because the scoringscales of these methods are different, we calculatethe standard deviation of the ranks of the correctchoice and its synonymous choices.
Table 3 shows the average standard deviationof the ranks. As expected, the average standarddeviation of SEQA is much lower than any othermethod on all the datasets, confirming that SEQAassigns more similar ranks and closer scores tosynonymous choices. We also observed that MI-QA provided relatively stable predictions comparedwith other baseline methods. A possible explana-tion is that, the normalization term PLM (A) helpsalleviate the influence of lexical perturbations.
4.6 Trends of Accuracy with Answer Length
Answer length is also a type of distracting factorwhich may mislead baseline methods. To exploreto which extent answer lengths affect the perfor-mance of methods, we divided the development setof CosmosQA into four subsets according to thelength of correct choice. Table 4 shows the resultsof SEQA and a robust baseline, MI-QA. Comparedwith MI-QA, SEQA has much more stable perfor-mance as answer lengths vary. The reason is that,SEQA focuses on semantic information so that ithas stronger resistance to such distracting factors.
Table 5: The before-attack (Bef) and after-attack (Aft)accuracy of SEQA with different temperatures.
4.7 Ablation Study
4.7.1 Analysis on TemperatureIn the previous experiments, the temperature T ofSEQA was set to 0.1 by default. To investigatethe influence of T , we varied T in a wide rangefrom 0.05 to 10 and report the results in Table 5.Considering that the temperature varied greatly,the performance of SEQA is relatively stable, in-dicating that SEQA is not so sensitive to the selec-tion of T . Another observation is that, althoughthe four datasets are different in domains and textlength, the trends of performance with temperatureon them are relatively similar, illustrating that thetemperature selected on one task can be generalizedto other tasks.
4.7.2 Analysis on Sample SizeFigure 3 shows the effect of the sample size K onSEQA. For comparison, Figure 3 also includes theresults of baselines in the settings of before- andafter-attack, respectively. Due to the limitation ofspace, the results on the other datasets are shown inAppendix. As expected, the before-attack and after-attack accuracy on SCT increased with the samplesize. In detail, the rapid increase in performanceoccurred when K < 100, and then the improve-ment slowed down when K > 100. Finally, SEQAachieved a stable and relatively high performance.
CGA and Self-Talk also leverage LMs to gen-erate some plausible answers. Different from ourmethod, they use the generated answers to formpart of the question, and then calculate the gener-ative probability of the choice based on the aug-mented question. We also tried different samplesizes for the two methods, and Figure 3 (a) shows
Figure 3: The before-attack (a) and after-attack accu-racy (b) of methods with different sample sizes on SCT.The after-attack accuracy of Pro-A, CGA and Self-Talkis below 5.0%, and thus omitted in (b).
ω(S|A) = 1f(1)
f (cos(hS , hA)) Bef Aftf(x) = I(x > α) 77.2 47.2f(x) = ReLU(x− β) 77.6 45.2f(x) = sigmoid( x
T) 75.6 48.6
f(x) = exp(
xT
)79.4 59.0
Table 6: The before-attack (Bef) and after-attack (Aft)accuracy of SEQA on the test set of COPA with differ-ent definitions of ω(S|A). α, β, T1, T2 are hyperparam-eters tuned on the development set of COPA.
that their accuracy will not stably increase with alarger sample size.
4.7.3 Analysis on ω(S|A)
ω(S|A) in SEQA can be defined in different forms,as long as the three requirements mentioned in Sec-tion 3.2 are met. Besides the default definition, weexplored another three forms of ω(S|A), and theexperiment results on COPA are shown in Table 6.Although the performance varies with ω(S|A), thebefore-attack accuracy of SEQA still outperformedmost of the baselines with any definition of ω(S|A).Moreover, SEQA maintains its obvious advantagein after-attack accuracy, which reflects the inherentrobustness of SEQA.
Table 7: SEQA’s accuracy with different feature ex-tractors and language models on COPA. Avg. GloVemeans the average pooling of the pre-trained word em-beddings (Pennington et al., 2014) over the sentence.
Table 8: Manual evaluation of the quality of voters(generated by GPT-2-xlarge conditioned on questions).Score 3/2/1 correspond to high, middle and low quality,respectively, in terms of grammar and logicality.
4.7.4 Analysis on Pre-trained LanguageModel and Feature Extractor
SEQA has no limit on the selection of the pre-trained language model and the feature extractor.Table 7 shows how the accuracy of SEQA on COPAvaried with the language model and the feature ex-tractor. As expected, more powerful extractor usu-ally led to higher accuracy under the same settingsof language models. Similar conclusion can be ob-tained for the language model. It can be inferredthat, if there are more powerful language modelsor feature extractors in the future, the performanceof SEQA may be further improved.
4.8 Analysis on the Quality of Voters
While the performance of SEQA served as an ex-trinsic evaluation for the quality of the voters (plau-sible answers sampled from PLM (·|Q), describedin Section 3.3), we were also interested in eval-uating it intrinsically. We sampled 125 votersfrom COPA. For each voter, we provided crowd-sourcing workers with the original question, andasked them: 1) whether the voter is grammati-cal, not entirely grammatical but understandable,or completely not understandable, 2) whether thevoter is a reasonable answer to the question, notreasonable but relevant, or completely irrelevant.These evaluation tasks comprehensively examinedthe voters in grammar and logicality. The annota-tion tasks were carried out in Amazon MechanicalTurk, and we aggregated annotations from 3 work-ers using majority vote.
Table 8 shows the results of the human evalua-tion of the voters. Score 3/2/1 correspond to thehigh, middle and low quality, respectively. Accord-ing to the grammar scores, 97.6% of the voters aregrammatical or at least understandable, for whichmost of the voters belong to the natural languagespace. In terms of logicality, 40.8% of the votersare reasonable answers to the questions, which maynot be very satisfying. However, in Section 4.9, wewill show that SEQA makes prediction based ona small part of voters, and hence SEQA is robust
Figure 4: The cumulative proportion of voters favor-ing the correct answer AC or the wrong answer AW
on COPA. Each point (δ, p) means that p% of voterssatisfy |ω(S|AC) − ω(S|AW )| ≥ δ, where S refers toa voter. The area between the two curves equals to thedifference of the semantic scores s(AC |Q)−s(AW |Q).
even though there are some irrelevant voters.
4.9 Voting Weight Distribution
We visualize the cumulative proportion of votersfavoring the correct or the wrong choices (see Fig-ure 4). The curve is averaged over all instances inthe test set of COPA, where we sampled 500 votersfor each instance and set T = 0.1.
From the curves, we can find several prop-erties of voters: (1) The voters favor the cor-rect choices over the wrong choices, where thecurve for correct choices is consistently above thecurve for wrong ones. The area between twocurves shows the difference of semantic scoress(AC |Q) − s(AW |Q), which is a large gap com-pared with the area under the bottom curve. (2)93.5% of voters do not strongly favor any choices(|ω(S|AC) − ω(S|AW )| < 0.05), indicating thatthey are semantically irrelevant to both candidatechoices. However, Table 8 shows that 40.8% ofvoters are logically reasonable, so many voters arereasonable but irrelevant to both answers. It sug-gests that there can be several reasonable answersfor a single question, and the sampled voters arediverse in the semantics. (3) Although there areonly 5.3% of voters strongly favoring the correctchoices, there are much less voters (1.2%) favoringthe wrong ones. It explains why our method is ableto predict the correct answer.
To help understand the relationship between vot-ers and choices, Table 9 provides an instance withvoters and their voting weights to the choices. Weshow four types of voters: favoring the correctchoice, favoring the wrong choice, logically rea-sonable but not favoring either choices, and unrea-sonable and irrelevant to both choices. We can see
3045
Q: The car ran out of gas. What happened as a result?AC : The driver was stranded on the road. (3)AW : The driver picked up a hitchhiker. (8)
ω(Si|AC) voter ω(Si|AW )
0.161 I had to park on a dead end road. 0.0080.008 We picked up a hitchhiker and
she drove us to the diner.0.137
0.013 We stopped at a gas station. 0.0110.018 It was time to hit the road again. 0.010
Table 9: An example of voters as well as their votingweights. AC is the correct choice, while AW is wrong.Si refers to a voter.
that the last two types of voters can hardly affect themethod’s prediction, because their voting weightsare much smaller than the first two types of voters.
5 Conclusion
We present a semantic-based question answeringmethod, SEQA, which can answer commonsensequestions more accurately and robustly in an unsu-pervised setting. Instead of directly scoring eachanswer choice, our method focuses on the prob-ability of observing a choice’s semantics. In theview of voting, SEQA first generates some plausi-ble answers (voters) and then utilizes them to votefor the correct choice by considering the seman-tic similarity between each choice and each voter.Experiment results show that SEQA achieves thebest performance on four datasets, and it is remark-ably more robust than all the baselines when beingattacked by TextFooler.
Acknowledgments
This work was partly supported by the NSFCprojects (Key project with No. 61936010 and reg-ular project with No. 61876096). This work wasalso supported by the Guoqiang Institute of Ts-inghua University, with Grant No. 2019GQG1 and2020GQG0005. This work was also supported byHuawei Noah’s Ark Lab.
ReferencesMostafa Abdou, Vinit Ravishankar, Maria Barrett,
Yonatan Belinkov, Desmond Elliott, and AndersSøgaard. 2020. The sensitivity of language modelsand humans to winograd schema perturbations. InACL, pages 7590–7604.
Pratyay Banerjee and Chitta Baral. 2020. Self-supervised knowledge triplet learning for zero-shotquestion answering. CoRR.
Antoine Bosselut and Yejin Choi. 2019. Dynamicknowledge graph construction for zero-shot com-monsense question answering. CoRR.
Antoine Bosselut, Hannah Rashkin, Maarten Sap, Chai-tanya Malaviya, Asli Celikyilmaz, and Yejin Choi.2019. COMET: commonsense transformers for au-tomatic knowledge graph construction. In ACL,pages 4762–4779.
Joe Davison, Joshua Feldman, and Alexander M. Rush.2019. Commonsense knowledge mining from pre-trained models. In EMNLP-IJCNLP, pages 1173–1178.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, andKristina Toutanova. 2019. BERT: pre-training ofdeep bidirectional transformers for language under-standing. In NAACL-HLT, pages 4171–4186.
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, andYejin Choi. 2020. The curious case of neural textdegeneration. In ICLR.
Lifu Huang, Ronan Le Bras, Chandra Bhagavatula, andYejin Choi. 2019. Cosmos QA: machine readingcomprehension with contextual commonsense rea-soning. In EMNLP, pages 2391–2401.
Di Jin, Zhijing Jin, Joey Tianyi Zhou, and PeterSzolovits. 2020. Is BERT really robust? A strongbaseline for natural language attack on text classifi-cation and entailment. In AAAI, pages 8018–8025.
Daniel Khashabi, Sewon Min, Tushar Khot, AshishSabharwal, Oyvind Tafjord, Peter Clark, and Han-naneh Hajishirzi. 2020. Unifiedqa: Crossing formatboundaries with a single QA system. In Findings,EMNLP, pages 1896–1907.
Vid Kocijan, Ana-Maria Cretu, Oana-Maria Camburu,Yordan Yordanov, and Thomas Lukasiewicz. 2019.A surprisingly robust trick for the winograd schemachallenge. In ACL, pages 4837–4842.
Bill Yuchen Lin, Xinyue Chen, Jamin Chen, and Xi-ang Ren. 2019. Kagnet: Knowledge-aware graphnetworks for commonsense reasoning. In EMNLP-IJCNLP, pages 2829–2839.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man-dar Joshi, Danqi Chen, Omer Levy, Mike Lewis,Luke Zettlemoyer, and Veselin Stoyanov. 2019.Roberta: A robustly optimized BERT pretraining ap-proach. CoRR.
Kaixin Ma, Filip Ilievski, Jonathan Francis, YonatanBisk, Eric Nyberg, and Alessandro Oltramari. 2020.Knowledge-driven data construction for zero-shotevaluation in commonsense question answering.CoRR.
Nasrin Mostafazadeh, Nathanael Chambers, XiaodongHe, Devi Parikh, Dhruv Batra, Lucy Vanderwende,Pushmeet Kohli, and James Allen. 2016. A cor-pus and cloze evaluation for deeper understandingof commonsense stories. In NAACL.
Jeffrey Pennington, Richard Socher, and Christopher D.Manning. 2014. Glove: Global vectors for word rep-resentation. In EMNLP, pages 1532–1543.
Fabio Petroni, Tim Rocktaschel, Sebastian Riedel,Patrick S. H. Lewis, Anton Bakhtin, Yuxiang Wu,and Alexander H. Miller. 2019. Language modelsas knowledge bases? In EMNLP-IJCNLP, pages2463–2473.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan,Dario Amodei, and Ilya Sutskever. 2019. Languagemodels are unsupervised multitask learners.
Nazneen Fatema Rajani, Bryan McCann, CaimingXiong, and Richard Socher. 2019. Explain yourself!leveraging language models for commonsense rea-soning. In ACL, pages 4932–4942.
Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. In EMNLP-IJCNLP, pages 3980–3990.
Melissa Roemmele, Cosmin Adrian Bejan, and An-drew S. Gordon. 2011. Choice of plausible alterna-tives: An evaluation of commonsense causal reason-ing. In AAAI.
Maarten Sap, Ronan Le Bras, Emily Allaway, Chan-dra Bhagavatula, Nicholas Lourie, Hannah Rashkin,Brendan Roof, Noah A. Smith, and Yejin Choi.2019a. ATOMIC: an atlas of machine commonsensefor if-then reasoning. In AAAI, pages 3027–3035.
Maarten Sap, Hannah Rashkin, Derek Chen, Ronan LeBras, and Yejin Choi. 2019b. Social iqa: Com-monsense reasoning about social interactions. InEMNLP, pages 4462–4472.
Vered Shwartz, Peter West, Ronan Le Bras, ChandraBhagavatula, and Yejin Choi. 2020. Unsupervisedcommonsense question answering with self-talk. InEMNLP.
Robyn Speer, Joshua Chin, and Catherine Havasi. 2017.Conceptnet 5.5: An open multilingual graph of gen-eral knowledge. In AAAI, pages 4444–4451.
Alon Talmor, Jonathan Herzig, Nicholas Lourie, andJonathan Berant. 2019. Commonsenseqa: A ques-tion answering challenge targeting commonsenseknowledge. In NAACL-HLT, pages 4149–4158.
Alexandre Tamborrino, Nicola Pellicano, Baptiste Pan-nier, Pascal Voitot, and Louise Naudin. 2020. Pre-training is (almost) all you need: An applicationto commonsense reasoning. In ACL, pages 3878–3887.
Trieu H. Trinh and Quoc V. Le. 2018. A simple methodfor commonsense reasoning. CoRR.
Peifeng Wang, Nanyun Peng, Filip Ilievski, Pedro A.Szekely, and Xiang Ren. 2020. Connecting the dots:A knowledgeable path generator for commonsensequestion answering. In Findings, EMNLP, pages4129–4140.
Shuohang Wang, Sheng Zhang, Yelong Shen, Xi-aodong Liu, Jingjing Liu, Jianfeng Gao, and JingJiang. 2019. Unsupervised deep structured seman-tic models for commonsense reasoning. In NAACL-HLT, pages 882–891.
Table 10: Statistic information of each dataset. Due to the removal of the choice “None of the above”, each instanceof CosmosQA may have 3 or 4 answer choices.
A Datasets
The four datasets used in this work are multiple-choice commonsense question answering tasks.
COPA2 (Roemmele et al., 2011) evaluates theability of causal reasoning about a certain event,which is expressed in a simple sentence. Each ques-tion is accompanied with two candidate choices.
StoryClozeTest (SCT)3 (Mostafazadeh et al.,2016) requires models to select the reasonable storyending, from two alternatives, conditioned on a de-scription about the story context.
SocialIQA4 (Sap et al., 2019b) evaluates the rea-soning ability on social events. In each example,the question describes a social event and asks mod-els to make some inferences based on the event,such as its cause or effect.
CosmosQA5 (Huang et al., 2019) is a read-ing comprehension task. Different from the threedatasets above, the examples of CosmosQA havelong and complex contexts. The original datasetcontains a type of choices “None of the above”to test whether models can identify unanswerablequestions. This is not the focus of our work, so weremoved such choices.
For COPA, we reported the results on its testset. As the test sets of SCT, SocialIQA and Cos-mosQA are hidden, for convenience of analysis,we reported the experiment results on their devel-opment sets. See Table 10 for statistic informationof each dataset.
B Templates for Rewriting Questions
We use the same templates for our method and allthe baselines. Note that the templates for rewritingquestions is not the focus of this paper, and we in-herit the templates from previous work if available.
Tamborrino et al. (2020) provides templates forCOPA (Table 11) and Shwartz et al. (2020) pro-vides templates for SocialIQA (Table 12). Sincethe instances in SCT have no questions, SCT doesnot need templates. There is no related work dis-cussing templates for CosmosQA, so we designsome templates by ourselves (Table 13). Sourcecode for rewriting questions and SEQA will bemade publicly available.
C Selection of Pre-trained Models
For each method, we tried to adopt different pre-trained models and find the pre-trained models thatmaximized the accuracy on the development set ofeach dataset. Table 14 shows the set of candidatepre-trained models for each method, with the se-lected models in bold. Because of the nature of Pro-Q, it can only use bidirectional language models,so we only evaluated Pro-Q with RoBERTa-largeand SentenceRoBERTa-large.
As shown in Table 14, for each method exceptCGA, the best selection of pre-trained models isconsistent on all the datasets. CGA achieved itsbest performance with COMET on SocialIQA andwith GPT2-xlarge on the other datasets.
D Hyperparameter Search
For SEQA, we only tuned the temperature T . To bemore specific, we selected T from five candidatevalues according to the accuracy on the develop-ment set of COPA. Table 15 shows that SEQA withT = 0.1 achieved the best performance on the de-velopment set of COPA. And then we evaluatedSEQA with T = 0.1 on the test set of COPA aswell as the development sets of SCT, SocialIQAand CosmosQA.
E Analysis on Sample Size
Figure 5,6,7 shows the effect of the sample sizeK on SEQA. For comparison, these figures alsoinclude the results of baselines in the settings ofbefore- and after-attack, respectively. On the over-all trend, the performance of SEQA improved as
Original Question RewriteWhat was the cause of this? becauseWhat happened as a result? soOriginal Example RewriteI saw my breath when I exhaled. What wasthe cause of this? The weather was chilly.
I saw my breath when I exhaled because theweather was chilly.
Table 11: Templates and a rewritten example of COPA. The templates are inherited from Tamborrino et al. (2020).
Original Question Rewrite 1 Rewrite 2What will [SUBJ] want to do next? As a result, [SUBJ] wanted to <xwant>How would [SUBJ] feel as a result? As a result, [SUBJ] felt <xeffect>What will [SUBJ] do next? [SUBJ] then <xreact>How would you describe [SUBJ]? [SUBJ] is seen as <xattr>Why did [SUBJ] do that? Before, [SUBJ] wanted <xintent>What does [SUBJ] need to do before? Before, [SUBJ] needed to <xneed>Original Example Rewrite 1 Rewrite 2Sydney went trick or treating and theothers joined him happily. What willOthers want to do next? get candy
Sydney went trick or treating and theothers joined him happily. As a result,Others wanted to get candy.
Sydney went trick or treating and theothers joined him happily. <xwant>get candy.
Table 12: Some templates and a rewritten example of SocialIQA. [SUBJ] refers to a subject. There are two groupsof templates, Rewrite1 for GPT-2 and Rewrite2 for COMET (Bosselut et al., 2019). The relations in Rewrite2are defined in Sap et al. (2019a) and used for training COMET. These templates are inherited from Shwartz et al.(2020). More details can be found in Shwartz et al. (2020) and https://github.com/vered1986/self talk.
the sample size increased. Another observation isthat a smaller sample size can already make SEQAoutperform most baseline methods.
Figure 5: The before-attack (a) and after-attack ac-curacy (b) of methods with different sample sizes onCOPA. The after-attack accuracy of Pro-A, CGA andSelf-Talk is below 10.0%, and thus omitted in (b).
Figure 6: The before-attack (a) and after-attack accu-racy (b) of methods with different sample sizes on So-cialIQA. The after-attack accuracy of Pro-A, CGA andSelf-Talk is below 20.0%, and thus omitted in (b).
Figure 7: The before-attack (a) and after-attack accu-racy (b) of methods with different sample sizes on Cos-mosQA. The after-attack accuracy of Pro-A, CGA andSelf-Talk is below 2.0%, and thus omitted in (b).
Original Question RewriteWhy [SENTENCE] [CLAUSE] ? [CLAUSE] [SENTENCE] becauseWhat [NOUN] [SENTENCE] [CLAUSE] ? [CLAUSE] the [NOUN] [SENTENCE] is thatWhat [SENTENCE] [CLAUSE] ? [CLAUSE] it [SENTENCE] thatOriginal Example Rewrite... He was conscious but seemed dazed and prob-ably intoxicated . Nearby there was a youngman dialing his cell phone . What may hap-pen after the young man makes his call ? Anambulance would likely come to the scene .
... He was conscious but seemed dazed and prob-ably intoxicated . Nearby there was a youngman dialing his cell phone . After the youngman makes his call , it may happen that an am-bulance would likely come to the scene .
Table 13: Templates and a rewritten example of CosmosQA. [NOUN], [SENTENCE] and [CLAUSE] refer to anoun, a sentence fragment and an adverbial clause, respectively.
Method Set of Candidate Pre-trained ModelsPro-A LM as QA model: (GPT2-xlarge, COMET, RoBERTa-large, SentenceRoBERTa-large)Pro-Q LM as QA model: (RoBERTa-large, SentenceRoBERTa-large)MI-QA LM as QA model: (GPT2-xlarge, COMET, RoBERTa-large, SentenceRoBERTa-large)
Self-talkLM as generator: (GPT2-xlarge, COMET)LM as QA model: (GPT2-xlarge, COMET, RoBERTa-large, SentenceRoBERTa-large)
CGA LM as QA model and generator: (GPT2-xlarge, COMET)
SEQALM as generator: (GPT2-xlarge, COMET)Feature Extractor: SentenceRoBERTa-large
Table 14: The set of candidate pre-trained models. The selected pre-trained models for each method are markedin bold. Note that CGA achieved its best performance with COMET on SocialIQA and with GPT2-xlarge on theother datasets.
T Dev Test10 70.0 75.61 70.4 76.40.2 71.8 77.00.1 75.4 79.40.05 74.4 80.2
Table 15: Hyperparameter Search of SEQA. The tem-perature is selected according to the accuracy on thedevelopment set of COPA.
