[ad_1]
Lately, pre-trained language fashions, akin to BERT and GPT-3, have seen widespread use in pure language processing (NLP). By coaching on giant volumes of textual content, language fashions purchase broad data in regards to the world, reaching robust efficiency on numerous NLP benchmarks. These fashions, nonetheless, are sometimes opaque in that it might not be clear why they carry out so nicely, which limits additional hypothesis-driven enchancment of the fashions. Therefore, a brand new line of scientific inquiry has arisen: what linguistic data is contained in these fashions?
Whereas there are various kinds of linguistic data that one might need to examine, a subject that gives a robust foundation for evaluation is the topic–verb settlement grammar rule in English, which requires that the grammatical quantity of a verb agree with that of the topic. For instance, the sentence “The canines run.” is grammatical as a result of “canines” and “run” are each plural, however “The canines runs.” is ungrammatical as a result of “runs” is a singular verb.
One framework for assessing the linguistic data of a language mannequin is focused syntactic analysis (TSE), wherein minimally totally different pairs of sentences, one grammatical and one ungrammatical, are proven to a mannequin, and the mannequin should decide which one is grammatical. TSE can be utilized to check data of the English topic–verb settlement rule by having the mannequin choose between two variations of the identical sentence: one the place a specific verb is written in its singular type, and the opposite wherein the verb is written in its plural type.
With the above context, in “Frequency Results on Syntactic Rule-Studying in Transformers”, printed at EMNLP 2021, we investigated how a BERT mannequin’s capability to accurately apply the English topic–verb settlement rule is affected by the variety of occasions the phrases are seen by the mannequin throughout pre-training. To check particular situations, we pre-trained BERT fashions from scratch utilizing fastidiously managed datasets. We discovered that BERT achieves good efficiency on topic–verb pairs that don’t seem collectively within the pre-training information, which signifies that it does be taught to use topic–verb settlement. Nevertheless, the mannequin tends to foretell the inaccurate type when it’s far more frequent than the proper type, indicating that BERT doesn’t deal with grammatical settlement as a rule that have to be adopted. These outcomes assist us to higher perceive the strengths and limitations of pre-trained language fashions.
Prior Work
Earlier work used TSE to measure English topic–verb settlement capability in a BERT mannequin. On this setup, BERT performs a fill-in-the-blank activity (e.g., “the canine _ throughout the park”) by assigning chances to each the singular and plural types of a given verb (e.g., “runs” and “run”). If the mannequin has accurately discovered to use the topic–verb settlement rule, then it ought to constantly assign increased chances to the verb kinds that make the sentences grammatically appropriate.
This earlier work evaluated BERT utilizing each pure sentences (drawn from Wikipedia) and nonce sentences, that are artificially constructed to be grammatically legitimate however semantically nonsensical, akin to Noam Chomsky’s well-known instance “colorless inexperienced concepts sleep furiously”. Nonce sentences are helpful when testing syntactic skills as a result of the mannequin can not simply fall again on superficial corpus statistics: for instance, whereas “canines run” is far more frequent than “canines runs”, “canines publish” and “canines publishes” will each be very uncommon, so a mannequin will not be more likely to have merely memorized the truth that certainly one of them is extra seemingly than the opposite.
BERT achieves an accuracy of greater than 80% on nonce sentences (much better than the random-chance baseline of fifty%), which was taken as proof that the mannequin had discovered to use the topic–verb settlement rule. In our paper, we went past this earlier work by pre-training BERT fashions underneath particular information situations, permitting us to dig deeper into these outcomes to see how sure patterns within the pre-training information have an effect on efficiency.
Unseen Topic–Verb Pairs
We first checked out how nicely the mannequin performs on topic–verb pairs that have been seen throughout pre-training, versus examples wherein the topic and verb have been by no means seen collectively in the identical sentence:
 |
| BERT’s error price on pure and nonce analysis sentences, stratified by whether or not a specific topic–verb (SV) pair was seen in the identical sentence throughout coaching or not. BERT’s efficiency on unseen SV pairs is much better than easy heuristics akin to selecting the extra frequent verb or selecting the extra frequent SV pair. |
BERT’s error price will increase barely for unseen topic–verb (SV) pairs, for each pure and nonce analysis sentences, however it’s nonetheless significantly better than naïve heuristics, akin to selecting the verb type that occurred extra usually within the pre-training information or selecting the verb type that occurred extra continuously with the topic noun. This tells us that BERT is not only reflecting again the issues that it sees throughout pre-training: making selections primarily based on extra than simply uncooked frequencies and generalizing to novel topic–verb pairs are indications that the mannequin has discovered to use some underlying rule regarding topic–verb settlement.
Frequency of Verbs
Subsequent, we went past simply seen versus unseen, and examined how the frequency of a phrase impacts BERT’s capability to make use of it accurately with the topic–verb settlement rule. For this research, we selected a set of 60 verbs, after which created a number of variations of the pre-training information, every engineered to include the 60 verbs at a particular frequency, guaranteeing that the singular and plural kinds appeared the identical variety of occasions. We then skilled BERT fashions from these totally different datasets and evaluated them on the topic–verb settlement activity:
 |
| BERT’s capability to comply with the topic–verb settlement rule depends upon the frequency of verbs within the coaching set. |
These outcomes point out that though BERT is ready to mannequin the topic–verb settlement rule, it must see a verb about 100 occasions earlier than it may well reliably use it with the rule.
Relative Frequency Between Verb Varieties
Lastly, we needed to know how the relative frequencies of the singular and plural types of a verb have an effect on BERT’s predictions. For instance, if one type of the verb (e.g., “fight”) appeared within the pre-training information far more continuously than the opposite verb type (e.g., “combats”), then BERT may be extra more likely to assign a excessive likelihood to the extra frequent type, even when it’s grammatically incorrect. To judge this, we once more used the identical 60 verbs, however this time we created manipulated variations of the pre-training information the place the frequency ratio between verb kinds various from 1:1 to 100:1. The determine beneath exhibits BERT’s efficiency for these various ranges of frequency imbalance:
 |
| Because the frequency ratio between verb kinds in coaching information turns into extra imbalanced, BERT’s capability to make use of these verbs grammatically decreases. |
These outcomes present that BERT achieves good accuracy at predicting the proper verb type when the 2 kinds are seen the identical variety of occasions throughout pre-training, however the outcomes turn into worse because the imbalance between the frequencies will increase. This suggests that despite the fact that BERT has discovered apply topic–verb settlement, it doesn’t essentially use it as a “rule”, as a substitute preferring to foretell high-frequency phrases no matter whether or not they violate the topic–verb settlement constraint.
Conclusions
Utilizing TSE to judge the efficiency of BERT reveals its linguistic skills on syntactic duties. Furthermore, learning its syntactic capability in relation to how usually phrases seem within the coaching dataset reveals the ways in which BERT handles competing priorities — it is aware of that topics and verbs ought to agree and that prime frequency phrases are extra seemingly, however doesn’t perceive that settlement is a rule that have to be adopted and that the frequency is simply a desire. We hope this work offers new perception into how language fashions mirror properties of the datasets on which they’re skilled.
Acknowledgements
It was a privilege to collaborate with Tal Linzen and Ellie Pavlick on this venture.
[ad_2]
