The ability to generalise is one of the primary desiderata of natural language processing (NLP).
Yet, how "good generalisation" should be defined and evaluated is not well understood,
nor are there common standards to evaluate generalisation.
As a consequence, newly proposed models are usually not systematically tested for their ability to generalise.
's mission is to make state-of-the-art generalisation testing the new status-quo in NLP.
As a first step, we present a generalisation taxonomy
, describing the underlying building blocks
of generalisation in NLP. We use the taxonomy to do an elaborate review of over 400 generalisation papers
and we make recommendations
for promising areas for the future.
You can find all this information in our taxonomy paper
On this website, you can learn about the taxonomy
you can visually explore our results
and get citations from our analysis
or contribute papers
that we will periodically add to our review.
|December 6, 2022
||GenBench presentation by Karim Lasri @ ENS
|November 14, 2022
||GenBench presentation by Christos Christodoulopoulos @Amazon
|November 10, 2022
||We are working on a *ACL workshop proposal, express your interest in our twitter poll, stay tuned for more info.
|November 2, 2022
||GenBench presentation by Dieuwke Hupkes @Deep Learning & AI Talks hosted by Qualcomm Amsterdam
|October 7, 2022
||We are live! Read a summary tweet about our taxonomy paper, browse our website to explore the results, and stay tuned!