Natural Language Processing (NLP) today - like most of Artificial Intelligence (AI) - is much more of an "engineering" discipline than it originally was, when it sought to develop a general theory of human language understanding that not only translates into language technology, but that is also linguistically meaningful and cognitively plausible.
At first glance, this trend seems to be clearly connected to recent rapid development. Such development, in the last ten years, was driven to a large extent by the adoption of deep learning techniques. However, it can be argued that the move towards deep learning has the potential of bringing NLP back to its roots after all. Some recent activities in this direction include:
? Techniques like multi-task learning have been used to integrate cognitive data as supervision in NLP tasks1;
? Pre-training/fine-tuning regimens are potentially interpretable in terms of cognitive mechanisms like general competencies applied to specific tasks2;
? The ability of modern models for 'few-shot' or even 'zero-shot' performance on novel tasks mirrors human performance3;
? Analysis of complex neural network architectures like transformer models has found evidence of unsupervised structure learning that mirrors classical linguistic structures using so-called 'probing studies'4,5.
The last generation of neural network architectures has allowed AI and NLP to make unprecedented progress in developing systems endowed with natural language capabilities. Such systems (e.g., GPT) are typically trained with huge computational infrastructures on large amounts of textual data from which they acquire knowledge thanks to their extraordinary ability to record and generalize the statistical patterns found in data. However, the debate about the human-like semantic abilities that such “juggernaut models” really acquire is still wide open. In fact, despite the figures typically reported to show the success of AI on various benchmarks, other research argues that their semantic competence is still very brittle6,7,8. Thus, an important limitation of current AI research is the lack of attention to the mechanisms behind human language understanding. The latter does not only consist in a brute-force, data-intensive processing of statistical regularities, but it is also governed by complex inferential mechanisms that integrate linguistic information and contextual knowledge coming from different sources and potentially different modalities ("grounding").
We posit that the possibility for new breakthroughs in the study of human and machine intelligence calls for a new alliance between NLP, AI, linguistic and cognitive research. The current computational paradigms can offer new ways to explore human language learning and processing, while linguistic and cognitive research can contribute by highlighting those aspects of human intelligence that systems need to model or incorporate within their architectures.
The current Research Topic aims at fostering this process by discussing perspectives forward for NLP, given the data and learning devices we have at hand and given the conflicting interests of the participating fields. Suitable topics include, but are not limited to:
? What can NLP do for linguistics, and vice versa?
? What can NLP do for cognitive science, and vice versa?
? How does modeling language relate to modeling general intelligence?
? How do we measure short-term long-term success in NLP?
? Is interdisciplinary research the way ahead for NLP? What are hallmarks for successful interdisciplinary research on language?
We invite not only empirical work but also theoretical (methodological) considerations and position papers.
Information for the authors:
- To ensure a quick and high-quality reviewing process, we invite authors to act as reviewers for other submissions to the collection.
- We encourage authors to submit an abstract by June 15th to allow the Guest Editors to assess the relevance of the paper to the collection.
References:
1. M. Barrett, and A. Søgaard. "Reading behavior predicts syntactic categories." Proceedings of the 19th conference on Computational Natural Language Learning. 2015.
2. T. Flesch, et al. "Comparing continual task learning in minds and machines." Proceedings of the National Academy of Sciences. 2018.
3. A. Lazaridou, et al. "Hubness and pollution: Delving into cross-space mapping for zero-shot learning." Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing. 2015.
4. J. Hewitt, and C. D. Manning. "A structural probe for finding syntax in word representations." Proceedings of the Annual Meeting of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2019.
5. I. Tenney, et al. "BERT Rediscovers the Classical NLP Pipeline." Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. 2019.
6. B. M. Lake, and M. Baroni. "Generalization without Systematicity: On the Compositional Skills of Sequence-to-Sequence Recurrent Networks”. Proceedings of the 35th International Conference on Machine Learning. 2018.
7. A. Ravichander, et al. "Probing the Probing Paradigm: Does Probing Accuracy Entail Task Relevance?." arXiv:2005.00719. 2020.
8. E. M. Bender, and A. Koller. "Climbing towards NLU: On meaning, form, and understanding in the age of data." Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. 2020.
Natural Language Processing (NLP) today - like most of Artificial Intelligence (AI) - is much more of an "engineering" discipline than it originally was, when it sought to develop a general theory of human language understanding that not only translates into language technology, but that is also linguistically meaningful and cognitively plausible.
At first glance, this trend seems to be clearly connected to recent rapid development. Such development, in the last ten years, was driven to a large extent by the adoption of deep learning techniques. However, it can be argued that the move towards deep learning has the potential of bringing NLP back to its roots after all. Some recent activities in this direction include:
? Techniques like multi-task learning have been used to integrate cognitive data as supervision in NLP tasks1;
? Pre-training/fine-tuning regimens are potentially interpretable in terms of cognitive mechanisms like general competencies applied to specific tasks2;
? The ability of modern models for 'few-shot' or even 'zero-shot' performance on novel tasks mirrors human performance3;
? Analysis of complex neural network architectures like transformer models has found evidence of unsupervised structure learning that mirrors classical linguistic structures using so-called 'probing studies'4,5.
The last generation of neural network architectures has allowed AI and NLP to make unprecedented progress in developing systems endowed with natural language capabilities. Such systems (e.g., GPT) are typically trained with huge computational infrastructures on large amounts of textual data from which they acquire knowledge thanks to their extraordinary ability to record and generalize the statistical patterns found in data. However, the debate about the human-like semantic abilities that such “juggernaut models” really acquire is still wide open. In fact, despite the figures typically reported to show the success of AI on various benchmarks, other research argues that their semantic competence is still very brittle6,7,8. Thus, an important limitation of current AI research is the lack of attention to the mechanisms behind human language understanding. The latter does not only consist in a brute-force, data-intensive processing of statistical regularities, but it is also governed by complex inferential mechanisms that integrate linguistic information and contextual knowledge coming from different sources and potentially different modalities ("grounding").
We posit that the possibility for new breakthroughs in the study of human and machine intelligence calls for a new alliance between NLP, AI, linguistic and cognitive research. The current computational paradigms can offer new ways to explore human language learning and processing, while linguistic and cognitive research can contribute by highlighting those aspects of human intelligence that systems need to model or incorporate within their architectures.
The current Research Topic aims at fostering this process by discussing perspectives forward for NLP, given the data and learning devices we have at hand and given the conflicting interests of the participating fields. Suitable topics include, but are not limited to:
? What can NLP do for linguistics, and vice versa?
? What can NLP do for cognitive science, and vice versa?
? How does modeling language relate to modeling general intelligence?
? How do we measure short-term long-term success in NLP?
? Is interdisciplinary research the way ahead for NLP? What are hallmarks for successful interdisciplinary research on language?
We invite not only empirical work but also theoretical (methodological) considerations and position papers.
Information for the authors:
- To ensure a quick and high-quality reviewing process, we invite authors to act as reviewers for other submissions to the collection.
- We encourage authors to submit an abstract by June 15th to allow the Guest Editors to assess the relevance of the paper to the collection.
References:
1. M. Barrett, and A. Søgaard. "Reading behavior predicts syntactic categories." Proceedings of the 19th conference on Computational Natural Language Learning. 2015.
2. T. Flesch, et al. "Comparing continual task learning in minds and machines." Proceedings of the National Academy of Sciences. 2018.
3. A. Lazaridou, et al. "Hubness and pollution: Delving into cross-space mapping for zero-shot learning." Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing. 2015.
4. J. Hewitt, and C. D. Manning. "A structural probe for finding syntax in word representations." Proceedings of the Annual Meeting of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2019.
5. I. Tenney, et al. "BERT Rediscovers the Classical NLP Pipeline." Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. 2019.
6. B. M. Lake, and M. Baroni. "Generalization without Systematicity: On the Compositional Skills of Sequence-to-Sequence Recurrent Networks”. Proceedings of the 35th International Conference on Machine Learning. 2018.
7. A. Ravichander, et al. "Probing the Probing Paradigm: Does Probing Accuracy Entail Task Relevance?." arXiv:2005.00719. 2020.
8. E. M. Bender, and A. Koller. "Climbing towards NLU: On meaning, form, and understanding in the age of data." Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. 2020.