Preregistering NLP Research

Autor: Emiel Krahmer, Emiel van Miltenburg, Chris van der Lee

Publikováno v: Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 613-623
STARTPAGE=613;ENDPAGE=623;TITLE=Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies
NAACL-HLT
Tilburg University-PURE

Preregistration refers to the practice of specifying what you are going to do, and what you expect to find in your study, before carrying out the study. This practice is increasingly common in medicine and psychology, but is rarely discussed in NLP.

Externí odkaz: https://explore.openaire.eu/search/publication?articleId=doi_dedup___::91875805f5bd8f2b58311d63b0defb6f
http://arxiv.org/abs/2103.06944

The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics

Autor: Thibault Sellam, Vikas Raunak, Cristina Garbacea, Simon Mille, Tatsunori Hashimoto, Miruna-Adriana Clinciu, Ankur P. Parikh, Angelina McMillan-Major, Samira Shaikh, Kaustubh Dhole, Pawan Sasanka Ammanamanchi, Anuoluwapo Aremu, Laura Perez-Beltrachini, Marco Antonio Sobrevilla Cabezudo, Juan Diego Rodriguez, Sashank Santhanam, Bodhisattwa Prasad Majumder, Dhruv Kumar, Esin Durmus, Dipanjan Das, Wei Xu, Wanyu Du, Vitaly Nikolaev, Faisal Ladhak, Moin Nadeem, Chris Chinenye Emezue, Varun Gangal, Diyi Yang, Yangfeng Ji, João Sedoc, Aman Madaan, Shailza Jolly, Saad Mahamood, Karmanya Aggarwal, Emiel van Miltenburg, Antoine Bosselut, Salomey Osei, Jiawei Zhou, Yufang Hou, Harsh Jhamtani, Niranjan Ramesh Rao, Anastasia Shimorina, Shashi Narayan, Akhila Yerukola, Nishant Subramani, Sebastian Gehrmann, Hendrik Strobelt, Khyathi Raghavi Chandu, Yacine Jernite, Tosin P. Adewumi, Mihir Kale, Khyati Mahajan, Andre Niyongabo Rubungo, Pedro Henrique Martins, Ondřej Dušek, Mounica Maddela

Publikováno v: Proceedings of the 1st Workshop on Natural Language Generation, Evaluation, and Metrics (GEM 2021)
Proceedings of the 1st Workshop on Natural Language Generation, Evaluation, and Metrics (GEM 2021), Aug 2021, Online, France. pp.96-120, ⟨10.18653/v1/2021.gem-1.10⟩
Tilburg University-PURE
Proceedings of the 1st Workshop on Natural Language Generation, Evaluation, and Metrics (GEM 2021), 96-120
STARTPAGE=96;ENDPAGE=120;TITLE=Proceedings of the 1st Workshop on Natural Language Generation, Evaluation, and Metrics (GEM 2021)

International audience; We introduce GEM, a living benchmark for natural language Generation (NLG), its Evaluation, and Metrics. Measuring progress in NLG relies on a constantly evolving ecosystem of automated metrics, datasets, and human evaluation

Externí odkaz: https://explore.openaire.eu/search/publication?articleId=doi_dedup___::56549f809a95ad121bc57aac47f790d8
https://doi.org/10.18653/v1/2021.gem-1.10

Gradations of Error Severity in Automatic Image Descriptions

Autor: Emiel van Miltenburg, Wei-Ting Lu, Emiel Krahmer, Albert Gatt, Guanyi Chen, Lin Li, Kees van Deemter

Publikováno v: Proceedings of the 13th International Conference on Natural Language Generation, 398-411
STARTPAGE=398;ENDPAGE=411;TITLE=Proceedings of the 13th International Conference on Natural Language Generation
Tilburg University-PURE

Earlier research has shown that evaluation metrics based on textual similarity (e.g., BLEU, CIDEr, Meteor) do not correlate well with human evaluation scores for automatically generated text. We carried out an experiment with Chinese speakers, where

Externí odkaz: https://explore.openaire.eu/search/publication?articleId=dedup_wf_001::8c879dd8e9639e5413368fb999a6e87c
https://research.tilburguniversity.edu/en/publications/3cef1ba5-5633-417f-8dfa-b9d5f571583a

Best practices for the human evaluation of automatically generated text

Autor: Emiel Krahmer, Albert Gatt, Sander Wubben, Chris van der Lee, Emiel van Miltenburg

Publikováno v: INLG
Tilburg University-PURE
Proceedings of the 12th International Conference on Natural Language Generation, 355-368
STARTPAGE=355;ENDPAGE=368;TITLE=Proceedings of the 12th International Conference on Natural Language Generation

Currently, there is little agreement as to how Natural Language Generation (NLG) systems should be evaluated. While there is some agreement regarding automatic metrics, there is a high degree of variation in the way that human evaluation is carried o

Externí odkaz: https://explore.openaire.eu/search/publication?articleId=doi_dedup___::2060b025c9829068e3da8e9f00a85701
https://doi.org/10.18653/v1/w19-8643

Interactive Exploration of Journalistic Video Footage Through Multimodal Semantic Matching

Autor: Arthur Câmara, William Thong, Pascal Mettes, Emiel van Miltenburg, Jiaojiao Zhao, Shuo Chen, Maurits van der Goes, Daan Odijk, Tanja Crijns, Sarah Ibrahimi, Yunlu Chen, Thomas Mensink, Devanshu Arya

Publikováno v: MM 2019-Proceedings of the 27th ACM International Conference on Multimedia
Proceedings of the 27th ACM International Conference on Multimedia, 2196-2198
STARTPAGE=2196;ENDPAGE=2198;TITLE=Proceedings of the 27th ACM International Conference on Multimedia
ACM Multimedia
MM'19: proceedings of the 27th ACM Conference on Multimedia : October 21-25, 2019, Nice, France, 2196-2198
STARTPAGE=2196;ENDPAGE=2198;TITLE=MM'19

This demo presents a system for journalists to explore video footage for broadcasts. Daily news broadcasts contain multiple news items that consist of many video shots and searching for relevant footage is a labor intensive task. Without the need for

Externí odkaz: https://explore.openaire.eu/search/publication?articleId=doi_dedup___::4c0bfd7ac069933c8779e543e3d5cb3c
https://doi.org/10.1145/3343031.3350597