RepEval 2017: The Second Workshop on Evaluating Vector Space Representations for NLP
Co-located with EMNLP 2017
This workshop deals with the evaluation of general-purpose vector representations for linguistic units (morphemes, words, phrases, sentences, etc). What distinguishes these representations (or embeddings) is that they are not trained with a specific application in mind, but rather to capture broadly useful features of the represented units. Another way to view their usage is through the lens of transfer learning: The embeddings are trained with one objective, but applied on others.
Evaluating general-purpose representation learning systems is fundamentally difficult. They can be trained on a variety of objectives, making simple intrinsic evaluations useless as a means of comparing methods. They are also meant to be applied to a variety of downstream tasks, which will place different demands on them, making no single extrinsic evaluation definitive. The best techniques for evaluating embedding methods in downstream tasks often require investing considerable time and resources in retraining large neural network models, making broad suites of downstream evaluations impractical. In many cases, especially for word-level embeddings, these constraints have led to the rise of dedicated evaluation tasks like similarity and analogy which are not directly related either to training objectives or to downstream tasks. Tasks like these can serve a valuable role in principle, but in practice performance on these tasks has not been highly predictive of downstream task performance.
This workshop aims foster discussion of these issues, and to support the search for high-quality general purpose representation learning techniques for NLP. The workshop will accept submissions through two tracks: a proposal track will showcase submitted proposals for new evaluation techniques, and a shared task will accept submissions of new general purpose sentence representation systems—for which standard evaluations are notably absent—which will be evaluated on a sentence understanding task.
All sessions in Skt. Hans Torv
09:00 Opening remarks
09:20 - 09:55 Shared task report
09:55 - 10:30 Yejin Choi (University of Washington)
10:30 - 11:00 Coffee break (set up posters)
11:00 - 11:35 Jakob Uszkoreit (Google Research)
11:35 - 12:10 Kyunghyun Cho (New York University)
12:10 - 12:30 Few minutes madness (Evaluation Proposals)
Xiaoyin Che, Nico Ring, Willi Raschkowski, Haojin Yang, Christoph Meinel Hasso Plattner Institute
Jeremy Auguste, Arnaud Rey, Benoit Favre
Anmol Gulati and Kumar Krishna Agrawal
12:30 - 14:00 Lunch (somewhere together if pos)
14:00 - 14:30 Contributed talks (shared task systems)
14:00 - 14:15
Qian Chen, Xiaodan Zhu, Zhen-Hua Ling, Si Wei, Hui Jiang, Diana Inkpen
14:15 - 14:30
Yixin Nie and Mohit Bansal
14:30 - 15:30 Posters and discussion
Han Yang, Marta R. Costa-jussà, José A. R. Fonollosa
Jorge Balazs, Edison Marrese-Taylor, Pablo Loyola, Yutaka Matsuo
Hoa Trong Vu, Thuong-Hai Pham, Xiaoyu Bai, Marc Tanti, Lonneke van der Plas, Albert Gatt
15:30 - 16:00 Working coffee break
16:00 - 17:30 Presentation of findings and panel discussion
Yejin Choi, Kyunghyun Cho, Jakob Uszkoreit and other great minds if they are up for it…
- 3/24: The shared task has launched.
This workshop follows the ACL anti-harassment policy. We encourage you to contact any or all of the organizers listed below if you witness or experience any harassment or hostile behavior related to this workshop.
Jose Camacho Collados