About
RepEval 2017: The Second Workshop on Evaluating Vector Space Representations for NLP
Co-located with EMNLP 2017
Description
This workshop deals with the evaluation of general-purpose vector representations for linguistic units (morphemes, words, phrases, sentences, etc). What distinguishes these representations (or embeddings) is that they are not trained with a specific application in mind, but rather to capture broadly useful features of the represented units. Another way to view their usage is through the lens of transfer learning: The embeddings are trained with one objective, but applied on others.
Evaluating general-purpose representation learning systems is fundamentally difficult. They can be trained on a variety of objectives, making simple intrinsic evaluations useless as a means of comparing methods. They are also meant to be applied to a variety of downstream tasks, which will place different demands on them, making no single extrinsic evaluation definitive. The best techniques for evaluating embedding methods in downstream tasks often require investing considerable time and resources in retraining large neural network models, making broad suites of downstream evaluations impractical. In many cases, especially for word-level embeddings, these constraints have led to the rise of dedicated evaluation tasks like similarity and analogy which are not directly related either to training objectives or to downstream tasks. Tasks like these can serve a valuable role in principle, but in practice performance on these tasks has not been highly predictive of downstream task performance.
This workshop aims foster discussion of these issues, and to support the search for high-quality general purpose representation learning techniques for NLP. The workshop will accept submissions through two tracks: a proposal track will showcase submitted proposals for new evaluation techniques, and a shared task will accept submissions of new general purpose sentence representation systems—for which standard evaluations are notably absent—which will be evaluated on a sentence understanding task.
Proceedings Papers
Available in the ACL Anthology
Schedule
All sessions in Skt. Hans Torv
09:00 Opening remarks
09:20 - 09:55 Shared task report
The RepEval 2017 Shared Task: Multi-Genre Natural Language Inference with Sentence Representations
09:55 - 10:30 Yejin Choi (University of Washington)
10:30 - 11:00 Coffee break (set up posters)
11:00 - 11:35 Jakob Uszkoreit (Google Research)
11:35 - 12:10 Kyunghyun Cho (New York University)
12:10 - 12:30 Few minutes madness (Evaluation Proposals)
Traversal-Free Word Vector Evaluation in Analogy Space
Xiaoyin Che, Nico Ring, Willi Raschkowski, Haojin Yang, Christoph Meinel Hasso Plattner Institute
Hypothesis Testing based Intrinsic Evaluation of Word Embeddings
Nishant Gurnani
Jeremy Auguste, Arnaud Rey, Benoit Favre
Playing with Embeddings : Evaluating embeddings for Robot Language Learning through MUD Games
Anmol Gulati and Kumar Krishna Agrawal
Recognizing Textual Entailment in Twitter Using Word Embeddings
Octavia-Maria Şulea
12:30 - 14:00 Lunch (somewhere together if pos)
14:00 - 14:30 Contributed talks (shared task systems)
14:00 - 14:15
Recurrent Neural Network-Based Sentence Encoder with Gated Attention for Natural Language Inference
Qian Chen, Xiaodan Zhu, Zhen-Hua Ling, Si Wei, Hui Jiang, Diana Inkpen
14:15 - 14:30
Shortcut-Stacked Sentence Encoders for Multi-Domain Inference
Yixin Nie and Mohit Bansal
14:30 - 15:30 Posters and discussion
Character-level Intra Attention Network for Natural Language Inference
Han Yang, Marta R. Costa-jussà, José A. R. Fonollosa
Refining Raw Sentence Representations for Textual Entailment Recognition via Attention
Jorge Balazs, Edison Marrese-Taylor, Pablo Loyola, Yutaka Matsuo
LCT-MALTA’s Submission to RepEval 2017 Shared Task
Hoa Trong Vu, Thuong-Hai Pham, Xiaoyu Bai, Marc Tanti, Lonneke van der Plas, Albert Gatt
15:30 - 16:00 Working coffee break
16:00 - 17:30 Presentation of findings and panel discussion
Yejin Choi, Kyunghyun Cho, Jakob Uszkoreit and other great minds if they are up for it…
News
- 3/24: The shared task has launched.
Anti-harassment policy
This workshop follows the ACL anti-harassment policy. We encourage you to contact any or all of the organizers listed below if you witness or experience any harassment or hostile behavior related to this workshop.
Organizers
Sam Bowman
Yoav Goldberg
Felix Hill
Angeliki Lazaridou
Omer Levy
Roi Reichart
Anders Søgaard
Programme Committee
Omri Abend
Mohit Bansal
Jose Camacho Collados
Billy Chiu
Georgiana Dinu
Allyson Ettinger
Sahar Ghannay
Anna Gladkova
Mohit Iyyer
Douwe Kiela
Arne Kohn
Andras Kornai
Tal Linzen
Farhana Liza
Oren Melamud
Dmitrijs Milajevs
Diarmuid O’Seaghdha
Denis Paperno
Ellie Pavlick
Marek Rei
Laura Rimell
Naomi Saphra
Roy Schwartz
Patrice Seyed
Gabriel Stanovsky
Pontus Stenetorp
Karl Stratos
Yulia Tsvetkov
Peter Turney
Ivan Vulic
Torsten Zesch