RepEval 2017: The Second Workshop on Evaluating Vector Space Representations for NLP

Co-located with EMNLP 2017


This workshop deals with the evaluation of general-purpose vector representations for linguistic units (morphemes, words, phrases, sentences, etc). What distinguishes these representations (or embeddings) is that they are not trained with a specific application in mind, but rather to capture broadly useful features of the represented units. Another way to view their usage is through the lens of transfer learning: The embeddings are trained with one objective, but applied on others.

Evaluating general-purpose representation learning systems is fundamentally difficult. They can be trained on a variety of objectives, making simple intrinsic evaluations useless as a means of comparing methods. They are also meant to be applied to a variety of downstream tasks, which will place different demands on them, making no single extrinsic evaluation definitive. The best techniques for evaluating embedding methods in downstream tasks often require investing considerable time and resources in retraining large neural network models, making broad suites of downstream evaluations impractical. In many cases, especially for word-level embeddings, these constraints have led to the rise of dedicated evaluation tasks like similarity and analogy which are not directly related either to training objectives or to downstream tasks. Tasks like these can serve a valuable role in principle, but in practice performance on these tasks has not been highly predictive of downstream task performance.

This workshop aims foster discussion of these issues, and to support the search for high-quality general purpose representation learning techniques for NLP. The workshop will accept submissions through two tracks: a proposal track will showcase submitted proposals for new evaluation techniques, and a shared task will accept submissions of new general purpose sentence representation systems—for which standard evaluations are notably absent—which will be evaluated on a sentence understanding task.

Proceedings Papers

Available in the ACL Anthology


All sessions in Skt. Hans Torv

09:00 Opening remarks

09:20 - 09:55 Shared task report

The RepEval 2017 Shared Task: Multi-Genre Natural Language Inference with Sentence Representations

09:55 - 10:30 Yejin Choi (University of Washington)

10:30 - 11:00 Coffee break (set up posters)

11:00 - 11:35 Jakob Uszkoreit (Google Research)

11:35 - 12:10 Kyunghyun Cho (New York University)

12:10 - 12:30 Few minutes madness (Evaluation Proposals)

Traversal-Free Word Vector Evaluation in Analogy Space

Xiaoyin Che, Nico Ring, Willi Raschkowski, Haojin Yang, Christoph Meinel Hasso Plattner Institute

Hypothesis Testing based Intrinsic Evaluation of Word Embeddings

Nishant Gurnani

Evaluation of word embeddings against cognitive processess: primed reaction times in lexical decision and naming tasks

Jeremy Auguste, Arnaud Rey, Benoit Favre

Playing with Embeddings : Evaluating embeddings for Robot Language Learning through MUD Games

Anmol Gulati and Kumar Krishna Agrawal

Recognizing Textual Entailment in Twitter Using Word Embeddings

Octavia-Maria Şulea

12:30 - 14:00 Lunch (somewhere together if pos)

14:00 - 14:30 Contributed talks (shared task systems)

14:00 - 14:15

Recurrent Neural Network-Based Sentence Encoder with Gated Attention for Natural Language Inference

Qian Chen, Xiaodan Zhu, Zhen-Hua Ling, Si Wei, Hui Jiang, Diana Inkpen

14:15 - 14:30

Shortcut-Stacked Sentence Encoders for Multi-Domain Inference

Yixin Nie and Mohit Bansal

14:30 - 15:30 Posters and discussion

Character-level Intra Attention Network for Natural Language Inference

Han Yang, Marta R. Costa-jussà, José A. R. Fonollosa

Refining Raw Sentence Representations for Textual Entailment Recognition via Attention

Jorge Balazs, Edison Marrese-Taylor, Pablo Loyola, Yutaka Matsuo

LCT-MALTA’s Submission to RepEval 2017 Shared Task

Hoa Trong Vu, Thuong-Hai Pham, Xiaoyu Bai, Marc Tanti, Lonneke van der Plas, Albert Gatt

15:30 - 16:00 Working coffee break

16:00 - 17:30 Presentation of findings and panel discussion

Yejin Choi, Kyunghyun Cho, Jakob Uszkoreit and other great minds if they are up for it…


  • 3/24: The shared task has launched.

Anti-harassment policy

This workshop follows the ACL anti-harassment policy. We encourage you to contact any or all of the organizers listed below if you witness or experience any harassment or hostile behavior related to this workshop.


Sam Bowman
Yoav Goldberg
Felix Hill
Angeliki Lazaridou
Omer Levy
Roi Reichart
Anders Søgaard

Programme Committee

Omri Abend
Mohit Bansal
Jose Camacho Collados
Billy Chiu
Georgiana Dinu
Allyson Ettinger
Sahar Ghannay
Anna Gladkova
Mohit Iyyer
Douwe Kiela
Arne Kohn
Andras Kornai
Tal Linzen
Farhana Liza
Oren Melamud
Dmitrijs Milajevs
Diarmuid O’Seaghdha
Denis Paperno
Ellie Pavlick
Marek Rei
Laura Rimell
Naomi Saphra
Roy Schwartz
Patrice Seyed
Gabriel Stanovsky
Pontus Stenetorp
Karl Stratos
Yulia Tsvetkov
Peter Turney
Ivan Vulic
Torsten Zesch