Automatic generation of tunable analogy benchmarks for word representations

Sakakini, Tarek J

Permalink

https://hdl.handle.net/2142/92859 Copy

Description

Title

Automatic generation of tunable analogy benchmarks for word representations

Author(s)

Sakakini, Tarek J

Issue Date

2016-07-20

Director of Research (if dissertation) or Advisor (if thesis)

• Viswanath, Pramod
• Bhat, Suma

Department of Study

Electrical & Computer Eng

Discipline

Electrical & Computer Engr

Degree Granting Institution

University of Illinois at Urbana-Champaign

Degree Name

M.S.

Degree Level

Thesis

Keyword(s)

• Natural Language Processing (NLP)
• word representations
• evaluation
• semantics
• morphology

Abstract

We present a method to automatically generate syntactic analogy datasets for the evaluation of word representations in an unsupervised manner. The automatic generation also allows for customization in terms of word-frequencies, syntactic rules, part-of-speech tags and size of the dataset. We show the ability of our method to generate cross-lingual analogy task datasets for languages other than English, where evaluation datasets are limited if not nonexistent, by constructing datasets for French, German, Spanish, Arabic and Hebrew. Our method clusters pairs of words into morphological rules in an unsupervised manner, using which we generate analogy questions for different rules. We show the quality of an automatically generated dataset by checking the correlation of the performance of different word representations on it with the performance of the same representations on the Google analogy dataset. The values exhibited a high correlation of 95%. Moreover, we showcase the benefits of customization through studying the performance of different word representations when varying the frequency of words in the dataset.

Graduation Semester

2016-08

Type of Resource

text

Permalink

http://hdl.handle.net/2142/92859

Copyright and License Information

Copyright 2016 Tarek Sakakini

Owning Collections