Ongoing series of evaluations of computational semantic analysis systems
SemEval
Academics
Disciplines:
Natural language processing Computational linguistics Semantics
Umbrella Organization:
ACL-SIGLEX hhdgdhc
Workshop Overview
Founded:
1998 (Senseval)
Latest:
SemEval-2015 NAACL @ Denver, USA
Upcoming:
SemEval-2018
History
Senseval-1
1998 @ Sussex
Senseval-2
2001 @ Toulouse
Senseval-3
2004 @ Barcelona
SemEval-2007
2007 @ Prague
SemEval-2010
2010 @ Uppsala
SemEval-2012
2012 @ Montreal
SemEval-2013
2013 @ Atlanta
SemEval-2014
2014 @ Dublin
SemEval-2015
2015 @ Denver
SemEval-2016
2016 @ San Diego
v
t
e
SemEval (Semantic Evaluation) is an ongoing series of evaluations of computational semantic analysis systems; it evolved from the Senseval word sense evaluation series. The evaluations are intended to explore the nature of meaning in language. While meaning is intuitive to humans, transferring those intuitions to computational analysis has proved elusive.
This series of evaluations is providing a mechanism to characterize in more precise terms exactly what is necessary to compute in meaning. As such, the evaluations provide an emergent mechanism to identify the problems and solutions for computations with meaning. These exercises have evolved to articulate more of the dimensions that are involved in our use of language. They began with apparently simple attempts to identify word senses computationally. They have evolved to investigate the interrelationships among the elements in a sentence (e.g., semantic role labeling), relations between sentences (e.g., coreference), and the nature of what we are saying (semantic relations and sentiment analysis).
The purpose of the SemEval and Senseval exercises is to evaluate semantic analysis systems. "Semantic Analysis" refers to a formal analysis of meaning, and "computational" refer to approaches that in principle support effective implementation.[1]
The first three evaluations, Senseval-1 through Senseval-3, were focused on word sense disambiguation (WSD), each time growing in the number of languages offered in the tasks and in the number of participating teams. Beginning with the fourth workshop, SemEval-2007 (SemEval-1), the nature of the tasks evolved to include semantic analysis tasks outside of word sense disambiguation.[2]
Triggered by the conception of the *SEM conference, the SemEval community had decided to hold the evaluation workshops yearly in association with the *SEM conference. It was also the decision that not every evaluation task will be run every year, e.g. none of the WSD tasks were included in the SemEval-2012 workshop.
^Blackburn, P., and Bos, J. (2005), Representation and Inference for Natural Language: A First Course in Computational Semantics, CSLI Publications. ISBN 1-57586-496-7.
^Navigli, R (2009). "Word sense disambiguation". ACM Computing Surveys. 41 (2): 1–69. doi:10.1145/1459352.1459355. S2CID 461624.
June). Semeval-2013 task 12: Multilingual word sense disambiguation. In Proceedings of the 7th International Workshop on Semantic Evaluation (SemEval 2013)...
D. McCarthy, R. Navigli. SemEval-2007 Task 10: English Lexical Substitution Task. Proc. of Semeval-2007 Workshop (SEMEVAL), in the 45th Annual Meeting...
neural models—and are under discussion at established workshops such as SemEval. Distributional semantic models have been applied successfully to the following...
(2017-08-01). "SemEval-2017 Task 4: Sentiment Analysis in Twitter". Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017). Workshop...
Computational Semantics and Semantic Evaluation (SemEval) Joint Conference on Lexical and Computational Semantics (*SEM) Workshop on Statistical Machine Translation...
Natural-language understanding Semantic compression Semantic parsing Semantic Web SemEval WordNet Blackburn, P., and Bos, J. (2005), Representation and Inference...
word embeddings and regularly participates in evaluations such as SemEval. At SemEval 2017, Luminoso participated in Task 2, measuring the semantic similarity...
and clinical text mining. The TempEval task series sets a shared temporal annotation task, and has run at SemEval three times, attracting system entries...
Proceedings of the Fourth International Workshop on Semantic Evaluations (SemEval-2007). Kim, S. M.; Hovy, E. H. (2006). "Identifying and Analyzing Judgment...
914–920, Oct. 2013. I. Segura-Bedmar, P. Martínez, and M. Herrero-Zazo, “SemEval-2013 Task 9: Extraction of drug-drug interactions from biomedical texts...
Computational problems intended to be solved with computer programs United States SemEval Association for Computational Linguistics SIGLEX Evaluations of computational...
scientific journals. Semeval-2012 task 6: A pilot on semantic textual similarity. E. Agirre, D. Cer, M. Diab, A. Gonzalez-Agirre. *SEM 2012: The First Joint...
Global Information