Task Description
Relation extraction is the task of extracting semantic relations between entities in a sentence. Generally, semantic relations connect two or more entities mentioned in the text. These relations can be of different semantic types.
Relation Extraction is a key component for building knowledge graphs, and it is of crucial significance to natural language processing applications such as structured search, sentiment analysis, question answering, and summarization.
Sub-Task A: Mono-Lingual Relation Extraction
In this task the training data is in Persian and the aim is to use this data set for training. This task is the same as SemEval-2010 task 8.
Sub-Task B: Bi-Lingual English-Persian Relation Extraction
In this subtask, the training data is a parallel English-Persian data set and the aim is to employ the bi-lingual data in the training model.
Both sub-tasks are classification, however, other learning methods such as distant supervision, and bootstrapping are also appreciated.
Input: A sentence with two tagged entities.
Output: The relation of two entities and direction of it.
Examples:
| The format of the data is illustrated by the following examples: Example 1: “این <e1>ترکیدگی </e1>ناشی از <e2>فشار </e2>ضربه قوچ است .” Cause-Effect(e2,e1) Example 2: “این کتاب در مورد چگونگی استفاده از <e1>بمب </e1>در <e2>دیپلماسی </e2>است .” Instrument-Agency(e1,e2) |
Data and tools
PERLEX http://farsbase.net/PERLEX.html is the translation of SemEval-2010 task 8 data set. Format of both data the same. Each sample is presented in three lines. The first line contains the Persian sentence itself inside quotation marks, preceded by a numerical identifier. Two other lines contains relation type and comments.
Each sample is annotated with four pieces of information:
(a) Two entity mentions in the sentence are tagged as e1 and e2 — the numbering simply reflects the order of the mentions in the sentence. The span of the tag corresponds to the “base NP” which may be smaller than the full NP denoting the entity.
(b) If one of the semantic relations 1-9 holds between e1 and e2, the sentence is labelled with this relation’s name and the order in which the relation arguments are filled by e1 and e2. For example, Cause-Effect(e1,e2) means that e1 is the Cause and e2 is the Effect, whereas Cause-Effect(e2,e1) means that e2 is the Cause and e1 is the Effect. If none of the relations 1-9 holds, the sentence is labelled “Other”. In total, then, 19 labels are possible.
(c) A comment may be provided to explain why the annotators chose a given label. Comments are intended for human readers and should be ignored by automatic systems participating in the task.
(d) An original is the English format of the example that you can use for bilingual models.
Note that the test release will be formatted similarly, but without lines for the relation label and for the comment.
There are nine classes of relation in this data set:
(1) Cause-Effect
(2) Instrument-Agency
(3) Product-Producer
(4) Content-Container
(5) Entity-Origin
(6) Entity-Destination
(7) Component-Whole
(8) Member-Collection
(9) Message-Topic
Please notice that PERLEX contains both the training and test data, as the SemEval-2010 task 8. However, the evaluations are performed using a test data rather than PERLEX test data.
The task evaluation baseline(s)
Sub-Task A:
We use the baseline method presented in [https://arxiv.org/abs/2005.06588] as the baseline of mono-lingual task.
Sub-Task B:
A CNN method trained with bi-lingual word embeddings.
The evaluation procedure The evaluations are performed using the standard script presented at SemEval 2010-task 8 for this task.
The predictions of the system should be in the following format:
1 Content-Container(e2,e1)
2 Other
3 Entity-Destination(e1,e2)
…
The official evaluation measures are accuracy over all examples and macro-averaged F-score over the 18 relation labels apart from Other. To calculate the F-score, 18 individual F-scores — one for each relation label — are calculated in the standard way and the average of these scores is taken. For each relation Rel, each sentence labelled Rel in the gold standard will count as either a true positive or a false negative, depending on whether it was correctly labelled by the system; each sentence labelled with a different relation or with Other will count as a true negative or false positive.
Data release:
To obtain PERLEX, please use the following like:
http://farsbase.net/PERLEX.html#
or contact:
nsr.taghizadeh@ut.ac.ir
Task Organizers
If you have any queries regarding this task, please refer to the task organizers:
Heshaam Faili, hfaili@ut.ac.ir
Nasrin Taghizadeh, nsr.taghizadeh@ut.ac.ir
Ali Ebrahimi, ali96ebrahimi@ut.ac.ir
