Elementary Dependency Match (EDM; see Dridan & Oepen, 2011) is a granular evaluation metric based on the so-called 'ltriples' format that can be exported from a [incr tsdb()] profile. These triples are derived from the variable-free reduction of MRSs known as Elementary Dependency Structures (EDS), described by Oepen & Lønning, 2006. EDS describes almost all the semantics contained in an MRS (excepting scopal information), and can be divided into three types of semantic information:

An EDM evaluation is measured over all three types of ED, but other combinations are possible. EDM_NA evaluates predicate names and arguments, and is closest to other metrics such as GR, CCG dependencies etc. The default ouput for the evaluation script shows precision, recall and f-score over each relation separately, as well as typical aggregations.

To use this evaluation (following the original procedure of Dridan, 2007), you first need to set the data up as described below. Alternatively, there is a more recent reference implementation of EDM available as part of mtool, the Swiss Army Knife of Meaning Representation.

Set up

1. Export gold:

2. Export test:

This should produce directories containing one gzipped file per item parsed. The ltriples should look like:

The links (eg. <10:17>) are necessary to the evaluation, and if your output doesn't have them, ask StephanOepen why.

Evaluate

The Perl implementation is available in SVN:

svn co http://svn.delph-in.net/mu/evaluation/EDM/trunk

Usage: cat <goldfilelist>|./edm_eval.pl [-i] [-v] [-p <num>] [-s] <export directory>

To evaluate a profile:

To evaluate a profile, only over files that received a parse:

To evaluate a single item:

To evaluate a specific analysis of a single item:

To examine the errors in a single item:

To produce the files needed for statistical significance testing:

Significance Testing

An implementation of the computationally-intensive randomisation test described in:

Usage:

ElementaryDependencyMatch (last edited 2020-01-14 23:56:36 by StephanOepen)

(The DELPH-IN infrastructure is hosted at the University of Oslo)