SC corpus sense annotation alignment

SC corpus has now been automatically aligned to the SemCor sense annotations. The alignment process found realpred or gpred matches for 96.3% of SemCor word forms. The remaining word forms were either mapping to elements treated by the ERG as semantically empty (e.g., copulas), or treated as MWE by the ERG but not by WordNet (‘such+as’, ‘right+then’, ‘not+even’)

However, only 36.3% of the ERG predicates emerged as sense-tagged: 55.6% of realpreds and 11.3% of gpreds.

The alignment program generated modified DMRS files, with an optional <sense> element:

<node nodeid='10002' cfrom='0' cto='6'>
   <realpred lemma='first' pos='a' sense='1'/>
   <sortinfo cvarsort='e' sf='prop' tense='untensed' mood='indicative' prog='minus' perf='minus'/>
   <sense wn='2' lexsn='5:00:00:ordinal:00' wn_lemma='first'/>

The sense-annotated DMRS output is available here

There is also an updated dmrs.dtd and SemCoreMapping.csv: a mapping from each SC corpus item to the annotated SemCor 3.0 concordance, context, and sentence number.

Semcor data from Rada Mihalcea

/!\ According to the SemCor readme:

lexsn vs offset

Synsets and Senses are typically referred to in one of two different ways:

offset-pos: 01234567-x

Here the 8 digit offset for a given wordnet, followed by its pos, is used to refer to a synset. A sense is the combination of the synset with a given lemma. This is becoming the standard interchange key between wordnets.

The advantage of the offset-pos is that it gives a handy way to refer to the whole synset, not just a sense.

Not particularly beatutiful code to go from offset-pos to synset using nltk

import nltk
from nltk.corpus import wordnet as ewn

def of2ss(offset):
    '''Look up a synset given offset-pos'''
    return ewn._synset_from_pos_and_offset(str(offset[-1:]), int(offset[:8]))

ss = of2ss('02614387-v')
print ss, ss.definition, ss.lexname, '(%08d-%s)' % (ss.offset, ss.pos) 

Based on code from Masato Hagiwara.

Sense Key Encoding

This is preferred by Princeton WN, described in WordNet's sense index Man Page

A sense is encoded in three parts: lemma, pos, sense number (in the corpus wn_lemma, lexsn, wn).

lex_sense (lexsn) = ss_type:lex_filenum:lex_id:head_word:head_id 
sense number (wn)



Part of Speech
















Concatenating the lemma and lex_sense fields of a semantically tagged word, using % as the concatenation character, creates the sense_key for that sense, which can in turn be used to search the sense index file.

According to PWN, a sense_key is the best way to represent a sense in semantic tagging or other systems that refer to WordNet senses. sense_keys are independent of WordNet sense numbers and synset_offsets, which vary between versions of the database. Using the sense index and a sense_key the corresponding synset (via the synset_offset) and WordNet sense number can easily be obtained.

Code to go from lemma, sense_key, sense_no to synset using nltk

import nltk
from nltk.corpus import wordnet as ewn

def sc2ss(sensekey):
    '''Look up a synset given the information from SemCor'''
    ### Assuming it is the same WN version (e.g. 3.0)
      return ewn.lemma_from_key(sensekey).synset
ss = sc2ss('live%2:42:06::')
print ss, ss.definition, ss.lexname, '(%08d-%s)' % (ss.offset, ss.pos) 

SemCor (last edited 2014-09-23 09:15:32 by LilingTan)

(The DELPH-IN infrastructure is hosted at the University of Oslo)