Note

columns

  1. ordinary: the ordinary grammar (using neither unknown word handling nor bridge rules)

  2. unk: unknown word handling (the YY mode (ZhongYYMode))

  3. br: robust parsing with bridge rules

  4. unk+br: unknown word handling and robust parsing with bridge rules

  5. generation: using the result of [1] as the input MRSes (# of generated items / # of parse items)

  6. end-to-end: # of generated items / # of input sentences

numbers

After using gTest, the numbers are described in the following format. Before that, only the total coverages are presented.

coverage-of-grammatical (readings-or-grammatical) / coverage-of-ungrammatical (readings-or-ungrammatical)

For example, 4.94 (1.5) / 7.89 (2.6667) means the coverage for grammatical items is 4.94%, the coverage for ungrammatical items is 7.89%, the average number of readinfs for grammatical items is 1.5, and the average number of readings for ungrammatical items is 2.6667.

Not all datasets have ungrammatical items. If there is no ungrammatical item in the dataset, the part after "/" is missing.

Note that with the numbers for grammatical items, the higher the better, while with those for ungrammatical ones the lower the better.

abbreviations

options

  1. nbest: 5
  2. timeout: 5 sec.
  3. max-chart-megabytes: 512mb (previously 256mb) for parsing, 2048mb for generation
  4. max-unpack-megabytes: 512mb (previously 256mb), 2048mb for generation

Development Data

mrs

Changes

Date

ordinary

unk

br

unk+br

generation

end-to-end

relatives

2015-07-17

88.79 (3.41)

88.79 (3.45)

100.0 (5.0)

100.0 (5.0)

90.53

80.38

frag+512mb

2015-04-24

94.39 (3.52)

94.39 (3.56)

100.0 (5.0)

100.0 (5.0)

95.05

89.72

NP+PP

2015-04-20

94.39 (3.55)

94.39 (3.59)

99.07 (5.0)

99.07 (5.0)

95.05

89.72

prep+lzg+gle

2015-04-13

91.59 (2.88)

91.59 (2.91)

100.0 (5.0)

100.0 (5.0)

97.96

89.72

nydva

2015-04-10

91.59 (2.71)

91.59 (2.74)

100.0 (5.0)

100.0 (5.0)

97.96

89.72

spr+ditr

2015-04-02

95.33 (3.4)

95.33 (3.41)

100.0 (5.0)

100.0 (5.0)

99.02

94.4

16K+pu

2015-03-27

95.33 (4.05)

95.33 (4.05)

100.0 (5.0)

100.0 (5.0)

99.02

94.4

gTest

2015-03-20

93.46 (3.27)

93.46 (3.27)

100.0 (5.00)

100.0 (5.00)

99.0

92.53

AABB

2015-03-13

93.5

88.8

100

94.4

N/A

N/A

interjections

2015-03-13

93.5

88.8

100

94.4

N/A

N/A

auto

2015-03-12

93.5

88.8

100

94.4

N/A

N/A

w/o auto

2015-03-12

38.3

38.3

97.2

93.5

N/A

N/A

change

YYYY-MM-DD

ordinary

unk

br

unk+br

generation

end-to-end

ntumc

Changes

Date

ordinary

unk

br

unk+br

generation

end-to-end

relatives

2015-07-17

2.71 (3.83) / 0.0 (0.0)

12.42 (4.09) / 20.0 (5.0)

10.88 (4.94) / 6.67 (5.0)

N/A

64.13

1.74

frag+512mb

2015-04-24

3.05 (3.91)

14.56 (4.23)

10.01 (4.93)

47.16 (4.89)

75.77

2.31

NP+PP

2015-04-20

2.4 (4.09)

11.08 (4.39)

8.95 (4.89)

37.07 (4.83)

63.1

1.51

prep+lzg+gle

2015-04-13

1.8 (3.71)

9.44 (3.93)

7.99 (4.84)

42.81 (4.82)

70.69

1.27

nydva

2015-04-10

1.65 (3.63)

8.46 (3.84)

8.01 (4.84)

47.24 (4.84)

78.05

1.29

spr+ditr

2015-04-02

1.52 (3.32)

6.77 (3.7)

7.59 (4.82)

37.96 (4.8)

60.0

0.91

16K+pu

2015-03-27

1.58 (3.42)

6.85 (3.77)

7.59 (4.83)

38.51 (4.8)

79.66

1.26

gTest

2015-03-20

0.32 (1.92)

4.04 (4.14)

1.13 (4.49)

9.7 (4.39)

100.0

0.32

AABB

2015-03-13

0.8

1.5

1.0

3.0

N/A

N/A

interjections

2015-03-13

0.8

1.5

1.0

2.7

N/A

N/A

auto

2015-03-12

0.8

1.5

1.0

2.8

N/A

N/A

w/o auto

2015-03-12

0.0

1.4

0.0

1.9

N/A

N/A

change

YYYY-MM-DD

ordinary

unk

br

unk+br

generation

end-to-end

pctb-dev (the first 5,000 sentences)

Changes

Date

ordinary

unk

br

unk+br

generation

end-to-end

relatives

2015-07-17

0.68 (4.68)

5.56 (4.65)

10.34 (4.97)

46.62 (4.98)

44.12

0.3

frag+512mb

2015-04-24

0.8 (4.75)

6.92 (4.74)

10.18 (4.97)

44.06 (4.97)

45.0

0.36

NP+PP

2015-04-20

0.74 (4.78)

6.02 (4.73)

9.66 (4.97)

35.08 (4.96)

43.24

0.32

prep+lzg+gle

2015-04-13

0.5 (4.44)

4.84 (4.53)

9.78 (4.97)

41.82 (4.96)

60.0

0.3

nydva

2015-04-10

0.36 (4.17)

4.12 (4.46)

10.56 (4.97)

51.7 (4.97)

61.11

0.22

spr+ditr

2015-04-02

0.38 (4.32)

9.92 (2.73)

3.28 (4.9)

35.74 (4.97)

63.16

0.24

16K+pu

2015-03-27

0.38 (4.21)

9.96 (2.75)

3.3 (4.9)

36.1 (4.97)

68.42

0.26

gTest

2015-03-20

0.04 (1.50)

3.08 (4.68)

0.06 (5.00)

7.3 (4.84)

100.0

0.04

AABB

2015-03-13

0.0

2.5

0.1

4.1

N/A

N/A

interjections

2015-03-13

0.0

2.5

0.1

4.1

N/A

N/A

auto

2015-03-12

0.0

2.5

0.1

4.1

N/A

N/A

w/o auto

2015-03-12

0.0

2.3

0.0

3.0

N/A

N/A

change

YYYY-MM-DD

ordinary

unk

br

unk+br

generation

end-to-end

sinica-dev (the first 5,000 sentences (nltk_data))

Changes

Date

ordinary

unk

br

unk+br

generation

end-to-end

relatives

2015-07-17

3.76 (3.78)

34.86 (4.34)

6.54 (4.65)

65.3 (4.94)

78.72

2.96

frag+512mb

2015-04-24

3.56 (3.79)

37.74 (4.52)

6.34 (4.57)

64.22 (4.92)

80.9

2.88

NP+PP

2015-04-20

3.3 (3.77)

35.38 (4.54)

6.32 (4.56)

62.62 (4.92)

77.58

2.56

prep+lzg+gle

2015-04-13

3.02 (3.32)

31.22 (4.15)

6.28 (4.53)

63.3 (4.91)

80.13

2.42

nydva

2015-04-10

2.96 (3.15)

28.96 (4.1)

6.36 (4.53)

63.74 (4.91)

81.08

2.4

spr+ditr

2015-04-02

2.9 (2.9)

23.64 (3.96)

6.24 (4.55)

56.6 (4.9)

81.38

2.36

16K+pu

2015-03-27

3.0 (3.01)

24.08 (4.05)

6.24 (4.56)

56.6 (4.9)

82.0

2.46

gTest

2015-03-20

1.36 (3.06)

23.06 (4.70)

2.8 (4.46)

30.86 (4.83)

95.59

1.3

AABB

2015-03-13

1.2

8.7

2.5

11.1

N/A

N/A

interjections

2015-03-13

1.2

8.6

2.5

11.1

N/A

N/A

auto

2015-03-12

1.2

8.6

2.4

11.1

N/A

N/A

w/o auto

2015-03-12

0.1

8.1

0.7

10.0

N/A

N/A

change

YYYY-MM-DD

ordinary

unk

br

unk+br

generation

end-to-end

Held-out Data

jec

Changes

Date

ordinary

unk

br

unk+br

generation

end-to-end

relatives

2015-07-17

10.43 (3.69)

32.77 (3.74)

27.19 (5.0)

79.85 (5.0)

72.88

7.6

frag+512mb

2015-04-24

11.92 (3.77)

37.71 (3.87)

26.6 (5.0)

78.96 (5.0)

93.08

11.1

NP+PP

2015-04-20

11.97 (3.81)

37.78 (3.89)

26.17 (5.0)

77.06 (5.0)

90.48

10.83

prep+lzg+gle

2015-04-13

10.07 (3.26)

32.33 (3.24)

24.49 (4.99)

77.51 (4.99)

95.88

9.66

nydva

2015-04-10

9.01 (3.13)

28.96 (3.13)

24.06 (4.99)

77.87 (4.99)

96.03

8.65

spr+ditr

2015-04-02

8.43 (2.79)

26.04 (2.84)

22.45 (4.99)

69.16 (4.99)

96.42

8.13

16K+pu

2015-03-27

8.54 (3.05)

26.21 (3.05)

22.45 (4.99)

69.19 (4.99)

95.54

8.16

gTest

2015-03-20

2.77 (2.29)

21.66 (3.88)

6.26 (5.00)

37.03 (5.00)

87.5

2.42

AABB

2015-03-13

2.7

20.5

4.5

31.1

N/A

N/A

interjections

2015-03-13

2.5

20.2

4.3

30.4

N/A

N/A

auto

2015-03-12

2.5

20.2

4.3

30.6

N/A

N/A

w/o auto

2015-03-12

0.0

18.3

0.1

22.7

N/A

N/A

change

YYYY-MM-DD

ordinary

unk

br

unk+br

generation

end-to-end

fu-berlin

Changes

Date

ordinary

unk

br

unk+br

generation

end-to-end

relatives

2015-07-17

22.22 (2.56) / 39.47 (2.67)

77.78 (2.84) / 68.42 (2.77)

22.22 (4.89) / 55.26 (5.0)

97.53 (4.97) / 100.0 (5.0)

81.82

18.18

frag+512mb

2015-04-24

22.22 (3.11) / 39.47 (3.07)

80.25 (3.12) / 68.42 (3.0)

22.22 (4.89) / 55.26 (5.0)

97.53 (4.97) / 100.0 (5.0)

90.91

20.2

NP+PP

2015-04-20

11.11 (3.0) / 18.42 (4.14)

67.9 (3.27) / 42.11 (3.44)

22.22 (4.89) / 55.26 (5.0)

97.53 (4.97) / 100.0 (5.0)

81.25

9.03

prep+lzg+gle

2015-04-13

8.64 (1.29) / 15.79 (2.0)

54.32 (2.0) / 34.21 (2.15)

16.05 (4.92) / 42.11 (4.75)

97.53 (4.99) / 100.0 (4.89)

100.0

8.64

NP+PP

2015-04-20

11.11 (3.0) / 18.42 (4.14)

67.9 (3.27) / 42.11 (3.44)

22.22 (4.89) / 55.26 (5.0)

97.53 (4.97) / 100.0 (5.0)

81.25

9.03

nydva

2015-04-10

8.64 (1.29) / 15.79 (1.83)

46.91 (1.74) / 31.58 (1.92)

16.05 (4.92) / 42.11 (4.75)

97.53 (4.99) / 100.0 (4.89)

100.0

8.64

spr+ditr

2015-04-02

8.64 (1.29) / 15.79 (1.83)

45.68 (1.68) / 26.32 (1.9)

16.05 (4.92) / 42.11 (4.31)

79.01 (4.98) / 78.95 (4.63)

100.0

8.64

16K+pu

2015-03-27

8.64 (2.14) / 15.79 (2.67)

43.21 (2.26) / 26.32 (2.5)

16.05 (4.92) / 42.11 (4.63)

79.01 (4.98) / 78.95 (4.8)

100.0

8.64

gTest

2015-03-20

4.94 (1.5000) / 7.89 (2.67)

37.04 (2.67) / 21.05 (2.38)

12.35 (4.10) / 36.84 (3.50)

74.07 (4.85) / 76.32 (4.28)

100.0

4.94

AABB

2015-03-13

4.9

39.5

9.9

65.4

N/A

N/A

interjections

2015-03-13

4.9

40.7

9.9

65.4

N/A

N/A

auto

2015-03-12

4.9

40.7

9.9

65.4

N/A

N/A

w/o auto

2015-03-12

1.2

48.1

3.7

64.2

N/A

N/A

change

YYYY-MM-DD

ordinary

unk

br

unk+br

generation

end-to-end

mcg-smallest

Changes

Date

ordinary

unk

br

unk+br

generation

end-to-end

relatives

2015-07-17

51.35 (3.16) / 61.11 (3.82)

57.66 (3.22) / 61.11 (3.82)

84.68 (5.0) / 94.44 (5.0)

100.0 (5.0) / 100.0 (5.0)

94.12

48.33

frag+512mb

2015-04-24

54.95 (3.34) / 61.11 (3.73)

61.26 (3.4) / 61.11 (3.73)

83.78 (5.0) / 94.44 (5.0)

99.1 (5.0) / 100.0 (5.0)

95.83

52.66

NP+PP

2015-04-20

55.86 (3.16) / 61.11 (3.73)

62.16 (3.23) / 61.11 (3.73)

83.78 (5.0) / 94.44 (5.0)

99.1 (5.0) / 100.0 (5.0)

95.89

53.56

prep+lzg+gle

2015-04-13

42.34 (2.7) / 38.89 (3.29)

50.45 (2.75) / 50.0 (2.78)

77.48 (5.0) / 72.22 (5.0)

98.2 (5.0) / 94.44 (5.0)

100.0

42.34

nydva

2015-04-10

36.94 (2.56) / 38.89 (3.29)

41.44 (2.67) / 50.0 (2.78)

77.48 (5.0) / 72.22 (5.0)

98.2 (5.0) / 94.44 (5.0)

100.0

36.94

spr+ditr

2015-04-02

34.23 (2.08) / 27.78 (3.4)

37.84 (2.12) / 27.78 (3.4)

70.27 (5.0) / 66.67 (5.0)

87.39 (5.0) / 72.22 (5.0)

100.0

34.23

16K+pu

2015-03-27

35.14 (2.49) / 27.78 (3.6)

38.74 (2.51) / 27.78 (3.6)

70.27 (5.0) / 66.67 (5.0)

87.39 (5.0) / 72.22 (5.0)

100.0

35.14

gTest

2015-03-20

24.32 (1.89) / 16.67 (2.33)

38.74 (2.81) / 22.22 (2.00)

45.05 (5.00) / 33.33 (5.00)

67.57 (5.00) / 44.44 (5.00)

100.0

24.32

AABB

2015-03-13

27.9

44.1

44.1

64.0

N/A

N/A

interjections

2015-03-13

26.1

42.3

42.3

62.2

N/A

N/A

auto

2015-03-12

26.1

42.3

42.3

62.2

N/A

N/A

w/o auto

2015-03-12

4.5

46.8

7.2

56.8

N/A

N/A

change

YYYY-MM-DD

ordinary

unk

br

unk+br

generation

end-to-end

mcg-wxl

Changes

Date

ordinary

unk

br

unk+br

generation

end-to-end

relatives

2015-07-17

54.75 (3.62) / 63.0 (3.1)

63.45 (3.56) / 63.0 (3.1)

82.44 (5.0) / 98.0 (5.0)

99.37 (5.0) / 100.0 (5.0)

91.69

50.2

frag+512mb

2015-04-24

56.17 (3.79) / 62.0 (3.29)

65.35 (3.77) / 62.0 (3.29)

80.7 (5.0) / 95.0 (5.0)

97.63 (5.0) / 97.0 (5.0)

93.05

52.27

NP+PP

2015-04-20

56.49 (3.74) / 62.0 (3.29)

65.35 (3.73) / 62.0 (3.29)

80.38 (5.0) / 95.0 (5.0)

97.31 (5.0) / 97.0 (5.0)

95.08

53.71

prep+lzg+gle

2015-04-13

43.99 (3.37) / 48.0 (2.81)

53.8 (3.36) / 53.0 (2.64)

75.47 (5.0) / 82.0 (5.0)

96.04 (5.0) / 95.0 (5.0)

98.77

43.45

nydva

2015-04-10

39.08 (3.19) / 48.0 (2.73)

43.99 (3.21) / 51.0 (2.63)

75.16 (5.0) / 82.0 (5.0)

96.04 (5.0) / 95.0 (5.0)

98.98

38.68

spr+ditr

2015-04-02

36.23 (2.71) / 43.0 (2.05)

43.2 (2.69) / 50.0 (2.26)

68.2 (5.0) / 72.0 (5.0)

87.18 (5.0) / 81.0 (5.0)

98.16

35.56

16K+pu

2015-03-27

35.92 (2.97) / 46.0 (2.52)

42.88 (2.91) / 53.0 (2.66)

68.2 (5.0) / 72.0 (5.0)

87.18 (5.0) / 81.0 (5.0)

99.19

35.63

gTest

2015-03-20

21.36 (2.21) / 23.0 (1.40)

34.97 (3.06) / 31.0 (2.19)

45.89 (5.00) / 45.0 (5.00)

70.09 (5.00) / 55.0 (5.00)

100.0

21.36

AABB

2015-03-13

23.9

37.8

44.5

66.3

N/A

N/A

interjections

2015-03-13

23.4

37.3

43.8

65.5

N/A

N/A

auto

2015-03-12

23.4

37.3

43.8

65.5

N/A

N/A

w/o auto

2015-03-12

3.8

46.6

5.5

56.2

N/A

N/A

change

YYYY-MM-DD

ordinary

unk

br

unk+br

generation

end-to-end

Test Data

pctb-test (the second 5,000 sentences)

Changes

Date

ordinary

unk

br

unk+br

generation

end-to-end

relatives

2015-07-17

0.88 (4.7)

6.26 (4.47)

10.76 (5.0)

44.26 (4.98)

45.45

0.4

frag+512mb

2015-04-24

1.06 (4.81)

7.88 (4.59)

10.68 (5.0)

41.6 (4.98)

40.0

0.42

NP+PP

2015-04-20

0.9 (4.53)

6.84 (4.59)

10.12 (5.0)

32.9 (4.97)

53.33

0.48

prep+lzg+gle

2015-04-13

0.7 (4.17)

5.68 (4.24)

10.14 (5.0)

39.46 (4.97)

60.0

0.42

nydva

2015-04-10

0.66 (4.12)

5.12 (4.22)

10.76 (5.0)

50.36 (4.98)

63.64

0.42

spr+ditr

2015-04-02

0.4 (4.25)

10.42 (2.6)

3.64 (5.0)

34.06 (4.97)

70.0

0.28

16K+pu

2015-03-27

0.4 (4.4)

10.44 (2.64)

3.62 (5.0)

34.22 (4.97)

75.0

0.3

gTest

2015-03-20

0.0

3.68 (4.41)

0.08 (5.00)

7.38 (4.89)

0.0

0.0

AABB

2015-03-13

0.0

8.5

0.0

9.9

N/A

N/A

interjections

2015-03-13

0.0

8.5

0.0

9.9

N/A

N/A

auto

2015-03-12

0.0

8.5

0.0

9.9

N/A

N/A

w/o auto

2015-03-12

0.0

9.2

0.0

8.5

N/A

N/A

change

YYYY-MM-DD

ordinary

unk

br

unk+br

generation

end-to-end

sinica-test (the second 5,000 sentences (nltk_data))

Changes

Date

ordinary

unk

br

unk+br

generation

end-to-end

relatives

2015-07-17

0.06 (4.0)

11.82 (4.87)

0.18 (4.56)

39.08 (4.96)

0.0

0.0

frag+512mb

2015-04-24

0.06 (4.0)

15.3 (4.93)

0.18 (4.56)

37.16 (4.96)

0.0

0.0

NP+PP

2015-04-20

0.04 (4.5)

13.92 (4.93)

0.18 (4.56)

29.3 (4.95)

50.0

0.02

prep+lzg+gle

2015-04-13

0.02 (5.0)

12.84 (4.85)

0.18 (4.56)

35.16 (4.96)

100.0

0.02

nydva

2015-04-10

0.02 (5.0)

11.42 (4.85)

0.18 (4.56)

36.92 (4.96)

100.0

0.02

spr+ditr

2015-04-02

0.02 (5.0)

7.76 (4.77)

0.16 (4.0)

29.4 (4.95)

100.0

0.02

16K+pu

2015-03-27

0.02 (5.0)

7.92 (4.79)

0.18 (3.67)

29.46 (4.95)

100.0

0.02

gTest

2015-03-20

0.0

5.14 (4.95)

0.02 (5.00)

3.74 (4.68)

0.0

0.0

AABB

2015-03-13

0.0

1.8

0.1

1.5

N/A

N/A

interjections

2015-03-13

0.0

1.7

0.1

1.5

N/A

N/A

auto

2015-03-12

0.0

1.7

0.1

1.5

N/A

N/A

w/o auto

2015-03-12

0.0

1.7

0.0

1.1

N/A

N/A

change

YYYY-MM-DD

ordinary

unk

br

unk+br

generation

end-to-end

ZhongHistory (last edited 2015-07-20 02:55:22 by SanghounSong)

(The DELPH-IN infrastructure is hosted at the University of Oslo)