New and revived features in the LKB FOS
Notes from session 2 of the LKB+FOS SIG. We again started off by reviewing the slides.
They show the differences between versions.
WP: "Random" unpacking ((setq *unpacking-scoring-hook* (constantly 0.0))) isn't really random, right?
JC: No, it gives an arbitrary - but fixed - order. #(lambda (&rest x) (random 1.0)) for example would give random ordering.
PX: Is there a configuration file for the control parameters?
JC: Yes, you can put them in .lkbrc
EMB: One of the problems we have with Ubuntu+LKB is you can't paste non-ASCII. Is this fixed in LKB-FOS?
JC: You can paste Unicode characters, but font may not support them in which case they would show up as boxes (aka “tofu”). It's not easy to change the font, because X11 wasn't designed with this in mind. The more difficult problem is typing them; none of the Linux input methods work. With macOS you can do some things with the keyboard widget, but not other input methods.
Glenn: Was able to enter and paste using Trollet; still supported?
JC: Started integrating Trollet, but turned out to be more difficult than expected (it's a brittle socket interface). Hopeful that the McCLIM developers will be adding input method support, but not there at the moment.
JC: Trollet integration probably needs a week or so of work, but I see it as a dead-end so not very keen to put the effort in.
EMB: Any reason not to have the next version of the Ubuntu+LKB VM to be LKB-FOS?
JC: I'd be very pleased. OE is keen to simplify the versions that are maintained. Ideally to merge LKB-FOS into the same codebase
EMB: I'll be aiming for that by the next iteration of Grammar Engineering class, which is next January.
WP: Can you use the Mac clickable-app version?
EMB: Yes, but I have to accommodate Windows users and it's helpful for debugging to be able to see exactly what the students are seeing.
JC: Windows version seems to be widely wanted, but difficult to work on right now.
DF: I think it's worth adding to the list of differences that only the LKB+FOS really supports TDL docstrings. It would make life much better to shift to LKB+FOS as "the" version of LKB.
DF: I have a list of patches that I’ve had to apply for the ERG, how to determine which are still needed?
JC: Once OE and I have merged codebases then we'll have a clearer idea of what patches are now extraneous.
WP: Feature request: not bad thing to continue efficiency quest to ultimately surpass ACE so I don't have to do anything.
JC: In the "evaluation" ACE is generally so far ahead. Difficult to conceive of being able to do generalization packing well, for example.
WP: threshold on generalisation level -- very conservative; probably not that complicated.
JC: I think go-faster improvements are at the end of the list of priority improvements.
GS: Would be interested in the algorithm also.
WP: Yes, should do at least an informal writeup.
JC: I wouldn't mind putting some work into visualizations to help in digging into a complex grammar
OZ: Unsolicited report on how I use the different versions: I have a Mac as my primary machine; my goal is to do as much as I can on this. I only resort to the VM/Ubuntu as a last resort when I absolutely can't do something on my Mac. Hard to be efficient in both. I use LKB primarily for debugging issues in grammars; for parsing & regression testing I now use the tool Michael wrote which calls ACE. Very happy to be using that over the "classic" regression testing system. Personally the GUI looking old-fashioned tends to not be a problem at all. Did notice: do you have an idea of why the file explorer becomes slower after a while? I think my version of LKB-FOS is from March. It's been fine for months, and then suddenly a few weeks ago it takes forever and sometimes I have to kill it. Any idea?
JC: March would be the July 2019 version, so that's a year out of date. There might be something that changed in the dialogs. There was at least one bug I reported in the McCLIM code, which they fixed.
OZ: That sounds like a different problem. Hesitant to change now until I finish what I'm working on.
Happy with what LKB+FOS brings for the mac.... (technical difficulties here)
DF: Invitation about imagining improvements in visualization. Exciting and have been brewing ideas for about a decade. Central use case: sentence where machine reports staggering amount of ambiguity or resources to parse, and my instinct as a grammarian is that that is wrong. Ideally would like to go to the parse chart and look for the densest part of the chart. This is really slow in LUI, but that doesn't seem the right way to look at this problem. I'm picturing a 20,000ft view of the chart that shows thicker/thinner, just to see where the problematic span is.
Next improvement would be to tell me the density of the rule constructions that are the mother nodes of those rules. Have to turn on packing to even get to see it, but packing hides the density that I'm trying to see.
In the best case, I'd like to know, based on MaxEnt or some other model, which edges never occur in any full parse? I'm sure it's the case in numeric expressions that just go haywire. e.g. you gave me two numbers, an adjective, and two nouns, but I don't see that anywhere in your gold sentences. "Surprisal"
WP: What do you think the right granularity is for matching those MaxEnt models to what's in the black hole?
DF: Something like a 3-4 level deep tree. Pretty sure it's not 7+, though. Might need to say which nodes are interesting for branching factor, interesting levels of grandparenting, etc. I agree it's a critical element to having a visualizer give me something useful.
Might have to work by example for a while -- looking at some examples and see where we can identify the salient properties. My guess is (based on other things we've played with Woodley) I might have to impose myself on you in person in Sussex to do a little of this by brute force. Probably heuristic strategies.
(back to olga after tech lacuna...)
OZ: I use LKB+FOS to debug any grammars that use Latin script; when I need to debug my Russian grammars I need to go back to the VM because I need the LUI version to work.
JC: I think if you upgrade MacLUI it should go away.
WP: I haven't looked at it in a few months, but I believe it should handle Unicode fine now. I'd like to not be in the VM at all.
JC: I'd love to bring [incr tsdb()] over to mac, but that seems pretty unlikely. It's not really an interesting project for a student to work on to improve, even.
OZ: Even getting to navigate the text field is a huge quality of life improvement.
JC: Which I prefer to wait for a real fix that appears to be coming rather than hack around.
(back to visualization topic)
JC: Dan, perhaps a virtual workshop on this?
DF: Maybe some emails to set up and coordinate. A few hours isn't a good way to tackle this, but perhaps some flexible scheduling where we can come together as we've all done homework or made some progress.
WP: The first path is pretty simple; scoring those with respect to a corpus is much more complex. Can probably spin something like that up in an afternoon, which might at least tell you something.
DF: It may be that not being able to see it at all is blocking me from even thinking about it futher. Not sure yet what pieces will help me to find the source of evil.
JC: It seems to not be a well-defined problem, so maybe iteratively leveraging the LUI features would be a good start.
GS: Would being able to turn on/off rules and seeing how that changes the density help?
DF: It might. I have done that in the past by hand, with an hypothesis about what might be wrong, to turn off some bits. I think it's probably the conspiracy of 2-3 rules together is more likely the problem, though, and that makes it harder to isolate individual rules. But it would be a valuable part of the toolbox to confirm hypotheses, rather than generating them.
XP: Trouble is no easy way to turn on/off without recompiling.
JC: True, but could also do some visualization pruning without rebuilding also. This idea is really an extension of filtering edges in the parse chart.
WP: Would be helpful to send a few example sentences via email that exhibit this "black hole" behavior.
GS: One example: "We admire the National Aeronautics and Space Administration's Ames Research Center at Moffett Field, Calif."
DF: Other wishlist items... Single biggest other item is in grammar development; tree for an analysis -> where did this feature come from? Where in my TDL file did this [INFL na] get set? I know that my parser knows this, because it did the unification, but I don't know that it remembers it. Especially important when I have a definition, but I've masked it several times because I'm an idiot. I'd like the machine to tell me hey, you've redefined this a fourth time.
WP: Interesting but non-trivial problem, but one I've worked on. It's implemented in ACE, and you can activate it from ACE: pretend you’re doing interactive unification and drag feature you're interested in onto anywhere in the AVM that contains it (e.g., would create a cycle) I interpret that as this request to compute provenance.
DF: What happens when I declare it in two places?
WP: I'd need a clearer example
[some exploration] loos like it's just one result
DF: Interesting enough in the vanilla case, but it would be especially helpful to see all of the cases. This might be a good opportunity for division of labor with the LKB+FOS -- does something come to mind about how to display that information?
JC: I'm not clear; how do you do that?
WP: It's been a couple of years so I can't tell you exactly what it's doing. I know the sequence of unifications.
JC: Do you record that normally?
WP: Not for an arbitrary AVM, but the fact that I grabbed it from a tree is easy enough, and the tree records the sequence of unifications.
JC: The LKB reconstructs its trees from the chart.
WP: Yes, same.
GS: The way I did that in AGREE is instrumenting the unification so that when a conflict occurs I can spit out where it got there.
WP: Yes, for unification failure, what about success?
GS: Right, I'm talking about re-unification. conflict was probably the wrong term.
DF: What about type inference? It's not in any of the definitions.
WP: I agree it's a wrinkle, but I don't remember if/how I solved it.
DF: Valuable for a grammarian of course, but also valuable for succession planning for a new grammarian coming in to understand an existing grammar.
WP: I think the functionality is in the released versions of ACE.
JC: One thing that might help you in future is unified grammar config. Allow conditioning on TDL fragments, so you wouldn't have to have overlays/patching.
DF: On any feature/value pairs?
JC: No, just by sign. So you could put the two rules side by side in the TDL file.
DF: That might help reduce the chance of having four copies of the same rule, though not eliminate it.
GS: I think I presented something on this idea of live editing at a previous DELPH-IN.
DF: Yes, I think that's where I started thinking about this idea.
OZ: Any other secrets in ACE?
WP: None, no secrets whatsoever.
OZ: I wonder if it's me doing it wrong, but sometimes it takes me a long time to find the unification failure. Is that normal, or am I missing some shortcut? Maybe it's the UI after all and the scrolling?
JC: Are you using LUI? That's Woodley's stuff.
OZ: If I turn off LUI, will it be faster?
JC: Not necessarily; you'll get the native LKB display. Which doesn't do drag-and-drop interactive unification; it has 'select' and 'unify' instead.
DF: Does it zoom in on the failure?
JC: No, but if it fails it will put up a dialog with the first path that fails.
WP: The unification is computed by the LKB in either case. I believe I zoom in on the first failure. R/L Arrow keys cycle through the unification failures in LUI. But ACE will only report one failure.
JC: In the LKB there's a *unify-debug* flag that gets rebound on for interactive unification.
DF: Some while ago WP humored me to come up with a way to type some of the words you think might be in the generation output, and would show some collection of the edges that would include those spans. It's powerful, but hard to use. Have you been playing around at all with better ways to visualize what the generator is doing?
JC: No, I haven't looked at it recently. What I have found is that selecting lexical edges in the LKB generator chart no longer highlights derived phrasal edges. I'll think about this visualisation problem and try to implement a prototype.
DF: There might be other ways do do this, such as being able to select parts of the MRS that should be in the output. On the other hand, it's much easier to manipulate tokens rather than MRSes. Also not easy to tell that I don't have a lexical entry for a given predication.
WP: ACE is supposed to print an error message when you try to generate without full coverage.
DF: It may be that there is *a* lexical entry, but not the one I need (transitive vs. complement-taking, eg)
JC: Perhaps select EPs, and show what edges are generated from that collection of EPs.
[more in-depth discussion of generation and controlling what to see]
JC: One question about releases: I do them 3x a year, but don't typically announce them. Would people be upset if I sent out notifications?
DF: No, I think people would welcome them.