|
Size: 4799
Comment:
|
Size: 6078
Comment: Interim save
|
| Deletions are marked like this. | Additions are marked like this. |
| Line 3: | Line 3: |
| Before you start: suitable for POWER USERS only. | The LexDB is useful for grammars with large lexicons. Rather than storing the lexicon in a .tdl file, which has to be loaded each time the lexicon is changed, the LexDB stores the lexical entries in a postgres database. The instructions for using the LexDB are somewhat sparse and it is not expected that novice LKB users will make use of the system. The best approach for users who want to make use of the LexDB for their own grammars is to start by getting a LexDB going with an existing grammar in order to become familiar with the set up. The initial choice that has to be made is whether to use the LexDB in the 'single user' or 'multiple user' modes. In the multiple user mode, the revisions are stored in the database itself. The assumption is that the database is running on a server to which multiple users may have access. This mode assumes that the database is set up with its own id (something that system administrators may dislike because of potential security risks). In single user mode, the database user may be the actual user. Revisions are not stored in the database (the assumption is that Subversion or CVS will be used to manage changes). |
| Line 5: | Line 6: |
| You must be running a LexDB-enabled build of the LKB: | * [http://www.cl.cam.ac.uk/~bmw20/Papers/ Papers] |
| Line 7: | Line 8: |
| 1. download the [http://lingo.stanford.edu/ftp/ LKB binary] and the [http://www.cl.cam.ac.uk/~bmw20/DT/lkb/latest/ LexDB setup files]; or 2. download the [http://lingo.stanford.edu/ftp/ LKB source code]. |
* LkbLexDbSingleUser for basic instructions for the SINGLE USER LexDB |
| Line 10: | Line 10: |
| If running the binary, ensure the environment variable `PSQL` is set: | * LkbLexDbMultiUser for basic instructions for the MULTI USER LexDB |
| Line 12: | Line 12: |
| {{{ export PSQL=t }}} | * JacyLexDb |
| Line 14: | Line 14: |
| If building from source, ensure your {{{.clinit.cl}}} file contains the following: | * LexDbEmacsInterface |
| Line 16: | Line 16: |
| {{{ (pushnew :psql *features*) }}} | * LexDbFieldMappings |
| Line 18: | Line 18: |
| InitializePostgres [[BR]] ["InitializeSkeletalLexDB"] |
* LexDbFunctions * LexDbInitialize * LexDbInternals * LexDbPsqlInitialize The remainder of this page contains miscellaneous information about use of the LexDb. == LKB Compilation == The LexDB code requires that *features* includes :psql. This is now true by default for supported platforms. |
| Line 22: | Line 34: |
| ''LexDB -> Filter'' | ''LexDB -> Set Filter'' |
| Line 24: | Line 36: |
| The filter specified will be interpreted as an SQL WHERE clause. Eg. |
The filter specified is interpreted as an SQL WHERE clause. Examples: |
| Line 35: | Line 45: |
| (Note: the default is {{{ TRUE }}}. This represents the empty condition and will select all available entries.) | (Note: the default is `NULL`. Change this to a non-NULL value to select a subset of the lexical database.) |
| Line 37: | Line 47: |
| The lexicon as seen by the login user is determined by that user's database filter. Only revision entries matching the conditions in filter can form part of the lexicon. In general multiple revisions for a given entry will be returned; the most recent will become part of the visible lexicon. | The filter determines the entries in the lexicon as seen by a particular user. Only revision entries matching the filter conditions in filter can form part of the lexicon (of these, the most recent revision is the one actually used). |
| Line 45: | Line 55: |
| (This will dump public schema tables to text files -- eg. {{{ lexdb.rev }}}, {{{lexdb.dfn}}}, and {{{lexdb.fld}}}) [[BR]](Note: a TDL dump will be performed also unless you set {{{*lexdb-dump-tdl*}}} to {{{nil}}}) [[BR]](Note: the database dump files are tab-separated with null as {{{\N}}}) |
(This will dump public schema tables to text files -- eg. `lexdb.rev`, `lexdb.rev_key`, `lexdb.dfn`, `lexdb.fld`, `lexdb.meta`) [[BR]](Note: the dump mechanism will also produce a `.tdl` file if `*lexdb-dump-tdl*` is set to `t`) [[BR]](Note: the database dump files are tab-separated with null as `\N`) |
| Line 49: | Line 59: |
| 2. Run the cvs commit command. Eg. | 2. Run the cvs commit command. E.g. |
| Line 51: | Line 61: |
| {{{cvs commit ~/erg/lexdb.*}}} | `cvs commit ~/erg/lexdb.*` |
| Line 55: | Line 65: |
| 1. Run the cvs update command to retrieve the latest dump file. Eg. | 1. Run the cvs update command to retrieve the latest dump file. E.g. |
| Line 59: | Line 69: |
| 2. ''LexDB -> Merge new entries'' | 2. ''LexDB -> Load ('rev' entries)'' |
| Line 61: | Line 71: |
| These steps update the LexDB (public schema) to include all new revisions stored in a CVS dump file. The new entries will be copied to the table public.revision_new. Any changes made to your copy of the LexDB since the last update will be preserved. | These steps update the LexDB (public schema) to include all new revisions stored in a CVS dump file. The new entries will be copied to the table public.rev_new. Any changes made to your copy of the LexDB since the last update will be preserved. |
| Line 63: | Line 73: |
| == HOW TO dump LexDB as TDL file == | == HOW TO export LexDB to TDL file == |
| Line 65: | Line 75: |
| ''LexDB -> Dump (TDL format)'' | ''LexDB -> Export (TDL file)'' |
| Line 67: | Line 77: |
| Dumps active LexDB entries (see filter) to {{{.tdl}}} file. | Dumps active LexDB entries (determined by filter) to `.tdl` file. |
| Line 71: | Line 81: |
| The LexDB-Emacs interface allows editing of lexical entries from within an Emacs environment (with browsing functionality, field completion, etc.). New revision entries are first stored in the users private schema, and hence are visible only to the particular user. To commit the entries to the public table (public.revision): | The LexDB-Emacs interface allows editing of lexical entries from within an Emacs environment (with browsing functionality, field completion, etc.). New revision entries are first stored in the users private schema, and hence are visible only to the particular user. |
| Line 73: | Line 83: |
| 0. Add the following line to your {{{.emacs}}} file: | 0. Add the following (path adjusted for your setup) to your `.emacs` file: |
| Line 75: | Line 85: |
| {{{(load "pg-interface")}}} |
{{{ (add-to-list 'load-path "/path/to/delphin/lkb/lexdb") (load "pg-interface") }}} |
| Line 83: | Line 95: |
| ''C-c'' : commit (edited/new revision of) lexical entry into LexDB | ''C-c C-c'' : commit (edited/new revision of) lexical entry into LexDB |
| Line 87: | Line 99: |
| ''M-TAB'' : get (ring of) (active) entries in LexDB where value of current field matches that in buffer | ''M-TAB l'' : get (ring of) entries in table lex where value of current field matches that in buffer ''M-TAB r'' : get (ring of) entries in union of rev tables where value of current field matches that in buffer |
| Line 95: | Line 109: |
| ''M-vs'' : view entries in user's scratch space | ''M-vs'' : view entries in user's privat rev |
| Line 97: | Line 111: |
| Note: To remove a lexical entry from the active grammar, create a (head) revision where the {{{flags}}} field is set to `0` (rather than `1`). This is necessary as in order to preserve revision history entry. No revision entry should ever be deleted from the lexical database itself.) | To remove a lexical entry from the current lexicon ''lex'', create a (head) revision where the `dead` field is set to `t` (true) rather than `f` (false). In this manner we keep a revision history even for entries which are no longer used (and such entries can be reactivated if necessary). No revision entry should ever be deleted from the lexical database itself. |
| Line 99: | Line 113: |
| == HOW TO load TDL entries into private schema == | == HOW TO import TDL entries from a file == |
| Line 101: | Line 115: |
| To add a small number of new (revision) entries from a `.tdl` file: ''LexDB -> Load TDL entries''. The grammatical fields of the LexDB will be obtained from the TDL code; you will be queried to provide values for other necessary fields. | To add a small number of new (revision) entries from a `.tdl` file: ''LexDB -> Import (TDL file)''. You will be queried to provide values for other certain non-grammar fields. The entries will go into the private ''rev'' table. |
| Line 103: | Line 117: |
| == HOW TO commit entries to public schema == | == HOW TO commit entries to public rev == |
| Line 105: | Line 119: |
| The LexDB consists of a single public schema and a set of private schemas, one per user. New (revision) entries are placed initially in your private schema. To commit (all) entries in your private schema to the public table: ''LexDB -> Commit scratch'' | The LexDB consists of a single public schema and a set of private schemas, one per user. New (revision) entries are placed initially in your private schema. To commit (all) entries in your private schema to the public table: ''LexDB -> Commit private 'rev' '' |
| Line 108: | Line 122: |
| == HOW TO list entries in private schema == | == HOW TO list entries in private rev == |
| Line 110: | Line 124: |
| From LKB: ''LexDB -> View scratch'' | From LKB: ''LexDB -> Display private 'rev' '' |
| Line 116: | Line 130: |
| == HOW TO clear entries in private schema == | == HOW TO clear entries in private rev == |
| Line 118: | Line 132: |
| ''LexDB -> Clear scratch'' | ''LexDB -> Clear private 'rev''' |
| Line 120: | Line 134: |
| == Further Topics == | == Troubleshooting == |
| Line 122: | Line 136: |
| ["InternalLexDBStructure"] [[BR]]["MWEs and Idiomatic Expressions"] [[BR]] [http://www.cl.cam.ac.uk/~bmw20/DT/Papers/ Papers] |
* If M-x lexdb results in "no connection to LexDb", make sure that a working grammar is loaded. |
LexDB Usage Instructions
The LexDB is useful for grammars with large lexicons. Rather than storing the lexicon in a .tdl file, which has to be loaded each time the lexicon is changed, the LexDB stores the lexical entries in a postgres database. The instructions for using the LexDB are somewhat sparse and it is not expected that novice LKB users will make use of the system. The best approach for users who want to make use of the LexDB for their own grammars is to start by getting a LexDB going with an existing grammar in order to become familiar with the set up. The initial choice that has to be made is whether to use the LexDB in the 'single user' or 'multiple user' modes. In the multiple user mode, the revisions are stored in the database itself. The assumption is that the database is running on a server to which multiple users may have access. This mode assumes that the database is set up with its own id (something that system administrators may dislike because of potential security risks). In single user mode, the database user may be the actual user. Revisions are not stored in the database (the assumption is that Subversion or CVS will be used to manage changes).
* [http://www.cl.cam.ac.uk/~bmw20/Papers/ Papers]
* LkbLexDbSingleUser for basic instructions for the SINGLE USER LexDB
* LkbLexDbMultiUser for basic instructions for the MULTI USER LexDB
The remainder of this page contains miscellaneous information about use of the LexDb.
LKB Compilation
The LexDB code requires that *features* includes :psql. This is now true by default for supported platforms.
HOW TO set the filter
LexDB -> Set Filter
The filter specified is interpreted as an SQL WHERE clause. Examples:
userid = 'danf'
userid = 'danf' AND dialect = 'my_dialect'
userid IN ('danf', 'aac')
confidence > 0.5(Note: the default is NULL. Change this to a non-NULL value to select a subset of the lexical database.)
The filter determines the entries in the lexicon as seen by a particular user. Only revision entries matching the filter conditions in filter can form part of the lexicon (of these, the most recent revision is the one actually used).
HOW TO store LexDB in CVS
The LexDB may be dumped to text files which can then be uploaded to storage in CVS.
1. LexDB -> Dump
(This will dump public schema tables to text files -- eg. lexdb.rev, lexdb.rev_key, lexdb.dfn, lexdb.fld, lexdb.meta) BR(Note: the dump mechanism will also produce a .tdl file if *lexdb-dump-tdl* is set to t) BR(Note: the database dump files are tab-separated with null as \N)
2. Run the cvs commit command. E.g.
cvs commit ~/erg/lexdb.*
HOW TO retrieve LexDB from CVS
1. Run the cvs update command to retrieve the latest dump file. E.g.
cvs update ~/erg/lexdb.*
2. LexDB -> Load ('rev' entries)
These steps update the LexDB (public schema) to include all new revisions stored in a CVS dump file. The new entries will be copied to the table public.rev_new. Any changes made to your copy of the LexDB since the last update will be preserved.
HOW TO export LexDB to TDL file
LexDB -> Export (TDL file)
Dumps active LexDB entries (determined by filter) to .tdl file.
HOW TO edit entries in the LexDB
The LexDB-Emacs interface allows editing of lexical entries from within an Emacs environment (with browsing functionality, field completion, etc.). New revision entries are first stored in the users private schema, and hence are visible only to the particular user.
0. Add the following (path adjusted for your setup) to your .emacs file:
(add-to-list 'load-path "/path/to/delphin/lkb/lexdb") (load "pg-interface")
1. In [http://www.gnu.org/software/emacs/emacs.html GNU Emacs]: M-x lexdb to enter LexDB major mode. Then see the PG menu.
Available commands in LexDB major mode are:
C-l : load (active revision of lexical entry) into Emacs
C-c C-c : commit (edited/new revision of) lexical entry into LexDB
TAB : field completion
M-TAB l : get (ring of) entries in table lex where value of current field matches that in buffer
M-TAB r : get (ring of) entries in union of rev tables where value of current field matches that in buffer
M-n : cycle through ring of entries obtained above
M-s : as M-TAB, but explicitly specify field value
M-va : view entries added in merge operation from dump file
M-vs : view entries in user's privat rev
To remove a lexical entry from the current lexicon lex, create a (head) revision where the dead field is set to t (true) rather than f (false). In this manner we keep a revision history even for entries which are no longer used (and such entries can be reactivated if necessary). No revision entry should ever be deleted from the lexical database itself.
HOW TO import TDL entries from a file
To add a small number of new (revision) entries from a .tdl file: LexDB -> Import (TDL file). You will be queried to provide values for other certain non-grammar fields. The entries will go into the private rev table.
HOW TO commit entries to public rev
The LexDB consists of a single public schema and a set of private schemas, one per user. New (revision) entries are placed initially in your private schema. To commit (all) entries in your private schema to the public table: LexDB -> Commit private 'rev'
HOW TO list entries in private rev
From LKB: LexDB -> Display private 'rev'
or
from Emacs LexDB major mode: M-vs
HOW TO clear entries in private rev
LexDB -> Clear private 'rev
If M-x lexdb results in "no connection to LexDb", make sure that a working grammar is loaded. Troubleshooting