Updating the Customization System
There are several steps to updating the customization system, although you won't necessarily do all of them for every change. We also have some instructions for committing or vivifying changes. There are also a few basic examples of things to play with if you are starting out.
Steps to Update the System
The Matrix questionnaire is a web application that presents a user-linguist with a linguistic-typological questions, and in response to the answers, the customization system creates a LKB-compatible grammar for the language described. When a developer wishes to make changes to the any part of the system, several different parts must all be updated at the same time in order to keep everything working.
The questionnaire is a CGI application written in Python that is stored in the Matrix svn repository (in matrix/gmcs/). Its most important files are:
The matrixdef file, which defines both the questionnaire and the list of choices -- that is, the name space of attributes and their ranges of values that are used to store the user's answers to the questionnaire.
matrix.cgi, a CGI script written in Python. The entry point for the questionnaire.
deffile.py, which defines the MatrixDefFile class. This class is responsible for creating web pages based on the contents of matrixdef.
choices.py, which defines the ChoicesFile class. This class is responsible for loading, saving, and providing access to the choices file, which contains the user's answers to the questionnaire.
validate.py, which contains code to confirm that the user's current choices are describe a coherent language. (For example, if the user says the language contains no determiners, they must not have defined one.)
customize.py, which contains the code that creates an LKB-compatible grammar based on the current choices.
tdl.py, which defines the TDLFile class.
Library modules in gmcs/linglib/
When there are changes to be made to the questionnaire, a likely course of action is the following:
Form fields for the new choices must be added to the matrixdef file.
If necessary, support for new form field types or behavior (e.g. a drop-down box) must be added to deffile.py.
If any old choices have become obsolete, the version number in choices.py (specifically, in the current_version method on the ChoicesFile class) must be incremented. Furthermore, a new up-revving method that converts choices files of version n-1 to the current version n must be added. See the comments in choices.py for details.
- Install the system to a test location (NOT the live site), then verify the changes look as expected
Write appropriate methods in validate.py (or a library's validate() method)
When changes are made to libraries:
- The linguistic phenomenon to be covered must be identified, along with its range of possible patterns in the world's languages, and based on this range a set of new choices (names and values) must be defined.
Write the processing code in customize.py (and/or in a library-specific module)
Write unit tests for the code
Write regression tests to test the output
When changing the underlying infrastructure:
Write the changes to matrix.cgi, customize.py, choices.py, or whatever module is needing change
Definitely write unit tests for these kinds of changes
- Update the documentation as necessary (e.g. these wiki pages)
Don't forget to commit the changes.
Installing a Test Instance of the Customization System
You can install an instance of the Grammar Customization System in a test location for local development using matrix.py
python matrix.py -C /path/to/local/gmcs install /path/to/install/location
Make sure there is a sessions directory where you install the matrix; if there is not, create it yourself and make writeable. It seems like without it, you can't run matrix.cgi from localhost. E.g. install the instance to a directory called matrix, and inside that directory, also create a directory sessions, and make it writealbe (e.g. chmod a+w sessions).
Committing and Vivifying Changes
- svn update
- resolve conflicts
run the regression tests: MatrixRegressionTesting
- fix any regressions (repeat as necessary)
- svn diff each file to find diffs and note in text file for svn comment
- svn update
- svn commit
- Close any associated Trac tickets
Vivify to the live site with the following command:
python matrix.py vivify
Caution: The vivify command doesn't know about svn, and uploads whatever is in your $CUSTOMIZATIONROOT directory. Make sure to vivify from a clean trunk!
Regarding Trac and SVN
We use a Trac instance for our bug/ticket tracking. Trac has some convenient functions, and it's good practice to make use of them. For instance, 'r' followed by a number generates a link to Trac's source code browser for that revision, and '#' followed by a number generates a link to a Trac ticket. A good way to use these is:
- In your SVN commit message, point to the Trac ticket it addresses, such as:
matrix.py: Added the ability to remove and rename regression tests. Fixes Trac #78
- When you close or modify a Trac ticket, point to the code revision of changes related to that ticket, such as:
Completed in r18481
- This way, users viewing tickets or code in Trac can easily see the related changes to the project for each revision or bug.
Occasionally you may find upon merging a branch to or from trunk that SVN claims files were modified that were not. This may be a problem with svn:mergeinfo properties on files, which get modified on merges (note, the property is modified, not necessarily the file itself). You can check this property with the following command:
svn propget svn:mergeinfo path/
In the Matrix repository, we merge to/from trunk and branches, not to/from subdirectories. If you merge to/from a subdirectory, spurious mergeinfo properties may be created, which will lead to these issues. These properties can be removed as follows:
svn propdel svn:mergeinfo path/
Be sure to remove the properties from all branches that copied them, or it might get copied back again.
Basic things to try
You can extend the subpages form with a placeholder for your new subpage. That's an easy and fun way to start making changes.
Suppose you are adding a Library called MyLibrary. In deffile.py, you will find the class MatrixDefFile's definition, with two dictionaries, called "sections" and "doclinks". Add a key-value pair for your library to each:
'mylib':'My Library', ...
Then, open web/matrixdef (it may look like a binary file but go ahead and open it in a text editor). There, add placeholder content between some existing sections. E.g.:
... Section mylib "My Library" This is a placeholder. ...
If you now deploy the updated files to the test location and renew the main matrix.cgi page, you should see "My Library" in the Sections form and should be able to click on the link and see the placeholder content.
Preliminary notes about attaching a debugger
- append the location of pycharm-debug.egg to your PYTHONPATH
create a remote-debug running configuration in PyCharm (note that you do need professional edition for this)
hit "debug" in PyCharm
- insert the import pydevd and the settrace statements (see the guide above) in the code (before the beginning of the main script part is fine)
- reload the webpage
-- You should see the debugger attach successfully and should be able to set breakpoints and step through the execution.