Updating the Customization System

There are several steps to updating the customization system, although you won't necessarily do all of them for every change. We also have some instructions for committing or vivifying changes. There are also a few basic examples of things to play with if you are starting out.

Steps to Update the System

The Matrix questionnaire is a web application that presents a user-linguist with a linguistic-typological questions, and in response to the answers, the customization system creates a LKB-compatible grammar for the language described. When a developer wishes to make changes to the any part of the system, several different parts must all be updated at the same time in order to keep everything working.

The questionnaire is a CGI application written in Python that is stored in the Matrix svn repository (in matrix/gmcs/). Its most important files are:

When there are changes to be made to the questionnaire, a likely course of action is the following:

  1. Form fields for the new choices must be added to the matrixdef file.

    1. If necessary, support for new form field types or behavior (e.g. a drop-down box) must be added to deffile.py.

  2. If any old choices have become obsolete, the version number in choices.py (specifically, in the current_version method on the ChoicesFile class) must be incremented. Furthermore, a new up-revving method that converts choices files of version n-1 to the current version n must be added. See the comments in choices.py for details.

  3. Install the system to a test location (NOT the live site), then verify the changes look as expected
  4. Write appropriate methods in validate.py (or a library's validate() method)

When changes are made to libraries:

  1. The linguistic phenomenon to be covered must be identified, along with its range of possible patterns in the world's languages, and based on this range a set of new choices (names and values) must be defined.
  2. Write the processing code in customize.py (and/or in a library-specific module)

  3. Write unit tests for the code

  4. Write regression tests to test the output

When changing the underlying infrastructure:

  1. Write the changes to matrix.cgi, customize.py, choices.py, or whatever module is needing change

  2. Definitely write unit tests for these kinds of changes

  3. Update the documentation as necessary (e.g. these wiki pages)

Don't forget to commit the changes.

Installing a Test Instance of the Customization System

You can install an instance of the Grammar Customization System in a test location for local development using matrix.py

python matrix.py -C /path/to/local/gmcs install /path/to/install/location

Make sure there is a sessions directory where you install the matrix; if there is not, create it yourself and make writeable. It seems like without it, you can't run matrix.cgi from localhost. E.g. install the instance to a directory called matrix, and inside that directory, also create a directory sessions, and make it writealbe (e.g. chmod a+w sessions).

Committing and Vivifying Changes

  1. svn update
  2. resolve conflicts
  3. run the regression tests: MatrixRegressionTesting

  4. fix any regressions (repeat as necessary)
  5. svn diff each file to find diffs and note in text file for svn comment
  6. svn update
  7. svn commit
  8. Close any associated Trac tickets

Vivify to the live site with the following command:

python matrix.py vivify

Caution: The vivify command doesn't know about svn, and uploads whatever is in your $CUSTOMIZATIONROOT directory. Make sure to vivify from a clean trunk!

Regarding Trac and SVN

We use a Trac instance for our bug/ticket tracking. Trac has some convenient functions, and it's good practice to make use of them. For instance, 'r' followed by a number generates a link to Trac's source code browser for that revision, and '#' followed by a number generates a link to a Trac ticket. A good way to use these is:

matrix.py: Added the ability to remove and rename regression tests.
           Fixes Trac #78

Completed in r18481

Merge Issues

Occasionally you may find upon merging a branch to or from trunk that SVN claims files were modified that were not. This may be a problem with svn:mergeinfo properties on files, which get modified on merges (note, the property is modified, not necessarily the file itself). You can check this property with the following command:

svn propget svn:mergeinfo path/

In the Matrix repository, we merge to/from trunk and branches, not to/from subdirectories. If you merge to/from a subdirectory, spurious mergeinfo properties may be created, which will lead to these issues. These properties can be removed as follows:

svn propdel svn:mergeinfo path/

Be sure to remove the properties from all branches that copied them, or it might get copied back again.

Basic things to try

You can extend the subpages form with a placeholder for your new subpage. That's an easy and fun way to start making changes.

Suppose you are adding a Library called MyLibrary. In deffile.py, you will find the class MatrixDefFile's definition, with two dictionaries, called "sections" and "doclinks". Add a key-value pair for your library to each:

'mylib':'My Library',
...

Then, open web/matrixdef (it may look like a binary file but go ahead and open it in a text editor). There, add placeholder content between some existing sections. E.g.:

...
Section mylib "My Library"
This is a placeholder.
...

If you now deploy the updated files to the test location and renew the main matrix.cgi page, you should see "My Library" in the Sections form and should be able to click on the link and see the placeholder content.

Preliminary notes about attaching a debugger

In general, this seems to be a good guide. In practice, what you need to do in, say, PyCharm is this (assuming your copy of the Matrix webpage is simply on your localhost):

-- You should see the debugger attach successfully and should be able to set breakpoints and step through the execution.

Updating_the_Customization_System (last edited 2017-09-11 21:59:47 by OlgaZamaraeva)

(The DELPH-IN infrastructure is hosted at the University of Oslo)