moved parts of doc to deprecated/doc

1 files changed, 827 insertions, 0 deletions

diff --git a/deprecated/Resource-HOWTO.txt b/deprecated/Resource-HOWTO.txt
new file mode 100644
index 000000000..8e50974a7
--- /dev/null
+++ b/deprecated/Resource-HOWTO.txt
@@ -0,0 +1,827 @@
+Resource grammar writing HOWTO
+Author: Aarne Ranta <aarne (at) cs.chalmers.se>
+Last update: %%date(%c)
+
+% NOTE: this is a txt2tags file.
+% Create an html file from this file using:
+% txt2tags --toc -thtml Resource-HOWTO.txt
+
+%!target:html
+
+**History**
+
+September 2008: updated for Version 1.5.
+
+October 2007: updated for Version 1.2.
+
+January 2006: first version.
+
+
+The purpose of this document is to tell how to implement the GF
+resource grammar API for a new language. We will //not// cover how
+to use the resource grammar, nor how to change the API. But we
+will give some hints how to extend the API.
+
+A manual for using the resource grammar is found in
+
+[``www.cs.chalmers.se/Cs/Research/Language-technology/GF/lib/resource/doc/synopsis.html`` ../lib/resource/doc/synopsis.html].
+
+A tutorial on GF, also introducing the idea of resource grammars, is found in
+
+[``www.cs.chalmers.se/Cs/Research/Language-technology/GF/doc/gf-tutorial.html`` ./gf-tutorial.html].
+
+This document concerns the API v. 1.5, while the current stable release is 1.4. 
+You can find the code for the stable release in 
+
+[``www.cs.chalmers.se/Cs/Research/Language-technology/GF/lib/resource/`` ../lib/resource]
+
+and the next release in
+
+[``www.cs.chalmers.se/Cs/Research/Language-technology/GF/next-lib/src/`` ../next-lib/src]
+
+It is recommended to build new grammars to match the next release.
+
+
+
+
+==The resource grammar structure==
+
+The library is divided into a bunch of modules, whose dependencies
+are given in the following figure.
+
+[Syntax.png] 
+
+Modules of different kinds are distinguished as follows:
+- solid contours: module seen by end users
+- dashed contours: internal module
+- ellipse: abstract/concrete pair of modules
+- rectangle: resource or instance
+- diamond: interface
+
+
+Put in another way:
+- solid rectangles and diamonds: user-accessible library API
+- solid ellipses: user-accessible top-level grammar for parsing and linearization
+- dashed contours: not visible to users
+
+
+The dashed ellipses form the main parts of the implementation, on which the resource
+grammar programmer has to work with. She also has to work on the ``Paradigms``
+module. The rest of the modules can be produced mechanically from corresponding
+modules for other languages, by just changing the language codes appearing in
+their module headers.
+
+The module structure is rather flat: most modules are direct
+parents of ``Grammar``. The idea
+is that the implementors can concentrate on one linguistic aspect at a time, or
+also distribute the work among several authors. The module ``Cat``
+defines the "glue" that ties the aspects together - a type system
+to which all the other modules conform, so that e.g. ``NP`` means
+the same thing in those modules that use ``NP``s and those that
+constructs them.
+
+
+===Library API modules===
+
+For the user of the library, these modules are the most important ones.
+In a typical application, it is enough to open ``Paradigms`` and ``Syntax``.
+The module ``Try`` combines these two, making it possible to experiment
+with combinations of syntactic and lexical constructors by using the
+``cc`` command in the GF shell. Here are short explanations of each API module:
+- ``Try``: the whole resource library for a language (``Paradigms``, ``Syntax``,
+  ``Irreg``, and ``Extra``); 
+  produced mechanically as a collection of modules
+- ``Syntax``: language-independent categories, syntax functions, and structural words;
+  produced mechanically as a collection of modules
+- ``Constructors``: language-independent syntax functions and structural words;
+  produced mechanically via functor instantiation
+- ``Paradigms``: language-dependent morphological paradigms
+
+
+
+
+
+===Phrase category modules===
+
+The immediate parents of ``Grammar`` will be called **phrase category modules**,
+since each of them concentrates on a particular phrase category (nouns, verbs,
+adjectives, sentences,...). A phrase category module tells 
+//how to construct phrases in that category//. You will find out that
+all functions in any of these modules have the same value type (or maybe
+one of a small number of different types). Thus we have
+- ``Noun``: construction of nouns and noun phrases
+- ``Adjective``: construction of adjectival phrases
+- ``Verb``: construction of verb phrases
+- ``Adverb``: construction of adverbial phrases
+- ``Numeral``: construction of cardinal and ordinal numerals
+- ``Sentence``: construction of sentences and imperatives
+- ``Question``: construction of questions
+- ``Relative``: construction of relative clauses
+- ``Conjunction``: coordination of phrases
+- ``Phrase``: construction of the major units of text and speech
+- ``Text``: construction of texts as sequences of phrases
+- ``Idiom``: idiomatic expressions such as existentials
+
+
+
+
+===Infrastructure modules===
+
+Expressions of each phrase category are constructed in the corresponding
+phrase category module. But their //use// takes mostly place in other modules.
+For instance, noun phrases, which are constructed in ``Noun``, are
+used as arguments of functions of almost all other phrase category modules. 
+How can we build all these modules independently of each other?
+
+As usual in typeful programming, the //only// thing you need to know
+about an object you use is its type. When writing a linearization rule
+for a GF abstract syntax function, the only thing you need to know is
+the linearization types of its value and argument categories. To achieve
+the division of the resource grammar to several parallel phrase category modules,
+what we need is an underlying definition of the linearization types. This
+definition is given as the implementation of
+- ``Cat``: syntactic categories of the resource grammar
+
+
+Any resource grammar implementation has first to agree on how to implement
+``Cat``. Luckily enough, even this can be done incrementally: you
+can skip the ``lincat`` definition of a category and use the default
+``{s : Str}`` until you need to change it to something else. In
+English, for instance, many categories do have this linearization type.
+
+
+
+===Lexical modules===
+
+What is lexical and what is syntactic is not as clearcut in GF as in
+some other grammar formalisms. Logically, lexical means atom, i.e. a
+``fun`` with no arguments. Linguistically, one may add to this
+that the ``lin`` consists of only one token (or of a table whose values
+are single tokens). Even in the restricted lexicon included in the resource
+API, the latter rule is sometimes violated in some languages. For instance,
+``Structural.both7and_DConj`` is an atom, but its linearization is
+two words e.g. //both - and//.
+
+Another characterization of lexical is that lexical units can be added
+almost //ad libitum//, and they cannot be defined in terms of already
+given rules. The lexical modules of the resource API are thus more like
+samples than complete lists. There are two such modules:
+- ``Structural``: structural words (determiners, conjunctions,...)
+- ``Lexicon``: basic everyday content words (nouns, verbs,...)
+
+
+The module ``Structural`` aims for completeness, and is likely to
+be extended in future releases of the resource. The module ``Lexicon``
+gives a "random" list of words, which enables testing the syntax.
+It also provides a check list for morphology, since those words are likely to include
+most morphological patterns of the language.
+
+In the case of ``Lexicon`` it may come out clearer than anywhere else
+in the API that it is impossible to give exact translation equivalents in
+different languages on the level of a resource grammar. This is no problem,
+since application grammars can use the resource in different ways for
+different languages.
+
+
+==Language-dependent syntax modules==
+
+In addition to the common API, there is room for language-dependent extensions
+of the resource. The top level of each languages looks as follows (with German 
+as example):
+```
+  abstract AllGerAbs = Lang, ExtraGerAbs, IrregGerAbs
+```
+where ``ExtraGerAbs`` is a collection of syntactic structures specific to German,
+and ``IrregGerAbs`` is a dictionary of irregular words of German 
+(at the moment, just verbs). Each of these language-specific grammars has 
+the potential to grow into a full-scale grammar of the language. These grammar
+can also be used as libraries, but the possibility of using functors is lost.
+
+To give a better overview of language-specific structures, 
+modules like ``ExtraGerAbs``
+are built from a language-independent module ``ExtraAbs`` 
+by restricted inheritance:
+```
+  abstract ExtraGerAbs = Extra [f,g,...]
+```
+Thus any category and function in ``Extra`` may be shared by a subset of all
+languages. One can see this set-up as a matrix, which tells 
+what ``Extra`` structures
+are implemented in what languages. For the common API in ``Grammar``, the matrix
+is filled with 1's (everything is implemented in every language).
+
+In a minimal resource grammar implementation, the language-dependent
+extensions are just empty modules, but it is good to provide them for
+the sake of uniformity.
+
+
+
+===The present-tense fragment===
+
+Some lines in the resource library are suffixed with the comment
+```
+  --# notpresent
+```
+which is used by a preprocessor to exclude those lines from 
+a reduced version of the full resource. This present-tense-only
+version is useful for applications in most technical text, since
+they reduce the grammar size and compilation time. It can also
+be useful to exclude those lines in a first version of resource
+implementation. To compile a grammar with present-tense-only, use
+```
+  make Present
+```
+with ``resource/Makefile``.
+
+
+
+==Phases of the work==
+
+===Putting up a directory===
+
+Unless you are writing an instance of a parametrized implementation
+(Romance or Scandinavian), which will be covered later, the
+simplest way is to follow roughly the following procedure. Assume you
+are building a grammar for the German language. Here are the first steps,
+which we actually followed ourselves when building the German implementation
+of resource v. 1.0 at Ubuntu linux. We have slightly modified them to
+match resource v. 1.5 and GF v. 3.0.
+
++ Create a sister directory for ``GF/lib/resource/english``, named
+     ``german``.
+```
+       cd GF/lib/resource/
+       mkdir german
+       cd german
+```
+
++ Check out the [ISO 639 3-letter language code 
+   http://www.w3.org/WAI/ER/IG/ert/iso639.htm] 
+   for German: both ``Ger`` and ``Deu`` are given, and we pick ``Ger``.
+   (We use the 3-letter codes rather than the more common 2-letter codes,
+    since they will suffice for many more languages!)
+
++ Copy the ``*Eng.gf`` files from ``english`` ``german``,
+     and rename them:
+```
+       cp ../english/*Eng.gf .
+       rename 's/Eng/Ger/' *Eng.gf
+```
+  If you don't have the ``rename`` command, you can use a bash script with ``mv``.
+
+
++ Change the ``Eng`` module references to ``Ger`` references
+     in all files:
+```
+       sed -i 's/English/German/g' *Ger.gf
+       sed -i 's/Eng/Ger/g' *Ger.gf
+```
+  The first line prevents changing the word ``English``, which appears
+  here and there in comments, to ``Gerlish``. The ``sed`` command syntax
+  may vary depending on your operating system.
+
++ This may of course change unwanted occurrences of the 
+     string ``Eng`` - verify this by
+```
+       grep Ger *.gf
+```
+     But you will have to make lots of manual changes in all files anyway!
+
++ Comment out the contents of these files:
+``` 
+       sed -i 's/^/--/' *Ger.gf
+```
+     This will give you a set of templates out of which the grammar
+     will grow as you uncomment and modify the files rule by rule.
+
++ In all ``.gf`` files, uncomment the module headers and brackets,
+  leaving the module bodies commented. Unfortunately, there is no
+  simple way to do this automatically (or to avoid commenting these
+  lines in the previous step) - but uncommenting the first
+  and the last lines will actually do the job for many of the files.
+
++ Uncomment the contents of the main grammar file:
+``` 
+       sed -i 's/^--//' LangGer.gf
+```
+
++ Now you can open the grammar ``LangGer`` in GF:
+``` 
+       gf LangGer.gf
+```
+  You will get lots of warnings on missing rules, but the grammar will compile.
+
++ At all the following steps you will now have a valid, but incomplete
+     GF grammar. The GF command
+``` 
+       pg -missing
+```
+     tells you what exactly is missing.
+
+
+Here is the module structure of ``LangGer``. It has been simplified by leaving out
+the majority of the phrase category modules. Each of them has the same dependencies
+as ``VerbGer``, whose complete dependencies are shown as an example.
+
+[German.png]
+
+
+===Direction of work===
+
+The real work starts now. There are many ways to proceed, the most obvious ones being
+- Top-down: start from the module ``Phrase`` and go down to ``Sentence``, then
+  ``Verb``, ``Noun``, and in the end ``Lexicon``. In this way, you are all the time
+  building complete phrases, and add them with more content as you proceed.
+  **This approach is not recommended**. It is impossible to test the rules if
+  you have no words to apply the constructions to.
+
+- Bottom-up: set as your first goal to implement ``Lexicon``. To this end, you
+  need to write ``ParadigmsGer``, which in turn needs parts of 
+  ``MorphoGer`` and ``ResGer``.
+  **This approach is not recommended**. You can get stuck to details of
+  morphology such as irregular words, and you don't have enough grasp about
+  the type system to decide what forms to cover in morphology.
+
+
+The practical working direction is thus a saw-like motion between the morphological
+and top-level modules. Here is a possible course of the work that gives enough
+test data and enough general view at any point:
++ Define ``Cat.N`` and the required parameter types in ``ResGer``. As we define
+```
+  lincat N  = {s : Number => Case => Str ; g : Gender} ;
+```
+we need the parameter types ``Number``, ``Case``, and ``Gender``. The definition
+of ``Number`` in [``common/ParamX``  ../lib/resource/common/ParamX.gf] 
+works for German, so we
+use it and just define ``Case`` and ``Gender`` in ``ResGer``.
+
++ Define some cases of ``mkN`` in ``ParadigmsGer``. In this way you can 
+already implement a huge amount of nouns correctly in ``LexiconGer``. Actually
+just adding the worst-case instance of ``mkN`` (the one taking the most
+arguments) should suffice for every noun - but, 
+since it is tedious to use, you
+might proceed to the next step before returning to morphology and defining the
+real work horse, ``mkN`` taking two forms and a gender.
+
++ While doing this, you may want to test the resource independently. Do this by
+  starting the GF shell in the ``resource`` directory, by the commands
+```
+  > i -retain german/ParadigmsGer
+  > cc -table mkN "Kirche"
+```
+
++ Proceed to determiners and pronouns in 
+``NounGer`` (``DetCN UsePron DetQuant NumSg DefArt IndefArt UseN``) and 
+``StructuralGer`` (``i_Pron this_Quant``). You also need some categories and
+parameter types. At this point, it is maybe not possible to find out the final
+linearization types of ``CN``, ``NP``, ``Det``, and ``Quant``, but at least you should
+be able to correctly inflect noun phrases such as //every airplane//:
+```
+  > i german/LangGer.gf
+  > l -table DetCN every_Det (UseN airplane_N)
+
+  Nom: jeder Flugzeug
+  Acc: jeden Flugzeug
+  Dat: jedem Flugzeug
+  Gen: jedes Flugzeugs
+```
+
++ Proceed to verbs: define ``CatGer.V``,  ``ResGer.VForm``, and
+``ParadigmsGer.mkV``. You may choose to exclude ``notpresent``
+cases at this point. But anyway, you will be able to inflect a good
+number of verbs in ``Lexicon``, such as
+``live_V`` (``mkV "leben"``).
+
++ Now you can soon form your first sentences: define ``VP`` and
+``Cl`` in ``CatGer``, ``VerbGer.UseV``, and ``SentenceGer.PredVP``.
+Even if you have excluded the tenses, you will be able to produce
+```
+  > i -preproc=./mkPresent german/LangGer.gf
+  > l -table PredVP (UsePron i_Pron) (UseV live_V)
+
+  Pres Simul Pos Main: ich lebe
+  Pres Simul Pos Inv:  lebe ich
+  Pres Simul Pos Sub:  ich lebe
+  Pres Simul Neg Main: ich lebe nicht
+  Pres Simul Neg Inv:  lebe ich nicht
+  Pres Simul Neg Sub:  ich nicht lebe
+```
+You should also be able to parse:
+```
+  > p -cat=Cl "ich lebe"
+  PredVP (UsePron i_Pron) (UseV live_V)
+```
+
++ Transitive verbs 
+(``CatGer.V2 CatGer.VPSlash ParadigmsGer.mkV2 VerbGer.ComplSlash VerbGer.SlashV2a``) 
+are a natural next step, so that you can
+produce ``ich liebe dich`` ("I love you").
+
++ Adjectives (``CatGer.A ParadigmsGer.mkA NounGer.AdjCN AdjectiveGer.PositA``) 
+will force you to think about strong and weak declensions, so that you can
+correctly inflect //mein neuer Wagen, dieser neue Wagen// 
+("my new car, this new car"). 
+
++ Once you have implemented the set
+(``Noun.DetCN Noun.AdjCN Verb.UseV Verb.ComplSlash Verb.SlashV2a Sentence.PredVP),
+you have overcome most of difficulties. You know roughly what parameters
+and dependences there are in your language, and you can now proceed very
+much in the order you please. 
+
+
+
+===The develop-test cycle===
+
+The following develop-test cycle will
+be applied most of the time, both in the first steps described above
+and in later steps where you are more on your own.
+
++ Select a phrase category module, e.g. ``NounGer``, and uncomment some
+  linearization rules (for instance, ``DetCN``, as above).
+
++ Write down some German examples of this rule, for instance translations
+     of "the dog", "the house", "the big house", etc. Write these in all their
+     different forms (two numbers and four cases).
+
++ Think about the categories involved (``CN, NP, N, Det``) and the
+     variations they have. Encode this in the lincats of ``CatGer``.
+     You may have to define some new parameter types in ``ResGer``.
+
++ To be able to test the construction, 
+     define some words you need to instantiate it
+     in ``LexiconGer``. You will also need some regular inflection patterns
+     in``ParadigmsGer``.
+
++ Test by parsing, linearization,
+     and random generation. In particular, linearization to a table should
+     be used so that you see all forms produced; the ``treebank`` option
+     preserves the tree
+```
+    > gr -cat=NP -number=20 | l -table -treebank
+```
+
++ Save some tree-linearization pairs for later regression testing. You can save
+  a gold standard treebank and use the Unix ``diff`` command to compare later
+  linearizations produced from the same list of trees. If you save the trees
+  in a file ``trees``, you can do as follows:
+```
+    > rf -file=trees -tree -lines | l -table -treebank | wf -file=treebank
+```
+
++ A file with trees testing all resource functions is included in the resource,
+  entitled ``resource/exx-resource.gft``. A treebank can be created from this by
+  the Unix command
+```
+  % runghc Make.hs test langs=Ger
+```
+
+
+
+You are likely to run this cycle a few times for each linearization rule
+you implement, and some hundreds of times altogether. There are roughly
+70 ``cat``s and
+600 ``funs`` in ``Lang`` at the moment; 170 of the ``funs`` are outside the two
+lexicon modules).
+
+
+===Auxiliary modules===
+
+These auxuliary ``resource`` modules will be written by you.
+
+- ``ResGer``: parameter types and auxiliary operations 
+(a resource for the resource grammar!)
+- ``ParadigmsGer``: complete inflection engine and most important regular paradigms
+- ``MorphoGer``: auxiliaries for ``ParadigmsGer`` and ``StructuralGer``. This need
+not be separate from ``ResGer``.
+
+
+These modules are language-independent and provided by the existing resource
+package.
+
+- ``ParamX``: parameter types used in many languages
+- ``CommonX``: implementation of language-uniform categories 
+    such as $Text$ and $Phr$, as well as of
+    the logical tense, anteriority, and polarity parameters
+- ``Coordination``: operations to deal with lists and coordination
+- ``Prelude``: general-purpose operations on strings, records,
+      truth values, etc.
+- ``Predef``: general-purpose operations with hard-coded definitions
+
+
+An important decision is what rules to implement in terms of operations in
+``ResGer``. The **golden rule of functional programming** says:
+- //Whenever you find yourself programming by copy and paste, write a function instead!//. 
+
+
+This rule suggests that an operation should be created if it is to be
+used at least twice. At the same time, a sound principle of **vicinity** says: 
+- //It should not require too much browsing to understand what a piece of code does.//
+
+
+From these two principles, we have derived the following practice:
+- If an operation is needed //in two different modules//, 
+  it should be created in as an ``oper`` in ``ResGer``. An example is ``mkClause``, 
+  used in ``Sentence``, ``Question``, and ``Relative``-
+- If an operation is needed //twice in the same module//, but never
+  outside, it should be created in the same module. Many examples are
+  found in ``Numerals``.
+- If an operation is needed //twice in the same judgement//, but never
+  outside, it should be created by a ``let`` definition. 
+- If an operation is only needed once, it should not be created as an ``oper``, 
+  but rather inlined. However, a ``let`` definition may well be in place just
+  to make the readable. 
+  Most functions in phrase category modules 
+  are implemented in this way. 
+
+
+This discipline is very different from the one followed in early
+versions of the library (up to 0.9). We then valued the principle of
+abstraction more than vicinity, creating layers of abstraction for
+almost everything. This led in practice to the duplication of almost
+all code on the ``lin`` and ``oper`` levels, and made the code
+hard to understand and maintain.
+
+
+
+===Morphology and lexicon===
+
+The paradigms needed to implement
+``LexiconGer`` are defined in
+``ParadigmsGer``.
+This module provides high-level ways to define the linearization of
+lexical items, of categories ``N, A, V`` and their complement-taking
+variants.
+
+For ease of use, the ``Paradigms`` modules follow a certain
+naming convention. Thus they for each lexical category, such as ``N``,
+the overloaded functions, such as ``mkN``, with the following cases:
+
+- the worst-case construction of ``N``. Its type signature
+     has the form
+```
+       mkN : Str -> ... -> Str -> P -> ... -> Q -> N
+```
+     with as many string and parameter arguments as can ever be needed to
+     construct an ``N``.
+- the most regular cases, with just one string argument:
+```
+       mkN : Str -> N
+```
+- A language-dependent (small) set of functions to handle mild irregularities
+     and common exceptions.
+
+
+For the complement-taking variants, such as ``V2``, we provide
+- a case that takes a ``V`` and all necessary arguments, such
+     as case and preposition:
+```
+       mkV2 : V -> Case -> Str -> V2 ;
+```
+- a case that takes a ``Str`` and produces a transitive verb with the direct
+  object case:
+```
+       mkV2 : Str -> V2 ;
+```
+- A language-dependent (small) set of functions to handle common special cases,
+  such as transitive verbs that are not regular:
+```
+       mkV2 : V -> V2 ;
+```
+
+
+The golden rule for the design of paradigms is that
+- //The user of the library will only need function applications with constants and strings, never any records or tables.//
+
+
+The discipline of data abstraction moreover requires that the user of the resource
+is not given access to parameter constructors, but only to constants that denote 
+them. This gives the resource grammarian the freedom to change the underlying
+data representation if needed. It means that the ``ParadigmsGer`` module has
+to define constants for those parameter types and constructors that 
+the application grammarian may need to use, e.g.
+```
+  oper 
+    Case : Type ;
+    nominative, accusative, genitive, dative : Case ;
+```
+These constants are defined in terms of parameter types and constructors
+in ``ResGer`` and ``MorphoGer``, which modules are not
+visible to the application grammarian.
+
+
+===Lock fields===
+
+An important difference between ``MorphoGer`` and
+``ParadigmsGer`` is that the former uses "raw" record types
+for word classes, whereas the latter used category symbols defined in
+``CatGer``. When these category symbols are used to denote
+record types in a resource modules, such as ``ParadigmsGer``,
+a **lock field** is added to the record, so that categories
+with the same implementation are not confused with each other.
+(This is inspired by the ``newtype`` discipline in Haskell.)
+For instance, the lincats of adverbs and conjunctions are the same
+in ``CommonX`` (and therefore in ``CatGer``, which inherits it):
+```
+  lincat Adv  = {s : Str} ;
+  lincat Conj = {s : Str} ;
+```
+But when these category symbols are used to denote their linearization 
+types in resource module, these definitions are translated to
+```
+  oper Adv  : Type = {s : Str  ; lock_Adv  : {}} ;
+  oper Conj : Type = {s : Str} ; lock_Conj : {}} ;
+```
+In this way, the user of a resource grammar cannot confuse adverbs with
+conjunctions. In other words, the lock fields force the type checker
+to function as grammaticality checker.
+
+When the resource grammar is ``open``ed in an application grammar, the
+lock fields are never seen (except possibly in type error messages),
+and the application grammarian should never write them herself. If she
+has to do this, it is a sign that the resource grammar is incomplete, and
+the proper way to proceed is to fix the resource grammar.
+
+The resource grammarian has to provide the dummy lock field values
+in her hidden definitions of constants in ``Paradigms``. For instance,
+```
+  mkAdv : Str -> Adv ;
+  -- mkAdv s = {s = s ; lock_Adv = <>} ;
+```
+
+
+===Lexicon construction===
+
+The lexicon belonging to ``LangGer`` consists of two modules:
+- ``StructuralGer``, structural words, built by using both
+  ``ParadigmsGer`` and ``MorphoGer``.
+- ``LexiconGer``, content words, built by using ``ParadigmsGer`` only.
+
+
+The reason why ``MorphoGer`` has to be used in ``StructuralGer``
+is that ``ParadigmsGer`` does not contain constructors for closed
+word classes such as pronouns and determiners. The reason why we
+recommend ``ParadigmsGer`` for building ``LexiconGer`` is that
+the coverage of the paradigms gets thereby tested and that the
+use of the paradigms in ``LexiconGer`` gives a good set of examples for
+those who want to build new lexica.
+
+
+
+
+
+==Lexicon extension==
+
+===The irregularity lexicon===
+
+It is useful in most languages to provide a separate module of irregular
+verbs and other words which are difficult for a lexicographer
+to handle. There are usually a limited number of such words - a
+few hundred perhaps. Building such a lexicon separately also
+makes it less important to cover //everything// by the
+worst-case variants of the paradigms ``mkV`` etc.
+
+
+
+===Lexicon extraction from a word list===
+
+You can often find resources such as lists of 
+irregular verbs on the internet. For instance, the
+Irregular German Verb page 
+previously found in 
+``http://www.iee.et.tu-dresden.de/~wernerr/grammar/verben_dt.html``
+page gives a list of verbs in the
+traditional tabular format, which begins as follows:
+```
+  backen (du bäckst, er bäckt)                   backte [buk]              gebacken
+  befehlen (du befiehlst, er befiehlt; befiehl!) befahl (beföhle; befähle) befohlen
+  beginnen                                       begann (begönne; begänne) begonnen
+  beißen                                         biß                       gebissen
+```
+All you have to do is to write a suitable verb paradigm
+```
+  irregV : (x1,_,_,_,_,x6 : Str) -> V ;
+```
+and a Perl or Python or Haskell script that transforms
+the table to
+```
+  backen_V   = irregV "backen" "bäckt" "back" "backte" "backte" "gebacken" ;
+  befehlen_V = irregV "befehlen" "befiehlt" "befiehl" "befahl" "beföhle" "befohlen" ;
+```
+
+When using ready-made word lists, you should think about
+coyright issues. All resource grammar material should
+be provided under GNU Lesser General Public License (LGPL).
+
+
+
+===Lexicon extraction from raw text data===
+
+This is a cheap technique to build a lexicon of thousands
+of words, if text data is available in digital format.
+See the [Extract Homepage http://www.cs.chalmers.se/~markus/extract/] 
+homepage for details.
+
+
+===Bootstrapping with smart paradigms===
+
+This is another cheap technique, where you need as input a list of words with
+part-of-speech marking. You initialize the lexicon by using the one-argument
+``mkN`` etc paradigms, and add forms to those words that do not come out right.
+This procedure is described in the paper
+
+A. Ranta.
+How predictable is Finnish morphology? An experiment on lexicon construction.
+In J. Nivre, M. Dahllöf and B. Megyesi (eds),
+//Resourceful Language Technology: Festschrift in Honor of Anna Sågvall Hein//,
+University of Uppsala,
+2008.
+Available from the [series homepage http://publications.uu.se/abstract.xsql?dbid=8933]
+
+
+
+
+==Extending the resource grammar API==
+
+Sooner or later it will happen that the resource grammar API
+does not suffice for all applications. A common reason is
+that it does not include idiomatic expressions in a given language.
+The solution then is in the first place to build language-specific
+extension modules, like ``ExtraGer``. 
+
+==Using parametrized modules==
+
+===Writing an instance of parametrized resource grammar implementation===
+
+Above we have looked at how a resource implementation is built by
+the copy and paste method (from English to German), that is, formally
+speaking, from scratch. A more elegant solution available for 
+families of languages such as Romance and Scandinavian is to
+use parametrized modules. The advantages are
+- theoretical: linguistic generalizations and insights
+- practical: maintainability improves with fewer components
+
+
+Here is a set of
+[slides http://www.cs.chalmers.se/~aarne/geocal2006.pdf]
+on the topic.
+
+
+===Parametrizing a resource grammar implementation===
+
+This is the most demanding form of resource grammar writing.
+We do //not// recommend the method of parametrizing from the
+beginning: it is easier to have one language first implemented 
+in the conventional way and then add another language of the
+same family by aprametrization. This means that the copy and
+paste method is still used, but at this time the differences
+are put into an ``interface`` module. 
+
+
+==Character encoding and transliterations==
+
+This section is relevant for languages using a non-ASCII character set. 
+
+==Coding conventions in GF==
+
+From version 3.0, GF follows a simple encoding convention:
+- GF source files may follow any encoding, such as isolatin-1 or UTF-8;
+  the default is isolatin-1, and UTF8 must be indicated by the judgement
+```
+    flags coding = utf8 ;
+``` 
+  in each source module.
+- for internal processing, all characters are converted to 16-bit unicode, 
+  as the first step of grammar compilation guided by the ``coding`` flag
+- as the last step of compilation, all characters are converted to UTF-8
+- thus, GF object files (``gfo``) and the Portable Grammar Format (``pgf``)
+  are in UTF-8
+
+
+Most current resource grammars use isolatin-1 in the source, but this does
+not affect their use in parallel with grammars written in other encodings.
+In fact, a grammar can be put up from modules using different codings.
+
+**Warning**. While string literals may contain any characters, identifiers
+must be isolatin-1 letters (or digits, underscores, or dashes). This has to
+do with the restrictions of the lexer tool that is used.
+
+
+==Transliterations==
+
+While UTF-8 is well supported by most web browsers, its use in terminals and
+text editors may cause disappointment. Many grammarians therefore prefer to
+use ASCII transliterations. GF 3.0beta2 provides the following built-in
+transliterations:
+- Arabic
+- Devanagari (Hindi)
+- Thai
+
+
+New transliterations can be defined in the GF source file
+[``GF/Text/Transliterations.hs`` ../src/GF/Text/Transliterations.hs].
+This file also gives instructions on how new ones are added.
+
+
+
+
+