diff options
| author | aarne <aarne@chalmers.se> | 2010-12-22 14:08:42 +0000 |
|---|---|---|
| committer | aarne <aarne@chalmers.se> | 2010-12-22 14:08:42 +0000 |
| commit | ce15ec7b787479ca4c7295863ea7fa5cfdd16755 (patch) | |
| tree | f47d9227ab535781d44d00e6232b8c62902167df /doc/Resource-HOWTO.txt | |
| parent | fb722fe8e2cedee3b42d7fb0c9da61ace74f3e22 (diff) | |
moved parts of doc to deprecated/doc
Diffstat (limited to 'doc/Resource-HOWTO.txt')
| -rw-r--r-- | doc/Resource-HOWTO.txt | 827 |
1 files changed, 0 insertions, 827 deletions
diff --git a/doc/Resource-HOWTO.txt b/doc/Resource-HOWTO.txt deleted file mode 100644 index 8e50974a7..000000000 --- a/doc/Resource-HOWTO.txt +++ /dev/null @@ -1,827 +0,0 @@ -Resource grammar writing HOWTO -Author: Aarne Ranta <aarne (at) cs.chalmers.se> -Last update: %%date(%c) - -% NOTE: this is a txt2tags file. -% Create an html file from this file using: -% txt2tags --toc -thtml Resource-HOWTO.txt - -%!target:html - -**History** - -September 2008: updated for Version 1.5. - -October 2007: updated for Version 1.2. - -January 2006: first version. - - -The purpose of this document is to tell how to implement the GF -resource grammar API for a new language. We will //not// cover how -to use the resource grammar, nor how to change the API. But we -will give some hints how to extend the API. - -A manual for using the resource grammar is found in - -[``www.cs.chalmers.se/Cs/Research/Language-technology/GF/lib/resource/doc/synopsis.html`` ../lib/resource/doc/synopsis.html]. - -A tutorial on GF, also introducing the idea of resource grammars, is found in - -[``www.cs.chalmers.se/Cs/Research/Language-technology/GF/doc/gf-tutorial.html`` ./gf-tutorial.html]. - -This document concerns the API v. 1.5, while the current stable release is 1.4. -You can find the code for the stable release in - -[``www.cs.chalmers.se/Cs/Research/Language-technology/GF/lib/resource/`` ../lib/resource] - -and the next release in - -[``www.cs.chalmers.se/Cs/Research/Language-technology/GF/next-lib/src/`` ../next-lib/src] - -It is recommended to build new grammars to match the next release. - - - - -==The resource grammar structure== - -The library is divided into a bunch of modules, whose dependencies -are given in the following figure. - -[Syntax.png] - -Modules of different kinds are distinguished as follows: -- solid contours: module seen by end users -- dashed contours: internal module -- ellipse: abstract/concrete pair of modules -- rectangle: resource or instance -- diamond: interface - - -Put in another way: -- solid rectangles and diamonds: user-accessible library API -- solid ellipses: user-accessible top-level grammar for parsing and linearization -- dashed contours: not visible to users - - -The dashed ellipses form the main parts of the implementation, on which the resource -grammar programmer has to work with. She also has to work on the ``Paradigms`` -module. The rest of the modules can be produced mechanically from corresponding -modules for other languages, by just changing the language codes appearing in -their module headers. - -The module structure is rather flat: most modules are direct -parents of ``Grammar``. The idea -is that the implementors can concentrate on one linguistic aspect at a time, or -also distribute the work among several authors. The module ``Cat`` -defines the "glue" that ties the aspects together - a type system -to which all the other modules conform, so that e.g. ``NP`` means -the same thing in those modules that use ``NP``s and those that -constructs them. - - -===Library API modules=== - -For the user of the library, these modules are the most important ones. -In a typical application, it is enough to open ``Paradigms`` and ``Syntax``. -The module ``Try`` combines these two, making it possible to experiment -with combinations of syntactic and lexical constructors by using the -``cc`` command in the GF shell. Here are short explanations of each API module: -- ``Try``: the whole resource library for a language (``Paradigms``, ``Syntax``, - ``Irreg``, and ``Extra``); - produced mechanically as a collection of modules -- ``Syntax``: language-independent categories, syntax functions, and structural words; - produced mechanically as a collection of modules -- ``Constructors``: language-independent syntax functions and structural words; - produced mechanically via functor instantiation -- ``Paradigms``: language-dependent morphological paradigms - - - - - -===Phrase category modules=== - -The immediate parents of ``Grammar`` will be called **phrase category modules**, -since each of them concentrates on a particular phrase category (nouns, verbs, -adjectives, sentences,...). A phrase category module tells -//how to construct phrases in that category//. You will find out that -all functions in any of these modules have the same value type (or maybe -one of a small number of different types). Thus we have -- ``Noun``: construction of nouns and noun phrases -- ``Adjective``: construction of adjectival phrases -- ``Verb``: construction of verb phrases -- ``Adverb``: construction of adverbial phrases -- ``Numeral``: construction of cardinal and ordinal numerals -- ``Sentence``: construction of sentences and imperatives -- ``Question``: construction of questions -- ``Relative``: construction of relative clauses -- ``Conjunction``: coordination of phrases -- ``Phrase``: construction of the major units of text and speech -- ``Text``: construction of texts as sequences of phrases -- ``Idiom``: idiomatic expressions such as existentials - - - - -===Infrastructure modules=== - -Expressions of each phrase category are constructed in the corresponding -phrase category module. But their //use// takes mostly place in other modules. -For instance, noun phrases, which are constructed in ``Noun``, are -used as arguments of functions of almost all other phrase category modules. -How can we build all these modules independently of each other? - -As usual in typeful programming, the //only// thing you need to know -about an object you use is its type. When writing a linearization rule -for a GF abstract syntax function, the only thing you need to know is -the linearization types of its value and argument categories. To achieve -the division of the resource grammar to several parallel phrase category modules, -what we need is an underlying definition of the linearization types. This -definition is given as the implementation of -- ``Cat``: syntactic categories of the resource grammar - - -Any resource grammar implementation has first to agree on how to implement -``Cat``. Luckily enough, even this can be done incrementally: you -can skip the ``lincat`` definition of a category and use the default -``{s : Str}`` until you need to change it to something else. In -English, for instance, many categories do have this linearization type. - - - -===Lexical modules=== - -What is lexical and what is syntactic is not as clearcut in GF as in -some other grammar formalisms. Logically, lexical means atom, i.e. a -``fun`` with no arguments. Linguistically, one may add to this -that the ``lin`` consists of only one token (or of a table whose values -are single tokens). Even in the restricted lexicon included in the resource -API, the latter rule is sometimes violated in some languages. For instance, -``Structural.both7and_DConj`` is an atom, but its linearization is -two words e.g. //both - and//. - -Another characterization of lexical is that lexical units can be added -almost //ad libitum//, and they cannot be defined in terms of already -given rules. The lexical modules of the resource API are thus more like -samples than complete lists. There are two such modules: -- ``Structural``: structural words (determiners, conjunctions,...) -- ``Lexicon``: basic everyday content words (nouns, verbs,...) - - -The module ``Structural`` aims for completeness, and is likely to -be extended in future releases of the resource. The module ``Lexicon`` -gives a "random" list of words, which enables testing the syntax. -It also provides a check list for morphology, since those words are likely to include -most morphological patterns of the language. - -In the case of ``Lexicon`` it may come out clearer than anywhere else -in the API that it is impossible to give exact translation equivalents in -different languages on the level of a resource grammar. This is no problem, -since application grammars can use the resource in different ways for -different languages. - - -==Language-dependent syntax modules== - -In addition to the common API, there is room for language-dependent extensions -of the resource. The top level of each languages looks as follows (with German -as example): -``` - abstract AllGerAbs = Lang, ExtraGerAbs, IrregGerAbs -``` -where ``ExtraGerAbs`` is a collection of syntactic structures specific to German, -and ``IrregGerAbs`` is a dictionary of irregular words of German -(at the moment, just verbs). Each of these language-specific grammars has -the potential to grow into a full-scale grammar of the language. These grammar -can also be used as libraries, but the possibility of using functors is lost. - -To give a better overview of language-specific structures, -modules like ``ExtraGerAbs`` -are built from a language-independent module ``ExtraAbs`` -by restricted inheritance: -``` - abstract ExtraGerAbs = Extra [f,g,...] -``` -Thus any category and function in ``Extra`` may be shared by a subset of all -languages. One can see this set-up as a matrix, which tells -what ``Extra`` structures -are implemented in what languages. For the common API in ``Grammar``, the matrix -is filled with 1's (everything is implemented in every language). - -In a minimal resource grammar implementation, the language-dependent -extensions are just empty modules, but it is good to provide them for -the sake of uniformity. - - - -===The present-tense fragment=== - -Some lines in the resource library are suffixed with the comment -``` - --# notpresent -``` -which is used by a preprocessor to exclude those lines from -a reduced version of the full resource. This present-tense-only -version is useful for applications in most technical text, since -they reduce the grammar size and compilation time. It can also -be useful to exclude those lines in a first version of resource -implementation. To compile a grammar with present-tense-only, use -``` - make Present -``` -with ``resource/Makefile``. - - - -==Phases of the work== - -===Putting up a directory=== - -Unless you are writing an instance of a parametrized implementation -(Romance or Scandinavian), which will be covered later, the -simplest way is to follow roughly the following procedure. Assume you -are building a grammar for the German language. Here are the first steps, -which we actually followed ourselves when building the German implementation -of resource v. 1.0 at Ubuntu linux. We have slightly modified them to -match resource v. 1.5 and GF v. 3.0. - -+ Create a sister directory for ``GF/lib/resource/english``, named - ``german``. -``` - cd GF/lib/resource/ - mkdir german - cd german -``` - -+ Check out the [ISO 639 3-letter language code - http://www.w3.org/WAI/ER/IG/ert/iso639.htm] - for German: both ``Ger`` and ``Deu`` are given, and we pick ``Ger``. - (We use the 3-letter codes rather than the more common 2-letter codes, - since they will suffice for many more languages!) - -+ Copy the ``*Eng.gf`` files from ``english`` ``german``, - and rename them: -``` - cp ../english/*Eng.gf . - rename 's/Eng/Ger/' *Eng.gf -``` - If you don't have the ``rename`` command, you can use a bash script with ``mv``. - - -+ Change the ``Eng`` module references to ``Ger`` references - in all files: -``` - sed -i 's/English/German/g' *Ger.gf - sed -i 's/Eng/Ger/g' *Ger.gf -``` - The first line prevents changing the word ``English``, which appears - here and there in comments, to ``Gerlish``. The ``sed`` command syntax - may vary depending on your operating system. - -+ This may of course change unwanted occurrences of the - string ``Eng`` - verify this by -``` - grep Ger *.gf -``` - But you will have to make lots of manual changes in all files anyway! - -+ Comment out the contents of these files: -``` - sed -i 's/^/--/' *Ger.gf -``` - This will give you a set of templates out of which the grammar - will grow as you uncomment and modify the files rule by rule. - -+ In all ``.gf`` files, uncomment the module headers and brackets, - leaving the module bodies commented. Unfortunately, there is no - simple way to do this automatically (or to avoid commenting these - lines in the previous step) - but uncommenting the first - and the last lines will actually do the job for many of the files. - -+ Uncomment the contents of the main grammar file: -``` - sed -i 's/^--//' LangGer.gf -``` - -+ Now you can open the grammar ``LangGer`` in GF: -``` - gf LangGer.gf -``` - You will get lots of warnings on missing rules, but the grammar will compile. - -+ At all the following steps you will now have a valid, but incomplete - GF grammar. The GF command -``` - pg -missing -``` - tells you what exactly is missing. - - -Here is the module structure of ``LangGer``. It has been simplified by leaving out -the majority of the phrase category modules. Each of them has the same dependencies -as ``VerbGer``, whose complete dependencies are shown as an example. - -[German.png] - - -===Direction of work=== - -The real work starts now. There are many ways to proceed, the most obvious ones being -- Top-down: start from the module ``Phrase`` and go down to ``Sentence``, then - ``Verb``, ``Noun``, and in the end ``Lexicon``. In this way, you are all the time - building complete phrases, and add them with more content as you proceed. - **This approach is not recommended**. It is impossible to test the rules if - you have no words to apply the constructions to. - -- Bottom-up: set as your first goal to implement ``Lexicon``. To this end, you - need to write ``ParadigmsGer``, which in turn needs parts of - ``MorphoGer`` and ``ResGer``. - **This approach is not recommended**. You can get stuck to details of - morphology such as irregular words, and you don't have enough grasp about - the type system to decide what forms to cover in morphology. - - -The practical working direction is thus a saw-like motion between the morphological -and top-level modules. Here is a possible course of the work that gives enough -test data and enough general view at any point: -+ Define ``Cat.N`` and the required parameter types in ``ResGer``. As we define -``` - lincat N = {s : Number => Case => Str ; g : Gender} ; -``` -we need the parameter types ``Number``, ``Case``, and ``Gender``. The definition -of ``Number`` in [``common/ParamX`` ../lib/resource/common/ParamX.gf] -works for German, so we -use it and just define ``Case`` and ``Gender`` in ``ResGer``. - -+ Define some cases of ``mkN`` in ``ParadigmsGer``. In this way you can -already implement a huge amount of nouns correctly in ``LexiconGer``. Actually -just adding the worst-case instance of ``mkN`` (the one taking the most -arguments) should suffice for every noun - but, -since it is tedious to use, you -might proceed to the next step before returning to morphology and defining the -real work horse, ``mkN`` taking two forms and a gender. - -+ While doing this, you may want to test the resource independently. Do this by - starting the GF shell in the ``resource`` directory, by the commands -``` - > i -retain german/ParadigmsGer - > cc -table mkN "Kirche" -``` - -+ Proceed to determiners and pronouns in -``NounGer`` (``DetCN UsePron DetQuant NumSg DefArt IndefArt UseN``) and -``StructuralGer`` (``i_Pron this_Quant``). You also need some categories and -parameter types. At this point, it is maybe not possible to find out the final -linearization types of ``CN``, ``NP``, ``Det``, and ``Quant``, but at least you should -be able to correctly inflect noun phrases such as //every airplane//: -``` - > i german/LangGer.gf - > l -table DetCN every_Det (UseN airplane_N) - - Nom: jeder Flugzeug - Acc: jeden Flugzeug - Dat: jedem Flugzeug - Gen: jedes Flugzeugs -``` - -+ Proceed to verbs: define ``CatGer.V``, ``ResGer.VForm``, and -``ParadigmsGer.mkV``. You may choose to exclude ``notpresent`` -cases at this point. But anyway, you will be able to inflect a good -number of verbs in ``Lexicon``, such as -``live_V`` (``mkV "leben"``). - -+ Now you can soon form your first sentences: define ``VP`` and -``Cl`` in ``CatGer``, ``VerbGer.UseV``, and ``SentenceGer.PredVP``. -Even if you have excluded the tenses, you will be able to produce -``` - > i -preproc=./mkPresent german/LangGer.gf - > l -table PredVP (UsePron i_Pron) (UseV live_V) - - Pres Simul Pos Main: ich lebe - Pres Simul Pos Inv: lebe ich - Pres Simul Pos Sub: ich lebe - Pres Simul Neg Main: ich lebe nicht - Pres Simul Neg Inv: lebe ich nicht - Pres Simul Neg Sub: ich nicht lebe -``` -You should also be able to parse: -``` - > p -cat=Cl "ich lebe" - PredVP (UsePron i_Pron) (UseV live_V) -``` - -+ Transitive verbs -(``CatGer.V2 CatGer.VPSlash ParadigmsGer.mkV2 VerbGer.ComplSlash VerbGer.SlashV2a``) -are a natural next step, so that you can -produce ``ich liebe dich`` ("I love you"). - -+ Adjectives (``CatGer.A ParadigmsGer.mkA NounGer.AdjCN AdjectiveGer.PositA``) -will force you to think about strong and weak declensions, so that you can -correctly inflect //mein neuer Wagen, dieser neue Wagen// -("my new car, this new car"). - -+ Once you have implemented the set -(``Noun.DetCN Noun.AdjCN Verb.UseV Verb.ComplSlash Verb.SlashV2a Sentence.PredVP), -you have overcome most of difficulties. You know roughly what parameters -and dependences there are in your language, and you can now proceed very -much in the order you please. - - - -===The develop-test cycle=== - -The following develop-test cycle will -be applied most of the time, both in the first steps described above -and in later steps where you are more on your own. - -+ Select a phrase category module, e.g. ``NounGer``, and uncomment some - linearization rules (for instance, ``DetCN``, as above). - -+ Write down some German examples of this rule, for instance translations - of "the dog", "the house", "the big house", etc. Write these in all their - different forms (two numbers and four cases). - -+ Think about the categories involved (``CN, NP, N, Det``) and the - variations they have. Encode this in the lincats of ``CatGer``. - You may have to define some new parameter types in ``ResGer``. - -+ To be able to test the construction, - define some words you need to instantiate it - in ``LexiconGer``. You will also need some regular inflection patterns - in``ParadigmsGer``. - -+ Test by parsing, linearization, - and random generation. In particular, linearization to a table should - be used so that you see all forms produced; the ``treebank`` option - preserves the tree -``` - > gr -cat=NP -number=20 | l -table -treebank -``` - -+ Save some tree-linearization pairs for later regression testing. You can save - a gold standard treebank and use the Unix ``diff`` command to compare later - linearizations produced from the same list of trees. If you save the trees - in a file ``trees``, you can do as follows: -``` - > rf -file=trees -tree -lines | l -table -treebank | wf -file=treebank -``` - -+ A file with trees testing all resource functions is included in the resource, - entitled ``resource/exx-resource.gft``. A treebank can be created from this by - the Unix command -``` - % runghc Make.hs test langs=Ger -``` - - - -You are likely to run this cycle a few times for each linearization rule -you implement, and some hundreds of times altogether. There are roughly -70 ``cat``s and -600 ``funs`` in ``Lang`` at the moment; 170 of the ``funs`` are outside the two -lexicon modules). - - -===Auxiliary modules=== - -These auxuliary ``resource`` modules will be written by you. - -- ``ResGer``: parameter types and auxiliary operations -(a resource for the resource grammar!) -- ``ParadigmsGer``: complete inflection engine and most important regular paradigms -- ``MorphoGer``: auxiliaries for ``ParadigmsGer`` and ``StructuralGer``. This need -not be separate from ``ResGer``. - - -These modules are language-independent and provided by the existing resource -package. - -- ``ParamX``: parameter types used in many languages -- ``CommonX``: implementation of language-uniform categories - such as $Text$ and $Phr$, as well as of - the logical tense, anteriority, and polarity parameters -- ``Coordination``: operations to deal with lists and coordination -- ``Prelude``: general-purpose operations on strings, records, - truth values, etc. -- ``Predef``: general-purpose operations with hard-coded definitions - - -An important decision is what rules to implement in terms of operations in -``ResGer``. The **golden rule of functional programming** says: -- //Whenever you find yourself programming by copy and paste, write a function instead!//. - - -This rule suggests that an operation should be created if it is to be -used at least twice. At the same time, a sound principle of **vicinity** says: -- //It should not require too much browsing to understand what a piece of code does.// - - -From these two principles, we have derived the following practice: -- If an operation is needed //in two different modules//, - it should be created in as an ``oper`` in ``ResGer``. An example is ``mkClause``, - used in ``Sentence``, ``Question``, and ``Relative``- -- If an operation is needed //twice in the same module//, but never - outside, it should be created in the same module. Many examples are - found in ``Numerals``. -- If an operation is needed //twice in the same judgement//, but never - outside, it should be created by a ``let`` definition. -- If an operation is only needed once, it should not be created as an ``oper``, - but rather inlined. However, a ``let`` definition may well be in place just - to make the readable. - Most functions in phrase category modules - are implemented in this way. - - -This discipline is very different from the one followed in early -versions of the library (up to 0.9). We then valued the principle of -abstraction more than vicinity, creating layers of abstraction for -almost everything. This led in practice to the duplication of almost -all code on the ``lin`` and ``oper`` levels, and made the code -hard to understand and maintain. - - - -===Morphology and lexicon=== - -The paradigms needed to implement -``LexiconGer`` are defined in -``ParadigmsGer``. -This module provides high-level ways to define the linearization of -lexical items, of categories ``N, A, V`` and their complement-taking -variants. - -For ease of use, the ``Paradigms`` modules follow a certain -naming convention. Thus they for each lexical category, such as ``N``, -the overloaded functions, such as ``mkN``, with the following cases: - -- the worst-case construction of ``N``. Its type signature - has the form -``` - mkN : Str -> ... -> Str -> P -> ... -> Q -> N -``` - with as many string and parameter arguments as can ever be needed to - construct an ``N``. -- the most regular cases, with just one string argument: -``` - mkN : Str -> N -``` -- A language-dependent (small) set of functions to handle mild irregularities - and common exceptions. - - -For the complement-taking variants, such as ``V2``, we provide -- a case that takes a ``V`` and all necessary arguments, such - as case and preposition: -``` - mkV2 : V -> Case -> Str -> V2 ; -``` -- a case that takes a ``Str`` and produces a transitive verb with the direct - object case: -``` - mkV2 : Str -> V2 ; -``` -- A language-dependent (small) set of functions to handle common special cases, - such as transitive verbs that are not regular: -``` - mkV2 : V -> V2 ; -``` - - -The golden rule for the design of paradigms is that -- //The user of the library will only need function applications with constants and strings, never any records or tables.// - - -The discipline of data abstraction moreover requires that the user of the resource -is not given access to parameter constructors, but only to constants that denote -them. This gives the resource grammarian the freedom to change the underlying -data representation if needed. It means that the ``ParadigmsGer`` module has -to define constants for those parameter types and constructors that -the application grammarian may need to use, e.g. -``` - oper - Case : Type ; - nominative, accusative, genitive, dative : Case ; -``` -These constants are defined in terms of parameter types and constructors -in ``ResGer`` and ``MorphoGer``, which modules are not -visible to the application grammarian. - - -===Lock fields=== - -An important difference between ``MorphoGer`` and -``ParadigmsGer`` is that the former uses "raw" record types -for word classes, whereas the latter used category symbols defined in -``CatGer``. When these category symbols are used to denote -record types in a resource modules, such as ``ParadigmsGer``, -a **lock field** is added to the record, so that categories -with the same implementation are not confused with each other. -(This is inspired by the ``newtype`` discipline in Haskell.) -For instance, the lincats of adverbs and conjunctions are the same -in ``CommonX`` (and therefore in ``CatGer``, which inherits it): -``` - lincat Adv = {s : Str} ; - lincat Conj = {s : Str} ; -``` -But when these category symbols are used to denote their linearization -types in resource module, these definitions are translated to -``` - oper Adv : Type = {s : Str ; lock_Adv : {}} ; - oper Conj : Type = {s : Str} ; lock_Conj : {}} ; -``` -In this way, the user of a resource grammar cannot confuse adverbs with -conjunctions. In other words, the lock fields force the type checker -to function as grammaticality checker. - -When the resource grammar is ``open``ed in an application grammar, the -lock fields are never seen (except possibly in type error messages), -and the application grammarian should never write them herself. If she -has to do this, it is a sign that the resource grammar is incomplete, and -the proper way to proceed is to fix the resource grammar. - -The resource grammarian has to provide the dummy lock field values -in her hidden definitions of constants in ``Paradigms``. For instance, -``` - mkAdv : Str -> Adv ; - -- mkAdv s = {s = s ; lock_Adv = <>} ; -``` - - -===Lexicon construction=== - -The lexicon belonging to ``LangGer`` consists of two modules: -- ``StructuralGer``, structural words, built by using both - ``ParadigmsGer`` and ``MorphoGer``. -- ``LexiconGer``, content words, built by using ``ParadigmsGer`` only. - - -The reason why ``MorphoGer`` has to be used in ``StructuralGer`` -is that ``ParadigmsGer`` does not contain constructors for closed -word classes such as pronouns and determiners. The reason why we -recommend ``ParadigmsGer`` for building ``LexiconGer`` is that -the coverage of the paradigms gets thereby tested and that the -use of the paradigms in ``LexiconGer`` gives a good set of examples for -those who want to build new lexica. - - - - - -==Lexicon extension== - -===The irregularity lexicon=== - -It is useful in most languages to provide a separate module of irregular -verbs and other words which are difficult for a lexicographer -to handle. There are usually a limited number of such words - a -few hundred perhaps. Building such a lexicon separately also -makes it less important to cover //everything// by the -worst-case variants of the paradigms ``mkV`` etc. - - - -===Lexicon extraction from a word list=== - -You can often find resources such as lists of -irregular verbs on the internet. For instance, the -Irregular German Verb page -previously found in -``http://www.iee.et.tu-dresden.de/~wernerr/grammar/verben_dt.html`` -page gives a list of verbs in the -traditional tabular format, which begins as follows: -``` - backen (du bäckst, er bäckt) backte [buk] gebacken - befehlen (du befiehlst, er befiehlt; befiehl!) befahl (beföhle; befähle) befohlen - beginnen begann (begönne; begänne) begonnen - beißen biß gebissen -``` -All you have to do is to write a suitable verb paradigm -``` - irregV : (x1,_,_,_,_,x6 : Str) -> V ; -``` -and a Perl or Python or Haskell script that transforms -the table to -``` - backen_V = irregV "backen" "bäckt" "back" "backte" "backte" "gebacken" ; - befehlen_V = irregV "befehlen" "befiehlt" "befiehl" "befahl" "beföhle" "befohlen" ; -``` - -When using ready-made word lists, you should think about -coyright issues. All resource grammar material should -be provided under GNU Lesser General Public License (LGPL). - - - -===Lexicon extraction from raw text data=== - -This is a cheap technique to build a lexicon of thousands -of words, if text data is available in digital format. -See the [Extract Homepage http://www.cs.chalmers.se/~markus/extract/] -homepage for details. - - -===Bootstrapping with smart paradigms=== - -This is another cheap technique, where you need as input a list of words with -part-of-speech marking. You initialize the lexicon by using the one-argument -``mkN`` etc paradigms, and add forms to those words that do not come out right. -This procedure is described in the paper - -A. Ranta. -How predictable is Finnish morphology? An experiment on lexicon construction. -In J. Nivre, M. Dahllöf and B. Megyesi (eds), -//Resourceful Language Technology: Festschrift in Honor of Anna Sågvall Hein//, -University of Uppsala, -2008. -Available from the [series homepage http://publications.uu.se/abstract.xsql?dbid=8933] - - - - -==Extending the resource grammar API== - -Sooner or later it will happen that the resource grammar API -does not suffice for all applications. A common reason is -that it does not include idiomatic expressions in a given language. -The solution then is in the first place to build language-specific -extension modules, like ``ExtraGer``. - -==Using parametrized modules== - -===Writing an instance of parametrized resource grammar implementation=== - -Above we have looked at how a resource implementation is built by -the copy and paste method (from English to German), that is, formally -speaking, from scratch. A more elegant solution available for -families of languages such as Romance and Scandinavian is to -use parametrized modules. The advantages are -- theoretical: linguistic generalizations and insights -- practical: maintainability improves with fewer components - - -Here is a set of -[slides http://www.cs.chalmers.se/~aarne/geocal2006.pdf] -on the topic. - - -===Parametrizing a resource grammar implementation=== - -This is the most demanding form of resource grammar writing. -We do //not// recommend the method of parametrizing from the -beginning: it is easier to have one language first implemented -in the conventional way and then add another language of the -same family by aprametrization. This means that the copy and -paste method is still used, but at this time the differences -are put into an ``interface`` module. - - -==Character encoding and transliterations== - -This section is relevant for languages using a non-ASCII character set. - -==Coding conventions in GF== - -From version 3.0, GF follows a simple encoding convention: -- GF source files may follow any encoding, such as isolatin-1 or UTF-8; - the default is isolatin-1, and UTF8 must be indicated by the judgement -``` - flags coding = utf8 ; -``` - in each source module. -- for internal processing, all characters are converted to 16-bit unicode, - as the first step of grammar compilation guided by the ``coding`` flag -- as the last step of compilation, all characters are converted to UTF-8 -- thus, GF object files (``gfo``) and the Portable Grammar Format (``pgf``) - are in UTF-8 - - -Most current resource grammars use isolatin-1 in the source, but this does -not affect their use in parallel with grammars written in other encodings. -In fact, a grammar can be put up from modules using different codings. - -**Warning**. While string literals may contain any characters, identifiers -must be isolatin-1 letters (or digits, underscores, or dashes). This has to -do with the restrictions of the lexer tool that is used. - - -==Transliterations== - -While UTF-8 is well supported by most web browsers, its use in terminals and -text editors may cause disappointment. Many grammarians therefore prefer to -use ASCII transliterations. GF 3.0beta2 provides the following built-in -transliterations: -- Arabic -- Devanagari (Hindi) -- Thai - - -New transliterations can be defined in the GF source file -[``GF/Text/Transliterations.hs`` ../src/GF/Text/Transliterations.hs]. -This file also gives instructions on how new ones are added. - - - - - |
