| Age | Commit message (Collapse) | Author |
|---|
|
Someone who is familiar with the Python translation should check this.
|
|
|
|
This is to avoid one trivial reason for failures in the test suite.
|
|
definitions
|
|
This means that the -old-comp and -new-comp flags are not recognized anymore.
The only functional difference is that printnames were still normalized with
the old partial evaluator. Now that is done with the new partial evaluator.
|
|
GF.Compile.TypeCheck.Primitives
Also move the list of primitives
|
|
Also simplified its type.
|
|
GF.Compile.Coding is not used any more.
|
|
This was a fairly simple change thanks to previous work on making the Ident
type abstract and the fact that PGF.CId already uses UTF-8-encoded
ByteStrings.
One potential pitfall is that Data.ByteString.UTF8 uses the same type for
ByteStrings as Data.ByteString. I renamed ident2bs to ident2utf8 and
bsCId to utf8CId, to make it clearer that they work with UTF-8-encoded
ByteStrings.
Since both the compiler input and identifiers are now UTF-8-encoded
ByteStrings, the lexer now creates identifiers without copying any characters.
**END OF DESCRIPTION***
Place the long patch description above the ***END OF DESCRIPTION*** marker.
The first line of this file will be the patch name.
This patch contains the following changes:
M ./src/compiler/GF/Compile/CheckGrammar.hs -3 +3
M ./src/compiler/GF/Compile/GrammarToPGF.hs -2 +2
M ./src/compiler/GF/Grammar/Binary.hs -5 +1
M ./src/compiler/GF/Grammar/Lexer.x -11 +13
M ./src/compiler/GF/Infra/Ident.hs -19 +36
M ./src/runtime/haskell/PGF.hs -1 +1
M ./src/runtime/haskell/PGF/CId.hs -2 +3
|
|
1. The default encoding is changed from Latin-1 to UTF-8.
2. Alternate encodings should be specified as "--# -coding=enc", the old
"flags coding=enc" declarations have no effect but are still checked for
consistency.
3. A transitional warning is generated for files that contain non-ASCII
characters without specifying a character encoding:
"Warning: default encoding has changed from Latin-1 to UTF-8"
4. Conversion to Unicode is now done *before* lexing. This makes it possible
to allow arbitrary Unicode characters in identifiers. But identifiers are
still stored as ByteStrings, so they are limited to Latin-1 characters
for now.
5. Lexer.hs is no longer part of the repository. We now generate the lexer
from Lexer.x with alex>=3. Some workarounds for bugs in alex-3.0 were
needed. These bugs might already be fixed in newer versions of alex, but
we should be compatible with what is shipped in the Haskell Platform.
|
|
Move source transcoding function GF.Compile to GF.Compile.GetGrammar, in
preparation for doing transcoding before lexing.
|
|
|
|
|
|
+ Eliminated vairous ad-hoc coersion functions between specific monads
(IO, Err, IOE, Check) in favor of more general lifting functions
(liftIO, liftErr).
+ Generalized many basic monadic operations from specific monads to
arbitrary monads in the appropriate class (MonadIO and/or ErrorMonad),
thereby completely eliminating the need for lifting functions in lots
of places.
This can be considered a small step forward towards a cleaner
compiler API and more malleable compiler code in general.
|
|
square brackets
Add proper type checking of course-of-values tables:
+ Make sure that all subterms have the same type.
+ Resolve overloaded operators.
Note though that the GF book states in C.4.12 that the "course-of-values
table [...] format is not recommended for GF source code, since the
ordering of parameter values is not specified and therefore a
compiler-internal decision."
|
|
between ordinary tokens. It is also used in the English RGL to attach the commas to the previous word.
|
|
This module should not be part of the public PGF library API, and it was only
used in GF.CompileToAPI, so the code was moved there. The module defined
constFuncs and syntaxFuncs, but only syntaxFuncs was used.
|
|
later is now retrained. Once the grammar is compiled with the .probs file now it doesn't need anything more to do robust parsing. The robustness itself is controlled by the flags 'heuristic_search_factor', 'meta_prob' and 'meta_token_prob' in ParseEngAbs.gf
|
|
+ References to modules under src/compiler have been eliminated from the PGF
library (under src/runtime/haskell). Only two functions had to be moved (from
GF.Data.Utilities to PGF.Utilities) to make this possible, other apparent
dependencies turned out to be vacuous.
+ In gf.cabal, the GF executable no longer directly depends on the PGF library
source directory, but only on the exposed library modules. This means that
there is less duplication in gf.cabal and that the 30 modules in the
PGF library will no longer be compiled twice while building GF.
To make this possible, additional PGF library modules have been exposed, even
though they should probably be considered for internal use only. They could
be collected in a PGF.Internal module, or marked as "unstable", to make
this explicit.
+ Also, by using the -fwarn-unused-imports flag, ~220 redundant imports were
found and removed, reducing the total number of imports by ~15%.
|
|
|
|
a type signature for GF.Compile.GrammarToPGF.genCncFuns in order to compile.
|
|
|
|
change in the PGF format!!!
The following are the outcomes:
- Predef.nonExist is fully supported by both the Haskell and the C runtimes
- Predef.BIND is now an internal compiler defined token. For now
it behaves just as usual for the Haskell runtime, i.e. it generates &+.
However, the special treatment will let us to handle it properly in
the C runtime.
- This required a major change in the PGF format since both
nonExist and BIND may appear inside 'pre' and this was not supported
before.
|
|
|
|
|
|
of just the module names. This saves extra lookups later
|
|
The fact that identifiers are represented as ByteStrings is now an internal
implentation detail in module GF.Infra.Ident. Conversion between ByteString
and identifiers is only needed in the lexer and the Binary instances.
|
|
Most of the explicit uses of ByteStrings were eliminated by using identS,
identS = identC . BS.pack
which was found in GF.Grammar.CF and moved to GF.Infra.Ident. The function
prefixIdent :: String -> Ident -> Ident
allowed one additional import of ByteString to be eliminated. The functions
isArgIdent :: Ident -> Bool
getArgIndex :: Ident -> Maybe Int
were needed to eliminate explicit pattern matching on Ident from two modules.
|
|
|
|
This problem showed up after delaing eta expansion until just before partial
evaluation
|
|
This seems to work for the most part, but a problem showed up in WordsCat.gf in
the phrasebook.
|
|
Before, they were silently converted to linear patterns.
Nonlinear patterns in MorphoCat.gf, ParadigmsGre.gf and ParadigmsFin.gf have
been make linear by renaming pattern variables.
|
|
|
|
|
|
The refresh pass does not correctly keep track of the scope of local variables
and can convert things like \x->(\x->x) x into \x1->(\x2->x2) x2. Fortunately,
it appears that the refresh pass is not needed anymore, so it has been removed.
|
|
detecting whether this is the last token.
|
|
|
|
|
|
|
|
|
|
The old partial evaluator has special rules to convert pattern macros in
pre { } expressions. These rules were missing in the new partial evaluator.
|
|
This should prevent errors like
Internal error in Compute.ConcreteNew:
Applying Predef.drop: Expected a value of type String, got VFV [VString "gewandt",VString "gewendet"]
|
|
Get rid of old-time depend (and ClockTime in favour of UTCTime).
time-compat helps to retain backward compatibility with directory-1.1
and lower.
|
|
The sequence operator (x+y) was implemented by splitting the string to be
matched at all positions and trying to match the parts against the two
subpatterns. To reduce the number of splits, we now estimate the minimum and
maximum length of the string that the subpatterns could match. For common
cases, where one of the subpatterns is a string of known length, like
in (x+"y") or (x + ("a"|"o"|"u"|"e")+"y"), only one split will be tried.
|
|
flag beam_size in the top-level concrete module
|
|
Instead of "Internal error in ...", you now get a proper error message with
a source location and a function name.
|
|
Add a conversion rule for ({ l1 = e } ** x).l2 in PMCFG generation. (A rule
for the symmetric case (x ** { l1 = e }).l2 was added some time ago.)
|
|
+ Instead of "Internal error in ...", you now get a proper error message with
a source location and a function name.
+ Also added some missing error value propagation in the partial evaluator.
+ Also some other minor cleanup and error handling fixes.
|
|
same as one of the expected ones: it shows full records rather than just lock fields.
|
|
"a"+("b"++"c") was simplified to "bb"++"c" instead of "ab"++c.
|
