gf-core.git/src/compiler/GF/Grammar/lexer, branch masterUnnamed repository; edit this file 'description' to name the repository.
https://git.adelon.net/gf-core.git/atom?h=master2013-11-25T21:12:11+00:00Change how GF deals with character encodings in grammar files2013-11-25T21:12:11+00:00hallgrenhallgren@chalmers.se2013-11-25T21:12:11+00:00urn:sha1:9d7fdf7c9a525a3b5659a566f76d26d151dcd664
1. The default encoding is changed from Latin-1 to UTF-8.
2. Alternate encodings should be specified as "--# -coding=enc", the old
"flags coding=enc" declarations have no effect but are still checked for
consistency.
3. A transitional warning is generated for files that contain non-ASCII
characters without specifying a character encoding:
"Warning: default encoding has changed from Latin-1 to UTF-8"
4. Conversion to Unicode is now done *before* lexing. This makes it possible
to allow arbitrary Unicode characters in identifiers. But identifiers are
still stored as ByteStrings, so they are limited to Latin-1 characters
for now.
5. Lexer.hs is no longer part of the repository. We now generate the lexer
from Lexer.x with alex>=3. Some workarounds for bugs in alex-3.0 were
needed. These bugs might already be fixed in newer versions of alex, but
we should be compatible with what is shipped in the Haskell Platform.
the GF syntax for identifiers is exteded with quoted forms, i.e. you could write for instance 'ab.c' and then everything between the quites is identifier. This includes Unicode characters and non-ASCII symbols. This is useful for automatically generated GF grammars.2013-11-22T13:30:18+00:00kr.angelovkr.angelov@gmail.com2013-11-22T13:30:18+00:00urn:sha1:8bcc70eac8af379ed3481039eb1bd5feea3cf195added the linref construction in GF. The PGF version number is now bumped2013-10-30T12:53:36+00:00kr.angelovkr.angelov@gmail.com2013-10-30T12:53:36+00:00urn:sha1:042243f08a321cd8ed5918ba94e83f22a8552adbIntroduce type RawIdent; only 9 imports of Data.ByteString.Char8 remain2013-09-19T20:48:10+00:00hallgrenhallgren@chalmers.se2013-09-19T20:48:10+00:00urn:sha1:021b5f06d3900fe2b10d5d3ccf6ac286a779ef16
The fact that identifiers are represented as ByteStrings is now an internal
implentation detail in module GF.Infra.Ident. Conversion between ByteString
and identifiers is only needed in the lexer and the Binary instances.
alex 3 incompatibility workaround2012-05-04T12:39:07+00:00hallgrenhallgren@chalmers.se2012-05-04T12:39:07+00:00urn:sha1:841e54e3dcf1373e9f462c55c0be93dda642321c
As a temporary workaround, alex is no longer invoked automatically when
building with cabal. Developers who want to modify the lexer need to run
alex on Lexer.x manually and record the modified Lexer.hs.
src/compiler/GF/Grammar/lexer/Lexer.x -- hidden from cabal
src/compiler/GF/Grammar/Lexer.hs -- update it manually