Improve function decorator parsing

REFERENCE.md

@@ -1,69 +1,5 @@
-Anselme
-=======
-
-The overengineered dialog scripting system in pure Lua.
-
-**Documentation and language are still WIP and will change.**
-
-Purpose
--------
-
-Once upon a time, I wanted to do a game with a branching story. I could store the current state in a bunch of variables and just write everything like the rest of my game's code. But no, that would be *too simple*. I briefly looked at [ink](https://github.com/inkle/ink), which looked nice but lacked some features I felt like I needed. Mainly, I wanted something more language independant and less linear. Also, I wasn't a fan of the syntax. And I'm a weirdo who make their game in Lua.
-
-So, here we go. Let's make a new scripting language.
-
-Anselme ended up with some features that are actually quite useful compared to the alternatives:
-
-* allows for concurently running scripts (a conversation bores you? why not start another at the same time!)
-* allows for script interuption with gracious fallback (so you can *finally* make that NPC shut up mid-sentence)
-* a mostly consistent and easy to read syntax based around lines and whitespace
-* easily extensible (at least from Lua ;))
-
-And most stuff you'd expect from such a language:
-
-* easy text writing, can integrate expressions into text, can assign tags to (part of) lines
-* choices that lead to differents paths
-* variables, functions, arbitrary expressions (not Lua, it's its own thing)
-* can pause the interpreter when needed
-* can save and restore state
-
-And things that are halfway there but *should* be there eventually (i.e., TODO):
-* language independant; scripts should (hopefully) be easily localizable into any language (it's possible, but doesn't provide any batteries for this right now)
-    Defaults variables use emoji and then it's expected to alias them; works but not the most satisfying solution.
-* a good documentation
-    Need to work on consistent naming of Anselme concepts
-    A step by step tutorial
-
-Things that Anselme is not:
-* a game engine. It's very specific to dialogs and text, so unless you make a text game you will need to do a lot of other stuff.
-* a language based on Lua. It's imperative and arrays start at 1 but there's not much else in common.
-
-Example
--------
-
-Sometimes we need some simplicity:
-
-```
-HELLO SIR, HOW ARE YOU TODAY
-> why are you yelling
-    I LIKE TO
-    > Well that's stupid.
-        I DO NOT LIKE YOU SIR.
-> I AM FINE AND YOU
-    I AM FINE THANK YOU
-
-    LOVELY WEATHER WE'RE HAVING, AREN'T WE?
-    > Sure is!
-        YEAH. YEAH.
-    > I've seen better.
-        NOT NICE.
-
-WELL, GOOD BYE.
-```
-
-Othertimes we don't:
-
-TODO: stupidly complex script
+Anselme reference
+=================
 
 Language reference
 ------------------
@@ -359,7 +295,7 @@ tagged # 42
 
 Every line can also be followed with decorators, which are appended at the end of the line and affect its behaviour. Decorators are just syntaxic sugar to make some common operations simpler to write.
 
-* `$`: function decorator. Same as a function line, behaving as if this line was it sole child, but also run the function.
+* `$`: function decorator. Same as a function line, behaving as if this line was it sole child, but also run the function. Function can not take arguments.
 
 ```
 text $ f
@@ -382,7 +318,7 @@ This is typically used for immediatletly running functions when defining them, f
     > Exit
 ```
 
-is equivalent to (since empty condition is assumed true):
+is equivalent to:
 
 ```
 $ loop
@@ -615,18 +551,19 @@ How conservions are handled from Lua to Anselme:
 
 * `boolean` -> `number`, 0 for false, 1 for true.
 
-#### Escapes codes
+#### Escape codes
 
 These can be used to represent some caracters in string and other text elements that would otherwise be difficult to express due to conflicts with Anselme syntax.
 
-* `\{` for `{`
-* `\~` for `~`
-* `\#` for `#`
-* `\$` for `$`
 * `\\` for `\`
 * `\"` for `"`
 * `\n` for a newline
 * `\t` for a tabulation
+* `\{` for `{`
+* `\[` for `[`
+* `\~` for `~`
+* `\#` for `#`
+* `\$` for `$`
 
 #### Truethness
 
@@ -889,13 +826,13 @@ This only works on strings:
 
 `a :: b`: evaluate a and b, returns a new typed value with a as value and b as type.
 
-`a ~ b`: evaluates b, if true evaluates and returns a, otherwise returns nil (lazy).
+`a ~ b`: evaluates b, if true evaluates a and returns it, otherwise returns nil (lazy).
 
-`a # b`: evaluates b, then evaluates a whith b added to the active tags.
+`a # b`: evaluates b, then evaluates a whith b added to the active tags. Returns a.
 
 `a(b)`: evaluate b (number), returns the value with this index in a (list). Use 1-based indexing. If b is a string, will search the first pair in the list with this string as its name. Operator is named `()`.
 
-`{}(v)`: function called when formatting a value in a text interpolation for printing
+`{}(v)`: function called when formatting a value in a text interpolation for printing.
 
 #### Built-in functions
 

notes.txt

@@ -21,7 +21,7 @@ Using other unicode symbols may be also alright, but there also should be a way
 
 TODO: add alias to §
 
-Reserved symbols that are still not used in expressions: ~`\_?@$#
+Reserved symbols that are still not used in expressions: `\_?@$
 
 Reserved symbols that are still not used as a line type: `^+-=</[]*{}|\_!?.,;)"&%
 

parser/common.lua

@@ -23,7 +23,7 @@ local disallowed_set = ("~`^+-=<>/[]*{}|\\_!?,;:()\"@&$#%"):gsub("[^%w]", "%%%1"
 common = {
 	--- valid identifier pattern
 	identifier_pattern = "%s*[^0-9%s"..disallowed_set.."][^"..disallowed_set.."]*",
-	-- names allowed for a function that aren't valide identifiers, mainly for overloading operators
+	-- names allowed for a function that aren't valid identifiers, mainly for overloading operators
 	special_functions_names = {
 		-- operators not included here:
 		-- * assignment operators (:=, +=, -=, //=, /=, *=, %=, ^=): handled with its own syntax (function assignment)
@@ -43,14 +43,20 @@ common = {
 	-- escapement code and their value in strings
 	-- I don't think there's a point in supporting form feed, carriage return, and other printer and terminal related codes
 	string_escapes = {
+		-- usual escape codes
 		["\\\\"] = "\\",
 		["\\\""] = "\"",
 		["\\n"] = "\n",
 		["\\t"] = "\t",
+		-- string interpolation
+		["\\{"] = "{",
+		-- subtext
+		["\\["] = "[",
+		-- end of text line expressions
 		["\\~"] = "~",
 		["\\#"] = "#",
-		["\\$"] = "$", -- FIXME
-		["\\{"] = "{"
+		-- decorators
+		["\\$"] = "$"
 	},
 	--- escape a string to be used as an exact match pattern
 	escape = function(str)
@@ -59,7 +65,7 @@ common = {
 		end
 		return escapeCache[str]
 	end,
-	--- trim a string
+	--- trim a string by removing whitespace at the start and end
 	trim = function(str)
 		return str:match("^%s*(.-)%s*$")
 	end,
@@ -153,13 +159,13 @@ common = {
 			local t, r = text:match(("^([^{%s]*)(.-)$"):format(delimiters))
 			-- text
 			if t ~= "" then
-				-- handle \{ escape: skip to next { until it's not escaped
+				-- handle \{ and binop escape: skip to next { until it's not escaped
 				while t:match("\\$") and r:match(("^[{%s]"):format(delimiters)) do
 					local t2, r2 = r:match(("^([{%s][^{%s]*)(.-)$"):format(delimiters, delimiters))
-					t = t:match("^(.-)\\$") .. t2
+					t = t .. t2 -- don't need to remove \ as it will be stripped with other escapes codes 3 lines later
 					r = r2
 				end
-				-- replace other escape codes
+				-- replace escape codes
 				local escaped = t:gsub("\\.", common.string_escapes)
 				table.insert(l, escaped)
 			end

parser/preparser.lua

@@ -326,9 +326,9 @@ local function transform_indented(indented)
 			table.remove(indented, i)
 		else
 			-- function decorator
-			if l.content:match("^.-[^\\]%$[^#~$]-$") then -- FIXME
+			if l.content:match("^.-[^\\]%$"..identifier_pattern.."$") then
 				local name
-				l.content, name = l.content:match("^(..-)%$([^#~$]-)$")
+				l.content, name = l.content:match("^(.-[^\\])%$("..identifier_pattern..")$")
 				indented[i] = { content = "~"..name, source = l.source }
 				table.insert(indented, i+1, { content = "$"..name, source = l.source, children = { l } })
 				i = i + 1 -- $ line should not contain any decorator anymore

stdlib/functions.lua

@@ -125,7 +125,7 @@ functions = {
 		end
 	},
 	-- alias
-	-- TODO: directly change global state, should new aliases be kept in case of interpreter error before a merge?
+	-- TODO: currently directly change global state, should new aliases be kept in case of interpreter error before a merge?
 	["alias(identifier::string, alias::string)"] = {
 		value = function(identifier, alias)
 			-- check identifiers

test/tests/text escaping.ans

@@ -5,3 +5,5 @@ quote \"
 other codes \n \\ \t
 
 decorators \# tag \~ condition \$ fn
+
+sub \[text]

test/tests/text escaping.lua

@@ -1,22 +1,26 @@
 local _={}
+_[21]={}
+_[20]={}
+_[19]={}
+_[18]={}
 _[17]={}
-_[16]={}
-_[15]={}
-_[14]={}
-_[13]={tags=_[17],text="decorators # tag ~ condition $ fn"}
-_[12]={tags=_[16],text="other codes \n \\ \9"}
-_[11]={tags=_[15],text="quote \""}
-_[10]={tags=_[14],text="expression {a}"}
-_[9]={_[13]}
-_[8]={_[12]}
-_[7]={_[11]}
-_[6]={_[10]}
-_[5]={"return"}
-_[4]={"text",_[9]}
-_[3]={"text",_[8]}
-_[2]={"text",_[7]}
-_[1]={"text",_[6]}
-return {_[1],_[2],_[3],_[4],_[5]}
+_[16]={text="sub [text]",tags=_[21]}
+_[15]={text="decorators # tag ~ condition $ fn",tags=_[20]}
+_[14]={text="other codes \n \\ \9",tags=_[19]}
+_[13]={text="quote \"",tags=_[18]}
+_[12]={text="expression {a}",tags=_[17]}
+_[11]={_[16]}
+_[10]={_[15]}
+_[9]={_[14]}
+_[8]={_[13]}
+_[7]={_[12]}
+_[6]={"return"}
+_[5]={"text",_[11]}
+_[4]={"text",_[10]}
+_[3]={"text",_[9]}
+_[2]={"text",_[8]}
+_[1]={"text",_[7]}
+return {_[1],_[2],_[3],_[4],_[5],_[6]}
 --[[
 { "text", { {
       tags = {},
@@ -34,5 +38,9 @@ return {_[1],_[2],_[3],_[4],_[5]}
       tags = {},
       text = "decorators # tag ~ condition $ fn"
     } } }
+{ "text", { {
+      tags = {},
+      text = "sub [text]"
+    } } }
 { "return" }
 ]]--