More configurable handling of whitespace before decorators

README.md

@@ -156,7 +156,7 @@ There's different types of lines, depending on their first character(s) (after i
     This is.
 ```
 
-* `>`: write a choice into the event buffer. Followed by arbitrary text. Support [text interpolation](#text-interpolation); if a text event is created during the text interpolation, it is added to the choice text content instead of the global event buffer.
+* `>`: write a choice into the [event buffer](#event-buffer). Followed by arbitrary text. Support [text interpolation](#text-interpolation); if a text event is created during the text interpolation, it is added to the choice text content instead of the global event buffer.
 
 ```
 $ f
@@ -323,7 +323,7 @@ $ f
 
 * empty line: flush the event buffer, i.e., if there are any pending lines of text or choices, send them to your game. See [Event buffer](#event-buffer). This line always keep the same identation as the last non-empty line, so you don't need to put invisible whitespace on an empty-looking line. Is also automatically added at the end of a file.
 
-* regular text: write some text into the event buffer. Support [text interpolation](#text-interpolation).
+* regular text: write some text into the [event buffer](#event-buffer). Support [text interpolation](#text-interpolation).
 
 ```
 Hello,
@@ -348,7 +348,7 @@ is equivalent to:
 ```
 $ fn
     ~ 👁️
-        run this line only once
+        run this line only once 
 ```
 
 * `#`: tag decorator. Same as a tag line, behaving as if this line was it sole child.
@@ -361,7 +361,7 @@ is equivalent to:
 
 ```
 # 42
-    tagged
+    tagged 
 ```
 
 * `$`: function decorator. Same as a function line, behaving as if this line was it sole child, but also run the function.
@@ -375,7 +375,7 @@ is equivalent to:
 ```
 ~ f
 $ f
-    text
+    text 
 ```
 
 This is typically used for immediatletly running functions when defining them, for example for a looping choice :
@@ -428,18 +428,20 @@ Hello {f}
 :b = "{f}"
 ```
 
-### Event buffer
+### Events
 
-Anselme need to give back control to the game at some point. This is done through events: the interpreter regularly yield its coroutine and returns a buch of data to your game (called the event buffer or just "event"). It's called an event buffer because, well, it's a buffer, and events are what we call whatever Anselme sends back to your game.
+Anselme need to give back control to the game at some point. This is done through events: the interpreter regularly yield its coroutine and returns a bunch of data to your game. This is the "event", it is what we call whatever Anselme sends back to your game.
 
-Each event have a type (`text`, `choice`, `return` or `error` by default, custom types can also be defined) and associated data; the data associated with each event depends on its type. For the default events this data is:
+Each event is composed of two elements: a type (string; `text`, `choice`, `return` or `error` by default, custom types can also be defined) and associated data; the data associated with each event depends on its type. For the default events this data is:
 
 * `text` (text to display) is a list of text elements, each with a `text` field, containing the text contents, and a `tags` field, containing the tags associated with this text.
 * `choice` (choices to choose from) is a list of tableas, each associated to a choice. Each of these choice is a list of text elements like for the `text` event.
 * `return` (when the script ends) is the returned value.
 * `error` (when the script error) is the error message.
 
-For some event types (`text` and `choice`), Anselme does not immediately sends the event as soon as they are available but appends them to a buffer of events that will be sent to your game on the next event flush line (empty lines).
+#### Event buffer
+
+For some event types (`text` and `choice`), Anselme does not immediately sends the event as soon as they are available but appends them to a buffer of events that will be sent to your game on the next event flush line (empty line): this is the "event buffer".
 
 ```
 Some text.
@@ -449,7 +451,7 @@ Another text.
 Text in another event.
 ```
 
-Beyond technical reasons, the event buffering also serves as a way to group together several lines. For example, choice A and B will be sent to the game at the same time and can therefore be assumed to be part of the same "choice block", as opposed to choice C wich will be sent alone:
+Beyond technical reasons, the event buffer serves as a way to group together several lines. For example, choice A and B will be sent to the game at the same time and can therefore be assumed to be part of the same "choice block", as opposed to choice C wich will be sent alone:
 
 ```
 > Choice A
@@ -476,6 +478,27 @@ Text
 > Choice
 ```
 
+By default, some processing is done on the event buffer before sending it to your game. You can disable these by disabling the associated features flages using `vm:disable` (see #api-reference).
+
+* strip trailing spaces: will remove any space caracters at the end of the text (for text event), or at the end of each choice (for choice event).
+
+```
+(There is a space between the text and the tag decorator that would be included in the text event otherwise.)
+Some text # tag
+```
+
+* strip duplicate spaces: will remove any duplicated space caracters between each element that constitute the text (for text event), or for each choice (for choice event).
+
+```
+(There is a space between the text and the tag decorator; but there is a space as well after the text interpolation in the last line. The two spaces are converted into a single space (the space will belong to the first text element, i.e. the "text " element).)
+$ f
+    text # tag
+
+Some {text} here.
+```
+
+TODO: check if spacing rules are language-specific and move this to language files if appropriate
+
 ### Identifiers
 
 Valid identifiers must be at least 1 caracters long and can contain anything except the caracters ``~`^+-=<>/[]*{}|\_!?,;:()"@&$#%`` (that is, every special caracter on a US keyboard except '). They can contain spaces. They can not start with a number.

anselme.lua

@@ -6,11 +6,11 @@ local anselme = {
 	-- api is incremented a each update which may break Lua API compatibility
 	versions = {
 		save = 1,
-		language = 16,
+		language = 17,
 		api = 2
 	},
 	-- version is incremented at each update
-	version = 17,
+	version = 18,
 	--- currently running interpreter
 	running = nil
 }
@@ -429,6 +429,26 @@ local vm_mt = {
 		return self
 	end,
 
+	--- enable feature flags
+	-- available flags:
+	-- * "strip trailing spaces": remove trailing spaces from choice and text events (enabled by default)
+	-- * "strip duplicate spaces": remove duplicated spaces between text elements from choice and text events (enabled by default)
+	-- returns self
+	enable = function(self, ...)
+		for _, flag in ipairs{...} do
+			self.state.feature_flags[flag] = true
+		end
+		return self
+	end,
+	--- disable features flags
+	-- returns self
+	disable = function(self, ...)
+		for _, flag in ipairs{...} do
+			self.state.feature_flags[flag] = nil
+		end
+		return self
+	end,
+
 	--- run code
 	-- expr: expression to evaluate (string or parsed expression), or a block to run
 	-- will merge state after successful execution
@@ -447,6 +467,7 @@ local vm_mt = {
 		local interpreter
 		interpreter = {
 			state = {
+				feature_flags = self.state.feature_flags,
 				builtin_aliases = self.state.builtin_aliases,
 				aliases = self.state.aliases,
 				functions = self.state.functions,
@@ -497,6 +518,10 @@ return setmetatable(anselme, {
 	__call = function()
 		-- global state
 		local state = {
+			feature_flags = {
+				["strip trailing spaces"] = true,
+				["strip duplicate spaces"] = true
+			},
 			builtin_aliases = {
 				-- ["👁️"] = "seen",
 				-- ["🔖"] = "checkpoint",

interpreter/interpreter.lua

@@ -2,6 +2,33 @@ local eval
 local truthy, merge_state, to_lua, escape, get_variable, eval_text_callback
 local run_line, run_block
 
+local function post_process_text(state, text)
+	-- remove trailing spaces
+	if state.feature_flags["strip trailing spaces"] then
+		local final = text[#text]
+		if final then
+			final.text = final.text:match("^(.-) *$")
+			if final.text == "" then
+				table.remove(text)
+			end
+		end
+	end
+	-- remove duplicate spaces
+	if state.feature_flags["strip duplicate spaces"] then
+		for i=1, #text-1 do
+			local a, b = text[i], text[i+1]
+			local na = #a.text:match(" *$")
+			local nb = #b.text:match(" *$")
+			if na > 0 and nb > 0 then -- remove duplicated spaces from second element first
+				b.text = b.text:match("^(.-) *$")
+			end
+			if na > 1 then
+				a.text = a.text:match("^(.- ) *$")
+			end
+		end
+	end
+end
+
 --- tag management
 local tags = {
 	--- push new tags on top of the stack, from Anselme values
@@ -134,6 +161,15 @@ local events = {
 					c._state = nil
 				end
 			end
+			-- text post processing
+			if type == "text" then
+				post_process_text(state, buffer)
+			end
+			if type == "choice" then
+				for _, c in ipairs(buffer) do
+					post_process_text(state, c)
+				end
+			end
 			-- yield event
 			coroutine.yield(type, buffer)
 			-- run choice

parser/preparser.lua

@@ -326,19 +326,19 @@ local function transform_indented(indented)
 			table.remove(indented, i)
 		else
 			-- condition decorator
-			if l.content:match("^.-%s*[^~]%~[^#~$]-$") then
+			if l.content:match("^.-[^~]%~[^#~$]-$") then
 				local decorator
-				l.content, decorator = l.content:match("^(..-)%s*(%~[^#~$]-)$")
+				l.content, decorator = l.content:match("^(..-)(%~[^#~$]-)$")
 				indented[i] = { content = decorator, source = l.source, children = { l } }
 			-- tag decorator
-			elseif l.content:match("^..-%s*%#[^#~$]-$") then
+			elseif l.content:match("^..-%#[^#~$]-$") then
 				local decorator
-				l.content, decorator = l.content:match("^(..-)%s*(%#[^#~$]-)$")
+				l.content, decorator = l.content:match("^(..-)(%#[^#~$]-)$")
 				indented[i] = { content = decorator, source = l.source, children = { l } }
 			-- function decorator
-			elseif l.content:match("^..-%s*%$[^#~$]-$") then
+			elseif l.content:match("^..-%$[^#~$]-$") then
 				local name
-				l.content, name = l.content:match("^(..-)%s*%$([^#~$]-)$")
+				l.content, name = l.content:match("^(..-)%$([^#~$]-)$")
 				indented[i] = { content = "~"..name, source = l.source }
 				table.insert(indented, i+1, { content = "$"..name, source = l.source, children = { l } })
 				i = i + 1 -- $ line should not contain any decorator anymore

test/tests/condition decorator.lua

@@ -1,8 +1,8 @@
 local _={}
 _[7]={}
 _[6]={}
-_[5]={tags=_[7],text="ok bis"}
-_[4]={tags=_[6],text="ok"}
+_[5]={text="ok bis",tags=_[7]}
+_[4]={text="ok ",tags=_[6]}
 _[3]={_[4],_[5]}
 _[2]={"return"}
 _[1]={"text",_[3]}
@@ -10,7 +10,7 @@ return {_[1],_[2]}
 --[[
 { "text", { {
       tags = {},
-      text = "ok"
+      text = "ok "
     }, {
       tags = {},
       text = "ok bis"

test/tests/function decorator scope implicit call.lua

@@ -1,17 +1,18 @@
 local _={}
+_[18]={}
 _[17]={}
 _[16]={}
 _[15]={}
 _[14]={}
-_[13]={}
-_[12]={tags=_[17],text="ok"}
-_[11]={tags=_[16],text="1"}
-_[10]={tags=_[16],text="a.\240\159\145\129\239\184\143: "}
-_[9]={tags=_[15],text="ko"}
-_[8]={tags=_[14],text="In function:"}
-_[7]={tags=_[13],text="0"}
-_[6]={tags=_[13],text="a.\240\159\145\129\239\184\143: "}
-_[5]={_[8],_[9],_[10],_[11],_[12]}
+_[13]={text="ok",tags=_[18]}
+_[12]={text=" ",tags=_[17]}
+_[11]={text="1",tags=_[17]}
+_[10]={text="a.\240\159\145\129\239\184\143: ",tags=_[17]}
+_[9]={text="ko",tags=_[16]}
+_[8]={text="In function:",tags=_[15]}
+_[7]={text="0",tags=_[14]}
+_[6]={text="a.\240\159\145\129\239\184\143: ",tags=_[14]}
+_[5]={_[8],_[9],_[10],_[11],_[12],_[13]}
 _[4]={_[6],_[7]}
 _[3]={"return"}
 _[2]={"text",_[5]}
@@ -37,6 +38,9 @@ return {_[1],_[2],_[3]}
     }, {
       tags = <table 1>,
       text = "1"
+    }, {
+      tags = <table 1>,
+      text = " "
     }, {
       tags = {},
       text = "ok"

test/tests/unseen line.lua

@@ -4,11 +4,11 @@ _[12]={}
 _[11]={}
 _[10]={}
 _[9]={}
-_[8]={tags=_[13],text="b"}
-_[7]={tags=_[12],text="a"}
-_[6]={tags=_[11],text="b"}
-_[5]={tags=_[10],text="seen only once"}
-_[4]={tags=_[9],text="a"}
+_[8]={text="b",tags=_[13]}
+_[7]={text="a",tags=_[12]}
+_[6]={text="b",tags=_[11]}
+_[5]={text="seen only once ",tags=_[10]}
+_[4]={text="a",tags=_[9]}
 _[3]={_[4],_[5],_[6],_[7],_[8]}
 _[2]={"return"}
 _[1]={"text",_[3]}
@@ -19,7 +19,7 @@ return {_[1],_[2]}
       text = "a"
     }, {
       tags = {},
-      text = "seen only once"
+      text = "seen only once "
     }, {
       tags = {},
       text = "b"