diff --git a/README.md b/README.md index 4e83256..fab71b1 100644 --- a/README.md +++ b/README.md @@ -1,144 +1,148 @@ -Lune -==== +Candran +======= +Candran is a dialect of the [Lua](http://www.lua.org) programming language which compiles to Lua. It adds a preprocessor and several useful syntax additions. -Lune is a simple [Lua](http://www.lua.org) dialect, which compile to normal Lua. It adds a preprocessor and some usefull syntax additions to the language, like += operators. - -Lune code example : +Candran code example : ````lua -#local language = args.lang or "en" -local a = 5 -a += 3 -#if language == "fr" then - print("Le resultat est "..a) -#elseif language == "en" then - print("The result is "..a) -#end +#import("lib.thing") +#local debug = args.debug or false + +local function debugArgs(func) + return function(...) + #if debug then + for _,arg in pairs({...}) do + print(arg, type(arg)) + end + #end + return func(...) + end +end + +@debugArgs +local function calculate() + local result = thing.do() + result += 25 + return result +end + +print(calculate()) ```` -This code will compile diffrently depending on the "lang" argument you pass to the compiler. - -Syntax details --------------- +The language +------------ ### Preprocessor -Before compiling, a preprocessor is run; it search the lines which start with a # and execute the Lune code after it. +Before compiling, Candran's preprocessor is run. It execute every line starting with a _#_ (ignoring whitespace) as Candran code. For example, ````lua #if args.lang == "fr" then - print("Ce programme a ete compile en francais") + print("Bonjour") #else - print("This program was compiled in english") + print("Hello") #end ```` -Output ````print("Ce programme a ete compile en francais")```` or ````print("This program was compiled in english")```` depending of the "lang" argument. +Will output ````print("Bonjour")```` or ````print("Hello")```` depending of the "lang" argument passed to the preprocessor. -In the preprocessor, the following global variables are available : -* ````lune```` : the Lune library table -* ````output```` : the preprocessor output string -* ````include(filename)```` : a function which copy the contents of the file filename to the output and add some code so it is equivalent to : +The preprocessor has access to the following variables : +* ````candran```` : the Candran library table. +* ````output```` : the preprocessor output string. +* ````import(module[, autoRequire])```` : a function which import a module. This is equivalent to use _require(module)_ in the Candran code, except the module will be embedded in the current file. _autoRequire_ (boolean, default true) indicate if the module should be automaticaly loaded in a local variable or not. If true, the local variable will have the name of the module. +* ````include(filename)```` : a function which copy the contents of the file _filename_ to the output. +* ````print(...)```` : instead of writing to stdout, _print(...)_ will write to the preprocessor output. For example, ````#print("hello()")```` will output ````hello()````. +* ````args```` : the arguments table passed to the compiler. Example use : ````withDebugTools = args["debug"]````. +* and every standard Lua library. - ````lua - filname = require("filename") or filename - ```` +### Syntax additions +After the preprocessor is run the Candran code is compiled to Lua. The Candran code adds the folowing syntax to Lua : +##### New assignment operators +* ````var += nb```` +* ````var -= nb```` +* ````var *= nb```` +* ````var /= nb```` +* ````var ^= nb```` +* ````var %= nb```` +* ````var ..= str```` - except that the required code is actually embedded in the file. -* ````rawInclude(filename)```` : a function which copy the contents of the file filename to the output, whithout modifications -* ````print(...)```` : instead of writing to stdout, write to the preprocessor output; for example, - - ````lua - local foo = "hello" - #print("foo ..= ' lune')") - print(foo) - ```` +For example, a ````var += nb```` assignment will be compiled into ````var = var + nb````. - will output : +##### Decorators +Candran supports function decorators similar to Python. A decorator is a function returning another function, and allows easy function modification with this syntax : +````lua +@decorator +function name(...) + ... +end +```` +This is equivalent to : +````lua +function name(...) + ... +end +name = decorator(name) +```` +The decorators can be chained. Note that Candran allows this syntax for every variable, not only functions. - ````lua - local foo = "hello" - foo = foo .. ' lune' - print(foo) - ```` - -* ````args```` : the arguments table passed to the compiler. Example use : - - ````lua - argumentValue = args["argumentName"] - ```` - -* And all the Lua standard libraries. - -### Compiler -After the preprocessor, the compiler is run; it translate Lune syntax to Lua syntax. What is translated to what : -* ````var += nb```` > ````var = var + nb```` -* ````var -= nb```` > ````var = var - nb```` -* ````var *= nb```` > ````var = var * nb```` -* ````var /= nb```` > ````var = var / nb```` -* ````var ^= nb```` > ````var = var ^ nb```` -* ````var %= nb```` > ````var = var % nb```` -* ````var ..= str```` > ````var = var .. str```` -* ````var++```` > ````var = var + 1```` -* ````var--```` > ````var = var - 1```` - -Command-line usage ------------------- +The library +----------- +### Command-line usage The library can be used standalone : - lua lune.lua +* ````lua candran.lua```` + + Display the information text (version and basic command-line usage). -Display a simple information text (version & basic command-line usage). +* ````lua candran.lua [arguments]```` + + Output to stdout the _filename_ Candran file, preprocessed (with _arguments_) and compiled to Lua. - lua lune.lua [arguments] + _arguments_ is of type ````--somearg value --anotherarg anothervalue ...````. -Output to stdout the Lune code compiled in Lua. -* arguments : - * input : input file name - * arguments : arguments to pass to the preprocessor (every argument is of type ````-- ````) -* example uses : + * example uses : - lua lune.lua foo.lune > foo.lua + ````lua candran.lua foo.can > foo.lua```` - compile foo.lune and write the result in foo.lua + preprocess and compile _foo.can_ and write the result in _foo.lua_. - lua lune.lua foo.lune --verbose true | lua + ````lua candran.lua foo.can --verbose true | lua```` - compile foo.lune with "verbose" set to true and execute it + preprocess _foo.can_ with _verbose_ set to _true_, compile it and execute it. -Library usage -------------- -Lune can also be used as a normal Lua library. For example, +### Library usage +Candran can also be used as a normal Lua library. For example, ````lua -local lune = require("lune") +local candran = require("candran") -local f = io.open("foo.lune") +local f = io.open("foo.can") local contents = f:read("*a") f:close() -local compiled = lune.make(contents, { lang = "fr" }) +local compiled = candran.make(contents, { lang = "fr" }) load(compiled)() ```` -will load Lune, read the file foo.lune, compile its contents with the argument "lang" set to "fr", and then execute the result. +Will load Candran, read the file _foo.can_, compile its contents with the argument _lang_ set to _"fr"_, and then execute the result. -Lune API : -* ````lune.VERSION```` : version string -* ````lune.syntax```` : syntax table used when compiling (TODO : need more explainations) -* ````lune.preprocess(code[, args])```` : return the Lune code preprocessed with args as argument table -* ````lune.compile(code)```` : return the Lune code compiled to Lua -* ````lune.make(code[, args])```` : return the Lune code preprocessed & compilled to Lua with args as argument table +The table returned by _require("candran")_ gives you access to : +* ````candran.VERSION```` : Candran's version string. +* ````candran.syntax```` : table containing all the syntax additions of Candran. +* ````candran.preprocess(code[, args])```` : return the Candran code _code_, preprocessed with _args_ as argument table. +* ````candran.compile(code)```` : return the Candran code compiled to Lua. +* ````candran.make(code[, args])```` : return the Candran code, preprocessed with _args_ as argument table and compilled to Lua. -Compiling Lune --------------- -Because the Lune compiler itself is written in Lune, you have to compile it with an already compiled version of Lune. This command will use the precompilled version in build/lune.lua to compile lune.lune and write the result in lune.lua : +### Compiling the library +The Candran library itself is written is Candran, so you have to compile it with an already compiled Candran library. + +This command will use the precompilled version of this repository (build/candran.lua) to compile _candran.can_ and write the result in _candran.lua_ : ```` -lua build/lune.lua lune.lune > lune.lua +lua build/candran.lua candran.can > candran.lua ```` -You can then test your build : +You can then run the tests on your build : ```` cd tests -lua test.lua ../lune.lua +lua test.lua ../candran.lua ```` \ No newline at end of file diff --git a/build/candran.lua b/build/candran.lua new file mode 100644 index 0000000..ac14ebe --- /dev/null +++ b/build/candran.lua @@ -0,0 +1,2764 @@ +--[[ +Candran language, preprocessor and compiler by Thomas99. + +LICENSE : +Copyright (c) 2015 Thomas99 + +This software is provided 'as-is', without any express or implied warranty. +In no event will the authors be held liable for any damages arising from the +use of this software. + +Permission is granted to anyone to use this software for any purpose, including +commercial applications, and to alter it and redistribute it freely, subject +to the following restrictions: + + 1. The origin of this software must not be misrepresented; you must not + claim that you wrote the original software. If you use this software in a + product, an acknowledgment in the product documentation would be appreciated + but is not required. + + 2. Altered source versions must be plainly marked as such, and must not be + misrepresented as being the original software. + + 3. This notice may not be removed or altered from any source distribution. +]] + +local candran = { + VERSION = "0.1.0", + syntax = { + assignment = { "+=", "-=", "*=", "/=", "^=", "%=", "..=" }, + decorator = "@" + } +} +package.loaded["candran"] = candran + +-- IMPORT OF MODULE "lib.table" -- +local function _() +--[[ +Table utility by Thomas99. + +LICENSE : +Copyright (c) 2015 Thomas99 + +This software is provided 'as-is', without any express or implied warranty. +In no event will the authors be held liable for any damages arising from the +use of this software. + +Permission is granted to anyone to use this software for any purpose, including +commercial applications, and to alter it and redistribute it freely, subject +to the following restrictions: + + 1. The origin of this software must not be misrepresented; you must not + claim that you wrote the original software. If you use this software in a + product, an acknowledgment in the product documentation would be appreciated + but is not required. + + 2. Altered source versions must be plainly marked as such, and must not be + misrepresented as being the original software. + + 3. This notice may not be removed or altered from any source distribution. +]] + +-- Diverses fonctions en rapport avec les tables. +-- v0.1.0 +-- +-- Changements : +-- - v0.1.0 : +-- Première version versionnée. Il a dû se passer des trucs avant mais j'ai pas noté :p + +-- Copie récursivement la table t dans la table dest (ou une table vide si non précisé) et la retourne +-- replace (false) : indique si oui ou non, les clefs existant déjà dans dest doivent être écrasées par celles de t +-- metatable (true) : copier ou non également les metatables +-- filter (function) : filtre, si retourne true copie l'objet, sinon ne le copie pas +-- Note : les metatables des objets ne sont jamais re-copiées (mais référence à la place), car sinon lors de la copie +-- la classe de ces objets changera pour une nouvelle classe, et c'est pas pratique :p +function table.copy(t, dest, replace, metatable, filter, copied) + local copied = copied or {} + local replace = replace or false + local metatable = (metatable==nil or metatable) and true + local filter = filter or function(name, source, destination) return true end + + if type(t) ~= "table" then + return t + elseif copied[t] then -- si la table a déjà été copiée + return copied[t] + end + + local dest = dest or {} -- la copie + + copied[t] = dest -- on marque la table comme copiée + + for k, v in pairs(t) do + if filter(k, t, dest) then + if replace then + dest[k] = table.copy(v, dest[k], replace, metatable, filter, copied) + else + if dest[k] == nil or type(v) == "table" then -- si la clef n'existe pas déjà dans dest ou si c'est une table à copier + dest[k] = table.copy(v, dest[k], replace, metatable, filter, copied) + end + end + end + end + + -- copie des metatables + if metatable then + if t.__classe then + setmetatable(dest, getmetatable(t)) + else + setmetatable(dest, table.copy(getmetatable(t), getmetatable(dest), replace, filter)) + end + end + + return dest +end + +-- retourne true si value est dans la table +function table.isIn(table, value) + for _,v in pairs(table) do + if v == value then + return true + end + end + return false +end + +-- retourne la longueur exacte d'une table (fonctionne sur les tables à clef) +function table.len(t) + local len=0 + for i in pairs(t) do + len=len+1 + end + return len +end + +-- Sépare str en éléments séparés par le pattern et retourne une table +function string.split(str, pattern) + local t = {} + local pos = 0 + + for i,p in string.gmatch(str, "(.-)"..pattern.."()") do + table.insert(t, i) + pos = p + end + + table.insert(t, str:sub(pos)) + + return t +end +end +local table = _() or table +package.loaded["lib.table"] = table or true +-- END OF IMPORT OF MODULE "lib.table" -- +-- IMPORT OF MODULE "lib.LuaMinify.Util" -- +local function _() +--[[ +This file is part of LuaMinify by stravant (https://github.com/stravant/LuaMinify). + +LICENSE : +The MIT License (MIT) + +Copyright (c) 2012-2013 + +Permission is hereby granted, free of charge, to any person obtaining a copy of +this software and associated documentation files (the "Software"), to deal in +the Software without restriction, including without limitation the rights to +use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of +the Software, and to permit persons to whom the Software is furnished to do so, +subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS +FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR +COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER +IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN +CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. +]] + +-- +-- Util.lua +-- +-- Provides some common utilities shared throughout the project. +-- + +local function lookupify(tb) + for _, v in pairs(tb) do + tb[v] = true + end + return tb +end + + +local function CountTable(tb) + local c = 0 + for _ in pairs(tb) do c = c + 1 end + return c +end + + +local function PrintTable(tb, atIndent) + if tb.Print then + return tb.Print() + end + atIndent = atIndent or 0 + local useNewlines = (CountTable(tb) > 1) + local baseIndent = string.rep(' ', atIndent+1) + local out = "{"..(useNewlines and '\n' or '') + for k, v in pairs(tb) do + if type(v) ~= 'function' then + --do + out = out..(useNewlines and baseIndent or '') + if type(k) == 'number' then + --nothing to do + elseif type(k) == 'string' and k:match("^[A-Za-z_][A-Za-z0-9_]*$") then + out = out..k.." = " + elseif type(k) == 'string' then + out = out.."[\""..k.."\"] = " + else + out = out.."["..tostring(k).."] = " + end + if type(v) == 'string' then + out = out.."\""..v.."\"" + elseif type(v) == 'number' then + out = out..v + elseif type(v) == 'table' then + out = out..PrintTable(v, atIndent+(useNewlines and 1 or 0)) + else + out = out..tostring(v) + end + if next(tb, k) then + out = out.."," + end + if useNewlines then + out = out..'\n' + end + end + end + out = out..(useNewlines and string.rep(' ', atIndent) or '').."}" + return out +end + + +local function splitLines(str) + if str:match("\n") then + local lines = {} + for line in str:gmatch("[^\n]*") do + table.insert(lines, line) + end + assert(#lines > 0) + return lines + else + return { str } + end +end + + +local function printf(fmt, ...) + return print(string.format(fmt, ...)) +end + + +return { + PrintTable = PrintTable, + CountTable = CountTable, + lookupify = lookupify, + splitLines = splitLines, + printf = printf, +} + +end +local Util = _() or Util +package.loaded["lib.LuaMinify.Util"] = Util or true +-- END OF IMPORT OF MODULE "lib.LuaMinify.Util" -- +-- IMPORT OF MODULE "lib.LuaMinify.Scope" -- +local function _() +--[[ +This file is part of LuaMinify by stravant (https://github.com/stravant/LuaMinify). + +LICENSE : +The MIT License (MIT) + +Copyright (c) 2012-2013 + +Permission is hereby granted, free of charge, to any person obtaining a copy of +this software and associated documentation files (the "Software"), to deal in +the Software without restriction, including without limitation the rights to +use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of +the Software, and to permit persons to whom the Software is furnished to do so, +subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS +FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR +COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER +IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN +CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. +]] + +local Scope = { + new = function(self, parent) + local s = { + Parent = parent, + Locals = { }, + Globals = { }, + oldLocalNamesMap = { }, + oldGlobalNamesMap = { }, + Children = { }, + } + + if parent then + table.insert(parent.Children, s) + end + + return setmetatable(s, { __index = self }) + end, + + AddLocal = function(self, v) + table.insert(self.Locals, v) + end, + + AddGlobal = function(self, v) + table.insert(self.Globals, v) + end, + + CreateLocal = function(self, name) + local v + v = self:GetLocal(name) + if v then return v end + v = { } + v.Scope = self + v.Name = name + v.IsGlobal = false + v.CanRename = true + v.References = 1 + self:AddLocal(v) + return v + end, + + GetLocal = function(self, name) + for k, var in pairs(self.Locals) do + if var.Name == name then return var end + end + + if self.Parent then + return self.Parent:GetLocal(name) + end + end, + + GetOldLocal = function(self, name) + if self.oldLocalNamesMap[name] then + return self.oldLocalNamesMap[name] + end + return self:GetLocal(name) + end, + + mapLocal = function(self, name, var) + self.oldLocalNamesMap[name] = var + end, + + GetOldGlobal = function(self, name) + if self.oldGlobalNamesMap[name] then + return self.oldGlobalNamesMap[name] + end + return self:GetGlobal(name) + end, + + mapGlobal = function(self, name, var) + self.oldGlobalNamesMap[name] = var + end, + + GetOldVariable = function(self, name) + return self:GetOldLocal(name) or self:GetOldGlobal(name) + end, + + RenameLocal = function(self, oldName, newName) + oldName = type(oldName) == 'string' and oldName or oldName.Name + local found = false + local var = self:GetLocal(oldName) + if var then + var.Name = newName + self:mapLocal(oldName, var) + found = true + end + if not found and self.Parent then + self.Parent:RenameLocal(oldName, newName) + end + end, + + RenameGlobal = function(self, oldName, newName) + oldName = type(oldName) == 'string' and oldName or oldName.Name + local found = false + local var = self:GetGlobal(oldName) + if var then + var.Name = newName + self:mapGlobal(oldName, var) + found = true + end + if not found and self.Parent then + self.Parent:RenameGlobal(oldName, newName) + end + end, + + RenameVariable = function(self, oldName, newName) + oldName = type(oldName) == 'string' and oldName or oldName.Name + if self:GetLocal(oldName) then + self:RenameLocal(oldName, newName) + else + self:RenameGlobal(oldName, newName) + end + end, + + GetAllVariables = function(self) + local ret = self:getVars(true) -- down + for k, v in pairs(self:getVars(false)) do -- up + table.insert(ret, v) + end + return ret + end, + + getVars = function(self, top) + local ret = { } + if top then + for k, v in pairs(self.Children) do + for k2, v2 in pairs(v:getVars(true)) do + table.insert(ret, v2) + end + end + else + for k, v in pairs(self.Locals) do + table.insert(ret, v) + end + for k, v in pairs(self.Globals) do + table.insert(ret, v) + end + if self.Parent then + for k, v in pairs(self.Parent:getVars(false)) do + table.insert(ret, v) + end + end + end + return ret + end, + + CreateGlobal = function(self, name) + local v + v = self:GetGlobal(name) + if v then return v end + v = { } + v.Scope = self + v.Name = name + v.IsGlobal = true + v.CanRename = true + v.References = 1 + self:AddGlobal(v) + return v + end, + + GetGlobal = function(self, name) + for k, v in pairs(self.Globals) do + if v.Name == name then return v end + end + + if self.Parent then + return self.Parent:GetGlobal(name) + end + end, + + GetVariable = function(self, name) + return self:GetLocal(name) or self:GetGlobal(name) + end, + + ObfuscateLocals = function(self, recommendedMaxLength, validNameChars) + recommendedMaxLength = recommendedMaxLength or 7 + local chars = validNameChars or "QWERTYUIOPASDFGHJKLZXCVBNMqwertyuioplkjhgfdsazxcvbnm_" + local chars2 = validNameChars or "QWERTYUIOPASDFGHJKLZXCVBNMqwertyuioplkjhgfdsazxcvbnm_1234567890" + for _, var in pairs(self.Locals) do + local id = "" + local tries = 0 + repeat + local n = math.random(1, #chars) + id = id .. chars:sub(n, n) + for i = 1, math.random(0, tries > 5 and 30 or recommendedMaxLength) do + local n = math.random(1, #chars2) + id = id .. chars2:sub(n, n) + end + tries = tries + 1 + until not self:GetVariable(id) + self:RenameLocal(var.Name, id) + end + end, +} + +return Scope + +end +local Scope = _() or Scope +package.loaded["lib.LuaMinify.Scope"] = Scope or true +-- END OF IMPORT OF MODULE "lib.LuaMinify.Scope" -- +-- IMPORT OF MODULE "lib.LuaMinify.ParseCandran" -- +local function _() +-- +-- CANDRAN +-- Based on the ParseLua.lua of LuaMinify. +-- Modified by Thomas99 to parse Candran code. +-- +-- Modified parts are marked with "-- CANDRAN" comments. +-- + +--[[ +This file is part of LuaMinify by stravant (https://github.com/stravant/LuaMinify). + +LICENSE : +The MIT License (MIT) + +Copyright (c) 2012-2013 + +Permission is hereby granted, free of charge, to any person obtaining a copy of +this software and associated documentation files (the "Software"), to deal in +the Software without restriction, including without limitation the rights to +use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of +the Software, and to permit persons to whom the Software is furnished to do so, +subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS +FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR +COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER +IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN +CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. +]] + +-- +-- ParseLua.lua +-- +-- The main lua parser and lexer. +-- LexLua returns a Lua token stream, with tokens that preserve +-- all whitespace formatting information. +-- ParseLua returns an AST, internally relying on LexLua. +-- + +--require'LuaMinify.Strict' -- CANDRAN : comment, useless here + +-- CANDRAN : add Candran syntaxic additions +local candran = require("candran").syntax + +local util = require 'lib.LuaMinify.Util' +local lookupify = util.lookupify + +local WhiteChars = lookupify{ ' ', '\n', '\t', '\r'} +local EscapeLookup = {['\r'] = '\\r', ['\n'] = '\\n', ['\t'] = '\\t', ['"'] = '\\"', ["'"] = "\\'"} +local LowerChars = lookupify{ 'a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'j', 'k', 'l', 'm', 'n', 'o', 'p', 'q', 'r', 's', 't', 'u', 'v', 'w', 'x', 'y', 'z'} +local UpperChars = lookupify{ 'A', 'B', 'C', 'D', 'E', 'F', 'G', 'H', 'I', 'J', 'K', 'L', 'M', 'N', 'O', 'P', 'Q', 'R', 'S', 'T', 'U', 'V', 'W', 'X', 'Y', 'Z'} +local Digits = lookupify{ '0', '1', '2', '3', '4', '5', '6', '7', '8', '9'} +local HexDigits = lookupify{ '0', '1', '2', '3', '4', '5', '6', '7', '8', '9', 'A', 'a', 'B', 'b', 'C', 'c', 'D', 'd', 'E', 'e', 'F', 'f'} + +local Symbols = lookupify{ '+', '-', '*', '/', '^', '%', ',', '{', '}', '[', ']', '(', ')', ';', '#', + table.unpack(candran.assignment)} -- CANDRAN : Candran symbols +local Scope = require'lib.LuaMinify.Scope' + +local Keywords = lookupify{ 'and', 'break', 'do', 'else', 'elseif', 'end', 'false', 'for', 'function', 'goto', 'if', 'in', 'local', 'nil', 'not', 'or', 'repeat', 'return', 'then', 'true', 'until', 'while', candran.decorator -- LUA : Candran keywords +}; + +local function LexLua(src) + --token dump + local tokens = {} + + local st, err = pcall(function() + --line / char / pointer tracking + local p = 1 + local line = 1 + local char = 1 + + --get / peek functions + local function get() + local c = src:sub(p,p) + if c == '\n' then + char = 1 + line = line + 1 + else + char = char + 1 + end + p = p + 1 + return c + end + local function peek(n) + n = n or 0 + return src:sub(p+n,p+n) + end + local function consume(chars) + local c = peek() + for i = 1, #chars do + if c == chars:sub(i,i) then return get() end + end + end + + --shared stuff + local function generateError(err) + return error(">> :"..line..":"..char..": "..err, 0) + end + + local function tryGetLongString() + local start = p + if peek() == '[' then + local equalsCount = 0 + local depth = 1 + while peek(equalsCount+1) == '=' do + equalsCount = equalsCount + 1 + end + if peek(equalsCount+1) == '[' then + --start parsing the string. Strip the starting bit + for _ = 0, equalsCount+1 do get() end + + --get the contents + local contentStart = p + while true do + --check for eof + if peek() == '' then + generateError("Expected `]"..string.rep('=', equalsCount).."]` near .", 3) + end + + --check for the end + local foundEnd = true + if peek() == ']' then + for i = 1, equalsCount do + if peek(i) ~= '=' then foundEnd = false end + end + if peek(equalsCount+1) ~= ']' then + foundEnd = false + end + else + if peek() == '[' then + -- is there an embedded long string? + local embedded = true + for i = 1, equalsCount do + if peek(i) ~= '=' then + embedded = false + break + end + end + if peek(equalsCount + 1) == '[' and embedded then + -- oh look, there was + depth = depth + 1 + for i = 1, (equalsCount + 2) do + get() + end + end + end + foundEnd = false + end + -- + if foundEnd then + depth = depth - 1 + if depth == 0 then + break + else + for i = 1, equalsCount + 2 do + get() + end + end + else + get() + end + end + + --get the interior string + local contentString = src:sub(contentStart, p-1) + + --found the end. Get rid of the trailing bit + for i = 0, equalsCount+1 do get() end + + --get the exterior string + local longString = src:sub(start, p-1) + + --return the stuff + return contentString, longString + else + return nil + end + else + return nil + end + end + + --main token emitting loop + while true do + --get leading whitespace. The leading whitespace will include any comments + --preceding the token. This prevents the parser needing to deal with comments + --separately. + local leading = { } + local leadingWhite = '' + local longStr = false + while true do + local c = peek() + if c == '#' and peek(1) == '!' and line == 1 then + -- #! shebang for linux scripts + get() + get() + leadingWhite = "#!" + while peek() ~= '\n' and peek() ~= '' do + leadingWhite = leadingWhite .. get() + end + local token = { + Type = 'Comment', + CommentType = 'Shebang', + Data = leadingWhite, + Line = line, + Char = char + } + token.Print = function() + return "<"..(token.Type .. string.rep(' ', 7-#token.Type)).." "..(token.Data or '').." >" + end + leadingWhite = "" + table.insert(leading, token) + end + if c == ' ' or c == '\t' then + --whitespace + --leadingWhite = leadingWhite..get() + local c2 = get() -- ignore whitespace + table.insert(leading, { Type = 'Whitespace', Line = line, Char = char, Data = c2 }) + elseif c == '\n' or c == '\r' then + local nl = get() + if leadingWhite ~= "" then + local token = { + Type = 'Comment', + CommentType = longStr and 'LongComment'or 'Comment', + Data = leadingWhite, + Line = line, + Char = char, + } + token.Print = function() + return "<"..(token.Type .. string.rep(' ', 7-#token.Type)).." "..(token.Data or '').." >" + end + table.insert(leading, token) + leadingWhite = "" + end + table.insert(leading, { Type = 'Whitespace', Line = line, Char = char, Data = nl }) + elseif c == '-' and peek(1) == '-' then + --comment + get() + get() + leadingWhite = leadingWhite .. '--' + local _, wholeText = tryGetLongString() + if wholeText then + leadingWhite = leadingWhite..wholeText + longStr = true + else + while peek() ~= '\n' and peek() ~= '' do + leadingWhite = leadingWhite..get() + end + end + else + break + end + end + if leadingWhite ~= "" then + local token = { + Type = 'Comment', + CommentType = longStr and 'LongComment'or 'Com mnment', + Data = leadingWhite, + Line = line, + Char = char, + } + token.Print = function() + return "<"..(token.Type .. string.rep(' ', 7-#token.Type)).." "..(token.Data or '').." >" + end + table.insert(leading, token) + end + + --get the initial char + local thisLine = line + local thisChar = char + local errorAt = ":"..line..":"..char..":> " + local c = peek() + + --symbol to emit + local toEmit = nil + + --branch on type + if c == '' then + --eof + toEmit = { Type = 'Eof' } + + -- CANDRAN : add decorator symbol (@) + elseif c == candran.decorator then + get() + toEmit = {Type = 'Keyword', Data = c} + + elseif UpperChars[c] or LowerChars[c] or c == '_' then + --ident or keyword + local start = p + repeat + get() + c = peek() + until not (UpperChars[c] or LowerChars[c] or Digits[c] or c == '_') + local dat = src:sub(start, p-1) + if Keywords[dat] then + toEmit = {Type = 'Keyword', Data = dat} + else + toEmit = {Type = 'Ident', Data = dat} + end + + elseif Digits[c] or (peek() == '.' and Digits[peek(1)]) then + --number const + local start = p + if c == '0' and peek(1) == 'x' then + get();get() + while HexDigits[peek()] do get() end + if consume('Pp') then + consume('+-') + while Digits[peek()] do get() end + end + else + while Digits[peek()] do get() end + if consume('.') then + while Digits[peek()] do get() end + end + if consume('Ee') then + consume('+-') + while Digits[peek()] do get() end + end + end + toEmit = {Type = 'Number', Data = src:sub(start, p-1)} + + elseif c == '\'' or c == '\"' then + local start = p + --string const + local delim = get() + local contentStart = p + while true do + local c = get() + if c == '\\' then + get() --get the escape char + elseif c == delim then + break + elseif c == '' then + generateError("Unfinished string near ") + end + end + local content = src:sub(contentStart, p-2) + local constant = src:sub(start, p-1) + toEmit = {Type = 'String', Data = constant, Constant = content} + + -- CANDRAN : accept 3 and 2 caracters symbols + elseif Symbols[c..peek(1)..peek(2)] then + local c = c..peek(1)..peek(2) + get() get() get() + toEmit = {Type = 'Symbol', Data = c} + elseif Symbols[c..peek(1)] then + local c = c..peek(1) + get() get() + toEmit = {Type = 'Symbol', Data = c} + + elseif c == '[' then + local content, wholetext = tryGetLongString() + if wholetext then + toEmit = {Type = 'String', Data = wholetext, Constant = content} + else + get() + toEmit = {Type = 'Symbol', Data = '['} + end + + elseif consume('>=<') then + if consume('=') then + toEmit = {Type = 'Symbol', Data = c..'='} + else + toEmit = {Type = 'Symbol', Data = c} + end + + elseif consume('~') then + if consume('=') then + toEmit = {Type = 'Symbol', Data = '~='} + else + generateError("Unexpected symbol `~` in source.", 2) + end + + elseif consume('.') then + if consume('.') then + if consume('.') then + toEmit = {Type = 'Symbol', Data = '...'} + else + toEmit = {Type = 'Symbol', Data = '..'} + end + else + toEmit = {Type = 'Symbol', Data = '.'} + end + + elseif consume(':') then + if consume(':') then + toEmit = {Type = 'Symbol', Data = '::'} + else + toEmit = {Type = 'Symbol', Data = ':'} + end + + elseif Symbols[c] then + get() + toEmit = {Type = 'Symbol', Data = c} + + else + local contents, all = tryGetLongString() + if contents then + toEmit = {Type = 'String', Data = all, Constant = contents} + else + generateError("Unexpected Symbol `"..c.."` in source.", 2) + end + end + + --add the emitted symbol, after adding some common data + toEmit.LeadingWhite = leading -- table of leading whitespace/comments + --for k, tok in pairs(leading) do + -- tokens[#tokens + 1] = tok + --end + + toEmit.Line = thisLine + toEmit.Char = thisChar + toEmit.Print = function() + return "<"..(toEmit.Type..string.rep(' ', 7-#toEmit.Type)).." "..(toEmit.Data or '').." >" + end + tokens[#tokens+1] = toEmit + + --halt after eof has been emitted + if toEmit.Type == 'Eof' then break end + end + end) + if not st then + return false, err + end + + --public interface: + local tok = {} + local savedP = {} + local p = 1 + + function tok:getp() + return p + end + + function tok:setp(n) + p = n + end + + function tok:getTokenList() + return tokens + end + + --getters + function tok:Peek(n) + n = n or 0 + return tokens[math.min(#tokens, p+n)] + end + function tok:Get(tokenList) + local t = tokens[p] + p = math.min(p + 1, #tokens) + if tokenList then + table.insert(tokenList, t) + end + return t + end + function tok:Is(t) + return tok:Peek().Type == t + end + + --save / restore points in the stream + function tok:Save() + savedP[#savedP+1] = p + end + function tok:Commit() + savedP[#savedP] = nil + end + function tok:Restore() + p = savedP[#savedP] + savedP[#savedP] = nil + end + + --either return a symbol if there is one, or return true if the requested + --symbol was gotten. + function tok:ConsumeSymbol(symb, tokenList) + local t = self:Peek() + if t.Type == 'Symbol' then + if symb then + if t.Data == symb then + self:Get(tokenList) + return true + else + return nil + end + else + self:Get(tokenList) + return t + end + else + return nil + end + end + + function tok:ConsumeKeyword(kw, tokenList) + local t = self:Peek() + if t.Type == 'Keyword' and t.Data == kw then + self:Get(tokenList) + return true + else + return nil + end + end + + function tok:IsKeyword(kw) + local t = tok:Peek() + return t.Type == 'Keyword' and t.Data == kw + end + + function tok:IsSymbol(s) + local t = tok:Peek() + return t.Type == 'Symbol' and t.Data == s + end + + function tok:IsEof() + return tok:Peek().Type == 'Eof' + end + + return true, tok +end + + +local function ParseLua(src) + local st, tok + if type(src) ~= 'table' then + st, tok = LexLua(src) + else + st, tok = true, src + end + if not st then + return false, tok + end + -- + local function GenerateError(msg) + local err = ">> :"..tok:Peek().Line..":"..tok:Peek().Char..": "..msg.."\n" + --find the line + local lineNum = 0 + if type(src) == 'string' then + for line in src:gmatch("[^\n]*\n?") do + if line:sub(-1,-1) == '\n' then line = line:sub(1,-2) end + lineNum = lineNum+1 + if lineNum == tok:Peek().Line then + err = err..">> `"..line:gsub('\t',' ').."`\n" + for i = 1, tok:Peek().Char do + local c = line:sub(i,i) + if c == '\t' then + err = err..' ' + else + err = err..' ' + end + end + err = err.." ^^^^" + break + end + end + end + return err + end + -- + local VarUid = 0 + -- No longer needed: handled in Scopes now local GlobalVarGetMap = {} + local VarDigits = { '_', 'a', 'b', 'c', 'd'} + local function CreateScope(parent) + --[[ + local scope = {} + scope.Parent = parent + scope.LocalList = {} + scope.LocalMap = {} + + function scope:ObfuscateVariables() + for _, var in pairs(scope.LocalList) do + local id = "" + repeat + local chars = "QWERTYUIOPASDFGHJKLZXCVBNMqwertyuioplkjhgfdsazxcvbnm_" + local chars2 = "QWERTYUIOPASDFGHJKLZXCVBNMqwertyuioplkjhgfdsazxcvbnm_1234567890" + local n = math.random(1, #chars) + id = id .. chars:sub(n, n) + for i = 1, math.random(0,20) do + local n = math.random(1, #chars2) + id = id .. chars2:sub(n, n) + end + until not GlobalVarGetMap[id] and not parent:GetLocal(id) and not scope.LocalMap[id] + var.Name = id + scope.LocalMap[id] = var + end + end + + scope.RenameVars = scope.ObfuscateVariables + + -- Renames a variable from this scope and down. + -- Does not rename global variables. + function scope:RenameVariable(old, newName) + if type(old) == "table" then -- its (theoretically) an AstNode variable + old = old.Name + end + for _, var in pairs(scope.LocalList) do + if var.Name == old then + var.Name = newName + scope.LocalMap[newName] = var + end + end + end + + function scope:GetLocal(name) + --first, try to get my variable + local my = scope.LocalMap[name] + if my then return my end + + --next, try parent + if scope.Parent then + local par = scope.Parent:GetLocal(name) + if par then return par end + end + + return nil + end + + function scope:CreateLocal(name) + --create my own var + local my = {} + my.Scope = scope + my.Name = name + my.CanRename = true + -- + scope.LocalList[#scope.LocalList+1] = my + scope.LocalMap[name] = my + -- + return my + end]] + local scope = Scope:new(parent) + scope.RenameVars = scope.ObfuscateLocals + scope.ObfuscateVariables = scope.ObfuscateLocals + scope.Print = function() return "" end + return scope + end + + local ParseExpr + local ParseStatementList + local ParseSimpleExpr, + ParseSubExpr, + ParsePrimaryExpr, + ParseSuffixedExpr + + local function ParseFunctionArgsAndBody(scope, tokenList) + local funcScope = CreateScope(scope) + if not tok:ConsumeSymbol('(', tokenList) then + return false, GenerateError("`(` expected.") + end + + --arg list + local argList = {} + local isVarArg = false + while not tok:ConsumeSymbol(')', tokenList) do + if tok:Is('Ident') then + local arg = funcScope:CreateLocal(tok:Get(tokenList).Data) + argList[#argList+1] = arg + if not tok:ConsumeSymbol(',', tokenList) then + if tok:ConsumeSymbol(')', tokenList) then + break + else + return false, GenerateError("`)` expected.") + end + end + elseif tok:ConsumeSymbol('...', tokenList) then + isVarArg = true + if not tok:ConsumeSymbol(')', tokenList) then + return false, GenerateError("`...` must be the last argument of a function.") + end + break + else + return false, GenerateError("Argument name or `...` expected") + end + end + + --body + local st, body = ParseStatementList(funcScope) + if not st then return false, body end + + --end + if not tok:ConsumeKeyword('end', tokenList) then + return false, GenerateError("`end` expected after function body") + end + local nodeFunc = {} + nodeFunc.AstType = 'Function' + nodeFunc.Scope = funcScope + nodeFunc.Arguments = argList + nodeFunc.Body = body + nodeFunc.VarArg = isVarArg + nodeFunc.Tokens = tokenList + -- + return true, nodeFunc + end + + + function ParsePrimaryExpr(scope) + local tokenList = {} + + if tok:ConsumeSymbol('(', tokenList) then + local st, ex = ParseExpr(scope) + if not st then return false, ex end + if not tok:ConsumeSymbol(')', tokenList) then + return false, GenerateError("`)` Expected.") + end + if false then + --save the information about parenthesized expressions somewhere + ex.ParenCount = (ex.ParenCount or 0) + 1 + return true, ex + else + local parensExp = {} + parensExp.AstType = 'Parentheses' + parensExp.Inner = ex + parensExp.Tokens = tokenList + return true, parensExp + end + + elseif tok:Is('Ident') then + local id = tok:Get(tokenList) + local var = scope:GetLocal(id.Data) + if not var then + var = scope:GetGlobal(id.Data) + if not var then + var = scope:CreateGlobal(id.Data) + else + var.References = var.References + 1 + end + else + var.References = var.References + 1 + end + -- + local nodePrimExp = {} + nodePrimExp.AstType = 'VarExpr' + nodePrimExp.Name = id.Data + nodePrimExp.Variable = var + nodePrimExp.Tokens = tokenList + -- + return true, nodePrimExp + else + return false, GenerateError("primary expression expected") + end + end + + function ParseSuffixedExpr(scope, onlyDotColon) + --base primary expression + local st, prim = ParsePrimaryExpr(scope) + if not st then return false, prim end + -- + while true do + local tokenList = {} + + if tok:IsSymbol('.') or tok:IsSymbol(':') then + local symb = tok:Get(tokenList).Data + if not tok:Is('Ident') then + return false, GenerateError(" expected.") + end + local id = tok:Get(tokenList) + local nodeIndex = {} + nodeIndex.AstType = 'MemberExpr' + nodeIndex.Base = prim + nodeIndex.Indexer = symb + nodeIndex.Ident = id + nodeIndex.Tokens = tokenList + -- + prim = nodeIndex + + elseif not onlyDotColon and tok:ConsumeSymbol('[', tokenList) then + local st, ex = ParseExpr(scope) + if not st then return false, ex end + if not tok:ConsumeSymbol(']', tokenList) then + return false, GenerateError("`]` expected.") + end + local nodeIndex = {} + nodeIndex.AstType = 'IndexExpr' + nodeIndex.Base = prim + nodeIndex.Index = ex + nodeIndex.Tokens = tokenList + -- + prim = nodeIndex + + elseif not onlyDotColon and tok:ConsumeSymbol('(', tokenList) then + local args = {} + while not tok:ConsumeSymbol(')', tokenList) do + local st, ex = ParseExpr(scope) + if not st then return false, ex end + args[#args+1] = ex + if not tok:ConsumeSymbol(',', tokenList) then + if tok:ConsumeSymbol(')', tokenList) then + break + else + return false, GenerateError("`)` Expected.") + end + end + end + local nodeCall = {} + nodeCall.AstType = 'CallExpr' + nodeCall.Base = prim + nodeCall.Arguments = args + nodeCall.Tokens = tokenList + -- + prim = nodeCall + + elseif not onlyDotColon and tok:Is('String') then + --string call + local nodeCall = {} + nodeCall.AstType = 'StringCallExpr' + nodeCall.Base = prim + nodeCall.Arguments = { tok:Get(tokenList) } + nodeCall.Tokens = tokenList + -- + prim = nodeCall + + elseif not onlyDotColon and tok:IsSymbol('{') then + --table call + local st, ex = ParseSimpleExpr(scope) + -- FIX: ParseExpr(scope) parses the table AND and any following binary expressions. + -- We just want the table + if not st then return false, ex end + local nodeCall = {} + nodeCall.AstType = 'TableCallExpr' + nodeCall.Base = prim + nodeCall.Arguments = { ex } + nodeCall.Tokens = tokenList + -- + prim = nodeCall + + else + break + end + end + return true, prim + end + + + function ParseSimpleExpr(scope) + local tokenList = {} + + if tok:Is('Number') then + local nodeNum = {} + nodeNum.AstType = 'NumberExpr' + nodeNum.Value = tok:Get(tokenList) + nodeNum.Tokens = tokenList + return true, nodeNum + + elseif tok:Is('String') then + local nodeStr = {} + nodeStr.AstType = 'StringExpr' + nodeStr.Value = tok:Get(tokenList) + nodeStr.Tokens = tokenList + return true, nodeStr + + elseif tok:ConsumeKeyword('nil', tokenList) then + local nodeNil = {} + nodeNil.AstType = 'NilExpr' + nodeNil.Tokens = tokenList + return true, nodeNil + + elseif tok:IsKeyword('false') or tok:IsKeyword('true') then + local nodeBoolean = {} + nodeBoolean.AstType = 'BooleanExpr' + nodeBoolean.Value = (tok:Get(tokenList).Data == 'true') + nodeBoolean.Tokens = tokenList + return true, nodeBoolean + + elseif tok:ConsumeSymbol('...', tokenList) then + local nodeDots = {} + nodeDots.AstType = 'DotsExpr' + nodeDots.Tokens = tokenList + return true, nodeDots + + elseif tok:ConsumeSymbol('{', tokenList) then + local v = {} + v.AstType = 'ConstructorExpr' + v.EntryList = {} + -- + while true do + -- CANDRAN : read decorator(s) + local decorated = false + local decoratorChain = {} + while tok:ConsumeKeyword(candran.decorator) do + if not tok:Is('Ident') then + return false, GenerateError("Decorator name expected") + end + -- CANDRAN : get decorator name + local st, decorator = ParseExpr(scope) + if not st then return false, ex end + + table.insert(decoratorChain, decorator) + decorated = true + end + + if tok:IsSymbol('[', tokenList) then + --key + tok:Get(tokenList) + local st, key = ParseExpr(scope) + if not st then + return false, GenerateError("Key Expression Expected") + end + if not tok:ConsumeSymbol(']', tokenList) then + return false, GenerateError("`]` Expected") + end + if not tok:ConsumeSymbol('=', tokenList) then + return false, GenerateError("`=` Expected") + end + local st, value = ParseExpr(scope) + if not st then + return false, GenerateError("Value Expression Expected") + end + v.EntryList[#v.EntryList+1] = { + Type = 'Key'; + Key = key; + Value = value; + } + + elseif tok:Is('Ident') then + --value or key + local lookahead = tok:Peek(1) + if lookahead.Type == 'Symbol' and lookahead.Data == '=' then + --we are a key + local key = tok:Get(tokenList) + if not tok:ConsumeSymbol('=', tokenList) then + return false, GenerateError("`=` Expected") + end + local st, value = ParseExpr(scope) + if not st then + return false, GenerateError("Value Expression Expected") + end + v.EntryList[#v.EntryList+1] = { + Type = 'KeyString'; + Key = key.Data; + Value = value; + } + + else + --we are a value + local st, value = ParseExpr(scope) + if not st then + return false, GenerateError("Value Exected") + end + v.EntryList[#v.EntryList+1] = { + Type = 'Value'; + Value = value; + } + + end + elseif tok:ConsumeSymbol('}', tokenList) then + break + + else + --value + local st, value = ParseExpr(scope) + v.EntryList[#v.EntryList+1] = { + Type = 'Value'; + Value = value; + } + if not st then + return false, GenerateError("Value Expected") + end + end + + -- CANDRAN : decorate entry + if decorated then + v.EntryList[#v.EntryList].Decorated = true + v.EntryList[#v.EntryList].DecoratorChain = decoratorChain + end + + if tok:ConsumeSymbol(';', tokenList) or tok:ConsumeSymbol(',', tokenList) then + --all is good + elseif tok:ConsumeSymbol('}', tokenList) then + break + else + return false, GenerateError("`}` or table entry Expected") + end + end + v.Tokens = tokenList + return true, v + + elseif tok:ConsumeKeyword('function', tokenList) then + local st, func = ParseFunctionArgsAndBody(scope, tokenList) + if not st then return false, func end + -- + func.IsLocal = true + return true, func + + else + return ParseSuffixedExpr(scope) + end + end + + + local unops = lookupify{ '-', 'not', '#'} + local unopprio = 8 + local priority = { + ['+'] = { 6, 6}; + ['-'] = { 6, 6}; + ['%'] = { 7, 7}; + ['/'] = { 7, 7}; + ['*'] = { 7, 7}; + ['^'] = { 10, 9}; + ['..'] = { 5, 4}; + ['=='] = { 3, 3}; + ['<'] = { 3, 3}; + ['<='] = { 3, 3}; + ['~='] = { 3, 3}; + ['>'] = { 3, 3}; + ['>='] = { 3, 3}; + ['and'] = { 2, 2}; + ['or'] = { 1, 1}; + } + function ParseSubExpr(scope, level) + --base item, possibly with unop prefix + local st, exp + if unops[tok:Peek().Data] then + local tokenList = {} + local op = tok:Get(tokenList).Data + st, exp = ParseSubExpr(scope, unopprio) + if not st then return false, exp end + local nodeEx = {} + nodeEx.AstType = 'UnopExpr' + nodeEx.Rhs = exp + nodeEx.Op = op + nodeEx.OperatorPrecedence = unopprio + nodeEx.Tokens = tokenList + exp = nodeEx + else + st, exp = ParseSimpleExpr(scope) + if not st then return false, exp end + end + + --next items in chain + while true do + local prio = priority[tok:Peek().Data] + if prio and prio[1] > level then + local tokenList = {} + local op = tok:Get(tokenList).Data + local st, rhs = ParseSubExpr(scope, prio[2]) + if not st then return false, rhs end + local nodeEx = {} + nodeEx.AstType = 'BinopExpr' + nodeEx.Lhs = exp + nodeEx.Op = op + nodeEx.OperatorPrecedence = prio[1] + nodeEx.Rhs = rhs + nodeEx.Tokens = tokenList + -- + exp = nodeEx + else + break + end + end + + return true, exp + end + + + ParseExpr = function(scope) + return ParseSubExpr(scope, 0) + end + + + local function ParseStatement(scope) + local stat = nil + local tokenList = {} + if tok:ConsumeKeyword('if', tokenList) then + --setup + local nodeIfStat = {} + nodeIfStat.AstType = 'IfStatement' + nodeIfStat.Clauses = {} + + --clauses + repeat + local st, nodeCond = ParseExpr(scope) + if not st then return false, nodeCond end + if not tok:ConsumeKeyword('then', tokenList) then + return false, GenerateError("`then` expected.") + end + local st, nodeBody = ParseStatementList(scope) + if not st then return false, nodeBody end + nodeIfStat.Clauses[#nodeIfStat.Clauses+1] = { + Condition = nodeCond; + Body = nodeBody; + } + until not tok:ConsumeKeyword('elseif', tokenList) + + --else clause + if tok:ConsumeKeyword('else', tokenList) then + local st, nodeBody = ParseStatementList(scope) + if not st then return false, nodeBody end + nodeIfStat.Clauses[#nodeIfStat.Clauses+1] = { + Body = nodeBody; + } + end + + --end + if not tok:ConsumeKeyword('end', tokenList) then + return false, GenerateError("`end` expected.") + end + + nodeIfStat.Tokens = tokenList + stat = nodeIfStat + + elseif tok:ConsumeKeyword('while', tokenList) then + --setup + local nodeWhileStat = {} + nodeWhileStat.AstType = 'WhileStatement' + + --condition + local st, nodeCond = ParseExpr(scope) + if not st then return false, nodeCond end + + --do + if not tok:ConsumeKeyword('do', tokenList) then + return false, GenerateError("`do` expected.") + end + + --body + local st, nodeBody = ParseStatementList(scope) + if not st then return false, nodeBody end + + --end + if not tok:ConsumeKeyword('end', tokenList) then + return false, GenerateError("`end` expected.") + end + + --return + nodeWhileStat.Condition = nodeCond + nodeWhileStat.Body = nodeBody + nodeWhileStat.Tokens = tokenList + stat = nodeWhileStat + + elseif tok:ConsumeKeyword('do', tokenList) then + --do block + local st, nodeBlock = ParseStatementList(scope) + if not st then return false, nodeBlock end + if not tok:ConsumeKeyword('end', tokenList) then + return false, GenerateError("`end` expected.") + end + + local nodeDoStat = {} + nodeDoStat.AstType = 'DoStatement' + nodeDoStat.Body = nodeBlock + nodeDoStat.Tokens = tokenList + stat = nodeDoStat + + elseif tok:ConsumeKeyword('for', tokenList) then + --for block + if not tok:Is('Ident') then + return false, GenerateError(" expected.") + end + local baseVarName = tok:Get(tokenList) + if tok:ConsumeSymbol('=', tokenList) then + --numeric for + local forScope = CreateScope(scope) + local forVar = forScope:CreateLocal(baseVarName.Data) + -- + local st, startEx = ParseExpr(scope) + if not st then return false, startEx end + if not tok:ConsumeSymbol(',', tokenList) then + return false, GenerateError("`,` Expected") + end + local st, endEx = ParseExpr(scope) + if not st then return false, endEx end + local st, stepEx; + if tok:ConsumeSymbol(',', tokenList) then + st, stepEx = ParseExpr(scope) + if not st then return false, stepEx end + end + if not tok:ConsumeKeyword('do', tokenList) then + return false, GenerateError("`do` expected") + end + -- + local st, body = ParseStatementList(forScope) + if not st then return false, body end + if not tok:ConsumeKeyword('end', tokenList) then + return false, GenerateError("`end` expected") + end + -- + local nodeFor = {} + nodeFor.AstType = 'NumericForStatement' + nodeFor.Scope = forScope + nodeFor.Variable = forVar + nodeFor.Start = startEx + nodeFor.End = endEx + nodeFor.Step = stepEx + nodeFor.Body = body + nodeFor.Tokens = tokenList + stat = nodeFor + else + --generic for + local forScope = CreateScope(scope) + -- + local varList = { forScope:CreateLocal(baseVarName.Data) } + while tok:ConsumeSymbol(',', tokenList) do + if not tok:Is('Ident') then + return false, GenerateError("for variable expected.") + end + varList[#varList+1] = forScope:CreateLocal(tok:Get(tokenList).Data) + end + if not tok:ConsumeKeyword('in', tokenList) then + return false, GenerateError("`in` expected.") + end + local generators = {} + local st, firstGenerator = ParseExpr(scope) + if not st then return false, firstGenerator end + generators[#generators+1] = firstGenerator + while tok:ConsumeSymbol(',', tokenList) do + local st, gen = ParseExpr(scope) + if not st then return false, gen end + generators[#generators+1] = gen + end + if not tok:ConsumeKeyword('do', tokenList) then + return false, GenerateError("`do` expected.") + end + local st, body = ParseStatementList(forScope) + if not st then return false, body end + if not tok:ConsumeKeyword('end', tokenList) then + return false, GenerateError("`end` expected.") + end + -- + local nodeFor = {} + nodeFor.AstType = 'GenericForStatement' + nodeFor.Scope = forScope + nodeFor.VariableList = varList + nodeFor.Generators = generators + nodeFor.Body = body + nodeFor.Tokens = tokenList + stat = nodeFor + end + + elseif tok:ConsumeKeyword('repeat', tokenList) then + local st, body = ParseStatementList(scope) + if not st then return false, body end + -- + if not tok:ConsumeKeyword('until', tokenList) then + return false, GenerateError("`until` expected.") + end + -- FIX: Used to parse in parent scope + -- Now parses in repeat scope + local st, cond = ParseExpr(body.Scope) + if not st then return false, cond end + -- + local nodeRepeat = {} + nodeRepeat.AstType = 'RepeatStatement' + nodeRepeat.Condition = cond + nodeRepeat.Body = body + nodeRepeat.Tokens = tokenList + stat = nodeRepeat + + -- CANDRAN : add decorator keyword (@) + elseif tok:ConsumeKeyword(candran.decorator, tokenList) then + if not tok:Is('Ident') then + return false, GenerateError("Decorator name expected") + end + + -- CANDRAN : get decorator name + local st, decorator = ParseExpr(scope) + if not st then return false, ex end + + -- CANDRAN : get decorated statement/decorator chain + local st, nodeStatement = ParseStatement(scope) + if not st then return false, nodeStatement end + + local nodeDecorator = {} + nodeDecorator.AstType = 'DecoratedStatement' + nodeDecorator.Decorator = decorator + nodeDecorator.Decorated = nodeStatement + nodeDecorator.Tokens = tokenList + stat = nodeDecorator + + elseif tok:ConsumeKeyword('function', tokenList) then + if not tok:Is('Ident') then + return false, GenerateError("Function name expected") + end + local st, name = ParseSuffixedExpr(scope, true) --true => only dots and colons + if not st then return false, name end + -- + local st, func = ParseFunctionArgsAndBody(scope, tokenList) + if not st then return false, func end + -- + func.IsLocal = false + func.Name = name + stat = func + + elseif tok:ConsumeKeyword('local', tokenList) then + if tok:Is('Ident') then + local varList = { tok:Get(tokenList).Data } + while tok:ConsumeSymbol(',', tokenList) do + if not tok:Is('Ident') then + return false, GenerateError("local var name expected") + end + varList[#varList+1] = tok:Get(tokenList).Data + end + + local initList = {} + if tok:ConsumeSymbol('=', tokenList) then + repeat + local st, ex = ParseExpr(scope) + if not st then return false, ex end + initList[#initList+1] = ex + until not tok:ConsumeSymbol(',', tokenList) + end + + --now patch var list + --we can't do this before getting the init list, because the init list does not + --have the locals themselves in scope. + for i, v in pairs(varList) do + varList[i] = scope:CreateLocal(v) + end + + local nodeLocal = {} + nodeLocal.AstType = 'LocalStatement' + nodeLocal.LocalList = varList + nodeLocal.InitList = initList + nodeLocal.Tokens = tokenList + -- + stat = nodeLocal + + elseif tok:ConsumeKeyword('function', tokenList) then + if not tok:Is('Ident') then + return false, GenerateError("Function name expected") + end + local name = tok:Get(tokenList).Data + local localVar = scope:CreateLocal(name) + -- + local st, func = ParseFunctionArgsAndBody(scope, tokenList) + if not st then return false, func end + -- + func.Name = localVar + func.IsLocal = true + stat = func + + else + return false, GenerateError("local var or function def expected") + end + + elseif tok:ConsumeSymbol('::', tokenList) then + if not tok:Is('Ident') then + return false, GenerateError('Label name expected') + end + local label = tok:Get(tokenList).Data + if not tok:ConsumeSymbol('::', tokenList) then + return false, GenerateError("`::` expected") + end + local nodeLabel = {} + nodeLabel.AstType = 'LabelStatement' + nodeLabel.Label = label + nodeLabel.Tokens = tokenList + stat = nodeLabel + + elseif tok:ConsumeKeyword('return', tokenList) then + local exList = {} + if not tok:IsKeyword('end') then + local st, firstEx = ParseExpr(scope) + if st then + exList[1] = firstEx + while tok:ConsumeSymbol(',', tokenList) do + local st, ex = ParseExpr(scope) + if not st then return false, ex end + exList[#exList+1] = ex + end + end + end + + local nodeReturn = {} + nodeReturn.AstType = 'ReturnStatement' + nodeReturn.Arguments = exList + nodeReturn.Tokens = tokenList + stat = nodeReturn + + elseif tok:ConsumeKeyword('break', tokenList) then + local nodeBreak = {} + nodeBreak.AstType = 'BreakStatement' + nodeBreak.Tokens = tokenList + stat = nodeBreak + + elseif tok:ConsumeKeyword('goto', tokenList) then + if not tok:Is('Ident') then + return false, GenerateError("Label expected") + end + local label = tok:Get(tokenList).Data + local nodeGoto = {} + nodeGoto.AstType = 'GotoStatement' + nodeGoto.Label = label + nodeGoto.Tokens = tokenList + stat = nodeGoto + + else + --statementParseExpr + local st, suffixed = ParseSuffixedExpr(scope) + if not st then return false, suffixed end + + --assignment or call? + -- CANDRAN : check if it is a Candran assignment symbol + local function isCandranAssignmentSymbol() + for _,s in ipairs(candran.assignment) do + if tok:IsSymbol(s) then + return true + end + end + return false + end + if tok:IsSymbol(',') or tok:IsSymbol('=') or isCandranAssignmentSymbol() then + --check that it was not parenthesized, making it not an lvalue + if (suffixed.ParenCount or 0) > 0 then + return false, GenerateError("Can not assign to parenthesized expression, is not an lvalue") + end + + --more processing needed + local lhs = { suffixed } + while tok:ConsumeSymbol(',', tokenList) do + local st, lhsPart = ParseSuffixedExpr(scope) + if not st then return false, lhsPart end + lhs[#lhs+1] = lhsPart + end + + --equals + -- CANDRAN : consume the Candran assignment symbol + local function consumeCandranAssignmentSymbol() + for _,s in ipairs(candran.assignment) do + if tok:ConsumeSymbol(s, tokenList) then + return true + end + end + return false + end + if not tok:ConsumeSymbol('=', tokenList) and not consumeCandranAssignmentSymbol() then + return false, GenerateError("`=` Expected.") + end + + --rhs + local rhs = {} + local st, firstRhs = ParseExpr(scope) + if not st then return false, firstRhs end + rhs[1] = firstRhs + while tok:ConsumeSymbol(',', tokenList) do + local st, rhsPart = ParseExpr(scope) + if not st then return false, rhsPart end + rhs[#rhs+1] = rhsPart + end + + --done + local nodeAssign = {} + nodeAssign.AstType = 'AssignmentStatement' + nodeAssign.Lhs = lhs + nodeAssign.Rhs = rhs + nodeAssign.Tokens = tokenList + stat = nodeAssign + + elseif suffixed.AstType == 'CallExpr' or + suffixed.AstType == 'TableCallExpr' or + suffixed.AstType == 'StringCallExpr' + then + --it's a call statement + local nodeCall = {} + nodeCall.AstType = 'CallStatement' + nodeCall.Expression = suffixed + nodeCall.Tokens = tokenList + stat = nodeCall + else + return false, GenerateError("Assignment Statement Expected") + end + end + + if tok:IsSymbol(';') then + stat.Semicolon = tok:Get( stat.Tokens ) + end + return true, stat + end + + + local statListCloseKeywords = lookupify{ 'end', 'else', 'elseif', 'until'} + + ParseStatementList = function(scope) + local nodeStatlist = {} + nodeStatlist.Scope = CreateScope(scope) + nodeStatlist.AstType = 'Statlist' + nodeStatlist.Body = { } + nodeStatlist.Tokens = { } + -- + --local stats = {} + -- + while not statListCloseKeywords[tok:Peek().Data] and not tok:IsEof() do + local st, nodeStatement = ParseStatement(nodeStatlist.Scope) + if not st then return false, nodeStatement end + --stats[#stats+1] = nodeStatement + nodeStatlist.Body[#nodeStatlist.Body + 1] = nodeStatement + end + + if tok:IsEof() then + local nodeEof = {} + nodeEof.AstType = 'Eof' + nodeEof.Tokens = { tok:Get() } + nodeStatlist.Body[#nodeStatlist.Body + 1] = nodeEof + end + + -- + --nodeStatlist.Body = stats + return true, nodeStatlist + end + + + local function mainfunc() + local topScope = CreateScope() + return ParseStatementList(topScope) + end + + local st, main = mainfunc() + --print("Last Token: "..PrintTable(tok:Peek())) + return st, main +end + +return { LexLua = LexLua, ParseLua = ParseLua } + +end +local ParseCandran = _() or ParseCandran +package.loaded["lib.LuaMinify.ParseCandran"] = ParseCandran or true +-- END OF IMPORT OF MODULE "lib.LuaMinify.ParseCandran" -- +-- IMPORT OF MODULE "lib.LuaMinify.FormatIdentityCandran" -- +local function _() +-- +-- CANDRAN +-- Based on the FormatIdentity.lua of LuaMinify. +-- Modified by Thomas99 to format valid Lua code from Candran AST. +-- +-- Modified parts are marked with "-- CANDRAN" comments. +-- + +--[[ +This file is part of LuaMinify by stravant (https://github.com/stravant/LuaMinify). + +LICENSE : +The MIT License (MIT) + +Copyright (c) 2012-2013 + +Permission is hereby granted, free of charge, to any person obtaining a copy of +this software and associated documentation files (the "Software"), to deal in +the Software without restriction, including without limitation the rights to +use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of +the Software, and to permit persons to whom the Software is furnished to do so, +subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS +FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR +COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER +IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN +CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. +]] + +--require'strict' -- CANDRAN : comment, useless here + +-- CANDRAN : add Candran syntaxic additions +local candran = require("candran").syntax + +require'lib.LuaMinify.ParseCandran' +local util = require'lib.LuaMinify.Util' + +local function debug_printf(...) + --[[ + util.printf(...) + --]] +end + +-- +-- FormatIdentity.lua +-- +-- Returns the exact source code that was used to create an AST, preserving all +-- comments and whitespace. +-- This can be used to get back a Lua source after renaming some variables in +-- an AST. +-- + +local function Format_Identity(ast) + local out = { + rope = {}, -- List of strings + line = 1, + char = 1, + + appendStr = function(self, str) + table.insert(self.rope, str) + + local lines = util.splitLines(str) + if #lines == 1 then + self.char = self.char + #str + else + self.line = self.line + #lines - 1 + local lastLine = lines[#lines] + self.char = #lastLine + end + end, + + -- CANDRAN : options + appendToken = function(self, token, options) + local options = options or {} -- CANDRAN + self:appendWhite(token, options) + --[*[ + --debug_printf("appendToken(%q)", token.Data) + local data = token.Data + local lines = util.splitLines(data) + while self.line + #lines < token.Line do + if not options.no_newline then self:appendStr('\n') end -- CANDRAN : options + self.line = self.line + 1 + self.char = 1 + end + --]] + if options.no_newline then data = data:gsub("[\n\r]*", "") end -- CANDRAN : options + if options.no_leading_white then data = data:gsub("^%s+", "") end + self:appendStr(data) + end, + + -- CANDRAN : options + appendTokens = function(self, tokens, options) + for _,token in ipairs(tokens) do + self:appendToken( token, options ) -- CANDRAN : options + end + end, + + -- CANDRAN : options + appendWhite = function(self, token, options) + if token.LeadingWhite then + self:appendTokens( token.LeadingWhite, options ) -- CANDRAN : options + --self.str = self.str .. ' ' + end + end + } + + local formatStatlist, formatExpr, formatStatement; + + -- CANDRAN : added options argument + -- CANDRAN : options = { no_newline = false, no_leading_white = false } + formatExpr = function(expr, options) + local options = options or {} -- CANDRAN + local tok_it = 1 + local function appendNextToken(str) + local tok = expr.Tokens[tok_it]; + if str and tok.Data ~= str then + error("Expected token '" .. str .. "'. Tokens: " .. util.PrintTable(expr.Tokens)) + end + out:appendToken( tok, options ) -- CANDRAN : options + tok_it = tok_it + 1 + options.no_leading_white = false -- CANDRAN : not the leading token anymore + end + local function appendToken(token) + out:appendToken( token, options ) -- CANDRAN : options + tok_it = tok_it + 1 + options.no_leading_white = false -- CANDRAN : not the leading token anymore + end + local function appendWhite() + local tok = expr.Tokens[tok_it]; + if not tok then error(util.PrintTable(expr)) end + out:appendWhite( tok, options ) -- CANDRAN : options + tok_it = tok_it + 1 + options.no_leading_white = false -- CANDRAN : not the leading token anymore + end + local function appendStr(str) + appendWhite() + out:appendStr(str) + end + local function peek() + if tok_it < #expr.Tokens then + return expr.Tokens[tok_it].Data + end + end + local function appendComma(mandatory, seperators) + if true then + seperators = seperators or { "," } + seperators = util.lookupify( seperators ) + if not mandatory and not seperators[peek()] then + return + end + assert(seperators[peek()], "Missing comma or semicolon") + appendNextToken() + else + local p = peek() + if p == "," or p == ";" then + appendNextToken() + end + end + end + + debug_printf("formatExpr(%s) at line %i", expr.AstType, expr.Tokens[1] and expr.Tokens[1].Line or -1) + + if expr.AstType == 'VarExpr' then + if expr.Variable then + appendStr( expr.Variable.Name ) + else + appendStr( expr.Name ) + end + + elseif expr.AstType == 'NumberExpr' then + appendToken( expr.Value ) + + elseif expr.AstType == 'StringExpr' then + appendToken( expr.Value ) + + elseif expr.AstType == 'BooleanExpr' then + appendNextToken( expr.Value and "true" or "false" ) + + elseif expr.AstType == 'NilExpr' then + appendNextToken( "nil" ) + + elseif expr.AstType == 'BinopExpr' then + formatExpr(expr.Lhs) + appendStr( expr.Op ) + formatExpr(expr.Rhs) + + elseif expr.AstType == 'UnopExpr' then + appendStr( expr.Op ) + formatExpr(expr.Rhs) + + elseif expr.AstType == 'DotsExpr' then + appendNextToken( "..." ) + + elseif expr.AstType == 'CallExpr' then + formatExpr(expr.Base) + appendNextToken( "(" ) + for i,arg in ipairs( expr.Arguments ) do + formatExpr(arg) + appendComma( i ~= #expr.Arguments ) + end + appendNextToken( ")" ) + + elseif expr.AstType == 'TableCallExpr' then + formatExpr( expr.Base ) + formatExpr( expr.Arguments[1] ) + + elseif expr.AstType == 'StringCallExpr' then + formatExpr(expr.Base) + appendToken( expr.Arguments[1] ) + + elseif expr.AstType == 'IndexExpr' then + formatExpr(expr.Base) + appendNextToken( "[" ) + formatExpr(expr.Index) + appendNextToken( "]" ) + + elseif expr.AstType == 'MemberExpr' then + formatExpr(expr.Base) + appendNextToken() -- . or : + appendToken(expr.Ident) + + elseif expr.AstType == 'Function' then + -- anonymous function + appendNextToken( "function" ) + appendNextToken( "(" ) + if #expr.Arguments > 0 then + for i = 1, #expr.Arguments do + appendStr( expr.Arguments[i].Name ) + if i ~= #expr.Arguments then + appendNextToken(",") + elseif expr.VarArg then + appendNextToken(",") + appendNextToken("...") + end + end + elseif expr.VarArg then + appendNextToken("...") + end + appendNextToken(")") + formatStatlist(expr.Body) + appendNextToken("end") + + elseif expr.AstType == 'ConstructorExpr' then + -- CANDRAN : function to get a value with its applied decorators + local function appendValue(entry) + out:appendStr(" ") + if entry.Decorated then + for _,d in ipairs(entry.DecoratorChain) do + formatExpr(d) + out:appendStr("(") + end + end + formatExpr(entry.Value, { no_leading_white = true }) + if entry.Decorated then + for _ in ipairs(entry.DecoratorChain) do + out:appendStr(")") + end + end + end + + appendNextToken( "{" ) + for i = 1, #expr.EntryList do + local entry = expr.EntryList[i] + if entry.Type == 'Key' then + appendNextToken( "[" ) + formatExpr(entry.Key) + appendNextToken( "]" ) + appendNextToken( "=" ) + appendValue(entry) -- CANDRAN : respect decorators + elseif entry.Type == 'Value' then + appendValue(entry) -- CANDRAN : respect decorators + elseif entry.Type == 'KeyString' then + appendStr(entry.Key) + appendNextToken( "=" ) + appendValue(entry) -- CANDRAN : respect decorators + end + appendComma( i ~= #expr.EntryList, { ",", ";" } ) + end + appendNextToken( "}" ) + + elseif expr.AstType == 'Parentheses' then + appendNextToken( "(" ) + formatExpr(expr.Inner) + appendNextToken( ")" ) + + else + print("Unknown AST Type: ", statement.AstType) + end + + assert(tok_it == #expr.Tokens + 1) + debug_printf("/formatExpr") + end + + formatStatement = function(statement) + local tok_it = 1 + local function appendNextToken(str) + local tok = statement.Tokens[tok_it]; + assert(tok, string.format("Not enough tokens for %q. First token at %i:%i", + str, statement.Tokens[1].Line, statement.Tokens[1].Char)) + assert(tok.Data == str, + string.format('Expected token %q, got %q', str, tok.Data)) + out:appendToken( tok ) + tok_it = tok_it + 1 + end + local function appendToken(token) + out:appendToken( str ) + tok_it = tok_it + 1 + end + local function appendWhite() + local tok = statement.Tokens[tok_it]; + out:appendWhite( tok ) + tok_it = tok_it + 1 + end + local function appendStr(str) + appendWhite() + out:appendStr(str) + end + local function appendComma(mandatory) + if mandatory + or (tok_it < #statement.Tokens and statement.Tokens[tok_it].Data == ",") then + appendNextToken( "," ) + end + end + + debug_printf("") + debug_printf(string.format("formatStatement(%s) at line %i", statement.AstType, statement.Tokens[1] and statement.Tokens[1].Line or -1)) + + if statement.AstType == 'AssignmentStatement' then + local newlineToCheck -- CANDRAN : position of a potential newline to eliminate in some edge cases + + for i,v in ipairs(statement.Lhs) do + formatExpr(v) + appendComma( i ~= #statement.Lhs ) + end + if #statement.Rhs > 0 then + -- CANDRAN : get the assignment operator used (default to =) + local assignmentToken = "=" + local candranAssignmentExists = util.lookupify(candran.assignment) + for i,v in pairs(statement.Tokens) do + if candranAssignmentExists[v.Data] then + assignmentToken = v.Data + break + end + end + appendNextToken(assignmentToken) -- CANDRAN : accept Candran assignments operators + --appendNextToken( "=" ) + newlineToCheck = #out.rope + 1 -- CANDRAN : the potential newline position afte the = + + if assignmentToken == "=" then + for i,v in ipairs(statement.Rhs) do + formatExpr(v) + appendComma( i ~= #statement.Rhs ) + end + else + out.rope[#out.rope] = "= " -- CANDRAN : remplace +=, -=, etc. with = + for i,v in ipairs(statement.Rhs) do + if i <= #statement.Lhs then -- CANDRAN : impossible to assign more variables than indicated in Lhs + formatExpr(statement.Lhs[i], { no_newline = true }) -- CANDRAN : write variable to assign + out:appendStr(" "..assignmentToken:gsub("=$","")) -- CANDRAN : assignment operation + formatExpr(v) -- CANDRAN : write variable to add/sub/etc. + if i ~= #statement.Rhs then -- CANDRAN : add comma to allow multi-assignment + appendComma( i ~= #statement.Rhs ) + if i >= #statement.Lhs then + out.rope[#out.rope] = "" -- CANDRAN : if this was the last element, remove the comma + end + end + end + end + end + end + + -- CANDRAN : eliminate the bad newlines + if out.rope[newlineToCheck] == "\n" then + out.rope[newlineToCheck] = "" + end + + elseif statement.AstType == 'CallStatement' then + formatExpr(statement.Expression) + + elseif statement.AstType == 'LocalStatement' then + appendNextToken( "local" ) + for i = 1, #statement.LocalList do + appendStr( statement.LocalList[i].Name ) + appendComma( i ~= #statement.LocalList ) + end + if #statement.InitList > 0 then + appendNextToken( "=" ) + for i = 1, #statement.InitList do + formatExpr(statement.InitList[i]) + appendComma( i ~= #statement.InitList ) + end + end + + elseif statement.AstType == 'IfStatement' then + appendNextToken( "if" ) + formatExpr( statement.Clauses[1].Condition ) + appendNextToken( "then" ) + formatStatlist( statement.Clauses[1].Body ) + for i = 2, #statement.Clauses do + local st = statement.Clauses[i] + if st.Condition then + appendNextToken( "elseif" ) + formatExpr(st.Condition) + appendNextToken( "then" ) + else + appendNextToken( "else" ) + end + formatStatlist(st.Body) + end + appendNextToken( "end" ) + + elseif statement.AstType == 'WhileStatement' then + appendNextToken( "while" ) + formatExpr(statement.Condition) + appendNextToken( "do" ) + formatStatlist(statement.Body) + appendNextToken( "end" ) + + elseif statement.AstType == 'DoStatement' then + appendNextToken( "do" ) + formatStatlist(statement.Body) + appendNextToken( "end" ) + + elseif statement.AstType == 'ReturnStatement' then + appendNextToken( "return" ) + for i = 1, #statement.Arguments do + formatExpr(statement.Arguments[i]) + appendComma( i ~= #statement.Arguments ) + end + + elseif statement.AstType == 'BreakStatement' then + appendNextToken( "break" ) + + elseif statement.AstType == 'RepeatStatement' then + appendNextToken( "repeat" ) + formatStatlist(statement.Body) + appendNextToken( "until" ) + formatExpr(statement.Condition) + + -- CANDRAN : add decorator support (@) + elseif statement.AstType == 'DecoratedStatement' then + -- CANDRAN : list of the chained decorators + local decoratorChain = { statement} + + -- CANDRAN : get the decorated statement + local decorated = statement.Decorated + while decorated.AstType == "DecoratedStatement" do + table.insert(decoratorChain, decorated) + decorated = decorated.Decorated + end + + -- CANDRAN : write the decorated statement like a normal statement + formatStatement(decorated) + + -- CANDRAN : mark the decorator token as used (and add whitespace) + appendNextToken(candran.decorator) + out.rope[#out.rope] = "" + + -- CANDRAN : get the variable(s) to decorate name(s) + local names = {} + if decorated.AstType == "Function" then + table.insert(names, decorated.Name.Name) + elseif decorated.AstType == "AssignmentStatement" then + for _,var in ipairs(decorated.Lhs) do + table.insert(names, var.Name) + end + elseif decorated.AstType == "LocalStatement" then + for _,var in ipairs(decorated.LocalList) do + table.insert(names, var.Name) + end + else + error("Invalid statement type to decorate : "..decorated.AstType) + end + + -- CANDRAN : redefine the variable(s) ( name, name2, ... = ... ) + for i,name in ipairs(names) do + out:appendStr(name) + if i ~= #names then out:appendStr(", ") end + end + out:appendStr(" = ") + + for i,name in ipairs(names) do + -- CANDRAN : write the decorator chain ( a(b(c(... ) + for _,v in pairs(decoratorChain) do + formatExpr(v.Decorator) + out:appendStr("(") + end + + -- CANDRAN : pass the undecorated variable name to the decorator chain + out:appendStr(name) + + -- CANDRAN : close parantheses + for _ in pairs(decoratorChain) do + out:appendStr(")") + end + + if i ~= #names then out:appendStr(", ") end + end + + elseif statement.AstType == 'Function' then + --print(util.PrintTable(statement)) + + if statement.IsLocal then + appendNextToken( "local" ) + end + appendNextToken( "function" ) + + if statement.IsLocal then + appendStr(statement.Name.Name) + else + formatExpr(statement.Name) + end + + appendNextToken( "(" ) + if #statement.Arguments > 0 then + for i = 1, #statement.Arguments do + appendStr( statement.Arguments[i].Name ) + appendComma( i ~= #statement.Arguments or statement.VarArg ) + if i == #statement.Arguments and statement.VarArg then + appendNextToken( "..." ) + end + end + elseif statement.VarArg then + appendNextToken( "..." ) + end + appendNextToken( ")" ) + + formatStatlist(statement.Body) + appendNextToken( "end" ) + + elseif statement.AstType == 'GenericForStatement' then + appendNextToken( "for" ) + for i = 1, #statement.VariableList do + appendStr( statement.VariableList[i].Name ) + appendComma( i ~= #statement.VariableList ) + end + appendNextToken( "in" ) + for i = 1, #statement.Generators do + formatExpr(statement.Generators[i]) + appendComma( i ~= #statement.Generators ) + end + appendNextToken( "do" ) + formatStatlist(statement.Body) + appendNextToken( "end" ) + + elseif statement.AstType == 'NumericForStatement' then + appendNextToken( "for" ) + appendStr( statement.Variable.Name ) + appendNextToken( "=" ) + formatExpr(statement.Start) + appendNextToken( "," ) + formatExpr(statement.End) + if statement.Step then + appendNextToken( "," ) + formatExpr(statement.Step) + end + appendNextToken( "do" ) + formatStatlist(statement.Body) + appendNextToken( "end" ) + + elseif statement.AstType == 'LabelStatement' then + appendNextToken( "::" ) + appendStr( statement.Label ) + appendNextToken( "::" ) + + elseif statement.AstType == 'GotoStatement' then + appendNextToken( "goto" ) + appendStr( statement.Label ) + + elseif statement.AstType == 'Eof' then + appendWhite() + + else + print("Unknown AST Type: ", statement.AstType) + end + + if statement.Semicolon then + appendNextToken(";") + end + + assert(tok_it == #statement.Tokens + 1) + debug_printf("/formatStatment") + end + + formatStatlist = function(statList) + for _, stat in ipairs(statList.Body) do + formatStatement(stat) + end + end + + formatStatlist(ast) + + return true, table.concat(out.rope) +end + +return Format_Identity + +end +local FormatIdentityCandran = _() or FormatIdentityCandran +package.loaded["lib.LuaMinify.FormatIdentityCandran"] = FormatIdentityCandran or true +-- END OF IMPORT OF MODULE "lib.LuaMinify.FormatIdentityCandran" -- + +-- Preprocessor +function candran.preprocess(input, args) + -- generate preprocessor + local preprocessor = "return function()\n" + + local lines = {} + for line in (input.."\n"):gmatch("(.-)\n") do + table.insert(lines, line) + -- preprocessor instructions (exclude shebang) + if line:match("^%s*#") and not (line:match("^#!") and #lines == 1) then + preprocessor = preprocessor .. line:gsub("^%s*#", "") .. "\n" + else + preprocessor = preprocessor .. "output ..= lines[" .. #lines .. "] .. \"\\n\"\n" + end + end + preprocessor = preprocessor .. "return output\nend" + + -- make preprocessor environement + local env = table.copy(_G) + env.candran = candran + env.output = "" + env.import = function(modpath, autoRequire) + local autoRequire = (autoRequire == nil) or autoRequire + + -- get module filepath + local filepath + for _,search in ipairs(package.searchers) do + local loader, path = search(modpath) + if type(loader) == "function" and type(path) == "string" then + filepath = path + break + end + end + if not filepath then error("No module named \""..modpath.."\"") end + + -- open module file + local f = io.open(filepath) + if not f then error("Can't open the module file to import") end + local modcontent = f:read("*a") + f:close() + + -- get module name (ex: module name of path.to.module is module) + local modname = modpath:match("[^%.]+$") + + -- + env.output = + -- + env.output .. + "-- IMPORT OF MODULE \""..modpath.."\" --\n".. + "local function _()\n".. + modcontent.."\n".. + "end\n".. + (autoRequire and "local "..modname.." = _() or "..modname.."\n" or "").. -- auto require + "package.loaded[\""..modpath.."\"] = "..(autoRequire and modname or "_()").." or true\n".. -- add to package.loaded + "-- END OF IMPORT OF MODULE \""..modpath.."\" --\n" + end + env.include = function(file) + local f = io.open(file) + if not f then error("Can't open the file to include") end + env.output = env.output .. f:read("*a").."\n" + f:close() + end + env.print = function(...) + env.output = env.output .. table.concat({ ...}, "\t") .. "\n" + end + env.args = args or {} + env.lines = lines + + -- load preprocessor + local preprocess, err = load(candran.compile(preprocessor), "Preprocessor", nil, env) + if not preprocess then error("Error while creating preprocessor :\n" .. err) end + + -- execute preprocessor + local success, output = pcall(preprocess()) + if not success then error("Error while preprocessing file :\n" .. output .. "\nWith preprocessor : \n" .. preprocessor) end + + return output +end + +-- Compiler +function candran.compile(input) + local parse = require("lib.LuaMinify.ParseCandran") + local format = require("lib.LuaMinify.FormatIdentityCandran") + + local success, ast = parse.ParseLua(input) + if not success then error("Error while parsing the file :\n"..tostring(ast)) end + + local success, output = format(ast) + if not success then error("Error while formating the file :\n"..tostring(output)) end + + return output +end + +-- Preprocess & compile +function candran.make(code, args) + local preprocessed = candran.preprocess(code, args or {}) + local output = candran.compile(preprocessed) + + return output +end + +-- Standalone mode +if debug.getinfo(3) == nil and arg then + -- Check args + if #arg < 1 then + print("Candran version "..candran.VERSION.." by Thomas99") + print("Command-line usage :") + print("lua candran.lua [preprocessor arguments]") + return candran + end + + -- Parse args + local inputFilepath = arg[1] + local args = {} + for i=2, #arg, 1 do + if arg[i]:sub(1,2) == "--" then + args[arg[i]:sub(3)] = arg[i+1] + i = i + 1 -- skip argument value + end + end + + -- Open & read input file + local inputFile, err = io.open(inputFilepath, "r") + if not inputFile then error("Error while opening input file : "..err) end + local input = inputFile:read("*a") + inputFile:close() + + -- Make + print(candran.make(input, args)) +end + +return candran + diff --git a/build/lune.lua b/build/lune.lua deleted file mode 100644 index 68517bd..0000000 --- a/build/lune.lua +++ /dev/null @@ -1,784 +0,0 @@ -#!/usr/bin/lua ---[[ -Lune language & compiler by Thomas99. - -LICENSE : -Copyright (c) 2014 Thomas99 - -This software is provided 'as-is', without any express or implied warranty. -In no event will the authors be held liable for any damages arising from the -use of this software. - -Permission is granted to anyone to use this software for any purpose, including -commercial applications, and to alter it and redistribute it freely, subject -to the following restrictions: - - 1. The origin of this software must not be misrepresented; you must not - claim that you wrote the original software. If you use this software in a - product, an acknowledgment in the product documentation would be appreciated - but is not required. - - 2. Altered source versions must be plainly marked as such, and must not be - misrepresented as being the original software. - - 3. This notice may not be removed or altered from any source distribution. -]] --- INCLUSION OF FILE "lib/lexer.lua" -- -local function _() ---[[ -This file is a part of Penlight (set of pure Lua libraries) - https://github.com/stevedonovan/Penlight - -LICENSE : -Copyright (C) 2009 Steve Donovan, David Manura. - -Permission is hereby granted, free of charge, to any person obtaining a copy of -this software and associated documentation files (the "Software"), to deal in the -Software without restriction, including without limitation the rights to use, copy, -modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, -and to permit persons to whom the Software is furnished to do so, subject to the -following conditions: - -The above copyright notice and this permission notice shall be included in all copies -or substantial portions of the Software. - -THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, -INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A -PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT -HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF -CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE -OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. -]] - ---- Lexical scanner for creating a sequence of tokens from text. --- `lexer.scan(s)` returns an iterator over all tokens found in the --- string `s`. This iterator returns two values, a token type string --- (such as 'string' for quoted string, 'iden' for identifier) and the value of the --- token. --- --- Versions specialized for Lua and C are available; these also handle block comments --- and classify keywords as 'keyword' tokens. For example: --- --- > s = 'for i=1,n do' --- > for t,v in lexer.lua(s) do print(t,v) end --- keyword for --- iden i --- = = --- number 1 --- , , --- iden n --- keyword do --- --- See the Guide for further @{06-data.md.Lexical_Scanning|discussion} --- @module pl.lexer - -local yield,wrap = coroutine.yield,coroutine.wrap -local strfind = string.find -local strsub = string.sub -local append = table.insert - -local function assert_arg(idx,val,tp) - if type(val) ~= tp then - error("argument "..idx.." must be "..tp, 2) - end -end - -local lexer = {} - -local NUMBER1 = '^[%+%-]?%d+%.?%d*[eE][%+%-]?%d+' -local NUMBER2 = '^[%+%-]?%d+%.?%d*' -local NUMBER3 = '^0x[%da-fA-F]+' -local NUMBER4 = '^%d+%.?%d*[eE][%+%-]?%d+' -local NUMBER5 = '^%d+%.?%d*' -local IDEN = '^[%a_][%w_]*' -local WSPACE = '^%s+' -local STRING0 = [[^(['\"]).-\\%1]] -local STRING1 = [[^(['\"]).-[^\]%1]] -local STRING3 = "^((['\"])%2)" -- empty string -local PREPRO = '^#.-[^\\]\n' - -local plain_matches,lua_matches,cpp_matches,lua_keyword,cpp_keyword - -local function tdump(tok) - return yield(tok,tok) -end - -local function ndump(tok,options) - if options and options.number then - tok = tonumber(tok) - end - return yield("number",tok) -end - --- regular strings, single or double quotes; usually we want them --- without the quotes -local function sdump(tok,options) - if options and options.string then - tok = tok:sub(2,-2) - end - return yield("string",tok) -end - --- long Lua strings need extra work to get rid of the quotes -local function sdump_l(tok,options) - if options and options.string then - tok = tok:sub(3,-3) - end - return yield("string",tok) -end - -local function chdump(tok,options) - if options and options.string then - tok = tok:sub(2,-2) - end - return yield("char",tok) -end - -local function cdump(tok) - return yield('comment',tok) -end - -local function wsdump (tok) - return yield("space",tok) -end - -local function pdump (tok) - return yield('prepro',tok) -end - -local function plain_vdump(tok) - return yield("iden",tok) -end - -local function lua_vdump(tok) - if lua_keyword[tok] then - return yield("keyword",tok) - else - return yield("iden",tok) - end -end - -local function cpp_vdump(tok) - if cpp_keyword[tok] then - return yield("keyword",tok) - else - return yield("iden",tok) - end -end - ---- create a plain token iterator from a string or file-like object. --- @param s the string --- @param matches an optional match table (set of pattern-action pairs) --- @param filter a table of token types to exclude, by default {space=true} --- @param options a table of options; by default, {number=true,string=true}, --- which means convert numbers and strip string quotes. -function lexer.scan (s,matches,filter,options) - --assert_arg(1,s,'string') - local file = type(s) ~= 'string' and s - filter = filter or {space=true} - options = options or {number=true,string=true} - if filter then - if filter.space then filter[wsdump] = true end - if filter.comments then - filter[cdump] = true - end - end - if not matches then - if not plain_matches then - plain_matches = { - {WSPACE,wsdump}, - {NUMBER3,ndump}, - {IDEN,plain_vdump}, - {NUMBER1,ndump}, - {NUMBER2,ndump}, - {STRING3,sdump}, - {STRING0,sdump}, - {STRING1,sdump}, - {'^.',tdump} - } - end - matches = plain_matches - end - local function lex () - local i1,i2,idx,res1,res2,tok,pat,fun,capt - local line = 1 - if file then s = file:read()..'\n' end - local sz = #s - local idx = 1 - --print('sz',sz) - while true do - for _,m in ipairs(matches) do - pat = m[1] - fun = m[2] - i1,i2 = strfind(s,pat,idx) - if i1 then - tok = strsub(s,i1,i2) - idx = i2 + 1 - if not (filter and filter[fun]) then - lexer.finished = idx > sz - res1,res2 = fun(tok,options) - end - if res1 then - local tp = type(res1) - -- insert a token list - if tp=='table' then - yield('','') - for _,t in ipairs(res1) do - yield(t[1],t[2]) - end - elseif tp == 'string' then -- or search up to some special pattern - i1,i2 = strfind(s,res1,idx) - if i1 then - tok = strsub(s,i1,i2) - idx = i2 + 1 - yield('',tok) - else - yield('','') - idx = sz + 1 - end - --if idx > sz then return end - else - yield(line,idx) - end - end - if idx > sz then - if file then - --repeat -- next non-empty line - line = line + 1 - s = file:read() - if not s then return end - --until not s:match '^%s*$' - s = s .. '\n' - idx ,sz = 1,#s - break - else - return - end - else break end - end - end - end - end - return wrap(lex) -end - -local function isstring (s) - return type(s) == 'string' -end - ---- insert tokens into a stream. --- @param tok a token stream --- @param a1 a string is the type, a table is a token list and --- a function is assumed to be a token-like iterator (returns type & value) --- @param a2 a string is the value -function lexer.insert (tok,a1,a2) - if not a1 then return end - local ts - if isstring(a1) and isstring(a2) then - ts = {{a1,a2}} - elseif type(a1) == 'function' then - ts = {} - for t,v in a1() do - append(ts,{t,v}) - end - else - ts = a1 - end - tok(ts) -end - ---- get everything in a stream upto a newline. --- @param tok a token stream --- @return a string -function lexer.getline (tok) - local t,v = tok('.-\n') - return v -end - ---- get current line number.
--- Only available if the input source is a file-like object. --- @param tok a token stream --- @return the line number and current column -function lexer.lineno (tok) - return tok(0) -end - ---- get the rest of the stream. --- @param tok a token stream --- @return a string -function lexer.getrest (tok) - local t,v = tok('.+') - return v -end - ---- get the Lua keywords as a set-like table. --- So res["and"] etc would be true. --- @return a table -function lexer.get_keywords () - if not lua_keyword then - lua_keyword = { - ["and"] = true, ["break"] = true, ["do"] = true, - ["else"] = true, ["elseif"] = true, ["end"] = true, - ["false"] = true, ["for"] = true, ["function"] = true, - ["if"] = true, ["in"] = true, ["local"] = true, ["nil"] = true, - ["not"] = true, ["or"] = true, ["repeat"] = true, - ["return"] = true, ["then"] = true, ["true"] = true, - ["until"] = true, ["while"] = true - } - end - return lua_keyword -end - - ---- create a Lua token iterator from a string or file-like object. --- Will return the token type and value. --- @param s the string --- @param filter a table of token types to exclude, by default {space=true,comments=true} --- @param options a table of options; by default, {number=true,string=true}, --- which means convert numbers and strip string quotes. -function lexer.lua(s,filter,options) - filter = filter or {space=true,comments=true} - lexer.get_keywords() - if not lua_matches then - lua_matches = { - {WSPACE,wsdump}, - {NUMBER3,ndump}, - {IDEN,lua_vdump}, - {NUMBER4,ndump}, - {NUMBER5,ndump}, - {STRING3,sdump}, - {STRING0,sdump}, - {STRING1,sdump}, - {'^%-%-%[%[.-%]%]',cdump}, - {'^%-%-.-\n',cdump}, - {'^%[%[.-%]%]',sdump_l}, - {'^==',tdump}, - {'^~=',tdump}, - {'^<=',tdump}, - {'^>=',tdump}, - {'^%.%.%.',tdump}, - {'^%.%.',tdump}, - {'^.',tdump} - } - end - return lexer.scan(s,lua_matches,filter,options) -end - ---- create a C/C++ token iterator from a string or file-like object. --- Will return the token type type and value. --- @param s the string --- @param filter a table of token types to exclude, by default {space=true,comments=true} --- @param options a table of options; by default, {number=true,string=true}, --- which means convert numbers and strip string quotes. -function lexer.cpp(s,filter,options) - filter = filter or {comments=true} - if not cpp_keyword then - cpp_keyword = { - ["class"] = true, ["break"] = true, ["do"] = true, ["sizeof"] = true, - ["else"] = true, ["continue"] = true, ["struct"] = true, - ["false"] = true, ["for"] = true, ["public"] = true, ["void"] = true, - ["private"] = true, ["protected"] = true, ["goto"] = true, - ["if"] = true, ["static"] = true, ["const"] = true, ["typedef"] = true, - ["enum"] = true, ["char"] = true, ["int"] = true, ["bool"] = true, - ["long"] = true, ["float"] = true, ["true"] = true, ["delete"] = true, - ["double"] = true, ["while"] = true, ["new"] = true, - ["namespace"] = true, ["try"] = true, ["catch"] = true, - ["switch"] = true, ["case"] = true, ["extern"] = true, - ["return"] = true,["default"] = true,['unsigned'] = true,['signed'] = true, - ["union"] = true, ["volatile"] = true, ["register"] = true,["short"] = true, - } - end - if not cpp_matches then - cpp_matches = { - {WSPACE,wsdump}, - {PREPRO,pdump}, - {NUMBER3,ndump}, - {IDEN,cpp_vdump}, - {NUMBER4,ndump}, - {NUMBER5,ndump}, - {STRING3,sdump}, - {STRING1,chdump}, - {'^//.-\n',cdump}, - {'^/%*.-%*/',cdump}, - {'^==',tdump}, - {'^!=',tdump}, - {'^<=',tdump}, - {'^>=',tdump}, - {'^->',tdump}, - {'^&&',tdump}, - {'^||',tdump}, - {'^%+%+',tdump}, - {'^%-%-',tdump}, - {'^%+=',tdump}, - {'^%-=',tdump}, - {'^%*=',tdump}, - {'^/=',tdump}, - {'^|=',tdump}, - {'^%^=',tdump}, - {'^::',tdump}, - {'^.',tdump} - } - end - return lexer.scan(s,cpp_matches,filter,options) -end - ---- get a list of parameters separated by a delimiter from a stream. --- @param tok the token stream --- @param endtoken end of list (default ')'). Can be '\n' --- @param delim separator (default ',') --- @return a list of token lists. -function lexer.get_separated_list(tok,endtoken,delim) - endtoken = endtoken or ')' - delim = delim or ',' - local parm_values = {} - local level = 1 -- used to count ( and ) - local tl = {} - local function tappend (tl,t,val) - val = val or t - append(tl,{t,val}) - end - local is_end - if endtoken == '\n' then - is_end = function(t,val) - return t == 'space' and val:find '\n' - end - else - is_end = function (t) - return t == endtoken - end - end - local token,value - while true do - token,value=tok() - if not token then return nil,'EOS' end -- end of stream is an error! - if is_end(token,value) and level == 1 then - append(parm_values,tl) - break - elseif token == '(' then - level = level + 1 - tappend(tl,'(') - elseif token == ')' then - level = level - 1 - if level == 0 then -- finished with parm list - append(parm_values,tl) - break - else - tappend(tl,')') - end - elseif token == delim and level == 1 then - append(parm_values,tl) -- a new parm - tl = {} - else - tappend(tl,token,value) - end - end - return parm_values,{token,value} -end - ---- get the next non-space token from the stream. --- @param tok the token stream. -function lexer.skipws (tok) - local t,v = tok() - while t == 'space' do - t,v = tok() - end - return t,v -end - -local skipws = lexer.skipws - ---- get the next token, which must be of the expected type. --- Throws an error if this type does not match! --- @param tok the token stream --- @param expected_type the token type --- @param no_skip_ws whether we should skip whitespace -function lexer.expecting (tok,expected_type,no_skip_ws) - assert_arg(1,tok,'function') - assert_arg(2,expected_type,'string') - local t,v - if no_skip_ws then - t,v = tok() - else - t,v = skipws(tok) - end - if t ~= expected_type then error ("expecting "..expected_type,2) end - return v -end - -return lexer - -end -local lexer = _() or lexer --- END OF INCLUDSION OF FILE "lib/lexer.lua" -- --- INCLUSION OF FILE "lib/table.lua" -- -local function _() ---[[ -Lua table utilities by Thomas99. - -LICENSE : -Copyright (c) 2014 Thomas99 - -This software is provided 'as-is', without any express or implied warranty. -In no event will the authors be held liable for any damages arising from the -use of this software. - -Permission is granted to anyone to use this software for any purpose, including -commercial applications, and to alter it and redistribute it freely, subject -to the following restrictions: - - 1. The origin of this software must not be misrepresented; you must not - claim that you wrote the original software. If you use this software in a - product, an acknowledgment in the product documentation would be appreciated - but is not required. - - 2. Altered source versions must be plainly marked as such, and must not be - misrepresented as being the original software. - - 3. This notice may not be removed or altered from any source distribution. -]] - --- Copie récursivement la table t dans la table dest (ou une table vide si non précisé) et la retourne --- replace (false) : indique si oui ou non, les clefs existant déjà dans dest doivent être écrasées par celles de t --- metatable (true) : copier ou non également les metatables --- filter (function) : filtre, si retourne true copie l'objet, sinon ne le copie pas --- Note : les metatables des objets ne sont jamais re-copiées (mais référence à la place), car sinon lors de la copie --- la classe de ces objets changera pour une nouvelle classe, et c'est pas pratique :p -function table.copy(t, dest, replace, metatable, filter, copied) - local copied = copied or {} - local replace = replace or false - local metatable = (metatable==nil or metatable) and true - local filter = filter or function(name, source, destination) return true end - - if type(t) ~= "table" then - return t - elseif copied[t] then -- si la table a déjà été copiée - return copied[t] - end - - local dest = dest or {} -- la copie - - copied[t] = dest -- on marque la table comme copiée - - for k, v in pairs(t) do - if filter(k, t, dest) then - if replace then - dest[k] = table.copy(v, dest[k], replace, metatable, filter, copied) - else - if dest[k] == nil or type(v) == "table" then -- si la clef n'existe pas déjà dans dest ou si c'est une table à copier - dest[k] = table.copy(v, dest[k], replace, metatable, filter, copied) - end - end - end - end - - -- copie des metatables - if metatable then - if t.__classe then - setmetatable(dest, getmetatable(t)) - else - setmetatable(dest, table.copy(getmetatable(t), getmetatable(dest), replace, filter)) - end - end - - return dest -end - --- retourne true si value est dans la table -function table.isIn(table, value) - for _,v in pairs(table) do - if v == value then - return true - end - end - return false -end - --- retourne true si la clé key est dans la table -function table.hasKey(table, key) - for k,_ in pairs(table) do - if k == key then - return true - end - end - return false -end - --- retourne la longueur exacte d'une table (fonctionne sur les tables à clef) -function table.len(t) - local len=0 - for i in pairs(t) do - len=len+1 - end - return len -end - --- Sépare str en éléments séparés par le pattern et retourne une table -function string.split(str, pattern) - local t = {} - local pos = 0 - - for i,p in string.gmatch(str, "(.-)"..pattern.."()") do - table.insert(t, i) - pos = p - end - - table.insert(t, str:sub(pos)) - - return t -end -end -local table = _() or table --- END OF INCLUDSION OF FILE "lib/table.lua" -- - -local lune = {} -lune.VERSION = "0.0.1" -lune.syntax = { - affectation = { ["+"] = "= %s +", ["-"] = "= %s -", ["*"] = "= %s *", ["/"] = "= %s /", - ["^"] = "= %s ^", ["%"] = "= %s %%", [".."] = "= %s .." }, - incrementation = { ["+"] = " = %s + 1" , ["-"] = " = %s - 1" }, -} - --- Preprocessor -function lune.preprocess(input, args) - -- generate preprocessor - local preprocessor = "return function()\n" - - local lines = {} - for line in (input.."\n"):gmatch("(.-)\n") do - table.insert(lines, line) - if line:sub(1,1) == "#" then - -- exclude shebang - if not (line:sub(1,2) == "#!" and #lines ==1) then - preprocessor = preprocessor .. line:sub(2) .. "\n" - else - preprocessor = preprocessor .. "output ..= lines[" .. #lines .. "] .. \"\\n\"\n" - end - else - preprocessor = preprocessor .. "output ..= lines[" .. #lines .. "] .. \"\\n\"\n" - end - end - preprocessor = preprocessor .. "return output\nend" - - -- make preprocessor environement - local env = table.copy(_G) - env.lune = lune - env.output = "" - env.include = function(file) - local f = io.open(file) - if not f then error("can't open the file to include") end - - local filename = file:match("([^%/%\\]-)%.[^%.]-$") - - env.output = env.output .. - "-- INCLUSION OF FILE \""..file.."\" --\n".. - "local function _()\n".. - f:read("*a").."\n".. - "end\n".. - "local "..filename.." = _() or "..filename.."\n".. - "-- END OF INCLUDSION OF FILE \""..file.."\" --\n" - - f:close() - end - env.rawInclude = function(file) - local f = io.open(file) - if not f then error("can't open the file to raw include") end - env.output = env.output .. f:read("*a").."\n" - f:close() - end - env.print = function(...) - env.output = env.output .. table.concat({...}, "\t") .. "\n" - end - env.args = args or {} - env.lines = lines - - -- load preprocessor - local preprocess, err = load(lune.compile(preprocessor), "Preprocessor", nil, env) - if not preprocess then error("Error while creating preprocessor :\n" .. err) end - - -- execute preprocessor - local success, output = pcall(preprocess()) - if not success then error("Error while preprocessing file :\n" .. output .. "\nWith preprocessor : \n" .. preprocessor) end - - return output -end - --- Compiler -function lune.compile(input) - local output = "" - - local last = {} - for t,v in lexer.lua(input, {}, {}) do - local toInsert = v - - -- affectation - if t == "=" then - if table.hasKey(lune.syntax.affectation, last.token) then - toInsert = string.format(lune.syntax.affectation[last.token], last.varName) - output = output:sub(1, -1 -#last.token) -- remove token before = - end - end - - -- self-incrementation - if table.hasKey(lune.syntax.incrementation, t) and t == last.token then - toInsert = string.format(lune.syntax.incrementation[last.token], last.varName) - output = output:sub(1, -#last.token*2) -- remove token ++/-- - end - - -- reconstitude full variable name (ex : ith.game.camera) - if t == "iden" then - if last.token == "." then - last.varName = last.varName .. "." .. v - else - last.varName = v - end - end - - last[t] = v - last.token = t - last.value = v - - output = output .. toInsert - end - - return output -end - --- Preprocess & compile -function lune.make(code, args) - local preprocessed = lune.preprocess(code, args or {}) - local output = lune.compile(preprocessed) - return output -end - --- Standalone mode -if debug.getinfo(3) == nil and arg then - -- Check args - if #arg < 1 then - print("Lune version "..lune.VERSION.." by Thomas99") - print("Command-line usage :") - print("lua lune.lua [preprocessor arguments]") - return lune - end - - -- Parse args - local inputFilePath = arg[1] - local args = {} - -- Parse compilation args - for i=2, #arg, 1 do - if arg[i]:sub(1,2) == "--" then - args[arg[i]:sub(3)] = arg[i+1] - i = i +1 -- skip argument value - end - end - - -- Open & read input file - local inputFile, err = io.open(inputFilePath, "r") - if not inputFile then error("Error while opening input file : "..err) end - local input = inputFile:read("*a") - inputFile:close() - - -- End - print(lune.make(input, args)) -end - -return lune - diff --git a/candran.can b/candran.can new file mode 100644 index 0000000..972e7b7 --- /dev/null +++ b/candran.can @@ -0,0 +1,170 @@ +--[[ +Candran language, preprocessor and compiler by Thomas99. + +LICENSE : +Copyright (c) 2015 Thomas99 + +This software is provided 'as-is', without any express or implied warranty. +In no event will the authors be held liable for any damages arising from the +use of this software. + +Permission is granted to anyone to use this software for any purpose, including +commercial applications, and to alter it and redistribute it freely, subject +to the following restrictions: + + 1. The origin of this software must not be misrepresented; you must not + claim that you wrote the original software. If you use this software in a + product, an acknowledgment in the product documentation would be appreciated + but is not required. + + 2. Altered source versions must be plainly marked as such, and must not be + misrepresented as being the original software. + + 3. This notice may not be removed or altered from any source distribution. +]] + +local candran = { + VERSION = "0.1.0", + syntax = { + assignment = { "+=", "-=", "*=", "/=", "^=", "%=", "..=" }, + decorator = "@" + } +} +package.loaded["candran"] = candran + +#import("lib.table") +#import("lib.LuaMinify.Util") +#import("lib.LuaMinify.Scope") +#import("lib.LuaMinify.ParseCandran") +#import("lib.LuaMinify.FormatIdentityCandran") + +-- Preprocessor +function candran.preprocess(input, args) + -- generate preprocessor + local preprocessor = "return function()\n" + + local lines = {} + for line in (input.."\n"):gmatch("(.-)\n") do + table.insert(lines, line) + -- preprocessor instructions (exclude shebang) + if line:match("^%s*#") and not (line:match("^#!") and #lines == 1) then + preprocessor ..= line:gsub("^%s*#", "") .. "\n" + else + preprocessor ..= "output ..= lines[" .. #lines .. "] .. \"\\n\"\n" + end + end + preprocessor ..= "return output\nend" + + -- make preprocessor environement + local env = table.copy(_G) + env.candran = candran + env.output = "" + env.import = function(modpath, autoRequire) + local autoRequire = (autoRequire == nil) or autoRequire + + -- get module filepath + local filepath + for _,search in ipairs(package.searchers) do + local loader, path = search(modpath) + if type(loader) == "function" and type(path) == "string" then + filepath = path + break + end + end + if not filepath then error("No module named \""..modpath.."\"") end + + -- open module file + local f = io.open(filepath) + if not f then error("Can't open the module file to import") end + local modcontent = f:read("*a") + f:close() + + -- get module name (ex: module name of path.to.module is module) + local modname = modpath:match("[^%.]+$") + + -- + env.output ..= + "-- IMPORT OF MODULE \""..modpath.."\" --\n".. + "local function _()\n".. + modcontent.."\n".. + "end\n".. + (autoRequire and "local "..modname.." = _() or "..modname.."\n" or "").. -- auto require + "package.loaded[\""..modpath.."\"] = "..(autoRequire and modname or "_()").." or true\n".. -- add to package.loaded + "-- END OF IMPORT OF MODULE \""..modpath.."\" --\n" + end + env.include = function(file) + local f = io.open(file) + if not f then error("Can't open the file to include") end + env.output ..= f:read("*a").."\n" + f:close() + end + env.print = function(...) + env.output ..= table.concat({...}, "\t") .. "\n" + end + env.args = args or {} + env.lines = lines + + -- load preprocessor + local preprocess, err = load(candran.compile(preprocessor), "Preprocessor", nil, env) + if not preprocess then error("Error while creating preprocessor :\n" .. err) end + + -- execute preprocessor + local success, output = pcall(preprocess()) + if not success then error("Error while preprocessing file :\n" .. output .. "\nWith preprocessor : \n" .. preprocessor) end + + return output +end + +-- Compiler +function candran.compile(input) + local parse = require("lib.LuaMinify.ParseCandran") + local format = require("lib.LuaMinify.FormatIdentityCandran") + + local success, ast = parse.ParseLua(input) + if not success then error("Error while parsing the file :\n"..tostring(ast)) end + + local success, output = format(ast) + if not success then error("Error while formating the file :\n"..tostring(output)) end + + return output +end + +-- Preprocess & compile +function candran.make(code, args) + local preprocessed = candran.preprocess(code, args or {}) + local output = candran.compile(preprocessed) + + return output +end + +-- Standalone mode +if debug.getinfo(3) == nil and arg then + -- Check args + if #arg < 1 then + print("Candran version "..candran.VERSION.." by Thomas99") + print("Command-line usage :") + print("lua candran.lua [preprocessor arguments]") + return candran + end + + -- Parse args + local inputFilepath = arg[1] + local args = {} + for i=2, #arg, 1 do + if arg[i]:sub(1,2) == "--" then + args[arg[i]:sub(3)] = arg[i+1] + i = i + 1 -- skip argument value + end + end + + -- Open & read input file + local inputFile, err = io.open(inputFilepath, "r") + if not inputFile then error("Error while opening input file : "..err) end + local input = inputFile:read("*a") + inputFile:close() + + -- Make + print(candran.make(input, args)) +end + +return candran \ No newline at end of file diff --git a/lib/LuaMinify/CommandLineBeautify.lua b/lib/LuaMinify/CommandLineBeautify.lua new file mode 100644 index 0000000..12b0ced --- /dev/null +++ b/lib/LuaMinify/CommandLineBeautify.lua @@ -0,0 +1,121 @@ +-- +-- beautify +-- +-- A command line utility for beautifying lua source code using the beautifier. +-- + +local util = require'Util' +local Parser = require'ParseLua' +local Format_Beautify = require'FormatBeautiful' +local ParseLua = Parser.ParseLua +local PrintTable = util.PrintTable + +local function splitFilename(name) + --table.foreach(arg, print) + if name:find(".") then + local p, ext = name:match("()%.([^%.]*)$") + if p and ext then + if #ext == 0 then + return name, nil + else + local filename = name:sub(1,p-1) + return filename, ext + end + else + return name, nil + end + else + return name, nil + end +end + +if #arg == 1 then + local name, ext = splitFilename(arg[1]) + local outname = name.."_formatted" + if ext then outname = outname.."."..ext end + -- + local inf = io.open(arg[1], 'r') + if not inf then + print("Failed to open '"..arg[1].."' for reading") + return + end + -- + local sourceText = inf:read('*all') + inf:close() + -- + local st, ast = ParseLua(sourceText) + if not st then + --we failed to parse the file, show why + print(ast) + return + end + -- + local outf = io.open(outname, 'w') + if not outf then + print("Failed to open '"..outname.."' for writing") + return + end + -- + outf:write(Format_Beautify(ast)) + outf:close() + -- + print("Beautification complete") + +elseif #arg == 2 then + --keep the user from accidentally overwriting their non-minified file with + if arg[1]:find("_formatted") then + print("Did you mix up the argument order?\n".. + "Current command will beautify '"..arg[1].."' and overwrite '"..arg[2].."' with the results") + while true do + io.write("Confirm (yes/no): ") + local msg = io.read('*line') + if msg == 'yes' then + break + elseif msg == 'no' then + return + end + end + end + local inf = io.open(arg[1], 'r') + if not inf then + print("Failed to open '"..arg[1].."' for reading") + return + end + -- + local sourceText = inf:read('*all') + inf:close() + -- + local st, ast = ParseLua(sourceText) + if not st then + --we failed to parse the file, show why + print(ast) + return + end + -- + if arg[1] == arg[2] then + print("Are you SURE you want to overwrite the source file with a beautified version?\n".. + "You will be UNABLE to get the original source back!") + while true do + io.write("Confirm (yes/no): ") + local msg = io.read('*line') + if msg == 'yes' then + break + elseif msg == 'no' then + return + end + end + end + local outf = io.open(arg[2], 'w') + if not outf then + print("Failed to open '"..arg[2].."' for writing") + return + end + -- + outf:write(Format_Beautify(ast)) + outf:close() + -- + print("Beautification complete") + +else + print("Invalid arguments!\nUsage: lua CommandLineLuaBeautify.lua source_file [destination_file]") +end diff --git a/lib/LuaMinify/CommandLineLiveBeautify.lua b/lib/LuaMinify/CommandLineLiveBeautify.lua new file mode 100644 index 0000000..b40a7b1 --- /dev/null +++ b/lib/LuaMinify/CommandLineLiveBeautify.lua @@ -0,0 +1,47 @@ + +-- +-- beautify.interactive +-- +-- For testing: Lets you enter lines of text to be beautified to verify the +-- correctness of their implementation. +-- + +local util = require'Util' +local Parser = require'ParseLua' +local Format_Beautify = require'FormatBeautiful' +local ParseLua = Parser.ParseLua +local PrintTable = util.PrintTable + +while true do + io.write('> ') + local line = io.read('*line') + local fileFrom, fileTo = line:match("^file (.*) (.*)") + if fileFrom and fileTo then + local file = io.open(fileFrom, 'r') + local fileTo = io.open(fileTo, 'w') + if file and fileTo then + local st, ast = ParseLua(file:read('*all')) + if st then + fileTo:write(Format_Beautify(ast)..'\n') + io.write("Beautification Complete\n") + else + io.write(""..tostring(ast).."\n") + end + file:close() + fileTo:close() + else + io.write("File does not exist\n") + end + else + local st, ast = ParseLua(line) + if st then + io.write("====== AST =======\n") + io.write(PrintTable(ast)..'\n') + io.write("==== BEAUTIFIED ====\n") + io.write(Format_Beautify(ast)) + io.write("==================\n") + else + io.write(""..tostring(ast).."\n") + end + end +end diff --git a/lib/LuaMinify/CommandLineLiveMinify.lua b/lib/LuaMinify/CommandLineLiveMinify.lua new file mode 100644 index 0000000..1eb4c24 --- /dev/null +++ b/lib/LuaMinify/CommandLineLiveMinify.lua @@ -0,0 +1,47 @@ + +-- +-- CommandLineLiveMinify.lua +-- +-- For testing: Lets you enter lines of text to be minified to verify the +-- correctness of their implementation. +-- + +local util = require'Util' +local Parser = require'ParseLua' +local Format_Mini = require'FormatMini' +local ParseLua = Parser.ParseLua +local PrintTable = util.PrintTable + +while true do + io.write('> ') + local line = io.read('*line') + local fileFrom, fileTo = line:match("^file (.*) (.*)") + if fileFrom and fileTo then + local file = io.open(fileFrom, 'r') + local fileTo = io.open(fileTo, 'w') + if file and fileTo then + local st, ast = ParseLua(file:read('*all')) + if st then + fileTo:write(Format_Mini(ast)..'\n') + io.write("Minification Complete\n") + else + io.write(""..tostring(ast).."\n") + end + file:close() + fileTo:close() + else + io.write("File does not exist\n") + end + else + local st, ast = ParseLua(line) + if st then + io.write("====== AST =======\n") + io.write(PrintTable(ast)..'\n') + io.write("==== MINIFIED ====\n") + io.write(Format_Mini(ast)..'\n') + io.write("==================\n") + else + io.write(""..tostring(ast).."\n") + end + end +end diff --git a/lib/LuaMinify/CommandLineMinify.lua b/lib/LuaMinify/CommandLineMinify.lua new file mode 100644 index 0000000..c239195 --- /dev/null +++ b/lib/LuaMinify/CommandLineMinify.lua @@ -0,0 +1,122 @@ + +-- +-- CommandlineMinify.lua +-- +-- A command line utility for minifying lua source code using the minifier. +-- + +local util = require'Util' +local Parser = require'ParseLua' +local Format_Mini = require'FormatMini' +local ParseLua = Parser.ParseLua +local PrintTable = util.PrintTable + +local function splitFilename(name) + table.foreach(arg, print) + if name:find(".") then + local p, ext = name:match("()%.([^%.]*)$") + if p and ext then + if #ext == 0 then + return name, nil + else + local filename = name:sub(1,p-1) + return filename, ext + end + else + return name, nil + end + else + return name, nil + end +end + +if #arg == 1 then + local name, ext = splitFilename(arg[1]) + local outname = name.."_min" + if ext then outname = outname.."."..ext end + -- + local inf = io.open(arg[1], 'r') + if not inf then + print("Failed to open `"..arg[1].."` for reading") + return + end + -- + local sourceText = inf:read('*all') + inf:close() + -- + local st, ast = ParseLua(sourceText) + if not st then + --we failed to parse the file, show why + print(ast) + return + end + -- + local outf = io.open(outname, 'w') + if not outf then + print("Failed to open `"..outname.."` for writing") + return + end + -- + outf:write(Format_Mini(ast)) + outf:close() + -- + print("Minification complete") + +elseif #arg == 2 then + --keep the user from accidentally overwriting their non-minified file with + if arg[1]:find("_min") then + print("Did you mix up the argument order?\n".. + "Current command will minify `"..arg[1].."` and OVERWRITE `"..arg[2].."` with the results") + while true do + io.write("Confirm (yes/cancel): ") + local msg = io.read('*line') + if msg == 'yes' then + break + elseif msg == 'cancel' then + return + end + end + end + local inf = io.open(arg[1], 'r') + if not inf then + print("Failed to open `"..arg[1].."` for reading") + return + end + -- + local sourceText = inf:read('*all') + inf:close() + -- + local st, ast = ParseLua(sourceText) + if not st then + --we failed to parse the file, show why + print(ast) + return + end + -- + if arg[1] == arg[2] then + print("Are you SURE you want to overwrite the source file with a minified version?\n".. + "You will be UNABLE to get the original source back!") + while true do + io.write("Confirm (yes/cancel): ") + local msg = io.read('*line') + if msg == 'yes' then + break + elseif msg == 'cancel' then + return + end + end + end + local outf = io.open(arg[2], 'w') + if not outf then + print("Failed to open `"..arg[2].."` for writing") + return + end + -- + outf:write(Format_Mini(ast)) + outf:close() + -- + print("Minification complete") + +else + print("Invalid arguments, Usage:\nLuaMinify source [destination]") +end diff --git a/lib/LuaMinify/FormatBeautiful.lua b/lib/LuaMinify/FormatBeautiful.lua new file mode 100644 index 0000000..1262202 --- /dev/null +++ b/lib/LuaMinify/FormatBeautiful.lua @@ -0,0 +1,347 @@ +-- +-- Beautifier +-- +-- Returns a beautified version of the code, including comments +-- + +local parser = require"ParseLua" +local ParseLua = parser.ParseLua +local util = require'Util' +local lookupify = util.lookupify + +local LowerChars = lookupify{'a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', + 'j', 'k', 'l', 'm', 'n', 'o', 'p', 'q', 'r', + 's', 't', 'u', 'v', 'w', 'x', 'y', 'z'} +local UpperChars = lookupify{'A', 'B', 'C', 'D', 'E', 'F', 'G', 'H', 'I', + 'J', 'K', 'L', 'M', 'N', 'O', 'P', 'Q', 'R', + 'S', 'T', 'U', 'V', 'W', 'X', 'Y', 'Z'} +local Digits = lookupify{'0', '1', '2', '3', '4', '5', '6', '7', '8', '9'} + +local function Format_Beautify(ast) + local formatStatlist, formatExpr + local indent = 0 + local EOL = "\n" + + local function getIndentation() + return string.rep(" ", indent) + end + + local function joinStatementsSafe(a, b, sep) + sep = sep or '' + local aa, bb = a:sub(-1,-1), b:sub(1,1) + if UpperChars[aa] or LowerChars[aa] or aa == '_' then + if not (UpperChars[bb] or LowerChars[bb] or bb == '_' or Digits[bb]) then + --bb is a symbol, can join without sep + return a .. b + elseif bb == '(' then + --prevent ambiguous syntax + return a..sep..b + else + return a..sep..b + end + elseif Digits[aa] then + if bb == '(' then + --can join statements directly + return a..b + else + return a..sep..b + end + elseif aa == '' then + return a..b + else + if bb == '(' then + --don't want to accidentally call last statement, can't join directly + return a..sep..b + else + return a..b + end + end + end + + formatExpr = function(expr) + local out = string.rep('(', expr.ParenCount or 0) + if expr.AstType == 'VarExpr' then + if expr.Variable then + out = out .. expr.Variable.Name + else + out = out .. expr.Name + end + + elseif expr.AstType == 'NumberExpr' then + out = out..expr.Value.Data + + elseif expr.AstType == 'StringExpr' then + out = out..expr.Value.Data + + elseif expr.AstType == 'BooleanExpr' then + out = out..tostring(expr.Value) + + elseif expr.AstType == 'NilExpr' then + out = joinStatementsSafe(out, "nil") + + elseif expr.AstType == 'BinopExpr' then + out = joinStatementsSafe(out, formatExpr(expr.Lhs)) .. " " + out = joinStatementsSafe(out, expr.Op) .. " " + out = joinStatementsSafe(out, formatExpr(expr.Rhs)) + + elseif expr.AstType == 'UnopExpr' then + out = joinStatementsSafe(out, expr.Op) .. (#expr.Op ~= 1 and " " or "") + out = joinStatementsSafe(out, formatExpr(expr.Rhs)) + + elseif expr.AstType == 'DotsExpr' then + out = out.."..." + + elseif expr.AstType == 'CallExpr' then + out = out..formatExpr(expr.Base) + out = out.."(" + for i = 1, #expr.Arguments do + out = out..formatExpr(expr.Arguments[i]) + if i ~= #expr.Arguments then + out = out..", " + end + end + out = out..")" + + elseif expr.AstType == 'TableCallExpr' then + out = out..formatExpr(expr.Base) .. " " + out = out..formatExpr(expr.Arguments[1]) + + elseif expr.AstType == 'StringCallExpr' then + out = out..formatExpr(expr.Base) .. " " + out = out..expr.Arguments[1].Data + + elseif expr.AstType == 'IndexExpr' then + out = out..formatExpr(expr.Base).."["..formatExpr(expr.Index).."]" + + elseif expr.AstType == 'MemberExpr' then + out = out..formatExpr(expr.Base)..expr.Indexer..expr.Ident.Data + + elseif expr.AstType == 'Function' then + -- anonymous function + out = out.."function(" + if #expr.Arguments > 0 then + for i = 1, #expr.Arguments do + out = out..expr.Arguments[i].Name + if i ~= #expr.Arguments then + out = out..", " + elseif expr.VarArg then + out = out..", ..." + end + end + elseif expr.VarArg then + out = out.."..." + end + out = out..")" .. EOL + indent = indent + 1 + out = joinStatementsSafe(out, formatStatlist(expr.Body)) + indent = indent - 1 + out = joinStatementsSafe(out, getIndentation() .. "end") + elseif expr.AstType == 'ConstructorExpr' then + out = out.."{ " + for i = 1, #expr.EntryList do + local entry = expr.EntryList[i] + if entry.Type == 'Key' then + out = out.."["..formatExpr(entry.Key).."] = "..formatExpr(entry.Value) + elseif entry.Type == 'Value' then + out = out..formatExpr(entry.Value) + elseif entry.Type == 'KeyString' then + out = out..entry.Key.." = "..formatExpr(entry.Value) + end + if i ~= #expr.EntryList then + out = out..", " + end + end + out = out.." }" + + elseif expr.AstType == 'Parentheses' then + out = out.."("..formatExpr(expr.Inner)..")" + + end + out = out..string.rep(')', expr.ParenCount or 0) + return out + end + + local formatStatement = function(statement) + local out = "" + if statement.AstType == 'AssignmentStatement' then + out = getIndentation() + for i = 1, #statement.Lhs do + out = out..formatExpr(statement.Lhs[i]) + if i ~= #statement.Lhs then + out = out..", " + end + end + if #statement.Rhs > 0 then + out = out.." = " + for i = 1, #statement.Rhs do + out = out..formatExpr(statement.Rhs[i]) + if i ~= #statement.Rhs then + out = out..", " + end + end + end + elseif statement.AstType == 'CallStatement' then + out = getIndentation() .. formatExpr(statement.Expression) + elseif statement.AstType == 'LocalStatement' then + out = getIndentation() .. out.."local " + for i = 1, #statement.LocalList do + out = out..statement.LocalList[i].Name + if i ~= #statement.LocalList then + out = out..", " + end + end + if #statement.InitList > 0 then + out = out.." = " + for i = 1, #statement.InitList do + out = out..formatExpr(statement.InitList[i]) + if i ~= #statement.InitList then + out = out..", " + end + end + end + elseif statement.AstType == 'IfStatement' then + out = getIndentation() .. joinStatementsSafe("if ", formatExpr(statement.Clauses[1].Condition)) + out = joinStatementsSafe(out, " then") .. EOL + indent = indent + 1 + out = joinStatementsSafe(out, formatStatlist(statement.Clauses[1].Body)) + indent = indent - 1 + for i = 2, #statement.Clauses do + local st = statement.Clauses[i] + if st.Condition then + out = getIndentation() .. joinStatementsSafe(out, getIndentation() .. "elseif ") + out = joinStatementsSafe(out, formatExpr(st.Condition)) + out = joinStatementsSafe(out, " then") .. EOL + else + out = joinStatementsSafe(out, getIndentation() .. "else") .. EOL + end + indent = indent + 1 + out = joinStatementsSafe(out, formatStatlist(st.Body)) + indent = indent - 1 + end + out = joinStatementsSafe(out, getIndentation() .. "end") .. EOL + elseif statement.AstType == 'WhileStatement' then + out = getIndentation() .. joinStatementsSafe("while ", formatExpr(statement.Condition)) + out = joinStatementsSafe(out, " do") .. EOL + indent = indent + 1 + out = joinStatementsSafe(out, formatStatlist(statement.Body)) + indent = indent - 1 + out = joinStatementsSafe(out, getIndentation() .. "end") .. EOL + elseif statement.AstType == 'DoStatement' then + out = getIndentation() .. joinStatementsSafe(out, "do") .. EOL + indent = indent + 1 + out = joinStatementsSafe(out, formatStatlist(statement.Body)) + indent = indent - 1 + out = joinStatementsSafe(out, getIndentation() .. "end") .. EOL + elseif statement.AstType == 'ReturnStatement' then + out = getIndentation() .. "return " + for i = 1, #statement.Arguments do + out = joinStatementsSafe(out, formatExpr(statement.Arguments[i])) + if i ~= #statement.Arguments then + out = out..", " + end + end + elseif statement.AstType == 'BreakStatement' then + out = getIndentation() .. "break" + elseif statement.AstType == 'RepeatStatement' then + out = getIndentation() .. "repeat" .. EOL + indent = indent + 1 + out = joinStatementsSafe(out, formatStatlist(statement.Body)) + indent = indent - 1 + out = joinStatementsSafe(out, getIndentation() .. "until ") + out = joinStatementsSafe(out, formatExpr(statement.Condition)) .. EOL + elseif statement.AstType == 'Function' then + if statement.IsLocal then + out = "local " + end + out = joinStatementsSafe(out, "function ") + out = getIndentation() .. out + if statement.IsLocal then + out = out..statement.Name.Name + else + out = out..formatExpr(statement.Name) + end + out = out.."(" + if #statement.Arguments > 0 then + for i = 1, #statement.Arguments do + out = out..statement.Arguments[i].Name + if i ~= #statement.Arguments then + out = out..", " + elseif statement.VarArg then + out = out..",..." + end + end + elseif statement.VarArg then + out = out.."..." + end + out = out..")" .. EOL + indent = indent + 1 + out = joinStatementsSafe(out, formatStatlist(statement.Body)) + indent = indent - 1 + out = joinStatementsSafe(out, getIndentation() .. "end") .. EOL + elseif statement.AstType == 'GenericForStatement' then + out = getIndentation() .. "for " + for i = 1, #statement.VariableList do + out = out..statement.VariableList[i].Name + if i ~= #statement.VariableList then + out = out..", " + end + end + out = out.." in " + for i = 1, #statement.Generators do + out = joinStatementsSafe(out, formatExpr(statement.Generators[i])) + if i ~= #statement.Generators then + out = joinStatementsSafe(out, ', ') + end + end + out = joinStatementsSafe(out, " do") .. EOL + indent = indent + 1 + out = joinStatementsSafe(out, formatStatlist(statement.Body)) + indent = indent - 1 + out = joinStatementsSafe(out, getIndentation() .. "end") .. EOL + elseif statement.AstType == 'NumericForStatement' then + out = getIndentation() .. "for " + out = out..statement.Variable.Name.." = " + out = out..formatExpr(statement.Start)..", "..formatExpr(statement.End) + if statement.Step then + out = out..", "..formatExpr(statement.Step) + end + out = joinStatementsSafe(out, " do") .. EOL + indent = indent + 1 + out = joinStatementsSafe(out, formatStatlist(statement.Body)) + indent = indent - 1 + out = joinStatementsSafe(out, getIndentation() .. "end") .. EOL + elseif statement.AstType == 'LabelStatement' then + out = getIndentation() .. "::" .. statement.Label .. "::" .. EOL + elseif statement.AstType == 'GotoStatement' then + out = getIndentation() .. "goto " .. statement.Label .. EOL + elseif statement.AstType == 'Comment' then + if statement.CommentType == 'Shebang' then + out = getIndentation() .. statement.Data + --out = out .. EOL + elseif statement.CommentType == 'Comment' then + out = getIndentation() .. statement.Data + --out = out .. EOL + elseif statement.CommentType == 'LongComment' then + out = getIndentation() .. statement.Data + --out = out .. EOL + end + elseif statement.AstType == 'Eof' then + -- Ignore + else + print("Unknown AST Type: ", statement.AstType) + end + return out + end + + formatStatlist = function(statList) + local out = '' + for _, stat in pairs(statList.Body) do + out = joinStatementsSafe(out, formatStatement(stat) .. EOL) + end + return out + end + + return formatStatlist(ast) +end + +return Format_Beautify diff --git a/lib/LuaMinify/FormatIdentity.lua b/lib/LuaMinify/FormatIdentity.lua new file mode 100644 index 0000000..8cbb59c --- /dev/null +++ b/lib/LuaMinify/FormatIdentity.lua @@ -0,0 +1,440 @@ +require'strict' +require'ParseLua' +local util = require'Util' + +local function debug_printf(...) + --[[ + util.printf(...) + --]] +end + +-- +-- FormatIdentity.lua +-- +-- Returns the exact source code that was used to create an AST, preserving all +-- comments and whitespace. +-- This can be used to get back a Lua source after renaming some variables in +-- an AST. +-- + +local function Format_Identity(ast) + local out = { + rope = {}, -- List of strings + line = 1, + char = 1, + + appendStr = function(self, str) + table.insert(self.rope, str) + + local lines = util.splitLines(str) + if #lines == 1 then + self.char = self.char + #str + else + self.line = self.line + #lines - 1 + local lastLine = lines[#lines] + self.char = #lastLine + end + end, + + appendToken = function(self, token) + self:appendWhite(token) + --[*[ + --debug_printf("appendToken(%q)", token.Data) + local data = token.Data + local lines = util.splitLines(data) + while self.line + #lines < token.Line do + print("Inserting extra line") + self.str = self.str .. '\n' + self.line = self.line + 1 + self.char = 1 + end + --]] + self:appendStr(token.Data) + end, + + appendTokens = function(self, tokens) + for _,token in ipairs(tokens) do + self:appendToken( token ) + end + end, + + appendWhite = function(self, token) + if token.LeadingWhite then + self:appendTokens( token.LeadingWhite ) + --self.str = self.str .. ' ' + end + end + } + + local formatStatlist, formatExpr; + + formatExpr = function(expr) + local tok_it = 1 + local function appendNextToken(str) + local tok = expr.Tokens[tok_it]; + if str and tok.Data ~= str then + error("Expected token '" .. str .. "'. Tokens: " .. util.PrintTable(expr.Tokens)) + end + out:appendToken( tok ) + tok_it = tok_it + 1 + end + local function appendToken(token) + out:appendToken( token ) + tok_it = tok_it + 1 + end + local function appendWhite() + local tok = expr.Tokens[tok_it]; + if not tok then error(util.PrintTable(expr)) end + out:appendWhite( tok ) + tok_it = tok_it + 1 + end + local function appendStr(str) + appendWhite() + out:appendStr(str) + end + local function peek() + if tok_it < #expr.Tokens then + return expr.Tokens[tok_it].Data + end + end + local function appendComma(mandatory, seperators) + if true then + seperators = seperators or { "," } + seperators = util.lookupify( seperators ) + if not mandatory and not seperators[peek()] then + return + end + assert(seperators[peek()], "Missing comma or semicolon") + appendNextToken() + else + local p = peek() + if p == "," or p == ";" then + appendNextToken() + end + end + end + + debug_printf("formatExpr(%s) at line %i", expr.AstType, expr.Tokens[1] and expr.Tokens[1].Line or -1) + + if expr.AstType == 'VarExpr' then + if expr.Variable then + appendStr( expr.Variable.Name ) + else + appendStr( expr.Name ) + end + + elseif expr.AstType == 'NumberExpr' then + appendToken( expr.Value ) + + elseif expr.AstType == 'StringExpr' then + appendToken( expr.Value ) + + elseif expr.AstType == 'BooleanExpr' then + appendNextToken( expr.Value and "true" or "false" ) + + elseif expr.AstType == 'NilExpr' then + appendNextToken( "nil" ) + + elseif expr.AstType == 'BinopExpr' then + formatExpr(expr.Lhs) + appendStr( expr.Op ) + formatExpr(expr.Rhs) + + elseif expr.AstType == 'UnopExpr' then + appendStr( expr.Op ) + formatExpr(expr.Rhs) + + elseif expr.AstType == 'DotsExpr' then + appendNextToken( "..." ) + + elseif expr.AstType == 'CallExpr' then + formatExpr(expr.Base) + appendNextToken( "(" ) + for i,arg in ipairs( expr.Arguments ) do + formatExpr(arg) + appendComma( i ~= #expr.Arguments ) + end + appendNextToken( ")" ) + + elseif expr.AstType == 'TableCallExpr' then + formatExpr( expr.Base ) + formatExpr( expr.Arguments[1] ) + + elseif expr.AstType == 'StringCallExpr' then + formatExpr(expr.Base) + appendToken( expr.Arguments[1] ) + + elseif expr.AstType == 'IndexExpr' then + formatExpr(expr.Base) + appendNextToken( "[" ) + formatExpr(expr.Index) + appendNextToken( "]" ) + + elseif expr.AstType == 'MemberExpr' then + formatExpr(expr.Base) + appendNextToken() -- . or : + appendToken(expr.Ident) + + elseif expr.AstType == 'Function' then + -- anonymous function + appendNextToken( "function" ) + appendNextToken( "(" ) + if #expr.Arguments > 0 then + for i = 1, #expr.Arguments do + appendStr( expr.Arguments[i].Name ) + if i ~= #expr.Arguments then + appendNextToken(",") + elseif expr.VarArg then + appendNextToken(",") + appendNextToken("...") + end + end + elseif expr.VarArg then + appendNextToken("...") + end + appendNextToken(")") + formatStatlist(expr.Body) + appendNextToken("end") + + elseif expr.AstType == 'ConstructorExpr' then + appendNextToken( "{" ) + for i = 1, #expr.EntryList do + local entry = expr.EntryList[i] + if entry.Type == 'Key' then + appendNextToken( "[" ) + formatExpr(entry.Key) + appendNextToken( "]" ) + appendNextToken( "=" ) + formatExpr(entry.Value) + elseif entry.Type == 'Value' then + formatExpr(entry.Value) + elseif entry.Type == 'KeyString' then + appendStr(entry.Key) + appendNextToken( "=" ) + formatExpr(entry.Value) + end + appendComma( i ~= #expr.EntryList, { ",", ";" } ) + end + appendNextToken( "}" ) + + elseif expr.AstType == 'Parentheses' then + appendNextToken( "(" ) + formatExpr(expr.Inner) + appendNextToken( ")" ) + + else + print("Unknown AST Type: ", statement.AstType) + end + + assert(tok_it == #expr.Tokens + 1) + debug_printf("/formatExpr") + end + + + local formatStatement = function(statement) + local tok_it = 1 + local function appendNextToken(str) + local tok = statement.Tokens[tok_it]; + assert(tok, string.format("Not enough tokens for %q. First token at %i:%i", + str, statement.Tokens[1].Line, statement.Tokens[1].Char)) + assert(tok.Data == str, + string.format('Expected token %q, got %q', str, tok.Data)) + out:appendToken( tok ) + tok_it = tok_it + 1 + end + local function appendToken(token) + out:appendToken( str ) + tok_it = tok_it + 1 + end + local function appendWhite() + local tok = statement.Tokens[tok_it]; + out:appendWhite( tok ) + tok_it = tok_it + 1 + end + local function appendStr(str) + appendWhite() + out:appendStr(str) + end + local function appendComma(mandatory) + if mandatory + or (tok_it < #statement.Tokens and statement.Tokens[tok_it].Data == ",") then + appendNextToken( "," ) + end + end + + debug_printf("") + debug_printf(string.format("formatStatement(%s) at line %i", statement.AstType, statement.Tokens[1] and statement.Tokens[1].Line or -1)) + + if statement.AstType == 'AssignmentStatement' then + for i,v in ipairs(statement.Lhs) do + formatExpr(v) + appendComma( i ~= #statement.Lhs ) + end + if #statement.Rhs > 0 then + appendNextToken( "=" ) + for i,v in ipairs(statement.Rhs) do + formatExpr(v) + appendComma( i ~= #statement.Rhs ) + end + end + + elseif statement.AstType == 'CallStatement' then + formatExpr(statement.Expression) + + elseif statement.AstType == 'LocalStatement' then + appendNextToken( "local" ) + for i = 1, #statement.LocalList do + appendStr( statement.LocalList[i].Name ) + appendComma( i ~= #statement.LocalList ) + end + if #statement.InitList > 0 then + appendNextToken( "=" ) + for i = 1, #statement.InitList do + formatExpr(statement.InitList[i]) + appendComma( i ~= #statement.InitList ) + end + end + + elseif statement.AstType == 'IfStatement' then + appendNextToken( "if" ) + formatExpr( statement.Clauses[1].Condition ) + appendNextToken( "then" ) + formatStatlist( statement.Clauses[1].Body ) + for i = 2, #statement.Clauses do + local st = statement.Clauses[i] + if st.Condition then + appendNextToken( "elseif" ) + formatExpr(st.Condition) + appendNextToken( "then" ) + else + appendNextToken( "else" ) + end + formatStatlist(st.Body) + end + appendNextToken( "end" ) + + elseif statement.AstType == 'WhileStatement' then + appendNextToken( "while" ) + formatExpr(statement.Condition) + appendNextToken( "do" ) + formatStatlist(statement.Body) + appendNextToken( "end" ) + + elseif statement.AstType == 'DoStatement' then + appendNextToken( "do" ) + formatStatlist(statement.Body) + appendNextToken( "end" ) + + elseif statement.AstType == 'ReturnStatement' then + appendNextToken( "return" ) + for i = 1, #statement.Arguments do + formatExpr(statement.Arguments[i]) + appendComma( i ~= #statement.Arguments ) + end + + elseif statement.AstType == 'BreakStatement' then + appendNextToken( "break" ) + + elseif statement.AstType == 'RepeatStatement' then + appendNextToken( "repeat" ) + formatStatlist(statement.Body) + appendNextToken( "until" ) + formatExpr(statement.Condition) + + elseif statement.AstType == 'Function' then + --print(util.PrintTable(statement)) + + if statement.IsLocal then + appendNextToken( "local" ) + end + appendNextToken( "function" ) + + if statement.IsLocal then + appendStr(statement.Name.Name) + else + formatExpr(statement.Name) + end + + appendNextToken( "(" ) + if #statement.Arguments > 0 then + for i = 1, #statement.Arguments do + appendStr( statement.Arguments[i].Name ) + appendComma( i ~= #statement.Arguments or statement.VarArg ) + if i == #statement.Arguments and statement.VarArg then + appendNextToken( "..." ) + end + end + elseif statement.VarArg then + appendNextToken( "..." ) + end + appendNextToken( ")" ) + + formatStatlist(statement.Body) + appendNextToken( "end" ) + + elseif statement.AstType == 'GenericForStatement' then + appendNextToken( "for" ) + for i = 1, #statement.VariableList do + appendStr( statement.VariableList[i].Name ) + appendComma( i ~= #statement.VariableList ) + end + appendNextToken( "in" ) + for i = 1, #statement.Generators do + formatExpr(statement.Generators[i]) + appendComma( i ~= #statement.Generators ) + end + appendNextToken( "do" ) + formatStatlist(statement.Body) + appendNextToken( "end" ) + + elseif statement.AstType == 'NumericForStatement' then + appendNextToken( "for" ) + appendStr( statement.Variable.Name ) + appendNextToken( "=" ) + formatExpr(statement.Start) + appendNextToken( "," ) + formatExpr(statement.End) + if statement.Step then + appendNextToken( "," ) + formatExpr(statement.Step) + end + appendNextToken( "do" ) + formatStatlist(statement.Body) + appendNextToken( "end" ) + + elseif statement.AstType == 'LabelStatement' then + appendNextToken( "::" ) + appendStr( statement.Label ) + appendNextToken( "::" ) + + elseif statement.AstType == 'GotoStatement' then + appendNextToken( "goto" ) + appendStr( statement.Label ) + + elseif statement.AstType == 'Eof' then + appendWhite() + + else + print("Unknown AST Type: ", statement.AstType) + end + + if statement.Semicolon then + appendNextToken(";") + end + + assert(tok_it == #statement.Tokens + 1) + debug_printf("/formatStatment") + end + + formatStatlist = function(statList) + for _, stat in ipairs(statList.Body) do + formatStatement(stat) + end + end + + formatStatlist(ast) + + return true, table.concat(out.rope) +end + +return Format_Identity diff --git a/lib/LuaMinify/FormatIdentityCandran.lua b/lib/LuaMinify/FormatIdentityCandran.lua new file mode 100644 index 0000000..b32808a --- /dev/null +++ b/lib/LuaMinify/FormatIdentityCandran.lua @@ -0,0 +1,601 @@ +-- +-- CANDRAN +-- Based on the FormatIdentity.lua of LuaMinify. +-- Modified by Thomas99 to format valid Lua code from Candran AST. +-- +-- Modified parts are marked with "-- CANDRAN" comments. +-- + +--[[ +This file is part of LuaMinify by stravant (https://github.com/stravant/LuaMinify). + +LICENSE : +The MIT License (MIT) + +Copyright (c) 2012-2013 + +Permission is hereby granted, free of charge, to any person obtaining a copy of +this software and associated documentation files (the "Software"), to deal in +the Software without restriction, including without limitation the rights to +use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of +the Software, and to permit persons to whom the Software is furnished to do so, +subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS +FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR +COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER +IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN +CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. +]] + +--require'strict' -- CANDRAN : comment, useless here + +-- CANDRAN : add Candran syntaxic additions +local candran = require("candran").syntax + +require'lib.LuaMinify.ParseCandran' +local util = require'lib.LuaMinify.Util' + +local function debug_printf(...) + --[[ + util.printf(...) + --]] +end + +-- +-- FormatIdentity.lua +-- +-- Returns the exact source code that was used to create an AST, preserving all +-- comments and whitespace. +-- This can be used to get back a Lua source after renaming some variables in +-- an AST. +-- + +local function Format_Identity(ast) + local out = { + rope = {}, -- List of strings + line = 1, + char = 1, + + appendStr = function(self, str) + table.insert(self.rope, str) + + local lines = util.splitLines(str) + if #lines == 1 then + self.char = self.char + #str + else + self.line = self.line + #lines - 1 + local lastLine = lines[#lines] + self.char = #lastLine + end + end, + + -- CANDRAN : options + appendToken = function(self, token, options) + local options = options or {} -- CANDRAN + self:appendWhite(token, options) + --[*[ + --debug_printf("appendToken(%q)", token.Data) + local data = token.Data + local lines = util.splitLines(data) + while self.line + #lines < token.Line do + if not options.no_newline then self:appendStr('\n') end -- CANDRAN : options + self.line = self.line + 1 + self.char = 1 + end + --]] + if options.no_newline then data = data:gsub("[\n\r]*", "") end -- CANDRAN : options + if options.no_leading_white then data = data:gsub("^%s+", "") end + self:appendStr(data) + end, + + -- CANDRAN : options + appendTokens = function(self, tokens, options) + for _,token in ipairs(tokens) do + self:appendToken( token, options ) -- CANDRAN : options + end + end, + + -- CANDRAN : options + appendWhite = function(self, token, options) + if token.LeadingWhite then + self:appendTokens( token.LeadingWhite, options ) -- CANDRAN : options + --self.str = self.str .. ' ' + end + end + } + + local formatStatlist, formatExpr, formatStatement; + + -- CANDRAN : added options argument + -- CANDRAN : options = { no_newline = false, no_leading_white = false } + formatExpr = function(expr, options) + local options = options or {} -- CANDRAN + local tok_it = 1 + local function appendNextToken(str) + local tok = expr.Tokens[tok_it]; + if str and tok.Data ~= str then + error("Expected token '" .. str .. "'. Tokens: " .. util.PrintTable(expr.Tokens)) + end + out:appendToken( tok, options ) -- CANDRAN : options + tok_it = tok_it + 1 + options.no_leading_white = false -- CANDRAN : not the leading token anymore + end + local function appendToken(token) + out:appendToken( token, options ) -- CANDRAN : options + tok_it = tok_it + 1 + options.no_leading_white = false -- CANDRAN : not the leading token anymore + end + local function appendWhite() + local tok = expr.Tokens[tok_it]; + if not tok then error(util.PrintTable(expr)) end + out:appendWhite( tok, options ) -- CANDRAN : options + tok_it = tok_it + 1 + options.no_leading_white = false -- CANDRAN : not the leading token anymore + end + local function appendStr(str) + appendWhite() + out:appendStr(str) + end + local function peek() + if tok_it < #expr.Tokens then + return expr.Tokens[tok_it].Data + end + end + local function appendComma(mandatory, seperators) + if true then + seperators = seperators or { "," } + seperators = util.lookupify( seperators ) + if not mandatory and not seperators[peek()] then + return + end + assert(seperators[peek()], "Missing comma or semicolon") + appendNextToken() + else + local p = peek() + if p == "," or p == ";" then + appendNextToken() + end + end + end + + debug_printf("formatExpr(%s) at line %i", expr.AstType, expr.Tokens[1] and expr.Tokens[1].Line or -1) + + if expr.AstType == 'VarExpr' then + if expr.Variable then + appendStr( expr.Variable.Name ) + else + appendStr( expr.Name ) + end + + elseif expr.AstType == 'NumberExpr' then + appendToken( expr.Value ) + + elseif expr.AstType == 'StringExpr' then + appendToken( expr.Value ) + + elseif expr.AstType == 'BooleanExpr' then + appendNextToken( expr.Value and "true" or "false" ) + + elseif expr.AstType == 'NilExpr' then + appendNextToken( "nil" ) + + elseif expr.AstType == 'BinopExpr' then + formatExpr(expr.Lhs) + appendStr( expr.Op ) + formatExpr(expr.Rhs) + + elseif expr.AstType == 'UnopExpr' then + appendStr( expr.Op ) + formatExpr(expr.Rhs) + + elseif expr.AstType == 'DotsExpr' then + appendNextToken( "..." ) + + elseif expr.AstType == 'CallExpr' then + formatExpr(expr.Base) + appendNextToken( "(" ) + for i,arg in ipairs( expr.Arguments ) do + formatExpr(arg) + appendComma( i ~= #expr.Arguments ) + end + appendNextToken( ")" ) + + elseif expr.AstType == 'TableCallExpr' then + formatExpr( expr.Base ) + formatExpr( expr.Arguments[1] ) + + elseif expr.AstType == 'StringCallExpr' then + formatExpr(expr.Base) + appendToken( expr.Arguments[1] ) + + elseif expr.AstType == 'IndexExpr' then + formatExpr(expr.Base) + appendNextToken( "[" ) + formatExpr(expr.Index) + appendNextToken( "]" ) + + elseif expr.AstType == 'MemberExpr' then + formatExpr(expr.Base) + appendNextToken() -- . or : + appendToken(expr.Ident) + + elseif expr.AstType == 'Function' then + -- anonymous function + appendNextToken( "function" ) + appendNextToken( "(" ) + if #expr.Arguments > 0 then + for i = 1, #expr.Arguments do + appendStr( expr.Arguments[i].Name ) + if i ~= #expr.Arguments then + appendNextToken(",") + elseif expr.VarArg then + appendNextToken(",") + appendNextToken("...") + end + end + elseif expr.VarArg then + appendNextToken("...") + end + appendNextToken(")") + formatStatlist(expr.Body) + appendNextToken("end") + + elseif expr.AstType == 'ConstructorExpr' then + -- CANDRAN : function to get a value with its applied decorators + local function appendValue(entry) + out:appendStr(" ") + if entry.Decorated then + for _,d in ipairs(entry.DecoratorChain) do + formatExpr(d) + out:appendStr("(") + end + end + formatExpr(entry.Value, { no_leading_white = true }) + if entry.Decorated then + for _ in ipairs(entry.DecoratorChain) do + out:appendStr(")") + end + end + end + + appendNextToken( "{" ) + for i = 1, #expr.EntryList do + local entry = expr.EntryList[i] + if entry.Type == 'Key' then + appendNextToken( "[" ) + formatExpr(entry.Key) + appendNextToken( "]" ) + appendNextToken( "=" ) + appendValue(entry) -- CANDRAN : respect decorators + elseif entry.Type == 'Value' then + appendValue(entry) -- CANDRAN : respect decorators + elseif entry.Type == 'KeyString' then + appendStr(entry.Key) + appendNextToken( "=" ) + appendValue(entry) -- CANDRAN : respect decorators + end + appendComma( i ~= #expr.EntryList, { ",", ";" } ) + end + appendNextToken( "}" ) + + elseif expr.AstType == 'Parentheses' then + appendNextToken( "(" ) + formatExpr(expr.Inner) + appendNextToken( ")" ) + + else + print("Unknown AST Type: ", statement.AstType) + end + + assert(tok_it == #expr.Tokens + 1) + debug_printf("/formatExpr") + end + + formatStatement = function(statement) + local tok_it = 1 + local function appendNextToken(str) + local tok = statement.Tokens[tok_it]; + assert(tok, string.format("Not enough tokens for %q. First token at %i:%i", + str, statement.Tokens[1].Line, statement.Tokens[1].Char)) + assert(tok.Data == str, + string.format('Expected token %q, got %q', str, tok.Data)) + out:appendToken( tok ) + tok_it = tok_it + 1 + end + local function appendToken(token) + out:appendToken( str ) + tok_it = tok_it + 1 + end + local function appendWhite() + local tok = statement.Tokens[tok_it]; + out:appendWhite( tok ) + tok_it = tok_it + 1 + end + local function appendStr(str) + appendWhite() + out:appendStr(str) + end + local function appendComma(mandatory) + if mandatory + or (tok_it < #statement.Tokens and statement.Tokens[tok_it].Data == ",") then + appendNextToken( "," ) + end + end + + debug_printf("") + debug_printf(string.format("formatStatement(%s) at line %i", statement.AstType, statement.Tokens[1] and statement.Tokens[1].Line or -1)) + + if statement.AstType == 'AssignmentStatement' then + local newlineToCheck -- CANDRAN : position of a potential newline to eliminate in some edge cases + + for i,v in ipairs(statement.Lhs) do + formatExpr(v) + appendComma( i ~= #statement.Lhs ) + end + if #statement.Rhs > 0 then + -- CANDRAN : get the assignment operator used (default to =) + local assignmentToken = "=" + local candranAssignmentExists = util.lookupify(candran.assignment) + for i,v in pairs(statement.Tokens) do + if candranAssignmentExists[v.Data] then + assignmentToken = v.Data + break + end + end + appendNextToken(assignmentToken) -- CANDRAN : accept Candran assignments operators + --appendNextToken( "=" ) + newlineToCheck = #out.rope + 1 -- CANDRAN : the potential newline position afte the = + + if assignmentToken == "=" then + for i,v in ipairs(statement.Rhs) do + formatExpr(v) + appendComma( i ~= #statement.Rhs ) + end + else + out.rope[#out.rope] = "= " -- CANDRAN : remplace +=, -=, etc. with = + for i,v in ipairs(statement.Rhs) do + if i <= #statement.Lhs then -- CANDRAN : impossible to assign more variables than indicated in Lhs + formatExpr(statement.Lhs[i], { no_newline = true }) -- CANDRAN : write variable to assign + out:appendStr(" "..assignmentToken:gsub("=$","")) -- CANDRAN : assignment operation + formatExpr(v) -- CANDRAN : write variable to add/sub/etc. + if i ~= #statement.Rhs then -- CANDRAN : add comma to allow multi-assignment + appendComma( i ~= #statement.Rhs ) + if i >= #statement.Lhs then + out.rope[#out.rope] = "" -- CANDRAN : if this was the last element, remove the comma + end + end + end + end + end + end + + -- CANDRAN : eliminate the bad newlines + if out.rope[newlineToCheck] == "\n" then + out.rope[newlineToCheck] = "" + end + + elseif statement.AstType == 'CallStatement' then + formatExpr(statement.Expression) + + elseif statement.AstType == 'LocalStatement' then + appendNextToken( "local" ) + for i = 1, #statement.LocalList do + appendStr( statement.LocalList[i].Name ) + appendComma( i ~= #statement.LocalList ) + end + if #statement.InitList > 0 then + appendNextToken( "=" ) + for i = 1, #statement.InitList do + formatExpr(statement.InitList[i]) + appendComma( i ~= #statement.InitList ) + end + end + + elseif statement.AstType == 'IfStatement' then + appendNextToken( "if" ) + formatExpr( statement.Clauses[1].Condition ) + appendNextToken( "then" ) + formatStatlist( statement.Clauses[1].Body ) + for i = 2, #statement.Clauses do + local st = statement.Clauses[i] + if st.Condition then + appendNextToken( "elseif" ) + formatExpr(st.Condition) + appendNextToken( "then" ) + else + appendNextToken( "else" ) + end + formatStatlist(st.Body) + end + appendNextToken( "end" ) + + elseif statement.AstType == 'WhileStatement' then + appendNextToken( "while" ) + formatExpr(statement.Condition) + appendNextToken( "do" ) + formatStatlist(statement.Body) + appendNextToken( "end" ) + + elseif statement.AstType == 'DoStatement' then + appendNextToken( "do" ) + formatStatlist(statement.Body) + appendNextToken( "end" ) + + elseif statement.AstType == 'ReturnStatement' then + appendNextToken( "return" ) + for i = 1, #statement.Arguments do + formatExpr(statement.Arguments[i]) + appendComma( i ~= #statement.Arguments ) + end + + elseif statement.AstType == 'BreakStatement' then + appendNextToken( "break" ) + + elseif statement.AstType == 'RepeatStatement' then + appendNextToken( "repeat" ) + formatStatlist(statement.Body) + appendNextToken( "until" ) + formatExpr(statement.Condition) + + -- CANDRAN : add decorator support (@) + elseif statement.AstType == 'DecoratedStatement' then + -- CANDRAN : list of the chained decorators + local decoratorChain = {statement} + + -- CANDRAN : get the decorated statement + local decorated = statement.Decorated + while decorated.AstType == "DecoratedStatement" do + table.insert(decoratorChain, decorated) + decorated = decorated.Decorated + end + + -- CANDRAN : write the decorated statement like a normal statement + formatStatement(decorated) + + -- CANDRAN : mark the decorator token as used (and add whitespace) + appendNextToken(candran.decorator) + out.rope[#out.rope] = "" + + -- CANDRAN : get the variable(s) to decorate name(s) + local names = {} + if decorated.AstType == "Function" then + table.insert(names, decorated.Name.Name) + elseif decorated.AstType == "AssignmentStatement" then + for _,var in ipairs(decorated.Lhs) do + table.insert(names, var.Name) + end + elseif decorated.AstType == "LocalStatement" then + for _,var in ipairs(decorated.LocalList) do + table.insert(names, var.Name) + end + else + error("Invalid statement type to decorate : "..decorated.AstType) + end + + -- CANDRAN : redefine the variable(s) ( name, name2, ... = ... ) + for i,name in ipairs(names) do + out:appendStr(name) + if i ~= #names then out:appendStr(", ") end + end + out:appendStr(" = ") + + for i,name in ipairs(names) do + -- CANDRAN : write the decorator chain ( a(b(c(... ) + for _,v in pairs(decoratorChain) do + formatExpr(v.Decorator) + out:appendStr("(") + end + + -- CANDRAN : pass the undecorated variable name to the decorator chain + out:appendStr(name) + + -- CANDRAN : close parantheses + for _ in pairs(decoratorChain) do + out:appendStr(")") + end + + if i ~= #names then out:appendStr(", ") end + end + + elseif statement.AstType == 'Function' then + --print(util.PrintTable(statement)) + + if statement.IsLocal then + appendNextToken( "local" ) + end + appendNextToken( "function" ) + + if statement.IsLocal then + appendStr(statement.Name.Name) + else + formatExpr(statement.Name) + end + + appendNextToken( "(" ) + if #statement.Arguments > 0 then + for i = 1, #statement.Arguments do + appendStr( statement.Arguments[i].Name ) + appendComma( i ~= #statement.Arguments or statement.VarArg ) + if i == #statement.Arguments and statement.VarArg then + appendNextToken( "..." ) + end + end + elseif statement.VarArg then + appendNextToken( "..." ) + end + appendNextToken( ")" ) + + formatStatlist(statement.Body) + appendNextToken( "end" ) + + elseif statement.AstType == 'GenericForStatement' then + appendNextToken( "for" ) + for i = 1, #statement.VariableList do + appendStr( statement.VariableList[i].Name ) + appendComma( i ~= #statement.VariableList ) + end + appendNextToken( "in" ) + for i = 1, #statement.Generators do + formatExpr(statement.Generators[i]) + appendComma( i ~= #statement.Generators ) + end + appendNextToken( "do" ) + formatStatlist(statement.Body) + appendNextToken( "end" ) + + elseif statement.AstType == 'NumericForStatement' then + appendNextToken( "for" ) + appendStr( statement.Variable.Name ) + appendNextToken( "=" ) + formatExpr(statement.Start) + appendNextToken( "," ) + formatExpr(statement.End) + if statement.Step then + appendNextToken( "," ) + formatExpr(statement.Step) + end + appendNextToken( "do" ) + formatStatlist(statement.Body) + appendNextToken( "end" ) + + elseif statement.AstType == 'LabelStatement' then + appendNextToken( "::" ) + appendStr( statement.Label ) + appendNextToken( "::" ) + + elseif statement.AstType == 'GotoStatement' then + appendNextToken( "goto" ) + appendStr( statement.Label ) + + elseif statement.AstType == 'Eof' then + appendWhite() + + else + print("Unknown AST Type: ", statement.AstType) + end + + if statement.Semicolon then + appendNextToken(";") + end + + assert(tok_it == #statement.Tokens + 1) + debug_printf("/formatStatment") + end + + formatStatlist = function(statList) + for _, stat in ipairs(statList.Body) do + formatStatement(stat) + end + end + + formatStatlist(ast) + + return true, table.concat(out.rope) +end + +return Format_Identity diff --git a/lib/LuaMinify/FormatMini.lua b/lib/LuaMinify/FormatMini.lua new file mode 100644 index 0000000..8bd9c28 --- /dev/null +++ b/lib/LuaMinify/FormatMini.lua @@ -0,0 +1,364 @@ + +local parser = require'ParseLua' +local ParseLua = parser.ParseLua +local util = require'Util' +local lookupify = util.lookupify + +-- +-- FormatMini.lua +-- +-- Returns the minified version of an AST. Operations which are performed: +-- - All comments and whitespace are ignored +-- - All local variables are renamed +-- + +local LowerChars = lookupify{'a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', + 'j', 'k', 'l', 'm', 'n', 'o', 'p', 'q', 'r', + 's', 't', 'u', 'v', 'w', 'x', 'y', 'z'} +local UpperChars = lookupify{'A', 'B', 'C', 'D', 'E', 'F', 'G', 'H', 'I', + 'J', 'K', 'L', 'M', 'N', 'O', 'P', 'Q', 'R', + 'S', 'T', 'U', 'V', 'W', 'X', 'Y', 'Z'} +local Digits = lookupify{'0', '1', '2', '3', '4', '5', '6', '7', '8', '9'} +local Symbols = lookupify{'+', '-', '*', '/', '^', '%', ',', '{', '}', '[', ']', '(', ')', ';', '#'} + +local function Format_Mini(ast) + local formatStatlist, formatExpr; + local count = 0 + -- + local function joinStatementsSafe(a, b, sep) + --print(a, b) + if count > 150 then + count = 0 + return a.."\n"..b + end + sep = sep or ' ' + local aa, bb = a:sub(-1,-1), b:sub(1,1) + if UpperChars[aa] or LowerChars[aa] or aa == '_' then + if not (UpperChars[bb] or LowerChars[bb] or bb == '_' or Digits[bb]) then + --bb is a symbol, can join without sep + return a..b + elseif bb == '(' then + print("==============>>>",aa,bb) + --prevent ambiguous syntax + return a..sep..b + else + return a..sep..b + end + elseif Digits[aa] then + if bb == '(' then + --can join statements directly + return a..b + elseif Symbols[bb] then + return a .. b + else + return a..sep..b + end + elseif aa == '' then + return a..b + else + if bb == '(' then + --don't want to accidentally call last statement, can't join directly + return a..sep..b + else + --print("asdf", '"'..a..'"', '"'..b..'"') + return a..b + end + end + end + + formatExpr = function(expr, precedence) + local precedence = precedence or 0 + local currentPrecedence = 0 + local skipParens = false + local out = "" + if expr.AstType == 'VarExpr' then + if expr.Variable then + out = out..expr.Variable.Name + else + out = out..expr.Name + end + + elseif expr.AstType == 'NumberExpr' then + out = out..expr.Value.Data + + elseif expr.AstType == 'StringExpr' then + out = out..expr.Value.Data + + elseif expr.AstType == 'BooleanExpr' then + out = out..tostring(expr.Value) + + elseif expr.AstType == 'NilExpr' then + out = joinStatementsSafe(out, "nil") + + elseif expr.AstType == 'BinopExpr' then + currentPrecedence = expr.OperatorPrecedence + out = joinStatementsSafe(out, formatExpr(expr.Lhs, currentPrecedence)) + out = joinStatementsSafe(out, expr.Op) + out = joinStatementsSafe(out, formatExpr(expr.Rhs)) + if expr.Op == '^' or expr.Op == '..' then + currentPrecedence = currentPrecedence - 1 + end + + if currentPrecedence < precedence then + skipParens = false + else + skipParens = true + end + --print(skipParens, precedence, currentPrecedence) + elseif expr.AstType == 'UnopExpr' then + out = joinStatementsSafe(out, expr.Op) + out = joinStatementsSafe(out, formatExpr(expr.Rhs)) + + elseif expr.AstType == 'DotsExpr' then + out = out.."..." + + elseif expr.AstType == 'CallExpr' then + out = out..formatExpr(expr.Base) + out = out.."(" + for i = 1, #expr.Arguments do + out = out..formatExpr(expr.Arguments[i]) + if i ~= #expr.Arguments then + out = out.."," + end + end + out = out..")" + + elseif expr.AstType == 'TableCallExpr' then + out = out..formatExpr(expr.Base) + out = out..formatExpr(expr.Arguments[1]) + + elseif expr.AstType == 'StringCallExpr' then + out = out..formatExpr(expr.Base) + out = out..expr.Arguments[1].Data + + elseif expr.AstType == 'IndexExpr' then + out = out..formatExpr(expr.Base).."["..formatExpr(expr.Index).."]" + + elseif expr.AstType == 'MemberExpr' then + out = out..formatExpr(expr.Base)..expr.Indexer..expr.Ident.Data + + elseif expr.AstType == 'Function' then + expr.Scope:ObfuscateVariables() + out = out.."function(" + if #expr.Arguments > 0 then + for i = 1, #expr.Arguments do + out = out..expr.Arguments[i].Name + if i ~= #expr.Arguments then + out = out.."," + elseif expr.VarArg then + out = out..",..." + end + end + elseif expr.VarArg then + out = out.."..." + end + out = out..")" + out = joinStatementsSafe(out, formatStatlist(expr.Body)) + out = joinStatementsSafe(out, "end") + + elseif expr.AstType == 'ConstructorExpr' then + out = out.."{" + for i = 1, #expr.EntryList do + local entry = expr.EntryList[i] + if entry.Type == 'Key' then + out = out.."["..formatExpr(entry.Key).."]="..formatExpr(entry.Value) + elseif entry.Type == 'Value' then + out = out..formatExpr(entry.Value) + elseif entry.Type == 'KeyString' then + out = out..entry.Key.."="..formatExpr(entry.Value) + end + if i ~= #expr.EntryList then + out = out.."," + end + end + out = out.."}" + + elseif expr.AstType == 'Parentheses' then + out = out.."("..formatExpr(expr.Inner)..")" + + end + --print(">>", skipParens, expr.ParenCount, out) + if not skipParens then + --print("hehe") + out = string.rep('(', expr.ParenCount or 0) .. out + out = out .. string.rep(')', expr.ParenCount or 0) + --print("", out) + end + count = count + #out + return --[[print(out) or]] out + end + + local formatStatement = function(statement) + local out = '' + if statement.AstType == 'AssignmentStatement' then + for i = 1, #statement.Lhs do + out = out..formatExpr(statement.Lhs[i]) + if i ~= #statement.Lhs then + out = out.."," + end + end + if #statement.Rhs > 0 then + out = out.."=" + for i = 1, #statement.Rhs do + out = out..formatExpr(statement.Rhs[i]) + if i ~= #statement.Rhs then + out = out.."," + end + end + end + + elseif statement.AstType == 'CallStatement' then + out = formatExpr(statement.Expression) + + elseif statement.AstType == 'LocalStatement' then + out = out.."local " + for i = 1, #statement.LocalList do + out = out..statement.LocalList[i].Name + if i ~= #statement.LocalList then + out = out.."," + end + end + if #statement.InitList > 0 then + out = out.."=" + for i = 1, #statement.InitList do + out = out..formatExpr(statement.InitList[i]) + if i ~= #statement.InitList then + out = out.."," + end + end + end + + elseif statement.AstType == 'IfStatement' then + out = joinStatementsSafe("if", formatExpr(statement.Clauses[1].Condition)) + out = joinStatementsSafe(out, "then") + out = joinStatementsSafe(out, formatStatlist(statement.Clauses[1].Body)) + for i = 2, #statement.Clauses do + local st = statement.Clauses[i] + if st.Condition then + out = joinStatementsSafe(out, "elseif") + out = joinStatementsSafe(out, formatExpr(st.Condition)) + out = joinStatementsSafe(out, "then") + else + out = joinStatementsSafe(out, "else") + end + out = joinStatementsSafe(out, formatStatlist(st.Body)) + end + out = joinStatementsSafe(out, "end") + + elseif statement.AstType == 'WhileStatement' then + out = joinStatementsSafe("while", formatExpr(statement.Condition)) + out = joinStatementsSafe(out, "do") + out = joinStatementsSafe(out, formatStatlist(statement.Body)) + out = joinStatementsSafe(out, "end") + + elseif statement.AstType == 'DoStatement' then + out = joinStatementsSafe(out, "do") + out = joinStatementsSafe(out, formatStatlist(statement.Body)) + out = joinStatementsSafe(out, "end") + + elseif statement.AstType == 'ReturnStatement' then + out = "return" + for i = 1, #statement.Arguments do + out = joinStatementsSafe(out, formatExpr(statement.Arguments[i])) + if i ~= #statement.Arguments then + out = out.."," + end + end + + elseif statement.AstType == 'BreakStatement' then + out = "break" + + elseif statement.AstType == 'RepeatStatement' then + out = "repeat" + out = joinStatementsSafe(out, formatStatlist(statement.Body)) + out = joinStatementsSafe(out, "until") + out = joinStatementsSafe(out, formatExpr(statement.Condition)) + + elseif statement.AstType == 'Function' then + statement.Scope:ObfuscateVariables() + if statement.IsLocal then + out = "local" + end + out = joinStatementsSafe(out, "function ") + if statement.IsLocal then + out = out..statement.Name.Name + else + out = out..formatExpr(statement.Name) + end + out = out.."(" + if #statement.Arguments > 0 then + for i = 1, #statement.Arguments do + out = out..statement.Arguments[i].Name + if i ~= #statement.Arguments then + out = out.."," + elseif statement.VarArg then + --print("Apply vararg") + out = out..",..." + end + end + elseif statement.VarArg then + out = out.."..." + end + out = out..")" + out = joinStatementsSafe(out, formatStatlist(statement.Body)) + out = joinStatementsSafe(out, "end") + + elseif statement.AstType == 'GenericForStatement' then + statement.Scope:ObfuscateVariables() + out = "for " + for i = 1, #statement.VariableList do + out = out..statement.VariableList[i].Name + if i ~= #statement.VariableList then + out = out.."," + end + end + out = out.." in" + for i = 1, #statement.Generators do + out = joinStatementsSafe(out, formatExpr(statement.Generators[i])) + if i ~= #statement.Generators then + out = joinStatementsSafe(out, ',') + end + end + out = joinStatementsSafe(out, "do") + out = joinStatementsSafe(out, formatStatlist(statement.Body)) + out = joinStatementsSafe(out, "end") + + elseif statement.AstType == 'NumericForStatement' then + out = "for " + out = out..statement.Variable.Name.."=" + out = out..formatExpr(statement.Start)..","..formatExpr(statement.End) + if statement.Step then + out = out..","..formatExpr(statement.Step) + end + out = joinStatementsSafe(out, "do") + out = joinStatementsSafe(out, formatStatlist(statement.Body)) + out = joinStatementsSafe(out, "end") + elseif statement.AstType == 'LabelStatement' then + out = getIndentation() .. "::" .. statement.Label .. "::" + elseif statement.AstType == 'GotoStatement' then + out = getIndentation() .. "goto " .. statement.Label + elseif statement.AstType == 'Comment' then + -- ignore + elseif statement.AstType == 'Eof' then + -- ignore + else + print("Unknown AST Type: " .. statement.AstType) + end + count = count + #out + return out + end + + formatStatlist = function(statList) + local out = '' + statList.Scope:ObfuscateVariables() + for _, stat in pairs(statList.Body) do + out = joinStatementsSafe(out, formatStatement(stat), ';') + end + return out + end + + ast.Scope:ObfuscateVariables() + return formatStatlist(ast) +end + +return Format_Mini diff --git a/lib/LuaMinify/LICENSE.md b/lib/LuaMinify/LICENSE.md new file mode 100644 index 0000000..8e9ca9e --- /dev/null +++ b/lib/LuaMinify/LICENSE.md @@ -0,0 +1,20 @@ +The MIT License (MIT) + +Copyright (c) 2012-2013 + +Permission is hereby granted, free of charge, to any person obtaining a copy of +this software and associated documentation files (the "Software"), to deal in +the Software without restriction, including without limitation the rights to +use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of +the Software, and to permit persons to whom the Software is furnished to do so, +subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS +FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR +COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER +IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN +CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. diff --git a/lib/LuaMinify/LuaMinify.bat b/lib/LuaMinify/LuaMinify.bat new file mode 100644 index 0000000..1630486 --- /dev/null +++ b/lib/LuaMinify/LuaMinify.bat @@ -0,0 +1,2 @@ +@echo off +lua CommandLineMinify.lua %* \ No newline at end of file diff --git a/lib/LuaMinify/LuaMinify.sh b/lib/LuaMinify/LuaMinify.sh new file mode 100644 index 0000000..75570c0 --- /dev/null +++ b/lib/LuaMinify/LuaMinify.sh @@ -0,0 +1,2 @@ +#!/bin/bash +lua CommandLineMinify.lua $@ \ No newline at end of file diff --git a/lib/LuaMinify/ParseCandran.lua b/lib/LuaMinify/ParseCandran.lua new file mode 100644 index 0000000..8105f1f --- /dev/null +++ b/lib/LuaMinify/ParseCandran.lua @@ -0,0 +1,1523 @@ +-- +-- CANDRAN +-- Based on the ParseLua.lua of LuaMinify. +-- Modified by Thomas99 to parse Candran code. +-- +-- Modified parts are marked with "-- CANDRAN" comments. +-- + +--[[ +This file is part of LuaMinify by stravant (https://github.com/stravant/LuaMinify). + +LICENSE : +The MIT License (MIT) + +Copyright (c) 2012-2013 + +Permission is hereby granted, free of charge, to any person obtaining a copy of +this software and associated documentation files (the "Software"), to deal in +the Software without restriction, including without limitation the rights to +use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of +the Software, and to permit persons to whom the Software is furnished to do so, +subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS +FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR +COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER +IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN +CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. +]] + +-- +-- ParseLua.lua +-- +-- The main lua parser and lexer. +-- LexLua returns a Lua token stream, with tokens that preserve +-- all whitespace formatting information. +-- ParseLua returns an AST, internally relying on LexLua. +-- + +--require'LuaMinify.Strict' -- CANDRAN : comment, useless here + +-- CANDRAN : add Candran syntaxic additions +local candran = require("candran").syntax + +local util = require 'lib.LuaMinify.Util' +local lookupify = util.lookupify + +local WhiteChars = lookupify{' ', '\n', '\t', '\r'} +local EscapeLookup = {['\r'] = '\\r', ['\n'] = '\\n', ['\t'] = '\\t', ['"'] = '\\"', ["'"] = "\\'"} +local LowerChars = lookupify{'a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', + 'j', 'k', 'l', 'm', 'n', 'o', 'p', 'q', 'r', + 's', 't', 'u', 'v', 'w', 'x', 'y', 'z'} +local UpperChars = lookupify{'A', 'B', 'C', 'D', 'E', 'F', 'G', 'H', 'I', + 'J', 'K', 'L', 'M', 'N', 'O', 'P', 'Q', 'R', + 'S', 'T', 'U', 'V', 'W', 'X', 'Y', 'Z'} +local Digits = lookupify{'0', '1', '2', '3', '4', '5', '6', '7', '8', '9'} +local HexDigits = lookupify{'0', '1', '2', '3', '4', '5', '6', '7', '8', '9', + 'A', 'a', 'B', 'b', 'C', 'c', 'D', 'd', 'E', 'e', 'F', 'f'} + +local Symbols = lookupify{'+', '-', '*', '/', '^', '%', ',', '{', '}', '[', ']', '(', ')', ';', '#', + table.unpack(candran.assignment)} -- CANDRAN : Candran symbols +local Scope = require'lib.LuaMinify.Scope' + +local Keywords = lookupify{ + 'and', 'break', 'do', 'else', 'elseif', + 'end', 'false', 'for', 'function', 'goto', 'if', + 'in', 'local', 'nil', 'not', 'or', 'repeat', + 'return', 'then', 'true', 'until', 'while', candran.decorator -- LUA : Candran keywords +}; + +local function LexLua(src) + --token dump + local tokens = {} + + local st, err = pcall(function() + --line / char / pointer tracking + local p = 1 + local line = 1 + local char = 1 + + --get / peek functions + local function get() + local c = src:sub(p,p) + if c == '\n' then + char = 1 + line = line + 1 + else + char = char + 1 + end + p = p + 1 + return c + end + local function peek(n) + n = n or 0 + return src:sub(p+n,p+n) + end + local function consume(chars) + local c = peek() + for i = 1, #chars do + if c == chars:sub(i,i) then return get() end + end + end + + --shared stuff + local function generateError(err) + return error(">> :"..line..":"..char..": "..err, 0) + end + + local function tryGetLongString() + local start = p + if peek() == '[' then + local equalsCount = 0 + local depth = 1 + while peek(equalsCount+1) == '=' do + equalsCount = equalsCount + 1 + end + if peek(equalsCount+1) == '[' then + --start parsing the string. Strip the starting bit + for _ = 0, equalsCount+1 do get() end + + --get the contents + local contentStart = p + while true do + --check for eof + if peek() == '' then + generateError("Expected `]"..string.rep('=', equalsCount).."]` near .", 3) + end + + --check for the end + local foundEnd = true + if peek() == ']' then + for i = 1, equalsCount do + if peek(i) ~= '=' then foundEnd = false end + end + if peek(equalsCount+1) ~= ']' then + foundEnd = false + end + else + if peek() == '[' then + -- is there an embedded long string? + local embedded = true + for i = 1, equalsCount do + if peek(i) ~= '=' then + embedded = false + break + end + end + if peek(equalsCount + 1) == '[' and embedded then + -- oh look, there was + depth = depth + 1 + for i = 1, (equalsCount + 2) do + get() + end + end + end + foundEnd = false + end + -- + if foundEnd then + depth = depth - 1 + if depth == 0 then + break + else + for i = 1, equalsCount + 2 do + get() + end + end + else + get() + end + end + + --get the interior string + local contentString = src:sub(contentStart, p-1) + + --found the end. Get rid of the trailing bit + for i = 0, equalsCount+1 do get() end + + --get the exterior string + local longString = src:sub(start, p-1) + + --return the stuff + return contentString, longString + else + return nil + end + else + return nil + end + end + + --main token emitting loop + while true do + --get leading whitespace. The leading whitespace will include any comments + --preceding the token. This prevents the parser needing to deal with comments + --separately. + local leading = { } + local leadingWhite = '' + local longStr = false + while true do + local c = peek() + if c == '#' and peek(1) == '!' and line == 1 then + -- #! shebang for linux scripts + get() + get() + leadingWhite = "#!" + while peek() ~= '\n' and peek() ~= '' do + leadingWhite = leadingWhite .. get() + end + local token = { + Type = 'Comment', + CommentType = 'Shebang', + Data = leadingWhite, + Line = line, + Char = char + } + token.Print = function() + return "<"..(token.Type .. string.rep(' ', 7-#token.Type)).." "..(token.Data or '').." >" + end + leadingWhite = "" + table.insert(leading, token) + end + if c == ' ' or c == '\t' then + --whitespace + --leadingWhite = leadingWhite..get() + local c2 = get() -- ignore whitespace + table.insert(leading, { Type = 'Whitespace', Line = line, Char = char, Data = c2 }) + elseif c == '\n' or c == '\r' then + local nl = get() + if leadingWhite ~= "" then + local token = { + Type = 'Comment', + CommentType = longStr and 'LongComment' or 'Comment', + Data = leadingWhite, + Line = line, + Char = char, + } + token.Print = function() + return "<"..(token.Type .. string.rep(' ', 7-#token.Type)).." "..(token.Data or '').." >" + end + table.insert(leading, token) + leadingWhite = "" + end + table.insert(leading, { Type = 'Whitespace', Line = line, Char = char, Data = nl }) + elseif c == '-' and peek(1) == '-' then + --comment + get() + get() + leadingWhite = leadingWhite .. '--' + local _, wholeText = tryGetLongString() + if wholeText then + leadingWhite = leadingWhite..wholeText + longStr = true + else + while peek() ~= '\n' and peek() ~= '' do + leadingWhite = leadingWhite..get() + end + end + else + break + end + end + if leadingWhite ~= "" then + local token = { + Type = 'Comment', + CommentType = longStr and 'LongComment' or 'Com mnment', + Data = leadingWhite, + Line = line, + Char = char, + } + token.Print = function() + return "<"..(token.Type .. string.rep(' ', 7-#token.Type)).." "..(token.Data or '').." >" + end + table.insert(leading, token) + end + + --get the initial char + local thisLine = line + local thisChar = char + local errorAt = ":"..line..":"..char..":> " + local c = peek() + + --symbol to emit + local toEmit = nil + + --branch on type + if c == '' then + --eof + toEmit = { Type = 'Eof' } + + -- CANDRAN : add decorator symbol (@) + elseif c == candran.decorator then + get() + toEmit = {Type = 'Keyword', Data = c} + + elseif UpperChars[c] or LowerChars[c] or c == '_' then + --ident or keyword + local start = p + repeat + get() + c = peek() + until not (UpperChars[c] or LowerChars[c] or Digits[c] or c == '_') + local dat = src:sub(start, p-1) + if Keywords[dat] then + toEmit = {Type = 'Keyword', Data = dat} + else + toEmit = {Type = 'Ident', Data = dat} + end + + elseif Digits[c] or (peek() == '.' and Digits[peek(1)]) then + --number const + local start = p + if c == '0' and peek(1) == 'x' then + get();get() + while HexDigits[peek()] do get() end + if consume('Pp') then + consume('+-') + while Digits[peek()] do get() end + end + else + while Digits[peek()] do get() end + if consume('.') then + while Digits[peek()] do get() end + end + if consume('Ee') then + consume('+-') + while Digits[peek()] do get() end + end + end + toEmit = {Type = 'Number', Data = src:sub(start, p-1)} + + elseif c == '\'' or c == '\"' then + local start = p + --string const + local delim = get() + local contentStart = p + while true do + local c = get() + if c == '\\' then + get() --get the escape char + elseif c == delim then + break + elseif c == '' then + generateError("Unfinished string near ") + end + end + local content = src:sub(contentStart, p-2) + local constant = src:sub(start, p-1) + toEmit = {Type = 'String', Data = constant, Constant = content} + + -- CANDRAN : accept 3 and 2 caracters symbols + elseif Symbols[c..peek(1)..peek(2)] then + local c = c..peek(1)..peek(2) + get() get() get() + toEmit = {Type = 'Symbol', Data = c} + elseif Symbols[c..peek(1)] then + local c = c..peek(1) + get() get() + toEmit = {Type = 'Symbol', Data = c} + + elseif c == '[' then + local content, wholetext = tryGetLongString() + if wholetext then + toEmit = {Type = 'String', Data = wholetext, Constant = content} + else + get() + toEmit = {Type = 'Symbol', Data = '['} + end + + elseif consume('>=<') then + if consume('=') then + toEmit = {Type = 'Symbol', Data = c..'='} + else + toEmit = {Type = 'Symbol', Data = c} + end + + elseif consume('~') then + if consume('=') then + toEmit = {Type = 'Symbol', Data = '~='} + else + generateError("Unexpected symbol `~` in source.", 2) + end + + elseif consume('.') then + if consume('.') then + if consume('.') then + toEmit = {Type = 'Symbol', Data = '...'} + else + toEmit = {Type = 'Symbol', Data = '..'} + end + else + toEmit = {Type = 'Symbol', Data = '.'} + end + + elseif consume(':') then + if consume(':') then + toEmit = {Type = 'Symbol', Data = '::'} + else + toEmit = {Type = 'Symbol', Data = ':'} + end + + elseif Symbols[c] then + get() + toEmit = {Type = 'Symbol', Data = c} + + else + local contents, all = tryGetLongString() + if contents then + toEmit = {Type = 'String', Data = all, Constant = contents} + else + generateError("Unexpected Symbol `"..c.."` in source.", 2) + end + end + + --add the emitted symbol, after adding some common data + toEmit.LeadingWhite = leading -- table of leading whitespace/comments + --for k, tok in pairs(leading) do + -- tokens[#tokens + 1] = tok + --end + + toEmit.Line = thisLine + toEmit.Char = thisChar + toEmit.Print = function() + return "<"..(toEmit.Type..string.rep(' ', 7-#toEmit.Type)).." "..(toEmit.Data or '').." >" + end + tokens[#tokens+1] = toEmit + + --halt after eof has been emitted + if toEmit.Type == 'Eof' then break end + end + end) + if not st then + return false, err + end + + --public interface: + local tok = {} + local savedP = {} + local p = 1 + + function tok:getp() + return p + end + + function tok:setp(n) + p = n + end + + function tok:getTokenList() + return tokens + end + + --getters + function tok:Peek(n) + n = n or 0 + return tokens[math.min(#tokens, p+n)] + end + function tok:Get(tokenList) + local t = tokens[p] + p = math.min(p + 1, #tokens) + if tokenList then + table.insert(tokenList, t) + end + return t + end + function tok:Is(t) + return tok:Peek().Type == t + end + + --save / restore points in the stream + function tok:Save() + savedP[#savedP+1] = p + end + function tok:Commit() + savedP[#savedP] = nil + end + function tok:Restore() + p = savedP[#savedP] + savedP[#savedP] = nil + end + + --either return a symbol if there is one, or return true if the requested + --symbol was gotten. + function tok:ConsumeSymbol(symb, tokenList) + local t = self:Peek() + if t.Type == 'Symbol' then + if symb then + if t.Data == symb then + self:Get(tokenList) + return true + else + return nil + end + else + self:Get(tokenList) + return t + end + else + return nil + end + end + + function tok:ConsumeKeyword(kw, tokenList) + local t = self:Peek() + if t.Type == 'Keyword' and t.Data == kw then + self:Get(tokenList) + return true + else + return nil + end + end + + function tok:IsKeyword(kw) + local t = tok:Peek() + return t.Type == 'Keyword' and t.Data == kw + end + + function tok:IsSymbol(s) + local t = tok:Peek() + return t.Type == 'Symbol' and t.Data == s + end + + function tok:IsEof() + return tok:Peek().Type == 'Eof' + end + + return true, tok +end + + +local function ParseLua(src) + local st, tok + if type(src) ~= 'table' then + st, tok = LexLua(src) + else + st, tok = true, src + end + if not st then + return false, tok + end + -- + local function GenerateError(msg) + local err = ">> :"..tok:Peek().Line..":"..tok:Peek().Char..": "..msg.."\n" + --find the line + local lineNum = 0 + if type(src) == 'string' then + for line in src:gmatch("[^\n]*\n?") do + if line:sub(-1,-1) == '\n' then line = line:sub(1,-2) end + lineNum = lineNum+1 + if lineNum == tok:Peek().Line then + err = err..">> `"..line:gsub('\t',' ').."`\n" + for i = 1, tok:Peek().Char do + local c = line:sub(i,i) + if c == '\t' then + err = err..' ' + else + err = err..' ' + end + end + err = err.." ^^^^" + break + end + end + end + return err + end + -- + local VarUid = 0 + -- No longer needed: handled in Scopes now local GlobalVarGetMap = {} + local VarDigits = {'_', 'a', 'b', 'c', 'd'} + local function CreateScope(parent) + --[[ + local scope = {} + scope.Parent = parent + scope.LocalList = {} + scope.LocalMap = {} + + function scope:ObfuscateVariables() + for _, var in pairs(scope.LocalList) do + local id = "" + repeat + local chars = "QWERTYUIOPASDFGHJKLZXCVBNMqwertyuioplkjhgfdsazxcvbnm_" + local chars2 = "QWERTYUIOPASDFGHJKLZXCVBNMqwertyuioplkjhgfdsazxcvbnm_1234567890" + local n = math.random(1, #chars) + id = id .. chars:sub(n, n) + for i = 1, math.random(0,20) do + local n = math.random(1, #chars2) + id = id .. chars2:sub(n, n) + end + until not GlobalVarGetMap[id] and not parent:GetLocal(id) and not scope.LocalMap[id] + var.Name = id + scope.LocalMap[id] = var + end + end + + scope.RenameVars = scope.ObfuscateVariables + + -- Renames a variable from this scope and down. + -- Does not rename global variables. + function scope:RenameVariable(old, newName) + if type(old) == "table" then -- its (theoretically) an AstNode variable + old = old.Name + end + for _, var in pairs(scope.LocalList) do + if var.Name == old then + var.Name = newName + scope.LocalMap[newName] = var + end + end + end + + function scope:GetLocal(name) + --first, try to get my variable + local my = scope.LocalMap[name] + if my then return my end + + --next, try parent + if scope.Parent then + local par = scope.Parent:GetLocal(name) + if par then return par end + end + + return nil + end + + function scope:CreateLocal(name) + --create my own var + local my = {} + my.Scope = scope + my.Name = name + my.CanRename = true + -- + scope.LocalList[#scope.LocalList+1] = my + scope.LocalMap[name] = my + -- + return my + end]] + local scope = Scope:new(parent) + scope.RenameVars = scope.ObfuscateLocals + scope.ObfuscateVariables = scope.ObfuscateLocals + scope.Print = function() return "" end + return scope + end + + local ParseExpr + local ParseStatementList + local ParseSimpleExpr, + ParseSubExpr, + ParsePrimaryExpr, + ParseSuffixedExpr + + local function ParseFunctionArgsAndBody(scope, tokenList) + local funcScope = CreateScope(scope) + if not tok:ConsumeSymbol('(', tokenList) then + return false, GenerateError("`(` expected.") + end + + --arg list + local argList = {} + local isVarArg = false + while not tok:ConsumeSymbol(')', tokenList) do + if tok:Is('Ident') then + local arg = funcScope:CreateLocal(tok:Get(tokenList).Data) + argList[#argList+1] = arg + if not tok:ConsumeSymbol(',', tokenList) then + if tok:ConsumeSymbol(')', tokenList) then + break + else + return false, GenerateError("`)` expected.") + end + end + elseif tok:ConsumeSymbol('...', tokenList) then + isVarArg = true + if not tok:ConsumeSymbol(')', tokenList) then + return false, GenerateError("`...` must be the last argument of a function.") + end + break + else + return false, GenerateError("Argument name or `...` expected") + end + end + + --body + local st, body = ParseStatementList(funcScope) + if not st then return false, body end + + --end + if not tok:ConsumeKeyword('end', tokenList) then + return false, GenerateError("`end` expected after function body") + end + local nodeFunc = {} + nodeFunc.AstType = 'Function' + nodeFunc.Scope = funcScope + nodeFunc.Arguments = argList + nodeFunc.Body = body + nodeFunc.VarArg = isVarArg + nodeFunc.Tokens = tokenList + -- + return true, nodeFunc + end + + + function ParsePrimaryExpr(scope) + local tokenList = {} + + if tok:ConsumeSymbol('(', tokenList) then + local st, ex = ParseExpr(scope) + if not st then return false, ex end + if not tok:ConsumeSymbol(')', tokenList) then + return false, GenerateError("`)` Expected.") + end + if false then + --save the information about parenthesized expressions somewhere + ex.ParenCount = (ex.ParenCount or 0) + 1 + return true, ex + else + local parensExp = {} + parensExp.AstType = 'Parentheses' + parensExp.Inner = ex + parensExp.Tokens = tokenList + return true, parensExp + end + + elseif tok:Is('Ident') then + local id = tok:Get(tokenList) + local var = scope:GetLocal(id.Data) + if not var then + var = scope:GetGlobal(id.Data) + if not var then + var = scope:CreateGlobal(id.Data) + else + var.References = var.References + 1 + end + else + var.References = var.References + 1 + end + -- + local nodePrimExp = {} + nodePrimExp.AstType = 'VarExpr' + nodePrimExp.Name = id.Data + nodePrimExp.Variable = var + nodePrimExp.Tokens = tokenList + -- + return true, nodePrimExp + else + return false, GenerateError("primary expression expected") + end + end + + function ParseSuffixedExpr(scope, onlyDotColon) + --base primary expression + local st, prim = ParsePrimaryExpr(scope) + if not st then return false, prim end + -- + while true do + local tokenList = {} + + if tok:IsSymbol('.') or tok:IsSymbol(':') then + local symb = tok:Get(tokenList).Data + if not tok:Is('Ident') then + return false, GenerateError(" expected.") + end + local id = tok:Get(tokenList) + local nodeIndex = {} + nodeIndex.AstType = 'MemberExpr' + nodeIndex.Base = prim + nodeIndex.Indexer = symb + nodeIndex.Ident = id + nodeIndex.Tokens = tokenList + -- + prim = nodeIndex + + elseif not onlyDotColon and tok:ConsumeSymbol('[', tokenList) then + local st, ex = ParseExpr(scope) + if not st then return false, ex end + if not tok:ConsumeSymbol(']', tokenList) then + return false, GenerateError("`]` expected.") + end + local nodeIndex = {} + nodeIndex.AstType = 'IndexExpr' + nodeIndex.Base = prim + nodeIndex.Index = ex + nodeIndex.Tokens = tokenList + -- + prim = nodeIndex + + elseif not onlyDotColon and tok:ConsumeSymbol('(', tokenList) then + local args = {} + while not tok:ConsumeSymbol(')', tokenList) do + local st, ex = ParseExpr(scope) + if not st then return false, ex end + args[#args+1] = ex + if not tok:ConsumeSymbol(',', tokenList) then + if tok:ConsumeSymbol(')', tokenList) then + break + else + return false, GenerateError("`)` Expected.") + end + end + end + local nodeCall = {} + nodeCall.AstType = 'CallExpr' + nodeCall.Base = prim + nodeCall.Arguments = args + nodeCall.Tokens = tokenList + -- + prim = nodeCall + + elseif not onlyDotColon and tok:Is('String') then + --string call + local nodeCall = {} + nodeCall.AstType = 'StringCallExpr' + nodeCall.Base = prim + nodeCall.Arguments = { tok:Get(tokenList) } + nodeCall.Tokens = tokenList + -- + prim = nodeCall + + elseif not onlyDotColon and tok:IsSymbol('{') then + --table call + local st, ex = ParseSimpleExpr(scope) + -- FIX: ParseExpr(scope) parses the table AND and any following binary expressions. + -- We just want the table + if not st then return false, ex end + local nodeCall = {} + nodeCall.AstType = 'TableCallExpr' + nodeCall.Base = prim + nodeCall.Arguments = { ex } + nodeCall.Tokens = tokenList + -- + prim = nodeCall + + else + break + end + end + return true, prim + end + + + function ParseSimpleExpr(scope) + local tokenList = {} + + if tok:Is('Number') then + local nodeNum = {} + nodeNum.AstType = 'NumberExpr' + nodeNum.Value = tok:Get(tokenList) + nodeNum.Tokens = tokenList + return true, nodeNum + + elseif tok:Is('String') then + local nodeStr = {} + nodeStr.AstType = 'StringExpr' + nodeStr.Value = tok:Get(tokenList) + nodeStr.Tokens = tokenList + return true, nodeStr + + elseif tok:ConsumeKeyword('nil', tokenList) then + local nodeNil = {} + nodeNil.AstType = 'NilExpr' + nodeNil.Tokens = tokenList + return true, nodeNil + + elseif tok:IsKeyword('false') or tok:IsKeyword('true') then + local nodeBoolean = {} + nodeBoolean.AstType = 'BooleanExpr' + nodeBoolean.Value = (tok:Get(tokenList).Data == 'true') + nodeBoolean.Tokens = tokenList + return true, nodeBoolean + + elseif tok:ConsumeSymbol('...', tokenList) then + local nodeDots = {} + nodeDots.AstType = 'DotsExpr' + nodeDots.Tokens = tokenList + return true, nodeDots + + elseif tok:ConsumeSymbol('{', tokenList) then + local v = {} + v.AstType = 'ConstructorExpr' + v.EntryList = {} + -- + while true do + -- CANDRAN : read decorator(s) + local decorated = false + local decoratorChain = {} + while tok:ConsumeKeyword(candran.decorator) do + if not tok:Is('Ident') then + return false, GenerateError("Decorator name expected") + end + -- CANDRAN : get decorator name + local st, decorator = ParseExpr(scope) + if not st then return false, ex end + + table.insert(decoratorChain, decorator) + decorated = true + end + + if tok:IsSymbol('[', tokenList) then + --key + tok:Get(tokenList) + local st, key = ParseExpr(scope) + if not st then + return false, GenerateError("Key Expression Expected") + end + if not tok:ConsumeSymbol(']', tokenList) then + return false, GenerateError("`]` Expected") + end + if not tok:ConsumeSymbol('=', tokenList) then + return false, GenerateError("`=` Expected") + end + local st, value = ParseExpr(scope) + if not st then + return false, GenerateError("Value Expression Expected") + end + v.EntryList[#v.EntryList+1] = { + Type = 'Key'; + Key = key; + Value = value; + } + + elseif tok:Is('Ident') then + --value or key + local lookahead = tok:Peek(1) + if lookahead.Type == 'Symbol' and lookahead.Data == '=' then + --we are a key + local key = tok:Get(tokenList) + if not tok:ConsumeSymbol('=', tokenList) then + return false, GenerateError("`=` Expected") + end + local st, value = ParseExpr(scope) + if not st then + return false, GenerateError("Value Expression Expected") + end + v.EntryList[#v.EntryList+1] = { + Type = 'KeyString'; + Key = key.Data; + Value = value; + } + + else + --we are a value + local st, value = ParseExpr(scope) + if not st then + return false, GenerateError("Value Exected") + end + v.EntryList[#v.EntryList+1] = { + Type = 'Value'; + Value = value; + } + + end + elseif tok:ConsumeSymbol('}', tokenList) then + break + + else + --value + local st, value = ParseExpr(scope) + v.EntryList[#v.EntryList+1] = { + Type = 'Value'; + Value = value; + } + if not st then + return false, GenerateError("Value Expected") + end + end + + -- CANDRAN : decorate entry + if decorated then + v.EntryList[#v.EntryList].Decorated = true + v.EntryList[#v.EntryList].DecoratorChain = decoratorChain + end + + if tok:ConsumeSymbol(';', tokenList) or tok:ConsumeSymbol(',', tokenList) then + --all is good + elseif tok:ConsumeSymbol('}', tokenList) then + break + else + return false, GenerateError("`}` or table entry Expected") + end + end + v.Tokens = tokenList + return true, v + + elseif tok:ConsumeKeyword('function', tokenList) then + local st, func = ParseFunctionArgsAndBody(scope, tokenList) + if not st then return false, func end + -- + func.IsLocal = true + return true, func + + else + return ParseSuffixedExpr(scope) + end + end + + + local unops = lookupify{'-', 'not', '#'} + local unopprio = 8 + local priority = { + ['+'] = {6,6}; + ['-'] = {6,6}; + ['%'] = {7,7}; + ['/'] = {7,7}; + ['*'] = {7,7}; + ['^'] = {10,9}; + ['..'] = {5,4}; + ['=='] = {3,3}; + ['<'] = {3,3}; + ['<='] = {3,3}; + ['~='] = {3,3}; + ['>'] = {3,3}; + ['>='] = {3,3}; + ['and'] = {2,2}; + ['or'] = {1,1}; + } + function ParseSubExpr(scope, level) + --base item, possibly with unop prefix + local st, exp + if unops[tok:Peek().Data] then + local tokenList = {} + local op = tok:Get(tokenList).Data + st, exp = ParseSubExpr(scope, unopprio) + if not st then return false, exp end + local nodeEx = {} + nodeEx.AstType = 'UnopExpr' + nodeEx.Rhs = exp + nodeEx.Op = op + nodeEx.OperatorPrecedence = unopprio + nodeEx.Tokens = tokenList + exp = nodeEx + else + st, exp = ParseSimpleExpr(scope) + if not st then return false, exp end + end + + --next items in chain + while true do + local prio = priority[tok:Peek().Data] + if prio and prio[1] > level then + local tokenList = {} + local op = tok:Get(tokenList).Data + local st, rhs = ParseSubExpr(scope, prio[2]) + if not st then return false, rhs end + local nodeEx = {} + nodeEx.AstType = 'BinopExpr' + nodeEx.Lhs = exp + nodeEx.Op = op + nodeEx.OperatorPrecedence = prio[1] + nodeEx.Rhs = rhs + nodeEx.Tokens = tokenList + -- + exp = nodeEx + else + break + end + end + + return true, exp + end + + + ParseExpr = function(scope) + return ParseSubExpr(scope, 0) + end + + + local function ParseStatement(scope) + local stat = nil + local tokenList = {} + if tok:ConsumeKeyword('if', tokenList) then + --setup + local nodeIfStat = {} + nodeIfStat.AstType = 'IfStatement' + nodeIfStat.Clauses = {} + + --clauses + repeat + local st, nodeCond = ParseExpr(scope) + if not st then return false, nodeCond end + if not tok:ConsumeKeyword('then', tokenList) then + return false, GenerateError("`then` expected.") + end + local st, nodeBody = ParseStatementList(scope) + if not st then return false, nodeBody end + nodeIfStat.Clauses[#nodeIfStat.Clauses+1] = { + Condition = nodeCond; + Body = nodeBody; + } + until not tok:ConsumeKeyword('elseif', tokenList) + + --else clause + if tok:ConsumeKeyword('else', tokenList) then + local st, nodeBody = ParseStatementList(scope) + if not st then return false, nodeBody end + nodeIfStat.Clauses[#nodeIfStat.Clauses+1] = { + Body = nodeBody; + } + end + + --end + if not tok:ConsumeKeyword('end', tokenList) then + return false, GenerateError("`end` expected.") + end + + nodeIfStat.Tokens = tokenList + stat = nodeIfStat + + elseif tok:ConsumeKeyword('while', tokenList) then + --setup + local nodeWhileStat = {} + nodeWhileStat.AstType = 'WhileStatement' + + --condition + local st, nodeCond = ParseExpr(scope) + if not st then return false, nodeCond end + + --do + if not tok:ConsumeKeyword('do', tokenList) then + return false, GenerateError("`do` expected.") + end + + --body + local st, nodeBody = ParseStatementList(scope) + if not st then return false, nodeBody end + + --end + if not tok:ConsumeKeyword('end', tokenList) then + return false, GenerateError("`end` expected.") + end + + --return + nodeWhileStat.Condition = nodeCond + nodeWhileStat.Body = nodeBody + nodeWhileStat.Tokens = tokenList + stat = nodeWhileStat + + elseif tok:ConsumeKeyword('do', tokenList) then + --do block + local st, nodeBlock = ParseStatementList(scope) + if not st then return false, nodeBlock end + if not tok:ConsumeKeyword('end', tokenList) then + return false, GenerateError("`end` expected.") + end + + local nodeDoStat = {} + nodeDoStat.AstType = 'DoStatement' + nodeDoStat.Body = nodeBlock + nodeDoStat.Tokens = tokenList + stat = nodeDoStat + + elseif tok:ConsumeKeyword('for', tokenList) then + --for block + if not tok:Is('Ident') then + return false, GenerateError(" expected.") + end + local baseVarName = tok:Get(tokenList) + if tok:ConsumeSymbol('=', tokenList) then + --numeric for + local forScope = CreateScope(scope) + local forVar = forScope:CreateLocal(baseVarName.Data) + -- + local st, startEx = ParseExpr(scope) + if not st then return false, startEx end + if not tok:ConsumeSymbol(',', tokenList) then + return false, GenerateError("`,` Expected") + end + local st, endEx = ParseExpr(scope) + if not st then return false, endEx end + local st, stepEx; + if tok:ConsumeSymbol(',', tokenList) then + st, stepEx = ParseExpr(scope) + if not st then return false, stepEx end + end + if not tok:ConsumeKeyword('do', tokenList) then + return false, GenerateError("`do` expected") + end + -- + local st, body = ParseStatementList(forScope) + if not st then return false, body end + if not tok:ConsumeKeyword('end', tokenList) then + return false, GenerateError("`end` expected") + end + -- + local nodeFor = {} + nodeFor.AstType = 'NumericForStatement' + nodeFor.Scope = forScope + nodeFor.Variable = forVar + nodeFor.Start = startEx + nodeFor.End = endEx + nodeFor.Step = stepEx + nodeFor.Body = body + nodeFor.Tokens = tokenList + stat = nodeFor + else + --generic for + local forScope = CreateScope(scope) + -- + local varList = { forScope:CreateLocal(baseVarName.Data) } + while tok:ConsumeSymbol(',', tokenList) do + if not tok:Is('Ident') then + return false, GenerateError("for variable expected.") + end + varList[#varList+1] = forScope:CreateLocal(tok:Get(tokenList).Data) + end + if not tok:ConsumeKeyword('in', tokenList) then + return false, GenerateError("`in` expected.") + end + local generators = {} + local st, firstGenerator = ParseExpr(scope) + if not st then return false, firstGenerator end + generators[#generators+1] = firstGenerator + while tok:ConsumeSymbol(',', tokenList) do + local st, gen = ParseExpr(scope) + if not st then return false, gen end + generators[#generators+1] = gen + end + if not tok:ConsumeKeyword('do', tokenList) then + return false, GenerateError("`do` expected.") + end + local st, body = ParseStatementList(forScope) + if not st then return false, body end + if not tok:ConsumeKeyword('end', tokenList) then + return false, GenerateError("`end` expected.") + end + -- + local nodeFor = {} + nodeFor.AstType = 'GenericForStatement' + nodeFor.Scope = forScope + nodeFor.VariableList = varList + nodeFor.Generators = generators + nodeFor.Body = body + nodeFor.Tokens = tokenList + stat = nodeFor + end + + elseif tok:ConsumeKeyword('repeat', tokenList) then + local st, body = ParseStatementList(scope) + if not st then return false, body end + -- + if not tok:ConsumeKeyword('until', tokenList) then + return false, GenerateError("`until` expected.") + end + -- FIX: Used to parse in parent scope + -- Now parses in repeat scope + local st, cond = ParseExpr(body.Scope) + if not st then return false, cond end + -- + local nodeRepeat = {} + nodeRepeat.AstType = 'RepeatStatement' + nodeRepeat.Condition = cond + nodeRepeat.Body = body + nodeRepeat.Tokens = tokenList + stat = nodeRepeat + + -- CANDRAN : add decorator keyword (@) + elseif tok:ConsumeKeyword(candran.decorator, tokenList) then + if not tok:Is('Ident') then + return false, GenerateError("Decorator name expected") + end + + -- CANDRAN : get decorator name + local st, decorator = ParseExpr(scope) + if not st then return false, ex end + + -- CANDRAN : get decorated statement/decorator chain + local st, nodeStatement = ParseStatement(scope) + if not st then return false, nodeStatement end + + local nodeDecorator = {} + nodeDecorator.AstType = 'DecoratedStatement' + nodeDecorator.Decorator = decorator + nodeDecorator.Decorated = nodeStatement + nodeDecorator.Tokens = tokenList + stat = nodeDecorator + + elseif tok:ConsumeKeyword('function', tokenList) then + if not tok:Is('Ident') then + return false, GenerateError("Function name expected") + end + local st, name = ParseSuffixedExpr(scope, true) --true => only dots and colons + if not st then return false, name end + -- + local st, func = ParseFunctionArgsAndBody(scope, tokenList) + if not st then return false, func end + -- + func.IsLocal = false + func.Name = name + stat = func + + elseif tok:ConsumeKeyword('local', tokenList) then + if tok:Is('Ident') then + local varList = { tok:Get(tokenList).Data } + while tok:ConsumeSymbol(',', tokenList) do + if not tok:Is('Ident') then + return false, GenerateError("local var name expected") + end + varList[#varList+1] = tok:Get(tokenList).Data + end + + local initList = {} + if tok:ConsumeSymbol('=', tokenList) then + repeat + local st, ex = ParseExpr(scope) + if not st then return false, ex end + initList[#initList+1] = ex + until not tok:ConsumeSymbol(',', tokenList) + end + + --now patch var list + --we can't do this before getting the init list, because the init list does not + --have the locals themselves in scope. + for i, v in pairs(varList) do + varList[i] = scope:CreateLocal(v) + end + + local nodeLocal = {} + nodeLocal.AstType = 'LocalStatement' + nodeLocal.LocalList = varList + nodeLocal.InitList = initList + nodeLocal.Tokens = tokenList + -- + stat = nodeLocal + + elseif tok:ConsumeKeyword('function', tokenList) then + if not tok:Is('Ident') then + return false, GenerateError("Function name expected") + end + local name = tok:Get(tokenList).Data + local localVar = scope:CreateLocal(name) + -- + local st, func = ParseFunctionArgsAndBody(scope, tokenList) + if not st then return false, func end + -- + func.Name = localVar + func.IsLocal = true + stat = func + + else + return false, GenerateError("local var or function def expected") + end + + elseif tok:ConsumeSymbol('::', tokenList) then + if not tok:Is('Ident') then + return false, GenerateError('Label name expected') + end + local label = tok:Get(tokenList).Data + if not tok:ConsumeSymbol('::', tokenList) then + return false, GenerateError("`::` expected") + end + local nodeLabel = {} + nodeLabel.AstType = 'LabelStatement' + nodeLabel.Label = label + nodeLabel.Tokens = tokenList + stat = nodeLabel + + elseif tok:ConsumeKeyword('return', tokenList) then + local exList = {} + if not tok:IsKeyword('end') then + local st, firstEx = ParseExpr(scope) + if st then + exList[1] = firstEx + while tok:ConsumeSymbol(',', tokenList) do + local st, ex = ParseExpr(scope) + if not st then return false, ex end + exList[#exList+1] = ex + end + end + end + + local nodeReturn = {} + nodeReturn.AstType = 'ReturnStatement' + nodeReturn.Arguments = exList + nodeReturn.Tokens = tokenList + stat = nodeReturn + + elseif tok:ConsumeKeyword('break', tokenList) then + local nodeBreak = {} + nodeBreak.AstType = 'BreakStatement' + nodeBreak.Tokens = tokenList + stat = nodeBreak + + elseif tok:ConsumeKeyword('goto', tokenList) then + if not tok:Is('Ident') then + return false, GenerateError("Label expected") + end + local label = tok:Get(tokenList).Data + local nodeGoto = {} + nodeGoto.AstType = 'GotoStatement' + nodeGoto.Label = label + nodeGoto.Tokens = tokenList + stat = nodeGoto + + else + --statementParseExpr + local st, suffixed = ParseSuffixedExpr(scope) + if not st then return false, suffixed end + + --assignment or call? + -- CANDRAN : check if it is a Candran assignment symbol + local function isCandranAssignmentSymbol() + for _,s in ipairs(candran.assignment) do + if tok:IsSymbol(s) then + return true + end + end + return false + end + if tok:IsSymbol(',') or tok:IsSymbol('=') or isCandranAssignmentSymbol() then + --check that it was not parenthesized, making it not an lvalue + if (suffixed.ParenCount or 0) > 0 then + return false, GenerateError("Can not assign to parenthesized expression, is not an lvalue") + end + + --more processing needed + local lhs = { suffixed } + while tok:ConsumeSymbol(',', tokenList) do + local st, lhsPart = ParseSuffixedExpr(scope) + if not st then return false, lhsPart end + lhs[#lhs+1] = lhsPart + end + + --equals + -- CANDRAN : consume the Candran assignment symbol + local function consumeCandranAssignmentSymbol() + for _,s in ipairs(candran.assignment) do + if tok:ConsumeSymbol(s, tokenList) then + return true + end + end + return false + end + if not tok:ConsumeSymbol('=', tokenList) and not consumeCandranAssignmentSymbol() then + return false, GenerateError("`=` Expected.") + end + + --rhs + local rhs = {} + local st, firstRhs = ParseExpr(scope) + if not st then return false, firstRhs end + rhs[1] = firstRhs + while tok:ConsumeSymbol(',', tokenList) do + local st, rhsPart = ParseExpr(scope) + if not st then return false, rhsPart end + rhs[#rhs+1] = rhsPart + end + + --done + local nodeAssign = {} + nodeAssign.AstType = 'AssignmentStatement' + nodeAssign.Lhs = lhs + nodeAssign.Rhs = rhs + nodeAssign.Tokens = tokenList + stat = nodeAssign + + elseif suffixed.AstType == 'CallExpr' or + suffixed.AstType == 'TableCallExpr' or + suffixed.AstType == 'StringCallExpr' + then + --it's a call statement + local nodeCall = {} + nodeCall.AstType = 'CallStatement' + nodeCall.Expression = suffixed + nodeCall.Tokens = tokenList + stat = nodeCall + else + return false, GenerateError("Assignment Statement Expected") + end + end + + if tok:IsSymbol(';') then + stat.Semicolon = tok:Get( stat.Tokens ) + end + return true, stat + end + + + local statListCloseKeywords = lookupify{'end', 'else', 'elseif', 'until'} + + ParseStatementList = function(scope) + local nodeStatlist = {} + nodeStatlist.Scope = CreateScope(scope) + nodeStatlist.AstType = 'Statlist' + nodeStatlist.Body = { } + nodeStatlist.Tokens = { } + -- + --local stats = {} + -- + while not statListCloseKeywords[tok:Peek().Data] and not tok:IsEof() do + local st, nodeStatement = ParseStatement(nodeStatlist.Scope) + if not st then return false, nodeStatement end + --stats[#stats+1] = nodeStatement + nodeStatlist.Body[#nodeStatlist.Body + 1] = nodeStatement + end + + if tok:IsEof() then + local nodeEof = {} + nodeEof.AstType = 'Eof' + nodeEof.Tokens = { tok:Get() } + nodeStatlist.Body[#nodeStatlist.Body + 1] = nodeEof + end + + -- + --nodeStatlist.Body = stats + return true, nodeStatlist + end + + + local function mainfunc() + local topScope = CreateScope() + return ParseStatementList(topScope) + end + + local st, main = mainfunc() + --print("Last Token: "..PrintTable(tok:Peek())) + return st, main +end + +return { LexLua = LexLua, ParseLua = ParseLua } + \ No newline at end of file diff --git a/lib/LuaMinify/ParseLua.lua b/lib/LuaMinify/ParseLua.lua new file mode 100644 index 0000000..3aa3e14 --- /dev/null +++ b/lib/LuaMinify/ParseLua.lua @@ -0,0 +1,1411 @@ + +-- +-- ParseLua.lua +-- +-- The main lua parser and lexer. +-- LexLua returns a Lua token stream, with tokens that preserve +-- all whitespace formatting information. +-- ParseLua returns an AST, internally relying on LexLua. +-- + +require'Strict' + +local util = require'Util' +local lookupify = util.lookupify + +local WhiteChars = lookupify{' ', '\n', '\t', '\r'} +local EscapeLookup = {['\r'] = '\\r', ['\n'] = '\\n', ['\t'] = '\\t', ['"'] = '\\"', ["'"] = "\\'"} +local LowerChars = lookupify{'a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', + 'j', 'k', 'l', 'm', 'n', 'o', 'p', 'q', 'r', + 's', 't', 'u', 'v', 'w', 'x', 'y', 'z'} +local UpperChars = lookupify{'A', 'B', 'C', 'D', 'E', 'F', 'G', 'H', 'I', + 'J', 'K', 'L', 'M', 'N', 'O', 'P', 'Q', 'R', + 'S', 'T', 'U', 'V', 'W', 'X', 'Y', 'Z'} +local Digits = lookupify{'0', '1', '2', '3', '4', '5', '6', '7', '8', '9'} +local HexDigits = lookupify{'0', '1', '2', '3', '4', '5', '6', '7', '8', '9', + 'A', 'a', 'B', 'b', 'C', 'c', 'D', 'd', 'E', 'e', 'F', 'f'} + +local Symbols = lookupify{'+', '-', '*', '/', '^', '%', ',', '{', '}', '[', ']', '(', ')', ';', '#'} +local Scope = require'Scope' + +local Keywords = lookupify{ + 'and', 'break', 'do', 'else', 'elseif', + 'end', 'false', 'for', 'function', 'goto', 'if', + 'in', 'local', 'nil', 'not', 'or', 'repeat', + 'return', 'then', 'true', 'until', 'while', +}; + +local function LexLua(src) + --token dump + local tokens = {} + + local st, err = pcall(function() + --line / char / pointer tracking + local p = 1 + local line = 1 + local char = 1 + + --get / peek functions + local function get() + local c = src:sub(p,p) + if c == '\n' then + char = 1 + line = line + 1 + else + char = char + 1 + end + p = p + 1 + return c + end + local function peek(n) + n = n or 0 + return src:sub(p+n,p+n) + end + local function consume(chars) + local c = peek() + for i = 1, #chars do + if c == chars:sub(i,i) then return get() end + end + end + + --shared stuff + local function generateError(err) + return error(">> :"..line..":"..char..": "..err, 0) + end + + local function tryGetLongString() + local start = p + if peek() == '[' then + local equalsCount = 0 + local depth = 1 + while peek(equalsCount+1) == '=' do + equalsCount = equalsCount + 1 + end + if peek(equalsCount+1) == '[' then + --start parsing the string. Strip the starting bit + for _ = 0, equalsCount+1 do get() end + + --get the contents + local contentStart = p + while true do + --check for eof + if peek() == '' then + generateError("Expected `]"..string.rep('=', equalsCount).."]` near .", 3) + end + + --check for the end + local foundEnd = true + if peek() == ']' then + for i = 1, equalsCount do + if peek(i) ~= '=' then foundEnd = false end + end + if peek(equalsCount+1) ~= ']' then + foundEnd = false + end + else + if peek() == '[' then + -- is there an embedded long string? + local embedded = true + for i = 1, equalsCount do + if peek(i) ~= '=' then + embedded = false + break + end + end + if peek(equalsCount + 1) == '[' and embedded then + -- oh look, there was + depth = depth + 1 + for i = 1, (equalsCount + 2) do + get() + end + end + end + foundEnd = false + end + -- + if foundEnd then + depth = depth - 1 + if depth == 0 then + break + else + for i = 1, equalsCount + 2 do + get() + end + end + else + get() + end + end + + --get the interior string + local contentString = src:sub(contentStart, p-1) + + --found the end. Get rid of the trailing bit + for i = 0, equalsCount+1 do get() end + + --get the exterior string + local longString = src:sub(start, p-1) + + --return the stuff + return contentString, longString + else + return nil + end + else + return nil + end + end + + --main token emitting loop + while true do + --get leading whitespace. The leading whitespace will include any comments + --preceding the token. This prevents the parser needing to deal with comments + --separately. + local leading = { } + local leadingWhite = '' + local longStr = false + while true do + local c = peek() + if c == '#' and peek(1) == '!' and line == 1 then + -- #! shebang for linux scripts + get() + get() + leadingWhite = "#!" + while peek() ~= '\n' and peek() ~= '' do + leadingWhite = leadingWhite .. get() + end + local token = { + Type = 'Comment', + CommentType = 'Shebang', + Data = leadingWhite, + Line = line, + Char = char + } + token.Print = function() + return "<"..(token.Type .. string.rep(' ', 7-#token.Type)).." "..(token.Data or '').." >" + end + leadingWhite = "" + table.insert(leading, token) + end + if c == ' ' or c == '\t' then + --whitespace + --leadingWhite = leadingWhite..get() + local c2 = get() -- ignore whitespace + table.insert(leading, { Type = 'Whitespace', Line = line, Char = char, Data = c2 }) + elseif c == '\n' or c == '\r' then + local nl = get() + if leadingWhite ~= "" then + local token = { + Type = 'Comment', + CommentType = longStr and 'LongComment' or 'Comment', + Data = leadingWhite, + Line = line, + Char = char, + } + token.Print = function() + return "<"..(token.Type .. string.rep(' ', 7-#token.Type)).." "..(token.Data or '').." >" + end + table.insert(leading, token) + leadingWhite = "" + end + table.insert(leading, { Type = 'Whitespace', Line = line, Char = char, Data = nl }) + elseif c == '-' and peek(1) == '-' then + --comment + get() + get() + leadingWhite = leadingWhite .. '--' + local _, wholeText = tryGetLongString() + if wholeText then + leadingWhite = leadingWhite..wholeText + longStr = true + else + while peek() ~= '\n' and peek() ~= '' do + leadingWhite = leadingWhite..get() + end + end + else + break + end + end + if leadingWhite ~= "" then + local token = { + Type = 'Comment', + CommentType = longStr and 'LongComment' or 'Com mnment', + Data = leadingWhite, + Line = line, + Char = char, + } + token.Print = function() + return "<"..(token.Type .. string.rep(' ', 7-#token.Type)).." "..(token.Data or '').." >" + end + table.insert(leading, token) + end + + --get the initial char + local thisLine = line + local thisChar = char + local errorAt = ":"..line..":"..char..":> " + local c = peek() + + --symbol to emit + local toEmit = nil + + --branch on type + if c == '' then + --eof + toEmit = { Type = 'Eof' } + + elseif UpperChars[c] or LowerChars[c] or c == '_' then + --ident or keyword + local start = p + repeat + get() + c = peek() + until not (UpperChars[c] or LowerChars[c] or Digits[c] or c == '_') + local dat = src:sub(start, p-1) + if Keywords[dat] then + toEmit = {Type = 'Keyword', Data = dat} + else + toEmit = {Type = 'Ident', Data = dat} + end + + elseif Digits[c] or (peek() == '.' and Digits[peek(1)]) then + --number const + local start = p + if c == '0' and peek(1) == 'x' then + get();get() + while HexDigits[peek()] do get() end + if consume('Pp') then + consume('+-') + while Digits[peek()] do get() end + end + else + while Digits[peek()] do get() end + if consume('.') then + while Digits[peek()] do get() end + end + if consume('Ee') then + consume('+-') + while Digits[peek()] do get() end + end + end + toEmit = {Type = 'Number', Data = src:sub(start, p-1)} + + elseif c == '\'' or c == '\"' then + local start = p + --string const + local delim = get() + local contentStart = p + while true do + local c = get() + if c == '\\' then + get() --get the escape char + elseif c == delim then + break + elseif c == '' then + generateError("Unfinished string near ") + end + end + local content = src:sub(contentStart, p-2) + local constant = src:sub(start, p-1) + toEmit = {Type = 'String', Data = constant, Constant = content} + + elseif c == '[' then + local content, wholetext = tryGetLongString() + if wholetext then + toEmit = {Type = 'String', Data = wholetext, Constant = content} + else + get() + toEmit = {Type = 'Symbol', Data = '['} + end + + elseif consume('>=<') then + if consume('=') then + toEmit = {Type = 'Symbol', Data = c..'='} + else + toEmit = {Type = 'Symbol', Data = c} + end + + elseif consume('~') then + if consume('=') then + toEmit = {Type = 'Symbol', Data = '~='} + else + generateError("Unexpected symbol `~` in source.", 2) + end + + elseif consume('.') then + if consume('.') then + if consume('.') then + toEmit = {Type = 'Symbol', Data = '...'} + else + toEmit = {Type = 'Symbol', Data = '..'} + end + else + toEmit = {Type = 'Symbol', Data = '.'} + end + + elseif consume(':') then + if consume(':') then + toEmit = {Type = 'Symbol', Data = '::'} + else + toEmit = {Type = 'Symbol', Data = ':'} + end + + elseif Symbols[c] then + get() + toEmit = {Type = 'Symbol', Data = c} + + else + local contents, all = tryGetLongString() + if contents then + toEmit = {Type = 'String', Data = all, Constant = contents} + else + generateError("Unexpected Symbol `"..c.."` in source.", 2) + end + end + + --add the emitted symbol, after adding some common data + toEmit.LeadingWhite = leading -- table of leading whitespace/comments + --for k, tok in pairs(leading) do + -- tokens[#tokens + 1] = tok + --end + + toEmit.Line = thisLine + toEmit.Char = thisChar + toEmit.Print = function() + return "<"..(toEmit.Type..string.rep(' ', 7-#toEmit.Type)).." "..(toEmit.Data or '').." >" + end + tokens[#tokens+1] = toEmit + + --halt after eof has been emitted + if toEmit.Type == 'Eof' then break end + end + end) + if not st then + return false, err + end + + --public interface: + local tok = {} + local savedP = {} + local p = 1 + + function tok:getp() + return p + end + + function tok:setp(n) + p = n + end + + function tok:getTokenList() + return tokens + end + + --getters + function tok:Peek(n) + n = n or 0 + return tokens[math.min(#tokens, p+n)] + end + function tok:Get(tokenList) + local t = tokens[p] + p = math.min(p + 1, #tokens) + if tokenList then + table.insert(tokenList, t) + end + return t + end + function tok:Is(t) + return tok:Peek().Type == t + end + + --save / restore points in the stream + function tok:Save() + savedP[#savedP+1] = p + end + function tok:Commit() + savedP[#savedP] = nil + end + function tok:Restore() + p = savedP[#savedP] + savedP[#savedP] = nil + end + + --either return a symbol if there is one, or return true if the requested + --symbol was gotten. + function tok:ConsumeSymbol(symb, tokenList) + local t = self:Peek() + if t.Type == 'Symbol' then + if symb then + if t.Data == symb then + self:Get(tokenList) + return true + else + return nil + end + else + self:Get(tokenList) + return t + end + else + return nil + end + end + + function tok:ConsumeKeyword(kw, tokenList) + local t = self:Peek() + if t.Type == 'Keyword' and t.Data == kw then + self:Get(tokenList) + return true + else + return nil + end + end + + function tok:IsKeyword(kw) + local t = tok:Peek() + return t.Type == 'Keyword' and t.Data == kw + end + + function tok:IsSymbol(s) + local t = tok:Peek() + return t.Type == 'Symbol' and t.Data == s + end + + function tok:IsEof() + return tok:Peek().Type == 'Eof' + end + + return true, tok +end + + +local function ParseLua(src) + local st, tok + if type(src) ~= 'table' then + st, tok = LexLua(src) + else + st, tok = true, src + end + if not st then + return false, tok + end + -- + local function GenerateError(msg) + local err = ">> :"..tok:Peek().Line..":"..tok:Peek().Char..": "..msg.."\n" + --find the line + local lineNum = 0 + if type(src) == 'string' then + for line in src:gmatch("[^\n]*\n?") do + if line:sub(-1,-1) == '\n' then line = line:sub(1,-2) end + lineNum = lineNum+1 + if lineNum == tok:Peek().Line then + err = err..">> `"..line:gsub('\t',' ').."`\n" + for i = 1, tok:Peek().Char do + local c = line:sub(i,i) + if c == '\t' then + err = err..' ' + else + err = err..' ' + end + end + err = err.." ^^^^" + break + end + end + end + return err + end + -- + local VarUid = 0 + -- No longer needed: handled in Scopes now local GlobalVarGetMap = {} + local VarDigits = {'_', 'a', 'b', 'c', 'd'} + local function CreateScope(parent) + --[[ + local scope = {} + scope.Parent = parent + scope.LocalList = {} + scope.LocalMap = {} + + function scope:ObfuscateVariables() + for _, var in pairs(scope.LocalList) do + local id = "" + repeat + local chars = "QWERTYUIOPASDFGHJKLZXCVBNMqwertyuioplkjhgfdsazxcvbnm_" + local chars2 = "QWERTYUIOPASDFGHJKLZXCVBNMqwertyuioplkjhgfdsazxcvbnm_1234567890" + local n = math.random(1, #chars) + id = id .. chars:sub(n, n) + for i = 1, math.random(0,20) do + local n = math.random(1, #chars2) + id = id .. chars2:sub(n, n) + end + until not GlobalVarGetMap[id] and not parent:GetLocal(id) and not scope.LocalMap[id] + var.Name = id + scope.LocalMap[id] = var + end + end + + scope.RenameVars = scope.ObfuscateVariables + + -- Renames a variable from this scope and down. + -- Does not rename global variables. + function scope:RenameVariable(old, newName) + if type(old) == "table" then -- its (theoretically) an AstNode variable + old = old.Name + end + for _, var in pairs(scope.LocalList) do + if var.Name == old then + var.Name = newName + scope.LocalMap[newName] = var + end + end + end + + function scope:GetLocal(name) + --first, try to get my variable + local my = scope.LocalMap[name] + if my then return my end + + --next, try parent + if scope.Parent then + local par = scope.Parent:GetLocal(name) + if par then return par end + end + + return nil + end + + function scope:CreateLocal(name) + --create my own var + local my = {} + my.Scope = scope + my.Name = name + my.CanRename = true + -- + scope.LocalList[#scope.LocalList+1] = my + scope.LocalMap[name] = my + -- + return my + end]] + local scope = Scope:new(parent) + scope.RenameVars = scope.ObfuscateLocals + scope.ObfuscateVariables = scope.ObfuscateLocals + scope.Print = function() return "" end + return scope + end + + local ParseExpr + local ParseStatementList + local ParseSimpleExpr, + ParseSubExpr, + ParsePrimaryExpr, + ParseSuffixedExpr + + local function ParseFunctionArgsAndBody(scope, tokenList) + local funcScope = CreateScope(scope) + if not tok:ConsumeSymbol('(', tokenList) then + return false, GenerateError("`(` expected.") + end + + --arg list + local argList = {} + local isVarArg = false + while not tok:ConsumeSymbol(')', tokenList) do + if tok:Is('Ident') then + local arg = funcScope:CreateLocal(tok:Get(tokenList).Data) + argList[#argList+1] = arg + if not tok:ConsumeSymbol(',', tokenList) then + if tok:ConsumeSymbol(')', tokenList) then + break + else + return false, GenerateError("`)` expected.") + end + end + elseif tok:ConsumeSymbol('...', tokenList) then + isVarArg = true + if not tok:ConsumeSymbol(')', tokenList) then + return false, GenerateError("`...` must be the last argument of a function.") + end + break + else + return false, GenerateError("Argument name or `...` expected") + end + end + + --body + local st, body = ParseStatementList(funcScope) + if not st then return false, body end + + --end + if not tok:ConsumeKeyword('end', tokenList) then + return false, GenerateError("`end` expected after function body") + end + local nodeFunc = {} + nodeFunc.AstType = 'Function' + nodeFunc.Scope = funcScope + nodeFunc.Arguments = argList + nodeFunc.Body = body + nodeFunc.VarArg = isVarArg + nodeFunc.Tokens = tokenList + -- + return true, nodeFunc + end + + + function ParsePrimaryExpr(scope) + local tokenList = {} + + if tok:ConsumeSymbol('(', tokenList) then + local st, ex = ParseExpr(scope) + if not st then return false, ex end + if not tok:ConsumeSymbol(')', tokenList) then + return false, GenerateError("`)` Expected.") + end + if false then + --save the information about parenthesized expressions somewhere + ex.ParenCount = (ex.ParenCount or 0) + 1 + return true, ex + else + local parensExp = {} + parensExp.AstType = 'Parentheses' + parensExp.Inner = ex + parensExp.Tokens = tokenList + return true, parensExp + end + + elseif tok:Is('Ident') then + local id = tok:Get(tokenList) + local var = scope:GetLocal(id.Data) + if not var then + var = scope:GetGlobal(id.Data) + if not var then + var = scope:CreateGlobal(id.Data) + else + var.References = var.References + 1 + end + else + var.References = var.References + 1 + end + -- + local nodePrimExp = {} + nodePrimExp.AstType = 'VarExpr' + nodePrimExp.Name = id.Data + nodePrimExp.Variable = var + nodePrimExp.Tokens = tokenList + -- + return true, nodePrimExp + else + return false, GenerateError("primary expression expected") + end + end + + function ParseSuffixedExpr(scope, onlyDotColon) + --base primary expression + local st, prim = ParsePrimaryExpr(scope) + if not st then return false, prim end + -- + while true do + local tokenList = {} + + if tok:IsSymbol('.') or tok:IsSymbol(':') then + local symb = tok:Get(tokenList).Data + if not tok:Is('Ident') then + return false, GenerateError(" expected.") + end + local id = tok:Get(tokenList) + local nodeIndex = {} + nodeIndex.AstType = 'MemberExpr' + nodeIndex.Base = prim + nodeIndex.Indexer = symb + nodeIndex.Ident = id + nodeIndex.Tokens = tokenList + -- + prim = nodeIndex + + elseif not onlyDotColon and tok:ConsumeSymbol('[', tokenList) then + local st, ex = ParseExpr(scope) + if not st then return false, ex end + if not tok:ConsumeSymbol(']', tokenList) then + return false, GenerateError("`]` expected.") + end + local nodeIndex = {} + nodeIndex.AstType = 'IndexExpr' + nodeIndex.Base = prim + nodeIndex.Index = ex + nodeIndex.Tokens = tokenList + -- + prim = nodeIndex + + elseif not onlyDotColon and tok:ConsumeSymbol('(', tokenList) then + local args = {} + while not tok:ConsumeSymbol(')', tokenList) do + local st, ex = ParseExpr(scope) + if not st then return false, ex end + args[#args+1] = ex + if not tok:ConsumeSymbol(',', tokenList) then + if tok:ConsumeSymbol(')', tokenList) then + break + else + return false, GenerateError("`)` Expected.") + end + end + end + local nodeCall = {} + nodeCall.AstType = 'CallExpr' + nodeCall.Base = prim + nodeCall.Arguments = args + nodeCall.Tokens = tokenList + -- + prim = nodeCall + + elseif not onlyDotColon and tok:Is('String') then + --string call + local nodeCall = {} + nodeCall.AstType = 'StringCallExpr' + nodeCall.Base = prim + nodeCall.Arguments = { tok:Get(tokenList) } + nodeCall.Tokens = tokenList + -- + prim = nodeCall + + elseif not onlyDotColon and tok:IsSymbol('{') then + --table call + local st, ex = ParseSimpleExpr(scope) + -- FIX: ParseExpr(scope) parses the table AND and any following binary expressions. + -- We just want the table + if not st then return false, ex end + local nodeCall = {} + nodeCall.AstType = 'TableCallExpr' + nodeCall.Base = prim + nodeCall.Arguments = { ex } + nodeCall.Tokens = tokenList + -- + prim = nodeCall + + else + break + end + end + return true, prim + end + + + function ParseSimpleExpr(scope) + local tokenList = {} + + if tok:Is('Number') then + local nodeNum = {} + nodeNum.AstType = 'NumberExpr' + nodeNum.Value = tok:Get(tokenList) + nodeNum.Tokens = tokenList + return true, nodeNum + + elseif tok:Is('String') then + local nodeStr = {} + nodeStr.AstType = 'StringExpr' + nodeStr.Value = tok:Get(tokenList) + nodeStr.Tokens = tokenList + return true, nodeStr + + elseif tok:ConsumeKeyword('nil', tokenList) then + local nodeNil = {} + nodeNil.AstType = 'NilExpr' + nodeNil.Tokens = tokenList + return true, nodeNil + + elseif tok:IsKeyword('false') or tok:IsKeyword('true') then + local nodeBoolean = {} + nodeBoolean.AstType = 'BooleanExpr' + nodeBoolean.Value = (tok:Get(tokenList).Data == 'true') + nodeBoolean.Tokens = tokenList + return true, nodeBoolean + + elseif tok:ConsumeSymbol('...', tokenList) then + local nodeDots = {} + nodeDots.AstType = 'DotsExpr' + nodeDots.Tokens = tokenList + return true, nodeDots + + elseif tok:ConsumeSymbol('{', tokenList) then + local v = {} + v.AstType = 'ConstructorExpr' + v.EntryList = {} + -- + while true do + if tok:IsSymbol('[', tokenList) then + --key + tok:Get(tokenList) + local st, key = ParseExpr(scope) + if not st then + return false, GenerateError("Key Expression Expected") + end + if not tok:ConsumeSymbol(']', tokenList) then + return false, GenerateError("`]` Expected") + end + if not tok:ConsumeSymbol('=', tokenList) then + return false, GenerateError("`=` Expected") + end + local st, value = ParseExpr(scope) + if not st then + return false, GenerateError("Value Expression Expected") + end + v.EntryList[#v.EntryList+1] = { + Type = 'Key'; + Key = key; + Value = value; + } + + elseif tok:Is('Ident') then + --value or key + local lookahead = tok:Peek(1) + if lookahead.Type == 'Symbol' and lookahead.Data == '=' then + --we are a key + local key = tok:Get(tokenList) + if not tok:ConsumeSymbol('=', tokenList) then + return false, GenerateError("`=` Expected") + end + local st, value = ParseExpr(scope) + if not st then + return false, GenerateError("Value Expression Expected") + end + v.EntryList[#v.EntryList+1] = { + Type = 'KeyString'; + Key = key.Data; + Value = value; + } + + else + --we are a value + local st, value = ParseExpr(scope) + if not st then + return false, GenerateError("Value Exected") + end + v.EntryList[#v.EntryList+1] = { + Type = 'Value'; + Value = value; + } + + end + elseif tok:ConsumeSymbol('}', tokenList) then + break + + else + --value + local st, value = ParseExpr(scope) + v.EntryList[#v.EntryList+1] = { + Type = 'Value'; + Value = value; + } + if not st then + return false, GenerateError("Value Expected") + end + end + + if tok:ConsumeSymbol(';', tokenList) or tok:ConsumeSymbol(',', tokenList) then + --all is good + elseif tok:ConsumeSymbol('}', tokenList) then + break + else + return false, GenerateError("`}` or table entry Expected") + end + end + v.Tokens = tokenList + return true, v + + elseif tok:ConsumeKeyword('function', tokenList) then + local st, func = ParseFunctionArgsAndBody(scope, tokenList) + if not st then return false, func end + -- + func.IsLocal = true + return true, func + + else + return ParseSuffixedExpr(scope) + end + end + + + local unops = lookupify{'-', 'not', '#'} + local unopprio = 8 + local priority = { + ['+'] = {6,6}; + ['-'] = {6,6}; + ['%'] = {7,7}; + ['/'] = {7,7}; + ['*'] = {7,7}; + ['^'] = {10,9}; + ['..'] = {5,4}; + ['=='] = {3,3}; + ['<'] = {3,3}; + ['<='] = {3,3}; + ['~='] = {3,3}; + ['>'] = {3,3}; + ['>='] = {3,3}; + ['and'] = {2,2}; + ['or'] = {1,1}; + } + function ParseSubExpr(scope, level) + --base item, possibly with unop prefix + local st, exp + if unops[tok:Peek().Data] then + local tokenList = {} + local op = tok:Get(tokenList).Data + st, exp = ParseSubExpr(scope, unopprio) + if not st then return false, exp end + local nodeEx = {} + nodeEx.AstType = 'UnopExpr' + nodeEx.Rhs = exp + nodeEx.Op = op + nodeEx.OperatorPrecedence = unopprio + nodeEx.Tokens = tokenList + exp = nodeEx + else + st, exp = ParseSimpleExpr(scope) + if not st then return false, exp end + end + + --next items in chain + while true do + local prio = priority[tok:Peek().Data] + if prio and prio[1] > level then + local tokenList = {} + local op = tok:Get(tokenList).Data + local st, rhs = ParseSubExpr(scope, prio[2]) + if not st then return false, rhs end + local nodeEx = {} + nodeEx.AstType = 'BinopExpr' + nodeEx.Lhs = exp + nodeEx.Op = op + nodeEx.OperatorPrecedence = prio[1] + nodeEx.Rhs = rhs + nodeEx.Tokens = tokenList + -- + exp = nodeEx + else + break + end + end + + return true, exp + end + + + ParseExpr = function(scope) + return ParseSubExpr(scope, 0) + end + + + local function ParseStatement(scope) + local stat = nil + local tokenList = {} + if tok:ConsumeKeyword('if', tokenList) then + --setup + local nodeIfStat = {} + nodeIfStat.AstType = 'IfStatement' + nodeIfStat.Clauses = {} + + --clauses + repeat + local st, nodeCond = ParseExpr(scope) + if not st then return false, nodeCond end + if not tok:ConsumeKeyword('then', tokenList) then + return false, GenerateError("`then` expected.") + end + local st, nodeBody = ParseStatementList(scope) + if not st then return false, nodeBody end + nodeIfStat.Clauses[#nodeIfStat.Clauses+1] = { + Condition = nodeCond; + Body = nodeBody; + } + until not tok:ConsumeKeyword('elseif', tokenList) + + --else clause + if tok:ConsumeKeyword('else', tokenList) then + local st, nodeBody = ParseStatementList(scope) + if not st then return false, nodeBody end + nodeIfStat.Clauses[#nodeIfStat.Clauses+1] = { + Body = nodeBody; + } + end + + --end + if not tok:ConsumeKeyword('end', tokenList) then + return false, GenerateError("`end` expected.") + end + + nodeIfStat.Tokens = tokenList + stat = nodeIfStat + + elseif tok:ConsumeKeyword('while', tokenList) then + --setup + local nodeWhileStat = {} + nodeWhileStat.AstType = 'WhileStatement' + + --condition + local st, nodeCond = ParseExpr(scope) + if not st then return false, nodeCond end + + --do + if not tok:ConsumeKeyword('do', tokenList) then + return false, GenerateError("`do` expected.") + end + + --body + local st, nodeBody = ParseStatementList(scope) + if not st then return false, nodeBody end + + --end + if not tok:ConsumeKeyword('end', tokenList) then + return false, GenerateError("`end` expected.") + end + + --return + nodeWhileStat.Condition = nodeCond + nodeWhileStat.Body = nodeBody + nodeWhileStat.Tokens = tokenList + stat = nodeWhileStat + + elseif tok:ConsumeKeyword('do', tokenList) then + --do block + local st, nodeBlock = ParseStatementList(scope) + if not st then return false, nodeBlock end + if not tok:ConsumeKeyword('end', tokenList) then + return false, GenerateError("`end` expected.") + end + + local nodeDoStat = {} + nodeDoStat.AstType = 'DoStatement' + nodeDoStat.Body = nodeBlock + nodeDoStat.Tokens = tokenList + stat = nodeDoStat + + elseif tok:ConsumeKeyword('for', tokenList) then + --for block + if not tok:Is('Ident') then + return false, GenerateError(" expected.") + end + local baseVarName = tok:Get(tokenList) + if tok:ConsumeSymbol('=', tokenList) then + --numeric for + local forScope = CreateScope(scope) + local forVar = forScope:CreateLocal(baseVarName.Data) + -- + local st, startEx = ParseExpr(scope) + if not st then return false, startEx end + if not tok:ConsumeSymbol(',', tokenList) then + return false, GenerateError("`,` Expected") + end + local st, endEx = ParseExpr(scope) + if not st then return false, endEx end + local st, stepEx; + if tok:ConsumeSymbol(',', tokenList) then + st, stepEx = ParseExpr(scope) + if not st then return false, stepEx end + end + if not tok:ConsumeKeyword('do', tokenList) then + return false, GenerateError("`do` expected") + end + -- + local st, body = ParseStatementList(forScope) + if not st then return false, body end + if not tok:ConsumeKeyword('end', tokenList) then + return false, GenerateError("`end` expected") + end + -- + local nodeFor = {} + nodeFor.AstType = 'NumericForStatement' + nodeFor.Scope = forScope + nodeFor.Variable = forVar + nodeFor.Start = startEx + nodeFor.End = endEx + nodeFor.Step = stepEx + nodeFor.Body = body + nodeFor.Tokens = tokenList + stat = nodeFor + else + --generic for + local forScope = CreateScope(scope) + -- + local varList = { forScope:CreateLocal(baseVarName.Data) } + while tok:ConsumeSymbol(',', tokenList) do + if not tok:Is('Ident') then + return false, GenerateError("for variable expected.") + end + varList[#varList+1] = forScope:CreateLocal(tok:Get(tokenList).Data) + end + if not tok:ConsumeKeyword('in', tokenList) then + return false, GenerateError("`in` expected.") + end + local generators = {} + local st, firstGenerator = ParseExpr(scope) + if not st then return false, firstGenerator end + generators[#generators+1] = firstGenerator + while tok:ConsumeSymbol(',', tokenList) do + local st, gen = ParseExpr(scope) + if not st then return false, gen end + generators[#generators+1] = gen + end + if not tok:ConsumeKeyword('do', tokenList) then + return false, GenerateError("`do` expected.") + end + local st, body = ParseStatementList(forScope) + if not st then return false, body end + if not tok:ConsumeKeyword('end', tokenList) then + return false, GenerateError("`end` expected.") + end + -- + local nodeFor = {} + nodeFor.AstType = 'GenericForStatement' + nodeFor.Scope = forScope + nodeFor.VariableList = varList + nodeFor.Generators = generators + nodeFor.Body = body + nodeFor.Tokens = tokenList + stat = nodeFor + end + + elseif tok:ConsumeKeyword('repeat', tokenList) then + local st, body = ParseStatementList(scope) + if not st then return false, body end + -- + if not tok:ConsumeKeyword('until', tokenList) then + return false, GenerateError("`until` expected.") + end + -- FIX: Used to parse in parent scope + -- Now parses in repeat scope + local st, cond = ParseExpr(body.Scope) + if not st then return false, cond end + -- + local nodeRepeat = {} + nodeRepeat.AstType = 'RepeatStatement' + nodeRepeat.Condition = cond + nodeRepeat.Body = body + nodeRepeat.Tokens = tokenList + stat = nodeRepeat + + elseif tok:ConsumeKeyword('function', tokenList) then + if not tok:Is('Ident') then + return false, GenerateError("Function name expected") + end + local st, name = ParseSuffixedExpr(scope, true) --true => only dots and colons + if not st then return false, name end + -- + local st, func = ParseFunctionArgsAndBody(scope, tokenList) + if not st then return false, func end + -- + func.IsLocal = false + func.Name = name + stat = func + + elseif tok:ConsumeKeyword('local', tokenList) then + if tok:Is('Ident') then + local varList = { tok:Get(tokenList).Data } + while tok:ConsumeSymbol(',', tokenList) do + if not tok:Is('Ident') then + return false, GenerateError("local var name expected") + end + varList[#varList+1] = tok:Get(tokenList).Data + end + + local initList = {} + if tok:ConsumeSymbol('=', tokenList) then + repeat + local st, ex = ParseExpr(scope) + if not st then return false, ex end + initList[#initList+1] = ex + until not tok:ConsumeSymbol(',', tokenList) + end + + --now patch var list + --we can't do this before getting the init list, because the init list does not + --have the locals themselves in scope. + for i, v in pairs(varList) do + varList[i] = scope:CreateLocal(v) + end + + local nodeLocal = {} + nodeLocal.AstType = 'LocalStatement' + nodeLocal.LocalList = varList + nodeLocal.InitList = initList + nodeLocal.Tokens = tokenList + -- + stat = nodeLocal + + elseif tok:ConsumeKeyword('function', tokenList) then + if not tok:Is('Ident') then + return false, GenerateError("Function name expected") + end + local name = tok:Get(tokenList).Data + local localVar = scope:CreateLocal(name) + -- + local st, func = ParseFunctionArgsAndBody(scope, tokenList) + if not st then return false, func end + -- + func.Name = localVar + func.IsLocal = true + stat = func + + else + return false, GenerateError("local var or function def expected") + end + + elseif tok:ConsumeSymbol('::', tokenList) then + if not tok:Is('Ident') then + return false, GenerateError('Label name expected') + end + local label = tok:Get(tokenList).Data + if not tok:ConsumeSymbol('::', tokenList) then + return false, GenerateError("`::` expected") + end + local nodeLabel = {} + nodeLabel.AstType = 'LabelStatement' + nodeLabel.Label = label + nodeLabel.Tokens = tokenList + stat = nodeLabel + + elseif tok:ConsumeKeyword('return', tokenList) then + local exList = {} + if not tok:IsKeyword('end') then + local st, firstEx = ParseExpr(scope) + if st then + exList[1] = firstEx + while tok:ConsumeSymbol(',', tokenList) do + local st, ex = ParseExpr(scope) + if not st then return false, ex end + exList[#exList+1] = ex + end + end + end + + local nodeReturn = {} + nodeReturn.AstType = 'ReturnStatement' + nodeReturn.Arguments = exList + nodeReturn.Tokens = tokenList + stat = nodeReturn + + elseif tok:ConsumeKeyword('break', tokenList) then + local nodeBreak = {} + nodeBreak.AstType = 'BreakStatement' + nodeBreak.Tokens = tokenList + stat = nodeBreak + + elseif tok:ConsumeKeyword('goto', tokenList) then + if not tok:Is('Ident') then + return false, GenerateError("Label expected") + end + local label = tok:Get(tokenList).Data + local nodeGoto = {} + nodeGoto.AstType = 'GotoStatement' + nodeGoto.Label = label + nodeGoto.Tokens = tokenList + stat = nodeGoto + + else + --statementParseExpr + local st, suffixed = ParseSuffixedExpr(scope) + if not st then return false, suffixed end + + --assignment or call? + if tok:IsSymbol(',') or tok:IsSymbol('=') then + --check that it was not parenthesized, making it not an lvalue + if (suffixed.ParenCount or 0) > 0 then + return false, GenerateError("Can not assign to parenthesized expression, is not an lvalue") + end + + --more processing needed + local lhs = { suffixed } + while tok:ConsumeSymbol(',', tokenList) do + local st, lhsPart = ParseSuffixedExpr(scope) + if not st then return false, lhsPart end + lhs[#lhs+1] = lhsPart + end + + --equals + if not tok:ConsumeSymbol('=', tokenList) then + return false, GenerateError("`=` Expected.") + end + + --rhs + local rhs = {} + local st, firstRhs = ParseExpr(scope) + if not st then return false, firstRhs end + rhs[1] = firstRhs + while tok:ConsumeSymbol(',', tokenList) do + local st, rhsPart = ParseExpr(scope) + if not st then return false, rhsPart end + rhs[#rhs+1] = rhsPart + end + + --done + local nodeAssign = {} + nodeAssign.AstType = 'AssignmentStatement' + nodeAssign.Lhs = lhs + nodeAssign.Rhs = rhs + nodeAssign.Tokens = tokenList + stat = nodeAssign + + elseif suffixed.AstType == 'CallExpr' or + suffixed.AstType == 'TableCallExpr' or + suffixed.AstType == 'StringCallExpr' + then + --it's a call statement + local nodeCall = {} + nodeCall.AstType = 'CallStatement' + nodeCall.Expression = suffixed + nodeCall.Tokens = tokenList + stat = nodeCall + else + return false, GenerateError("Assignment Statement Expected") + end + end + + if tok:IsSymbol(';') then + stat.Semicolon = tok:Get( stat.Tokens ) + end + return true, stat + end + + + local statListCloseKeywords = lookupify{'end', 'else', 'elseif', 'until'} + + ParseStatementList = function(scope) + local nodeStatlist = {} + nodeStatlist.Scope = CreateScope(scope) + nodeStatlist.AstType = 'Statlist' + nodeStatlist.Body = { } + nodeStatlist.Tokens = { } + -- + --local stats = {} + -- + while not statListCloseKeywords[tok:Peek().Data] and not tok:IsEof() do + local st, nodeStatement = ParseStatement(nodeStatlist.Scope) + if not st then return false, nodeStatement end + --stats[#stats+1] = nodeStatement + nodeStatlist.Body[#nodeStatlist.Body + 1] = nodeStatement + end + + if tok:IsEof() then + local nodeEof = {} + nodeEof.AstType = 'Eof' + nodeEof.Tokens = { tok:Get() } + nodeStatlist.Body[#nodeStatlist.Body + 1] = nodeEof + end + + -- + --nodeStatlist.Body = stats + return true, nodeStatlist + end + + + local function mainfunc() + local topScope = CreateScope() + return ParseStatementList(topScope) + end + + local st, main = mainfunc() + --print("Last Token: "..PrintTable(tok:Peek())) + return st, main +end + +return { LexLua = LexLua, ParseLua = ParseLua } + \ No newline at end of file diff --git a/lib/LuaMinify/README.md b/lib/LuaMinify/README.md new file mode 100644 index 0000000..584abf8 --- /dev/null +++ b/lib/LuaMinify/README.md @@ -0,0 +1,44 @@ +Lua Parsing and Refactorization tools +========= + +A collection of tools for working with Lua source code. Primarily a Lua source code minifier, but also includes some static analysis tools and a general Lua lexer and parser. + +Currently the minifier performs: + +- Stripping of all comments and whitespace +- True semantic renaming of all local variables to a reduced form +- Reduces the source to the minimal spacing, spaces are only inserted where actually needed. + + +LuaMinify Command Line Utility Usage +------------------------------------ + +The `LuaMinify` shell and batch files are given as shortcuts to running a command line instance of the minifier with the following usage: + + LuaMinify sourcefile [destfile] + +Which will minify to a given destination file, or to a copy of the source file with _min appended to the filename if no output file is given. + + +LuaMinify Roblox Plugin Usage +----------------------------- + +First, download the source code, which you can do by hitting this button: + +![Click That](http://github.com/stravant/LuaMinify/raw/master/RobloxPluginInstructions.png) + +Then copy the `RobloxPlugin` folder from the source into your Roblox Plugins directory, which can be found by hitting `Tools->Open Plugins Folder` in Roblox Studio. + +Features/Todo +------------- +Features: + + - Lua scanner/parser, which generates a full AST + - Lua reconstructor + - minimal + - full reconstruction (TODO: options, comments) + - TODO: exact reconstructor + - support for embedded long strings/comments e.g. [[abc [[ def ]] ghi]] + +Todo: + - use table.concat instead of appends in the reconstructors \ No newline at end of file diff --git a/lib/LuaMinify/RobloxPlugin/Minify.lua b/lib/LuaMinify/RobloxPlugin/Minify.lua new file mode 100644 index 0000000..bbd1dbd --- /dev/null +++ b/lib/LuaMinify/RobloxPlugin/Minify.lua @@ -0,0 +1,1570 @@ + +-- +-- Minify.lua +-- +-- A compilation of all of the neccesary code to Minify a source file, all into one single +-- script for usage on Roblox. Needed to deal with Roblox' lack of `require`. +-- + +function lookupify(tb) + for _, v in pairs(tb) do + tb[v] = true + end + return tb +end + +function CountTable(tb) + local c = 0 + for _ in pairs(tb) do c = c + 1 end + return c +end + +function PrintTable(tb, atIndent) + if tb.Print then + return tb.Print() + end + atIndent = atIndent or 0 + local useNewlines = (CountTable(tb) > 1) + local baseIndent = string.rep(' ', atIndent+1) + local out = "{"..(useNewlines and '\n' or '') + for k, v in pairs(tb) do + if type(v) ~= 'function' then + out = out..(useNewlines and baseIndent or '') + if type(k) == 'number' then + --nothing to do + elseif type(k) == 'string' and k:match("^[A-Za-z_][A-Za-z0-9_]*$") then + out = out..k.." = " + elseif type(k) == 'string' then + out = out.."[\""..k.."\"] = " + else + out = out.."["..tostring(k).."] = " + end + if type(v) == 'string' then + out = out.."\""..v.."\"" + elseif type(v) == 'number' then + out = out..v + elseif type(v) == 'table' then + out = out..PrintTable(v, atIndent+(useNewlines and 1 or 0)) + else + out = out..tostring(v) + end + if next(tb, k) then + out = out.."," + end + if useNewlines then + out = out..'\n' + end + end + end + out = out..(useNewlines and string.rep(' ', atIndent) or '').."}" + return out +end + +local WhiteChars = lookupify{' ', '\n', '\t', '\r'} +local EscapeLookup = {['\r'] = '\\r', ['\n'] = '\\n', ['\t'] = '\\t', ['"'] = '\\"', ["'"] = "\\'"} +local LowerChars = lookupify{'a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', + 'j', 'k', 'l', 'm', 'n', 'o', 'p', 'q', 'r', + 's', 't', 'u', 'v', 'w', 'x', 'y', 'z'} +local UpperChars = lookupify{'A', 'B', 'C', 'D', 'E', 'F', 'G', 'H', 'I', + 'J', 'K', 'L', 'M', 'N', 'O', 'P', 'Q', 'R', + 'S', 'T', 'U', 'V', 'W', 'X', 'Y', 'Z'} +local Digits = lookupify{'0', '1', '2', '3', '4', '5', '6', '7', '8', '9'} +local HexDigits = lookupify{'0', '1', '2', '3', '4', '5', '6', '7', '8', '9', + 'A', 'a', 'B', 'b', 'C', 'c', 'D', 'd', 'E', 'e', 'F', 'f'} + +local Symbols = lookupify{'+', '-', '*', '/', '^', '%', ',', '{', '}', '[', ']', '(', ')', ';', '#'} + +local Keywords = lookupify{ + 'and', 'break', 'do', 'else', 'elseif', + 'end', 'false', 'for', 'function', 'goto', 'if', + 'in', 'local', 'nil', 'not', 'or', 'repeat', + 'return', 'then', 'true', 'until', 'while', +}; + +function LexLua(src) + --token dump + local tokens = {} + + local st, err = pcall(function() + --line / char / pointer tracking + local p = 1 + local line = 1 + local char = 1 + + --get / peek functions + local function get() + local c = src:sub(p,p) + if c == '\n' then + char = 1 + line = line + 1 + else + char = char + 1 + end + p = p + 1 + return c + end + local function peek(n) + n = n or 0 + return src:sub(p+n,p+n) + end + local function consume(chars) + local c = peek() + for i = 1, #chars do + if c == chars:sub(i,i) then return get() end + end + end + + --shared stuff + local function generateError(err) + return error(">> :"..line..":"..char..": "..err, 0) + end + + local function tryGetLongString() + local start = p + if peek() == '[' then + local equalsCount = 0 + while peek(equalsCount+1) == '=' do + equalsCount = equalsCount + 1 + end + if peek(equalsCount+1) == '[' then + --start parsing the string. Strip the starting bit + for _ = 0, equalsCount+1 do get() end + + --get the contents + local contentStart = p + while true do + --check for eof + if peek() == '' then + generateError("Expected `]"..string.rep('=', equalsCount).."]` near .", 3) + end + + --check for the end + local foundEnd = true + if peek() == ']' then + for i = 1, equalsCount do + if peek(i) ~= '=' then foundEnd = false end + end + if peek(equalsCount+1) ~= ']' then + foundEnd = false + end + else + foundEnd = false + end + -- + if foundEnd then + break + else + get() + end + end + + --get the interior string + local contentString = src:sub(contentStart, p-1) + + --found the end. Get rid of the trailing bit + for i = 0, equalsCount+1 do get() end + + --get the exterior string + local longString = src:sub(start, p-1) + + --return the stuff + return contentString, longString + else + return nil + end + else + return nil + end + end + + --main token emitting loop + while true do + --get leading whitespace. The leading whitespace will include any comments + --preceding the token. This prevents the parser needing to deal with comments + --separately. + local leadingWhite = '' + while true do + local c = peek() + if WhiteChars[c] then + --whitespace + leadingWhite = leadingWhite..get() + elseif c == '-' and peek(1) == '-' then + --comment + get();get() + leadingWhite = leadingWhite..'--' + local _, wholeText = tryGetLongString() + if wholeText then + leadingWhite = leadingWhite..wholeText + else + while peek() ~= '\n' and peek() ~= '' do + leadingWhite = leadingWhite..get() + end + end + else + break + end + end + + --get the initial char + local thisLine = line + local thisChar = char + local errorAt = ":"..line..":"..char..":> " + local c = peek() + + --symbol to emit + local toEmit = nil + + --branch on type + if c == '' then + --eof + toEmit = {Type = 'Eof'} + + elseif UpperChars[c] or LowerChars[c] or c == '_' then + --ident or keyword + local start = p + repeat + get() + c = peek() + until not (UpperChars[c] or LowerChars[c] or Digits[c] or c == '_') + local dat = src:sub(start, p-1) + if Keywords[dat] then + toEmit = {Type = 'Keyword', Data = dat} + else + toEmit = {Type = 'Ident', Data = dat} + end + + elseif Digits[c] or (peek() == '.' and Digits[peek(1)]) then + --number const + local start = p + if c == '0' and peek(1) == 'x' then + get();get() + while HexDigits[peek()] do get() end + if consume('Pp') then + consume('+-') + while Digits[peek()] do get() end + end + else + while Digits[peek()] do get() end + if consume('.') then + while Digits[peek()] do get() end + end + if consume('Ee') then + consume('+-') + while Digits[peek()] do get() end + end + end + toEmit = {Type = 'Number', Data = src:sub(start, p-1)} + + elseif c == '\'' or c == '\"' then + local start = p + --string const + local delim = get() + local contentStart = p + while true do + local c = get() + if c == '\\' then + get() --get the escape char + elseif c == delim then + break + elseif c == '' then + generateError("Unfinished string near ") + end + end + local content = src:sub(contentStart, p-2) + local constant = src:sub(start, p-1) + toEmit = {Type = 'String', Data = constant, Constant = content} + + elseif c == '[' then + local content, wholetext = tryGetLongString() + if wholetext then + toEmit = {Type = 'String', Data = wholetext, Constant = content} + else + get() + toEmit = {Type = 'Symbol', Data = '['} + end + + elseif consume('>=<') then + if consume('=') then + toEmit = {Type = 'Symbol', Data = c..'='} + else + toEmit = {Type = 'Symbol', Data = c} + end + + elseif consume('~') then + if consume('=') then + toEmit = {Type = 'Symbol', Data = '~='} + else + generateError("Unexpected symbol `~` in source.", 2) + end + + elseif consume('.') then + if consume('.') then + if consume('.') then + toEmit = {Type = 'Symbol', Data = '...'} + else + toEmit = {Type = 'Symbol', Data = '..'} + end + else + toEmit = {Type = 'Symbol', Data = '.'} + end + + elseif consume(':') then + if consume(':') then + toEmit = {Type = 'Symbol', Data = '::'} + else + toEmit = {Type = 'Symbol', Data = ':'} + end + + elseif Symbols[c] then + get() + toEmit = {Type = 'Symbol', Data = c} + + else + local contents, all = tryGetLongString() + if contents then + toEmit = {Type = 'String', Data = all, Constant = contents} + else + generateError("Unexpected Symbol `"..c.."` in source.", 2) + end + end + + --add the emitted symbol, after adding some common data + toEmit.LeadingWhite = leadingWhite + toEmit.Line = thisLine + toEmit.Char = thisChar + toEmit.Print = function() + return "<"..(toEmit.Type..string.rep(' ', 7-#toEmit.Type)).." "..(toEmit.Data or '').." >" + end + tokens[#tokens+1] = toEmit + + --halt after eof has been emitted + if toEmit.Type == 'Eof' then break end + end + end) + if not st then + return false, err + end + + --public interface: + local tok = {} + local savedP = {} + local p = 1 + + --getters + function tok:Peek(n) + n = n or 0 + return tokens[math.min(#tokens, p+n)] + end + function tok:Get() + local t = tokens[p] + p = math.min(p + 1, #tokens) + return t + end + function tok:Is(t) + return tok:Peek().Type == t + end + + --save / restore points in the stream + function tok:Save() + savedP[#savedP+1] = p + end + function tok:Commit() + savedP[#savedP] = nil + end + function tok:Restore() + p = savedP[#savedP] + savedP[#savedP] = nil + end + + --either return a symbol if there is one, or return true if the requested + --symbol was gotten. + function tok:ConsumeSymbol(symb) + local t = self:Peek() + if t.Type == 'Symbol' then + if symb then + if t.Data == symb then + self:Get() + return true + else + return nil + end + else + self:Get() + return t + end + else + return nil + end + end + + function tok:ConsumeKeyword(kw) + local t = self:Peek() + if t.Type == 'Keyword' and t.Data == kw then + self:Get() + return true + else + return nil + end + end + + function tok:IsKeyword(kw) + local t = tok:Peek() + return t.Type == 'Keyword' and t.Data == kw + end + + function tok:IsSymbol(s) + local t = tok:Peek() + return t.Type == 'Symbol' and t.Data == s + end + + function tok:IsEof() + return tok:Peek().Type == 'Eof' + end + + return true, tok +end + + +function ParseLua(src) + local st, tok = LexLua(src) + if not st then + return false, tok + end + -- + local function GenerateError(msg) + local err = ">> :"..tok:Peek().Line..":"..tok:Peek().Char..": "..msg.."\n" + --find the line + local lineNum = 0 + for line in src:gmatch("[^\n]*\n?") do + if line:sub(-1,-1) == '\n' then line = line:sub(1,-2) end + lineNum = lineNum+1 + if lineNum == tok:Peek().Line then + err = err..">> `"..line:gsub('\t',' ').."`\n" + for i = 1, tok:Peek().Char do + local c = line:sub(i,i) + if c == '\t' then + err = err..' ' + else + err = err..' ' + end + end + err = err.." ^---" + break + end + end + return err + end + -- + local VarUid = 0 + local GlobalVarGetMap = {} + local VarDigits = {'_', 'a', 'b', 'c', 'd'} + local function CreateScope(parent) + local scope = {} + scope.Parent = parent + scope.LocalList = {} + scope.LocalMap = {} + function scope:RenameVars() + for _, var in pairs(scope.LocalList) do + local id; + VarUid = 0 + repeat + VarUid = VarUid + 1 + local varToUse = VarUid + id = '' + while varToUse > 0 do + local d = varToUse % #VarDigits + varToUse = (varToUse - d) / #VarDigits + id = id..VarDigits[d+1] + end + until not GlobalVarGetMap[id] and not parent:GetLocal(id) and not scope.LocalMap[id] + var.Name = id + scope.LocalMap[id] = var + end + end + function scope:GetLocal(name) + --first, try to get my variable + local my = scope.LocalMap[name] + if my then return my end + + --next, try parent + if scope.Parent then + local par = scope.Parent:GetLocal(name) + if par then return par end + end + + return nil + end + function scope:CreateLocal(name) + --create my own var + local my = {} + my.Scope = scope + my.Name = name + my.CanRename = true + -- + scope.LocalList[#scope.LocalList+1] = my + scope.LocalMap[name] = my + -- + return my + end + scope.Print = function() return "" end + return scope + end + + local ParseExpr; + local ParseStatementList; + + local function ParseFunctionArgsAndBody(scope) + local funcScope = CreateScope(scope) + if not tok:ConsumeSymbol('(') then + return false, GenerateError("`(` expected.") + end + + --arg list + local argList = {} + local isVarArg = false + while not tok:ConsumeSymbol(')') do + if tok:Is('Ident') then + local arg = funcScope:CreateLocal(tok:Get().Data) + argList[#argList+1] = arg + if not tok:ConsumeSymbol(',') then + if tok:ConsumeSymbol(')') then + break + else + return false, GenerateError("`)` expected.") + end + end + elseif tok:ConsumeSymbol('...') then + isVarArg = true + if not tok:ConsumeSymbol(')') then + return false, GenerateError("`...` must be the last argument of a function.") + end + break + else + return false, GenerateError("Argument name or `...` expected") + end + end + + --body + local st, body = ParseStatementList(funcScope) + if not st then return false, body end + + --end + if not tok:ConsumeKeyword('end') then + return false, GenerateError("`end` expected after function body") + end + + local nodeFunc = {} + nodeFunc.AstType = 'Function' + nodeFunc.Scope = funcScope + nodeFunc.Arguments = argList + nodeFunc.Body = body + nodeFunc.VarArg = isVarArg + -- + return true, nodeFunc + end + + + local function ParsePrimaryExpr(scope) + if tok:ConsumeSymbol('(') then + local st, ex = ParseExpr(scope) + if not st then return false, ex end + if not tok:ConsumeSymbol(')') then + return false, GenerateError("`)` Expected.") + end + --save the information about parenthesized expressions somewhere + ex.ParenCount = (ex.ParenCount or 0) + 1 + return true, ex + + elseif tok:Is('Ident') then + local id = tok:Get() + local var = scope:GetLocal(id.Data) + if not var then + GlobalVarGetMap[id.Data] = true + end + -- + local nodePrimExp = {} + nodePrimExp.AstType = 'VarExpr' + nodePrimExp.Name = id.Data + nodePrimExp.Local = var + -- + return true, nodePrimExp + else + return false, GenerateError("primary expression expected") + end + end + + + local function ParseSuffixedExpr(scope, onlyDotColon) + --base primary expression + local st, prim = ParsePrimaryExpr(scope) + if not st then return false, prim end + -- + while true do + if tok:IsSymbol('.') or tok:IsSymbol(':') then + local symb = tok:Get().Data + if not tok:Is('Ident') then + return false, GenerateError(" expected.") + end + local id = tok:Get() + local nodeIndex = {} + nodeIndex.AstType = 'MemberExpr' + nodeIndex.Base = prim + nodeIndex.Indexer = symb + nodeIndex.Ident = id + -- + prim = nodeIndex + + elseif not onlyDotColon and tok:ConsumeSymbol('[') then + local st, ex = ParseExpr(scope) + if not st then return false, ex end + if not tok:ConsumeSymbol(']') then + return false, GenerateError("`]` expected.") + end + local nodeIndex = {} + nodeIndex.AstType = 'IndexExpr' + nodeIndex.Base = prim + nodeIndex.Index = ex + -- + prim = nodeIndex + + elseif not onlyDotColon and tok:ConsumeSymbol('(') then + local args = {} + while not tok:ConsumeSymbol(')') do + local st, ex = ParseExpr(scope) + if not st then return false, ex end + args[#args+1] = ex + if not tok:ConsumeSymbol(',') then + if tok:ConsumeSymbol(')') then + break + else + return false, GenerateError("`)` Expected.") + end + end + end + local nodeCall = {} + nodeCall.AstType = 'CallExpr' + nodeCall.Base = prim + nodeCall.Arguments = args + -- + prim = nodeCall + + elseif not onlyDotColon and tok:Is('String') then + --string call + local nodeCall = {} + nodeCall.AstType = 'StringCallExpr' + nodeCall.Base = prim + nodeCall.Arguments = {tok:Get()} + -- + prim = nodeCall + + elseif not onlyDotColon and tok:IsSymbol('{') then + --table call + local st, ex = ParseExpr(scope) + if not st then return false, ex end + local nodeCall = {} + nodeCall.AstType = 'TableCallExpr' + nodeCall.Base = prim + nodeCall.Arguments = {ex} + -- + prim = nodeCall + + else + break + end + end + return true, prim + end + + + local function ParseSimpleExpr(scope) + if tok:Is('Number') then + local nodeNum = {} + nodeNum.AstType = 'NumberExpr' + nodeNum.Value = tok:Get() + return true, nodeNum + + elseif tok:Is('String') then + local nodeStr = {} + nodeStr.AstType = 'StringExpr' + nodeStr.Value = tok:Get() + return true, nodeStr + + elseif tok:ConsumeKeyword('nil') then + local nodeNil = {} + nodeNil.AstType = 'NilExpr' + return true, nodeNil + + elseif tok:IsKeyword('false') or tok:IsKeyword('true') then + local nodeBoolean = {} + nodeBoolean.AstType = 'BooleanExpr' + nodeBoolean.Value = (tok:Get().Data == 'true') + return true, nodeBoolean + + elseif tok:ConsumeSymbol('...') then + local nodeDots = {} + nodeDots.AstType = 'DotsExpr' + return true, nodeDots + + elseif tok:ConsumeSymbol('{') then + local v = {} + v.AstType = 'ConstructorExpr' + v.EntryList = {} + -- + while true do + if tok:IsSymbol('[') then + --key + tok:Get() + local st, key = ParseExpr(scope) + if not st then + return false, GenerateError("Key Expression Expected") + end + if not tok:ConsumeSymbol(']') then + return false, GenerateError("`]` Expected") + end + if not tok:ConsumeSymbol('=') then + return false, GenerateError("`=` Expected") + end + local st, value = ParseExpr(scope) + if not st then + return false, GenerateError("Value Expression Expected") + end + v.EntryList[#v.EntryList+1] = { + Type = 'Key'; + Key = key; + Value = value; + } + + elseif tok:Is('Ident') then + --value or key + local lookahead = tok:Peek(1) + if lookahead.Type == 'Symbol' and lookahead.Data == '=' then + --we are a key + local key = tok:Get() + if not tok:ConsumeSymbol('=') then + return false, GenerateError("`=` Expected") + end + local st, value = ParseExpr(scope) + if not st then + return false, GenerateError("Value Expression Expected") + end + v.EntryList[#v.EntryList+1] = { + Type = 'KeyString'; + Key = key.Data; + Value = value; + } + + else + --we are a value + local st, value = ParseExpr(scope) + if not st then + return false, GenerateError("Value Exected") + end + v.EntryList[#v.EntryList+1] = { + Type = 'Value'; + Value = value; + } + + end + elseif tok:ConsumeSymbol('}') then + break + + else + --value + local st, value = ParseExpr(scope) + v.EntryList[#v.EntryList+1] = { + Type = 'Value'; + Value = value; + } + if not st then + return false, GenerateError("Value Expected") + end + end + + if tok:ConsumeSymbol(';') or tok:ConsumeSymbol(',') then + --all is good + elseif tok:ConsumeSymbol('}') then + break + else + return false, GenerateError("`}` or table entry Expected") + end + end + return true, v + + elseif tok:ConsumeKeyword('function') then + local st, func = ParseFunctionArgsAndBody(scope) + if not st then return false, func end + -- + func.IsLocal = true + return true, func + + else + return ParseSuffixedExpr(scope) + end + end + + + local unops = lookupify{'-', 'not', '#'} + local unopprio = 8 + local priority = { + ['+'] = {6,6}; + ['-'] = {6,6}; + ['%'] = {7,7}; + ['/'] = {7,7}; + ['*'] = {7,7}; + ['^'] = {10,9}; + ['..'] = {5,4}; + ['=='] = {3,3}; + ['<'] = {3,3}; + ['<='] = {3,3}; + ['~='] = {3,3}; + ['>'] = {3,3}; + ['>='] = {3,3}; + ['and'] = {2,2}; + ['or'] = {1,1}; + } + local function ParseSubExpr(scope, level) + --base item, possibly with unop prefix + local st, exp + if unops[tok:Peek().Data] then + local op = tok:Get().Data + st, exp = ParseSubExpr(scope, unopprio) + if not st then return false, exp end + local nodeEx = {} + nodeEx.AstType = 'UnopExpr' + nodeEx.Rhs = exp + nodeEx.Op = op + exp = nodeEx + else + st, exp = ParseSimpleExpr(scope) + if not st then return false, exp end + end + + --next items in chain + while true do + local prio = priority[tok:Peek().Data] + if prio and prio[1] > level then + local op = tok:Get().Data + local st, rhs = ParseSubExpr(scope, prio[2]) + if not st then return false, rhs end + local nodeEx = {} + nodeEx.AstType = 'BinopExpr' + nodeEx.Lhs = exp + nodeEx.Op = op + nodeEx.Rhs = rhs + -- + exp = nodeEx + else + break + end + end + + return true, exp + end + + + ParseExpr = function(scope) + return ParseSubExpr(scope, 0) + end + + + local function ParseStatement(scope) + local stat = nil + if tok:ConsumeKeyword('if') then + --setup + local nodeIfStat = {} + nodeIfStat.AstType = 'IfStatement' + nodeIfStat.Clauses = {} + + --clauses + repeat + local st, nodeCond = ParseExpr(scope) + if not st then return false, nodeCond end + if not tok:ConsumeKeyword('then') then + return false, GenerateError("`then` expected.") + end + local st, nodeBody = ParseStatementList(scope) + if not st then return false, nodeBody end + nodeIfStat.Clauses[#nodeIfStat.Clauses+1] = { + Condition = nodeCond; + Body = nodeBody; + } + until not tok:ConsumeKeyword('elseif') + + --else clause + if tok:ConsumeKeyword('else') then + local st, nodeBody = ParseStatementList(scope) + if not st then return false, nodeBody end + nodeIfStat.Clauses[#nodeIfStat.Clauses+1] = { + Body = nodeBody; + } + end + + --end + if not tok:ConsumeKeyword('end') then + return false, GenerateError("`end` expected.") + end + + stat = nodeIfStat + + elseif tok:ConsumeKeyword('while') then + --setup + local nodeWhileStat = {} + nodeWhileStat.AstType = 'WhileStatement' + + --condition + local st, nodeCond = ParseExpr(scope) + if not st then return false, nodeCond end + + --do + if not tok:ConsumeKeyword('do') then + return false, GenerateError("`do` expected.") + end + + --body + local st, nodeBody = ParseStatementList(scope) + if not st then return false, nodeBody end + + --end + if not tok:ConsumeKeyword('end') then + return false, GenerateError("`end` expected.") + end + + --return + nodeWhileStat.Condition = nodeCond + nodeWhileStat.Body = nodeBody + stat = nodeWhileStat + + elseif tok:ConsumeKeyword('do') then + --do block + local st, nodeBlock = ParseStatementList(scope) + if not st then return false, nodeBlock end + if not tok:ConsumeKeyword('end') then + return false, GenerateError("`end` expected.") + end + + local nodeDoStat = {} + nodeDoStat.AstType = 'DoStatement' + nodeDoStat.Body = nodeBlock + stat = nodeDoStat + + elseif tok:ConsumeKeyword('for') then + --for block + if not tok:Is('Ident') then + return false, GenerateError(" expected.") + end + local baseVarName = tok:Get() + if tok:ConsumeSymbol('=') then + --numeric for + local forScope = CreateScope(scope) + local forVar = forScope:CreateLocal(baseVarName.Data) + -- + local st, startEx = ParseExpr(scope) + if not st then return false, startEx end + if not tok:ConsumeSymbol(',') then + return false, GenerateError("`,` Expected") + end + local st, endEx = ParseExpr(scope) + if not st then return false, endEx end + local st, stepEx; + if tok:ConsumeSymbol(',') then + st, stepEx = ParseExpr(scope) + if not st then return false, stepEx end + end + if not tok:ConsumeKeyword('do') then + return false, GenerateError("`do` expected") + end + -- + local st, body = ParseStatementList(forScope) + if not st then return false, body end + if not tok:ConsumeKeyword('end') then + return false, GenerateError("`end` expected") + end + -- + local nodeFor = {} + nodeFor.AstType = 'NumericForStatement' + nodeFor.Scope = forScope + nodeFor.Variable = forVar + nodeFor.Start = startEx + nodeFor.End = endEx + nodeFor.Step = stepEx + nodeFor.Body = body + stat = nodeFor + else + --generic for + local forScope = CreateScope(scope) + -- + local varList = {forScope:CreateLocal(baseVarName.Data)} + while tok:ConsumeSymbol(',') do + if not tok:Is('Ident') then + return false, GenerateError("for variable expected.") + end + varList[#varList+1] = forScope:CreateLocal(tok:Get().Data) + end + if not tok:ConsumeKeyword('in') then + return false, GenerateError("`in` expected.") + end + local generators = {} + local st, firstGenerator = ParseExpr(scope) + if not st then return false, firstGenerator end + generators[#generators+1] = firstGenerator + while tok:ConsumeSymbol(',') do + local st, gen = ParseExpr(scope) + if not st then return false, gen end + generators[#generators+1] = gen + end + if not tok:ConsumeKeyword('do') then + return false, GenerateError("`do` expected.") + end + local st, body = ParseStatementList(forScope) + if not st then return false, body end + if not tok:ConsumeKeyword('end') then + return false, GenerateError("`end` expected.") + end + -- + local nodeFor = {} + nodeFor.AstType = 'GenericForStatement' + nodeFor.Scope = forScope + nodeFor.VariableList = varList + nodeFor.Generators = generators + nodeFor.Body = body + stat = nodeFor + end + + elseif tok:ConsumeKeyword('repeat') then + local st, body = ParseStatementList(scope) + if not st then return false, body end + -- + if not tok:ConsumeKeyword('until') then + return false, GenerateError("`until` expected.") + end + -- + local st, cond = ParseExpr(scope) + if not st then return false, cond end + -- + local nodeRepeat = {} + nodeRepeat.AstType = 'RepeatStatement' + nodeRepeat.Condition = cond + nodeRepeat.Body = body + stat = nodeRepeat + + elseif tok:ConsumeKeyword('function') then + if not tok:Is('Ident') then + return false, GenerateError("Function name expected") + end + local st, name = ParseSuffixedExpr(scope, true) --true => only dots and colons + if not st then return false, name end + -- + local st, func = ParseFunctionArgsAndBody(scope) + if not st then return false, func end + -- + func.IsLocal = false + func.Name = name + stat = func + + elseif tok:ConsumeKeyword('local') then + if tok:Is('Ident') then + local varList = {tok:Get().Data} + while tok:ConsumeSymbol(',') do + if not tok:Is('Ident') then + return false, GenerateError("local var name expected") + end + varList[#varList+1] = tok:Get().Data + end + + local initList = {} + if tok:ConsumeSymbol('=') then + repeat + local st, ex = ParseExpr(scope) + if not st then return false, ex end + initList[#initList+1] = ex + until not tok:ConsumeSymbol(',') + end + + --now patch var list + --we can't do this before getting the init list, because the init list does not + --have the locals themselves in scope. + for i, v in pairs(varList) do + varList[i] = scope:CreateLocal(v) + end + + local nodeLocal = {} + nodeLocal.AstType = 'LocalStatement' + nodeLocal.LocalList = varList + nodeLocal.InitList = initList + -- + stat = nodeLocal + + elseif tok:ConsumeKeyword('function') then + if not tok:Is('Ident') then + return false, GenerateError("Function name expected") + end + local name = tok:Get().Data + local localVar = scope:CreateLocal(name) + -- + local st, func = ParseFunctionArgsAndBody(scope) + if not st then return false, func end + -- + func.Name = localVar + func.IsLocal = true + stat = func + + else + return false, GenerateError("local var or function def expected") + end + + elseif tok:ConsumeSymbol('::') then + if not tok:Is('Ident') then + return false, GenerateError('Label name expected') + end + local label = tok:Get().Data + if not tok:ConsumeSymbol('::') then + return false, GenerateError("`::` expected") + end + local nodeLabel = {} + nodeLabel.AstType = 'LabelStatement' + nodeLabel.Label = label + stat = nodeLabel + + elseif tok:ConsumeKeyword('return') then + local exList = {} + if not tok:IsKeyword('end') then + local st, firstEx = ParseExpr(scope) + if st then + exList[1] = firstEx + while tok:ConsumeSymbol(',') do + local st, ex = ParseExpr(scope) + if not st then return false, ex end + exList[#exList+1] = ex + end + end + end + + local nodeReturn = {} + nodeReturn.AstType = 'ReturnStatement' + nodeReturn.Arguments = exList + stat = nodeReturn + + elseif tok:ConsumeKeyword('break') then + local nodeBreak = {} + nodeBreak.AstType = 'BreakStatement' + stat = nodeBreak + + elseif tok:IsKeyword('goto') then + if not tok:Is('Ident') then + return false, GenerateError("Label expected") + end + local label = tok:Get().Data + local nodeGoto = {} + nodeGoto.AstType = 'GotoStatement' + nodeGoto.Label = label + stat = nodeGoto + + else + --statementParseExpr + local st, suffixed = ParseSuffixedExpr(scope) + if not st then return false, suffixed end + + --assignment or call? + if tok:IsSymbol(',') or tok:IsSymbol('=') then + --check that it was not parenthesized, making it not an lvalue + if (suffixed.ParenCount or 0) > 0 then + return false, GenerateError("Can not assign to parenthesized expression, is not an lvalue") + end + + --more processing needed + local lhs = {suffixed} + while tok:ConsumeSymbol(',') do + local st, lhsPart = ParseSuffixedExpr(scope) + if not st then return false, lhsPart end + lhs[#lhs+1] = lhsPart + end + + --equals + if not tok:ConsumeSymbol('=') then + return false, GenerateError("`=` Expected.") + end + + --rhs + local rhs = {} + local st, firstRhs = ParseExpr(scope) + if not st then return false, firstRhs end + rhs[1] = firstRhs + while tok:ConsumeSymbol(',') do + local st, rhsPart = ParseExpr(scope) + if not st then return false, rhsPart end + rhs[#rhs+1] = rhsPart + end + + --done + local nodeAssign = {} + nodeAssign.AstType = 'AssignmentStatement' + nodeAssign.Lhs = lhs + nodeAssign.Rhs = rhs + stat = nodeAssign + + elseif suffixed.AstType == 'CallExpr' or + suffixed.AstType == 'TableCallExpr' or + suffixed.AstType == 'StringCallExpr' + then + --it's a call statement + local nodeCall = {} + nodeCall.AstType = 'CallStatement' + nodeCall.Expression = suffixed + stat = nodeCall + else + return false, GenerateError("Assignment Statement Expected") + end + end + + stat.HasSemicolon = tok:ConsumeSymbol(';') + return true, stat + end + + + local statListCloseKeywords = lookupify{'end', 'else', 'elseif', 'until'} + ParseStatementList = function(scope) + local nodeStatlist = {} + nodeStatlist.Scope = CreateScope(scope) + nodeStatlist.AstType = 'Statlist' + -- + local stats = {} + -- + while not statListCloseKeywords[tok:Peek().Data] and not tok:IsEof() do + local st, nodeStatement = ParseStatement(nodeStatlist.Scope) + if not st then return false, nodeStatement end + stats[#stats+1] = nodeStatement + end + -- + nodeStatlist.Body = stats + return true, nodeStatlist + end + + + local function mainfunc() + local topScope = CreateScope() + return ParseStatementList(topScope) + end + + local st, main = mainfunc() + --print("Last Token: "..PrintTable(tok:Peek())) + return st, main +end + +local LowerChars = lookupify{'a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', + 'j', 'k', 'l', 'm', 'n', 'o', 'p', 'q', 'r', + 's', 't', 'u', 'v', 'w', 'x', 'y', 'z'} +local UpperChars = lookupify{'A', 'B', 'C', 'D', 'E', 'F', 'G', 'H', 'I', + 'J', 'K', 'L', 'M', 'N', 'O', 'P', 'Q', 'R', + 'S', 'T', 'U', 'V', 'W', 'X', 'Y', 'Z'} +local Digits = lookupify{'0', '1', '2', '3', '4', '5', '6', '7', '8', '9'} + +function Format_Mini(ast) + local formatStatlist, formatExpr; + local count = 0 + -- + local function joinStatementsSafe(a, b, sep) + if count > 150 then + count = 0 + return a.."\n"..b + end + sep = sep or ' ' + local aa, bb = a:sub(-1,-1), b:sub(1,1) + if UpperChars[aa] or LowerChars[aa] or aa == '_' then + if not (UpperChars[bb] or LowerChars[bb] or bb == '_' or Digits[bb]) then + --bb is a symbol, can join without sep + return a..b + elseif bb == '(' then + print("==============>>>",aa,bb) + --prevent ambiguous syntax + return a..sep..b + else + return a..sep..b + end + elseif Digits[aa] then + if bb == '(' then + --can join statements directly + return a..b + else + return a..sep..b + end + elseif aa == '' then + return a..b + else + if bb == '(' then + --don't want to accidentally call last statement, can't join directly + return a..sep..b + else + return a..b + end + end + end + + formatExpr = function(expr) + local out = string.rep('(', expr.ParenCount or 0) + if expr.AstType == 'VarExpr' then + if expr.Local then + out = out..expr.Local.Name + else + out = out..expr.Name + end + + elseif expr.AstType == 'NumberExpr' then + out = out..expr.Value.Data + + elseif expr.AstType == 'StringExpr' then + out = out..expr.Value.Data + + elseif expr.AstType == 'BooleanExpr' then + out = out..tostring(expr.Value) + + elseif expr.AstType == 'NilExpr' then + out = joinStatementsSafe(out, "nil") + + elseif expr.AstType == 'BinopExpr' then + out = joinStatementsSafe(out, formatExpr(expr.Lhs)) + out = joinStatementsSafe(out, expr.Op) + out = joinStatementsSafe(out, formatExpr(expr.Rhs)) + + elseif expr.AstType == 'UnopExpr' then + out = joinStatementsSafe(out, expr.Op) + out = joinStatementsSafe(out, formatExpr(expr.Rhs)) + + elseif expr.AstType == 'DotsExpr' then + out = out.."..." + + elseif expr.AstType == 'CallExpr' then + out = out..formatExpr(expr.Base) + out = out.."(" + for i = 1, #expr.Arguments do + out = out..formatExpr(expr.Arguments[i]) + if i ~= #expr.Arguments then + out = out.."," + end + end + out = out..")" + + elseif expr.AstType == 'TableCallExpr' then + out = out..formatExpr(expr.Base) + out = out..formatExpr(expr.Arguments[1]) + + elseif expr.AstType == 'StringCallExpr' then + out = out..formatExpr(expr.Base) + out = out..expr.Arguments[1].Data + + elseif expr.AstType == 'IndexExpr' then + out = out..formatExpr(expr.Base).."["..formatExpr(expr.Index).."]" + + elseif expr.AstType == 'MemberExpr' then + out = out..formatExpr(expr.Base)..expr.Indexer..expr.Ident.Data + + elseif expr.AstType == 'Function' then + expr.Scope:RenameVars() + out = out.."function(" + if #expr.Arguments > 0 then + for i = 1, #expr.Arguments do + out = out..expr.Arguments[i].Name + if i ~= #expr.Arguments then + out = out.."," + elseif expr.VarArg then + out = out..",..." + end + end + elseif expr.VarArg then + out = out.."..." + end + out = out..")" + out = joinStatementsSafe(out, formatStatlist(expr.Body)) + out = joinStatementsSafe(out, "end") + + elseif expr.AstType == 'ConstructorExpr' then + out = out.."{" + for i = 1, #expr.EntryList do + local entry = expr.EntryList[i] + if entry.Type == 'Key' then + out = out.."["..formatExpr(entry.Key).."]="..formatExpr(entry.Value) + elseif entry.Type == 'Value' then + out = out..formatExpr(entry.Value) + elseif entry.Type == 'KeyString' then + out = out..entry.Key.."="..formatExpr(entry.Value) + end + if i ~= #expr.EntryList then + out = out.."," + end + end + out = out.."}" + + end + out = out..string.rep(')', expr.ParenCount or 0) + count = count + #out + return out + end + + local formatStatement = function(statement) + local out = '' + if statement.AstType == 'AssignmentStatement' then + for i = 1, #statement.Lhs do + out = out..formatExpr(statement.Lhs[i]) + if i ~= #statement.Lhs then + out = out.."," + end + end + if #statement.Rhs > 0 then + out = out.."=" + for i = 1, #statement.Rhs do + out = out..formatExpr(statement.Rhs[i]) + if i ~= #statement.Rhs then + out = out.."," + end + end + end + + elseif statement.AstType == 'CallStatement' then + out = formatExpr(statement.Expression) + + elseif statement.AstType == 'LocalStatement' then + out = out.."local " + for i = 1, #statement.LocalList do + out = out..statement.LocalList[i].Name + if i ~= #statement.LocalList then + out = out.."," + end + end + if #statement.InitList > 0 then + out = out.."=" + for i = 1, #statement.InitList do + out = out..formatExpr(statement.InitList[i]) + if i ~= #statement.InitList then + out = out.."," + end + end + end + + elseif statement.AstType == 'IfStatement' then + out = joinStatementsSafe("if", formatExpr(statement.Clauses[1].Condition)) + out = joinStatementsSafe(out, "then") + out = joinStatementsSafe(out, formatStatlist(statement.Clauses[1].Body)) + for i = 2, #statement.Clauses do + local st = statement.Clauses[i] + if st.Condition then + out = joinStatementsSafe(out, "elseif") + out = joinStatementsSafe(out, formatExpr(st.Condition)) + out = joinStatementsSafe(out, "then") + else + out = joinStatementsSafe(out, "else") + end + out = joinStatementsSafe(out, formatStatlist(st.Body)) + end + out = joinStatementsSafe(out, "end") + + elseif statement.AstType == 'WhileStatement' then + out = joinStatementsSafe("while", formatExpr(statement.Condition)) + out = joinStatementsSafe(out, "do") + out = joinStatementsSafe(out, formatStatlist(statement.Body)) + out = joinStatementsSafe(out, "end") + + elseif statement.AstType == 'DoStatement' then + out = joinStatementsSafe(out, "do") + out = joinStatementsSafe(out, formatStatlist(statement.Body)) + out = joinStatementsSafe(out, "end") + + elseif statement.AstType == 'ReturnStatement' then + out = "return" + for i = 1, #statement.Arguments do + out = joinStatementsSafe(out, formatExpr(statement.Arguments[i])) + if i ~= #statement.Arguments then + out = out.."," + end + end + + elseif statement.AstType == 'BreakStatement' then + out = "break" + + elseif statement.AstType == 'RepeatStatement' then + out = "repeat" + out = joinStatementsSafe(out, formatStatlist(statement.Body)) + out = joinStatementsSafe(out, "until") + out = joinStatementsSafe(out, formatExpr(statement.Condition)) + + elseif statement.AstType == 'Function' then + statement.Scope:RenameVars() + if statement.IsLocal then + out = "local" + end + out = joinStatementsSafe(out, "function ") + if statement.IsLocal then + out = out..statement.Name.Name + else + out = out..formatExpr(statement.Name) + end + out = out.."(" + if #statement.Arguments > 0 then + for i = 1, #statement.Arguments do + out = out..statement.Arguments[i].Name + if i ~= #statement.Arguments then + out = out.."," + elseif statement.VarArg then + print("Apply vararg") + out = out..",..." + end + end + elseif statement.VarArg then + out = out.."..." + end + out = out..")" + out = joinStatementsSafe(out, formatStatlist(statement.Body)) + out = joinStatementsSafe(out, "end") + + elseif statement.AstType == 'GenericForStatement' then + statement.Scope:RenameVars() + out = "for " + for i = 1, #statement.VariableList do + out = out..statement.VariableList[i].Name + if i ~= #statement.VariableList then + out = out.."," + end + end + out = out.." in" + for i = 1, #statement.Generators do + out = joinStatementsSafe(out, formatExpr(statement.Generators[i])) + if i ~= #statement.Generators then + out = joinStatementsSafe(out, ',') + end + end + out = joinStatementsSafe(out, "do") + out = joinStatementsSafe(out, formatStatlist(statement.Body)) + out = joinStatementsSafe(out, "end") + + elseif statement.AstType == 'NumericForStatement' then + out = "for " + out = out..statement.Variable.Name.."=" + out = out..formatExpr(statement.Start)..","..formatExpr(statement.End) + if statement.Step then + out = out..","..formatExpr(statement.Step) + end + out = joinStatementsSafe(out, "do") + out = joinStatementsSafe(out, formatStatlist(statement.Body)) + out = joinStatementsSafe(out, "end") + + end + count = count + #out + return out + end + + formatStatlist = function(statList) + local out = '' + statList.Scope:RenameVars() + for _, stat in pairs(statList.Body) do + out = joinStatementsSafe(out, formatStatement(stat), ';') + end + return out + end + + ast.Scope:RenameVars() + return formatStatlist(ast) +end + +_G.Minify = function(src) + local st, ast = ParseLua(src) + if not st then return false, ast end + return true, Format_Mini(ast) +end \ No newline at end of file diff --git a/lib/LuaMinify/RobloxPlugin/MinifyButtonIcon.png b/lib/LuaMinify/RobloxPlugin/MinifyButtonIcon.png new file mode 100644 index 0000000..5127c97 Binary files /dev/null and b/lib/LuaMinify/RobloxPlugin/MinifyButtonIcon.png differ diff --git a/lib/LuaMinify/RobloxPlugin/MinifyToolbar.lua b/lib/LuaMinify/RobloxPlugin/MinifyToolbar.lua new file mode 100644 index 0000000..3e7494c --- /dev/null +++ b/lib/LuaMinify/RobloxPlugin/MinifyToolbar.lua @@ -0,0 +1,93 @@ +-- +-- MinifyToolbar.lua +-- +-- The main script that generates a toolbar for studio that allows minification of selected +-- scripts, calling on the _G.Minify function defined in `Minify.lua` +-- + +local plugin = PluginManager():CreatePlugin() +local toolbar = plugin:CreateToolbar("Minify") +local minifyButton = toolbar:CreateButton("", "Minify Selected Script", 'MinifyButtonIcon.png') +local toggleReplaceButton = toolbar:CreateButton("Replace", "If enabled, selected script will be REPLACED ".. + "with a minified version", + 'ReplaceButtonIcon.png') + +local replace = false + +toggleReplaceButton.Click:connect(function() + replace = not replace + toggleReplaceButton:SetActive(replace) +end) + +minifyButton.Click:connect(function() + for _, o in pairs(game.Selection:Get()) do + if o:IsA('Script') then + --can't read linkedsource, bail out + if o.LinkedSource ~= '' then + Spawn(function() + error("Minify Plugin: Cannot Minify a script with a LinkedSource", 0) + end) + return + end + + --see if it has been minified + if o.Name:sub(-4,-1) == '_Min' then + local original = o:FindFirstChild(o.Name:sub(1,-5)) + if original then + local st, min = _G.Minify(original.Source) + if st then + game:GetService("ChangeHistoryService"):SetWaypoint("Minify `"..original.Name.."`") + if replace then + o.Source = min + original:Destroy() + else + o.Source = min + end + else + Spawn(function() + error("Minify Plugin: "..min, 0) + end) + return + end + else + if replace then + local st, min = _G.Minify(o.Source) + if st then + game:GetService("ChangeHistoryService"):SetWaypoint("Minify `"..original.Name.."`") + o.Source = min + else + Spawn(function() + error("Minify Plugin: "..min, 0) + end) + return + end + else + Spawn(function() + error("Minify Plugin: Missing original script `"..o.Name:sub(1,-5).."`", 0) + end) + end + end + else + local st, min = _G.Minify(o.Source) + if st then + game:GetService("ChangeHistoryService"):SetWaypoint("Minify `"..o.Name.."`") + if replace then + o.Source = min + o.Name = o.Name.."_Min" + else + local original = o:Clone() + original.Parent = o + original.Disabled = true + o.Name = o.Name.."_Min" + o.Source = min + end + else + Spawn(function() + error("Minify Plugin: "..min, 0) + end) + return + end + end + end + end +end) diff --git a/lib/LuaMinify/RobloxPlugin/ReplaceButtonIcon.png b/lib/LuaMinify/RobloxPlugin/ReplaceButtonIcon.png new file mode 100644 index 0000000..956f436 Binary files /dev/null and b/lib/LuaMinify/RobloxPlugin/ReplaceButtonIcon.png differ diff --git a/lib/LuaMinify/RobloxPluginInstructions.png b/lib/LuaMinify/RobloxPluginInstructions.png new file mode 100644 index 0000000..fe8a4bb Binary files /dev/null and b/lib/LuaMinify/RobloxPluginInstructions.png differ diff --git a/lib/LuaMinify/Scope.lua b/lib/LuaMinify/Scope.lua new file mode 100644 index 0000000..0e74a78 --- /dev/null +++ b/lib/LuaMinify/Scope.lua @@ -0,0 +1,221 @@ +--[[ +This file is part of LuaMinify by stravant (https://github.com/stravant/LuaMinify). + +LICENSE : +The MIT License (MIT) + +Copyright (c) 2012-2013 + +Permission is hereby granted, free of charge, to any person obtaining a copy of +this software and associated documentation files (the "Software"), to deal in +the Software without restriction, including without limitation the rights to +use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of +the Software, and to permit persons to whom the Software is furnished to do so, +subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS +FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR +COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER +IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN +CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. +]] + +local Scope = { + new = function(self, parent) + local s = { + Parent = parent, + Locals = { }, + Globals = { }, + oldLocalNamesMap = { }, + oldGlobalNamesMap = { }, + Children = { }, + } + + if parent then + table.insert(parent.Children, s) + end + + return setmetatable(s, { __index = self }) + end, + + AddLocal = function(self, v) + table.insert(self.Locals, v) + end, + + AddGlobal = function(self, v) + table.insert(self.Globals, v) + end, + + CreateLocal = function(self, name) + local v + v = self:GetLocal(name) + if v then return v end + v = { } + v.Scope = self + v.Name = name + v.IsGlobal = false + v.CanRename = true + v.References = 1 + self:AddLocal(v) + return v + end, + + GetLocal = function(self, name) + for k, var in pairs(self.Locals) do + if var.Name == name then return var end + end + + if self.Parent then + return self.Parent:GetLocal(name) + end + end, + + GetOldLocal = function(self, name) + if self.oldLocalNamesMap[name] then + return self.oldLocalNamesMap[name] + end + return self:GetLocal(name) + end, + + mapLocal = function(self, name, var) + self.oldLocalNamesMap[name] = var + end, + + GetOldGlobal = function(self, name) + if self.oldGlobalNamesMap[name] then + return self.oldGlobalNamesMap[name] + end + return self:GetGlobal(name) + end, + + mapGlobal = function(self, name, var) + self.oldGlobalNamesMap[name] = var + end, + + GetOldVariable = function(self, name) + return self:GetOldLocal(name) or self:GetOldGlobal(name) + end, + + RenameLocal = function(self, oldName, newName) + oldName = type(oldName) == 'string' and oldName or oldName.Name + local found = false + local var = self:GetLocal(oldName) + if var then + var.Name = newName + self:mapLocal(oldName, var) + found = true + end + if not found and self.Parent then + self.Parent:RenameLocal(oldName, newName) + end + end, + + RenameGlobal = function(self, oldName, newName) + oldName = type(oldName) == 'string' and oldName or oldName.Name + local found = false + local var = self:GetGlobal(oldName) + if var then + var.Name = newName + self:mapGlobal(oldName, var) + found = true + end + if not found and self.Parent then + self.Parent:RenameGlobal(oldName, newName) + end + end, + + RenameVariable = function(self, oldName, newName) + oldName = type(oldName) == 'string' and oldName or oldName.Name + if self:GetLocal(oldName) then + self:RenameLocal(oldName, newName) + else + self:RenameGlobal(oldName, newName) + end + end, + + GetAllVariables = function(self) + local ret = self:getVars(true) -- down + for k, v in pairs(self:getVars(false)) do -- up + table.insert(ret, v) + end + return ret + end, + + getVars = function(self, top) + local ret = { } + if top then + for k, v in pairs(self.Children) do + for k2, v2 in pairs(v:getVars(true)) do + table.insert(ret, v2) + end + end + else + for k, v in pairs(self.Locals) do + table.insert(ret, v) + end + for k, v in pairs(self.Globals) do + table.insert(ret, v) + end + if self.Parent then + for k, v in pairs(self.Parent:getVars(false)) do + table.insert(ret, v) + end + end + end + return ret + end, + + CreateGlobal = function(self, name) + local v + v = self:GetGlobal(name) + if v then return v end + v = { } + v.Scope = self + v.Name = name + v.IsGlobal = true + v.CanRename = true + v.References = 1 + self:AddGlobal(v) + return v + end, + + GetGlobal = function(self, name) + for k, v in pairs(self.Globals) do + if v.Name == name then return v end + end + + if self.Parent then + return self.Parent:GetGlobal(name) + end + end, + + GetVariable = function(self, name) + return self:GetLocal(name) or self:GetGlobal(name) + end, + + ObfuscateLocals = function(self, recommendedMaxLength, validNameChars) + recommendedMaxLength = recommendedMaxLength or 7 + local chars = validNameChars or "QWERTYUIOPASDFGHJKLZXCVBNMqwertyuioplkjhgfdsazxcvbnm_" + local chars2 = validNameChars or "QWERTYUIOPASDFGHJKLZXCVBNMqwertyuioplkjhgfdsazxcvbnm_1234567890" + for _, var in pairs(self.Locals) do + local id = "" + local tries = 0 + repeat + local n = math.random(1, #chars) + id = id .. chars:sub(n, n) + for i = 1, math.random(0, tries > 5 and 30 or recommendedMaxLength) do + local n = math.random(1, #chars2) + id = id .. chars2:sub(n, n) + end + tries = tries + 1 + until not self:GetVariable(id) + self:RenameLocal(var.Name, id) + end + end, +} + +return Scope diff --git a/lib/LuaMinify/Util.lua b/lib/LuaMinify/Util.lua new file mode 100644 index 0000000..6a24d02 --- /dev/null +++ b/lib/LuaMinify/Util.lua @@ -0,0 +1,116 @@ +--[[ +This file is part of LuaMinify by stravant (https://github.com/stravant/LuaMinify). + +LICENSE : +The MIT License (MIT) + +Copyright (c) 2012-2013 + +Permission is hereby granted, free of charge, to any person obtaining a copy of +this software and associated documentation files (the "Software"), to deal in +the Software without restriction, including without limitation the rights to +use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of +the Software, and to permit persons to whom the Software is furnished to do so, +subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS +FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR +COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER +IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN +CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. +]] + +-- +-- Util.lua +-- +-- Provides some common utilities shared throughout the project. +-- + +local function lookupify(tb) + for _, v in pairs(tb) do + tb[v] = true + end + return tb +end + + +local function CountTable(tb) + local c = 0 + for _ in pairs(tb) do c = c + 1 end + return c +end + + +local function PrintTable(tb, atIndent) + if tb.Print then + return tb.Print() + end + atIndent = atIndent or 0 + local useNewlines = (CountTable(tb) > 1) + local baseIndent = string.rep(' ', atIndent+1) + local out = "{"..(useNewlines and '\n' or '') + for k, v in pairs(tb) do + if type(v) ~= 'function' then + --do + out = out..(useNewlines and baseIndent or '') + if type(k) == 'number' then + --nothing to do + elseif type(k) == 'string' and k:match("^[A-Za-z_][A-Za-z0-9_]*$") then + out = out..k.." = " + elseif type(k) == 'string' then + out = out.."[\""..k.."\"] = " + else + out = out.."["..tostring(k).."] = " + end + if type(v) == 'string' then + out = out.."\""..v.."\"" + elseif type(v) == 'number' then + out = out..v + elseif type(v) == 'table' then + out = out..PrintTable(v, atIndent+(useNewlines and 1 or 0)) + else + out = out..tostring(v) + end + if next(tb, k) then + out = out.."," + end + if useNewlines then + out = out..'\n' + end + end + end + out = out..(useNewlines and string.rep(' ', atIndent) or '').."}" + return out +end + + +local function splitLines(str) + if str:match("\n") then + local lines = {} + for line in str:gmatch("[^\n]*") do + table.insert(lines, line) + end + assert(#lines > 0) + return lines + else + return { str } + end +end + + +local function printf(fmt, ...) + return print(string.format(fmt, ...)) +end + + +return { + PrintTable = PrintTable, + CountTable = CountTable, + lookupify = lookupify, + splitLines = splitLines, + printf = printf, +} diff --git a/lib/LuaMinify/strict.lua b/lib/LuaMinify/strict.lua new file mode 100644 index 0000000..c67243f --- /dev/null +++ b/lib/LuaMinify/strict.lua @@ -0,0 +1,39 @@ +-- From http://metalua.luaforge.net/src/lib/strict.lua.html +-- +-- strict.lua +-- checks uses of undeclared global variables +-- All global variables must be 'declared' through a regular assignment +-- (even assigning nil will do) in a main chunk before being used +-- anywhere or assigned to inside a function. +-- + +local mt = getmetatable(_G) +if mt == nil then + mt = {} + setmetatable(_G, mt) +end + +__STRICT = true +mt.__declared = {} + +mt.__newindex = function (t, n, v) + if __STRICT and not mt.__declared[n] then + local w = debug.getinfo(2, "S").what + if w ~= "main" and w ~= "C" then + error("assign to undeclared variable '"..n.."'", 2) + end + mt.__declared[n] = true + end + rawset(t, n, v) +end + +mt.__index = function (t, n) + if not mt.__declared[n] and debug.getinfo(2, "S").what ~= "C" then + error("variable '"..n.."' is not declared", 2) + end + return rawget(t, n) +end + +function global(...) + for _, v in ipairs{...} do mt.__declared[v] = true end +end diff --git a/lib/LuaMinify/tests/test_beautifier.lua b/lib/LuaMinify/tests/test_beautifier.lua new file mode 100644 index 0000000..062d9e4 --- /dev/null +++ b/lib/LuaMinify/tests/test_beautifier.lua @@ -0,0 +1,60 @@ +-- Adapted from Yueliang + +package.path = "../?.lua;" .. package.path +local util = require'Util' +local Parser = require'ParseLua' +local Format = require'FormatBeautiful' + +for w in io.lines("test_lines.txt") do + --print(w) + local success, ast = Parser.ParseLua(w) + if w:find("FAIL") then + --[[if success then + print("ERROR PARSING LINE:") + print("Should fail: true. Did fail: " .. tostring(not success)) + print("Line: " .. w) + else + --print("Suceeded!") + end]] + else + if not success then + print("ERROR PARSING LINE:") + print("Should fail: false. Did fail: " .. tostring(not success)) + print("Line: " .. w) + else + success, ast = Format(ast) + --print(success, ast) + if not success then + print("ERROR BEAUTIFYING LINE:") + print("Message: " .. ast) + print("Line: " .. w) + end + local success_ = success + success, ast = loadstring(success) + if not success then + print("ERROR PARSING BEAUTIFIED LINE:") + print("Message: " .. ast) + print("Line: " .. success_) + end + --print("Suceeded!") + end + end +end +print"Done!" +os.remove("tmp") + + +--[[ +function readAll(file) + local f = io.open(file, "rb") + local content = f:read("*all") + f:close() + return content +end + +local text = readAll('../ParseLua.lua') +local success, ast = Parser.ParseLua(text) +local nice +nice = Format(ast) +print(nice) +--]] \ No newline at end of file diff --git a/lib/LuaMinify/tests/test_identity.lua b/lib/LuaMinify/tests/test_identity.lua new file mode 100644 index 0000000..6658cd4 --- /dev/null +++ b/lib/LuaMinify/tests/test_identity.lua @@ -0,0 +1,124 @@ +package.path = "../?.lua;" .. package.path +local Parser = require'ParseLua' +local util = require'Util' +local FormatIdentity = require'FormatIdentity' +local FormatMini = require'FormatMini' +local FormatBeautiful = require'FormatBeautiful' +require'strict' + +function readAll(file) + local f = io.open(file, "rb") + local content = f:read("*all") + f:close() + return content +end + +local g_lexTime = 0 +local g_parseTime = 0 +local g_reconstructTime = 0 + +function reconstructText(text) + local preLex = os.clock() + + local success, tokens, ast, reconstructed + success, tokens = Parser.LexLua(text) + if not success then + print("ERROR: " .. tokens) + return + end + + local preParse = os.clock() + + success, ast = Parser.ParseLua(tokens) + if not success then + print("ERROR: " .. ast) + return + end + + local preReconstruct = os.clock() + + local DO_MINI = false + local DO_CHECK = false + + if DO_MINI then + success, reconstructed = FormatMini(ast) + else + success, reconstructed = FormatIdentity(ast) + end + + if not success then + print("ERROR: " .. reconstructed) + return + end + + local post = os.clock() + g_lexTime = g_lexTime + (preParse - preLex) + g_parseTime = g_parseTime + (preReconstruct - preParse) + g_reconstructTime = g_reconstructTime + (post - preReconstruct) + + if DO_CHECK then + --[[ + print() + print("Reconstructed: ") + print("--------------------") + print(reconstructed) + print("--------------------") + print("Done. ") + --]] + + if reconstructed == text then + --print("Reconstruction succeeded") + else + print("Reconstruction failed") + + local inputLines = util.splitLines(text) + local outputLines = util.splitLines(reconstructed) + local n = math.max(#inputLines, #outputLines) + for i = 1,n do + if inputLines[i] ~= outputLines[i] then + util.printf("ERROR on line %i", i) + util.printf("Input: %q", inputLines[i]) + util.printf("Output: %q", outputLines[i]) + break + end + end + end + end +end + + +--[*[ +local files = { + "../ParseLua.lua", + "../FormatIdentity.lua", + "../Scope.lua", + "../strict.lua", + "../Type.lua", + "Test_identity.lua" +} + +for _,path in ipairs(files) do + print(path) + local text = readAll(path) + reconstructText(text) +end + +--]] + +print("test_lines.txt") + +local line_nr = 0 +for text in io.lines("test_lines.txt") do + line_nr = line_nr + 1 + if not text:find("FAIL") then + --util.printf("\nText: %q", text) + reconstructText(text) + end +end + + +reconstructText('function a(p,q,r,...) end') + +util.printf("Lex time: %f s", g_lexTime) +util.printf("Parse time: %f s", g_parseTime) +util.printf("Format time: %f s", g_reconstructTime) diff --git a/lib/LuaMinify/tests/test_lines.txt b/lib/LuaMinify/tests/test_lines.txt new file mode 100644 index 0000000..3f00471 --- /dev/null +++ b/lib/LuaMinify/tests/test_lines.txt @@ -0,0 +1,523 @@ +; -- FAIL +local -- FAIL +local; -- FAIL +local = -- FAIL +local end -- FAIL +local a +local a; +local a, b, c +local a; local b local c; +local a = 1 +local a local b = a +local a, b = 1, 2 +local a, b, c = 1, 2, 3 +local a, b, c = 1 +local a = 1, 2, 3 +local a, local -- FAIL +local 1 -- FAIL +local "foo" -- FAIL +local a = local -- FAIL +local a, b, = -- FAIL +local a, b = 1, local -- FAIL +local a, b = , local -- FAIL +do -- FAIL +end -- FAIL +do end +do ; end -- FAIL +do 1 end -- FAIL +do "foo" end -- FAIL +do local a, b end +do local a local b end +do local a; local b; end +do local a = 1 end +do do end end +do do end; end +do do do end end end +do do do end; end; end +do do do return end end end +do end do -- FAIL +do end end -- FAIL +do return end +do return return end -- FAIL +do break end -- FAIL +while -- FAIL +while do -- FAIL +while = -- FAIL +while 1 do -- FAIL +while 1 do end +while 1 do local a end +while 1 do local a local b end +while 1 do local a; local b; end +while 1 do 2 end -- FAIL +while 1 do "foo" end -- FAIL +while true do end +while 1 do ; end -- FAIL +while 1 do while -- FAIL +while 1 end -- FAIL +while 1 2 do -- FAIL +while 1 = 2 do -- FAIL +while 1 do return end +while 1 do return return end -- FAIL +while 1 do do end end +while 1 do do return end end +while 1 do break end +while 1 do break break end -- FAIL +while 1 do do break end end +repeat -- FAIL +repeat until -- FAIL +repeat until 0 +repeat until false +repeat until local -- FAIL +repeat end -- FAIL +repeat 1 -- FAIL +repeat = -- FAIL +repeat local a until 1 +repeat local a local b until 0 +repeat local a; local b; until 0 +repeat ; until 1 -- FAIL +repeat 2 until 1 -- FAIL +repeat "foo" until 1 -- FAIL +repeat return until 0 +repeat return return until 0 -- FAIL +repeat break until 0 +repeat break break until 0 -- FAIL +repeat do end until 0 +repeat do return end until 0 +repeat do break end until 0 +for -- FAIL +for do -- FAIL +for end -- FAIL +for 1 -- FAIL +for a -- FAIL +for true -- FAIL +for a, in -- FAIL +for a in -- FAIL +for a do -- FAIL +for a in do -- FAIL +for a in b do -- FAIL +for a in b end -- FAIL +for a in b, do -- FAIL +for a in b do end +for a in b do local a local b end +for a in b do local a; local b; end +for a in b do 1 end -- FAIL +for a in b do "foo" end -- FAIL +for a b in -- FAIL +for a, b, c in p do end +for a, b, c in p, q, r do end +for a in 1 do end +for a in true do end +for a in "foo" do end +for a in b do break end +for a in b do break break end -- FAIL +for a in b do return end +for a in b do return return end -- FAIL +for a in b do do end end +for a in b do do break end end +for a in b do do return end end +for = -- FAIL +for a = -- FAIL +for a, b = -- FAIL +for a = do -- FAIL +for a = 1, do -- FAIL +for a = p, q, do -- FAIL +for a = p q do -- FAIL +for a = b do end -- FAIL +for a = 1, 2, 3, 4 do end -- FAIL +for a = p, q do end +for a = 1, 2 do end +for a = 1, 2 do local a local b end +for a = 1, 2 do local a; local b; end +for a = 1, 2 do 3 end -- FAIL +for a = 1, 2 do "foo" end -- FAIL +for a = p, q, r do end +for a = 1, 2, 3 do end +for a = p, q do break end +for a = p, q do break break end -- FAIL +for a = 1, 2 do return end +for a = 1, 2 do return return end -- FAIL +for a = p, q do do end end +for a = p, q do do break end end +for a = p, q do do return end end +break -- FAIL +return +return; +return return -- FAIL +return 1 +return local -- FAIL +return "foo" +return 1, -- FAIL +return 1,2,3 +return a,b,c,d +return 1,2; +return ... +return 1,a,... +if -- FAIL +elseif -- FAIL +else -- FAIL +then -- FAIL +if then -- FAIL +if 1 -- FAIL +if 1 then -- FAIL +if 1 else -- FAIL +if 1 then else -- FAIL +if 1 then elseif -- FAIL +if 1 then end +if 1 then local a end +if 1 then local a local b end +if 1 then local a; local b; end +if 1 then else end +if 1 then local a else local b end +if 1 then local a; else local b; end +if 1 then elseif 2 -- FAIL +if 1 then elseif 2 then -- FAIL +if 1 then elseif 2 then end +if 1 then local a elseif 2 then local b end +if 1 then local a; elseif 2 then local b; end +if 1 then elseif 2 then else end +if 1 then else if 2 then end end +if 1 then else if 2 then end -- FAIL +if 1 then break end -- FAIL +if 1 then return end +if 1 then return return end -- FAIL +if 1 then end; if 1 then end; +function -- FAIL +function 1 -- FAIL +function end -- FAIL +function a -- FAIL +function a end -- FAIL +function a( end -- FAIL +function a() end +function a(1 -- FAIL +function a("foo" -- FAIL +function a(p -- FAIL +function a(p,) -- FAIL +function a(p q -- FAIL +function a(p) end +function a(p,q,) end -- FAIL +function a(p,q,r) end +function a(p,q,1 -- FAIL +function a(p) do -- FAIL +function a(p) 1 end -- FAIL +function a(p) return end +function a(p) break end -- FAIL +function a(p) return return end -- FAIL +function a(p) do end end +function a.( -- FAIL +function a.1 -- FAIL +function a.b() end +function a.b, -- FAIL +function a.b.( -- FAIL +function a.b.c.d() end +function a: -- FAIL +function a:1 -- FAIL +function a:b() end +function a:b: -- FAIL +function a:b. -- FAIL +function a.b.c:d() end +function a(...) end +function a(..., -- FAIL +function a(p,...) end +function a(p,q,r,...) end +function a() local a local b end +function a() local a; local b; end +function a() end; function a() end; +local function -- FAIL +local function 1 -- FAIL +local function end -- FAIL +local function a -- FAIL +local function a end -- FAIL +local function a( end -- FAIL +local function a() end +local function a(1 -- FAIL +local function a("foo" -- FAIL +local function a(p -- FAIL +local function a(p,) -- FAIL +local function a(p q -- FAIL +local function a(p) end +local function a(p,q,) end -- FAIL +local function a(p,q,r) end +local function a(p,q,1 -- FAIL +local function a(p) do -- FAIL +local function a(p) 1 end -- FAIL +local function a(p) return end +local function a(p) break end -- FAIL +local function a(p) return return end -- FAIL +local function a(p) do end end +local function a. -- FAIL +local function a: -- FAIL +local function a(...) end +local function a(..., -- FAIL +local function a(p,...) end +local function a(p,q,r,...) end +local function a() local a local b end +local function a() local a; local b; end +local function a() end; local function a() end; +a -- FAIL +a, -- FAIL +a,b,c -- FAIL +a,b = -- FAIL +a = 1 +a = 1,2,3 +a,b,c = 1 +a,b,c = 1,2,3 +a.b = 1 +a.b.c = 1 +a[b] = 1 +a[b][c] = 1 +a.b[c] = 1 +a[b].c = 1 +0 = -- FAIL +"foo" = -- FAIL +true = -- FAIL +(a) = -- FAIL +{} = -- FAIL +a:b() = -- FAIL +a() = -- FAIL +a.b:c() = -- FAIL +a[b]() = -- FAIL +a = a b -- FAIL +a = 1 2 -- FAIL +a = a = 1 -- FAIL +a( -- FAIL +a() +a(1) +a(1,) -- FAIL +a(1,2,3) +1() -- FAIL +a()() +a.b() +a[b]() +a.1 -- FAIL +a.b -- FAIL +a[b] -- FAIL +a.b.( -- FAIL +a.b.c() +a[b][c]() +a[b].c() +a.b[c]() +a:b() +a:b -- FAIL +a:1 -- FAIL +a.b:c() +a[b]:c() +a:b: -- FAIL +a:b():c() +a:b().c[d]:e() +a:b()[c].d:e() +(a)() +()() -- FAIL +(1)() +("foo")() +(true)() +(a)()() +(a.b)() +(a[b])() +(a).b() +(a)[b]() +(a):b() +(a).b[c]:d() +(a)[b].c:d() +(a):b():c() +(a):b().c[d]:e() +(a):b()[c].d:e() +a"foo" +a[[foo]] +a.b"foo" +a[b]"foo" +a:b"foo" +a{} +a.b{} +a[b]{} +a:b{} +a()"foo" +a"foo"() +a"foo".b() +a"foo"[b]() +a"foo":c() +a"foo""bar" +a"foo"{} +(a):b"foo".c[d]:e"bar" +(a):b"foo"[c].d:e"bar" +a(){} +a{}() +a{}.b() +a{}[b]() +a{}:c() +a{}"foo" +a{}{} +(a):b{}.c[d]:e{} +(a):b{}[c].d:e{} +a = -- FAIL +a = a +a = nil +a = false +a = 1 +a = "foo" +a = [[foo]] +a = {} +a = (a) +a = (nil) +a = (true) +a = (1) +a = ("foo") +a = ([[foo]]) +a = ({}) +a = a.b +a = a.b. -- FAIL +a = a.b.c +a = a:b -- FAIL +a = a[b] +a = a[1] +a = a["foo"] +a = a[b][c] +a = a.b[c] +a = a[b].c +a = (a)[b] +a = (a).c +a = () -- FAIL +a = a() +a = a.b() +a = a[b]() +a = a:b() +a = (a)() +a = (a).b() +a = (a)[b]() +a = (a):b() +a = a"foo" +a = a{} +a = function -- FAIL +a = function 1 -- FAIL +a = function a -- FAIL +a = function end -- FAIL +a = function( -- FAIL +a = function() end +a = function(1 -- FAIL +a = function(p) end +a = function(p,) -- FAIL +a = function(p q -- FAIL +a = function(p,q,r) end +a = function(p,q,1 -- FAIL +a = function(...) end +a = function(..., -- FAIL +a = function(p,...) end +a = function(p,q,r,...) end +a = ... +a = a, b, ... +a = (...) +a = ..., 1, 2 +a = function() return ... end -- FAIL +a = -10 +a = -"foo" +a = -a +a = -nil +a = -true +a = -{} +a = -function() end +a = -a() +a = -(a) +a = - -- FAIL +a = not 10 +a = not "foo" +a = not a +a = not nil +a = not true +a = not {} +a = not function() end +a = not a() +a = not (a) +a = not -- FAIL +a = #10 +a = #"foo" +a = #a +a = #nil +a = #true +a = #{} +a = #function() end +a = #a() +a = #(a) +a = # -- FAIL +a = 1 + 2; a = 1 - 2 +a = 1 * 2; a = 1 / 2 +a = 1 ^ 2; a = 1 % 2 +a = 1 .. 2 +a = 1 + -- FAIL +a = 1 .. -- FAIL +a = 1 * / -- FAIL +a = 1 + -2; a = 1 - -2 +a = 1 * - -- FAIL +a = 1 * not 2; a = 1 / not 2 +a = 1 / not -- FAIL +a = 1 * #"foo"; a = 1 / #"foo" +a = 1 / # -- FAIL +a = 1 + 2 - 3 * 4 / 5 % 6 ^ 7 +a = ((1 + 2) - 3) * (4 / (5 % 6 ^ 7)) +a = (1 + (2 - (3 * (4 / (5 % 6 ^ ((7))))))) +a = ((1 -- FAIL +a = ((1 + 2) -- FAIL +a = 1) -- FAIL +a = a + b - c +a = "foo" + "bar" +a = "foo".."bar".."baz" +a = true + false - nil +a = {} * {} +a = function() end / function() end +a = a() ^ b() +a = ... % ... +a = 1 == 2; a = 1 ~= 2 +a = 1 < 2; a = 1 <= 2 +a = 1 > 2; a = 1 >= 2 +a = 1 < 2 < 3 +a = 1 >= 2 >= 3 +a = 1 == -- FAIL +a = ~= 2 -- FAIL +a = "foo" == "bar" +a = "foo" > "bar" +a = a ~= b +a = true == false +a = 1 and 2; a = 1 or 2 +a = 1 and -- FAIL +a = or 1 -- FAIL +a = 1 and 2 and 3 +a = 1 or 2 or 3 +a = 1 and 2 or 3 +a = a and b or c +a = a() and (b)() or c.d +a = "foo" and "bar" +a = true or false +a = {} and {} or {} +a = (1) and ("foo") or (nil) +a = function() end == function() end +a = function() end or function() end +a = { -- FAIL +a = {} +a = {,} -- FAIL +a = {;} -- FAIL +a = {,,} -- FAIL +a = {;;} -- FAIL +a = {{ -- FAIL +a = {{{}}} +a = {{},{},{{}},} +a = { 1 } +a = { 1, } +a = { 1; } +a = { 1, 2 } +a = { a, b, c, } +a = { true; false, nil; } +a = { a.b, a[b]; a:c(), } +a = { 1 + 2, a > b, "a" or "b" } +a = { a=1, } +a = { a=1, b="foo", c=nil } +a = { a -- FAIL +a = { a= -- FAIL +a = { a=, -- FAIL +a = { a=; -- FAIL +a = { 1, a="foo" -- FAIL +a = { 1, a="foo"; b={}, d=true; } +a = { [ -- FAIL +a = { [1 -- FAIL +a = { [1] -- FAIL +a = { [a]= -- FAIL +a = { ["foo"]="bar" } +a = { [1]=a, [2]=b, } +a = { true, a=1; ["foo"]="bar", } \ No newline at end of file diff --git a/lib/LuaMinify/tests/test_minifier.lua b/lib/LuaMinify/tests/test_minifier.lua new file mode 100644 index 0000000..666961e --- /dev/null +++ b/lib/LuaMinify/tests/test_minifier.lua @@ -0,0 +1,61 @@ +-- Adapted from Yueliang + +package.path = "../?.lua;" .. package.path +local util = require'Util' +local Parser = require'ParseLua' +local Format_Mini = require'FormatMini' +local line_nr = 0 + +for w in io.lines("test_lines.txt") do + line_nr = line_nr + 1 + --print(w) + local success, ast = Parser.ParseLua(w) + if w:find("FAIL") then + --[[if success then + print("ERROR PARSING LINE:") + print("Should fail: true. Did fail: " .. tostring(not success)) + print("Line: " .. w) + else + --print("Suceeded!") + end]] + else + if not success then + print("ERROR PARSING LINE:") + print("Should fail: false. Did fail: " .. tostring(not success)) + print("Line: " .. w) + else + success, ast = Format_Mini(ast) + --print(success, ast) + if not success then + print("ERROR MINIFYING LINE:") + print("Message: " .. ast) + print("Line: " .. w) + end + success, ast = loadstring(success) + if not success then + print("ERROR PARSING MINIFIED LINE:") + print("Message: " .. ast) + print("Line nr: " .. line_nr) + print("Line: " .. w) + end + --print("Suceeded!") + end + end +end +print"Done!" +os.remove("tmp") + +--[[ +function readAll(file) + local f = io.open(file, "rb") + local content = f:read("*all") + f:close() + return content +end + +local text = readAll('../ParseLua.lua') +local success, ast = Parser.ParseLua(text) +local nice +nice = Format_Mini(ast) +print(nice) +--]] diff --git a/lib/LuaMinify/tests/test_parser.lua b/lib/LuaMinify/tests/test_parser.lua new file mode 100644 index 0000000..000fd01 --- /dev/null +++ b/lib/LuaMinify/tests/test_parser.lua @@ -0,0 +1,561 @@ +-- Adapted from Yueliang + +local source = [=[ +; -- FAIL +local -- FAIL +local; -- FAIL +local = -- FAIL +local end -- FAIL +local a +local a; +local a, b, c +local a; local b local c; +local a = 1 +local a local b = a +local a, b = 1, 2 +local a, b, c = 1, 2, 3 +local a, b, c = 1 +local a = 1, 2, 3 +local a, local -- FAIL +local 1 -- FAIL +local "foo" -- FAIL +local a = local -- FAIL +local a, b, = -- FAIL +local a, b = 1, local -- FAIL +local a, b = , local -- FAIL +do -- FAIL +end -- FAIL +do end +do ; end -- FAIL +do 1 end -- FAIL +do "foo" end -- FAIL +do local a, b end +do local a local b end +do local a; local b; end +do local a = 1 end +do do end end +do do end; end +do do do end end end +do do do end; end; end +do do do return end end end +do end do -- FAIL +do end end -- FAIL +do return end +do return return end -- FAIL +do break end -- FAIL +while -- FAIL +while do -- FAIL +while = -- FAIL +while 1 do -- FAIL +while 1 do end +while 1 do local a end +while 1 do local a local b end +while 1 do local a; local b; end +while 1 do 2 end -- FAIL +while 1 do "foo" end -- FAIL +while true do end +while 1 do ; end -- FAIL +while 1 do while -- FAIL +while 1 end -- FAIL +while 1 2 do -- FAIL +while 1 = 2 do -- FAIL +while 1 do return end +while 1 do return return end -- FAIL +while 1 do do end end +while 1 do do return end end +while 1 do break end +while 1 do break break end -- FAIL +while 1 do do break end end +repeat -- FAIL +repeat until -- FAIL +repeat until 0 +repeat until false +repeat until local -- FAIL +repeat end -- FAIL +repeat 1 -- FAIL +repeat = -- FAIL +repeat local a until 1 +repeat local a local b until 0 +repeat local a; local b; until 0 +repeat ; until 1 -- FAIL +repeat 2 until 1 -- FAIL +repeat "foo" until 1 -- FAIL +repeat return until 0 +repeat return return until 0 -- FAIL +repeat break until 0 +repeat break break until 0 -- FAIL +repeat do end until 0 +repeat do return end until 0 +repeat do break end until 0 +for -- FAIL +for do -- FAIL +for end -- FAIL +for 1 -- FAIL +for a -- FAIL +for true -- FAIL +for a, in -- FAIL +for a in -- FAIL +for a do -- FAIL +for a in do -- FAIL +for a in b do -- FAIL +for a in b end -- FAIL +for a in b, do -- FAIL +for a in b do end +for a in b do local a local b end +for a in b do local a; local b; end +for a in b do 1 end -- FAIL +for a in b do "foo" end -- FAIL +for a b in -- FAIL +for a, b, c in p do end +for a, b, c in p, q, r do end +for a in 1 do end +for a in true do end +for a in "foo" do end +for a in b do break end +for a in b do break break end -- FAIL +for a in b do return end +for a in b do return return end -- FAIL +for a in b do do end end +for a in b do do break end end +for a in b do do return end end +for = -- FAIL +for a = -- FAIL +for a, b = -- FAIL +for a = do -- FAIL +for a = 1, do -- FAIL +for a = p, q, do -- FAIL +for a = p q do -- FAIL +for a = b do end -- FAIL +for a = 1, 2, 3, 4 do end -- FAIL +for a = p, q do end +for a = 1, 2 do end +for a = 1, 2 do local a local b end +for a = 1, 2 do local a; local b; end +for a = 1, 2 do 3 end -- FAIL +for a = 1, 2 do "foo" end -- FAIL +for a = p, q, r do end +for a = 1, 2, 3 do end +for a = p, q do break end +for a = p, q do break break end -- FAIL +for a = 1, 2 do return end +for a = 1, 2 do return return end -- FAIL +for a = p, q do do end end +for a = p, q do do break end end +for a = p, q do do return end end +break -- FAIL +return +return; +return return -- FAIL +return 1 +return local -- FAIL +return "foo" +return 1, -- FAIL +return 1,2,3 +return a,b,c,d +return 1,2; +return ... +return 1,a,... +if -- FAIL +elseif -- FAIL +else -- FAIL +then -- FAIL +if then -- FAIL +if 1 -- FAIL +if 1 then -- FAIL +if 1 else -- FAIL +if 1 then else -- FAIL +if 1 then elseif -- FAIL +if 1 then end +if 1 then local a end +if 1 then local a local b end +if 1 then local a; local b; end +if 1 then else end +if 1 then local a else local b end +if 1 then local a; else local b; end +if 1 then elseif 2 -- FAIL +if 1 then elseif 2 then -- FAIL +if 1 then elseif 2 then end +if 1 then local a elseif 2 then local b end +if 1 then local a; elseif 2 then local b; end +if 1 then elseif 2 then else end +if 1 then else if 2 then end end +if 1 then else if 2 then end -- FAIL +if 1 then break end -- FAIL +if 1 then return end +if 1 then return return end -- FAIL +if 1 then end; if 1 then end; +function -- FAIL +function 1 -- FAIL +function end -- FAIL +function a -- FAIL +function a end -- FAIL +function a( end -- FAIL +function a() end +function a(1 -- FAIL +function a("foo" -- FAIL +function a(p -- FAIL +function a(p,) -- FAIL +function a(p q -- FAIL +function a(p) end +function a(p,q,) end -- FAIL +function a(p,q,r) end +function a(p,q,1 -- FAIL +function a(p) do -- FAIL +function a(p) 1 end -- FAIL +function a(p) return end +function a(p) break end -- FAIL +function a(p) return return end -- FAIL +function a(p) do end end +function a.( -- FAIL +function a.1 -- FAIL +function a.b() end +function a.b, -- FAIL +function a.b.( -- FAIL +function a.b.c.d() end +function a: -- FAIL +function a:1 -- FAIL +function a:b() end +function a:b: -- FAIL +function a:b. -- FAIL +function a.b.c:d() end +function a(...) end +function a(..., -- FAIL +function a(p,...) end +function a(p,q,r,...) end +function a() local a local b end +function a() local a; local b; end +function a() end; function a() end; +local function -- FAIL +local function 1 -- FAIL +local function end -- FAIL +local function a -- FAIL +local function a end -- FAIL +local function a( end -- FAIL +local function a() end +local function a(1 -- FAIL +local function a("foo" -- FAIL +local function a(p -- FAIL +local function a(p,) -- FAIL +local function a(p q -- FAIL +local function a(p) end +local function a(p,q,) end -- FAIL +local function a(p,q,r) end +local function a(p,q,1 -- FAIL +local function a(p) do -- FAIL +local function a(p) 1 end -- FAIL +local function a(p) return end +local function a(p) break end -- FAIL +local function a(p) return return end -- FAIL +local function a(p) do end end +local function a. -- FAIL +local function a: -- FAIL +local function a(...) end +local function a(..., -- FAIL +local function a(p,...) end +local function a(p,q,r,...) end +local function a() local a local b end +local function a() local a; local b; end +local function a() end; local function a() end; +a -- FAIL +a, -- FAIL +a,b,c -- FAIL +a,b = -- FAIL +a = 1 +a = 1,2,3 +a,b,c = 1 +a,b,c = 1,2,3 +a.b = 1 +a.b.c = 1 +a[b] = 1 +a[b][c] = 1 +a.b[c] = 1 +a[b].c = 1 +0 = -- FAIL +"foo" = -- FAIL +true = -- FAIL +(a) = -- FAIL +{} = -- FAIL +a:b() = -- FAIL +a() = -- FAIL +a.b:c() = -- FAIL +a[b]() = -- FAIL +a = a b -- FAIL +a = 1 2 -- FAIL +a = a = 1 -- FAIL +a( -- FAIL +a() +a(1) +a(1,) -- FAIL +a(1,2,3) +1() -- FAIL +a()() +a.b() +a[b]() +a.1 -- FAIL +a.b -- FAIL +a[b] -- FAIL +a.b.( -- FAIL +a.b.c() +a[b][c]() +a[b].c() +a.b[c]() +a:b() +a:b -- FAIL +a:1 -- FAIL +a.b:c() +a[b]:c() +a:b: -- FAIL +a:b():c() +a:b().c[d]:e() +a:b()[c].d:e() +(a)() +()() -- FAIL +(1)() +("foo")() +(true)() +(a)()() +(a.b)() +(a[b])() +(a).b() +(a)[b]() +(a):b() +(a).b[c]:d() +(a)[b].c:d() +(a):b():c() +(a):b().c[d]:e() +(a):b()[c].d:e() +a"foo" +a[[foo]] +a.b"foo" +a[b]"foo" +a:b"foo" +a{} +a.b{} +a[b]{} +a:b{} +a()"foo" +a"foo"() +a"foo".b() +a"foo"[b]() +a"foo":c() +a"foo""bar" +a"foo"{} +(a):b"foo".c[d]:e"bar" +(a):b"foo"[c].d:e"bar" +a(){} +a{}() +a{}.b() +a{}[b]() +a{}:c() +a{}"foo" +a{}{} +(a):b{}.c[d]:e{} +(a):b{}[c].d:e{} +a = -- FAIL +a = a +a = nil +a = false +a = 1 +a = "foo" +a = [[foo]] +a = {} +a = (a) +a = (nil) +a = (true) +a = (1) +a = ("foo") +a = ([[foo]]) +a = ({}) +a = a.b +a = a.b. -- FAIL +a = a.b.c +a = a:b -- FAIL +a = a[b] +a = a[1] +a = a["foo"] +a = a[b][c] +a = a.b[c] +a = a[b].c +a = (a)[b] +a = (a).c +a = () -- FAIL +a = a() +a = a.b() +a = a[b]() +a = a:b() +a = (a)() +a = (a).b() +a = (a)[b]() +a = (a):b() +a = a"foo" +a = a{} +a = function -- FAIL +a = function 1 -- FAIL +a = function a -- FAIL +a = function end -- FAIL +a = function( -- FAIL +a = function() end +a = function(1 -- FAIL +a = function(p) end +a = function(p,) -- FAIL +a = function(p q -- FAIL +a = function(p,q,r) end +a = function(p,q,1 -- FAIL +a = function(...) end +a = function(..., -- FAIL +a = function(p,...) end +a = function(p,q,r,...) end +a = ... +a = a, b, ... +a = (...) +a = ..., 1, 2 +a = function() return ... end -- FAIL +a = -10 +a = -"foo" +a = -a +a = -nil +a = -true +a = -{} +a = -function() end +a = -a() +a = -(a) +a = - -- FAIL +a = not 10 +a = not "foo" +a = not a +a = not nil +a = not true +a = not {} +a = not function() end +a = not a() +a = not (a) +a = not -- FAIL +a = #10 +a = #"foo" +a = #a +a = #nil +a = #true +a = #{} +a = #function() end +a = #a() +a = #(a) +a = # -- FAIL +a = 1 + 2; a = 1 - 2 +a = 1 * 2; a = 1 / 2 +a = 1 ^ 2; a = 1 % 2 +a = 1 .. 2 +a = 1 + -- FAIL +a = 1 .. -- FAIL +a = 1 * / -- FAIL +a = 1 + -2; a = 1 - -2 +a = 1 * - -- FAIL +a = 1 * not 2; a = 1 / not 2 +a = 1 / not -- FAIL +a = 1 * #"foo"; a = 1 / #"foo" +a = 1 / # -- FAIL +a = 1 + 2 - 3 * 4 / 5 % 6 ^ 7 +a = ((1 + 2) - 3) * (4 / (5 % 6 ^ 7)) +a = (1 + (2 - (3 * (4 / (5 % 6 ^ ((7))))))) +a = ((1 -- FAIL +a = ((1 + 2) -- FAIL +a = 1) -- FAIL +a = a + b - c +a = "foo" + "bar" +a = "foo".."bar".."baz" +a = true + false - nil +a = {} * {} +a = function() end / function() end +a = a() ^ b() +a = ... % ... +a = 1 == 2; a = 1 ~= 2 +a = 1 < 2; a = 1 <= 2 +a = 1 > 2; a = 1 >= 2 +a = 1 < 2 < 3 +a = 1 >= 2 >= 3 +a = 1 == -- FAIL +a = ~= 2 -- FAIL +a = "foo" == "bar" +a = "foo" > "bar" +a = a ~= b +a = true == false +a = 1 and 2; a = 1 or 2 +a = 1 and -- FAIL +a = or 1 -- FAIL +a = 1 and 2 and 3 +a = 1 or 2 or 3 +a = 1 and 2 or 3 +a = a and b or c +a = a() and (b)() or c.d +a = "foo" and "bar" +a = true or false +a = {} and {} or {} +a = (1) and ("foo") or (nil) +a = function() end == function() end +a = function() end or function() end +a = { -- FAIL +a = {} +a = {,} -- FAIL +a = {;} -- FAIL +a = {,,} -- FAIL +a = {;;} -- FAIL +a = {{ -- FAIL +a = {{{}}} +a = {{},{},{{}},} +a = { 1 } +a = { 1, } +a = { 1; } +a = { 1, 2 } +a = { a, b, c, } +a = { true; false, nil; } +a = { a.b, a[b]; a:c(), } +a = { 1 + 2, a > b, "a" or "b" } +a = { a=1, } +a = { a=1, b="foo", c=nil } +a = { a -- FAIL +a = { a= -- FAIL +a = { a=, -- FAIL +a = { a=; -- FAIL +a = { 1, a="foo" -- FAIL +a = { 1, a="foo"; b={}, d=true; } +a = { [ -- FAIL +a = { [1 -- FAIL +a = { [1] -- FAIL +a = { [a]= -- FAIL +a = { ["foo"]="bar" } +a = { [1]=a, [2]=b, } +a = { true, a=1; ["foo"]="bar", } +]=] + +package.path = "../?.lua;" .. package.path +local util = require'Util' +local Parser = require'ParseLua' +local Format_Mini = require'FormatMini' + +local f = io.open("tmp", 'wb') +f:write(source) +f:close() +for w in io.lines("tmp") do + --print(w) + local success, ast = Parser.ParseLua(w) + if w:find("FAIL") then + if success then + print("ERROR PARSING LINE:") + print("Should fail: true. Did fail: " .. tostring(not success)) + --print("Message: " .. ast) + print("Line: " .. w) + else + --print("Suceeded!") + end + else + if not success then + print("ERROR PARSING LINE:") + print("Should fail: false. Did fail: " .. tostring(not success)) + print("Message: " .. ast) + print("Line: " .. w) + else + --print("Suceeded!") + end + end +end +print"Done!" +os.remove("tmp") diff --git a/lib/lexer.lua b/lib/lexer.lua deleted file mode 100644 index e3539fe..0000000 --- a/lib/lexer.lua +++ /dev/null @@ -1,480 +0,0 @@ ---[[ -This file is a part of Penlight (set of pure Lua libraries) - https://github.com/stevedonovan/Penlight - -LICENSE : -Copyright (C) 2009 Steve Donovan, David Manura. - -Permission is hereby granted, free of charge, to any person obtaining a copy of -this software and associated documentation files (the "Software"), to deal in the -Software without restriction, including without limitation the rights to use, copy, -modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, -and to permit persons to whom the Software is furnished to do so, subject to the -following conditions: - -The above copyright notice and this permission notice shall be included in all copies -or substantial portions of the Software. - -THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, -INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A -PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT -HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF -CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE -OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. -]] - ---- Lexical scanner for creating a sequence of tokens from text. --- `lexer.scan(s)` returns an iterator over all tokens found in the --- string `s`. This iterator returns two values, a token type string --- (such as 'string' for quoted string, 'iden' for identifier) and the value of the --- token. --- --- Versions specialized for Lua and C are available; these also handle block comments --- and classify keywords as 'keyword' tokens. For example: --- --- > s = 'for i=1,n do' --- > for t,v in lexer.lua(s) do print(t,v) end --- keyword for --- iden i --- = = --- number 1 --- , , --- iden n --- keyword do --- --- See the Guide for further @{06-data.md.Lexical_Scanning|discussion} --- @module pl.lexer - -local yield,wrap = coroutine.yield,coroutine.wrap -local strfind = string.find -local strsub = string.sub -local append = table.insert - -local function assert_arg(idx,val,tp) - if type(val) ~= tp then - error("argument "..idx.." must be "..tp, 2) - end -end - -local lexer = {} - -local NUMBER1 = '^[%+%-]?%d+%.?%d*[eE][%+%-]?%d+' -local NUMBER2 = '^[%+%-]?%d+%.?%d*' -local NUMBER3 = '^0x[%da-fA-F]+' -local NUMBER4 = '^%d+%.?%d*[eE][%+%-]?%d+' -local NUMBER5 = '^%d+%.?%d*' -local IDEN = '^[%a_][%w_]*' -local WSPACE = '^%s+' -local STRING0 = [[^(['\"]).-\\%1]] -local STRING1 = [[^(['\"]).-[^\]%1]] -local STRING3 = "^((['\"])%2)" -- empty string -local PREPRO = '^#.-[^\\]\n' - -local plain_matches,lua_matches,cpp_matches,lua_keyword,cpp_keyword - -local function tdump(tok) - return yield(tok,tok) -end - -local function ndump(tok,options) - if options and options.number then - tok = tonumber(tok) - end - return yield("number",tok) -end - --- regular strings, single or double quotes; usually we want them --- without the quotes -local function sdump(tok,options) - if options and options.string then - tok = tok:sub(2,-2) - end - return yield("string",tok) -end - --- long Lua strings need extra work to get rid of the quotes -local function sdump_l(tok,options) - if options and options.string then - tok = tok:sub(3,-3) - end - return yield("string",tok) -end - -local function chdump(tok,options) - if options and options.string then - tok = tok:sub(2,-2) - end - return yield("char",tok) -end - -local function cdump(tok) - return yield('comment',tok) -end - -local function wsdump (tok) - return yield("space",tok) -end - -local function pdump (tok) - return yield('prepro',tok) -end - -local function plain_vdump(tok) - return yield("iden",tok) -end - -local function lua_vdump(tok) - if lua_keyword[tok] then - return yield("keyword",tok) - else - return yield("iden",tok) - end -end - -local function cpp_vdump(tok) - if cpp_keyword[tok] then - return yield("keyword",tok) - else - return yield("iden",tok) - end -end - ---- create a plain token iterator from a string or file-like object. --- @param s the string --- @param matches an optional match table (set of pattern-action pairs) --- @param filter a table of token types to exclude, by default {space=true} --- @param options a table of options; by default, {number=true,string=true}, --- which means convert numbers and strip string quotes. -function lexer.scan (s,matches,filter,options) - --assert_arg(1,s,'string') - local file = type(s) ~= 'string' and s - filter = filter or {space=true} - options = options or {number=true,string=true} - if filter then - if filter.space then filter[wsdump] = true end - if filter.comments then - filter[cdump] = true - end - end - if not matches then - if not plain_matches then - plain_matches = { - {WSPACE,wsdump}, - {NUMBER3,ndump}, - {IDEN,plain_vdump}, - {NUMBER1,ndump}, - {NUMBER2,ndump}, - {STRING3,sdump}, - {STRING0,sdump}, - {STRING1,sdump}, - {'^.',tdump} - } - end - matches = plain_matches - end - local function lex () - local i1,i2,idx,res1,res2,tok,pat,fun,capt - local line = 1 - if file then s = file:read()..'\n' end - local sz = #s - local idx = 1 - --print('sz',sz) - while true do - for _,m in ipairs(matches) do - pat = m[1] - fun = m[2] - i1,i2 = strfind(s,pat,idx) - if i1 then - tok = strsub(s,i1,i2) - idx = i2 + 1 - if not (filter and filter[fun]) then - lexer.finished = idx > sz - res1,res2 = fun(tok,options) - end - if res1 then - local tp = type(res1) - -- insert a token list - if tp=='table' then - yield('','') - for _,t in ipairs(res1) do - yield(t[1],t[2]) - end - elseif tp == 'string' then -- or search up to some special pattern - i1,i2 = strfind(s,res1,idx) - if i1 then - tok = strsub(s,i1,i2) - idx = i2 + 1 - yield('',tok) - else - yield('','') - idx = sz + 1 - end - --if idx > sz then return end - else - yield(line,idx) - end - end - if idx > sz then - if file then - --repeat -- next non-empty line - line = line + 1 - s = file:read() - if not s then return end - --until not s:match '^%s*$' - s = s .. '\n' - idx ,sz = 1,#s - break - else - return - end - else break end - end - end - end - end - return wrap(lex) -end - -local function isstring (s) - return type(s) == 'string' -end - ---- insert tokens into a stream. --- @param tok a token stream --- @param a1 a string is the type, a table is a token list and --- a function is assumed to be a token-like iterator (returns type & value) --- @param a2 a string is the value -function lexer.insert (tok,a1,a2) - if not a1 then return end - local ts - if isstring(a1) and isstring(a2) then - ts = {{a1,a2}} - elseif type(a1) == 'function' then - ts = {} - for t,v in a1() do - append(ts,{t,v}) - end - else - ts = a1 - end - tok(ts) -end - ---- get everything in a stream upto a newline. --- @param tok a token stream --- @return a string -function lexer.getline (tok) - local t,v = tok('.-\n') - return v -end - ---- get current line number.
--- Only available if the input source is a file-like object. --- @param tok a token stream --- @return the line number and current column -function lexer.lineno (tok) - return tok(0) -end - ---- get the rest of the stream. --- @param tok a token stream --- @return a string -function lexer.getrest (tok) - local t,v = tok('.+') - return v -end - ---- get the Lua keywords as a set-like table. --- So res["and"] etc would be true. --- @return a table -function lexer.get_keywords () - if not lua_keyword then - lua_keyword = { - ["and"] = true, ["break"] = true, ["do"] = true, - ["else"] = true, ["elseif"] = true, ["end"] = true, - ["false"] = true, ["for"] = true, ["function"] = true, - ["if"] = true, ["in"] = true, ["local"] = true, ["nil"] = true, - ["not"] = true, ["or"] = true, ["repeat"] = true, - ["return"] = true, ["then"] = true, ["true"] = true, - ["until"] = true, ["while"] = true - } - end - return lua_keyword -end - - ---- create a Lua token iterator from a string or file-like object. --- Will return the token type and value. --- @param s the string --- @param filter a table of token types to exclude, by default {space=true,comments=true} --- @param options a table of options; by default, {number=true,string=true}, --- which means convert numbers and strip string quotes. -function lexer.lua(s,filter,options) - filter = filter or {space=true,comments=true} - lexer.get_keywords() - if not lua_matches then - lua_matches = { - {WSPACE,wsdump}, - {NUMBER3,ndump}, - {IDEN,lua_vdump}, - {NUMBER4,ndump}, - {NUMBER5,ndump}, - {STRING3,sdump}, - {STRING0,sdump}, - {STRING1,sdump}, - {'^%-%-%[%[.-%]%]',cdump}, - {'^%-%-.-\n',cdump}, - {'^%[%[.-%]%]',sdump_l}, - {'^==',tdump}, - {'^~=',tdump}, - {'^<=',tdump}, - {'^>=',tdump}, - {'^%.%.%.',tdump}, - {'^%.%.',tdump}, - {'^.',tdump} - } - end - return lexer.scan(s,lua_matches,filter,options) -end - ---- create a C/C++ token iterator from a string or file-like object. --- Will return the token type type and value. --- @param s the string --- @param filter a table of token types to exclude, by default {space=true,comments=true} --- @param options a table of options; by default, {number=true,string=true}, --- which means convert numbers and strip string quotes. -function lexer.cpp(s,filter,options) - filter = filter or {comments=true} - if not cpp_keyword then - cpp_keyword = { - ["class"] = true, ["break"] = true, ["do"] = true, ["sizeof"] = true, - ["else"] = true, ["continue"] = true, ["struct"] = true, - ["false"] = true, ["for"] = true, ["public"] = true, ["void"] = true, - ["private"] = true, ["protected"] = true, ["goto"] = true, - ["if"] = true, ["static"] = true, ["const"] = true, ["typedef"] = true, - ["enum"] = true, ["char"] = true, ["int"] = true, ["bool"] = true, - ["long"] = true, ["float"] = true, ["true"] = true, ["delete"] = true, - ["double"] = true, ["while"] = true, ["new"] = true, - ["namespace"] = true, ["try"] = true, ["catch"] = true, - ["switch"] = true, ["case"] = true, ["extern"] = true, - ["return"] = true,["default"] = true,['unsigned'] = true,['signed'] = true, - ["union"] = true, ["volatile"] = true, ["register"] = true,["short"] = true, - } - end - if not cpp_matches then - cpp_matches = { - {WSPACE,wsdump}, - {PREPRO,pdump}, - {NUMBER3,ndump}, - {IDEN,cpp_vdump}, - {NUMBER4,ndump}, - {NUMBER5,ndump}, - {STRING3,sdump}, - {STRING1,chdump}, - {'^//.-\n',cdump}, - {'^/%*.-%*/',cdump}, - {'^==',tdump}, - {'^!=',tdump}, - {'^<=',tdump}, - {'^>=',tdump}, - {'^->',tdump}, - {'^&&',tdump}, - {'^||',tdump}, - {'^%+%+',tdump}, - {'^%-%-',tdump}, - {'^%+=',tdump}, - {'^%-=',tdump}, - {'^%*=',tdump}, - {'^/=',tdump}, - {'^|=',tdump}, - {'^%^=',tdump}, - {'^::',tdump}, - {'^.',tdump} - } - end - return lexer.scan(s,cpp_matches,filter,options) -end - ---- get a list of parameters separated by a delimiter from a stream. --- @param tok the token stream --- @param endtoken end of list (default ')'). Can be '\n' --- @param delim separator (default ',') --- @return a list of token lists. -function lexer.get_separated_list(tok,endtoken,delim) - endtoken = endtoken or ')' - delim = delim or ',' - local parm_values = {} - local level = 1 -- used to count ( and ) - local tl = {} - local function tappend (tl,t,val) - val = val or t - append(tl,{t,val}) - end - local is_end - if endtoken == '\n' then - is_end = function(t,val) - return t == 'space' and val:find '\n' - end - else - is_end = function (t) - return t == endtoken - end - end - local token,value - while true do - token,value=tok() - if not token then return nil,'EOS' end -- end of stream is an error! - if is_end(token,value) and level == 1 then - append(parm_values,tl) - break - elseif token == '(' then - level = level + 1 - tappend(tl,'(') - elseif token == ')' then - level = level - 1 - if level == 0 then -- finished with parm list - append(parm_values,tl) - break - else - tappend(tl,')') - end - elseif token == delim and level == 1 then - append(parm_values,tl) -- a new parm - tl = {} - else - tappend(tl,token,value) - end - end - return parm_values,{token,value} -end - ---- get the next non-space token from the stream. --- @param tok the token stream. -function lexer.skipws (tok) - local t,v = tok() - while t == 'space' do - t,v = tok() - end - return t,v -end - -local skipws = lexer.skipws - ---- get the next token, which must be of the expected type. --- Throws an error if this type does not match! --- @param tok the token stream --- @param expected_type the token type --- @param no_skip_ws whether we should skip whitespace -function lexer.expecting (tok,expected_type,no_skip_ws) - assert_arg(1,tok,'function') - assert_arg(2,expected_type,'string') - local t,v - if no_skip_ws then - t,v = tok() - else - t,v = skipws(tok) - end - if t ~= expected_type then error ("expecting "..expected_type,2) end - return v -end - -return lexer diff --git a/lib/table.lua b/lib/table.lua index 6711ba3..aa2771e 100644 --- a/lib/table.lua +++ b/lib/table.lua @@ -1,8 +1,8 @@ --[[ -Lua table utilities by Thomas99. +Table utility by Thomas99. LICENSE : -Copyright (c) 2014 Thomas99 +Copyright (c) 2015 Thomas99 This software is provided 'as-is', without any express or implied warranty. In no event will the authors be held liable for any damages arising from the @@ -12,17 +12,24 @@ Permission is granted to anyone to use this software for any purpose, including commercial applications, and to alter it and redistribute it freely, subject to the following restrictions: - 1. The origin of this software must not be misrepresented; you must not - claim that you wrote the original software. If you use this software in a - product, an acknowledgment in the product documentation would be appreciated - but is not required. + 1. The origin of this software must not be misrepresented; you must not + claim that you wrote the original software. If you use this software in a + product, an acknowledgment in the product documentation would be appreciated + but is not required. - 2. Altered source versions must be plainly marked as such, and must not be - misrepresented as being the original software. + 2. Altered source versions must be plainly marked as such, and must not be + misrepresented as being the original software. - 3. This notice may not be removed or altered from any source distribution. + 3. This notice may not be removed or altered from any source distribution. ]] +-- Diverses fonctions en rapport avec les tables. +-- v0.1.0 +-- +-- Changements : +-- - v0.1.0 : +-- Première version versionnée. Il a dû se passer des trucs avant mais j'ai pas noté :p + -- Copie récursivement la table t dans la table dest (ou une table vide si non précisé) et la retourne -- replace (false) : indique si oui ou non, les clefs existant déjà dans dest doivent être écrasées par celles de t -- metatable (true) : copier ou non également les metatables @@ -79,16 +86,6 @@ function table.isIn(table, value) return false end --- retourne true si la clé key est dans la table -function table.hasKey(table, key) - for k,_ in pairs(table) do - if k == key then - return true - end - end - return false -end - -- retourne la longueur exacte d'une table (fonctionne sur les tables à clef) function table.len(t) local len=0 diff --git a/lune.lune b/lune.lune deleted file mode 100644 index 7348c4b..0000000 --- a/lune.lune +++ /dev/null @@ -1,180 +0,0 @@ -#!/usr/bin/lua ---[[ -Lune language & compiler by Thomas99. - -LICENSE : -Copyright (c) 2014 Thomas99 - -This software is provided 'as-is', without any express or implied warranty. -In no event will the authors be held liable for any damages arising from the -use of this software. - -Permission is granted to anyone to use this software for any purpose, including -commercial applications, and to alter it and redistribute it freely, subject -to the following restrictions: - - 1. The origin of this software must not be misrepresented; you must not - claim that you wrote the original software. If you use this software in a - product, an acknowledgment in the product documentation would be appreciated - but is not required. - - 2. Altered source versions must be plainly marked as such, and must not be - misrepresented as being the original software. - - 3. This notice may not be removed or altered from any source distribution. -]] -#include("lib/lexer.lua") -#include("lib/table.lua") - -local lune = {} -lune.VERSION = "0.0.1" -lune.syntax = { - affectation = { ["+"] = "= %s +", ["-"] = "= %s -", ["*"] = "= %s *", ["/"] = "= %s /", - ["^"] = "= %s ^", ["%"] = "= %s %%", [".."] = "= %s .." }, - incrementation = { ["+"] = " = %s + 1" , ["-"] = " = %s - 1" }, -} - --- Preprocessor -function lune.preprocess(input, args) - -- generate preprocessor - local preprocessor = "return function()\n" - - local lines = {} - for line in (input.."\n"):gmatch("(.-)\n") do - table.insert(lines, line) - if line:sub(1,1) == "#" then - -- exclude shebang - if not (line:sub(1,2) == "#!" and #lines ==1) then - preprocessor ..= line:sub(2) .. "\n" - else - preprocessor ..= "output ..= lines[" .. #lines .. "] .. \"\\n\"\n" - end - else - preprocessor ..= "output ..= lines[" .. #lines .. "] .. \"\\n\"\n" - end - end - preprocessor ..= "return output\nend" - - -- make preprocessor environement - local env = table.copy(_G) - env.lune = lune - env.output = "" - env.include = function(file) - local f = io.open(file) - if not f then error("can't open the file to include") end - - local filename = file:match("([^%/%\\]-)%.[^%.]-$") - - env.output ..= - "-- INCLUSION OF FILE \""..file.."\" --\n".. - "local function _()\n".. - f:read("*a").."\n".. - "end\n".. - "local "..filename.." = _() or "..filename.."\n".. - "-- END OF INCLUDSION OF FILE \""..file.."\" --\n" - - f:close() - end - env.rawInclude = function(file) - local f = io.open(file) - if not f then error("can't open the file to raw include") end - env.output ..= f:read("*a").."\n" - f:close() - end - env.print = function(...) - env.output ..= table.concat({...}, "\t") .. "\n" - end - env.args = args or {} - env.lines = lines - - -- load preprocessor - local preprocess, err = load(lune.compile(preprocessor), "Preprocessor", nil, env) - if not preprocess then error("Error while creating preprocessor :\n" .. err) end - - -- execute preprocessor - local success, output = pcall(preprocess()) - if not success then error("Error while preprocessing file :\n" .. output .. "\nWith preprocessor : \n" .. preprocessor) end - - return output -end - --- Compiler -function lune.compile(input) - local output = "" - - local last = {} - for t,v in lexer.lua(input, {}, {}) do - local toInsert = v - - -- affectation - if t == "=" then - if table.hasKey(lune.syntax.affectation, last.token) then - toInsert = string.format(lune.syntax.affectation[last.token], last.varName) - output = output:sub(1, -1 -#last.token) -- remove token before = - end - end - - -- self-incrementation - if table.hasKey(lune.syntax.incrementation, t) and t == last.token then - toInsert = string.format(lune.syntax.incrementation[last.token], last.varName) - output = output:sub(1, -#last.token*2) -- remove token ++/-- - end - - -- reconstitude full variable name (ex : ith.game.camera) - if t == "iden" then - if last.token == "." then - last.varName ..= "." .. v - else - last.varName = v - end - end - - last[t] = v - last.token = t - last.value = v - - output ..= toInsert - end - - return output -end - --- Preprocess & compile -function lune.make(code, args) - local preprocessed = lune.preprocess(code, args or {}) - local output = lune.compile(preprocessed) - return output -end - --- Standalone mode -if debug.getinfo(3) == nil and arg then - -- Check args - if #arg < 1 then - print("Lune version "..lune.VERSION.." by Thomas99") - print("Command-line usage :") - print("lua lune.lua [preprocessor arguments]") - return lune - end - - -- Parse args - local inputFilePath = arg[1] - local args = {} - -- Parse compilation args - for i=2, #arg, 1 do - if arg[i]:sub(1,2) == "--" then - args[arg[i]:sub(3)] = arg[i+1] - i = i +1 -- skip argument value - end - end - - -- Open & read input file - local inputFile, err = io.open(inputFilePath, "r") - if not inputFile then error("Error while opening input file : "..err) end - local input = inputFile:read("*a") - inputFile:close() - - -- End - print(lune.make(input, args)) -end - -return lune \ No newline at end of file diff --git a/tests/test.lua b/tests/test.lua index 9d07983..c0c090d 100644 --- a/tests/test.lua +++ b/tests/test.lua @@ -1,17 +1,17 @@ -print("=====================") -print("|| LUNE TESTS ||") -print("=====================") +print("========================") +print("|| CANDRAN TESTS ||") +print("========================") -local lune = dofile(arg[1] or "../build/lune.lua") +local candran = dofile(arg[1] or "../build/candran.lua") -- test helper local results = {} -- tests result -local function test(name, luneCode, result, args) +local function test(name, candranCode, result, args) results[name] = { result = "not finished", message = "no info" } local self = results[name] -- make code - local success, code = pcall(lune.make, luneCode, args) + local success, code = pcall(candran.make, candranCode, args) if not success then self.result = "error" self.message = "error while making code :\n"..code @@ -70,12 +70,12 @@ test("preprocessor print function", [[ #print("local a = true") return a ]], true) -test("preprocessor include function", [[ -#include("toInclude.lua") -return a +test("preprocessor import function", [[ +#import("toInclude") +return toInclude ]], 5) -test("preprocessor rawInclude function", "a = [[\n#rawInclude('toInclude.lua')\n]]\nreturn a", - "a = 5\n") +test("preprocessor include function", "a = [[\n#include('toInclude.lua')\n]]\nreturn a", + "local a = 5\nreturn a\n") test("+=", [[ local a = 5 @@ -113,16 +113,84 @@ a ..= " world" return a ]], "hello world") -test("++", [[ -local a = 5 -a++ -return a -]], 6) -test("--", [[ -local a = 5 -a-- -return a -]], 4) +test("decorator", [[ +local a = function(func) + local wrapper = function(...) + local b = func(...) + return b + 5 + end + return wrapper +end +@a +function c(nb) + return nb^2 +end +return c(5) +]], 30) +test("decorator with arguments", [[ +local a = function(add) + local b = function(func) + local wrapper = function(...) + local c = func(...) + return c + add + end + return wrapper + end + return b +end +@a(10) +function d(nb) + return nb^2 +end +return d(5) +]], 35) +test("multiple decorators", [[ +local a = function(func) + local wrapper = function(...) + local b = func(...) + return b + 5 + end + return wrapper +end +local c = function(func) + local wrapper = function(...) + local d = func(...) + return d * 2 + end + return wrapper +end +@a +@c +function e(nb) + return nb^2 +end +return e(5) +]], 55) +test("multiple decorators with arguments", [[ +local a = function(func) + local wrapper = function(...) + local b = func(...) + return b + 5 + end + return wrapper +end +local c = function(mul) + local d = function(func) + local wrapper = function(...) + local e = func(...) + return e * mul + end + return wrapper + end + return d +end +@a +@c(3) +function f(nb) + return nb^2 +end +return f(5) +]], 80) -- results print("=====================") diff --git a/tests/toInclude.lua b/tests/toInclude.lua index 2c7b797..fc2ae55 100644 --- a/tests/toInclude.lua +++ b/tests/toInclude.lua @@ -1 +1,2 @@ -a = 5 \ No newline at end of file +local a = 5 +return a \ No newline at end of file