Lavendeux Parser

Extensible inline parser engine, the backend parsing engine for Lavendeux.

Description

lavendeux-parser is an exensible parsing engine for mathematical expressions. It supports variable and function assignments, a variety of datatypes, and can be extended easily at runtime through extensions written in javascript.

Extensions are run in a sandboxed environment with no host or network access. This project is the engine behind Lavendeux.

Programming language: Rust

License: MIT License

Tags: Parser     Parsing     Grammar     Mathematics     Text     Text processing     Javascript    

Latest version: v0.7.1

README

Lavendeux Parser - Extensible inline parser engine

lavendeux-parser is an exensible parsing engine for mathematical expressions. It supports variable and function assignments, a variety of datatypes, and can be extended easily at runtime through extensions written in javascript.

Extensions are run in a sandboxed environment with no host or network access. This project is the engine behind Lavendeux.

Getting Started

To use it, create a ParserState object, and use it to tokenize input with Token::new:

use lavendeux_parser::{ParserState, ParserError, Token, Value};

fn main() -> Result<(), ParserError> {
    // Create a new parser, and tokenize 2 lines
    let mut state : ParserState = ParserState::new();
    let lines = Token::new("x=9\nsqrt(x) @bin", &mut state)?;

    // The resulting token contains the resulting values and text
    assert_eq!(lines.text(), "9\n0b11");
    assert_eq!(lines.child(1).unwrap().value(), Value::Integer(3));

    Ok(())
}

The result will be a Token object:

use lavendeux_parser::{ParserState, ParserError, Token, Value};

fn main() -> Result<(), ParserError> {
    let mut state : ParserState = ParserState::new();
    let lines = Token::new("x=9\nsqrt(x) @bin", &mut state)?;

    // String representation of the full result
    assert_eq!(lines.text(), "9\n0b11"); 

    // String representation of the first line's result
    assert_eq!(lines.child(0).unwrap().text(), "9");

    // Actual value of the first line's result
    // Values are integers, floats, booleans or strings
    let value = lines.child(0).unwrap().value();
    assert_eq!(value.as_int().unwrap(), 9);
    assert_eq!(true, matches!(value, Value::Integer(_)));

    Ok(())
}

A number of functions and @decorators are available for expressions to use - add more using the state:

use lavendeux_parser::{ParserState, ParserError, DecoratorDefinition, FunctionDefinition, FunctionArgument, Value};
use lavendeux_parser::errors::*;

let mut state : ParserState = ParserState::new();
state.decorators.register(DecoratorDefinition {
    name: &["upper", "uppercase"],
    description: "Outputs an uppercase version of the input",
    argument: ExpectedTypes::Any,
    handler: |_, input| Ok(input.as_string().to_uppercase())
});

// Functions take in an array of values, and return a single value
state.functions.register(FunctionDefinition {
    name: "echo",
    description: "Echo back the provided input",
    arguments: || vec![
        FunctionArgument::new_required("input", ExpectedTypes::String),
    ],
    handler: |_, args: &[Value]| {
        Ok(Value::String(args[0].as_string()))
    }
});

// Expressions being parsed can now call new_function(), and use the @new_decorator
```rust
use lavendeux_parser::{ParserState, ParserError, Value, Token};

fn main() -> Result<(), ParserError> {
    let mut state : ParserState = ParserState::new();

    // Load one extension
    state.extensions.load("example_extensions/colour_utils.js")?;

    // Load a whole directory
    state.extensions.load_all("./example_extensions")?;

    // Once loaded, functions and @decorators decribed in the extensions
    // can be called in expressions being parsed
    let token = Token::new("complement(0xFF0000) @colour", &mut state)?;
    assert_eq!(token.text(), "#ffff00");
    Ok(())
}

Using Extensions

Extensions give a more flexible way of adding functionality at runtime. Extensions are written in javascript.

Extensions are enabled by default, and can be excluded by disabling the crate's "extensions" feature

Extensions can be loaded as follows:

use lavendeux_parser::{ParserState, ParserError, Value, Token};

fn main() -> Result<(), ParserError> {
    let mut state : ParserState = ParserState::new();

    // Load one extension
    state.extensions.load("example_extensions/colour_utils.js")?;

    // Load a whole directory
    state.extensions.load_all("./example_extensions")?;

    // Once loaded, functions and @decorators decribed in the extensions
    // can be called in expressions being parsed
    let token = Token::new("complement(0xFF0000) @colour", &mut state)?;
    assert_eq!(token.text(), "#ffff00");
    Ok(())
}

Syntax

Expressions can be composed of integers, floats, strings, as well as numbers of various bases:

// Integer, floating point or scientific notation numbers
5 + 5.56 + .2e+3

// Currency values
// Note that no exchange rate is being applied automatically
$1,000.00 == ¥1,000.00

// Scientific numbers can be represented a number of ways
5.6e+7 - .6E7 + .2e-3

// Booleans
in_range = 5 > 3 && 5 < 10
true || false

// Integers can also be represented in base 2, 8 or 16
0xFFA & 0b110 & 0777

// Strings are also supported
concat("foo", "bar")

// Arrays can be composed of any combination of types
[10, 12] + [1.2, 1.3]
2 * [10, 5] // Operations can also be applied between scalar values and arrays
[false, 0, true] == true // An array evaluates to true if any element is true

Beyond the simpler operators, the following operations are supported:

5 ** 2 // Exponentiation
6 % 2 // Modulo
3! // Factorial

// Bitwise operators AND, OR, and XOR:
0xF & 0xA | 0x2 ^ 0xF

// Bitwise SHIFT, and NOT
0xF << 1
0x1 >> 2
~0xA

// Boolean operators
true || false && true
1 < 2 > 5 // true

You can also assign values to variables to be used later:
They are case sensitive, and can be composed of underscores or alphanumeric characters

// You can also assign values to variables to be used later
x = 0xFFA & 0xFF0
x - 55 // The result will be 200

// A few constants are also pre-defined
value = pi * e * tau

// You can also define functions
f(x) = 2*x**2 + 3*x + 5
f(2.3)

// Functions work well with arrays
sum(a) = element(a, 0) + ( len(a)>1 ? sum(dequeue(a)) : 0 )
sum([10, 10, 11])

// Recursive functions work too!
factorial(x) = x==0 ? 1 : (x * factorial(x - 1) )
factorial(5)

Decorators can be put at the end of a line to change the output format. Valid decorators include:
@bin, @oct, @hex, @int, @float, or @sci

255 @hex // The result will be 0xFF
8 @oct // The result will be 0o10
5 @float // The result will be 5.0
5 @usd // Also works with @dollars @cad, @aud, @yen, @pounds, or @euros
1647950086 @utc // 2022-03-22 11:54:46

The following functions are supported by default:

help() // List all functions and decorators
help("strlen") // Get help for a specific function by name

// String functions
concat("s1", "s2", ...) | strlen("string") | substr("string", start, [length])
uppercase("s1") | lowercase("S1") | trim("    s1    ")

// Regular expressions
regex("foo.*", "foobar") // foobar
regex("foo(.*)", "foobar", 1) // bar

// Array functions
len(a) | is_empty(a)
pop(a) | push(a) | dequeue(a) | enqueue(a)
remove(a, index) | element(a, index)
merge(a1, a2)

// Rounding functions
ceil(n) | floor(n) | round(n, precision)

// Trigonometric functions - values are in radians, use to_radians to convert
tan(r), cos(r), sin(r), atan(r), acos(r), asin(r), tanh(r), cosh(r), sinh(r)

// Rounding functions
ln(n) | log10(n) | log(n, base)
sqrt(n) | root(n, base)

// Typecasting
bool(n) | array(n) | int(n) | float(n)

// RNG functions
choose("argument 1", 2, 3.0, ...) | rand() | rand(min, max)

// Networking functions
get(url, ["header-name=value", ...]) | post(url, ["header-name=value", ...]) | resolve(hostname)

// Misc. functions
to_radians(degree_value) | to_degrees(radian_value) | abs(n) | tail(filename, [lines]) | time()

Lavendeux can be extended with javascript. Extensions are run in a sandboxed environment, with no network or host access.
Below is an example of a simple extension:

/**
* This function tells Lavendeux about this extension.
* It must return an object similar to the one below.
* @returns Object
*/
function extension() }
    return {
        name: "Extension Name",
        author: "Author's name",
        version: "0.0.0",

        functions: {,
            "callable_name": "js_function_name"
        },

        decorator: {,
            "callable_name": "js_decorator_name"
        },
    }
}

/**
* This function can be called from Lavendeux as callable_name(...)
* args is an array of value objects with either the key Integer, Float or String
* It must also return an object of that kind, or throw an exception
* @returns Object
*/
function js_function_name(args) }
    return {
        "Integer": 5,
    };
}

/**
* This decorator can be called from Lavendeux as @callable_name
* arg is a value object with either the key Integer, Float or String
* It must return a string, or throw an exception
* @returns String
*/
function js_decorator_name(arg) {
    return "formatted value";
}