Jokolay And Plugins / Modules / Scripting / Addons / Extensions

[coderedart]

Considering blish already supports Modules, This feature might be requested eventually.

Short Answer: NO. no support for any sort of addons. it is too big in scope and we will consider it after getting the basic overlay stuff working.
Long Answer: Maybe. scripting will help add features easily when they are lightweight. but

1. exposing it for plugins means we need to stabilize an external api and that should not be done hastily, or we will eventually be trapped in backwards compatibility.
2. anyone can make plugins and as we cannot be checking the quality of the plugin, it will be a bad user experience. i know that its not Jokolay's fault for the plugin crash, but we are enabling bad plugins to exist.
3. The most important reason is that we would need to select a language runtime/external api and it is a hard place to be.

Lua obviously makes sense. its popular in gamedev / game modding communitites, has a lot of libraries, fast af. but lifetimes are a pain and rust doesn't have smooth integration with lua yet.
Javascript. nothing needs to be said. every programmer and their mother know this. will be heavily popular with plugin creators. but package management will probably be a little painful to integrate. again, no smooth integration with rust yet.
Python. not as fast as js or lua. but has an okayish integration with python. has lots of libraries and will be easy for newbies that want to make plugins.

Finally, the real candidate would be Web Assembly. one advantage is that we now have speed AND the integration would be smooth considering rust support is first class for wasm. But more importantly, we can write wasm modules in any language and have them compile to wasm, and pack it for rust. But exposing the plugin API in wasm might not be an easy task at all.

No matter which language we select, we will need to provide a sample template repo on github, a guide for setting up the dev environment for plugins, finally a package format for use with Jokolay. thinking in these terms, self contained wasm files should be ez to distribute/package. we could just refer to already established guides about wasm dev setup for different languages. especially kotlin/rust/js/lua which already work with wasm.

too many decisions and we didn't even consider what we want to expose in plugin api, or security, or what happens if a plugin is too slow, do we wait or do we just skip rendering it this frame? or what kinda refactoring would we need in Jokolay itself to keep it clean and finally if we want to cooperate and expose similar api like Blish/ or make our own.

[coderedart]

first experiment

we were successfully able to use rhai to script egui and embed it into our app using a little bit of unsafe, but rhai doesn't support providing &mut T types as arguments to host functions, which limits our ability to provide bindings for egui's text_edit or checkbox widgets which take in &mut bool or &mut String. https://github.com/coderedart/rhai_egui
script editing and live reload:
The main hurdle at the moment is the closure based api of egui, as well as the use of ref mut Types as arguments. and ofcourse, the unsafe usage kind makes things sadder.

if it is possible with rhai, it might be possible with mlua too.
another option is deno_core which will have a blogpost guide about embedding it in rust by april 2022 (according to their discord).

wasm might be possible in the near future, you can already do some awesome things. the benefit of wasm is that we can just use rust for plugins too, but the dev-setup becomes a little bit complex for rust devs
to write a plugin in wasm using rust: refer https://docs.wasmtime.dev/wasm-rust.html
we will be using wasm-wasi as target to generate wasm files it seems.

to embed/run these wasm files, we will need to choose between wasmer / wamstime, both of which support cranelift backend for great speed.

irrespective of the language we choose, egui is still too hard to make bindings for at the moment. two problems in particular.

1. exposing the callback api, like Window::new("hello").show(&ctx, |ui| ui.label("something"));. both the Window and Ui are temporary objects and scripting apis like to take values rather than references, and prefer to use Clone types instead.
2. they also don't like giving you mutable strings or such types, so you need to somehow plan for the ui.textedit(&mut String) like apis.

for the ref mut types, we can just expose an explicit api like request_string_input() and we can present a dialog to user so that they can enter the input and when they click Ok, we can present it to the plugin.

OTOH, if we instead just use raw rust cdylibs, and expose a egui bindings crate with a safety abi mentioning the invariants to be upheld, we might be able to get away with a easier (but slightly unsafe) plugin api. the advantage of performance and rust safety would be great.

if these things are hard, we can instead go with a grpc route which is much easier to make plugins for, the hard part would be exposing the egui to them.

An alternative is to instead just make a nodejs with napi-rs / neon bindings (or deno )to our jokolay rust library, that we can just add other plugins as js scripts. we can even use imgui-js as bindings instead of egui, which would make it a lot faster to develop for and push the performance sensitive stuff into rust.

[coderedart]

so, was successfully able to use mlua to bind egui. callbacks work and more importantly, by using the https://docs.rs/mlua/latest/mlua/struct.Lua.html#method.scope we can actually avoid unsafe as lua can take &mut Ui directly /yay.

as for the egui api which takes in things like &mut T, we can simply take a Clone of T and a callback to which we provide the T and lua can decide what to do with it. it seems strings are immutable in rhai as well as mlua anyway.

Egui MLua

lua is looking very attractive at the moment. we will need to check other things (like loading cdylibs), but if it works out, lua might be a really good option to start with.

and we can also look into whether we can use rust as lua dylibs, that would allow us to extract a lot of functionality like markers into an external crate and draw using lua.

after asking around in The Programmer's Hangout discord #lua channel and discussing with Theros#7648 we are roughly looking at this design.

Design 1

1. define a layout for a plugin package. eg: a plugin called watch

watch/
    init.lua // will be loaded by jokolay
    package.json // some kind of manifest like plugin name, version etc.. to use for checking updates
    deps/
        lua/ // will be set as the package.path
        dylib_win/ // will be set as package.cpath if on windows
        dylib_lin/ // will be set as package.cpath if on linux

2. Theros's design was to have the init.lua be something like

return {
  name = "myplugin",
  description = "my cool plugin",
  version = "1.0.0",
  exports = {
    ... -- various "public" plugin methods
  },
  OnLoad = function(context) end,
  OnUnload = function(context) end,
}

the context is just jokolay related data that the plugin needs to initialize its state or destroy its state. the Onload and OnUnload functions are to enable/disable a module and init/cleanup its own state.
the exports field contains the functions and data structures that the plugin wants to expose.
a sample plugin of his as an example

3. inside the exports, we could have a tick() function and a gui_tick() function. first we call gui_tick() with egui context so that the module can add its own user interface, as well as draw anything with wgpu. and then we immediately send plugin to another thread and call tick() function. and next frame, we wait for plugin's tick to be done and repeat. so, gui_tick() must be as fast as possible to let others use egui too. and tick() might take upto 16 milliseconds (assuming 60 fps) to do any other expensive work, but still be as fast as possible to not block the next frame which needs the plugin for gui_tick(). this is on the assumption that wgpu::Surface::get_current_texture() would block due to vsync and thus we let the plugins use this time for some thing useful rather than just idling.

The main hurdle at the moment with lua is the lack of multi-threading.
tick() is not enough for any expensive calculations and co_routines will only help so much. so, we need some sort of multi-threading with lua for parallel computations.

Parallel Processing Methods

1. plugins can have a lightweight.lua and heavyweight.lua scripts, and we use the lightweight for gui_tick() and tick() while with heavyweight, we just let it run in another thread. The two lua modules could communicate with some sort of channels or any rpc based api like zero_mq or grpc or jsonrpc or such. this requires jokolay to do the setup the channels or take care to make them both quit at the same time etc..
2. plugins can just launch their own process via os.execute() lua function and communicate between themselves using grpc or pipes or sockets or whatever. the important thing is that, jokolay has nothing to do and the plugins manage that complexity themselves.
3. plugins can use native modules like cdylibs and do their expensive work (as well as threads ) inside those libraries. and just check for results using coroutines. this is probably the simplest and cleanest method yet. might be better for native rust modules to not use any async stuff and simply use std::thread or rayon::thread_pool to spawn any closures, or we run the risk of complications between tokio in jokolay and tokio in the native module interacting in weird ways.

[coderedart]

Just checked and all lua things work properly like require for lua modules as well as native lua modules (rust mlua or C). now, we just need to refine it a little bit and start finalizing the api. then, we can start exposing the required functionality for plugins as requested by plugin devs (fortunately, its not much work :D )

Parallel Processing Methods

2 and 3 are both valid methods. I would prefer 3 as its much simpler. it seems native rust modules work well as long as we make sure that the mlua's runtime feature "lua54 or luajit or luau" etc.. match the ones used by jokolay when compiling. threads work too and we can simply use Channels for a lot of these off-threading operations for now. there's also coroutines which can be used to expose reqwest or other async apis like filesystem.

[coderedart]

changed due date to April 06, 2022

[coderedart]

marked this issue as related to #24

[coderedart]

marked this issue as related to #31

[coderedart]

wasm WASI is still too young. none of the proposals are standardized yet. https://github.com/WebAssembly/WASI/blob/main/Proposals.md

wasm, though useful to enable plugins, cannot be used as a scripting language directly. so, even if we want to use wasm, we will need to decide on a language which has live wasm interpreter like https://github.com/RustPython/RustPython

[coderedart]

so, there's a bevy scripting api being developed (still in alpha stages) at https://github.com/makspll/bevy_mod_scripting . the design seems to be going for an event based trigger.

this sort of design would seems to be popular. lua scripts would be "attached" to a parent object (an individual Lua VM), and it will register the events that it is interested in and the name of the function that it wants run on that event. when the event is emitted, the game will call the right function and pass it the event details. the function then does stuff in the background while the game continues running. if there's any other events emitted while the lua script is still executing, they will be queued to be dealt with after the script is done.

so, each script is technically its own parallel thread, which communicates with the main game for input and outputs.

the issue we are facing is that egui is immediate mode. it means the UI code is blocking / synchronous and must be run every frame.

until now, we planned to run each plugin in a single VM. but if the vm is being used to respond to some events, we will need to wait until the current execution is halted, to run the UI function.

bevy_mod_scripting uses a priority event mechanism, so we can keep UI events as top priority to skip any other pending events and use the VM immediately after the current execution finishes. but it still limits the plugins to workaround the 16 milliseconds limit (60 fps assumed) and make the plugin dev harder (non intuitive).

but, if we could split a plugin into multiple scripts and use a dedicated vm each for each script, then UI script can have its own VM without having to deal with any delays due to other functions executing in response to events.

Plugins would need some sort of way to share a data between different scripts (VMs). could be a simple Arc<RwLock> for json or a channel network between all scripts or a plugin declared number of Strings/Vec or even a simple HashMap<String, String>. what the data means should be interpreted by the plugin itself.

[coderedart]

lua doesn't expose UserData methods which take self as argument (consuming methods). so, we cannot pass structs like egui::Window<'open> which are builders and take the self argument in the final method (show() for Window) into lua. thus, for configuation of Window, we will simply pass a lua Table<'lua> which can be iterated and the options applied before calling show on the rust side itself. then, we can pass in the Uias it is never consumed as self, but always used as &mut self.

we could technically, make a wrapper struct Window(Option<egui::Window>), and just use Option::takewhich is okay with&mut self`.

EDIT on 9/7/22: taking a lua table is bad:

1. we need to make sure that the values are of the right type and panic otherwise.
2. for some functions which take multiple arguments, there's no obvious way to set the values in the table for that functions. we have to add prefixes like fn_name_align2, fn_name_vec2 etc..

so, we are going with the Option approach for now. and taking a window configuration function.

[coderedart]

we still need to figure out how to deal with plugins on web. is it possible to use rhai (or other scripting langauges) for plugins so that they will be able to run on web? or can we use wasmer / wasmtime embedded ?

this requires a new issue which will take a while. either way, the performance will not be good enough compared to luajit. so, it makes sense to start with lua for now. if someone figures out a way to run lua on wasm, we can look at porting the lua plugin functionality to web then.

one thing to remember is that we can use wasm as a simple stdin / stdout piped plugin or a jsonrpc plugin too. like https://zellij.dev/documentation/plugin-rust.html.

we need to keep a persistent json which can be edited by sending events by the plugin. and we can build the Ui for the plugin using the Json.
a sample of how it would look:

{
"window_one" : {
  "container_type" : "Window", // we use this object to build a window
  "title" : "My Window", // fields of this object will be the options to be used for the window
  "id" : 3242, // some ID decided by plugin to be used for sending related events and uniquely identifying this object.
  "open" : true, // whether the window is open. if user closes window, we set this to false and send the close event with id to the plugin
  "ui_tree" : { // this is how we fill the window with Ui
    "alpha" : { 
      "widget_type" : "DragValue", // type of widget to use for this object
      "id" : 2345, // unique id by plugin
      "value" : 0.5,  // value to use for this widget. when changed, we send event to plugin. when plugin wants to, it can update this value by sending host an event.
    } // alpha closed
  }// ui_tree closed
} // window closed
} // root closed

this can help with highly performant and safe wasm based plugins. or plugins in any language technically speaking including native rust. we just need the events / signals to be repr(C).

the only issue is that we will need to write a LOT of events and map them properly to the respective egui apis. we will be simulating a retained mode on top of immediate mode.

[coderedart]

created branch 3-jokolay-and-plugins-modules-scripting-addons-extensions to address this issue

[coderedart]

fortunately, mlua developer plans to port C lua53 to rust. this will enable us to use lua on web and desktop.

although Luau would be great for its inbuilt features (safety / sandboxing) and performance, it is still c++ and much harder to port to rust. and it doesn't support native modules.

ideally, we should try to keep within the subset of Lua 5.1 so that we can target Luau + LuaJIT + Lua53 (rust) in future based on our needs.

On web, we can only use Lua53 (rust). because C/Cpp bindings cannot work on wasm32-unknown-unknown abi. although C/Cpp bindings can be compiled using wasm32-unknown-emscripten, most of the rust projects don't support this target (winit, bevy etc..).
we cannot use native modules, so only pure lua plugins are allowed.

On desktop, there are tradeoffs to be considered:

1. native modules: require unsafe and locking ourselves to a lua version (Luau / Jit / 53)
2. sandboxing: if we ignore the native modules feature and just focus on pure sandboxed plugins, we can provide better security.

we need to focus on basic scripting support so as to enable the other parts of the Jokolay (like #24 ) to move forward.

also need to create a basic template repo for the plugins development.

there's a lot of different projects which transpile into Lua. better to start thinking about how to support them.

Tier 1

https://github.com/TypeScriptToLua/TypeScriptToLua // typescript
https://github.com/teal-language/tl // teal (first class support with tealr)

Tier 2

https://github.com/pigpigyyy/Yuescript // really clean
https://github.com/leafo/moonscript
https://github.com/HaxeFoundation/haxe // seems popular?

just for completion, we have these too.
https://github.com/yanghuan/CSharp.lua
https://github.com/ClueLang/Clue

[coderedart]

Lua

We are waiting for a while for the lua port to rust. meanwhile, i tried to port Lua myself and it was a horrible experience.

1. Lua strings are byte arrays. so, their source files are not valid utf-8 or ascii
2. Lua C Source is full of macros. very hard to know what the code is actually doing.
3. there are no popular/complete garbage collection crates in rust. and making one is super hard.

The worst part is that i don't even like Lua. one indexing, lack of types, lack of proper arrays, lack of proper strings (utf-8), breaking changes leading to ecosystem split between Luau, Luajit, Lua reference impl. Life would be better with javascript (or python) which are much more mainstream. The whole reason to use Lua was mlua / tealr. If only rustpython / boa were more mature, we could have used them.

V8

One of the main reasons to chose Lua over JS was because we wanted to support plugins on web too. but if we sacrifice that, we can just use V8 for desktop scripting. if we had a mlua like nice wrapper over v8, it could be a much better alternative.

1. V8 is a god tier in terms of performance.
2. javascript is super popular. and i would prefer to learn JS inside out rather than Lua.

WASM Revisited

one of the reasons we ignored wasm was the lack of component model (and will take atleast another year before it settles down). This makes integrating wasm / egui hard.
BUT what if we go the WoW route and split the UI from the scripts.
just create a struct to describe UI. and jokolay will render it. send any changes in values to plugin and plugin sends any changes to jokolay.

struct JsonUI {
	/// This contains all the shared data like strings used for labels or text edit buffers and so on.
	data: Vec<JsonValue>,
	ui_structure: Vec<ContainerType>,
}
enum ContainerType {
	Window,
	Area,
	//...
}
struct Container {
	kind: ContainerType,
	name: u32, // index into the data field above for the actual string (JsonValue::String). 
	id_source: Option<u32>,
	child_ui: 
}
enum UI {
	ChildUI(UI),
	Label {
		data: u32,
	},
	TextEdit {
		data: u32,
	},
	//...
}

1. This will allow us to separate plugins from UI. so, we can run plugins along with jokolay, or separate thread or even processes. 2. This allows us to communicate with plugins using some form of RPC which is non-blocking in nature.
2. We can use wasmtime to sandbox plugins (and run them on a separate thread maybe) allowing us to write plugins in Rust.

We can also just use rpc over channels or sockets to communicate with external plugins which can be written in any language (js/python/cpp etc..).

There's json schema to help users find out any faults while they are still writing the raw json.

Anyway, V8 would still be super cool because it allows scripts which want more performance / integration / customization. so, that should be our preference over wasm.