documentation | extensions | examples

Toolips is an extensible web and server-development framework for the Julia programming language.

• HTTPS capable Can be deployed with SSL.
• Extensible server platform.
• Declarative and composable files, html, Javascript, and CSS templating syntax provided by ToolipsServables.
• Modular servers -- toolips applications are regular Julia Modules.
• Versatilility -- toolips can be used for all use-cases, from full-stack web-development to simple endpoints.
• Parallel Computing -- Declarative process management provided by parametric processes.

Toolips is able to create ...

• Endpoints
• File servers
• Fullstack web applications
• Other HTTP/HTTPS servers (e.g. Proxy server)
• UDP servers and services (e.g. Data-base cursors, DNS servers)

using Pkg; Pkg.add("Toolips")

julia> # Press ] to enter your Pkg REPL
julia> ]
pkg> add Toolips

map

• get started
  • documentation
  • quick start
    • projects
      • routing
      • extensions
    • responses
      • files
      • components
      • templating
• creating extensions
  • connection extensions
  • routing extensions
  • server extensions
• multi-threading
• built with toolips
• contributing

---

• toolips requires julia. julia installation instructions

get started

Toolips is available in three different version flavors:

• Latest (main) -- The main working version of toolips.
• stable (#stable) -- Faster, more frequent updates than main; stable... but some new features are not fully implemented.
• and Unstable (#Unstable) -- Latest updates, packages could be temporarily broken in different ways from time to time.

using Pkg
# Latest 
Pkg.add("Toolips")
Pkg.add("Toolips", rev = "stable")
Pkg.add("Toolips", rev = "Unstable")

Alternatively, you can add the latest of each breaking version using an x revision.

using Pkg
Pkg.add("Toolips", rev = "0.1.x")
Pkg.add("Toolips", rev = "0.2.x")
Pkg.add("Toolips", rev = "0.3.x")

• toolips primarily targets full-stack web-development, but does so through extensions -- the intention being to use Toolips for both simple APIs and complex web-apps. This being considered, it is important to look into toolips extensions to realize the full capabilities of this package! ToolipsSession provides Toolips with full-stack callbacks, for example.
• Check out our toolips projects and examples for some examples of use-cases for the framework.
• Check out creating-extensions for more information on building extensions.

documentation

Awesome documentation website coming soon

• For now, you can use ?Toolips to see a full list of exports.

projects

In Toolips, projects are modules which export Toolips types. These special types are

• Any sub-type of AbstractRoute.
• Any sub-type of Extension.
• or a Vector{<:AbstractRoute}

Here is a simple " hello world" project.

module HelloWorld
using Toolips
# hello world in toolips
home = route("/") do c::Connection
    write!(c, "hello world")
end

export start!, home
end

Here we use route to create a Route{Connection}, home. home is then exported, along with start! -- which is used to start our server.

# starts our server:
using HelloWorld; start!(HelloWorld)
# providing IP
using HelloWorld
start!(HelloWorld, "127.0.0.1":8000)

routing

home = route("/") do c::Connection
    write!(c, "hello world!")
end

To create a Route, we provide the route Function with a target, a String path starting at / to mount the website's base URL and a Function passed through do. The general Toolips process on a route is creating data and then writing it to the Connection with write!. The Function we provide will take a <: of an AbstractConnection. We are able to annotate this argument in our route call to change our route's functionality based on the dispatch. This creates what is effectively multiple dispatch routing, consider the example below:

module HelloWorld
using Toolips

desktop_home = route("/") do c::Connection
    write!(c, "hello world")
end

mobile_home = route("/") do c::MobileConnection
    write!(c, "hello world")
end

# multi-routing our home
home = route(mobile_home, desktop_home)
export start!, home
end

In the case above, mobile clients will be redirected to the latter Function, as their Connection will convert into a MobileConnection.

Routes are stored in the Connection under Connection.routes. We can dynamically change our routes by mutating this Vector{<:AbstractRoute}.

module ToolipsServer
using Toolips
using Toolips.Components

home = route("/") do c::Connection
    new_route = route("/newpage") do c::Connection
        write!(c, "second page")
    end
    push!(c.routes, new_route)
    # creating a quick page to link to our route
    lnk = a("othersite", text = "visit new route", align = "center", href = "/newpage")
    style!(lnk, "margin-top" => 10percent)
    write!(c, lnk)
end

export default_404, home
end

There are several "getter" methods associated with the Connection, here is a comprehensive list:

get_args
get_heading
get_ip
get_post
get_method
get_post
get_parent
get_client_system

All of these take a Connection and are pretty self explanatory with the exception of get_client_system. This will provide the system of the client, but also whether or not the client is on a mobile system. Note that the operating system is given as the request header gives it, of course.

client_operating_system_name, ismobile = get_client_system(c)

There's also

proxy_pass!(c::Connection, url::String)
startread!(c::AbstractConnection)
download!(c::AbstractConnection, uri::String)
respond!(c::AbstractConnection, args ...)

Routes can be exported as any Vector{<:AbstractRoute} or AbstractRoute. Only routes which are exported will be loaded, exporting names which do not actually exist in the project will break the server. The following functions/methods may be used to create new routes with base Toolips:

# creates a regular route
route(::Function, ::String) -> ::Route{<:AbstractConnection}
# creates a `multi-route`
route(::Route ...) -> ::MultiRoute
# mounts the file or directory in the value to the path in the key.
mount(::Pair{String, String}) -> ::Route{AbstractConnection}

module ServerSample
  route()
end

extensions

Extensions appear in Toolips in four main forms:

• Connection extensions,
• routing extensions,
• server extensions,
• and Component extensions.

Connection extensions allow us to utilize MultiRoute with new multiple dispatch Connection configurations. Routing extensions allow us to change the functionality of the Toolips router in different instances. Server extensions allow us to add autoloaded data, or perform actions alongside before our routes whenever a Connection is served. Finally, Component extensions give us more composable Component types to work with, and more high-level web-development capabilities.

Connection extensions are typically used through MultiRoute. This is done by providing multiple routes to route, which will call different routes depending on the incoming client Connection. For example, the MobileConnection is the quintessential Connection extension provided by Toolips.

module Sample
using Toolips

desktop = route("/") do c::Connection
    write!(c, "this page is only served to mobile users")
end

mob = route("/") do c::MobileConnection
    write!(c, "this page is only served to mobile users")
end

# make multiroute
mult_rt = route(desktop, mob)

export mult_rt, start!
end

• creating connection extensions

responses

Like most web-development frameworks, creating websites or APIs with Toolips primarily revolves around creating a response. In the case of an API, this is actually pretty simple. write! will convert any provided type to a String and then write it to the incoming Connection stream.

module Multiply
using Toolips

home = route("/") do c::Connection
    args = get_args(c)
    arg_keys = keys(args)
    if ~(:y in arg_keys) || ~(:x in arg_keys)
        write!(c, "you have not provided `x` or `y`.")
    end
    write!(c, string(x * y))
end

• Note the use of get_args, get_post might also be important for APIs.

module Crystals
using Toolips
import Base: getindex, in

mutable struct APIClient
    ip::String
    requests::Int64
    max::Int64
    name::String
end

getindex(apc::Vector{APIClient}, ip::String) = begin
    found_client = findfirst(c::APIClient -> c.ip == ip, apc)
    if isnothing(found_client)
        throw(KeyError(ip))
    end
    apc[found_client]
end
end

in(ip::String, apc::Vector{APIClient}) = begin
  found_client = findfirst(c::APIClient -> c.ip == ip, apc)
  ~(isnothing(found_client))
end

clients = Vector{APIClient}()

verify = route("/") do c::AbstractConnection
  nm = get_post(c)
  allnames = [client.name for client in clients]
  if length(nm) > 3 && replace(nm, " " => "") != "" && ~(nm in allnames)
     write!(c, "you are verified, $nm ! Have fun with the crystal API!"
     push!(clients, APIClient(get_ip(c), 1, 50, nm))
  end
end

crystals_api = route("/crystals") do c::AbstractConnection
    args = get_args(c)
    arg_keys = keys(args)
    if ~(get_ip(c) in clients)
        write!(c, "You are not verified! Please POST your name to our home-page to identify yourself.")
    end
end

export crystals_api
end

For more detailed websites, we might be building a more complicated response. Toolips provides the Components Module, ToolipsServables. This Module includes the File type for easily serving parametrically files by path and AbstractComponent types for high-level parametric HTML and CSS templating.

files

Files in Toolips can either be built manually with the File constructor or can be directly mounted to a route with mount. mount takes a Vector{Pair{String, String}}, and will return a Route or a Vector{<:AbstractRoute} -- depending on whether or not the provided path is a file or a directory. A directory will be recursively routed, creating a route for each file in each sub-directory below it...

When created manually, a File is able to be written with write!, like normal. This also gives us the ability to use interpolate!, which will interpolate Components by name or interpolate values by using interpolate! in place of write!.

function interpolate!(c::AbstractConnection, f::File{<:Any}, components::AbstractComponent ...; args ...)

For example, using this Method to interpolate HTML with components and values...

<body>
<div>
<h2>hello client</h2>
<a>your ip address is $ip</a>
<h4>would you like to name yourself?</h4>
$namebutton
</div>

This example interpolates HTML -- but is the catchall, or top-level function (binded to File{<:Any} -- meaning you could also write different methods to change behavior depending on file type.

function interpolate!(c::AbstractConnection, f::File{:md}, components::AbstractComponent ...; args ...)
    raw::String = string(f)
    interp_positions = findall("```", raw)
    ...
end

components

This package also allows us to create callbacks for these components...

And ToolipsSession expands on this by providing server-side callbacks and some pretty extreme fullstack capabilities.

templating

As demonstrated in this README thus far, Toolips has a diverse set of a capabilities when it comes to templating. Templating in Toolips is done by constructing and composing components into a body and then writing it to the Connection, or interpolating a file via the interpolate! function.

creating extensions

connection extensions

A Connection extension creates a new Connection which can be used with multi-route, or otherwise with a new router. The running example of this inside Toolips is the MobileConnection.

mutable struct MobileConnection{T} <: AbstractConnection
    stream::Any
    data::Dict{Symbol, Any}
    routes::Vector{AbstractRoute}
    MobileConnection(stream::Any, data::Dict{Symbol, <:Any}, routes::Vector{<:AbstractRoute}) = begin
        new{typeof(stream)}(stream, data, routes)
    end
end

The MobileConnection is created whenever an incoming client is on mobile. This is determined by get_client_system. Two functions are used for this; convert and convert!. convert is called on the Connection to see if the Connection should convert. If it should convert, then convert! is called.

function convert(c::AbstractConnection, routes::Routes, into::Type{MobileConnection})
    get_client_system(c)[2]::Bool
end

function convert!(c::AbstractConnection, routes::Routes, into::Type{MobileConnection})
    MobileConnection(c.stream, c.data, routes)::MobileConnection{typeof(c.stream)}
end

# for IO Connection specifically...
function convert!(c::IOConnection, routes::Routes, into::Type{MobileConnection})
    stream = Dict{Symbol, String}(:stream => c.stream, :args => get_args(c), :post => get_post(c), 
    :ip => get_ip(c), :method => get_method(c), :target => get_target(c), :host => get_host(c))
    MobileConnection(stream, c.data, routes)::MobileConnection{Dict{Symbol, String}}
end

Note that the MobileConnection is actually a MobileConnection{<:Any}. We build a data dictionary in order to turn the IOConnection into a MobileConnection, whereas in the case of the Connection we are provided the standard HTTP.Stream directly. This simple system facilitates both types. Beyond this, you are free to extend other Connection functions to enhance your interface if they are not compatible with your current Connection. Not implementing this will mean that the Connection will not work with multi-threading.

get_ip(c::MobileConnection{Dict{Symbol, String}}) = c.stream[:ip]
get_method(c::MobileConnection{Dict{Symbol, String}}) = c.stream[:method]
get_args(c::MobileConnection{Dict{Symbol, String}}) = c.stream[:args]
get_target(c::MobileConnection{Dict{Symbol, String}}) = c.stream[:target]
get_host(c::MobileConnection{Dict{Symbol, String}}) = c.stream[:host]
write!(c::MobileConnection{Dict{Symbol, String}}, a::Any ...) = c.stream[:stream] = c.stream[:stream] * join(string(obj) for obj in a)

Let's implement a PostConnection in order to demonstrate this:

module PostConnections
using Toolips
import Toolips: AbstractConnection, convert, convert!
mutable struct PostConnection{T} <: AbstractConnection
    stream::Any
    data::Dict{Symbol, Any}
    routes::Vector{AbstractRoute}
    PostConnection(stream::Any, data::Dict{Symbol, <:Any}, routes::Vector{<:AbstractRoute}) = begin
        new{typeof(stream)}(stream, data, routes)
    end
end

function convert(c::AbstractConnection, routes::Routes, into::Type{PostConnection})
    get_method(c) == "POST"
end

function convert!(c::AbstractConnection, routes::Routes, into::Type{PostConnection})
    PostConnection(c.stream, c.data, routes)::PostConnection{typeof(c.stream)}
end

function convert!(c::IOConnection, routes::Routes, into::Type{PostConnection})
    stream = Dict{Symbol, String}(:stream => c.stream, :args => get_args(c), :post => get_post(c), 
    :ip => get_ip(c), :method => get_method(c), :target => get_target(c), :host => get_host(c))
    PostConnection(stream, c.data, routes)::PostConnection{Dict{Symbol, String}}
end

Now let's use it:

module PostSample
using Toolips
using Main.PostConnections
using Toolips.Components

# regular `GET`
home_main = route("/") do c::Connection
    write!(c, h2("main", text = "you landed!", align = "center"))
end

home_post = route("/") do c::PostConnection
    write!(c, "welcome to the API :)")
end

home = route(home_main, home_post)

export home_main, home_post, home
end

routing extensions

Another type of extension that can be created for toolips is the routing extension. Routing extensions are created by extending the route! function. This function may be extended by adding new methods for Route types (<:AbstractRoute), Connection types (<:AbstractConnection), a Vector with <:AbstractRoute as its type parameter, or extension types (<:AbstractExtension).

• on an incoming Connection, route! is initially called on each extension using route!(c, ::AbstractExtension) (only if the binding exists) before the main routing process begins -- giving extensions the first oppurtunity to write! to the Connection.
• route! is called twice during the routing process, first on the Connection and the Vector{<:AbstractRoute} that holds the routes. This is where the startup printout of Toolips comes to relevance:

julia> Toolips.start!(Sample)
🌷 toolips> loaded router type: Vector{Toolips.Route{Connection}}
🌷 toolips> server listening at http://127.0.0.1:8000

• route! is also called again on a MultiRoute if a MultiRoute is being used. In the binding for the quintessential MultiRoute type, for example, the incoming Connection checks for conversion into any of the dispatched functions.

All of these considered, there are a lot of ways to extend the routing of Toolips.

server extensions

Finally, for server extensions wqe will want to

multi-threading

Toolips includes a distributed computing implementation built atop ParametricProcesses. This implementation of multi-threading allows us to serve each incoming connection on a different thread simply by providing the number of threads to utilize. Providing threads will simply add additional workers to our ProcessManager. These workers can then be used with distribute! and assign! or assign_open! -- all functions extended to work with the Connection from ParametricProcesses. By default, the number of router_threads will be -2, 3 responses from the base thread, and then however many threaded workers are provided. But of course, if we provided -- say 1:1 then we would only get the threads in the ProcessManager.

start!(mod::Module = Main, ip::IP4 = ip4_cli(Main.ARGS);
    threads::Int64 = 1, router_threads::UnitRange{Int64} = -2:threads)

module MySampleServer
using Toolips
home = route("/") do c::Connection
    write!(c, "hello")
end

export home, start!
end

using MySampleServer; start!(MySampleServer, router_threads

For the most part, this is straightforward -- but there are some things to be aware of...

• When a server is multi-threaded, its routes will be passed an IOConnection -- not a regular Connection. Routes will need to be annotated as an AbstractConnection (to work with single or multiple threads,) an IOConnection (to work with multi-threaded servers only,) or an AbstractConnection to work with multi-threaded servers.
• A multi-threaded server must be a project. The Module cannot be defined below Main, it must have its own Project.toml file. This is because your Module needs to be used across multiple threads from the same environment; ParametricProcesses will not be able to serialize your entire server and send it over to all of your threads. Instead, it is used via the environment. An environment compatible with this is of course set up for you when new_app is used.
• Finally, only certain objects will be serialized across threads. This means that we must be weary of what is in our IOConnection.data, or we might run into problems serializing across threads. This will primarily happen with functions. For example, consider the following Session callback:

module ThreadedSampleServer
using Toolips
using Toolips.Components
using ToolipsSession

session = Session(["/"]) # <- active route "/"

main = route("/") do c::Connection
    mainbody = body("mainbod")
    clickable = h3("sample", text = "hello")
    style!(h3, "transition" => 2seconds)
    push!(mainbody, clickable)
    on(c, clickable, "click") do cm::ComponentModifier
        alert!(cm, "goodbye!")
        style!(cm, "sample", "opacity" => 0percent)
    end
    write!(c, mainbody)
end

function load_alert(cm::ComponentModifier)
  
end

export session, main

end

This is not multi-threading compatible for two different reasons; our c is annotated to Connection, and our Session callback has a Function inside of it we will need to serialize. To avoid this with Function callbacks, we simply need to define the Function in our Module, as it is already loaded to our threads.

module ThreadedSampleServer
using Toolips
using Toolips.Components
using ToolipsSession

session = Session(["/"]) # <- active route "/"

main = route("/") do c::Toolips.AbstractConnection
    mainbody = body("mainbod")
    clickable = h3("sample", text = "hello")
    style!(h3, "transition" => 2seconds)
    push!(mainbody, clickable)
    on(load_alert, c, clickable, "click")
    write!(c, mainbody)
end

function load_alert(cm::ComponentModifier)
  alert!(cm, "goodbye!")
  style!(cm, "sample", "opacity" => 0percent)
end

export session, main

end

From here, we simply provide the threads key-word argument to start!

julia> using ThreadedSampleServer; ThreadedSampleServer.start!(ThreadedSampleServer, "192.168.1.15":8000, threads = 4)
[ Info: Precompiling ThreadedSampleServer [307046d4-7f21-496b-9a80-f3bfb096e574]
🌷 toolips> loaded router type: Vector{Toolips.Route{Toolips.AbstractConnection}}
🌷 toolips> server listening at http://192.168.1.15:8000
      Active manifest files: 9 found
      Active artifact files: 3 found
      Active scratchspaces: 0 found
     Deleted no artifacts, repos, packages or scratchspaces
🌷 toolips> adding 4 threaded workers ...
🌷 toolips> spawned threaded workers: 2|3|4|5
[ Info: Listening on: 192.168.1.15:8000, thread id: 4
   pid                 process type                        name active
  –––– –––––––––––––––––––––––––––– ––––––––––––––––––––––––––– ––––––
  2080    ParametricProcesses.Async ThreadedSampleServer router   true
     2 ParametricProcesses.Threaded                           1  false
     3 ParametricProcesses.Threaded                           2  false
     4 ParametricProcesses.Threaded                           3  false
     5 ParametricProcesses.Threaded                           4  false

built with toolips

Because Tooips was built primarily to drive other chifi software, ChifiSource has created a number of projects with Toolips. Here is a list of large projects we have created based on Toolips, along with their repository links.

• Olive Olive is the reason that Toolips was created in the first place. Olive is a parametric extensible notebook editor for Julia. This is a great example to demonstrate a full-scale project.
  • Using: Toolips, ToolipsServables, ToolipsSession
• Gattino Gattino is Toolips-based, or rather ToolipsServables-based SVG data visualizations for Julia. A look into this project may give insight on how ToolipsServables and Toolips might be used without a WebServer.
  • Using: ToolipsServables
• JLChat JLChat is emma's Toolips-built chatroom demonstration. This example is great for demonstrating how to create a small application in Toolips, along with using ToolipsSession and its RPC feature.
  • Using: Toolips, ToolipsServables, ToolipsSession
• ChiProxy ChiProxy is a Toolips-bound proxy server for Julia. This proxy server demonstrates replacing the Toolips router by extending functions, allowing for routes to be routed by host rather than just target -- as well as a plethora of other special capabilities.
  • Using: Toolips
• ChiNS ChiNS is a Domain Name Server built with Toolips. This project provides a running example of ToolipsUDP, as well as a pretty nice demonstration of how to create a DNS server.
  • Using: ToolipsUDP

contributing

Toolips is a totally awesome project, and with more contributors becomes even better even faster. You may contribute to this project by...

• using Toolips in your own project 🌷
• creating extensions for the toolips ecosystem 💐
• forking this project contributing guidelines
• submitting issues
• contributing to other chifi projects
• supporting chifi creators

I thank you for all of your help with our project, or just for considering contributing! I want to stress further that we are not picky -- allowing us all to express ourselves in different ways is part of the key methodology behind the entire chifi ecosystem. Feel free to contribute, we would love to see your art! Issues marked with good first issue might be a great place to start!

guidelines

When submitting issues or pull-requests for Toolips, it is important to make sure of a few things. We are not super strict, but making sure of these few things will be helpful for maintainers!

1. You have replicated the issue on Unstable
2. The issue does not currently exist... or does not have a planned implementation different to your own. In these cases, please collaborate on the issue, express your idea and we will select the best choice.
3. Pull Request TO UNSTABLE
4. Be specific about your issue -- if you are experiencing multiple issues, open multiple issues. It is better to have a high quantity of issues that specifically describe things than a low quantity of issues that describe multiple things.
5. If you have a new issue, open a new issue. It is not best to comment your issue under an unrelated issue; even a case where you are experiencing that issue, if you want to mention another issue, open a new issue.
6. Questions are fine, but not questions answered inside of this README.