Using Pluto as an inputs / outputs pipeline

[heetbeet]

I want to use Pluto for something alternative to a GUI Notebook, I have a calculation pipeline that I want to automate, so I want to use the internals of Pluto, like Notebook and update_run! directly. I want to create a large graph of variable relationships (a typical Notebook as you would have in the Puto GUI), but I want to change some of the "input" variables's values millions of times, and aggregate on some of the output variables.

My question is, how efficient is the Pluto Notebook pipeline for this?

[heetbeet]

Hmm, now that I think of it, it is probably as efficient as the effect of adjusting a slider

[ctrekker]

I think Pluto could be quite efficient for this! Like you say, it's rather similar to adjusting a slider but with any given variable instead. Importing Pluto in a script and using it as an API (with update_run! as you mentioned) would probably be the best approach.

What you describe surrounding changing variables and seeing the response in other "output" variables is exactly what I've been working on with "What you see is what you rest", although I would advise against its use in your particular scenario. There's minimal overhead in the API, but over millions of iterations that overhead adds up quickly.

[heetbeet]

@ctrekker I've had a look at the committed code and, wow, what you have done is awesome! I'm going to try it out as soon as possible.

[heetbeet]

Currently I have no working code for my idea, so your overhead is still better than nothing. Plus I can always use this approach to reverse something thinner. What's really nice is all the Notebook helper functions, that alone is worth a lot

[fonsp]

@heetbeet Could you tell us more about your intended application? It might help us (@ctrekker) during the design process of WYSIWYR

[heetbeet]

Yes, I'll quickly share on what I'm trying to achieve, but it's very rough draft at the moment.

Non-Pluto side of things (just for background and understanding)

We want to build models from a list of formulae defining time variant vectors. This is similar to how actuaries work out stuff in Excel or other type of cash-flow modeling systems. Here is an example (note, these are not strictly in Julia syntax):

age_t0 = 30
duration_t0 = 6
premium_t0 = 100
sum_assured_t0 = 10000
age = int(age_t0 + t/12)
duration = duration_t0 + t
qx = age / 1000
lapse_rate_m = 1 / qx / 100
no_pols_som = IF(t==1,1,no_pols_eom[t-1])
no_deaths = no_pols_som*qx/12*(1-0.5*lapse_rate_m)
no_surrenders = no_pols_som*lapse_rate_m*(1-0.5*qx)
no_pols_eom = IF(t==1,1,no_pols_eom[t-1])-no_deaths-no_surrenders
premium_income = premium_t0*no_pols_eom/12
claims_outgo = sum_assured_t0*no_deaths
profit = premium_income-claims_outgo
yield_curve = 10/100
profit_discounted = profit*(1+yield_curve)^(-t)
liability = -profit_discounted+IF(t<600,liability[t+1],0)

What our Julia system does is

• works out the dependencies between these formulae (similar to what Pluto does),
• group cyclic dependent formulae into a single clusters/cell
• wrap each cell in a Julia macro @T to make sense of the time t variable (adds dimensionality)
• output this as a Pluto Notebook

For example:

The @T macro places the formulae in a loop t in T and shuffles then around a bit to make sense (bear with me, this is still quite a rough process):

But in summary, we have formulae as inputs, and we have Julia code in a Pluto notebook as output. This is nice for converting an Excel-like table-like workflow into code.

The Pluto side of things

What all of this also gives us is a nice applied actuarial/mathematical model of the underlying formulae. This made me think, hmm, what else can we do with it... and I immediately though of this as an input-output model:

• what if we use this model in optimization, e.g. import the model into Julia, make some of the variables unknown, and optimize for something like maximum profit w.r.t. those variable.
• what if we can use this to describe a data transformation step and want to apply this to a million data point, e.g. describe all the transformations i.t.o. formulae, loop over some data from like a database, feed it to the model, and aggregate over one of the "output" variables.

I can't really think of more things at the moment, but for this to work, I will have to figure out a good way to call/update/run a Pluto notebook from Julia, and be sure that it is efficient. I'll also need to figure out how to use the notebook for input/output. I already found an inefficiency with my workflow when I tested some concept by attaching some of the variables to sliders, and that is that the @T macro's code generation is run at each point, but that is easy for me to fix with some caching behavior.

I hope this helps in any way

[lungben]

One idea would be to define one Pluto notebook with all the formulas for a single time step and then call this notebook for every time step with the new REST API.
However, this may be rather slow for millions of time steps.