Automatic timeout for cells to enforce reactivity

[karlwessel]

This is an idea I would like to discuss.

The intention of Pluto, as far as I understood it, is to be an reactive notebook where you get more or less instant feedback to changes so you immediately see the consequences of them.

If you have a cell that takes a long time to execute it really breaks the whole reactivity and collides with automatic reevaluation of cells. And often the only solution for long running cells seems to be to stop/kill Pluto and restart it (and julia).

I wonder therefore if it would be beneficial to consider long running cells (which are not compiling) an implementation error against which to guard the notebook per default, since most times they really are. For example when one underestimated the complexity of a method or messed up some loop ending condition.

A possible guard against that could be a really low (like 0.5 to 5 seconds) automatic timeout for each cell after which execution is stopped and an error is thrown telling the user to either reduce the load for that method or maybe set an explicit longer timeout for that cell in case it really needs to take that long.

A possible Problem could be compilation times, they should not count into the timeout.

Even cooler would be a mechanism by which the user can implement a cell so that it can return a partial result after the timeout instead of throwing an error.

[fonsp]

Cool idea, long-running cells is something that we should do something about!

I'm not sure about automatically interrupting, this would lead to bad things when importing a package, writing to a file, downloading a file, and other code with side effects.

So I would say that Pluto should prompt the user whether they want to stop a cell that is taking too long. Would you agree? And have you used the stop button that's currently implemented? You shouldn't have to shut down Pluto completely.

A common worry about Pluto is that (intentionally) long-running code will "not work in Pluto" or will be re-evaluated when they don't want to. I'm not sure what to say about the first one, but the second one could be solved with more informative UI.

My idea for this was to warn users when one of the downstream cells is expected to have a long runtime, based on its previous runtime, and prompt whether they still want to run a cell. This would be useful in cases where you are changing a parameter, without realising that your long-running code depends on that parameter. In that case, you might want to do something different, like define a new parameter.

[karlwessel]

You are right, automatic interruption would probably be to risky (also in the sense that it might make users just angry :-). Some prompt or warning might be good, but could lead also to the problem that users get tired of having to click away warnings or prompts all the time...

The most important thing probably is an informative UI. So that the user isn't surprised about things happening. One thing I'm often wondering is which cell(s) currently are executed and which are just waiting in the queue waiting for other cells. Also sometimes the stop button seems to disappear although the cell seems to be running.
Of course I'm also would like to know somehow which cell depends on which, but I'm not sure how one could communicate that information in a long notebook. Maybe some kind of birds view of the notebook?

Another important thing would be that interrupting a cell is as robust as one can be. I used the stop button several times. Often it works but also very often it results in a chain of error messages from which I have the impression that Pluto doesn't recover from and I have to restart it. However I never systematically tested those cases. I will try to do this the next time.

[fonsp]

I agree that informative UI is a priority! I'd love to hear more about the birds-eye view idea! (#13)

Interrupting code is quite difficult because Julia creates tight loops (from code like while true end) that never check whether they are interrupted. This is one of the reasons that the notebook code needs to run on a separate worker process. The current way to stop is pretty hacky, and doesn't work on Windows, but the idea is that it sends Ctrl+C once, then waits 5 seconds, and then starts trying to brute force an exit by sending Ctrl+C in short sequences (these are treated differently by the Julia process, try it out in a REPL!). The (arbitrary) 5 second delay is there to give code a chance to shut down nicely (although I have no concrete examples of when this is a problem), but it makes it feel unresponsive. It can also happen that these Ctrl+C sequences accidentally shut down the Julia process 🙈

There's some tweaking that can be done to the timing, but ideally we would have 1. concrete examples of code that needs to be shut down 'carefully' or where the current approach doesn't work, or even better 2. an entirely different approach that doesn't rely on Ctrl+C, but is able to send the different signals SIGTERM and SIGINT directly. And on Windows too!

Right now, Pluto is using Distributed.addprocs to create (asynchronous!) worker processes, but it would be great if we can use something more serious. I was planning to look into Julia's Test package, it is mysteriously able to run code in an isolated scope, and can even bring you back to the REPL after a segfault. 😮

The stopping function is not great, but i feel like it's still better than Jupyter, where you have to shut down the kernel if you run while true end!

[fonsp]

I should say that I feel like informative UI (#13) has higher priority than a better stop function, but please post your error messages in Issues when the stop button breaks again!

[fonsp]

I also had another solution to the long-running problem, which is the option to lock a cell from being edited, which would also lock all of its upstream cells (i.e. dependencies). You can always remove the lock, of course, but it might give users more confidence in the reactivity system. The lock could even be added automatically when the runtime is really long (say >1 minute).

A disadvantage, of course, is that this introduces a new UI element to worry about. Although the lock function would also be useful in education: a teacher could prepare a notebook with a number of locked cells, to tell students which cells are part of the exercise, and which aren't. (Students can remove the locks if they want to!)

[karlwessel]

I agree that improving the stop function is not as important, especially since I can imagine from own experience how hard/impossible it can be to get that right. Reducing the need to stop cells in the first place by a more informative UI is definitely the better solution.

[TheCedarPrince]

Hi guys,

Great project here! I looked pretty extensively through the UI and have a couple thoughts about how you could possibly improve the UI to convey long run times.

The Julia VSCode team have made a wonderful implementation of Debugger.jl in VSCode (attached is an image). I really encourage you guys to check out their implementation as well as UI of the Debugger. A screenshot of it is shown here:

I wonder if you could add a sort of "run with debugger" button. This could enable a user to step through their code with breakpoints potentially so they can "see" what is happening in the cell.

[fonsp]

Thanks for the suggestion!

Debugger.jl looks like a cool project! Implementing it would be an interesting experiment!

Of course (in the spirit of an experimental IDE), I don't think that the traditional debugging UI - having a different layout mode with a number of windows - would work well in Pluto. Instead, this debugging information should be available at the position on your screen where it has its relevance - this is a general UI goal for Pluto. Also, why have a debugging mode - isn't all programming debugging?

I'd just like to add that I think that having a notebook with separate cells, with intermediate output, is already a huge help in debugging. And having the notebook be reactive takes away an entire dimension of things to worry about when retracing results. Before implementing any debugging tools, it would be nice to know if my assumption is correct, and how common it is for errors to be too deeply buried for traditional debugging tools to be necessary.

[TheCedarPrince]

Hey @fonsp,

I agree with you 💯 percent on the need to keep Pluto.jl minimal. I personally think that is one of the biggest features is the lack of features or minimal bloat. If you want all the features, use something like VSCode or another editor/traditional IDE.

However, I do think a minimalistic debugger would still be useful. What I imagine Pluto to be is minimal but not necessarily feature-less. Like the project, spaceship terminal, it is minimal enough to where it doesn't get in the way, but sophisticated enough to provide powerful tools right when you need them. Going along with @karlwessel point, I think the minimal debugger, perhaps in the form of a temporary debugger cell that pops up upon selection, would be useful.

[dralletje]

Having Debugger.jl backed into Pluto itself seems a bit overkill and un-plutonic.
However, if we get better reactivity working, one can make a @debug (or to use Debugger.jl's lingua, @enter) macro that shows a debugger UI as cell output.
This seems a lot better, also because the interpreter that Debugger.jl uses is not at all as fast as the compiled code.

On the other hand, this could be a bigger project where we have a mode that runs all code interpreted by default (with a macro to enable compilation of certain block) to prevent notebooks from freezing. (Currently Pluto has to actually kill/interup the process by repeatedly sending SIGINTS for Julia to pick up on). This would make it more attractive to people starting out with Julia or programming in general. (Even for myself as I barely need compiled code in what I do, I think)

[fonsp]

Cool suggestion! Running Julia interpreted could actually make most cells faster, because we avoid the first-time-compilation overhead. This mostly matters for @bind notebooks with high update rate.

[dralletje]

@fonsp
Was thinking the same, but nah

First run

Thereafter

[fonsp]

i meant compilation of your code, which isn't included in those timings

[grero]

Somewhat tangential, but still related, I think. Is there a way to get the hash of the dependencies of a cell? The reason I ask is that I do in fact have some long running computation taking place in a cell, that depend on various functions that I've defined around my notebook. I would like to cache the result of this cell, so that, if none of the dependencies have changed, I'll just load some previously computed data. I think this would be mostly helpful when first running a notebook.