Cells with resource heavy code

[ixzh]

It is more of a personal problem:

I have a cell which reads 400+ of text files[10k lines per file on average] into array of strings, and another cell parsing them into array of dataframes. These two cells will take quite some time to run (1 mins to read, 10+ mins to parse into tables). Usually I will just wait for them [and the compiling overheads] to complete before I can start work.

The problem is the cells depend on functions in other cells [Reference Cell?]. When I need to change the functions, or sometimes just the comments inside functions, it will trigger these two cells to re-run.

I only need to read and parse the strings once. No change to the read data required.

What is the right way to prevent such cells from rerunning but generating the same data? Thanks!

[fonsp]

Hey! We will answer this question next week during our JuliaCon talk, is it okay to wait until then?

You can register for JuliaCon for free (do it!) or watch the talks on youtube after the week.

[fonsp]

See also #13 and #116

[ixzh]

Back to this issue again, as I've been haunted by a few cells which took too long to run...

I tried to @save (from JLD2) all variables to prevent parsing strings into DataFrames in future, but the macro is ignored... I have to read in the data as strings and parse them again each time restarting julia. (as I move around a lot)

Any suggestion to resolve resource heavy cells in a notebook?

[fonsp]

Did you trigger a re-run without knowing about it, or do you want to disable reactivity?

[ixzh]

I know the cell will be triggle by some action like adjusting output format of its dependant cells. Even first time loading is unavoidable. would appreciate some intelligence in limiting rerun the same results, and macro support to load variables from JLD2 file to eradicate the problem?

[karlwessel]

Would it be possible for Pluto to detect if only comments in a cell have changed and recognise that it doesn't have to rerun the code? That could reduce unnecessary reruns in some cases.

Otherwise I can only recommend to thoroughly test resource heavy cells with small datasets in the beginning until you are pretty sure that you don't have to change them to often later when using the real dataset.

[ixzh]

test resource heavy cells with small datasets in the beginning until you are pretty sure that you don't have to change them

not only in the current cell, but also upstream dependant cells. It's hard to make such avoidance upfront.

[fonsp]

#298