Converting .jl to Pluto notebook

[dpsanders]

I tried to open a standard .jl file in Pluto (from the Pluto startup page) and got

Failed to load notebook:

MethodError(union, (), 0x0000000000006c87)

<a href="/">Go back</a>

It would be great to be able to import a normal .jl and have it Plutified. This should e.g. take blocks of code and make them a single cell.

[fonsp]

I like your optimism in using the Open button and seeing what happens!

I guess that this is easy to add in, but hard to do right. With that I mean that there is lots of new UI needed to explain to the user: what they are doing, what the result will be (no two-way conversion), what went wrong (if the import fails) and where they want to save the notebook after importing (overwrite?).

Still, this might be worthwhile to add in, since it makes it easier to get started with Pluto.

[dpsanders]

I like your optimism in using the Open button and seeing what happens!

Ha. Yes it does seem like a good way to get started with Pluto, but I agree that it seems non-trivial to do correctly.

You could add _pluto to the filename for example, e.g. hello.jl -> hello_pluto.jl, since it's easy enough for the user to change. You don't want to just overwrite their file I guess.

[fonsp]

To anyone taking this on: see also #14, maybe this can be combined with Weave.jl and pandoc and Literate.jl and friends

[j-fu]

I think a possible way would be through Literate.jl, which has an option to output to Juypter, that means there is already a way to define markdown and Julia cells. By no means I think that a workflow .jl -> pluto should pass through .ipynb, though.

But may be it is even simpler: IMHO it would be sufficient to have syntax to mark cell boundaries, e.g. by # === which is recognized by Pluto on the first load and converted into the hash voodoo separators. Then we would have the notion of an "empty", "untouched" "proto-" notebook or however we call it and a notebook with recognized cells and manifest cell order.
This could even go into a Pluto.plutoify() method outside of the browser which takes in a proto notebook and spits out a real one (and complains about errors/ambiguities), very much like Literate itself works. This then could even be used for having a cli tool for this purpose.

[fonsp]

Support for writing notebook files by hand is not of this project, and making conversion part of Pluto is requires some carefully designed UI (#132 (comment)). But just like Jupyter conversion (https://observablehq.com/@olivier_plas/pluto-jl-jupyter-conversion and https://github.com/vdayanand/Jupyter2Pluto.jl), this can be an external tool, which is very simple to write! Maybe someone could try making it themselves?

If you run Meta.parse(code_string, 1), then it parses as much as it can into a single expression, and it returns the index of the next expression:

julia> code="s = 1\nr=2"
"s = 1\nr=2"

julia> Meta.parse(code, 1)
(:(s = 1), 7)

You can repeat this until you have found the start indices of every expression, and you then turn those into Pluto cells.

[j-fu]

Hi is there an API to turn something into a pluto cell (md or code) and to write a bunch of cells to a file? I think this is almost all I need...

[fonsp]

Yep:

julia> import Pluto

julia> code = ["hello", "world()"];

julia> n = Pluto.Notebook(Pluto.Cell.(code));

julia> path = "hello.jl";

julia> Pluto.save_notebook(n, path);

This gives:

julia> read(path, String) |> Text
### A Pluto.jl notebook ###
# v0.11.4

using Markdown
using InteractiveUtils

# ╔═╡ 844da824-dcb6-11ea-1b3d-95c52106a0d2
hello

# ╔═╡ 844da856-dcb6-11ea-2061-a59a23dd029f
world()

# ╔═╡ Cell order:
# ╠═844da824-dcb6-11ea-1b3d-95c52106a0d2
# ╠═844da856-dcb6-11ea-2061-a59a23dd029f

[j-fu]

Ok, this makes it easy to tweak Literate.script(), see my first attempt below.
I am not sure if @fredrikekre wants to have a Pluto dependency in Literate.jl. It could also go into Pluto it a Literate dependency is ok, or else into a separate package.
To put this into a separate package or into pluto creates some dependencies on internals of Literate, but perhaps this is the best bet.

function pluto(inputfile, outputdir=pwd(); config::Dict=Dict(), kwargs...)
    # Create configuration by merging default and userdefined
    config = Literate.create_configuration(inputfile; user_config=config, user_kwargs=kwargs)

    # normalize paths
    inputfile = normpath(inputfile)
    isfile(inputfile) || throw(ArgumentError("cannot find inputfile `$(inputfile)`"))
    inputfile = realpath(abspath(inputfile))
    mkpath(outputdir)
    outputdir = realpath(abspath(outputdir))

    @info "generating plain script file from `$(Base.contractuser(inputfile))`"
    # read content
    content = read(inputfile, String)

    # run custom pre-processing from user
    content = config["preprocess"](content)

    # default replacements
    content = Literate.replace_default(content, :jl; config=config)

    # create the script file
    chunks = Literate.parse(content)
    code=[]
    for chunk in chunks
        cell = IOBuffer()
        if isa(chunk, Literate.CodeChunk)
            for line in chunk.lines
                write(cell, line, '\n')
            end
            write(cell, '\n') # add a newline between each chunk
        elseif isa(chunk, Literate.MDChunk)
            write(cell,"md\"")
            for line in chunk.lines
                write(cell, rstrip(line.first * line.second * '\n'))
            end
            write(cell, "\"\n") # add a newline between each chunk
        end
        push!(code,String(take!(cell)))
    end

    # custom post-processing from user
    # content = config["postprocess"](String(take!(ioscript)))

    # write to file
    isdir(outputdir) || error("not a directory: $(outputdir)")
    outputfile = joinpath(outputdir, config["name"]::String * ".jl")

    if inputfile == outputfile
        throw(ArgumentError("outputfile (`$outputfile`) is identical to inputfile (`$inputfile`)"))
    end

    @info "writing result to `$(Base.contractuser(outputfile))`"
    nb = Pluto.Notebook(Pluto.Cell.(code));
    Pluto.save_notebook(nb, outputfile);

    return outputfile
end

[fonsp]

Great! I will try it out soon!

About the dependencies problem - I made PlutoUtils.jl for scripts like these, having the Literate.jl dependency is fine there. Can you write a PR?

[j-fu]

Great! I will try it out soon!

About the dependencies problem - I made PlutoUtils.jl for scripts like these, having the Literate.jl dependency is fine there. Can you write a PR?

Yeah I will do this within a day or two...

[fredrikekre]

It should be pretty trivilal to support something like

Literate.notebook(...; flavor = :pluto)

to Literate without having Pluto as a dependency.

[j-fu]

For writing a Pluto notebook to disk there is a Pluto API call
While it seems to be not too hard to re-create this in Literate.jl without a Pluto dependency, IMHO his would be hard to maintain
during the evolution of Pluto. So I think it is better to follow Fons' suggestion to put this into PlutoUtils.

[fonsp]

We talked about it here: #311 - the file format will always be backwards compatible, or conversion code is included in Pluto

[fredrikekre]

WIP support for using Literate.jl to create Pluto notebooks from .jl files: fredrikekre/Literate.jl#120. Please try it out!

[j-fu]

... so you have been faster than my PR to PlutoUtils using the Pluto.Notebook constructor... Have no time for testing this immediately. What I would like to see added are some options like fold_markdown_cells and fold_code_cells (which I however could add in a later PR).
But see also my 2ct in #311 ...

[a9lara]

I tried to open a standard .jl file in Pluto (from the Pluto startup page) and got

Failed to load notebook:

MethodError(union, (), 0x0000000000006c87)

<a href="/">Go back</a>

It would be great to be able to import a normal .jl and have it Plutified. This should e.g. take blocks of code and make them a single cell.

Or perhaps just add a "begin" and "end", then run it on Pluto, and once it is opened in Pluto, delete begin and end and ask Pluto to divide the code into cells.

[fonsp]

This won't work, but you can paste the contents into a cell

[jdadavid]

For the record, I wrote a simple utility for converting plain j.l file to notebook, available at https://github.com/jdadavid/Jl2pluto.jl

[jdadavid]

See discussion at https://discourse.julialang.org/t/import-already-existing-jl-file-in-pluto/44484