Can we make built-in modules reloadable?

[Totktonada]

The idea

If we unload a built-in module, we unable to require it again:

tarantool> package.loaded.clock = nil
---
...

tarantool> require('clock')
---
- error: "module 'clock' not found:\n\tno field package.preload['clock']\n\tno file
    '/home/alex/clock.lua'\n\tno file '/home/alex/clock/init.lua'\n\tno file '/home/alex/clock.so'\n\tno
    file '/home/alex/.rocks/share/tarantool/clock.lua'\n\tno file '/home/alex/.rocks/share/tarantool/clock/init.lua'\n\tno
    file '/home/.rocks/share/tarantool/clock.lua'\n\tno file '/home/.rocks/share/tarantool/clock/init.lua'\n\tno
    file '/.rocks/share/tarantool/clock.lua'\n\tno file '/.rocks/share/tarantool/clock/init.lua'\n\tno
    file '/home/alex/.rocks/lib/tarantool/clock.so'\n\tno file '/home/.rocks/lib/tarantool/clock.so'\n\tno
    file '/.rocks/lib/tarantool/clock.so'\n\tno file '/home/alex/.luarocks/share/lua/5.1/clock.lua'\n\tno
    file '/home/alex/.luarocks/share/lua/5.1/clock/init.lua'\n\tno file '/home/alex/.luarocks/share/lua/clock.lua'\n\tno
    file '/home/alex/.luarocks/share/lua/clock/init.lua'\n\tno file '/usr/local/share/tarantool/clock.lua'\n\tno
    file '/usr/local/share/tarantool/clock/init.lua'\n\tno file '/usr/share/tarantool/clock.lua'\n\tno
    file '/usr/share/tarantool/clock/init.lua'\n\tno file '/usr/lib64/lua/luarocks/share/tarantool/clock.lua'\n\tno
    file '/usr/lib64/lua/luarocks/share/tarantool/clock/init.lua'\n\tno file '/usr/local/share/lua/5.1/clock.lua'\n\tno
    file '/usr/local/share/lua/5.1/clock/init.lua'\n\tno file '/usr/share/lua/5.1/clock.lua'\n\tno
    file '/usr/share/lua/5.1/clock/init.lua'\n\tno file '/usr/lib64/lua/luarocks/share/lua/5.1/clock.lua'\n\tno
    file '/usr/lib64/lua/luarocks/share/lua/5.1/clock/init.lua'\n\tno file '/home/alex/.luarocks/lib/lua/5.1/clock.so'\n\tno
    file '/home/alex/.luarocks/lib/lua/clock.so'\n\tno file '/usr/local/lib64/tarantool/clock.so'\n\tno
    file '/usr/lib64/tarantool/clock.so'\n\tno file '/usr/lib64/lua/luarocks/lib/tarantool/clock.so'\n\tno
    file '/usr/local/lib64/lua/5.1/clock.so'\n\tno file '/usr/lib64/lua/5.1/clock.so'\n\tno
    file '/usr/lib64/lua/luarocks/lib/lua/5.1/clock.so'"
...

There is no reason why we can't load it second time. The Lua code is built into tarantool's executable and we can add a loader that reads it¹.

Why one may need it? There is a typical application upgrade scenario: unload all modules, load it again and reinitialize the application (some applications also kill fibers, cleanup global variables and perform specific actions like http server routes removal). Such code should leave built-in modules loaded: either hardcode a list of them or collect the list early during startup. Moreover, there are modules that reloads modules:

• https://github.com/moonlibs/package-reload/ (see also https://github.com/moonlibs/stash)
• https://github.com/tarantool/reload/ (Is it used anywhere? I don't know.)
• https://github.com/tarantool/cartridge/blob/master/cartridge/hotreload.lua (see [2pt] Support roles hot-reload cartridge#1100)

The idea is to make built-in modules reloadable and simplify application reloading. Hopefully even eliminate the need in the modules that reloads modules.

There are two things that disquiets me:

• Built-in modules in Lua and LuaJIT are not reloadable. If we'll unload a built-in module (say, os), the next require will fail. But there is no practical reason to make tarantool's built-in modules reloadable and leave luajit's ones non-reloadable. Technically the change, which allows to reload luajit's built-in modules, is not backward compatible.
• We anyway may need to perform module specific clean up: say, kill fibers.

So, maybe, it would be better to agree on some common reload API for modules (say, package.loaded.foo.reload()) and deprecate package.loaded.foo = nil require('foo') way as inaccurate one? At least old captures will be available in the reload function without hacks with burying them inside _G.

(NB: Think, how to better handle modules with several submodules: foo, foo.bar, foo.baz.)

References

Those thoughts appear in my head during writting the relevant issue https://github.com/tarantool/conf/issues/2.

Footnotes

Footnotes

1. In theory we can use package.preload, but it is accessible for a user that may lead to some collision. Aside of this, the preload loader is the first one: a user may remove a built-in module from package.loaded to load an external replacement module with the same name. ↩

[tsafin]

I believe there is no any reasonable explanation why we could not implement reload for Tarantool builtin modules. Certainly, package.loaded.foo.reload() looks less verbose and more concise, but also there is no hard complications foreseen even for older idiom package.loaded.foo = nil require('foo').

[Why not simply keep the list of unloaded builtin modules, where to look into first the next moment it will be required. This is pure speculation though - there should be multiple ways to implement it if needed]

[filonenko-mikhail]

The case which I met two days ago is that sometimes I want monkey patch built-in module at runtime.
The case is about

• Tarantool 2.7.1 from centos repo, it means compile release flags enabled
• Repro uses json and base64
• Repro make fio fiber and coio thread stuck, probably memory corrupt at global variables
• Repro does not work on hand crafted tarantool, because it's hard to enable release flags