Discussion: Strategy to remove shell-exec from compilation path

[lxfontes]

TinyGo's current behavior is to handle module -> component conversion for -target wasip2.

Where is shell exec today?

wit-bindgen-go

https://github.com/bytecodealliance/wasm-tools-go/blob/main/wit/load.go#L54

wit: parsing / world resolver

Uses wasm-tools to convert wit -> resolved world json.
The resulting json is converted into structs for codegen.

TinyGo

https://github.com/tinygo-org/tinygo/blob/release/builder/build.go#L864

wit: parsing / world resolver
wasm metadata: attach wit to wasm (wasm-tools component embed)
wasm component encoder: convert wasm module to component ( wasm-tools component new )

Uses wasm-tools to adapt a wasm module to component.

Discussion

Should TinyGo / Go generate core wasm modules and leave metadata/component encoding to 3rd party tools; or should they be part of a wasi target ( or goos/goarch combination )?

Assuming we implement wit parsing / wasm patching in Go:

• How do we keep up with wit spec?
• In what sequence we'd create these packages?

[ydnar]

TinyGo, by design and convention, has depended on external tools to handle parts of compilation (LLVM, asyncify, wasm-ld, among others). Based on statements made by the Go team and by historical precedent, I (and others, I believe) assume that this strategy would not apply to mainline ("big") Go. The go command (and associated toolchain) contains the necessary capability to compile (and cross-compile) binaries for a variety of platforms, including wasm.

• Is the issue that shell exec exists at all in the current Component Model tooling for Go?
• Do we (the authors and maintainers of this tooling, TinyGo, and Go) agree that this is an issue worth addressing?
• Wasm in Go has so far taken an approach of "prototype in TinyGo, learn and implement later in big Go."
• With respect to tooling and compilation, what if TinyGo skipped the wasm-tools component steps, and generated a wasm32-wasip2-module form suitable for consumption by wasm-tools or conforming runtimes?

[lxfontes]

we chatted about it during BA's meeting and no decisive actions other than researching what wasm-tools is actually doing under the hood and follow up next meeting.

We identified a few options to pass down wit:

• Continue with wit directory so wasm-tools (or something else) can perform embedding
• Nuke the wit directory and rely on information from generated code in downstream tooling. Options:
  • go embed per package: Attaching each interface wit to its corresponding go package
  • go embed per project: Attach the whole wit to the entrypoint package under compilation
  • A combination of go:wasm{import,export} and markers to carry type information, so the wit can be discarded after codegen

With respect to tooling and compilation, what if TinyGo skipped the wasm-tools component steps, and generated a WebAssembly/component-model#378 form suitable for consumption by wasm-tools or conforming runtimes?

This is a 🔑 point. We touched briefly on the point wrt making TinyGo / BigGo emit wasm32 module and leave wasm-tools steps to other tools. It would allow us to defer implementations based on upstream stability ( and yes, I'm looking at you WIT spec! 😂 )

The interesting bit here is wasm-tools-go is already a requirement, so something like this is not far from reality:

# generate bindings
go run wasm-tools/cmd/wit-bindgen-go

# compile module ( core wasm )
tinygo build ... -o module.wasm

# componetize it
go run wasm-tools/cmd/assemble-component -in module.wasm -out component.wasm

wasm-tools-go as pre & post compilation steps, having wit-bindgen-go and assemble-component defining their own api / data passing mechanism.

The first incarnation of assemble-component could be a full shell exec into wasm-tools, and feels this is a pattern acceptable in Big Go.

again, no decisions made but wanted to relay the discussion.

[ydnar]

I have WIT generation working with tree shaking in a branch, so we can emit minimal WIT files containing a single world to a world's Go package directory.

wasm-tools-go is only a prerequisite for code generation, so it's not really required to compile and build a Go program. This is similar to other tools used by big Go projects like protoc or avo.

[ydnar]

Work is progressing here:

1. @Mossaka has put up a draft PR to run wasm-tools as a WASI 0.1 module inside of wit-bindgen-go: internal/wasmtools: run wasm-tools.wasm in embedded Wazero runtime #252
2. @lukewagner is working on new module mode linker name mangling spec: Add BuildTargets.md WebAssembly/component-model#378
3. @alexcrichton merged early support for the new linker name mangling into wasm-tools: Implement support for the standard mangling scheme in the component model wasm-tools#1828