Allow lazy evaluation of some expressions

[Rangi42]

This is an internal requirement for features including:

• [Feature request] User-defined functions #201: User-defined functions (e.g. def f(x) = x * n where f(5) will multiply by whatever n currently is)
• [Feature request] WHILE loops #560: WHILE loops
• Make && and || short-circuiting #619: Short-circuiting && and || (so 2+2==4 || foo/bar==1 will evaluate to 1 even if foo is undefined or bar is 0)
• [Feature request] Ternary operator #621: Ternary operator (cond ? then : else)

[Rangi42]

I don't know if constructing a whole RPN tree would be necessary for these, at least not for short-circuiting logic operators. Instead, consider an isEvaluating flag for the parser. When given false && X, as soon as it reaches the && is sets isEvaluating = false for the duration of parsing X; likewise for true || X, and for both cases of ?:. This flag would disable symbol lookup and actual math and just keep evaluating a trivial 0 value, so syntax errors would still occur but not evaluation errors.

User-defined functions are more complicated but might also be doable without RPN trees. Their body would be captured like a macro body when they're defined. Then when they're called, the parser would expand the function body similar to a macro body, but instead of substituting the parameters \1, \2, etc at lex time, it would evaluate the parameters like any other symbols at parse time. This would mean keeping a context of which function it's in so that the parameter names map to the right argument values (symbol "layering" as mentioned in #662).

[ISSOtm]

The proposed solution for user-defined functions would imply re-parsing them, which is slow, and can be avoided since the expression tree won't change. This is why I'm pushing so hard to decouple expression evaluation from parsing.

Evaluation can still be eager for purely numeric expressions (1 + 1 will never change), and yield a "bad" RPN expression (say, pointing to NULL, so it doesn't require allocating a 1-byte buffer) for e.g. 1 / 0; that expression could be discarded due to short-circuiting, or turned into an error when trying to get a number out of it.

I believe this is necessary for a proper user-defined function implementation, notably to avoid technical debt, and since I think it also encompasses the other usages, I think we shouldn't use "partial" impls. This is why I've been reluctant to merge #665.

[ISSOtm]

Here are my thoughts on implementing this:

• Change the RPN IDs to have $80+ be errors; so the current immediates ($80 and $81) would become e.g. $70 and $71.
• When encountering a hard error, such as a division by 0, emit a RPN "error" byte, say $80 for RPN_DIV_BY_ZERO.
• RPN errors get turned into hard errors when:
  • The expression's value is extracted (either by RGBASM in the const productions, or by RGBLINK)
  • Attempting to emit a RPN expression solely containing an error (to catch db 1 / 0 early)

This way, db 1 || 2 / 0 would first generate the RPN buffer RPN_CONST 1 RPN_CONST 2 RPN_CONST 0 DIV, which RGBASM would collapse to RPN_CONST 1 RPN_DIV_BY_0, and seeing the ||, RGBASM would see the first constant, and ignore the RHS.
(Note that RGBASM would actually internally store the expressions differently due to their "const" status, but the principle stays the same.)
Assuming 1 is an unknown (wNonZeroLabel), RGBLINK would basically perform the same, though instead having to skip the RHS in the buffer, but that should be doable by keeping track of pushes and pops in the RPN stack, without actually evaluating anything.

Am I missing anything there?

[Rangi42]

To avoid reserving half of the IDs for errors, the defined error values could count down from $FF, and check for a minimum threshold value instead of the high bit.