MutationObserver flag to observe mutations in (open) shadow trees to facilitate polyfills of HTML syntax

[LeaVerou]

What problem are you trying to solve?

It is currently exceedingly difficult (and slow) to implement polyfills of certain features that affect HTML syntax across both light and shadow DOM because it requires observing mutations in shadow trees as well, which is currently incredibly difficult, and all solutions have prohibitive performance.

Some examples:

• Any of the custom attributes proposals (example)
• Any new HTML element or attribute

What solutions exist today?

The current options are:

1. 🤷🏽‍♀️ The shrug method: Use a regular document-wide mutation observer and simply accept that the polyfill will not work correctly when shadow trees are involved
2. 🩹 The monkey patch method: Wrap HTMLElement.prototype.attachShadow() with code that calls mutationObserver.observe() on it, like so:

let originalAttachShadow = HTMLElement.prototype.attachShadow;
HTMLElement.prototype.attachShadow = function(...args) {
	let node = originalAttachShadow.call(this, ...args);
	mutationObserver.observe(node, options);
	return node;
}

3. 🤕 The painful method: When the mutation observer detects new elements, check if they have a shadowRoot and call observe() on that too. Loop over all existing elements and attach mutation observers to the shadow roots of those with one.

As you can see, all of these are pretty terrible. They are slow, obtrusive, and wasteful. Can we do better?

How would you solve it?

Option 1

This is the most straightforward, lowest effort option: Simply add a new option to MutationObserverInit to observe (open) shadow trees. Name TBB, some ideas might be shadow, shadowRoots.

The MVP could be a boolean option, which could later be extended to more granularity (e.g. how many levels deep? What types of shadow trees (open, open-stylable etc)).

Option 2

It could be argued that MutationObserver.prototype.observe() is the wrong tool for the job. It was designed to observe mutations under a specific root, and has been since contorted to cater to these types of use cases. Perhaps a different method could be used to signal "observe this everywhere you can", e.g. MutationObserver.prototype.globalObserve().

Anything else?

No response

[WebReflection]

As author of all polyfills from v0 to current state, I fully support this proposal. My workaround is not super slow but it requires me patching the global attachShadow method on native classes which is a practice I usually avoid at all costs and something I've been advocating against for just about forever ... yet polyfill gotta polyfill, so that if this new proposal is the only thing that might need a greedy polyfill around until everyone is onboard, it's definitively worth the effort to me.

On a side note: the closed shadow might still be addressed somehow and I feel like the elephant in the room here is that we don't have a way for Websites authors to decide what is trusted as script that could patch natives, and what isn't ... CSP is not a great answer to this disambiguation, but maybe this discussion should be done in other venues.

[trusktr]

An alternative to this might be to be able to observe the composed tree, because in some cases we might want to know the nodes that will participate in the final output:

I've had to implement composed tree tracking so that webgl-rendered custom elements would render correct results when composed via slots into shadow roots. I'd like to release some of these tools separate libs, maybe it'll help ideate.

Even without custom rendering, it might be beneficial to react to composed nodes, avoiding extra work for trees that are not active. F.e. suppose a client-side route switcher modifies slot attributes to show/hide content, and we want to observe only content in the active tree.

[trusktr]

Over in

• observing the composed tree synchronously webqit/realdom#3

@ox-harris implements options.subtree = 'cross-roots' in the realdom lib for synchronously observing added/removed elements across shadow root boundaries, and gets us a little closer to synchronous observation of the composed tree (f.e. detect composed children, composed parent, etc).

Not only is the usage simpler than with MutationObserver (because MO events fire out of order) and lead to simpler code, but composed tree tracking code will also be as simple when that's ready (ComposedChildrenObserver, ComposedParentObserver, ComposedMutationObserver, etc).

[smaug----]

@LeaVerou could you explain a bit more what is the use case here?

[LeaVerou]

@LeaVerou could you explain a bit more what is the use case here?

Sure, what is unclear in my examples?

[smaug----]

"to implement polyfills of certain features that affect HTML syntax across both light and shadow DOM because it requires observing mutations in shadow trees as well" ... "Any new HTML element or attribute"

So what issue there exactly are you trying to solve?

[LeaVerou]

"to implement polyfills of certain features that affect HTML syntax across both light and shadow DOM because it requires observing mutations in shadow trees as well" ... "Any new HTML element or attribute"

So what issue there exactly are you trying to solve?

To implement polyfills of new HTML syntax, you need to be able to react to relevant changes in the DOM. E.g. for a new attribute foobar, you need to be able to react when someone adds a foobar attribute anywhere, including deeply nested Shadow DOM. Doing this with the current MutationObserver API is a hack (I listed how in the OP). Does that make sense? If not, it would help to try and formulate an actual question on what part you find confusing.

[smaug----]

I see, you want to polyfill new global attributes basically.

[LeaVerou]

I see, you want to polyfill new global attributes basically.

That's one use case. Or new native elements. Or new attributes on native elements.

[LeaVerou]

Just realized, this wouldn't only help with polyfilling HTML syntax, but also attributes like element.currentLang: whatwg/html#7039

[ox-harris]

Over in

• observing the composed tree synchronously webqit/realdom#3

@ox-harris implements options.subtree = 'cross-roots' in the realdom lib for synchronously observing added/removed elements across shadow root boundaries, and gets us a little closer to synchronous observation of the composed tree (f.e. detect composed children, composed parent, etc).

Not only is the usage simpler than with MutationObserver (because MO events fire out of order) and lead to simpler code, but composed tree tracking code will also be as simple when that's ready (ComposedChildrenObserver, ComposedParentObserver, ComposedMutationObserver, etc).

Thanks @trusktr for the mention.

Re: the above, there's been a clear misfit between what I do and what the MutationObserver APi does. For example, I sometimes just want to say:

• observer the style attribute (or draggable attribute, etc) globally. Which exactly means "also within Shadow trees".
  • do that synchronously, so that I can closely mimic native timing for the behaviour I want to attach to the element.
    • and in some of those cases: do that ahead of native timing; i.e. let me intercept this attribute before the DOM sees it. E.g. to rewrite animation properties in the style attribute before the 'animationstart` event fires.

(A reall case in point: implementing the upcoming OOHTML proposal.)

It turns out to be extremely difficult and mostly impossible to use the MutationObserver API in those scenarios. Writing the above referenced library - RealDOM - a realtime, and more declarative, DOM API, was the logical answer. Now, while I didn't pursue that as a proposal, perhaps some of ideas there could help in the ideation in here.

PSS: I didn't pursue that as a proposal because I was unsure how much typical my usecase was. But I think it's more of a moving story now.

[patrick-mcdougle]

Just in case another use case could inspire why this could be compelling...

Let's say I have a component auto-loader (like shoelace autoload) that detects custom elements being inserted into the DOM using MutationObserver. It'd be swell if some of my custom components (or perhaps 3rd party components) that use shoelace components in open shadowTrees could also trigger the auto-loading mechanism.

[WebReflection]

As WebKit / Safari is landing scoped Custom Elements we should be careful with polyfills or observing custom elements in there because the inner registry might already know, or take care of, those elements ... so that <shoe-thing> in there might not need to be bootstrapped, when in ShadowRoot, like the <shoe-thing> on the main light-dom, actually this is a very risky thing to do for whoever is dealing with patched attachShadow too, if that logic assumes all those potential custom elements require outer main DOM intervention.