[Feature] 3D Parallax using Depth map

[rocksdanister]

Describe the problem

There are not many easy methods of personalizing native application, adding a background image is one popular method.. but making it look 3D would take it one step further!

Use cases:

1. Change background when page changes. Example: Nature catalogue app
2. Change background when listview item selection changes. Example: Ambie music app
3. Depth cards (see preview.)
4. Apps with gyroscope support.

Describe the solution

Creating a 3D scene can get complex and not practical; instead we can use depth map (image) and shaders to create a similar effect by shifting the pixels at different speeds based on the depth.

Demo running on opensource app Ambie:

Ambie.Depth-1.mp4

Demo depth cards:

depth_cards.mp4

Advantages of depth map:

• Small size: no additional dependency other than Win2D; depth map is a simple image (can be smaller size than the input.)
• Simple: depth map can be generated using machine learning, captured using smart phone camera or drawn using paint.
• Performant: simple pixel shifting using shader.

Alternatives

Creating a 3D scene.

ParallaxView for 2D effect.

Additional info

I am still learning Win2D, will post a sample project in a bit (need some cleaning.)

Help us help you

Yes, I'd like to be assigned to work on this item.

[ghost]

Hello, 'rocksdanister! Thanks for submitting a new feature request. I've automatically added a vote 👍 reaction to help get things started. Other community members can vote to help us prioritize this feature in the future!

[rocksdanister]

Made a sample project:
https://github.com/rocksdanister/DeptEffectDemo

[michael-hawker]

Awesome @rocksdanister, thanks for sharing! Looks pretty straightforward for UWP. CanvasAnimatedControl isn't supported in WinUI 3 yet though I think right? Is that the challenge there? @Sergio0694 any ideas when the Win2D team is planning to enable that scenario? Is this the best issue to track that on their GitHub for that?

@rocksdanister are there common tools to create these depth maps for this effect? It seems like all a control would need is the 3 asset files, correct? Just wondering how folks produce those assets?

[Sergio0694]

"any ideas when the Win2D team is planning to enable that scenario?"

We don't currently have an ETA for CanvasAnimatedControl in WinUI 3 at the moment. If you really need animated rendering on WinUI 3 using Win2D right now you'd have to use a CanvasSwapChain and handle all the rendering thread yourself 😅

[rocksdanister]

are there common tools to create these depth maps for this effect? It seems like all a control would need is the 3 asset files, correct? Just wondering how folks produce those assets?

Smartphone camera
Some smartphones like Google Pixel include this data, particularly in bokeh mode:
https://www.photopea.com/tuts/extract-depth-map-from-photo-image/
https://www.gsmarena.com/feast_your_eyes_on_these_nokia_9_pureview_camera_samples-news-35690.php

Approximation using ML
There are machine learning algorithms that can approximate it:
https://github.com/isl-org/MiDaS
I have a C# implementation on my app (GPL license be warned~) using ONNX Runtime:
https://github.com/rocksdanister/lively/tree/core-separation/src/Lively/Lively.ML/DepthEstimate
(the ml model is around 60mb in size.)

There are online versions of this ML
https://huggingface.co/spaces/jonjhiggins/MiDaS

[michael-hawker]

@Sergio0694 I know you called out ComputeSharp as a helper here at least for the shader compilation step. Since ComputeSharp has it's own interop with Win2D directly for UWP/WinUI3 that'd get around the CanvasAnimatedControl limitation in WinUI 3 right?

Though we'd have to do the odd subclass aliasing of the XAML control if we make it a templated control: Sergio0694/ComputeSharp#442 - unless we just inherit from AnimatedComputeShaderPanel directly and then can just have the new control under a single namespace for consumers at least.

[Sergio0694]

No unfortunately that's orthogonal. Using ComputeSharp would make the code simpler and nicer (no need to use HLSL, no need for the custom build script to compile the shader, and you'd also get properly strongly-typed properties for the shader), but you'd still end up with a Win2D effect you'd have to draw. Which means that on WinUI 3 you'd still not have the animated canvas to draw that with. Also worth noting though, taking a dependency on ComputeSharp would likely not be ideal here if we want to publish this as a standalone control/library 😅

[michael-hawker]

Oh, I thought the AnimatedComputeShaderPanel could draw the effect as well still?

[Sergio0694]

Ooh, I get what you mean now, sorry ahah
Yeah, that'd work fine on WinUI 3, but that's using DirectX 12 compute shaders. This proposal instead uses Direct2D pixel shaders. You can of course implement the same control on top of DirectX 12, and do that via ComputeSharp as you mentioned, and yeah in that case WinUI 3 would be identical to UWP. II thought you were talking about the Win2D version of ComputeSharp earlier 😅

[michael-hawker]

I'm not as up-to-date on the latest offerings in ComputeSharp, so I think I was mixing the two concepts between DX12 and Win2D versions. I guess I didn't realize there's no equivalent panel for Win2D yet like there is for the DX12 shaders. 🙂 Would be cool if we you had a Win2D brush interop helper or something... 😉

[rocksdanister]

To me these are the todo/blockers:

• WinUI 3 support: In WinUI3 CanvasAnimatedControl is not present in Win2D, I didn't use ComputeSharp to avoid dependency since it is a simple control. So either we have to build our own animation loop or like mentioned this can be written DX12 compute shader using ComputeSharp.
• Gyro and touch support: I don't have devices with touch or gyroscope, the touch implementation should be identical to mouse but gyro no idea.
• Getting data: The animation won't work if there are control elements ontop since mousemove won't fire, the mouse/touch data needs a source as input or get from parent(?)

[rocksdanister]

Updated WinUI3 with ComputeSharp demo.

[michael-hawker]

@rocksdanister pretty cool! @Sergio0694 this is where I was thinking it'd be cool if the ComputeSharp panels could just pass 'inputs' like the mouse to the shader, kind of like what happens on ShaderToy, and there's just an interface you can add to your shader that adds the mouse coords or something...? Then all this control becomes is the shader code wrapped around a control which takes the image/depth map as input... (at least for a simple version leveraging ComputeSharp).