True DMABUF support

[gnif]

Now that the sources are no longer restricted from using GPL symbols, can we please have true Linux DMABUF import support?

Currently NVIDIA reports EGL_EXT_image_dma_buf_import however this is not true DMABUF support according to the Linux gfx documentation for DMABUF. Instead the driver is using some internal proprietary format that makes it impossible to use a DMABUF provided by another driver, such as Intel/AMD or some other external device like a capture card or even just local system RAM.

In the past NVIDIA have made several attempts to get true DMABUF support into the kernel which was rejected due to the GPL symbols being unavailable to them, see:

https://www.phoronix.com/scan.php?page=news_item&px=MTIwNDI
https://www.phoronix.com/scan.php?page=news_item&px=MTA1OTU

This would greatly benefit projects like Looking Glass where we are importing full screen textures directly from memory shared with a VM which is writing into the shared memory.

Note: This is already working on AMD and Intel GPUs . Currently we recommend against using NVidia GPUs to our user base due to the lack of this feature.

See our kernel module here where we create a valid DMABUF from shared RAM: https://github.com/gnif/LookingGlass/tree/master/module

[ghost]

REALLY WANT THIS!!!!!

[wertig0n]

+1

[ghost]

+∞

[ad1rie1]

Hello, Are you sure is not already the case? , the GPL function mentioned in the Phoronix article is used.

open-gpu-kernel-modules/kernel-open/nvidia/nv-dmabuf.c

Line 721 in d8f3bcf

buf = dma_buf_export(priv, &nv_dma_buf_ops,

open-gpu-kernel-modules/kernel-open/nvidia/nv-dmabuf.c

Line 739 in d8f3bcf

rc = dma_buf_fd(buf, O_RDWR | O_CLOEXEC);

you can see the use of the GPL function.

[gnif]

Hello, Are you sure is not already the case? , the GPL function mentioned in the Phoronix article is used.

open-gpu-kernel-modules/kernel-open/nvidia/nv-dmabuf.c

Line 721 in d8f3bcf

buf = dma_buf_export(priv, &nv_dma_buf_ops,

open-gpu-kernel-modules/kernel-open/nvidia/nv-dmabuf.c

Line 739 in d8f3bcf

rc = dma_buf_fd(buf, O_RDWR | O_CLOEXEC);

you can see the use of the GPL function.

Yes, it will create a DMABUF for export, but will not use one from another source for import. You can see that it's not even implemented as an IOCTL, only export is, see:

open-gpu-kernel-modules/kernel-open/nvidia/nv.c

Line 2339 in d8f3bcf

case NV_ESC_EXPORT_TO_DMABUF_FD:

The proprietary closed driver has an import implemented, but only if the fd used came from the same kernel module, it does not allow import from other sources. The symbol that is needed to be used is dma_buf_get which is only used once in this code base to check if there is a duplicate DMABUF before creating a new one for export.

[shoffmeister]

I am under the impression that hybrid GPU laptops - i.e. laptops with an integrated Intel GPU ("iGPU") and a discrete Nvidia GPU ("dGPU") - might be suffering from a lacking DMABUF implementation, too.

In a scenario where the Nvidia GPU drives the external monitor (due to physical wiring), but where the Intel GPU continues to supply the data to be displayed (reverse PRIME configuration - see https://wiki.archlinux.org/title/PRIME#Reverse_PRIME), a very high amount of CPU is expended by the X server to do "something" - copying of data into the Nvidia framebuffer, for displaying.

All this CPU consumption takes away available CPU resources, adds heat output, and increases fan noise.

The other way around - running the Nvidia GPU as the primary renderer ("PRIME") - does not exhibit this massive consumption of CPU, even when rendering to a 3072 x 1920 (3K) laptop screen.

[mirh]

PRIME was already working in 2013 dudes, please (even though userspace didn't catch up everything until 1-2 years ago)
There's also no "egregiously negative mention" of reverse PRIME in that page. Just a displaylink bug that got already fixed.
(conversely I could find a similar bug report for intel+amd)

[mirh]

I really don't know what userspace APIs have to do with kernel documentation (where dmabuf hasn't been a problem for almost a decade btw).
And if EGL is not for you, I think you are looking for zwp_linux_dmabuf_v1 (which is what they already use to communicate with mesa IIRC)

[gnif]

I really don't know what userspace APIs have to do with kernel documentation (where dmabuf hasn't been a problem for almost a decade btw). And if EGL is not for you, I think you are looking for zwp_linux_dmabuf_v1 (which is what they already use to communicate with mesa IIRC)

Sorry what are you going on about here?

We ARE using EGLImage in our usage of DMABUF, but NVIDIA do not allow imports of a DMABUF file descriptor that came from a source other then NVIDIA's own closed driver. In the case of the open driver, there is zero import capability at all, as shown above.

https://github.com/gnif/LookingGlass/blob/master/client/renderers/EGL/texture_dmabuf.c#L125-L141

[mirh]

Did you use EGL_EXT_image_dma_buf_import_modifiers?
Everything seems just good for others (plus some quirks).
The guy that also replied to you, got everything nice and dandy a week later.

And whatever you might be doing in the kernel, can't you use the PRIME helpers functions?
I guess it checks out that their module is not calling/doing dmabuf directly.. but it doesn't mean that they cannot "eventually" do the thing anyway. Why don't you check nvidia-drm-gem-dma-buf.c?

[gnif]

Sigh... from NVidia directly as I have discussed this with their engineers already via email:

That said, our driver simply doesn't currently support importing dma-buf FDs created by other drivers, which is just another of these potential limitations. For example, you might be less surprised if the radeon driver failed to import a dma-buf exported by our driver that represented a region of local memory on our card (I.e., video memory). The radeon driver/GPU simply has no way to address this memory, so it clearly won't work. There is nothing incorrect about the radeon driver reporting support for the general dma-buf import mechanism from EGL in such cases. The limitation in our driver is similar, in that our driver has no way to map memory it hasn't allocated into our GPU's address space at the moment. As I mentioned, this is a SW limitation, not a hardware one, and it will very likely be resolved in a future driver release, but I do not consider the current behavior incorrect.

can't you use the PRIME helpers functions

No, this has NOTHING to do with PRIME at all, we have implemented the linux defined standard for a DMABUF which works with every GPU except for NVIDIA. We are moving data out of a Virtual Machine via shared RAM, our use case is special and does not relate at all to anything you are referencing.

https://01.org/linuxgraphics/gfx-docs/drm/driver-api/dma-buf.html

This is used, for example, by drm "prime" multi-GPU support, but is of course not limited to GPU use cases.

If the drivers are going to report support for EGL_EXT_image_dma_buf_import which specifically states:

This extension allows creating an EGLImage from a Linux dma_buf file descriptor or multiple file descriptors in the case of multi-plane YUV images.

And you give it a dma_buf file descriptor that conforms to the standards and it is rejected, then the device does not support EGL_EXT_image_dma_buf_import and should not claim to.

As for our method of import using EGLImage, it's not just our usage of it there, we also use it in an OBS plugin using the OBS API:

https://github.com/gnif/LookingGlass/blob/master/obs/lg.c#L641:L645

These usages both fail ONLY on NVIDIA hardware when the DMABUF comes from a non-nvidia source.

[cubanismo]

As @erik-kz noted below, this particular usage (Importing foreign linear image memory that can export a scatter/gather list) will work soon. However,

And you give it a dma_buf file descriptor that conforms to the standards and it is rejected, then the device does not support EGL_EXT_image_dma_buf_import and should not claim to.

Is there a standard that defines what a dma-buf is and can do other than the Linux kernel code? The EGL extension spec in question is missing an error code for this case, but it's also missing an error code for the case where you pass it an FD that comes from open(/dev/null), so I wouldn't put to much stock in that. As was stated in our earlier correspondence apparently, dma-buf import can fail for any number of reasons at the kernel level or in userspace. Some devices simply can't talk to other device's memory for any number of reasons. Some devices hide more of the complexity in their drivers than ours do by migrating memory around as-needed, a choice which may or may not be to the end-user's benefit overall, but some limitations simply can't be overcome.

[gnif]

This is completely understood, there are obviously going to be circumstances where the FD provided will not be compatible, however in our usage example we are providing it with flat system RAM, this is not memory coming from another device which may have usage restrictions, and as such one would expect that EGL_EXT_image_dma_buf_import would at a bare minimum support this most primitive and basic of usages.

Seeing as EGL_EXT_image_dma_buf_import declares usage of the Linux kernel dma-buf, the kernel's documentation and specification on what and how a dmabuf works would be considered the authority/standard.

See: https://01.org/linuxgraphics/gfx-docs/drm/driver-api/dma-buf.html

As we have implemented the dma-buf interface as per specification, one would expect that anything advertising support for EGL_EXT_image_dma_buf_import would fully support the provided fd in such a basic usage example. Both AMD and Intel both conform and fully support this extension, only NVIDIA have been problematic with it.

This said, we are very excited to see this fixed in the next release and very appreciative of your efforts. We would love to see this in the closed driver also for those running older generation GPUs too, specifically those on laptops where dma-buf has the greatest performance impact for our users.

[erik-kz]

So you're right that, at the moment, importing a DMA-BUF from another driver as an EGLImage doesn't work. However we actually do have the infrastructure to do this in our kernel modules. Our X driver uses it for reverse-PRIME support, for example. The problem is just that this wasn't wired up in EGL.

Fortunately, this has been fixed and should be working as of the next driver release (525).

Unfortunately, looking at https://github.com/gnif/LookingGlass/blob/master/client/renderers/EGL/texture_dmabuf.c#L125-L141 it seems that you then proceed to bind the EGLImage to GL_TEXTURE_2D. If no format modifier is provided when creating the EGLImage, we assume it's pitch-linear. The problem is that we only allow linear buffers to be bound to GL_TEXTURE_EXTERNAL_OES. This is indicated by the fact that eglQueryDmaBufModifiersEXT sets the external_only flag for DRM_FORMAT_MOD_LINEAR.

Depending on how you use this texture, it may be possible to simply bind it to GL_TEXTURE_EXTERNAL_OES instead of GL_TEXTURE_2D. I believe the only restriction is that you can't attach it to an FBO.

[gnif]

Thanks for the information, I have just added support for DRM_FORMAT_MOD_LINEAR now and updated our code base in a experimental branch to use GL_TEXTURE_EXTERNAL_OES. Hopefully we now will meet the required criteria for this when it's ready.

[erik-kz]

Note that you should make sure linear is really what you want. Who will be rendering to these buffers? will it be an NVIDIA GPU? If so, the image data would actually be using our block-linear (tiled) layout and trying to interpret it as linear might not work. For use-cases like PRIME where we share a buffer with another GPU we copy / convert the rendered image to a linear buffer, but that needs to be done explicitly.

If you are simply sharing a buffer between processes on the same GPU, it would also be more efficient to avoid the conversion to linear and plumb the block-linear format modifier through.

[gnif]

The buffers are linear buffers coming from a guest virtual machine where a process is capturing the screen via either DXGI Desktop Duplication or NvFBC and copying the frame into a virtual device that has a block of ram mapped to it's BAR the host can access.

Unfortunately we do not have any more direct means of getting the data from the guest VM that is accessible to the general public on consumer hardware, so we are limited to what these APIs provide us with

[absolutelynothelix]

@erik-kz, @cubanismo, any plans on implementing the x dri3 extension? currently there is no way to export an x pixmap as a dma buf to e.g. import it in vulkan and that's... a little bit sad because other ways are incredibly slow and inefficient.

[cubanismo]

Correct, and we also support the closely related VK_EXT_image_drm_format_modifier extension since the same driver release. Basically everyone should always set modeset=1 at this point until we can work through the few remaining internal issues to make that the default. It adds more than just the DRM-KMS modeset functionality implied by its name.

[absolutelynothelix]

i'll try this tomorrow and report back, but without this vulkaninfo doesn't report the VK_EXT_external_memory_dma_buf extension's presence (but reports the VK_EXT_image_drm_format_modifier one).

also, while i have a rare opportunity to contact nvidia engineers directly, do you have any suggestions on participating in the driver's improvement? from a compositor's perspective it's very hard to work with the nvidia's driver because of a lot of features are missing and a lot of bugs are present instead. for example, all of the mechanisms that could be used to implement frame pacing and that are supported by the nvidia's driver seem to be buggy: the x present extension is unreliable (a question on nvidia developer forums about this) and the GLX_SGI_video_sync glx extension starts to misbehave after the pc is suspended and resumed. we've tried to create topics on the nvidia developer forums but rarely got an answer so it's questionable whether we should continue doing this, especially if it requires some time to create a minimal example to reproduce an issue. we're happy to help but we don't know how.

[cubanismo]

Note VK_EXT_image_drm_format_modifier is still reported with dma-buf support because while the two pair nicely, DRM format modifiers work fine with other external handle types in Vulkan as well.

In general, we always appreciate bug reports and feedback. We can't always provide immediate fixes or implementations of feature requests. However, as Erik notes, just having the demand expressed somewhere in writing, especially with well-reasoned motivations included as is done here, helps prioritize work internally. Even if no one responds, we can mine that data through searches later if additional requests come in or we arrive on the same ideas ourselves and need to build a case for working on it.

Officially, the developer forums are the correct place for notifying us of Linux driver issues, and this github repository is supposed to be specific to the open RM codebases, which is arguably how this issue started out. There are unfortunately only a handful of us monitoring both the forums and this repo, and we have finite time. That said, we are working on improving responsiveness in the forums. For developer-style questions, especially those related to window-system integration stuff like compositors, you may have better luck mentioning us somewhere on github or FDO gitlab, as appropriate.

[absolutelynothelix]

sorry for the delay. i've booted the system with the nvidia_drm.modeset=1 kernel parameter and vulkaninfo indeed reports the presence of the VK_EXT_external_memory_dma_buf extension now (and, surprisingly, it didn't break anything). thanks!

For developer-style questions, especially those related to window-system integration stuff like compositors, you may have better luck mentioning us somewhere on github or FDO gitlab, as appropriate.

@cubanismo, does this mean that if we (picom developers) encounter an issue that we consider a bug in the nvidia's driver we can ping you or erik in a corresponding issue in picom's github repository for a clarification?

[cubanismo]

With my current github settings, yes, that will generate another entry in my inbox. Erik and others would have to answer for themselves. As I say, no guarantee we'll be able to respond to everything in finite time given we're finite humans, but I'll at least read it.

[hgaiser]

Does this change anything for your purposes @gnif ?

Added support for mmap of exported DMA-BUF objects.

https://www.nvidia.com/en-us/drivers/details/233008/

[cubanismo]

I consider this resolved. That changelog entry isn't directly related, but we've supported importing "foreign" dma-bufs for several releases now via EGL and Vulkan. Note this is done through the kernel's PRIME helpers by way of nvidia-drm, not the separate dma-buf code in resource manager, which isn't used by the OpenGL or Vulkan drivers, so the code quoted earlier in this issue and various claims extrapolated based on it are all unrelated and inaccurate. If there are remaining issues with dma-buf support, please report it in a new issue.

[gnif]

For our purposes this was resolved after I made the changes to our code as seen here: #243 (reply in thread)

[ioquatix]

Thank you to everyone involved in resolving this issue. I’m happy to report that, with the latest drivers and kernel updates, both streaming and screen recording have improved significantly over the past year. I understand that reaching this point took a tremendous amount of work across multiple teams and projects, and I’m deeply grateful for all the effort that went into making this happen.