Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ubuntu and Mellanox OFED version that works with Moongen? #301

Open
Assar-Westman opened this issue Mar 9, 2021 · 10 comments
Open

Ubuntu and Mellanox OFED version that works with Moongen? #301

Assar-Westman opened this issue Mar 9, 2021 · 10 comments

Comments

@Assar-Westman
Copy link

Thanks for a really nice high speed packet generator.
I used the Moongen packet generator a lot a year ago on Mellanox mlx5 100 Gbe cards. Now I needed to do some more benchmarks using Moongen using these 100 Gbe cards. During last year I updated the computer to latest version of ubuntu and also the Mellanox OFED was updated to latest version. Now when I try to use Moongen I get this error

PMD: net_mlx5: ibv_exp_query_device() failed

So I wonder what is the latest version of Ubuntu and Mellanox OFED that are supported by Moongen?

Best regards Assar W.

@emmericp
Copy link
Owner

sorry, our support for mellanox is somwhat bad due to their terrible driver model. Can you try to use the branch dpdk-19.05?

@Assar-Westman
Copy link
Author

Thanks for the suggestion. When I using the 19.05 branch I get a lot of compile error like

libmoon/deps/dpdk/drivers/net/mlx5/mlx5_glue.c:754: undefined reference to `mlx5dv_open_device'
I have added the preferred deb packages, except
libmlx5[...].deb
libmlx5-dev[...].deb
which are missing in the used Mellanox OFED 5.2 distribution
It feels like some Mellanox library is missing maybe it is supplied with libmlx5...

Best regards Assar W.

@Assar-Westman
Copy link
Author

Now things looks much better. I have been able to compile MoonGen on my box using the DPDK 19.05 branch. What It did was to edit libmoon/CMakeLists.txt and add to add two lines after

#Add Mellanox libraries if options are specified
if(USE_MLX5 OR USE_MLX4)
SET(DPDK_LIBS ${DPDK_LIBS} ibverbs)

so it looks like this

#Add Mellanox libraries if options are specified
if(USE_MLX5 OR USE_MLX4)
SET(DPDK_LIBS ${DPDK_LIBS} ibverbs)
SET(DPDK_LIBS ${DPDK_LIBS} mnl)
SET(DPDK_LIBS ${DPDK_LIBS} mlx5)

I really needed the LUA jit stuff as I have to attach current time to to the crafted data part of the UDP package.

Again thanks for a really nice software package. Assar W.

@michael-kaplan
Copy link

Did you succeed? I'm curious if you got DPDK to include the Mellanox drivers, --mlx5 flag to build.sh not seem to be working for me. have very similar setup.

@Assar-Westman
Copy link
Author

My moongen setup now works with Mellanox OFED driver version 5.2-2.2.0.0, Ubuntu version 20.04.2 LTS and kernel 5.8.0-44-generic. For me the crucial thing was to use branch 19.05, install the whole Mellanox OFED driver package and also to install libmnl-dev package, and of course edit libmoon/CMakeLists.txt as seen above
AW

@michael-kaplan
Copy link

Fascinating. I am using Mellanox OFED driver 5.3-OFED.5.3.1.0.0.1, Ubuntu 20.04.2 LTS, and kernel 5.4.0-26-generic. I changed the MoonGen branch to 19.05 and built with ./build.sh --mlx5 (after updating CMAKELists.txt, built successfully).

Yet for whatever reason I continue to get a 0 device error. Arg! I have a MT28880, do you know which you have? I also find some of the complaints concerning, but do not know what to make of them. I suppose there's also updating the Ubuntu kernel.

[INFO]  Initializing DPDK. This will take a few seconds...
EAL: Detected 64 lcore(s)
EAL: Detected 2 NUMA nodes
EAL: Multi-process socket /run/user/10002/dpdk/rte/mp_socket
EAL: No available hugepages reported in hugepages-1048576kB
EAL: Probing VFIO support...
EAL: VFIO support initialized
EAL: Cannot obtain physical addresses: Success. Only vfio will function.
EAL: Couldn't get fd on hugepage file
EAL: Couldn't get fd on hugepage file
error allocating rte services array
EAL: FATAL: rte_service_init() failed
EAL: rte_service_init() failed
[INFO]  Found 0 usable devices:
[FATAL] Lua error in task master
...me/mkaplan/prod3/MoonGen/build/../libmoon/lua/device.lua:100: there are only 0 ports, tried to configure port id 0

@Assar-Westman
Copy link
Author

I have a MT28800 Family ConnectX-5 PCie 4.0 I see in my notes that I only compile Moongen with GCC version 7.5
Of course you have run both bind-interfaces.sh and setup-hugetlbfs.sh
AW

@liqia
Copy link

liqia commented Dec 18, 2021

Now things looks much better. I have been able to compile MoonGen on my box using the DPDK 19.05 branch. What It did was to edit libmoon/CMakeLists.txt and add to add two lines after

#Add Mellanox libraries if options are specified if(USE_MLX5 OR USE_MLX4) SET(DPDK_LIBS ${DPDK_LIBS} ibverbs)

so it looks like this

#Add Mellanox libraries if options are specified if(USE_MLX5 OR USE_MLX4) SET(DPDK_LIBS ${DPDK_LIBS} ibverbs) SET(DPDK_LIBS ${DPDK_LIBS} mnl) SET(DPDK_LIBS ${DPDK_LIBS} mlx5)

I really needed the LUA jit stuff as I have to attach current time to to the crafted data part of the UDP package.

Again thanks for a really nice software package. Assar W.

thanks!!!!

@Ramla-I
Copy link

Ramla-I commented Jan 28, 2022

Hi, have you found a solution for this? I'm also able to to build MoonGen with the mlx5 flag, but when I try to run it I get the 0 usable devices error.

@tom-wegener
Copy link

I hope Ramla-I solved his problem but for people looking into this later:
because of the special driver model of the mlx-devices the devices do not have to bind by the script and the script reports 0 devices anyway. I could just my script after following the steps from Assar-Westman .

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

6 participants