I am trying to use a Broadcom BCM57508 200G NIC with DPDK. However, I am facing an ethdev initialisation failed error when I try to use Pktgen or even testpmd. I have tried updating the NIC firmware but it didn't help. I can not turn on IOMMU for my setup. The host machine is AMD EPYC 7763. DPDK version 24.11. I can also see Cannot collect crash dump as TEE_BNXT_FW config option is not enabled. in the dmesg for 01:00.0 but not for 01:00.1.
EAL: Detected CPU lcores: 128
EAL: Detected NUMA nodes: 2
EAL: Auto-detected process type: PRIMARY
EAL: Detected shared linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/pg/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: VFIO support initialized
EAL: Using IOMMU type 8 (No-IOMMU)
ETHDEV: ethdev initialisation failed
PCI_BUS: Requested device 0000:01:00.0 cannot be used
EAL: Using IOMMU type 8 (No-IOMMU)
ETHDEV: ethdev initialisation failed
PCI_BUS: Requested device 0000:01:00.1 cannot be used
EAL: Bus (pci) probe failed.
EDIT:
dpdk-devbind.py -s gives the following output
Network devices using DPDK-compatible driver
============================================
0000:01:00.0 'BCM57508 NetXtreme-E 10Gb/25Gb/40Gb/50Gb/100Gb/200Gb Ethernet 1750' numa_node=0 drv=vfio-pci unused=bnxt_en
0000:01:00.1 'BCM57508 NetXtreme-E 10Gb/25Gb/40Gb/50Gb/100Gb/200Gb Ethernet 1750' numa_node=0 drv=vfio-pci unused=bnxt_en
Network devices using kernel driver
===================================
0000:81:00.0 'Ethernet Controller E810-C for QSFP 1592' numa_node=1 if=ens6f0np0 drv=ice unused=vfio-pci *Active*
0000:81:00.1 'Ethernet Controller E810-C for QSFP 1592' numa_node=1 if=ens6f1np1 drv=ice unused=vfio-pci
0000:a1:00.0 'Ethernet Controller E810-C for QSFP 1592' numa_node=1 if=ens5f0np0 drv=ice unused=vfio-pci
0000:a1:00.1 'Ethernet Controller E810-C for QSFP 1592' numa_node=1 if=ens5f1np1 drv=ice unused=vfio-pci
The PCIe ID in the lspci output is as follows
01:00.0 Ethernet controller [0200]: Broadcom Inc. and subsidiaries BCM57508 NetXtreme-E 10Gb/25Gb/40Gb/50Gb/100Gb/200Gb Ethernet [14e4:1750] (rev 11)
Subsystem: Broadcom Inc. and subsidiaries NetXtreme-E Dual-port 100G QSFP56 Ethernet PCIe4.0 x16 Adapter (BCM957508-P2100G) [14e4:2100]
Kernel driver in use: vfio-pci
Kernel modules: bnxt_en
Verbose printing of EAL output gave the following which might help in debugging:
(Skipping log level changed from info to debug and Detected lcore 0 as core 0 on socket 0 messages)
EAL: Maximum logical cores by configuration: 128
EAL: Detected CPU lcores: 128
EAL: Detected NUMA nodes: 2
EAL: Checking presence of .so 'librte_eal.so.25.0'
EAL: Checking presence of .so 'librte_eal.so.25'
EAL: Checking presence of .so 'librte_eal.so'
EAL: Detected static linkage of DPDK
DPAA_BUS: rte_dpaa_bus_parse(): Parse device name (01:00.1)
DPAA2_BUS: rte_fslmc_parse(): Invalid device for matching (01:00.1).
DPAA_BUS: rte_dpaa_bus_parse(): Parse device name (01:00.0)
DPAA2_BUS: rte_fslmc_parse(): Invalid device for matching (01:00.0).
EAL: Ask a virtual area of 0x7000 bytes
EAL: Virtual area found at 0x100000000 (size = 0x7000)
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
CDX_BUS: cdx_scan(): opendir failed: No such file or directory
DPAA_BUS: rte_dpaa_bus_scan(): >>
DPAA_BUS: DPAA Bus not present. Skipping.
DPAA2_BUS: rte_fslmc_scan(): DPAA2: DPRC not available
DPAA2_BUS: rte_fslmc_scan(): FSLMC Bus Not Available. Skipping (-22)
UACCE_BUS: open /sys/class/uacce failed!
EAL: Bus auxiliary wants IOVA as 'DC'
EAL: Bus cdx wants IOVA as 'DC'
EAL: Bus dpaa_bus wants IOVA as 'DC'
EAL: Bus fslmc wants IOVA as 'DC'
PCI_BUS: PCI driver net_bnxt for device 0000:01:00.0 wants IOVA as 'PA'
PCI_BUS: PCI driver net_bnxt for device 0000:01:00.1 wants IOVA as 'PA'
EAL: Bus pci wants IOVA as 'PA'
EAL: Bus platform wants IOVA as 'DC'
EAL: Bus vdev wants IOVA as 'DC'
EAL: Bus dsa wants IOVA as 'DC'
EAL: Selected IOVA mode 'PA'
EAL: Probing VFIO support...
EAL: IOMMU type 1 (Type 1) is supported
EAL: IOMMU type 7 (sPAPR) is not supported
EAL: IOMMU type 8 (No-IOMMU) is supported
EAL: VFIO support initialized
EAL: Ask a virtual area of 0x2e000 bytes
EAL: Virtual area found at 0x100007000 (size = 0x2e000)
EAL: Setting up physically contiguous memory...
EAL: Setting maximum number of open files to 500000
EAL: Detected memory type: socket_id:0 hugepage_sz:2097152
EAL: Detected memory type: socket_id:1 hugepage_sz:2097152
EAL: Creating 4 segment lists: n_segs:8192 socket_id:0 hugepage_sz:2097152
EAL: Ask a virtual area of 0x61000 bytes
EAL: Virtual area found at 0x100035000 (size = 0x61000)
EAL: Memseg list allocated at socket 0, page size 0x800kB
EAL: Ask a virtual area of 0x400000000 bytes
EAL: Virtual area found at 0x100200000 (size = 0x400000000)
EAL: VA reserved for memseg list at 0x100200000, size 400000000
EAL: Ask a virtual area of 0x61000 bytes
EAL: Virtual area found at 0x500200000 (size = 0x61000)
EAL: Memseg list allocated at socket 0, page size 0x800kB
EAL: Ask a virtual area of 0x400000000 bytes
EAL: Virtual area found at 0x500400000 (size = 0x400000000)
EAL: VA reserved for memseg list at 0x500400000, size 400000000
EAL: Ask a virtual area of 0x61000 bytes
EAL: Virtual area found at 0x900400000 (size = 0x61000)
EAL: Memseg list allocated at socket 0, page size 0x800kB
EAL: Ask a virtual area of 0x400000000 bytes
EAL: Virtual area found at 0x900600000 (size = 0x400000000)
EAL: VA reserved for memseg list at 0x900600000, size 400000000
EAL: Ask a virtual area of 0x61000 bytes
EAL: Virtual area found at 0xd00600000 (size = 0x61000)
EAL: Memseg list allocated at socket 0, page size 0x800kB
EAL: Ask a virtual area of 0x400000000 bytes
EAL: Virtual area found at 0xd00800000 (size = 0x400000000)
EAL: VA reserved for memseg list at 0xd00800000, size 400000000
EAL: Creating 4 segment lists: n_segs:8192 socket_id:1 hugepage_sz:2097152
EAL: Ask a virtual area of 0x61000 bytes
EAL: Virtual area found at 0x1100800000 (size = 0x61000)
EAL: Memseg list allocated at socket 1, page size 0x800kB
EAL: Ask a virtual area of 0x400000000 bytes
EAL: Virtual area found at 0x1100a00000 (size = 0x400000000)
EAL: VA reserved for memseg list at 0x1100a00000, size 400000000
EAL: Ask a virtual area of 0x61000 bytes
EAL: Virtual area found at 0x1500a00000 (size = 0x61000)
EAL: Memseg list allocated at socket 1, page size 0x800kB
EAL: Ask a virtual area of 0x400000000 bytes
EAL: Virtual area found at 0x1500c00000 (size = 0x400000000)
EAL: VA reserved for memseg list at 0x1500c00000, size 400000000
EAL: Ask a virtual area of 0x61000 bytes
EAL: Virtual area found at 0x1900c00000 (size = 0x61000)
EAL: Memseg list allocated at socket 1, page size 0x800kB
EAL: Ask a virtual area of 0x400000000 bytes
EAL: Virtual area found at 0x1900e00000 (size = 0x400000000)
EAL: VA reserved for memseg list at 0x1900e00000, size 400000000
EAL: Ask a virtual area of 0x61000 bytes
EAL: Virtual area found at 0x1d00e00000 (size = 0x61000)
EAL: Memseg list allocated at socket 1, page size 0x800kB
EAL: Ask a virtual area of 0x400000000 bytes
EAL: Virtual area found at 0x1d01000000 (size = 0x400000000)
EAL: VA reserved for memseg list at 0x1d01000000, size 400000000
EAL: TSC frequency is ~2445400 KHz
EAL: Main lcore 0 is ready (tid=70e6f9858c00;cpuset=[0])
EAL: lcore 1 is ready (tid=70e6f6800400;cpuset=[1])
EAL: lcore 2 is ready (tid=70e6f5e00400;cpuset=[2])
EAL: lcore 3 is ready (tid=70e6f5400400;cpuset=[3])
EAL: lcore 4 is ready (tid=70e6f4a00400;cpuset=[4])
EAL: lcore 5 is ready (tid=70e6f4000400;cpuset=[5])
EAL: lcore 6 is ready (tid=70e6f3600400;cpuset=[6])
EAL: lcore 7 is ready (tid=70e6f2c00400;cpuset=[7])
EAL: lcore 8 is ready (tid=70e6f2200400;cpuset=[8])
EAL: lcore 9 is ready (tid=70e6f1800400;cpuset=[9])
EAL: lcore 11 is ready (tid=70e6f0400400;cpuset=[11])
EAL: lcore 10 is ready (tid=70e6f0e00400;cpuset=[10])
EAL: lcore 12 is ready (tid=70e6efa00400;cpuset=[12])
EAL: lcore 13 is ready (tid=70e6ef000400;cpuset=[13])
EAL: lcore 14 is ready (tid=70e6ee600400;cpuset=[14])
EAL: lcore 15 is ready (tid=70e6edc00400;cpuset=[15])
EAL: lcore 16 is ready (tid=70e6ed200400;cpuset=[16])
EAL: lcore 19 is ready (tid=70e6eb400400;cpuset=[19])
EAL: lcore 17 is ready (tid=70e6ec800400;cpuset=[17])
EAL: lcore 20 is ready (tid=70e6eaa00400;cpuset=[20])
EAL: lcore 18 is ready (tid=70e6ebe00400;cpuset=[18])
EAL: lcore 21 is ready (tid=70e6ea000400;cpuset=[21])
EAL: lcore 22 is ready (tid=70e6e9600400;cpuset=[22])
EAL: lcore 23 is ready (tid=70e6e8c00400;cpuset=[23])
EAL: lcore 25 is ready (tid=70e6e7800400;cpuset=[25])
EAL: lcore 27 is ready (tid=70e6e6400400;cpuset=[27])
EAL: lcore 26 is ready (tid=70e6e6e00400;cpuset=[26])
EAL: lcore 24 is ready (tid=70e6e8200400;cpuset=[24])
EAL: lcore 28 is ready (tid=70e6e5a00400;cpuset=[28])
EAL: lcore 29 is ready (tid=70e6e5000400;cpuset=[29])
EAL: lcore 30 is ready (tid=70e6e4600400;cpuset=[30])
EAL: lcore 31 is ready (tid=70e6e3c00400;cpuset=[31])
EAL: lcore 32 is ready (tid=70e6e3200400;cpuset=[32])
EAL: lcore 33 is ready (tid=70e6e2800400;cpuset=[33])
EAL: lcore 34 is ready (tid=70e6e1e00400;cpuset=[34])
EAL: lcore 35 is ready (tid=70e6e1400400;cpuset=[35])
EAL: lcore 36 is ready (tid=70e6e0a00400;cpuset=[36])
EAL: lcore 37 is ready (tid=70e6e0000400;cpuset=[37])
EAL: lcore 38 is ready (tid=70e6df600400;cpuset=[38])
EAL: lcore 39 is ready (tid=70e6dec00400;cpuset=[39])
EAL: lcore 40 is ready (tid=70e6de200400;cpuset=[40])
EAL: lcore 41 is ready (tid=70e6dd800400;cpuset=[41])
EAL: lcore 42 is ready (tid=70e6dce00400;cpuset=[42])
EAL: lcore 45 is ready (tid=70e6db000400;cpuset=[45])
EAL: lcore 51 is ready (tid=70e6d7400400;cpuset=[51])
EAL: lcore 44 is ready (tid=70e6dba00400;cpuset=[44])
EAL: lcore 48 is ready (tid=70e6d9200400;cpuset=[48])
EAL: lcore 49 is ready (tid=70e6d8800400;cpuset=[49])
EAL: lcore 46 is ready (tid=70e6da600400;cpuset=[46])
EAL: lcore 43 is ready (tid=70e6dc400400;cpuset=[43])
EAL: lcore 47 is ready (tid=70e6d9c00400;cpuset=[47])
EAL: lcore 52 is ready (tid=70e6d6a00400;cpuset=[52])
EAL: lcore 50 is ready (tid=70e6d7e00400;cpuset=[50])
EAL: lcore 53 is ready (tid=70e6d6000400;cpuset=[53])
EAL: lcore 55 is ready (tid=70e6d4c00400;cpuset=[55])
EAL: lcore 54 is ready (tid=70e6d5600400;cpuset=[54])
EAL: lcore 58 is ready (tid=70e6d2e00400;cpuset=[58])
EAL: lcore 57 is ready (tid=70e6d3800400;cpuset=[57])
EAL: lcore 59 is ready (tid=70e6d2400400;cpuset=[59])
EAL: lcore 56 is ready (tid=70e6d4200400;cpuset=[56])
EAL: lcore 60 is ready (tid=70e6d1a00400;cpuset=[60])
EAL: lcore 61 is ready (tid=70e6d1000400;cpuset=[61])
EAL: lcore 62 is ready (tid=70e6d0600400;cpuset=[62])
EAL: lcore 63 is ready (tid=70e6cfc00400;cpuset=[63])
EAL: lcore 64 is ready (tid=70e6cf200400;cpuset=[64])
EAL: Trying to obtain current memory policy.
EAL: Setting policy MPOL_PREFERRED for socket 0
EAL: Restoring previous memory policy: 0
EAL: request: mp_malloc_sync
EAL: Heap on socket 0 was expanded by 2MB
EAL: Allocated 2112 bytes of per-lcore data with a 64-byte alignment
PCI_BUS: PCI device 0000:01:00.0 on NUMA socket 0
PCI_BUS: probe driver: 14e4:1750 net_bnxt
EAL: Set IOMMU type 1 (Type 1) failed, error 19 (No such device)
EAL: Set IOMMU type 7 (sPAPR) failed, error 19 (No such device)
EAL: Using IOMMU type 8 (No-IOMMU)
EAL: Mem event callback 'vfio_mem_event_clb:(nil)' registered
EAL: Installed memory event callback for VFIO
PCI_BUS: VFIO reports MSI-X BAR as mappable
PCI_BUS: PCI memory mapped at 0x2101000000
PCI_BUS: PCI memory mapped at 0x2101010000
PCI_BUS: PCI memory mapped at 0x2102010000
PCI_BUS: Probe PCI driver: net_bnxt (14e4:1750) device: 0000:01:00.0 (socket 0)
ETHDEV: Total devargs parsed 0
BNXT: bnxt_pci_probe(): nb_representor_ports = 0
BNXT: bnxt_dev_init(): Broadcom NetXtreme driver bnxt
BNXT: bnxt_dev_uninit(): Calling Device uninit
ETHDEV: ethdev initialisation failed
EAL: Mem event callback 'vfio_mem_event_clb:(nil)' unregistered
PCI_BUS: Releasing PCI mapped resource for 0000:01:00.0
PCI_BUS: Calling pci_unmap_resource for 0000:01:00.0 at 0x2101000000
PCI_BUS: PCI memory unmapped at 0x2101000000
PCI_BUS: Calling pci_unmap_resource for 0000:01:00.0 at 0x2101010000
PCI_BUS: PCI memory unmapped at 0x2101010000
PCI_BUS: Calling pci_unmap_resource for 0000:01:00.0 at 0x2102010000
PCI_BUS: PCI memory unmapped at 0x2102010000
PCI_BUS: Requested device 0000:01:00.0 cannot be used
PCI_BUS: PCI device 0000:01:00.1 on NUMA socket 0
PCI_BUS: probe driver: 14e4:1750 net_bnxt
EAL: Set IOMMU type 1 (Type 1) failed, error 19 (No such device)
EAL: Set IOMMU type 7 (sPAPR) failed, error 19 (No such device)
EAL: Using IOMMU type 8 (No-IOMMU)
EAL: Mem event callback 'vfio_mem_event_clb:(nil)' registered
EAL: Installed memory event callback for VFIO
PCI_BUS: VFIO reports MSI-X BAR as mappable
PCI_BUS: PCI memory mapped at 0x2102018000
PCI_BUS: PCI memory mapped at 0x2102028000
PCI_BUS: PCI memory mapped at 0x2103028000
PCI_BUS: Probe PCI driver: net_bnxt (14e4:1750) device: 0000:01:00.1 (socket 0)
ETHDEV: Total devargs parsed 0
BNXT: bnxt_pci_probe(): nb_representor_ports = 0
BNXT: bnxt_dev_uninit(): Calling Device uninit
ETHDEV: ethdev initialisation failed
EAL: Mem event callback 'vfio_mem_event_clb:(nil)' unregistered
PCI_BUS: Releasing PCI mapped resource for 0000:01:00.1
PCI_BUS: Calling pci_unmap_resource for 0000:01:00.1 at 0x2102018000
PCI_BUS: PCI memory unmapped at 0x2102018000
PCI_BUS: Calling pci_unmap_resource for 0000:01:00.1 at 0x2102028000
PCI_BUS: PCI memory unmapped at 0x2102028000
PCI_BUS: Calling pci_unmap_resource for 0000:01:00.1 at 0x2103028000
PCI_BUS: PCI memory unmapped at 0x2103028000
PCI_BUS: Requested device 0000:01:00.1 cannot be used
EAL: Bus (pci) probe failed.
TELEMETRY: Attempting socket bind to path '/var/run/dpdk/rte/dpdk_telemetry.v2'
TELEMETRY: Socket creation and binding ok
TELEMETRY: Telemetry initialized ok
TELEMETRY: No legacy callbacks, legacy socket not created
EAL: Trying to obtain current memory policy.
EAL: Setting policy MPOL_PREFERRED for socket 0
EAL: Restoring previous memory policy: 0
EAL: request: mp_malloc_sync
EAL: Heap on socket 0 was expanded by 6MB
EAL: Trying to obtain current memory policy.
EAL: Setting policy MPOL_PREFERRED for socket 0
EAL: Restoring previous memory policy: 0
EAL: request: mp_malloc_sync
EAL: Heap on socket 0 was expanded by 6MB
testpmd: No probed ethernet devices
testpmd: create a new mbuf pool <mb_pool_0>: n=659456, size=2176, socket=0
testpmd: preferred mempool ops selected: ring_mp_mc
EAL: Trying to obtain current memory policy.
EAL: Setting policy MPOL_PREFERRED for socket 0
EAL: Restoring previous memory policy: 0
EAL: request: mp_malloc_sync
EAL: Heap on socket 0 was expanded by 10MB
EAL: Trying to obtain current memory policy.
EAL: Setting policy MPOL_PREFERRED for socket 0
EAL: alloc_seg(): allocation happened on wrong socket (wanted 0, got 1)
EAL: Ask a virtual area of 0x200000 bytes
EAL: Virtual area found at 0x140200000 (size = 0x200000)
EAL: attempted to allocate 746 segments, but only 500 were allocated
EAL: Restoring previous memory policy: 0
EAL: Trying to obtain current memory policy.
EAL: Setting policy MPOL_PREFERRED for socket 0
EAL: Restoring previous memory policy: 0
EAL: request: mp_malloc_sync
EAL: Heap on socket 0 was expanded by 746MB
EAL: Trying to obtain current memory policy.
EAL: Setting policy MPOL_PREFERRED for socket 0
EAL: alloc_seg(): allocation happened on wrong socket (wanted 0, got 1)
EAL: Ask a virtual area of 0x200000 bytes
EAL: Virtual area found at 0x140200000 (size = 0x200000)
EAL: attempted to allocate 373 segments, but only 127 were allocated
EAL: Restoring previous memory policy: 0
EAL: Trying to obtain current memory policy.
EAL: Setting policy MPOL_PREFERRED for socket 0
EAL: alloc_seg(): allocation happened on wrong socket (wanted 0, got 1)
EAL: Ask a virtual area of 0x200000 bytes
EAL: Virtual area found at 0x140200000 (size = 0x200000)
EAL: attempted to allocate 187 segments, but only 127 were allocated
EAL: Restoring previous memory policy: 0
EAL: Trying to obtain current memory policy.
EAL: Setting policy MPOL_PREFERRED for socket 0
EAL: Restoring previous memory policy: 0
EAL: request: mp_malloc_sync
EAL: Heap on socket 0 was expanded by 188MB
EAL: Trying to obtain current memory policy.
EAL: Setting policy MPOL_PREFERRED for socket 0
EAL: alloc_seg(): allocation happened on wrong socket (wanted 0, got 1)
EAL: Ask a virtual area of 0x200000 bytes
EAL: Virtual area found at 0x140200000 (size = 0x200000)
EAL: attempted to allocate 94 segments, but only 33 were allocated
EAL: Restoring previous memory policy: 0
EAL: Trying to obtain current memory policy.
EAL: Setting policy MPOL_PREFERRED for socket 0
EAL: alloc_seg(): allocation happened on wrong socket (wanted 0, got 1)
EAL: Ask a virtual area of 0x200000 bytes
EAL: Virtual area found at 0x140200000 (size = 0x200000)
EAL: attempted to allocate 47 segments, but only 33 were allocated
EAL: Restoring previous memory policy: 0
EAL: Trying to obtain current memory policy.
EAL: Setting policy MPOL_PREFERRED for socket 0
EAL: Restoring previous memory policy: 0
EAL: request: mp_malloc_sync
EAL: Heap on socket 0 was expanded by 48MB
EAL: Trying to obtain current memory policy.
EAL: Setting policy MPOL_PREFERRED for socket 0
EAL: alloc_seg(): allocation happened on wrong socket (wanted 0, got 1)
EAL: Ask a virtual area of 0x200000 bytes
EAL: Virtual area found at 0x140200000 (size = 0x200000)
EAL: attempted to allocate 24 segments, but only 9 were allocated
EAL: Restoring previous memory policy: 0
EAL: Trying to obtain current memory policy.
EAL: Setting policy MPOL_PREFERRED for socket 0
EAL: alloc_seg(): allocation happened on wrong socket (wanted 0, got 1)
EAL: Ask a virtual area of 0x200000 bytes
EAL: Virtual area found at 0x140200000 (size = 0x200000)
EAL: attempted to allocate 12 segments, but only 9 were allocated
EAL: Restoring previous memory policy: 0
EAL: Trying to obtain current memory policy.
EAL: Setting policy MPOL_PREFERRED for socket 0
EAL: Restoring previous memory policy: 0
EAL: request: mp_malloc_sync
EAL: Heap on socket 0 was expanded by 12MB
EAL: Trying to obtain current memory policy.
EAL: Setting policy MPOL_PREFERRED for socket 0
EAL: alloc_seg(): allocation happened on wrong socket (wanted 0, got 1)
EAL: Ask a virtual area of 0x200000 bytes
EAL: Virtual area found at 0x140200000 (size = 0x200000)
EAL: attempted to allocate 6 segments, but only 3 were allocated
EAL: Restoring previous memory policy: 0
EAL: Trying to obtain current memory policy.
EAL: Setting policy MPOL_PREFERRED for socket 0
EAL: Restoring previous memory policy: 0
EAL: request: mp_malloc_sync
EAL: Heap on socket 0 was expanded by 6MB
EAL: Trying to obtain current memory policy.
EAL: Setting policy MPOL_PREFERRED for socket 0
EAL: alloc_seg(): allocation happened on wrong socket (wanted 0, got 1)
EAL: Ask a virtual area of 0x200000 bytes
EAL: Virtual area found at 0x140200000 (size = 0x200000)
EAL: attempted to allocate 3 segments, but only 0 were allocated
EAL: Restoring previous memory policy: 0
EAL: Trying to obtain current memory policy.
EAL: Setting policy MPOL_PREFERRED for socket 0
EAL: alloc_seg(): allocation happened on wrong socket (wanted 0, got 1)
EAL: Ask a virtual area of 0x200000 bytes
EAL: Virtual area found at 0x140200000 (size = 0x200000)
EAL: attempted to allocate 2 segments, but only 0 were allocated
EAL: Restoring previous memory policy: 0
EAL: Trying to obtain current memory policy.
EAL: Setting policy MPOL_PREFERRED for socket 0
EAL: alloc_seg(): allocation happened on wrong socket (wanted 0, got 1)
EAL: Ask a virtual area of 0x200000 bytes
EAL: Virtual area found at 0x140200000 (size = 0x200000)
EAL: attempted to allocate 1 segments, but only 0 were allocated
EAL: Restoring previous memory policy: 0
EAL: Trying to obtain current memory policy.
EAL: Setting policy MPOL_PREFERRED for socket 0
EAL: alloc_seg(): allocation happened on wrong socket (wanted 0, got 1)
EAL: Ask a virtual area of 0x200000 bytes
EAL: Virtual area found at 0x140200000 (size = 0x200000)
EAL: attempted to allocate 1 segments, but only 0 were allocated
EAL: Restoring previous memory policy: 0
EAL: Trying to obtain current memory policy.
EAL: Setting policy MPOL_PREFERRED for socket 0
EAL: alloc_seg(): allocation happened on wrong socket (wanted 0, got 1)
EAL: Ask a virtual area of 0x200000 bytes
EAL: Virtual area found at 0x140200000 (size = 0x200000)
EAL: attempted to allocate 1 segments, but only 0 were allocated
EAL: Restoring previous memory policy: 0
EAL: Trying to obtain current memory policy.
EAL: Setting policy MPOL_PREFERRED for socket 0
EAL: alloc_seg(): allocation happened on wrong socket (wanted 0, got 1)
EAL: Ask a virtual area of 0x200000 bytes
EAL: Virtual area found at 0x140200000 (size = 0x200000)
EAL: attempted to allocate 1 segments, but only 0 were allocated
EAL: Restoring previous memory policy: 0
EAL: Trying to obtain current memory policy.
EAL: Setting policy MPOL_PREFERRED for socket 0
EAL: alloc_seg(): allocation happened on wrong socket (wanted 0, got 1)
EAL: Ask a virtual area of 0x200000 bytes
EAL: Virtual area found at 0x140200000 (size = 0x200000)
EAL: attempted to allocate 1 segments, but only 0 were allocated
EAL: Restoring previous memory policy: 0
EAL: Trying to obtain current memory policy.
EAL: Setting policy MPOL_PREFERRED for socket 0
EAL: alloc_seg(): allocation happened on wrong socket (wanted 0, got 1)
EAL: Ask a virtual area of 0x200000 bytes
EAL: Virtual area found at 0x140200000 (size = 0x200000)
EAL: attempted to allocate 1 segments, but only 0 were allocated
EAL: Restoring previous memory policy: 0
EAL: Trying to obtain current memory policy.
EAL: Setting policy MPOL_PREFERRED for socket 0
EAL: alloc_seg(): allocation happened on wrong socket (wanted 0, got 1)
EAL: Ask a virtual area of 0x200000 bytes
EAL: Virtual area found at 0x140200000 (size = 0x200000)
EAL: attempted to allocate 1 segments, but only 0 were allocated
EAL: Restoring previous memory policy: 0
EAL: Trying to obtain current memory policy.
EAL: Setting policy MPOL_PREFERRED for socket 0
EAL: alloc_seg(): allocation happened on wrong socket (wanted 0, got 1)
EAL: Ask a virtual area of 0x200000 bytes
EAL: Virtual area found at 0x140200000 (size = 0x200000)
EAL: attempted to allocate 1 segments, but only 0 were allocated
EAL: Restoring previous memory policy: 0
EAL: Trying to obtain current memory policy.
EAL: Setting policy MPOL_PREFERRED for socket 0
EAL: alloc_seg(): allocation happened on wrong socket (wanted 0, got 1)
EAL: Ask a virtual area of 0x200000 bytes
EAL: Virtual area found at 0x140200000 (size = 0x200000)
EAL: attempted to allocate 1 segments, but only 0 were allocated
EAL: Restoring previous memory policy: 0
EAL: Trying to obtain current memory policy.
EAL: Setting policy MPOL_PREFERRED for socket 0
EAL: alloc_seg(): allocation happened on wrong socket (wanted 0, got 1)
EAL: Ask a virtual area of 0x200000 bytes
EAL: Virtual area found at 0x140200000 (size = 0x200000)
EAL: attempted to allocate 1 segments, but only 0 were allocated
EAL: Restoring previous memory policy: 0
EAL: request: mp_malloc_sync
EAL: Heap on socket 0 was shrunk by 744MB
EAL: request: mp_malloc_sync
EAL: Heap on socket 0 was shrunk by 186MB
EAL: request: mp_malloc_sync
EAL: Heap on socket 0 was shrunk by 46MB
EAL: request: mp_malloc_sync
EAL: Heap on socket 0 was shrunk by 10MB
EAL: request: mp_malloc_sync
EAL: Heap on socket 0 was shrunk by 4MB
EAL: request: mp_malloc_sync
EAL: Heap on socket 0 was shrunk by 2MB
EAL: request: mp_malloc_sync
EAL: Heap on socket 0 was shrunk by 2MB
EAL: request: mp_malloc_sync
EAL: Heap on socket 0 was shrunk by 2MB
EAL: request: mp_malloc_sync
EAL: Heap on socket 0 was shrunk by 2MB
EAL: request: mp_malloc_sync
EAL: Heap on socket 0 was shrunk by 2MB
EAL: request: mp_malloc_sync
EAL: Heap on socket 0 was shrunk by 10MB
EAL: Error - exiting with code: 1
Creation of mbuf pool for socket 0 failed: Cannot allocate memory
DPAA2_BUS: fslmc_vfio_close_group: Get fd by name((null)) failed(-19)
DPAA2_BUS: Unable to close devices -19
Interactive-mode selected
I am trying to use a Broadcom BCM57508 200G NIC with DPDK. However, I am facing an ethdev initialisation failed error when I try to use Pktgen or even testpmd. I have tried updating the NIC firmware but it didn't help. I can not turn on IOMMU for my setup. The host machine is AMD EPYC 7763. DPDK version 24.11. I can also see Cannot collect crash dump as TEE_BNXT_FW config option is not enabled. in the dmesg for 01:00.0 but not for 01:00.1.
EAL: Detected CPU lcores: 128
EAL: Detected NUMA nodes: 2
EAL: Auto-detected process type: PRIMARY
EAL: Detected shared linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/pg/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: VFIO support initialized
EAL: Using IOMMU type 8 (No-IOMMU)
ETHDEV: ethdev initialisation failed
PCI_BUS: Requested device 0000:01:00.0 cannot be used
EAL: Using IOMMU type 8 (No-IOMMU)
ETHDEV: ethdev initialisation failed
PCI_BUS: Requested device 0000:01:00.1 cannot be used
EAL: Bus (pci) probe failed.
EDIT:
dpdk-devbind.py -s gives the following output
Network devices using DPDK-compatible driver
============================================
0000:01:00.0 'BCM57508 NetXtreme-E 10Gb/25Gb/40Gb/50Gb/100Gb/200Gb Ethernet 1750' numa_node=0 drv=vfio-pci unused=bnxt_en
0000:01:00.1 'BCM57508 NetXtreme-E 10Gb/25Gb/40Gb/50Gb/100Gb/200Gb Ethernet 1750' numa_node=0 drv=vfio-pci unused=bnxt_en
Network devices using kernel driver
===================================
0000:81:00.0 'Ethernet Controller E810-C for QSFP 1592' numa_node=1 if=ens6f0np0 drv=ice unused=vfio-pci *Active*
0000:81:00.1 'Ethernet Controller E810-C for QSFP 1592' numa_node=1 if=ens6f1np1 drv=ice unused=vfio-pci
0000:a1:00.0 'Ethernet Controller E810-C for QSFP 1592' numa_node=1 if=ens5f0np0 drv=ice unused=vfio-pci
0000:a1:00.1 'Ethernet Controller E810-C for QSFP 1592' numa_node=1 if=ens5f1np1 drv=ice unused=vfio-pci
The PCIe ID in the lspci output is as follows
01:00.0 Ethernet controller [0200]: Broadcom Inc. and subsidiaries BCM57508 NetXtreme-E 10Gb/25Gb/40Gb/50Gb/100Gb/200Gb Ethernet [14e4:1750] (rev 11)
Subsystem: Broadcom Inc. and subsidiaries NetXtreme-E Dual-port 100G QSFP56 Ethernet PCIe4.0 x16 Adapter (BCM957508-P2100G) [14e4:2100]
Kernel driver in use: vfio-pci
Kernel modules: bnxt_en
Verbose printing of EAL output gave the following which might help in debugging:
(Skipping log level changed from info to debug and Detected lcore 0 as core 0 on socket 0 messages)
EAL: Maximum logical cores by configuration: 128
EAL: Detected CPU lcores: 128
EAL: Detected NUMA nodes: 2
EAL: Checking presence of .so 'librte_eal.so.25.0'
EAL: Checking presence of .so 'librte_eal.so.25'
EAL: Checking presence of .so 'librte_eal.so'
EAL: Detected static linkage of DPDK
DPAA_BUS: rte_dpaa_bus_parse(): Parse device name (01:00.1)
DPAA2_BUS: rte_fslmc_parse(): Invalid device for matching (01:00.1).
DPAA_BUS: rte_dpaa_bus_parse(): Parse device name (01:00.0)
DPAA2_BUS: rte_fslmc_parse(): Invalid device for matching (01:00.0).
EAL: Ask a virtual area of 0x7000 bytes
EAL: Virtual area found at 0x100000000 (size = 0x7000)
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
CDX_BUS: cdx_scan(): opendir failed: No such file or directory
DPAA_BUS: rte_dpaa_bus_scan(): >>
DPAA_BUS: DPAA Bus not present. Skipping.
DPAA2_BUS: rte_fslmc_scan(): DPAA2: DPRC not available
DPAA2_BUS: rte_fslmc_scan(): FSLMC Bus Not Available. Skipping (-22)
UACCE_BUS: open /sys/class/uacce failed!
EAL: Bus auxiliary wants IOVA as 'DC'
EAL: Bus cdx wants IOVA as 'DC'
EAL: Bus dpaa_bus wants IOVA as 'DC'
EAL: Bus fslmc wants IOVA as 'DC'
PCI_BUS: PCI driver net_bnxt for device 0000:01:00.0 wants IOVA as 'PA'
PCI_BUS: PCI driver net_bnxt for device 0000:01:00.1 wants IOVA as 'PA'
EAL: Bus pci wants IOVA as 'PA'
EAL: Bus platform wants IOVA as 'DC'
EAL: Bus vdev wants IOVA as 'DC'
EAL: Bus dsa wants IOVA as 'DC'
EAL: Selected IOVA mode 'PA'
EAL: Probing VFIO support...
EAL: IOMMU type 1 (Type 1) is supported
EAL: IOMMU type 7 (sPAPR) is not supported
EAL: IOMMU type 8 (No-IOMMU) is supported
EAL: VFIO support initialized
EAL: Ask a virtual area of 0x2e000 bytes
EAL: Virtual area found at 0x100007000 (size = 0x2e000)
EAL: Setting up physically contiguous memory...
EAL: Setting maximum number of open files to 500000
EAL: Detected memory type: socket_id:0 hugepage_sz:2097152
EAL: Detected memory type: socket_id:1 hugepage_sz:2097152
EAL: Creating 4 segment lists: n_segs:8192 socket_id:0 hugepage_sz:2097152
EAL: Ask a virtual area of 0x61000 bytes
EAL: Virtual area found at 0x100035000 (size = 0x61000)
EAL: Memseg list allocated at socket 0, page size 0x800kB
EAL: Ask a virtual area of 0x400000000 bytes
EAL: Virtual area found at 0x100200000 (size = 0x400000000)
EAL: VA reserved for memseg list at 0x100200000, size 400000000
EAL: Ask a virtual area of 0x61000 bytes
EAL: Virtual area found at 0x500200000 (size = 0x61000)
EAL: Memseg list allocated at socket 0, page size 0x800kB
EAL: Ask a virtual area of 0x400000000 bytes
EAL: Virtual area found at 0x500400000 (size = 0x400000000)
EAL: VA reserved for memseg list at 0x500400000, size 400000000
EAL: Ask a virtual area of 0x61000 bytes
EAL: Virtual area found at 0x900400000 (size = 0x61000)
EAL: Memseg list allocated at socket 0, page size 0x800kB
EAL: Ask a virtual area of 0x400000000 bytes
EAL: Virtual area found at 0x900600000 (size = 0x400000000)
EAL: VA reserved for memseg list at 0x900600000, size 400000000
EAL: Ask a virtual area of 0x61000 bytes
EAL: Virtual area found at 0xd00600000 (size = 0x61000)
EAL: Memseg list allocated at socket 0, page size 0x800kB
EAL: Ask a virtual area of 0x400000000 bytes
EAL: Virtual area found at 0xd00800000 (size = 0x400000000)
EAL: VA reserved for memseg list at 0xd00800000, size 400000000
EAL: Creating 4 segment lists: n_segs:8192 socket_id:1 hugepage_sz:2097152
EAL: Ask a virtual area of 0x61000 bytes
EAL: Virtual area found at 0x1100800000 (size = 0x61000)
EAL: Memseg list allocated at socket 1, page size 0x800kB
EAL: Ask a virtual area of 0x400000000 bytes
EAL: Virtual area found at 0x1100a00000 (size = 0x400000000)
EAL: VA reserved for memseg list at 0x1100a00000, size 400000000
EAL: Ask a virtual area of 0x61000 bytes
EAL: Virtual area found at 0x1500a00000 (size = 0x61000)
EAL: Memseg list allocated at socket 1, page size 0x800kB
EAL: Ask a virtual area of 0x400000000 bytes
EAL: Virtual area found at 0x1500c00000 (size = 0x400000000)
EAL: VA reserved for memseg list at 0x1500c00000, size 400000000
EAL: Ask a virtual area of 0x61000 bytes
EAL: Virtual area found at 0x1900c00000 (size = 0x61000)
EAL: Memseg list allocated at socket 1, page size 0x800kB
EAL: Ask a virtual area of 0x400000000 bytes
EAL: Virtual area found at 0x1900e00000 (size = 0x400000000)
EAL: VA reserved for memseg list at 0x1900e00000, size 400000000
EAL: Ask a virtual area of 0x61000 bytes
EAL: Virtual area found at 0x1d00e00000 (size = 0x61000)
EAL: Memseg list allocated at socket 1, page size 0x800kB
EAL: Ask a virtual area of 0x400000000 bytes
EAL: Virtual area found at 0x1d01000000 (size = 0x400000000)
EAL: VA reserved for memseg list at 0x1d01000000, size 400000000
EAL: TSC frequency is ~2445400 KHz
EAL: Main lcore 0 is ready (tid=70e6f9858c00;cpuset=[0])
EAL: lcore 1 is ready (tid=70e6f6800400;cpuset=[1])
EAL: lcore 2 is ready (tid=70e6f5e00400;cpuset=[2])
EAL: lcore 3 is ready (tid=70e6f5400400;cpuset=[3])
EAL: lcore 4 is ready (tid=70e6f4a00400;cpuset=[4])
EAL: lcore 5 is ready (tid=70e6f4000400;cpuset=[5])
EAL: lcore 6 is ready (tid=70e6f3600400;cpuset=[6])
EAL: lcore 7 is ready (tid=70e6f2c00400;cpuset=[7])
EAL: lcore 8 is ready (tid=70e6f2200400;cpuset=[8])
EAL: lcore 9 is ready (tid=70e6f1800400;cpuset=[9])
EAL: lcore 11 is ready (tid=70e6f0400400;cpuset=[11])
EAL: lcore 10 is ready (tid=70e6f0e00400;cpuset=[10])
EAL: lcore 12 is ready (tid=70e6efa00400;cpuset=[12])
EAL: lcore 13 is ready (tid=70e6ef000400;cpuset=[13])
EAL: lcore 14 is ready (tid=70e6ee600400;cpuset=[14])
EAL: lcore 15 is ready (tid=70e6edc00400;cpuset=[15])
EAL: lcore 16 is ready (tid=70e6ed200400;cpuset=[16])
EAL: lcore 19 is ready (tid=70e6eb400400;cpuset=[19])
EAL: lcore 17 is ready (tid=70e6ec800400;cpuset=[17])
EAL: lcore 20 is ready (tid=70e6eaa00400;cpuset=[20])
EAL: lcore 18 is ready (tid=70e6ebe00400;cpuset=[18])
EAL: lcore 21 is ready (tid=70e6ea000400;cpuset=[21])
EAL: lcore 22 is ready (tid=70e6e9600400;cpuset=[22])
EAL: lcore 23 is ready (tid=70e6e8c00400;cpuset=[23])
EAL: lcore 25 is ready (tid=70e6e7800400;cpuset=[25])
EAL: lcore 27 is ready (tid=70e6e6400400;cpuset=[27])
EAL: lcore 26 is ready (tid=70e6e6e00400;cpuset=[26])
EAL: lcore 24 is ready (tid=70e6e8200400;cpuset=[24])
EAL: lcore 28 is ready (tid=70e6e5a00400;cpuset=[28])
EAL: lcore 29 is ready (tid=70e6e5000400;cpuset=[29])
EAL: lcore 30 is ready (tid=70e6e4600400;cpuset=[30])
EAL: lcore 31 is ready (tid=70e6e3c00400;cpuset=[31])
EAL: lcore 32 is ready (tid=70e6e3200400;cpuset=[32])
EAL: lcore 33 is ready (tid=70e6e2800400;cpuset=[33])
EAL: lcore 34 is ready (tid=70e6e1e00400;cpuset=[34])
EAL: lcore 35 is ready (tid=70e6e1400400;cpuset=[35])
EAL: lcore 36 is ready (tid=70e6e0a00400;cpuset=[36])
EAL: lcore 37 is ready (tid=70e6e0000400;cpuset=[37])
EAL: lcore 38 is ready (tid=70e6df600400;cpuset=[38])
EAL: lcore 39 is ready (tid=70e6dec00400;cpuset=[39])
EAL: lcore 40 is ready (tid=70e6de200400;cpuset=[40])
EAL: lcore 41 is ready (tid=70e6dd800400;cpuset=[41])
EAL: lcore 42 is ready (tid=70e6dce00400;cpuset=[42])
EAL: lcore 45 is ready (tid=70e6db000400;cpuset=[45])
EAL: lcore 51 is ready (tid=70e6d7400400;cpuset=[51])
EAL: lcore 44 is ready (tid=70e6dba00400;cpuset=[44])
EAL: lcore 48 is ready (tid=70e6d9200400;cpuset=[48])
EAL: lcore 49 is ready (tid=70e6d8800400;cpuset=[49])
EAL: lcore 46 is ready (tid=70e6da600400;cpuset=[46])
EAL: lcore 43 is ready (tid=70e6dc400400;cpuset=[43])
EAL: lcore 47 is ready (tid=70e6d9c00400;cpuset=[47])
EAL: lcore 52 is ready (tid=70e6d6a00400;cpuset=[52])
EAL: lcore 50 is ready (tid=70e6d7e00400;cpuset=[50])
EAL: lcore 53 is ready (tid=70e6d6000400;cpuset=[53])
EAL: lcore 55 is ready (tid=70e6d4c00400;cpuset=[55])
EAL: lcore 54 is ready (tid=70e6d5600400;cpuset=[54])
EAL: lcore 58 is ready (tid=70e6d2e00400;cpuset=[58])
EAL: lcore 57 is ready (tid=70e6d3800400;cpuset=[57])
EAL: lcore 59 is ready (tid=70e6d2400400;cpuset=[59])
EAL: lcore 56 is ready (tid=70e6d4200400;cpuset=[56])
EAL: lcore 60 is ready (tid=70e6d1a00400;cpuset=[60])
EAL: lcore 61 is ready (tid=70e6d1000400;cpuset=[61])
EAL: lcore 62 is ready (tid=70e6d0600400;cpuset=[62])
EAL: lcore 63 is ready (tid=70e6cfc00400;cpuset=[63])
EAL: lcore 64 is ready (tid=70e6cf200400;cpuset=[64])
EAL: Trying to obtain current memory policy.
EAL: Setting policy MPOL_PREFERRED for socket 0
EAL: Restoring previous memory policy: 0
EAL: request: mp_malloc_sync
EAL: Heap on socket 0 was expanded by 2MB
EAL: Allocated 2112 bytes of per-lcore data with a 64-byte alignment
PCI_BUS: PCI device 0000:01:00.0 on NUMA socket 0
PCI_BUS: probe driver: 14e4:1750 net_bnxt
EAL: Set IOMMU type 1 (Type 1) failed, error 19 (No such device)
EAL: Set IOMMU type 7 (sPAPR) failed, error 19 (No such device)
EAL: Using IOMMU type 8 (No-IOMMU)
EAL: Mem event callback 'vfio_mem_event_clb:(nil)' registered
EAL: Installed memory event callback for VFIO
PCI_BUS: VFIO reports MSI-X BAR as mappable
PCI_BUS: PCI memory mapped at 0x2101000000
PCI_BUS: PCI memory mapped at 0x2101010000
PCI_BUS: PCI memory mapped at 0x2102010000
PCI_BUS: Probe PCI driver: net_bnxt (14e4:1750) device: 0000:01:00.0 (socket 0)
ETHDEV: Total devargs parsed 0
BNXT: bnxt_pci_probe(): nb_representor_ports = 0
BNXT: bnxt_dev_init(): Broadcom NetXtreme driver bnxt
BNXT: bnxt_dev_uninit(): Calling Device uninit
ETHDEV: ethdev initialisation failed
EAL: Mem event callback 'vfio_mem_event_clb:(nil)' unregistered
PCI_BUS: Releasing PCI mapped resource for 0000:01:00.0
PCI_BUS: Calling pci_unmap_resource for 0000:01:00.0 at 0x2101000000
PCI_BUS: PCI memory unmapped at 0x2101000000
PCI_BUS: Calling pci_unmap_resource for 0000:01:00.0 at 0x2101010000
PCI_BUS: PCI memory unmapped at 0x2101010000
PCI_BUS: Calling pci_unmap_resource for 0000:01:00.0 at 0x2102010000
PCI_BUS: PCI memory unmapped at 0x2102010000
PCI_BUS: Requested device 0000:01:00.0 cannot be used
PCI_BUS: PCI device 0000:01:00.1 on NUMA socket 0
PCI_BUS: probe driver: 14e4:1750 net_bnxt
EAL: Set IOMMU type 1 (Type 1) failed, error 19 (No such device)
EAL: Set IOMMU type 7 (sPAPR) failed, error 19 (No such device)
EAL: Using IOMMU type 8 (No-IOMMU)
EAL: Mem event callback 'vfio_mem_event_clb:(nil)' registered
EAL: Installed memory event callback for VFIO
PCI_BUS: VFIO reports MSI-X BAR as mappable
PCI_BUS: PCI memory mapped at 0x2102018000
PCI_BUS: PCI memory mapped at 0x2102028000
PCI_BUS: PCI memory mapped at 0x2103028000
PCI_BUS: Probe PCI driver: net_bnxt (14e4:1750) device: 0000:01:00.1 (socket 0)
ETHDEV: Total devargs parsed 0
BNXT: bnxt_pci_probe(): nb_representor_ports = 0
BNXT: bnxt_dev_uninit(): Calling Device uninit
ETHDEV: ethdev initialisation failed
EAL: Mem event callback 'vfio_mem_event_clb:(nil)' unregistered
PCI_BUS: Releasing PCI mapped resource for 0000:01:00.1
PCI_BUS: Calling pci_unmap_resource for 0000:01:00.1 at 0x2102018000
PCI_BUS: PCI memory unmapped at 0x2102018000
PCI_BUS: Calling pci_unmap_resource for 0000:01:00.1 at 0x2102028000
PCI_BUS: PCI memory unmapped at 0x2102028000
PCI_BUS: Calling pci_unmap_resource for 0000:01:00.1 at 0x2103028000
PCI_BUS: PCI memory unmapped at 0x2103028000
PCI_BUS: Requested device 0000:01:00.1 cannot be used
EAL: Bus (pci) probe failed.
TELEMETRY: Attempting socket bind to path '/var/run/dpdk/rte/dpdk_telemetry.v2'
TELEMETRY: Socket creation and binding ok
TELEMETRY: Telemetry initialized ok
TELEMETRY: No legacy callbacks, legacy socket not created
EAL: Trying to obtain current memory policy.
EAL: Setting policy MPOL_PREFERRED for socket 0
EAL: Restoring previous memory policy: 0
EAL: request: mp_malloc_sync
EAL: Heap on socket 0 was expanded by 6MB
EAL: Trying to obtain current memory policy.
EAL: Setting policy MPOL_PREFERRED for socket 0
EAL: Restoring previous memory policy: 0
EAL: request: mp_malloc_sync
EAL: Heap on socket 0 was expanded by 6MB
testpmd: No probed ethernet devices
testpmd: create a new mbuf pool <mb_pool_0>: n=659456, size=2176, socket=0
testpmd: preferred mempool ops selected: ring_mp_mc
EAL: Trying to obtain current memory policy.
EAL: Setting policy MPOL_PREFERRED for socket 0
EAL: Restoring previous memory policy: 0
EAL: request: mp_malloc_sync
EAL: Heap on socket 0 was expanded by 10MB
EAL: Trying to obtain current memory policy.
EAL: Setting policy MPOL_PREFERRED for socket 0
EAL: alloc_seg(): allocation happened on wrong socket (wanted 0, got 1)
EAL: Ask a virtual area of 0x200000 bytes
EAL: Virtual area found at 0x140200000 (size = 0x200000)
EAL: attempted to allocate 746 segments, but only 500 were allocated
EAL: Restoring previous memory policy: 0
EAL: Trying to obtain current memory policy.
EAL: Setting policy MPOL_PREFERRED for socket 0
EAL: Restoring previous memory policy: 0
EAL: request: mp_malloc_sync
EAL: Heap on socket 0 was expanded by 746MB
EAL: Trying to obtain current memory policy.
EAL: Setting policy MPOL_PREFERRED for socket 0
EAL: alloc_seg(): allocation happened on wrong socket (wanted 0, got 1)
EAL: Ask a virtual area of 0x200000 bytes
EAL: Virtual area found at 0x140200000 (size = 0x200000)
EAL: attempted to allocate 373 segments, but only 127 were allocated
EAL: Restoring previous memory policy: 0
EAL: Trying to obtain current memory policy.
EAL: Setting policy MPOL_PREFERRED for socket 0
EAL: alloc_seg(): allocation happened on wrong socket (wanted 0, got 1)
EAL: Ask a virtual area of 0x200000 bytes
EAL: Virtual area found at 0x140200000 (size = 0x200000)
EAL: attempted to allocate 187 segments, but only 127 were allocated
EAL: Restoring previous memory policy: 0
EAL: Trying to obtain current memory policy.
EAL: Setting policy MPOL_PREFERRED for socket 0
EAL: Restoring previous memory policy: 0
EAL: request: mp_malloc_sync
EAL: Heap on socket 0 was expanded by 188MB
EAL: Trying to obtain current memory policy.
EAL: Setting policy MPOL_PREFERRED for socket 0
EAL: alloc_seg(): allocation happened on wrong socket (wanted 0, got 1)
EAL: Ask a virtual area of 0x200000 bytes
EAL: Virtual area found at 0x140200000 (size = 0x200000)
EAL: attempted to allocate 94 segments, but only 33 were allocated
EAL: Restoring previous memory policy: 0
EAL: Trying to obtain current memory policy.
EAL: Setting policy MPOL_PREFERRED for socket 0
EAL: alloc_seg(): allocation happened on wrong socket (wanted 0, got 1)
EAL: Ask a virtual area of 0x200000 bytes
EAL: Virtual area found at 0x140200000 (size = 0x200000)
EAL: attempted to allocate 47 segments, but only 33 were allocated
EAL: Restoring previous memory policy: 0
EAL: Trying to obtain current memory policy.
EAL: Setting policy MPOL_PREFERRED for socket 0
EAL: Restoring previous memory policy: 0
EAL: request: mp_malloc_sync
EAL: Heap on socket 0 was expanded by 48MB
EAL: Trying to obtain current memory policy.
EAL: Setting policy MPOL_PREFERRED for socket 0
EAL: alloc_seg(): allocation happened on wrong socket (wanted 0, got 1)
EAL: Ask a virtual area of 0x200000 bytes
EAL: Virtual area found at 0x140200000 (size = 0x200000)
EAL: attempted to allocate 24 segments, but only 9 were allocated
EAL: Restoring previous memory policy: 0
EAL: Trying to obtain current memory policy.
EAL: Setting policy MPOL_PREFERRED for socket 0
EAL: alloc_seg(): allocation happened on wrong socket (wanted 0, got 1)
EAL: Ask a virtual area of 0x200000 bytes
EAL: Virtual area found at 0x140200000 (size = 0x200000)
EAL: attempted to allocate 12 segments, but only 9 were allocated
EAL: Restoring previous memory policy: 0
EAL: Trying to obtain current memory policy.
EAL: Setting policy MPOL_PREFERRED for socket 0
EAL: Restoring previous memory policy: 0
EAL: request: mp_malloc_sync
EAL: Heap on socket 0 was expanded by 12MB
EAL: Trying to obtain current memory policy.
EAL: Setting policy MPOL_PREFERRED for socket 0
EAL: alloc_seg(): allocation happened on wrong socket (wanted 0, got 1)
EAL: Ask a virtual area of 0x200000 bytes
EAL: Virtual area found at 0x140200000 (size = 0x200000)
EAL: attempted to allocate 6 segments, but only 3 were allocated
EAL: Restoring previous memory policy: 0
EAL: Trying to obtain current memory policy.
EAL: Setting policy MPOL_PREFERRED for socket 0
EAL: Restoring previous memory policy: 0
EAL: request: mp_malloc_sync
EAL: Heap on socket 0 was expanded by 6MB
EAL: Trying to obtain current memory policy.
EAL: Setting policy MPOL_PREFERRED for socket 0
EAL: alloc_seg(): allocation happened on wrong socket (wanted 0, got 1)
EAL: Ask a virtual area of 0x200000 bytes
EAL: Virtual area found at 0x140200000 (size = 0x200000)
EAL: attempted to allocate 3 segments, but only 0 were allocated
EAL: Restoring previous memory policy: 0
EAL: Trying to obtain current memory policy.
EAL: Setting policy MPOL_PREFERRED for socket 0
EAL: alloc_seg(): allocation happened on wrong socket (wanted 0, got 1)
EAL: Ask a virtual area of 0x200000 bytes
EAL: Virtual area found at 0x140200000 (size = 0x200000)
EAL: attempted to allocate 2 segments, but only 0 were allocated
EAL: Restoring previous memory policy: 0
EAL: Trying to obtain current memory policy.
EAL: Setting policy MPOL_PREFERRED for socket 0
EAL: alloc_seg(): allocation happened on wrong socket (wanted 0, got 1)
EAL: Ask a virtual area of 0x200000 bytes
EAL: Virtual area found at 0x140200000 (size = 0x200000)
EAL: attempted to allocate 1 segments, but only 0 were allocated
EAL: Restoring previous memory policy: 0
EAL: Trying to obtain current memory policy.
EAL: Setting policy MPOL_PREFERRED for socket 0
EAL: alloc_seg(): allocation happened on wrong socket (wanted 0, got 1)
EAL: Ask a virtual area of 0x200000 bytes
EAL: Virtual area found at 0x140200000 (size = 0x200000)
EAL: attempted to allocate 1 segments, but only 0 were allocated
EAL: Restoring previous memory policy: 0
EAL: Trying to obtain current memory policy.
EAL: Setting policy MPOL_PREFERRED for socket 0
EAL: alloc_seg(): allocation happened on wrong socket (wanted 0, got 1)
EAL: Ask a virtual area of 0x200000 bytes
EAL: Virtual area found at 0x140200000 (size = 0x200000)
EAL: attempted to allocate 1 segments, but only 0 were allocated
EAL: Restoring previous memory policy: 0
EAL: Trying to obtain current memory policy.
EAL: Setting policy MPOL_PREFERRED for socket 0
EAL: alloc_seg(): allocation happened on wrong socket (wanted 0, got 1)
EAL: Ask a virtual area of 0x200000 bytes
EAL: Virtual area found at 0x140200000 (size = 0x200000)
EAL: attempted to allocate 1 segments, but only 0 were allocated
EAL: Restoring previous memory policy: 0
EAL: Trying to obtain current memory policy.
EAL: Setting policy MPOL_PREFERRED for socket 0
EAL: alloc_seg(): allocation happened on wrong socket (wanted 0, got 1)
EAL: Ask a virtual area of 0x200000 bytes
EAL: Virtual area found at 0x140200000 (size = 0x200000)
EAL: attempted to allocate 1 segments, but only 0 were allocated
EAL: Restoring previous memory policy: 0
EAL: Trying to obtain current memory policy.
EAL: Setting policy MPOL_PREFERRED for socket 0
EAL: alloc_seg(): allocation happened on wrong socket (wanted 0, got 1)
EAL: Ask a virtual area of 0x200000 bytes
EAL: Virtual area found at 0x140200000 (size = 0x200000)
EAL: attempted to allocate 1 segments, but only 0 were allocated
EAL: Restoring previous memory policy: 0
EAL: Trying to obtain current memory policy.
EAL: Setting policy MPOL_PREFERRED for socket 0
EAL: alloc_seg(): allocation happened on wrong socket (wanted 0, got 1)
EAL: Ask a virtual area of 0x200000 bytes
EAL: Virtual area found at 0x140200000 (size = 0x200000)
EAL: attempted to allocate 1 segments, but only 0 were allocated
EAL: Restoring previous memory policy: 0
EAL: Trying to obtain current memory policy.
EAL: Setting policy MPOL_PREFERRED for socket 0
EAL: alloc_seg(): allocation happened on wrong socket (wanted 0, got 1)
EAL: Ask a virtual area of 0x200000 bytes
EAL: Virtual area found at 0x140200000 (size = 0x200000)
EAL: attempted to allocate 1 segments, but only 0 were allocated
EAL: Restoring previous memory policy: 0
EAL: Trying to obtain current memory policy.
EAL: Setting policy MPOL_PREFERRED for socket 0
EAL: alloc_seg(): allocation happened on wrong socket (wanted 0, got 1)
EAL: Ask a virtual area of 0x200000 bytes
EAL: Virtual area found at 0x140200000 (size = 0x200000)
EAL: attempted to allocate 1 segments, but only 0 were allocated
EAL: Restoring previous memory policy: 0
EAL: Trying to obtain current memory policy.
EAL: Setting policy MPOL_PREFERRED for socket 0
EAL: alloc_seg(): allocation happened on wrong socket (wanted 0, got 1)
EAL: Ask a virtual area of 0x200000 bytes
EAL: Virtual area found at 0x140200000 (size = 0x200000)
EAL: attempted to allocate 1 segments, but only 0 were allocated
EAL: Restoring previous memory policy: 0
EAL: request: mp_malloc_sync
EAL: Heap on socket 0 was shrunk by 744MB
EAL: request: mp_malloc_sync
EAL: Heap on socket 0 was shrunk by 186MB
EAL: request: mp_malloc_sync
EAL: Heap on socket 0 was shrunk by 46MB
EAL: request: mp_malloc_sync
EAL: Heap on socket 0 was shrunk by 10MB
EAL: request: mp_malloc_sync
EAL: Heap on socket 0 was shrunk by 4MB
EAL: request: mp_malloc_sync
EAL: Heap on socket 0 was shrunk by 2MB
EAL: request: mp_malloc_sync
EAL: Heap on socket 0 was shrunk by 2MB
EAL: request: mp_malloc_sync
EAL: Heap on socket 0 was shrunk by 2MB
EAL: request: mp_malloc_sync
EAL: Heap on socket 0 was shrunk by 2MB
EAL: request: mp_malloc_sync
EAL: Heap on socket 0 was shrunk by 2MB
EAL: request: mp_malloc_sync
EAL: Heap on socket 0 was shrunk by 10MB
EAL: Error - exiting with code: 1
Creation of mbuf pool for socket 0 failed: Cannot allocate memory
DPAA2_BUS: fslmc_vfio_close_group: Get fd by name((null)) failed(-19)
DPAA2_BUS: Unable to close devices -19
Interactive-mode selected
2 Answers
Reset to default 0- Consider to reload VFIO drivers in
no-IOMMU mode(first loadvfiowith the no-IOMMU parameter, thenvfio-pci) and re-bind the NIC. - The log suggests EAL probing multiple NICs. For the sake of clarity, it would be much nicer to first identify the target NIC by looking at
./usertools/dpdk-devbind.py --statusoutput (one should also make sure the NIC in question has been bound to the right kernel driver,vfio-pci, for instance) and instruct DPDK to only probe that particular device. For example, should the target NIC be listed as01:00.0in that output, pass argument-a 01:00.0via EAL section (before the--separator) when runningtest-pmd. - Perhaps enable verbose debug printouts by passing EAL argument
--log-level .*,8. That could help with further debugging. - Consider to post PCI IDs (vendor ID and device ID) of the NIC in question, as per
lspcioutput.
Sadly, the function that fails does not have debug printouts in place to clarify which precise step hasn't been successful. One may insert debug printouts of their own and recompile, to understand what happens.
Alternatively, consider to use uio_pci_generic instead of vfio-pci, as the former is also listed among supported drivers by the vendor.