I've read that, in order to use DPDK (in my case, with the AWS ENA poll-mode driver, developed by AWS as part of DPDK) for a kernel-bypass networking application, you need to either have an IOMMU hardware unit enabled, or use the vfio-pci kernel module in NO-IOMMU mode, by rebuilding the kernel module for that mode.

What I can't figure out is, do the AWS Enhanced Networking instances, such as c5.xlarge, have an IOMMU hardware and have it enabled by default, or do I need to somehow get it enabled manually, or do I simply have to run my DPDK kernel bypass system with vfio-pci module running in no-iommu mode?

I've already figured out how to enable booting with vfio-pci and how to enable the module at kernel runtime. I've also already figured out how to reserve hugepages instead of 4k pages. I've also figured out how to build the DPDK C examples with makefiles instead of meson/ninja. I'm almost there.

I just can't get the basic forwarding example in DPDK to work because my vfio-pci setup seems wrong, I don't know whether I have to enable IOMMU hardware in my AWS instance or not, and whether I have to run in no-iommu mode or not.

I've read that, in order to use DPDK (in my case, with the AWS ENA poll-mode driver, developed by AWS as part of DPDK) for a kernel-bypass networking application, you need to either have an IOMMU hardware unit enabled, or use the vfio-pci kernel module in NO-IOMMU mode, by rebuilding the kernel module for that mode.

What I can't figure out is, do the AWS Enhanced Networking instances, such as c5.xlarge, have an IOMMU hardware and have it enabled by default, or do I need to somehow get it enabled manually, or do I simply have to run my DPDK kernel bypass system with vfio-pci module running in no-iommu mode?

I've already figured out how to enable booting with vfio-pci and how to enable the module at kernel runtime. I've also already figured out how to reserve hugepages instead of 4k pages. I've also figured out how to build the DPDK C examples with makefiles instead of meson/ninja. I'm almost there.

I just can't get the basic forwarding example in DPDK to work because my vfio-pci setup seems wrong, I don't know whether I have to enable IOMMU hardware in my AWS instance or not, and whether I have to run in no-iommu mode or not.

• network-programming
• amazon-ec2
• dpdk
• vfio
• dpdk-pmd

4 Answers 4

In the running system, in order to check whether IOMMU is enabled or not, one can take a look at dmesg output as follows: dmesg -T | grep -i iommu (that may require root privileges). If that displays messages saying that IOMMU support is initialised and perhaps also lists IOMMU groups, then most likely the feature is enabled and should work just fine.

One may also check the current kernel cmdline by looking at parameters listed in cat /proc/cmdline output. Typically, this cmdline string can be tweaked by means of editing the bootloader configuration file, for example, such of the GRUB bootloader. DPDK documentation suggests that, in most cases, specifying iommu=on as kernel parameter should be enough to configure the Linux kernel to use IOMMU. Of course, reboot is required for the change to take effect.

However, when IOMMU support is unavailable, one still may use no-IOMMU mode with VFIO. Once module vfio is loaded (and before loading vfio-pci), it should be possible to enter the mode by issuing the following command (requires root privileges):

echo 1 > /sys/module/vfio/parameters/enable_unsafe_noiommu_mode

Subsequent loading of vfio-pci should result in no-IOMMU operation. One is advised to check with dmesg -T output after loading/unloading modules to learn the status from the messages that they produce.

Last, but not least, when running DPDK application examples, one may pass a EAL argument (before --) to enable verbose logging, which also applies to VFIO initialisation:

./build/dpdk-testpmd --log-level .*,8 -- -i

For more information, please consider to refer to the above mentioned documentation page.

One can use igb_uio instead of vfio-pci, yes, but that is not recommended. Consider to follow advice from the previous answer to get vfio-pci working.

As for the issue with the ninja build file missing (from the comment), one shall enter the build directory containing this file first (created by meson <build-directory-name> command) either as a separate step, via cd <build-directory-name> (and then invoke ninja), or as part of ninja invocation: ninja -C <build-directory-name>.

Regarding the vfio-pci advice: consider to post dmesg outputs that concern vfio, as explained in the original answer, and also consider posting verbose output from EAL initialisation. Without such information, it is very hard to guess about what might have gone wrong with the particular setup.

In what comes to build errors (which, by the way, make another question unrelated to the topic of the OP), it is hard to advise on third-party instructions that are not part of the upstream DPDK project: if those tutorials worked previously, consider reproducing them step-by-step, exactly as when they worked. Yes, vendor-specific tarballs may differ significantly from the upstream DPDK contents. So one shall make sure that all versions and sources from which the code comes, as well as command sequences used, follow those outlined in the tutorial.

To try the latest tarball with support from the vendor in question might be a nice idea, yes. With regard to the most recent compilation errors (a warning about some kind of function being implicitly defined, -- as stated in the comment), -- please consider either to edit the OP to add the build log so we could talk sense or to post a new question with these details: the tarball/repository and version used + commands ran in the terminal that are known to produce such output. Otherwise, it's quite hard to guess which function that might be that is implicitly defined.