I am working on a project where we have a kernel module that allocates memory using kmalloc(), and maps it to userspace using remap_pfn_range(). In remap_pfn_range() the vm_area_structs has its vm_flags field set to VM_IO | VM_PFNMAP (as well as some other flags that seem irrelevant to my problem). When I call sendmsg() using the address returned by our mmap() implementation (which is where the remap_pfn_range() call happens) the sendmsg() call fails with errno EFAULT.

I have tracked down the location in the kernel where -EFAULT is first returned, it happens in check_vma_flags() in mm/gup.c, the problem is that the above mentioned flags are set in the vm_area_struct:

static int check_vma_flags(struct vm_area_struct *vma, unsigned long gup_flags)
{
    vm_flags_t vm_flags = vma->vm_flags;
    int write = (gup_flags & FOLL_WRITE);
    int foreign = (gup_flags & FOLL_REMOTE);

    if (vm_flags & (VM_IO | VM_PFNMAP))
        return -EFAULT; // <= This error propagates all the way back to userspace.

For background, the allocated memory is shared between the CPU and an FPGA in a Cyclone V SOC-FPGA. The FPGA writes the bulk of the data to the buffers and requires the memory to be physically contiguous. We are currently using v5.3-rc8 of the linux kernel.

The memory mapping works well for regular send() calls without any attempts at zero-copy, but now we need to improve the network transmission speed for our data. In investigating the zero-copy mode for sendmsg() we have observed up to 50% improvements in data transfer rate, when we are using regular userspace allocated memory through malloc().

I have tried unsetting the vm_area_struct.vm_flags bits that correspond to VM_IO and VM_PFNMAP after the call to remap_pfn_range(). This results in sendmsg() actually "succeeding", the syscall returns the size of the buffer that I wished to send. However, no data arrives on the client side. Also, when I reboot the kernel, there are a lot of errors printed on the serial console.

I don't expect what I did to be the correct way to do this. I have to assume that the vm_flags are set for a very good reason, and there are no conditions in remap_pfn_range() that would prevent the flags from being set.

Hence, my question in the title: Is there a way to map memory obtained form kmalloc() to userspace such that it can be passed to sendmsg() with MSG_ZEROCOPY?

Update: I see that there have been a few posts regarding this subject already:

• Zero-copy user-space TCP send of dma_mmap_coherent() mapped memory

• Map physical memory to userspace as normal, struct page backed mapping

I will try the suggested answers from those posts and report back here.