cross-vm network connectivity issue when using socket_vmnet

[Ranjandas]

Description

I have frequently hit this issue when running VMs with lima:shared networking (socker_vmnet).

When this happens, the cross-vm traffic completely stalls. I am happy to provide further details that would help get to the bottom of this issue.

Aug 15 00:53:22 lima-murphy-cli-01 kernel: ------------[ cut here ]------------
Aug 15 00:53:22 lima-murphy-cli-01 kernel: NETDEV WATCHDOG: lima0 (virtio_net): transmit queue 0 timed out 6080 ms
Aug 15 00:53:22 lima-murphy-cli-01 kernel: WARNING: CPU: 2 PID: 0 at net/sched/sch_generic.c:525 dev_watchdog+0x2a8/0x2b0
Aug 15 00:53:22 lima-murphy-cli-01 kernel: Modules linked in: xt_multiport xt_nat xt_mark xt_comment veth tls xt_conntrack xt_MASQUERADE nf_conntrack_netlink nfnetlink ip6table_nat ip6table_filter ip6_tables iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 xt_addrtype iptable_filter ip_tables br_netfilter bridge stp llc overlay rfkill isofs vfat fat virtio_gpu virtio_dma_buf drm_shmem_helper drm_kms_helper fb_sys_fop
s syscopyarea sysfillrect sysimgblt virtio_input drm fuse xfs libcrc32c sr_mod cdrom sg crct10dif_ce ghash_ce sha2_ce sha256_arm64 sha1_ce virtio_blk virtio_console virtio_net net_failover virtio_scsi failover virtio_mmio sunrpc dm_mirror dm_region_hash dm_log dm_mod
Aug 15 00:53:22 lima-murphy-cli-01 kernel: CPU: 2 PID: 0 Comm: swapper/2 Not tainted 5.14.0-427.13.1.el9_4.aarch64 #1
Aug 15 00:53:22 lima-murphy-cli-01 kernel: Hardware name: QEMU QEMU Virtual Machine, BIOS edk2-stable202402-prebuilt.qemu.org 02/14/2024
Aug 15 00:53:22 lima-murphy-cli-01 kernel: pstate: 61400005 (nZCv daif +PAN -UAO -TCO +DIT -SSBS BTYPE=--)
Aug 15 00:53:22 lima-murphy-cli-01 kernel: pc : dev_watchdog+0x2a8/0x2b0
Aug 15 00:53:22 lima-murphy-cli-01 kernel: lr : dev_watchdog+0x2a8/0x2b0
Aug 15 00:53:22 lima-murphy-cli-01 kernel: sp : ffff800080013de0
Aug 15 00:53:22 lima-murphy-cli-01 kernel: x29: ffff800080013de0 x28: ffffc0360445a9c0 x27: ffff800080013e88
Aug 15 00:53:22 lima-murphy-cli-01 kernel: x26: ffffc036052f2008 x25: 00000000000017c0 x24: ffffc03605757000
Aug 15 00:53:22 lima-murphy-cli-01 kernel: x23: 0000000000000000 x22: ffff162d080e03dc x21: ffff162d080e0000
Aug 15 00:53:22 lima-murphy-cli-01 kernel: x20: ffff162d080e0488 x19: ffff162d00412a00 x18: ffffffffffffffff
Aug 15 00:53:22 lima-murphy-cli-01 kernel: x17: ffff55f73a0d6000 x16: ffff800080010000 x15: ffffc03605d443f9
Aug 15 00:53:22 lima-murphy-cli-01 kernel: x14: ffffffffffffffff x13: ffffc03605d443f0 x12: 696d736e61727420
Aug 15 00:53:22 lima-murphy-cli-01 kernel: x11: 656820747563205b x10: 0000000000000028 x9 : ffffc03603a94680
Aug 15 00:53:22 lima-murphy-cli-01 kernel: x8 : 000000000a736d20 x7 : 2074756f2064656d x6 : 000000000a736d20
Aug 15 00:53:22 lima-murphy-cli-01 kernel: x5 : ffff162d3f3cb950 x4 : 0000000000001000 x3 : 0000000000000040
Aug 15 00:53:22 lima-murphy-cli-01 kernel: x2 : 0000000000000000 x1 : 0000000000000000 x0 : ffff162d002f6900
Aug 15 00:53:22 lima-murphy-cli-01 kernel: Call trace:
Aug 15 00:53:22 lima-murphy-cli-01 kernel:  dev_watchdog+0x2a8/0x2b0
Aug 15 00:53:22 lima-murphy-cli-01 kernel:  call_timer_fn+0x38/0x1d0
Aug 15 00:53:22 lima-murphy-cli-01 kernel:  __run_timers.part.0+0x22c/0x300
Aug 15 00:53:22 lima-murphy-cli-01 kernel:  run_timer_softirq+0x40/0x74
Aug 15 00:53:22 lima-murphy-cli-01 kernel:  __do_softirq+0x118/0x3d8
Aug 15 00:53:22 lima-murphy-cli-01 kernel:  ____do_softirq+0x14/0x20
Aug 15 00:53:22 lima-murphy-cli-01 kernel:  call_on_irq_stack+0x24/0x30
Aug 15 00:53:22 lima-murphy-cli-01 kernel:  do_softirq_own_stack+0x20/0x30
Aug 15 00:53:22 lima-murphy-cli-01 kernel:  __irq_exit_rcu+0xfc/0x124
Aug 15 00:53:22 lima-murphy-cli-01 kernel:  irq_exit_rcu+0x14/0x1c
Aug 15 00:53:22 lima-murphy-cli-01 kernel:  el1_interrupt+0x48/0xa4
Aug 15 00:53:22 lima-murphy-cli-01 kernel:  el1h_64_irq_handler+0x14/0x20
Aug 15 00:53:22 lima-murphy-cli-01 kernel:  el1h_64_irq+0x78/0x7c
Aug 15 00:53:22 lima-murphy-cli-01 kernel:  cpuidle_idle_call+0xb4/0x1c0
Aug 15 00:53:22 lima-murphy-cli-01 kernel:  do_idle+0xb4/0x120
Aug 15 00:53:22 lima-murphy-cli-01 kernel:  cpu_startup_entry+0x28/0x30
Aug 15 00:53:22 lima-murphy-cli-01 kernel:  secondary_start_kernel+0x148/0x150
Aug 15 00:53:22 lima-murphy-cli-01 kernel:  __secondary_switched+0x6c/0x70
Aug 15 00:53:22 lima-murphy-cli-01 kernel: ---[ end trace 7ce1380af930c7d1 ]---
Aug 15 00:53:22 lima-murphy-cli-01 kernel: virtio_net virtio2 lima0: TX timeout on queue: 0, sq: output.0, vq: 0x1, name: output.0, 6080000 usecs ago
Aug 15 00:53:27 lima-murphy-cli-01 kernel: virtio_net virtio2 lima0: TX timeout on queue: 0, sq: output.0, vq: 0x1, name: output.0, 11040000 usecs ago

The below message repeats for ever.

Aug 15 00:53:27 lima-murphy-cli-01 kernel: virtio_net virtio2 lima0: TX timeout on queue: 0, sq: output.0, vq: 0x1, name: output.0, xxxxxxxx usecs ago

[ranjan@lima-murphy-cli-01 ~]$ cat /etc/os-release
NAME="Rocky Linux"
VERSION="9.4 (Blue Onyx)"
ID="rocky"
ID_LIKE="rhel centos fedora"
VERSION_ID="9.4"
PLATFORM_ID="platform:el9"
PRETTY_NAME="Rocky Linux 9.4 (Blue Onyx)"
ANSI_COLOR="0;32"
LOGO="fedora-logo-icon"
CPE_NAME="cpe:/o:rocky:rocky:9::baseos"
HOME_URL="https://rockylinux.org/"
BUG_REPORT_URL="https://bugs.rockylinux.org/"
SUPPORT_END="2032-05-31"
ROCKY_SUPPORT_PRODUCT="Rocky-Linux-9"
ROCKY_SUPPORT_PRODUCT_VERSION="9.4"
REDHAT_SUPPORT_PRODUCT="Rocky Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="9.4"

$ limactl --version
limactl version 0.22.0

$ brew info socket_vmnet lima| head -n 1
==> socket_vmnet: stable 1.1.4 (bottled), HEAD [keg-only]

$ sw_vers
ProductName:            macOS
ProductVersion:         14.5
BuildVersion:           23F79

$ sysctl -n machdep.cpu.brand_string

Apple M3 Pro