guest agent events closed unexpectedly error="rpc error: code = Unavailable desc = error reading from server: EOF"

[shefmarkh]

Description

Hello,

I am working on a M1 MacBook with Sonoma 14.6.1

Recently I updated Lima from 0.21.0 to 0.23.1 via brew. Subsequently when starting my container up I see:

WARN[0012] [hostagent] failed to set up forwarding tcp port 111 (negligible if already forwarded) error="listen tcp4 0.0.0.0:111: bind: address already in use"
WARN[0012] [hostagent] guest agent events closed unexpectedly error="rpc error: code = Unavailable desc = error reading from server: EOF"

Then workflows in the container which used to work now fail because attempted connections to a remote database started timing out. I would therefore suspect this might be related to the errors I see above.

When stopping the container I also now see a new error:

ERRO[0001] [hostagent] accept tcp 127.0.0.1:58584: use of closed network connection

As far as I could workout brew does not support reverting versions of software, so I have no way to go back to Lima 0.21? Is there any suggestion as to how to fix the above warnings and errors?

Thanks,

Mark

[shefmarkh]

I compiled v0.21.0 from the source code and when I use that the database access issues resolve themselves.

When starting the container I no longer see:

WARN[0012] [hostagent] guest agent events closed unexpectedly error="rpc error: code = Unavailable desc = error reading from server: EOF"

though I do still see:

WARN[0012] [hostagent] failed to set up forwarding tcp port 111 (negligible if already forwarded) error="listen tcp4 0.0.0.0:111: bind: address already in use"

So it seems the problem is related to this issue:

WARN[0012] [hostagent] guest agent events closed unexpectedly error="rpc error: code = Unavailable desc = error reading from server: EOF"

Is this something that could be fixed in the next lima version? Or could the problem be actually something on my MacBook that I should resolve? If so, can you advise how to resolve that?

Thanks,

Mark

[AkihiroSuda]

Needs reproduction steps.

Also, can you try bisecting?

[shefmarkh]

You could create the container via:

limactl start https://gitlab.cern.ch/atlas-sit/containers/-/raw/master/Lima/almalinux9.yaml --name=alma9

and then when you start it you should see the messages I described, assuming the problem is entirely with lima 0.23.1 and its not also some interdependent issue that also correlates with Mac OS version...

I don't know what bisecting is. How would I do that?

Thanks,

Mark

[AkihiroSuda]

limactl start https://gitlab.cern.ch/atlas-sit/containers/-/raw/master/Lima/almalinux9.yaml --name=alma9

Can you minimize the reproducer?
Is it reproducible with the bare template in https://github.com/lima-vm/lima/tree/master/examples ?
Or reproduction depends on some CERN stuff?

I don't know what bisecting is. How would I do that?

https://git-scm.com/docs/git-bisect

[AkihiroSuda]

Anyway https://gitlab.cern.ch/atlas-sit/containers/-/raw/master/Lima/almalinux9.yaml doesn't seem to work outside CERN's network

[shefmarkh]

I'm unsure what you mean by "not work". I am using it outside the cern network and it does work for me.

[shefmarkh]

I found out a few other things.

1. I cannot reproduce this with the baseline example you suggested ( https://github.com/lima-vm/lima/blob/master/examples/almalinux-9.yaml). Trying 5 times I don't see the error.
2. With lima 0.21.0 I see that sometimes I see the error when starting my container and sometimes not. So there was no correlation with lima version, just by coincidence it worked with lima 0.21.0 when I tested and did not work with 0.23.1. So the issue is not fully reproducible.

So it must be something specific to the container setup I have, but somehow I never saw this issue until a few days ago whilst I have been using this container for many months without seeing this issue.