driver: add a bootstrap timeout

[jsternberg]

Drivers will now fail with a timeout if they take over 20 seconds to
boot. This process repeats itself 3 times for a total of a minute.

Fixes #2120.

[tonistiigi]

I don't think we should set a general limit for boot. Eg. if you are on slow connection maybe pulling buildkit image takes a lot of time, or who knows what may affect the boot time for k8s driver. I think we should concentrate on the initial report of detecting unresponsive remote server or server that is not buildkit at all.

[jsternberg]

We can put the cancellation on the Wait() call instead and make it driver specific so this would only apply to the remote. It might also be better to include it as an option to the driver instead of just a default parameter. That way, we're also not changing the existing behavior.

The main problem with this that we run into is that GRPC doesn't really tell us if the GRPC service is unavailable in a permanent or temporary way. We'd have to parse the error message explicitly. GRPC's reconnection mechanism also seems to only be for reconnections and doesn't apply to the initial connection.

Another option that might help might be to include a status message for the bootstrap process. Right now, it just kind of stalls and doesn't show any message about the progress of the bootstrap. It might help if we use the progress writer to at least say how long the process is taking.

[tonistiigi]

We can put the cancellation on the Wait() call instead and make it driver specific so this would only apply to the remote.

If you mean the client.Wait() in boot then that seems fine.

It might help if we use the progress writer to at least say how long the process is taking.

That would mean collecting such messages from the grpc level I think. I'm not sure how possible it is to capture that, and keep it in the form that it is useful for the user.

[jsternberg]

I've created a PR for an alternative to this one: #2130

If the service is already up, it doesn't change the output at all. If the service is invalid like in the original PR, it looks like this:

[+] Building 20.0s (1/1) FINISHED
 => ERROR [internal] waiting for connection                                                                                 20.0s
------
 > [internal] waiting for connection:
------
ERROR: context deadline exceeded

For ctrl+c:

[+] Building 0.9s (1/1) FINISHED
 => CANCELED [internal] waiting for connection                                                                               0.9s
ERROR: context canceled

[jsternberg]

Closing this in favor of #2130.