Etcd exits without a trace

[purpleidea]

I'm hacking on #5430
I've added a method to etcdmain, and for fun, I imported it and tested it. It compiles and runs, but I knew that I was lazy and was calling a method on a nil variable, and was expecting a panic... My program runs, and dies, but no trace is seen!

Indeed the program seems to be dying when it calls a method on the nil variable, but how come I don't get a trace? I'm assuming etcd has somewhere hidden some "runtime" hack to disable this.

Could point me to where this is happening, it's making hacking on etcd painful!

[purpleidea]

Alternatively, if this isn't the case, something weird is happening, and I broke golang!

[xiang90]

Are you running with systemd?

[purpleidea]

Nope, I'm running interactively on the cli, although this is a build where I've called NewServer in my code since I'm embedding etcd server.

[xiang90]

Then I am not sure. All I can think of is a logging issue. Go should always panic with tracing.

[gyuho]

@purpleidea What's your OS? Or can you share that code to reproduce this?

[purpleidea]

On Mon, May 23, 2016 at 2:26 PM, Gyu-Ho Lee notifications@github.com
wrote:

@purpleidea https://github.com/purpleidea What's your OS? Or can you
share that code to reproduce this?

Fedora GNU/Linux 23, with stock distro golang. After my patch is finished,
I will share and see if the issue still exists. I was fairly certain it was
just some etcd hidden call to the runtime lib or similar, but maybe not!

[gyuho]

@purpleidea We don't silence any runtime panics, as far as I know. Please share the logs whenever it's ready.

Thanks!

[purpleidea]

I seem to have found a reproducer, although I have experienced this with different code before.

Doing the following:

bug := len(response.Events) // this line causes the abrupt unlogged quit.
_ = bug

where response is a

etcd.WatchResponse

which is returned when an watch event comes in. I don't have a trimmed down code example of this at the moment, but it should be fairly straightforward to throw together if someone is interested in confirming this.

FWIW I'm using:

• Fedora 23
• go version go1.5.4 linux/amd64
• etcd v.3.0.0-beta.0

Cheers

[gyuho]

bug := len(response.Events) // this line causes the abrupt unlogged quit.

This cannot happen unless response is nil. Are you sure that it was etcd that exited?

Even if response is nil, it will exit with runtime panic stacks as below

package main

import (
    "fmt"
)

type WatchResponse struct {
    Events []string
}

func main() {
    resp := &WatchResponse{}
    resp = nil
    bug := len(resp.Events)
    _ = bug
    fmt.Println("done")
}

/*
panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xb code=0xffffffff addr=0x0 pc=0x20044]

goroutine 1 [running]:
panic(0x13f480, 0x1040a038)
    /usr/local/go/src/runtime/panic.go:481 +0x700
main.main()
    /tmp/sandbox289640460/main.go:14 +0x44
*/

[purpleidea]

@gyuho Yeah, I agree. Maybe this is a golang bug I'm hitting. It's definitely strange.

#5431

[gyuho]

@purpleidea I don't think it's go bug either. Please re-open this when you can reliably reproduce.

I think there must be something related in your project. We had never seen this in etcd before :0

Thanks!

[purpleidea]

On Sun, May 29, 2016 at 6:14 PM, Gyu-Ho Lee notifications@github.com
wrote:

@purpleidea https://github.com/purpleidea I don't think it's go bug
either. Please re-open this when you can reliably reproduce. But I think
there must be something related in your project. We had never seen this in
etcd before :0

Thanks!

Fair enough! Sorry for the noise.

[gyuho]

Np! Please feel free to ask anything!

[purpleidea]

On Sun, May 29, 2016 at 6:24 PM, Gyu-Ho Lee notifications@github.com
wrote:

Np! Please feel free to ask anything!

You all have been immensely helpful during my etcd hacking. Let me know if
you're ever in Montreal since I'm pretty sure I owe you a few beverages or
montreal $food by now.