Core dump analysis in Go using Delve

If there is a crash in Golang application, how will you debug ? Delve is the answer.

Delve is a Golang debugging tool which can be used to debug core crashes, live golang applications etc.

Delve is supported from Go version 1.16+.

How to bring up Delve

Install Golang version >= 1.16
$ mkdir -p ~/Downloads && cd ~/Downloads
$ wget https://golang.org/dl/go1.16.6.linux-amd64.tar.gz
$ sudo rm -rf /usr/local/go/
$ sudo tar -C /usr/local -xzf go1.16.6.linux-amd64.tar.gz
$ go version
go version go1.16.6 linux/amd64
Install Delve
$ git clone https://github.com/go-delve/delve
$ cd delve
$ go install github.com/go-delve/delve/cmd/dlv
$ dlv version

Enable application to generate core dump in the application running folder.
By default ulimit value will be set to 0. We have to increase it to allow applications to store core dumps.

$ ulimit -c                 // Checking the current ulimit value 
0
$ ulimit -c unlimited       // Setting the ulimit value to unlimited

Sample Source Code

package main
import (
 "fmt"
)
type A struct {
 Name string
}
func (a *A) Greetings() {
 fmt.Println("Hello : ", a.Name)
}
func (a *A) DivideByZero() {
 for i := 0; i < 10; i++ {
  fmt.Println("Divide by 0 : ", (100 / (7 - i)))
 }
}
func sayGreetings(a *A) {
 fmt.Println("sayGreetings called ")
 a.Greetings()
}
func divide(a *A) {
 fmt.Println("divide called ")
 a.DivideByZero()
}
var a *A
var b *A
func main() {
 fmt.Println("Main called...")
 a = nil
 b = new(A)
 b.Name = "Prince"
 fmt.Println("B : ", b)
 divide(a)         // Helps to make app crash by a local variable.
 //sayGreetings(a) // Helps to make app crash by a package variable.
}

Building the binary

$ go mod init
$ go mod tidy
# Generates a binary. But while debugging, value of variables will not be clear due to compiler optimization.
$ go build -o binary
OR
# Generates a binary. But while debugging, value of variables will be clear due to compiler optimization disabled.
$ go build -gcflags=all="-N -l" -o binary

Run the program using GOTRACEBACK=crash and core dump will be generated in same folder.

$ GOTRACEBACK=crash ./binary
$ ls
binary  core  go.mod  main.go

Will rename the core dump from “core” to “core-dump” just for readability purpose.

$ mv core core-dump
$ ls
binary  core-dump  go.mod  main.go

How to debug using Delve ? Load the core dump to binary using dlv tool for analyzing.

# dlv  : is the command
# core : is the option of delve to debug core dump
# binary : is the binary built
# core-dump : is the core dump generated after crash
$ dlv core ./binary core-dump
# You will be entering delve command line.
(dlv)

What are the commands ?

# Do a backtrace first to list the frames and see the final frame where the know source code is visible
(dlv) bt    // Backtrace: Prints the stacktrace
8  0x0000000000432325 in runtime.gopanic
    at /usr/local/go/src/runtime/panic.go:1065
 9  0x000000000043035b in runtime.panicdivide
    at /usr/local/go/src/runtime/panic.go:191
10  0x000000000049785a in main.(*A).DivideByZero
    at ./main.go:17
11  0x0000000000497907 in main.divide
    at ./main.go:28
12  0x0000000000497a85 in main.main
    at ./main.go:40
13  0x0000000000434f76 in runtime.main
    at /usr/local/go/src/runtime/proc.go:225
# Left side numbers represents Frame numbers.
# Pick the frame number of last known code and enter the frame.
(dlv) frame 10
Frame 10: ./main.go:17 (PC: 49785a)
    12:  fmt.Println("Hello : ", a.Name)
    13: }
    14: 
    15: func (a *A) DivideByZero() {
    16:  for i := 0; i < 10; i++ {
=>  17:   fmt.Println("Divide by 0 : ", (100 / (7 - i)))
    18:  }
    19: }
# It will point out the point of crash.
# Check the values of all local variables.
(dlv) locals
i = 7
(dlv)
# If want to check the value of specific local variable which caused crash.
# 'p' means 'print'
(dlv) p i
7
(dlv)
# If the reason for the crash is a package variable, you can dump package variables using 'vars' command.
(dlv) vars <Package name>
(dlv) vars <Package name>.<variable name>
(dlv) vars main
runtime.main_init_done = chan bool 0/0
runtime.mainStarted = true
main.a = *main.A nil
main.b = (*main.A)(0xc000010250)
(dlv)
(dlv) vars main.b
main.b = (*main.A)(0xc000010250)
(dlv)
(dlv) p main.b
*main.A {Name: "Prince"}
(dlv)

How to generate core dump without crashing application ?

$ ./binary &
$ sudo gcore 7024 # 546 is the 7024 of binary.
OR
$ sudo gcore $(pgrep binary) # This is to run the gcore command in a shell script