Go Proposal: Exporting Goroutine Metrics for Enhanced Observability

Go Programming

Discover the upcoming Go 1.26 changes introducing new `runtime/metrics` for detailed goroutine states and thread counts, enhancing observability and debugging capabilities.

This article is part of the Accepted! series, offering clear explanations of upcoming Go changes.

We delve into the proposal for exporting goroutine-related metrics directly from the Go runtime. This feature, expected in Go 1.26, is a standard library addition with a medium impact, significantly enhancing observability for Go applications.

Summary

The runtime/metrics package will introduce new metrics, providing deeper insights into goroutine scheduling and thread management:

Total goroutines created: Since program inception.
Goroutines by state: Breakdown of goroutines in various states.
Active threads: Count of threads managed by the Go runtime.

Motivation

While Go's runtime/metrics package already offers extensive runtime statistics, it currently lacks detailed metrics for goroutine states and thread counts. These per-state goroutine metrics are crucial for diagnosing common production issues:

An increasing /sched/goroutines/waiting:goroutines count can signal a lock contention problem.
A high /sched/goroutines/not-in-go:goroutines count often indicates goroutines are blocked in syscalls or Cgo calls.
A growing /sched/goroutines/runnable:goroutines backlog suggests that CPU resources are insufficient to meet demand.

By tracking these counters, observability systems can proactively identify regressions, pinpoint scheduler bottlenecks, and trigger alerts when goroutine behavior deviates from normal patterns. This allows developers to catch and address problems early, often without requiring full traces.

Description

The following new metrics will be added to the runtime/metrics package, all utilizing uint64 counters:

/sched/goroutines-created:goroutines: The total count of goroutines created since the program began.
/sched/goroutines/not-in-go:goroutines: An approximate count of goroutines currently executing or blocked within a system call or Cgo call.
/sched/goroutines/runnable:goroutines: An approximate count of goroutines ready to execute but not currently running.
/sched/goroutines/running:goroutines: An approximate count of goroutines actively executing. This value will always be less than or equal to /sched/gomaxprocs:threads.
/sched/goroutines/waiting:goroutines: An approximate count of goroutines paused while waiting on a resource, such as I/O operations or synchronization primitives.
/sched/threads/total:threads: The current count of live threads managed and owned by the Go runtime.

It's important to note that the sum of the per-state goroutine numbers (not-in-go, runnable, running, waiting) is not guaranteed to equal the total live goroutine count (/sched/goroutines:goroutines, which has been available since Go 1.16).

Example

Here's an example demonstrating how to retrieve and print these new metrics after 100 milliseconds of activity:

func main() {
	go work() // omitted for brevity
	time.Sleep(100 * time.Millisecond)

	fmt.Println("Goroutine metrics:")
	printMetric("/sched/goroutines-created:goroutines", "Created")
	printMetric("/sched/goroutines:goroutines", "Live")
	printMetric("/sched/goroutines/not-in-go:goroutines", "Syscall/CGO")
	printMetric("/sched/goroutines/runnable:goroutines", "Runnable")
	printMetric("/sched/goroutines/running:goroutines", "Running")
	printMetric("/sched/goroutines/waiting:goroutines", "Waiting")

	fmt.Println("Thread metrics:")
	printMetric("/sched/gomaxprocs:threads", "Max")
	printMetric("/sched/threads/total:threads", "Live")
}

func printMetric(name string, descr string) {
	sample := []metrics.Sample{{
		Name: name,
	}}
	metrics.Read(sample)

	// Assuming a uint64 value; don't do this in production.
	// Instead, check sample[0].Value.Kind and handle accordingly.
	fmt.Printf("  %s: %v
", descr, sample[0].Value.Uint64())
}

The example output typically looks like this:

Goroutine metrics:
  Created: 52
  Live: 12
  Syscall/CGO: 0
  Runnable: 0
  Running: 4
  Waiting: 8
Thread metrics:
  Max: 8
  Live: 4

As expected, accessing these new metric values follows the established method using metrics.Read.

Go Proposal: Exporting Goroutine Metrics for Enhanced Observability

Summary

Motivation

Description

Example

Further Reading