Metrics & Profiling

Ghostunnel has a notion of “status port”, a TCP port (or UNIX socket) that can be used to expose status and metrics information over HTTPS. The status port feature can be controlled via the --status flag. Profiling endpoints on the status port can be enabled with --enable-pprof.

The X.509 certificate on the status port will be the same as the certificate used for proxying (either the client or server certificate). This means you can use the status port to inspect/verify the certificate that is being used, which can be useful for orchestration systems.

Example invocation with status port enabled:

ghostunnel client \
    --listen localhost:8080 \
    --target localhost:8443 \
    --keystore test-keys/client-keystore.p12 \
    --cacert test-keys/cacert.pem \
    --status localhost:6060

Note that we set the status port to “localhost:6060”. Ghostunnel will start an internal HTTPS server and listen for connections on the given host/port. You can also specify a UNIX socket instead of a TCP port.

How to check status and read connection metrics:

# Status information (JSON)
curl --cacert test-keys/cacert.pem https://localhost:6060/_status

# Metrics information (JSON)
curl --cacert test-keys/cacert.pem 'https://localhost:6060/_metrics/json'

# Metrics information (Prometheus)
curl --cacert test-keys/cacert.pem 'https://localhost:6060/_metrics/prometheus'

How to use profiling endpoints, if --enable-pprof is set:

# Human-readable goroutine dump
curl --cacert test-keys/cacert.pem 'https://localhost:6060/debug/pprof/goroutine?debug=1'

# Analyze execution trace using pprof tool
go tool pprof -seconds 5 https+insecure://localhost:6060/debug/pprof/profile

Note that go tool pprof does not support setting CA certificates at the moment, hence the use of the https+insecure scheme in the last example. You can use the standard https scheme if your Ghostunnel is using a certificate trusted by your system (see golang/go#20939). For more information on profiling via pprof, see the runtime/pprof and net/http/pprof docs.

Shutdown Endpoint

Available since v1.8.1.

If --enable-shutdown is set, a /_shutdown endpoint is available on the status port. Sending an HTTP POST request to this endpoint will trigger a graceful shutdown of the Ghostunnel process. Any other HTTP method returns 405 Method Not Allowed. For details on what happens after shutdown is triggered, including signal handling, connection draining, and the --shutdown-timeout flag, see Graceful Shutdown.

Backend Healthchecks

The /_status endpoint includes a backend healthcheck (server mode only). By default, Ghostunnel performs a TCP connection check against the --target address. You can override this with --target-status=URL (must use http:// or https:// scheme) to perform an HTTP GET against the given URL instead. Ghostunnel expects an HTTP 200 response.

The /_status JSON response includes:

• backend_ok: boolean indicating if the backend check passed
• backend_status: string of ok or critical
• backend_error: string of error message if the check failed

If the backend check fails, the /_status endpoint returns HTTP 503.

Metric Names

Ghostunnel exports the following base metrics:

The --metrics-prefix flag (default: ghostunnel) is prepended to all metric names. How the prefix and metric names are formatted depends on the output format (see below).

JSON format (/_metrics/json)

JSON output uses dot-separated names. Counters and gauges are emitted as a single value. Timers are expanded into count, min/max/mean, and percentile sub-metrics:

Each metric is returned as a JSON object with timestamp, metric, value, and hostname fields.

Prometheus format (/_metrics/prometheus)

Prometheus output replaces dots, dashes, and other special characters with underscores to comply with Prometheus naming conventions. All metrics are exposed as Prometheus gauges. Timers additionally include rate gauges, statistical gauges, and a summary histogram:

Prometheus scrape config

To scrape Ghostunnel metrics with Prometheus, add a job to your prometheus.yml:

scrape_configs:
  - job_name: ghostunnel
    scheme: https
    tls_config:
      ca_file: /path/to/cacert.pem
      cert_file: /path/to/client-cert.pem
      key_file: /path/to/client-key.pem
    metrics_path: /_metrics/prometheus
    static_configs:
      - targets: ['localhost:6060']

If the status port uses HTTP (see below), set scheme: http and drop the tls_config block.

Metrics Export

Metrics are always available via the status port endpoints (/_metrics/json, /_metrics/prometheus). Additionally, metrics can be pushed to external systems:

• --metrics-graphite=ADDR: push to a Graphite instance via raw TCP (dot-separated names, same as JSON format)
• --metrics-url=URL: push via HTTP POST (JSON format) at the interval set by --metrics-interval (default: 30s)

Exposing Status Port with HTTP Instead of HTTPS

By default, Ghostunnel uses HTTPS for the status port. You can force it to use HTTP by prefixing the status address with “http://”.

For example:

# Status flag passed to Ghostunnel
ghostunnel server ... --status http://localhost:6060

# Status information (JSON)
curl http://localhost:6060/_status

# Metrics information (JSON)
curl http://localhost:6060/_metrics/json

# Metrics information (Prometheus)
curl http://localhost:6060/_metrics/prometheus