[Editor – The solution described in this blog relies on the NGINX Plus Status module (enabled by the status directive). That module is replaced and deprecated by the NGINX Plus API in NGINX Plus R13 and later, and will not be available after NGINX Plus R15. For the solution to continue working, update the code that refers to the deprecated module.]
In this day and age, having an infrastructure that is scalable and reliable is crucial to your site’s success. Monitoring NGINX performance and the health of your load balanced applications is critical. Knowing how much traffic your virtual servers are receiving enables you to scale when needed and keeping an eye on error rates allows you to triage your applications effectively. Monitoring and acting on these metrics help provide your users with a reliable and satisfying experience.
The live activity monitoring feature in NGINX Plus makes it easy to get key load and performance metrics in real time. The large amount of useful data about the traffic flowing through NGINX Plus is available both on the built‑in dashboard and in an easy‑to‑parse JSON format through an HTTP‑based API.
There are many effective ways to use the data provided by the live activity monitoring API. For example, you might want to monitor a particular virtual server’s upstream traffic and autoscale Docker containers using the NGINX Plus dynamic reconfiguration API. You might want to dump the metrics directly to standard out and write them to a log file so you can send metrics to an ELK stack or Splunk cluster. Or perhaps you want to collect and send the API data to a data aggregator like statsd and collectd for other graphing or logging purposes.
There are a large number of packages written by Go Community members that might provide the functionality you are looking for. As another example, I also used a third party package called go-statsd-client to send some of my statistics directly to StatsD.
StatsD is a network daemon that runs on the Node.js platform and listens for statistics, like counters and timers, sent over UDP or TCP and sends aggregates to one or more pluggable backend services.
About Go
Go, also known as golang, is an open source programming language developed by Google that makes it easy to build simple, reliable, and efficient software. If you are interested in learning more about Go, check out How to Write Go Code and Go by Example. The Go site also has extensive documentation on the various packages available. The goal of this blog is to show you some basic methods and Go packages that can be used to collect NGINX Plus live activity monitoring API metrics.
I tested the sample script against Go version 1.4.2 and NGINX Plus R7 (based on NGINX Open Source 1.9.4).
$ go version
go version go1.4.2 linux/amd64
$ sudo /usr/sbin/nginx -V
nginx version: nginx/1.9.4 (nginx-plus-r7)Importing Go Packages
The first thing we define in the script is the package name and the packages we want to import at run or build time.
package main
import (
"encoding/json"
"errors"
"fmt"
"github.com/cactus/go-statsd-client/statsd"
"io/ioutil"
"log"
"net/http"
"time"
)In my sample script I use a handful of packages that are available in the standard Go library. Here is a list of those packages along with a brief description of what they are used for.
-
encoding/json– Encoding and decoding of JSON objects -
errors– Manipulating error messages -
fmt– Formatting I/O functions (printing data to the terminal) -
io/ioutil– Implementing I/O utility functions -
log– Logging and printing errors -
net/httpv – Making an HTTP client connection to the API endpoint - time – Creating a sleep mechanism in our script
Defining the JSON Data Structure
In Go you must define the data structure of your JSON. Note that Go simply ignores any JSON data that you do not define in your data structure. In this example I am only grabbing the connection statistics from the NGINX Plus API.
type NginxResponse struct {
Connections struct {
Accepted int64 `json:"accepted"`
Active int64 `json:"active"`
Dropped int64 `json:"dropped"`
Idle int64 `json:"idle"`
} `json:"connections"`
}If you decide to build on this JSON data structure and add upstream and server zone values, keep in mind that some of the data field names will be unique to your NGINX Plus instance and configuration. There are a lot of tools online that will easily convert a JSON dump into a Go struct for you, but once you get the hang of how the structure is defined it is very easy to create your own.
Defining the NginxStatus and SendStatsD Functions
My Go script consists of three functions: NginxStatus, SendStatsD, and the main function, which I’ll discuss in that order. The NginxStatus function makes a call to the NGINX Plus live activity monitoring API.
I first define the NGINX Plus server, then perform a GET request to the live activity monitoring API. I verify that the status code in the response is 200 OK and use the errors package to log the issue if not. I also defer the close of the connection until my function completes, at which point it will close the connection automatically.
I then read the response body into a variable and decode the JSON using the struct that I defined in Defining the JSON Data Structure. I pass the data back to the main function. I also print out the statistics on screen and pass them to my SendStatsD function.
func NginxStatus() (*NginxResponse, error) {
// assign variable for NGINX Plus server
var nginxStatusServer string = "my.nginx.server.com"
// perform a request to the NGINX Plus status API
resp, err := http.Get(fmt.Sprintf("http://%s/status", nginxStatusServer))
if err != nil {
return nil, err
}
if resp.StatusCode != http.StatusOK {
return nil, errors.New("Non 200 OK")
}
// clean up the connection
defer resp.Body.Close()
// read the body of the request into a variable
data, err := ioutil.ReadAll(resp.Body)
if err != nil {
return nil, err
}
// unmarshall the JSON data into a variable
var er NginxResponse
if err := json.Unmarshal(data, &er); err != nil {
return nil, err
}
return &er, nil
}In my SendStatsD function I define the variables and type that I wish to send directly to StatsD. At the beginning of the function I define the StatsD server and port I will be connecting to, along with the client name. I then establish a connection, log any unexpected errors, and again defer the closing of the connection. I then assign a variable to the flush interval for StatsD and another to part of the metric name. The flush interval tells StatsD how long to wait before sending data to its backend. Finally, at the end of my SendStatsD function I send all of the data to StatsD using the built‑in Inc function.
func SendStatsD(gs string, gt string, gv int64) {
// assign variables for statsd server and metric name prefix
var statsdServer string = "127.0.0.1:8125"
var gk string = "nginx"
// connect to statsd
client, err := statsd.NewClient(statsdServer, gk)
if err != nil {
log.Println(err)
}
// clean up the connections
defer client.Close()
// assign variables for metric name and interval
var fi float32 = 1.0
var st string = "status"
var sn string = "my_nginx_server_com"
// send metrics to StatsD
client.Inc(st+"."+sn+"."+gs+"."+gt, gv, fi)
}It is very important this is properly constructed because it tells StatsD how to organize your data. Think of the metric name like a folder structure with the period as the separator between each level. How you name the metric namespace determines where the data is stored and available in the Graphite web UI. The way you store your data is up to you, but I recommended you use the following naming convention when defining your metric names. I use angle brackets here to make it easier to see the segments of the name; they aren’t used in actual names.
<namespace>.<section-type>.<server-name>.<target noun>.<stat adjective or verb>for example:
nginx.status_api.my_nginx_server_com.connections.activeThis example tells StatsD to send data from the NGINX Plus status API about connections that are currently active on a server named my.nginx.server.com. Notice that you replace any periods or spaces within a segment of the name with the underscore character.
Defining the main Function
In the main function I create an infinite loop and then simply make calls to the functions I described above to easily receive and send data as needed.
func main() {
for {
// query the NginxStatus function to get data
nr, err := NginxStatus()
if err != nil {
log.Println(err)
}
// print the statistics on screen
fmt.Println("Connections Accepted:", nr.Connections.Accepted)
fmt.Println("Connections Dropped:", nr.Connections.Dropped)
fmt.Println("Connections Active:", nr.Connections.Active)
fmt.Println("Connections Idle", nr.Connections.Idle)
// send metrics to the SendStatsD function
go SendStatsD("connections", "accepted", nr.Connections.Accepted)
go SendStatsD("connections", "dropped", nr.Connections.Dropped)
go SendStatsD("connections", "active", nr.Connections.Active)
go SendStatsD("connections", "idle", nr.Connections.Idle)
// sleep for one second
time.Sleep(time.Millisecond * 1000)
}
}Here I assign a variable to the results of an API call to NGINX Plus using the NginxStatus function. Then I use the fmt package to print some of my connection statistics on screen and my SendStatsD function to feed that data to Graphite.
I also run the sleep function to insert a delay of one second between each request. In this blog I am not covering in detail how to handle connection or time management. When creating your script you will need to determine how you want to deal with connection timeouts or latency and how to account for the time between each request.
Here is an example of the screen output after running the script. The data sent to StatsD is transparent and can be verified by checking your Graphite web UI.
$ go run main.go
Connections Accepted: 36480962
Connections Dropped: 0
Connections Active: 13
Connections Idle 30The Complete Script
Here is the complete text of the script for reference. Spacing matters in Go, so be sure to check the syntax if you are getting any errors.
package main
import (
"encoding/json"
"errors"
"fmt"
"github.com/cactus/go-statsd-client/statsd"
"io/ioutil"
"log"
"net/http"
"time"
)
type NginxResponse struct {
Connections struct {
Accepted int64 `json:"accepted"`
Active int64 `json:"active"`
Dropped int64 `json:"dropped"`
Idle int64 `json:"idle"`
} `json:"connections"`
}
func NginxStatus() (*NginxResponse, error) {
// assign variable for NGINX Plus server
var nginxStatusServer string = "my.nginx.server.com"
// perform a request to the NGINX Plus status API
resp, err := http.Get(fmt.Sprintf("http://%s/status", nginxStatusServer))
if err != nil {
return nil, err
}
if resp.StatusCode != http.StatusOK {
return nil, errors.New("Non 200 OK")
}
// clean up the connection
defer resp.Body.Close()
// read the body of the request into a variable
data, err := ioutil.ReadAll(resp.Body)
if err != nil {
return nil, err
}
// unmarshall the JSON data into a variable
var er NginxResponse
if err := json.Unmarshal(data, &er); err != nil {
return nil, err
}
return &er, nil
}
func SendStatsD(gs string, gt string, gv int64) {
// assign variables for statsd server and metric name prefix
var statsdServer string = "127.0.0.1:8125"
var gk string = "nginx"
// connect to statsd
client, err := statsd.NewClient(statsdServer, gk)
if err != nil {
log.Println(err)
}
// clean up the connections
defer client.Close()
// assign variables for metric name and interval
var fi float32 = 1.0
var st string = "status"
var sn string = "my_nginx_server_com"
// send metrics to statsd
client.Inc(st+"."+sn+"."+gs+"."+gt, gv, fi)
}
func main() {
for {
// query the NginxStatus function to get data
nr, err := NginxStatus()
if err != nil {
log.Println(err)
}
// print the statistics on screen
fmt.Println("Connections Accepted:", nr.Connections.Accepted)
fmt.Println("Connections Dropped:", nr.Connections.Dropped)
fmt.Println("Connections Active:", nr.Connections.Active)
fmt.Println("Connections Idle", nr.Connections.Idle)
// send metrics to the SendStatsD function
go SendStatsD("connections", "accepted", nr.Connections.Accepted)
go SendStatsD("connections", "dropped", nr.Connections.Dropped)
go SendStatsD("connections", "active", nr.Connections.Active)
go SendStatsD("connections", "idle", nr.Connections.Idle)
// sleep for one second
time.Sleep(time.Millisecond * 1000)
}
}Summary
Using Go to access the NGINX Plus live activity monitoring API can provide some additional flexibility around the data already available to you in NGINX Plus. The statistical data you pull from NGINX Plus can help provide better logging, create automation around live activity monitoring, and create graphs or charts based on the performance history of NGINX Plus.
To try NGINX Plus, start your free 30-day trial today or contact us to discuss your use cases.
