NAME

VCL-probe - Configuring Backend Health Probes

BACKEND HEALTH PROBES

Varnish can be configured to periodically send a request to test if a backend is answering and thus "healthy".

Probes can be configured per backend:

backend foo {


    [...]


    .probe = {


        [...]


    }
}

They can be named and shared between backends:

probe light {


    [...]
}
backend foo {


    .probe = light;
}
backend bar {


    .probe = light;
}

Or a default probe can be defined, which applies to all backends without a specific .probe configured:

probe default {


    [...]
}

The basic syntax is the same as for backends:

probe name {


    .attribute1 = value;


    .attribute2 = "value";


    [...]
}

There are no mandatory attributes, they all have defaults.

ATTRIBUTE .URL

The URL to query. Defaults to /:

.url = "/health-probe";

ATTRIBUTE .REQUEST

Can be used to specify a full HTTP/1.1 request to be sent:

.request = "GET / HTTP/1.1"


    "Host: example.com"


    "X-Magic: We're fine with this."


    "Connection: close";

Each of the strings will have CRNL appended and a final HTTP header block terminating CRNL will be appended as well.

Because connection shutdown is part of the health check, Connection: close is mandatory.

ATTRIBUTE .EXPECTED_RESPONSE

The expected HTTP status, defaults to 200:

.expected_response = 418;

ATTRIBUTE .TIMEOUT

How fast the probe must succeed, default is two seconds:

.timeout = 10s;

ATTRIBUTE .INTERVAL

Time between probes, default is five seconds:

.interval = 1m;

THE BACKEND HEALTH SHIFT REGISTER

Backend health probes uses a 64 stage shift register to remember the most recent health probes and to evaluate the total health of the backend.

In the CLI, a good backend health status looks like this:

varnish> backend.list -p boot.backend
Backend name    Admin    Probe    Health     Last change
boot.backend    probe    5/5      healthy    Wed, 13 Jan 2021 10:31:50 GMT


 Current states  good:  5 threshold:  4 window:  5


  Average response time of good probes: 0.000793


  Oldest ================================================== Newest


  4444444444444444444444444444444444444444444444444444444444444444 Good IPv4


  XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX Good Xmit


  RRRRRRRRRRRRRRRRRRRRRRR----RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR Good Recv


  HHHHHHHHHHHHHHHHHHHHHHH--------HHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHH Happy

Starting from the bottom, the last line shows that this backend has been declared "Happy" for most the 64 health probes, but there were some trouble some while ago.

However, in this case the .window is configured to five, and the .threshold is set to four, so at this point in time, the backend is considered fully healthy.

An additional .initial fills that many "happy" entries in the shift register when the VCL is loaded, so that backends can quickly become healthy, even if their health is normally considered over many samples:

.interval = 1s;
.window = 60;
.threshold = 45;
.initial = 43;

This backend will be considered healthy if three out of four health probes in the last minute were good, but it becomes healthy as soon as two good probes have happened after the VCL was loaded.

The default values are:

Note that the default .initial means that the backend will be marked unhealthy until the first probe response come back successful. This means that for backends created on demand (by vmods) cannot use the default value for .initial, as the freshly created backend would very likely still be unhealthy when the backend request happens.