| NGHTTPX(1) | nghttp2 | NGHTTPX(1) |
nghttpx - HTTP/2 proxy
nghttpx [OPTIONS]... [<PRIVATE_KEY> <CERT>]
A reverse proxy for HTTP/3, HTTP/2, and HTTP/1.
The options are categorized into several groups.
Optionally, if <PATTERN>s are given, the backend address is only used if request matches the pattern. The pattern matching is closely designed to ServeMux in net/http package of Go programming language. <PATTERN> consists of path, host + path or just host. The path must start with "/". If it ends with "/", it matches all request path in its subtree. To deal with the request to the directory without trailing slash, the path which ends with "/" also matches the request path which only lacks trailing '/' (e.g., path "/foo/" matches request path "/foo"). If it does not end with "/", it performs exact match against the request path. If host is given, it performs a match against the request host. For a request received on the frontend listener with "sni-fwd" parameter enabled, SNI host is used instead of a request host. If host alone is given, "/" is appended to it, so that it matches all request paths under the host (e.g., specifying "nghttp2.org" equals to "nghttp2.org/"). CONNECT method is treated specially. It does not have path, and we don't allow empty path. To workaround this, we assume that CONNECT method has "/" as path.
Patterns with host take precedence over patterns with just path. Then, longer patterns take precedence over shorter ones.
Host can include "*" in the left most position to indicate wildcard match (only suffix match is done). The "*" must match at least one character. For example, host pattern "*.nghttp2.org" matches against "www.nghttp2.org" and "git.ngttp2.org", but does not match against "nghttp2.org". The exact hosts match takes precedence over the wildcard hosts match.
If path part ends with "*", it is treated as wildcard path. The wildcard path behaves differently from the normal path. For normal path, match is made around the boundary of path component separator,"/". On the other hand, the wildcard path does not take into account the path component separator. All paths which include the wildcard path without last "*" as prefix, and are strictly longer than wildcard path without last "*" are matched. "*" must match at least one character. For example, the pattern "/foo*" matches "/foo/" and "/foobar". But it does not match "/foo", or "/fo".
If <PATTERN> is omitted or empty string, "/" is used as pattern, which matches all request paths (catch-all pattern). The catch-all backend must be given.
When doing a match, nghttpx made some normalization to pattern, request host and path. For host part, they are converted to lower case. For path part, percent-encoded unreserved characters defined in RFC 3986 are decoded, and any dot-segments (".." and ".") are resolved and removed.
For example, -b'127.0.0.1,8080;nghttp2.org/httpbin/' matches the request host "nghttp2.org" and the request path "/httpbin/get", but does not match the request host "nghttp2.org" and the request path "/index.html".
The multiple <PATTERN>s can be specified, delimiting them by ":". Specifying -b'127.0.0.1,8080;nghttp2.org:www.nghttp2.org' has the same effect to specify -b'127.0.0.1,8080;nghttp2.org' and -b'127.0.0.1,8080;www.nghttp2.org'.
The backend addresses sharing same <PATTERN> are grouped together forming load balancing group.
Several parameters <PARAM> are accepted after <PATTERN>. The parameters are delimited by ";". The available parameters are: "proto=<PROTO>", "tls", "sni=<SNI_HOST>", "fall=<N>", "rise=<N>", "affinity=<METHOD>", "dns", "redirect-if-not-tls", "upgrade-scheme", "mruby=<PATH>", "read-timeout=<DURATION>", "write-timeout=<DURATION>", "group=<GROUP>", "group-weight=<N>", "weight=<N>", and "dnf". The parameter consists of keyword, and optionally followed by "=" and value. For example, the parameter "proto=h2" consists of the keyword "proto" and value "h2". The parameter "tls" consists of the keyword "tls" without value. Each parameter is described as follows.
The backend application protocol can be specified using optional "proto" parameter, and in the form of "proto=<PROTO>". <PROTO> should be one of the following list without quotes: "h2", "http/1.1". The default value of <PROTO> is "http/1.1". Note that usually "h2" refers to HTTP/2 over TLS. But in this option, it may mean HTTP/2 over cleartext TCP unless "tls" keyword is used (see below).
TLS can be enabled by specifying optional "tls" parameter. TLS is not enabled by default.
With "sni=<SNI_HOST>" parameter, it can override the TLS SNI field value with given <SNI_HOST>. This will default to the backend <HOST> name
The feature to detect whether backend is online or offline can be enabled using optional "fall" and "rise" parameters. Using "fall=<N>" parameter, if nghttpx cannot connect to a this backend <N> times in a row, this backend is assumed to be offline, and it is excluded from load balancing. If <N> is 0, this backend never be excluded from load balancing whatever times nghttpx cannot connect to it, and this is the default. There is also "rise=<N>" parameter. After backend was excluded from load balancing group, nghttpx periodically attempts to make a connection to the failed backend, and if the connection is made successfully <N> times in a row, the backend is assumed to be online, and it is now eligible for load balancing target. If <N> is 0, a backend is permanently offline, once it goes in that state, and this is the default behaviour.
The session affinity is enabled using "affinity=<METHOD>" parameter. If "ip" is given in <METHOD>, client IP based session affinity is enabled. If "cookie" is given in <METHOD>, cookie based session affinity is enabled. If "none" is given in <METHOD>, session affinity is disabled, and this is the default. The session affinity is enabled per <PATTERN>. If at least one backend has "affinity" parameter, and its <METHOD> is not "none", session affinity is enabled for all backend servers sharing the same <PATTERN>. It is advised to set "affinity" parameter to all backend explicitly if session affinity is desired. The session affinity may break if one of the backend gets unreachable, or backend settings are reloaded or replaced by API.
If "affinity=cookie" is used, the additional configuration is required. "affinity-cookie-name=<NAME>" must be used to specify a name of cookie to use. Optionally, "affinity-cookie-path=<PATH>" can be used to specify a path which cookie is applied. The optional "affinity-cookie-secure=<SECURE>" controls the Secure attribute of a cookie. The default value is "auto", and the Secure attribute is determined by a request scheme. If a request scheme is "https", then Secure attribute is set. Otherwise, it is not set. If <SECURE> is "yes", the Secure attribute is always set. If <SECURE> is "no", the Secure attribute is always omitted. "affinity-cookie-stickiness=<STICKINESS>" controls stickiness of this affinity. If <STICKINESS> is "loose", removing or adding a backend server might break the affinity and the request might be forwarded to a different backend server. If <STICKINESS> is "strict", removing the designated backend server breaks affinity, but adding new backend server does not cause breakage. If the designated backend server becomes unavailable, new backend server is chosen as if the request does not have an affinity cookie. <STICKINESS> defaults to "loose".
By default, name resolution of backend host name is done at start up, or reloading configuration. If "dns" parameter is given, name resolution takes place dynamically. This is useful if backend address changes frequently. If "dns" is given, name resolution of backend host name at start up, or reloading configuration is skipped.
If "redirect-if-not-tls" parameter is used, the matched backend requires that frontend connection is TLS encrypted. If it isn't, nghttpx responds to the request with 308 status code, and https URI the client should use instead is included in Location header field. The port number in redirect URI is 443 by default, and can be changed using --redirect-https-port option. If at least one backend has "redirect-if-not-tls" parameter, this feature is enabled for all backend servers sharing the same <PATTERN>. It is advised to set "redirect-if-no-tls" parameter to all backends explicitly if this feature is desired.
If "upgrade-scheme" parameter is used along with "tls" parameter, HTTP/2 :scheme pseudo header field is changed to "https" from "http" when forwarding a request to this particular backend. This is a workaround for a backend server which requires "https" :scheme pseudo header field on TLS encrypted connection.
"mruby=<PATH>" parameter specifies a path to mruby script file which is invoked when this pattern is matched. All backends which share the same pattern must have the same mruby path.
"read-timeout=<DURATION>" and "write-timeout=<DURATION>" parameters specify the read and write timeout of the backend connection when this pattern is matched. All backends which share the same pattern must have the same timeouts. If these timeouts are entirely omitted for a pattern, --backend-read-timeout and --backend-write-timeout are used.
"group=<GROUP>" parameter specifies the name of group this backend address belongs to. By default, it belongs to the unnamed default group. The name of group is unique per pattern. "group-weight=<N>" parameter specifies the weight of the group. The higher weight gets more frequently selected by the load balancing algorithm. <N> must be [1, 256] inclusive. The weight 8 has 4 times more weight than 2. <N> must be the same for all addresses which share the same <GROUP>. If "group-weight" is omitted in an address, but the other address which belongs to the same group specifies "group-weight", its weight is used. If no "group-weight" is specified for all addresses, the weight of a group becomes 1. "group" and "group-weight" are ignored if session affinity is enabled.
"weight=<N>" parameter specifies the weight of the backend address inside a group which this address belongs to. The higher weight gets more frequently selected by the load balancing algorithm. <N> must be [1, 256] inclusive. The weight 8 has 4 times more weight than weight 2. If this parameter is omitted, weight becomes 1. "weight" is ignored if session affinity is enabled.
If "dnf" parameter is specified, an incoming request is not forwarded to a backend and just consumed along with the request body (actually a backend server never be contacted). It is expected that the HTTP response is generated by mruby script (see "mruby=<PATH>" parameter above). "dnf" is an abbreviation of "do not forward".
Since ";" and ":" are used as delimiter, <PATTERN> must not contain these characters. In order to include ":" in <PATTERN>, one has to specify "%3A" (which is percent-encoded from of ":") instead. Since ";" has special meaning in shell, the option value must be quoted.
Default: 127.0.0.1,80
This option can take 0 or more parameters, which are described below. Note that "api" and "healthmon" parameters are mutually exclusive.
Optionally, TLS can be disabled by specifying "no-tls" parameter. TLS is enabled by default.
If "sni-fwd" parameter is used, when performing a match to select a backend server, SNI host name received from the client is used instead of the request host. See --backend option about the pattern match.
To make this frontend as API endpoint, specify "api" parameter. This is disabled by default. It is important to limit the access to the API frontend. Otherwise, someone may change the backend server, and break your services, or expose confidential information to the outside the world.
To make this frontend as health monitor endpoint, specify "healthmon" parameter. This is disabled by default. Any requests which come through this address are replied with 200 HTTP status, without no body.
To accept PROXY protocol version 1 and 2 on frontend connection, specify "proxyproto" parameter. This is disabled by default.
To receive HTTP/3 (QUIC) traffic, specify "quic" parameter. It makes nghttpx listen on UDP port rather than TCP port. UNIX domain socket, "api", and "healthmon" parameters cannot be used with "quic" parameter.
Default: *,3000
Default: 65536
Default: auto
Default: 1
Default: 0
Default: 0
Default: 0
Default: 0
Default: 0
Default: 0
Default: 0
Default: 0
Default: 0
Default: 8
Default: 0
Default: 0
Default: 0
Default: 16K
Default: 128K
Default: 0
Default: 3m
Default: 3m
Default: 1m
Default: 30s
Default: 1m
Default: 0
Default: 1m
Default: 1m
Default: 30s
Default: 30s
Default: 2s
Default: 30s
Default: 10s
Default: 10s
Default: 2m
Default: ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384
Default: TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256
Default: ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384
Default: TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256
Default: X25519:P-256:P-384:P-521
Additional parameter can be specified in <PARAM>. The available <PARAM> is "sct-dir=<DIR>".
"sct-dir=<DIR>" specifies the path to directory which contains *.sct files for TLS signed_certificate_timestamp extension (RFC 6962). This feature requires OpenSSL >= 1.0.2. See also --tls-sct-dir option.
Default: h2,h2-16,h2-14,http/1.1
Default: TLSv1.2
Default: TLSv1.3
Default: auto
Default: 10m
Default: 3
Default: 2
Default: /usr/local/share/nghttp2/fetch-ocsp-response
Default: 4h
Default: auto
Default: 1M
Default: 1s
Default: 16K
Default: 100
Default: 100
Default: 65535
Default: 65535
Default: 65535
Default: 2147483647
Default: 4K
Default: 4K
Default: 4K
Default: 4K
Default: NOTICE
The variable can be enclosed by "{" and "}" for disambiguation (e.g., ${remote_addr}).
Default: $remote_addr - - [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent"
Default: /dev/stderr
Default: daemon
Default: obfuscated
Default: obfuscated
Default: 64K
Default: 100
Default: 64K
Default: 500
Default: nghttpx
Default: 443
Default: 32M
Default: 10s
Default: 5s
Default: 2
Default: 30s
Default: /usr/local/lib/nghttp2/reuseport_kern.o
Default: cubic
Default: 333ms
Default: 256K
Default: 1M
Default: 6M
Default: 8M
Default: 100
Default: /etc/nghttpx/nghttpx.conf
The <SIZE> argument is an integer and an optional unit (e.g., 10K is 10 * 1024). Units are K, M and G (powers of 1024).
The <DURATION> argument is an integer and an optional unit (e.g., 1s is 1 second and 500ms is 500 milliseconds). Units are h, m, s or ms (hours, minutes, seconds and milliseconds, respectively). If a unit is omitted, a second is used as unit.
Those lines which are staring # are treated as comment.
The option name in the configuration file is the long command-line option name with leading -- stripped (e.g., frontend). Put = between option name and value. Don't put extra leading or trailing spaces.
When specifying arguments including characters which have special meaning to a shell, we usually use quotes so that shell does not interpret them. When writing this configuration file, quotes for this purpose must not be used. For example, specify additional request header field, do this:
add-request-header=foo: bar
instead of:
add-request-header="foo: bar"
The options which do not take argument in the command-line take argument in the configuration file. Specify yes as an argument (e.g., http2-proxy=yes). If other string is given, it is ignored.
To specify private key and certificate file which are given as positional arguments in command-line, use private-key-file and certificate-file.
--conf option cannot be used in the configuration file and will be ignored if specified.
<datetime> <main-pid> <current-pid> <thread-id> <level> (<filename>:<line>) <msg>
SIGUSR2
The difference between SIGUSR2 (+ SIGQUIT) and SIGHUP is that former is usually used to execute new binary, and the main process is newly spawned. On the other hand, the latter just reloads configuration file, and the same main process continues to exist.
NOTE:
nghttpx supports HTTP/2 server push in default mode with Link header field. nghttpx looks for Link header field (RFC 5988) in response headers from backend server and extracts URI-reference with parameter rel=preload (see preload) and pushes those URIs to the frontend client. Here is a sample Link header field to initiate server push:
Link: </fonts/font.woff>; rel=preload Link: </css/theme.css>; rel=preload
Currently, the following restriction is applied for server push:
This limitation may be loosened in the future release.
nghttpx also supports server push if both frontend and backend are HTTP/2 in default mode. In this case, in addition to server push via Link header field, server push from backend is forwarded to frontend HTTP/2 session.
HTTP/2 server push will be disabled if --http2-proxy is used.
nghttpx supports UNIX domain socket with a filename for both frontend and backend connections.
Please note that current nghttpx implementation does not delete a socket with a filename. And on start up, if nghttpx detects that the specified socket already exists in the file system, nghttpx first deletes it. However, if SIGUSR2 is used to execute new binary and both old and new configurations use same filename, new binary does not delete the socket and continues to use it.
OCSP query is done using external Python script fetch-ocsp-response, which has been originally developed in Perl as part of h2o project (https://github.com/h2o/h2o), and was translated into Python.
The script file is usually installed under $(prefix)/share/nghttp2/ directory. The actual path to script can be customized using --fetch-ocsp-response-file option.
If OCSP query is failed, previous OCSP response, if any, is continued to be used.
--fetch-ocsp-response-file option provides wide range of possibility to manage OCSP response. It can take an arbitrary script or executable. The requirement is that it supports the command-line interface of fetch-ocsp-response script, and it must return a valid DER encoded OCSP response on success. It must return exit code 0 on success, and 75 for temporary error, and the other error code for generic failure. For large cluster of servers, it is not efficient for each server to perform OCSP query using fetch-ocsp-response. Instead, you can retrieve OCSP response in some way, and store it in a disk or a shared database. Then specify a program in --fetch-ocsp-response-file to fetch it from those stores. This could provide a way to share the OCSP response between fleet of servers, and also any OCSP query strategy can be applied which may be beyond the ability of nghttpx itself or fetch-ocsp-response script.
nghttpx supports TLS session resumption through both session ID and session ticket.
By default, session ID is shared by all worker threads.
If --tls-session-cache-memcached is given, nghttpx will insert serialized session data to memcached with nghttpx:tls-session-cache: + lowercase hex string of session ID as a memcached entry key, with expiry time 12 hours. Session timeout is set to 12 hours.
By default, connections to memcached server are not encrypted. To enable encryption, use tls keyword in --tls-session-cache-memcached option.
By default, session ticket is shared by all worker threads. The automatic key rotation is also enabled by default. Every an hour, new encryption key is generated, and previous encryption key becomes decryption only key. We set session timeout to 12 hours, and thus we keep at most 12 keys.
If --tls-ticket-key-memcached is given, encryption keys are retrieved from memcached. nghttpx just reads keys from memcached; one has to deploy key generator program to update keys frequently (e.g., every 1 hour). The example key generator tlsticketupdate.go is available under contrib directory in nghttp2 archive. The memcached entry key is nghttpx:tls-ticket-key. The data format stored in memcached is the binary format described below:
+--------------+-------+----------------+ | VERSION (4) |LEN (2)|KEY(48 or 80) ... +--------------+-------+----------------+
^ |
| |
+------------------------+
(LEN, KEY) pair can be repeated
All numbers in the above figure is bytes. All integer fields are network byte order.
First 4 bytes integer VERSION field, which must be 1. The 2 bytes integer LEN field gives the length of following KEY field, which contains key. If --tls-ticket-key-cipher=aes-128-cbc is used, LEN must be 48. If --tls-ticket-key-cipher=aes-256-cbc is used, LEN must be 80. LEN and KEY pair can be repeated multiple times to store multiple keys. The key appeared first is used as encryption key. All the remaining keys are used as decryption only.
By default, connections to memcached server are not encrypted. To enable encryption, use tls keyword in --tls-ticket-key-memcached option.
If --tls-ticket-key-file is given, encryption key is read from the given file. In this case, nghttpx does not rotate key automatically. To rotate key, one has to restart nghttpx (see SIGNALS).
nghttpx supports TLS signed_certificate_timestamp extension (RFC 6962). The relevant options are --tls-sct-dir and sct-dir parameter in --subcert. They takes a directory, and nghttpx reads all files whose extension is .sct under the directory. The *.sct files are encoded as SignedCertificateTimestamp struct described in section 3.2 of RFC 69662. This format is the same one used by nginx-ct and mod_ssl_ct. ct-submit can be used to submit certificates to log servers, and obtain the SignedCertificateTimestamp struct which can be used with nghttpx.
WARNING:
WARNING:
nghttpx allows users to extend its capability using mruby scripts. nghttpx has 2 hook points to execute mruby script: request phase and response phase. The request phase hook is invoked after all request header fields are received from client. The response phase hook is invoked after all response header fields are received from backend server. These hooks allows users to modify header fields, or common HTTP variables, like authority or request path, and even return custom response without forwarding request to backend servers.
There are 2 levels of mruby script invocations: global and per-pattern. The global mruby script is set by --mruby-file option and is called for all requests. The per-pattern mruby script is set by "mruby" parameter in -b option. It is invoked for a request which matches the particular pattern. The order of hook invocation is: global request phase hook, per-pattern request phase hook, per-pattern response phase hook, and finally global response phase hook. If a hook returns a response, any later hooks are not invoked. The global request hook is invoked before the pattern matching is made and changing request path may affect the pattern matching.
Please note that request and response hooks of per-pattern mruby script for a single request might not come from the same script. This might happen after a request hook is executed, backend failed for some reason, and at the same time, backend configuration is replaced by API request, and then the request uses new configuration on retry. The response hook from new configuration, if it is specified, will be invoked.
The all mruby script will be evaluated once per thread on startup, and it must instantiate object and evaluate it as the return value (e.g., App.new). This object is called app object. If app object defines on_req method, it is called with Nghttpx::Env object on request hook. Similarly, if app object defines on_resp method, it is called with Nghttpx::Env object on response hook. For each method invocation, user can can access Nghttpx::Request and Nghttpx::Response objects via Nghttpx::Env#req and Nghttpx::Env#resp respectively.
Modify request path:
class App
def on_req(env)
env.req.path = "/apps#{env.req.path}"
end end App.new
Don't forget to instantiate and evaluate object at the last line.
Restrict permission of viewing a content to a specific client addresses:
class App
def on_req(env)
allowed_clients = ["127.0.0.1", "::1"]
if env.req.path.start_with?("/log/") &&
!allowed_clients.include?(env.remote_addr) then
env.resp.status = 404
env.resp.return "permission denied"
end
end end App.new
nghttpx exposes API endpoints to manipulate it via HTTP based API. By default, API endpoint is disabled. To enable it, add a dedicated frontend for API using --frontend option with "api" parameter. All requests which come from this frontend address, will be treated as API request.
The response is normally JSON dictionary, and at least includes the following keys:
Additionally, depending on the API endpoint, data key may be present, and its value contains the API endpoint specific data.
We wrote "normally", since nghttpx may return ordinal HTML response in some cases where the error has occurred before reaching API endpoint (e.g., header field is too large).
The following section describes available API endpoints.
This API replaces the current backend server settings with the requested ones. The request method should be POST, but PUT is also acceptable. The request body must be nghttpx configuration file format. For configuration file format, see FILES section. The line separator inside the request body must be single LF (0x0A). Currently, only backend option is parsed, the others are simply ignored. The semantics of this API is replace the current backend with the backend options in request body. Describe the desired set of backend severs, and nghttpx makes it happen. If there is no backend option is found in request body, the current set of backend is replaced with the backend option's default value, which is 127.0.0.1,80.
The replacement is done instantly without breaking existing connections or requests. It also avoids any process creation as is the case with hot swapping with signals.
The one limitation is that only numeric IP address is allowed in backend in request body unless "dns" parameter is used while non numeric hostname is allowed in command-line or configuration file is read using --conf.
This API returns configuration revision of the current nghttpx. The configuration revision is opaque string, and it changes after each reloading by SIGHUP. With this API, an external application knows that whether nghttpx has finished reloading its configuration by comparing the configuration revisions between before and after reloading. It is recommended to disable persistent (keep-alive) connection for this purpose in order to avoid to send a request using the reused connection which may bound to an old process.
This API returns response including data key. Its value is JSON object, and it contains at least the following key:
Tatsuhiro Tsujikawa
2012, 2015, 2016, Tatsuhiro Tsujikawa
| February 13, 2023 | 1.52.0 |