NAME

http - Client-side implementation of the HTTP/1.1 protocol

SYNOPSIS

package require http ?2.10?
::http::config ?-option value ...?
::http::geturl url ?-option value ...?
::http::formatQuery key value ?key value ...?
::http::quoteString value
::http::reset token ?why?
::http::wait token
::http::status token
::http::size token
::http::error token
::http::postError token
::http::cleanup token
::http::requestLine token
::http::requestHeaders token ?headerName?
::http::requestHeaderValue token headerName
::http::responseLine token
::http::responseCode token
::http::reasonPhrase code
::http::responseHeaders token ?headerName?
::http::responseHeaderValue token headerName
::http::responseInfo token
::http::responseBody token
::http::register proto port command ?socketCmdVarName? ?useSockThread? ?endToEndProxy?
::http::registerError sock ?message?
::http::unregister proto
::http::code token
::http::data token
::http::meta token ?headerName?
::http::metaValue token headerName
::http::ncode token

EXPORTED COMMANDS

Namespace http exports the commands config, formatQuery, geturl, postError, quoteString, reasonPhrase, register, registerError, requestHeaders, requestHeaderValue, requestLine, responseBody, responseCode, responseHeaders, responseHeaderValue, responseInfo, responseLine, reset, unregister, and wait.

It does not export the commands cleanup, code, data, error, meta, metaValue, ncode, size, or status.

DESCRIPTION

The http package provides the client side of the HTTP/1.1 protocol, as defined in RFC 9110 to 9112, which supersede RFC 7230 to RFC 7235, which in turn supersede RFC 2616. The package implements the GET, POST, and HEAD operations of HTTP/1.1. It allows configuration of a proxy host to get through firewalls. The package is compatible with the Safesock security policy, so it can be used by untrusted applets to do URL fetching from a restricted set of hosts. This package can be extended to support additional HTTP transport protocols, such as HTTPS, by providing a custom socket command, via ::http::register.

The ::http::geturl procedure does a HTTP transaction. Its options determine whether a GET, POST, or HEAD transaction is performed. The return value of ::http::geturl is a token for the transaction. The token can be supplied as an argument to other commands, to manage the transaction and examine its results.

If the -command option is specified, then the HTTP operation is done in the background. ::http::geturl returns immediately after generating the HTTP request and the -command callback is invoked when the transaction completes. For this to work, the Tcl event loop must be active. In Tk applications this is always true. For pure-Tcl applications, the caller can use ::http::wait after calling ::http::geturl to start the event loop.

Note: The event queue is even used without the -command option. As a side effect, arbitrary commands may be processed while http::geturl is running.

When the HTTP server has replied to the request, call the command ::http::responseInfo, which returns a dict of metadata that is essential for identifying a successful transaction and making use of the response. See section METADATA for details of the information returned. The response itself is returned by command ::http::responseBody, unless it has been redirected to a file by the -channel option of ::http::geturl.

COMMANDS

::http::config ?options?

The ::http::config command is used to set and query the name of the proxy server and port, and the User-Agent name used in the HTTP requests. If no options are specified, then the current configuration is returned. If a single argument is specified, then it should be one of the flags described below. In this case the current value of that setting is returned. Otherwise, the options should be a set of flags and values that define the configuration:

::http::geturl url ?options?

The ::http::geturl command is the main procedure in the package. The -query or -querychannel option causes a POST operation and the -validate option causes a HEAD operation; otherwise, a GET operation is performed. The ::http::geturl command returns a token value that can be passed as an argument to other commands to get information about the transaction. See the METADATA and ERRORS section for details. The ::http::geturl command blocks until the operation completes, unless the -command option specifies a callback that is invoked when the HTTP transaction completes. ::http::geturl takes several options:

proc httpCallback {token} {


    upvar 0 $token state


    # Access state as a Tcl array defined in this proc


    ...


    return
}

proc httpHandlerCallback {socket token} {


    upvar 0 $token state


    # Access socket, and state as a Tcl array defined in this proc


    # For example...


    ...


    set data [read $socket 1000]


    set nbytes [string length $data]


    ...


    return $nbytes
}

Pragma: no-cache

-headers {Content-Length 0}

proc httpProgress {token total current} {


    upvar 0 $token state


    # Access state as a Tcl array defined in this proc


    ...


    return
}

::http::formatQuery key value ?key value ...?

This procedure does x-url-encoding of query data. It takes an even number of arguments that are the keys and values of the query. It encodes the keys and values, and generates one string that has the proper & and = separators. The result is suitable for the -query value passed to ::http::geturl.

::http::quoteString value

This procedure does x-url-encoding of string. It takes a single argument and encodes it.

::http::reset token ?why?

This command resets the HTTP transaction identified by token, if any. This sets the state(status) value to why, which defaults to reset, and then calls the registered -command callback.

::http::wait token

This command blocks and waits for the transaction to complete. This only works in trusted code because it uses vwait. Also, it is not useful for the case where ::http::geturl is called without the -command option because in this case the ::http::geturl call does not return until the HTTP transaction is complete, and thus there is nothing to wait for.

::http::status token

This command returns a description of the status of the HTTP transaction. The return value is the empty string until the HTTP transaction is completed; after completion it has one of the values ok, eof, error, timeout, and reset. The meaning of these values is described in the section ERRORS (below).

::http::size token

This command returns the number of bytes received so far from the URL in the ::http::geturl call.

::http::error token

This command returns the error information if the HTTP transaction failed, or the empty string if there was no error. The information is a Tcl list of the error message, stack trace, and error code.

::http::postError token

A POST request is a call to ::http::geturl with either the -query or -querychannel option. The ::http::postError command returns the error information generated when a HTTP POST request sends its request-body to the server; or the empty string if there was no error. The information is a Tcl list of the error message, stack trace, and error code. When this type of error occurs, the ::http::geturl command continues the transaction and attempts to receive a response from the server.

::http::cleanup token

This procedure cleans up the state associated with the connection identified by token. After this call, the procedures like ::http::responseBody cannot be used to get information about the operation. It is strongly recommended that you call this function after you are done with a given HTTP request. Not doing so will result in memory not being freed, and if your app calls ::http::geturl enough times, the memory leak could cause a performance hit...or worse.

::http::requestLine token

This command returns the "request line" sent to the server. The "request line" is the first line of a HTTP client request, and has three elements separated by spaces: the HTTP method, the URL relative to the server, and the HTTP version. Examples:

::http::requestHeaders token ?headerName?

This command returns the HTTP request header names and values, in the order that they were sent to the server, as a Tcl list of the form ?name value ...? Header names are case-insensitive and are converted to lower case. The return value is not a dict because some header names may occur more than once. If one argument is supplied, all request headers are returned. If two arguments are supplied, the second provides the value of a header name. Only headers with the requested name (converted to lower case) are returned. If no such headers are found, an empty list is returned.

::http::requestHeaderValue token headerName

This command returns the value of the HTTP request header named headerName. Header names are case-insensitive and are converted to lower case. If no such header exists, the return value is the empty string. If there are multiple headers named headerName, the result is obtained by joining the individual values with the string ", " (comma and space), preserving their order.

::http::responseLine token

This command returns the first line of the server response: the HTTP "status line". The "status line" has three elements separated by spaces: the HTTP version, a three-digit numerical "status code", and a "reason phrase". Only the reason phrase may contain spaces. Examples:

::http::responseCode token

This command returns the "status code" (200, 404, etc.) of the server "status line". If a three-digit code cannot be found, the full status line is returned. See command ::http::responseLine for more information on the "status line".

::http::reasonPhrase code

This command returns the IANA recommended "reason phrase" for a particular "status code" returned by a HTTP server. The argument code is a valid status code, and therefore is an integer in the range 100 to 599 inclusive. For numbers in this range with no assigned meaning, the command returns the value "Unassigned". Several status codes are used only in response to the methods defined by HTTP extensions such as WebDAV, and not in response to a HEAD, GET, or POST request method.

::http::responseHeaders token ?headerName?

The response from a HTTP server includes metadata headers that describe the response body and the transaction itself. This command returns the HTTP response header names and values, in the order that they were received from the server, as a Tcl list of the form ?name value ...? Header names are case-insensitive and are converted to lower case. The return value is not a dict because some header names may occur more than once, notably Set-Cookie. If the second argument is not supplied, all response headers are returned. If the second argument is supplied, it provides the value of a header name. Only headers with the requested name (converted to lower case) are returned. If no such headers are found, an empty list is returned. See section METADATA for more information.

::http::responseHeaderValue token headerName

This command returns the value of the HTTP response header named headerName. Header names are case-insensitive and are converted to lower case. If no such header exists, the return value is the empty string. If there are multiple headers named headerName, the result is obtained by joining the individual values with the string ", " (comma and space), preserving their order. Multiple headers with the same name may be processed in this manner, except Set-Cookie which does not conform to the comma-separated-list syntax and cannot be combined into a single value. Each Set-Cookie header must be treated individually, e.g. by processing the return value of ::http::responseHeaders token Set-Cookie.

::http::responseInfo token

This command returns a dict of selected response metadata that are essential for identifying a successful transaction and making use of the response, along with other metadata that are informational. The keys of the dict are stage, status, responseCode, reasonPhrase, contentType, binary, redirection, upgrade, error, postError, method, charset, compression, httpRequest, httpResponse, url, connectionRequest, connectionResponse, connectionActual, transferEncoding, totalPost, currentPost, totalSize, and currentSize. The meaning of these keys is described in the section METADATA below.

::http::responseBody token

This command returns the entity sent by the HTTP server (unless -channel was used, in which case the entity was delivered to the channel, and the command returns the empty string).

::http::register proto port command ?socketCmdVarName? ?useSockThread? ?endToEndProxy?

This procedure allows one to provide custom HTTP transport types such as HTTPS, by registering a prefix, the default port, and the command to execute to create the Tcl channel. The optional arguments configure how http uses the custom transport, and have default values that are compatible with older versions of http in which ::http::register has no optional arguments.

package require http
package require tls
::http::register https 443 ::tls::socket ::tls::socketCmd 1 1
set token [::http::geturl https://my.secure.site/]

::http::registerError sock ?message?

This procedure allows a registered protocol handler to deliver an error message for use by http. Calling this command does not raise an error. The command is useful when a registered protocol detects an problem (for example, an invalid TLS certificate) that will cause an error to propagate to http. The command allows http to provide a precise error message rather than a general one. The command returns the value provided by the last call with argument message, or the empty string if no such call has been made.

::http::unregister proto

This procedure unregisters a protocol handler that was previously registered via ::http::register, returning a six-item list of the values that were previously supplied to ::http::register if there was such a handler, and an error if there was no such handler.

::http::code token

An alternative name for the command ::http::responseLine

::http::data token

An alternative name for the command ::http::responseBody.

::http::meta token ?headerName?

An alternative name for the command ::http::responseHeaders

::http::ncode token

An alternative name for the command ::http::responseCode

ERRORS

The ::http::geturl procedure will raise errors in the following cases: invalid command line options, or an invalid URL. These errors mean that it cannot even start the network transaction. For synchronous ::http::geturl calls (where -command is not specified), it will raise an error if the URL is on a non-existent host or at a bad port on an existing host. It will also raise an error for any I/O errors while writing out the HTTP request line and headers, or reading the HTTP reply headers or data. Because ::http::geturl does not return a token in these cases, it does all the required cleanup and there is no issue of your app having to call ::http::cleanup.

For asynchronous ::http::geturl calls, all of the above error situations apply, except that if there is any error while reading the HTTP reply headers or data, no exception is thrown. This is because after writing the HTTP headers, ::http::geturl returns, and the rest of the HTTP transaction occurs in the background. The command callback can check if any error occurred during the read by calling ::http::responseInfo to check the transaction status.

Alternatively, if the main program flow reaches a point where it needs to know the result of the asynchronous HTTP request, it can call ::http::wait and then check status and error, just as the synchronous call does.

The ::http::geturl command runs the -command, -handler, and -proxyfilter callbacks inside a catch command. Therefore an error in the callback command does not call the bgerror handler. When debugging one of these callbacks, it may be convenient to report errors by using a catch command within the callback command itself, e.g. to write an error message to stdout.

In any case, you must still call ::http::cleanup to delete the state array when you are done.

There are other possible results of the HTTP transaction determined by examining the status from ::http::status (or the value of the status key in the dictionary returned by ::http::responseInfo). These are described below.

ok

If the HTTP transaction completes entirely, then status will be ok. However, you should still check the ::http::responseLine value to get the HTTP status. The ::http::responseCode procedure provides just the numeric error (e.g., 200, 404 or 500) while the ::http::responseLine procedure returns a value like “HTTP 404 File not found”.

eof

If the server closes the socket without replying, then no error is raised, but the status of the transaction will be eof.

error

The error message, stack trace, and error code are accessible via ::http::error. The error message is also provided by the value of the error key in the dictionary returned by ::http::responseInfo.

timeout

A timeout occurred before the transaction could complete.

reset

The user has called ::http::reset.

""

(empty string) The transaction has not yet finished.

Another error possibility is that ::http::geturl failed to write the whole of the POST request body (-query or -querychannel data) to the server. ::http::geturl stores the error message for later retrieval by the ::http::postError or ::http::responseInfo commands, and then attempts to complete the transaction. If it can read the server's response the status will be ok, but it is important to call ::http::postError or ::http::responseInfo after every POST to check that the data was sent in full. If the server has closed the connection the status will be eof.

METADATA

STATE ARRAY

The ::http::geturl procedure returns a token that can be used as an argument to other ::http::* commands, which examine and manage the state of the HTTP transaction. For most purposes these commands are sufficient. The token can also be used to access the internal state of the transaction, which is stored in a Tcl array. This facility is most useful when writing callback commands for the options -command, -handler, -progress, or -queryprogress. Use the following command inside the proc to define an easy-to-use array state as a local variable within the proc

upvar 0 $token state

Once the data associated with the URL is no longer needed, the state array should be unset to free up storage. The ::http::cleanup procedure is provided for that purpose.

The following elements of the array are supported, and are the origin of the values returned by commands as described below. When a dictionary key is mentioned below, this refers to the dict value returned by command ::http::responseInfo.

PERSISTENT CONNECTIONS

COOKIE JAR PROTOCOL

Cookies are short key-value pairs used to implement sessions within the otherwise-stateless HTTP protocol. (See RFC 6265 for details; Tcl does not implement the Cookie2 protocol as that is rarely seen in the wild.)

Cookie storage management commands — “cookie jars” — must support these subcommands which form the HTTP cookie storage management protocol. Note that cookieJar below does not have to be a command name; it is properly a command prefix (a Tcl list of words that will be expanded in place) and admits many possible implementations.

Though not formally part of the protocol, it is expected that particular values of cookieJar will correspond to sessions; it is up to the caller of ::http::config to decide what session applies and to manage the deletion of said sessions when they are no longer desired (which should be when they not configured as the current cookie jar).

cookieJar getCookies protocol host requestPath

This command asks the cookie jar what cookies should be supplied for a particular request. It should take the protocol (typically http or https), host name and requestPath (parsed from the url argument to ::http::geturl) and return a list of cookie keys and values that describe the cookies to supply to the remote host. The list must have an even number of elements.

cookieJar storeCookie cookieDictionary

This command asks the cookie jar to store a particular cookie that was returned by a request; the result of this command is ignored. The cookie (which will have been parsed by the http package) is described by a dictionary, cookieDictionary, that may have the following keys:

PROTOCOL UPGRADES

The HTTP/1.1 Connection and Upgrade request headers inform the server that the client wishes to change the protocol used over the existing connection (RFC 7230). This mechanism can be used to request a WebSocket (RFC 6455), a higher version of the HTTP protocol (HTTP 2), or TLS encryption. If the server accepts the upgrade request, its response code will be 101.

To request a protocol upgrade when calling http::geturl, the -headers option must supply appropriate values for Connection and Upgrade, and the -command option must supply a command that implements the requested protocol and can also handle the server response if the server refuses the protocol upgrade. For upgrade requests http::geturl ignores the value of option -keepalive, and always uses the value 0 so that the upgrade request is not made over a connection that is intended for multiple HTTP requests.

The Tcllib library websocket implements WebSockets, and makes the necessary calls to commands in the http package.

There is currently no native Tcl client library for HTTP/2.

The Upgrade mechanism is not used to request TLS in web browsers, because http and https are served over different ports. It is used by protocols such as Internet Printing Protocol (IPP) that are built on top of http(s) and use the same TCP port number for both secure and insecure traffic.

In browsers, opportunistic encryption is instead implemented by the Upgrade-Insecure-Requests client header. If a secure service is available, the server response code is a 307 redirect, and the response header Location specifies the target URL. The browser must call http::geturl again in order to fetch this URL. See https://w3c.github.io/webappsec-upgrade-insecure-requests/

THREADS

EXAMPLE

This example creates a procedure to copy a URL to a file while printing a progress meter, and prints the response headers associated with the URL.

proc httpcopy { url file {chunk 4096} } {


    set out [open $file w]


    set token [::http::geturl $url -channel $out \


            -progress httpCopyProgress -blocksize $chunk]


    close $out


    # This ends the line started by httpCopyProgress


    puts stderr ""


    upvar 0 $token state


    set max 0


    foreach {name value} $state(meta) {


        if {[string length $name] > $max} {


            set max [string length $name]


        }


        if {[regexp -nocase ^location$ $name]} {


            # Handle URL redirects


            puts stderr "Location:$value"


            return [httpcopy [string trim $value] $file $chunk]


        }


    }


    incr max


    foreach {name value} $state(meta) {


        puts [format "%-*s %s" $max $name: $value]


    }


    return $token
}
proc httpCopyProgress {args} {


    puts -nonewline stderr .


    flush stderr
}

SEE ALSO

safe(3tcl), socket(3tcl), safesock(3tcl)

KEYWORDS

internet, security policy, socket, www