DOKK / manpages / debian 11 / wget2-dev / libwget-robots.3.en
libwget-robots(3) wget2 libwget-robots(3)

libwget-robots - Robots Exclusion file parser


wget_robots_t * wget_robots_parse (const char *data, const char *client)
void wget_robots_free (wget_robots_t **robots)

The purpose of this set of functions is to parse a Robots Exlusion Standard file into a data structure for easy access.

wget_robots_t* wget_robots_parse (const char * data, const char * client)

Parameters

data Memory with robots.txt content (with trailing 0-byte)
client Name of the client / user-agent

Returns

Return an allocated ROBOTS structure or NULL on error

The function parses the robots.txt data and returns a ROBOTS structure including a list of the disallowed paths and including a list of the sitemap files.

The ROBOTS structure has to be freed by calling wget_robots_free().

Parameters

robots Pointer to Pointer to ROBOTS structure

wget_robots_free() free's the formerly allocated ROBOTS structure.

Generated automatically by Doxygen for wget2 from the source code.

Tue Jan 26 2021 Version 1.99.1