libwget-robots(3) | wget2 | libwget-robots(3) |
libwget-robots
wget_robots_t * wget_robots_parse (const char *data,
const char *client)
void wget_robots_free (wget_robots_t **robots)
The purpose of this set of functions is to parse a Robots Exlusion Standard file into a data structure for easy access.
Parameters:
Returns:
The function parses the robots.txt data and returns a ROBOTS structure including a list of the disallowed paths and including a list of the sitemap files.
The ROBOTS structure has to be freed by calling wget_robots_free().
Parameters:
wget_robots_free() free's the formerly allocated ROBOTS structure.
Generated automatically by Doxygen for wget2 from the source code.
Thu May 17 2018 | Version 1.99.1 |