| HTTP::OAI::Harvester(3pm) | User Contributed Perl Documentation | HTTP::OAI::Harvester(3pm) |
HTTP::OAI::Harvester - Agent for harvesting from Open Archives version 1.0, 1.1, 2.0 and static ('2.0s') compatible repositories
"HTTP::OAI::Harvester" is the harvesting front-end in the OAI-PERL library.
To harvest from an OAI-PMH compliant repository create an "HTTP::OAI::Harvester" object using the baseURL option and then call OAI-PMH methods to request data from the repository. To handle version 1.0/1.1 repositories automatically you must request "Identify()" first.
It is recommended that you request an Identify from the Repository and use the "repository()" method to update the Identify object used by the harvester.
When making OAI requests the underlying HTTP::OAI::UserAgent module will take care of automatic redirection (http code 302) and retry-after (http code 503). OAI-PMH flow control (i.e. resumption tokens) is handled transparently by "HTTP::OAI::Response".
Static repositories are automatically and transparently supported within the existing API. To harvest a static repository specify the repository XML file using the baseURL argument to HTTP::OAI::Harvester. An initial request is made that determines whether the base URL specifies a static repository or a normal OAI 1.x/2.0 CGI repository. To prevent this initial request state the OAI version using an HTTP::OAI::Identify object e.g.
$h = HTTP::OAI::Harvester->new(
repository=>HTTP::OAI::Identify->new(
baseURL => 'http://arXiv.org/oai2',
version => '2.0',
));
If a static repository is found the response is cached, and further requests are served by that cache. Static repositories do not support sets, and will result in a noSetHierarchy error if you try to use sets. You can determine whether the repository is static by checking the version ($ha->repository->version), which will be "2.0s" for static repositories.
You should refer to the Open Archives Protocol version 2.0 and other OAI documentation, available from http://www.openarchives.org/.
Note OAI-PMH 1.0 and 1.1 are deprecated.
In the examples I use arXiv.org's and cogprints OAI interfaces. To avoid causing annoyance to their server administrators please contact them before performing testing or large downloads (or use other, less loaded, servers for testing).
use HTTP::OAI;
my $h = new HTTP::OAI::Harvester(baseURL=>'http://arXiv.org/oai2');
my $response = $h->Identify;
if( $response->is_error ) {
print "Error requesting Identify:\n",
$response->code . " " . $response->message, "\n";
exit;
}
# Note: repositoryVersion will always be 2.0, $r->version returns
# the actual version the repository is running
print "Repository supports protocol version ", $response->version, "\n";
# Version 1.x repositories don't support metadataPrefix,
# but OAI-PERL will drop the prefix automatically
# if an Identify was requested first (as above)
$response = $h->ListIdentifiers(
metadataPrefix=>'oai_dc',
from=>'2001-02-03',
until=>'2001-04-10'
);
if( $response->is_error ) {
die("Error harvesting: " . $response->message . "\n");
}
print "responseDate => ", $response->responseDate, "\n",
"requestURL => ", $response->requestURL, "\n";
while( my $id = $response->next ) {
print "identifier => ", $id->identifier;
# Only available from OAI 2.0 repositories
print " (", $id->datestamp, ")" if $id->datestamp;
print " (", $id->status, ")" if $id->status;
print "\n";
# Only available from OAI 2.0 repositories
for( $id->setSpec ) {
print "\t", $_, "\n";
}
}
# Using a handler
$response = $h->ListRecords(
metadataPrefix=>'oai_dc',
handlers=>{metadata=>'HTTP::OAI::Metadata::OAI_DC'},
onRecord=>sub {
my $rec = shift;
printf"%s\t%s\t%s\n"
, $rec->identifier
, $rec->datestamp
, join(',', @{$rec->metadata->dc->{'title'}});
}
);
# Offline parsing
$I = HTTP::OAI::Identify->new();
$I->parse_string($content);
$I->parse_file($fh);
Any other parameters are passed to the HTTP::OAI::UserAgent module, and from there to the LWP::UserAgent module.
$h = HTTP::OAI::Harvester->new(
baseURL => 'http://arXiv.org/oai2',
resume=>0, # Suppress automatic resumption
)
$id = $h->repository();
$h->repository($h->Identify);
$h = HTTP::OAI::Harvester->new(
HTTP::OAI::Identify->new(
baseURL => 'http://arXiv.org/oai2',
));
The 6 OAI-PMH Verbs are the requests supported by an OAI-PMH interface.
Use "is_success()" or "is_error()" on the returned object to determine whether an error occurred (see HTTP::OAI::Response).
"code()" and "message()" return the error code (200 is success) and a human-readable message respectively. Errors returned by the repository can be retrieved using the "errors()" method:
foreach my $error ($r->errors) {
print $error->code, "\t", $error->message, "\n";
}
Note: "is_success()" is true for the OAI Error Code "noRecordsMatch" (i.e. empty set), although "errors()" will still contain the OAI error.
If the response contained a resumption token this can be retrieved using the $r->resumptionToken method.
These methods return an object subclassed from HTTP::Response (where the class corresponds to the verb requested, e.g. "GetRecord" requests return an "HTTP::OAI::GetRecord" object).
$gr = $h->GetRecord(
identifier => 'oai:arXiv:hep-th/0001001', # Required
metadataPrefix => 'oai_dc' # Required
);
$rec = $gr->next;
die $rec->message if $rec->is_error;
printf("%s (%s)\n", $rec->identifier, $rec->datestamp);
$dom = $rec->metadata->dom;
$id = $h->Identify();
print join ',', $id->adminEmail;
$lr = $h->ListIdentifiers(
metadataPrefix => 'oai_dc', # Required
from => '2001-10-01',
until => '2001-10-31',
set=>'physics:hep-th',
);
while($rec = $lr->next)
{
{ ... do something with $rec ... }
}
die $lr->message if $lr->is_error;
$lmdf = $h->ListMetadataFormats(
identifier => 'oai:arXiv.org:hep-th/0001001'
);
for($lmdf->metadataFormat) {
print $_->metadataPrefix, "\n";
}
die $lmdf->message if $lmdf->is_error;
$lr = $h->ListRecords(
metadataPrefix=>'oai_dc', # Required
from => '2001-10-01',
until => '2001-10-01',
set => 'physics:hep-th',
);
while($rec = $lr->next)
{
{ ... do something with $rec ... }
}
die $lr->message if $lr->is_error;
$ls = $h->ListSets();
while($set = $ls->next)
{
print $set->setSpec, "\n";
}
die $ls->message if $ls->is_error;
These modules have been written by Tim Brody <tdb01r@ecs.soton.ac.uk>.
| 2021-02-27 | perl v5.32.1 |