| String::Tokenizer(3pm) | User Contributed Perl Documentation | String::Tokenizer(3pm) |
String::Tokenizer - A simple string tokenizer.
use String::Tokenizer;
# create the tokenizer and tokenize input
my $tokenizer = String::Tokenizer->new("((5+5) * 10)", '+*()');
# create tokenizer
my $tokenizer = String::Tokenizer->new();
# ... then tokenize the string
$tokenizer->tokenize("((5 + 5) - 10)", '()');
# will print '(, (, 5, +, 5, ), -, 10, )'
print join ", " => $tokenizer->getTokens();
# create tokenizer which retains whitespace
my $st = String::Tokenizer->new(
'this is a test with, (significant) whitespace',
',()',
String::Tokenizer->RETAIN_WHITESPACE
);
# this will print:
# 'this', ' ', 'is', ' ', 'a', ' ', 'test', ' ', 'with', ' ', '(', 'significant', ')', ' ', 'whitespace'
print "'" . (join "', '" => $tokenizer->getTokens()) . "'";
# get a token iterator
my $i = $tokenizer->iterator();
while ($i->hasNextToken()) {
my $next = $i->nextToken();
# peek ahead at the next token
my $look_ahead = $i->lookAheadToken();
# ...
# skip the next 2 tokens
$i->skipTokens(2);
# ...
# then backtrack 1 token
my $previous = $i->prevToken();
# ...
# get the current token
my $current = $i->currentToken();
# ...
}
A simple string tokenizer which takes a string and splits it on whitespace. It also optionally takes a string of characters to use as delimiters, and returns them with the token set as well. This allows for splitting the string in many different ways.
This is a very basic tokenizer, so more complex needs should be either addressed with a custom written tokenizer or post-processing of the output generated by this module. Basically, this will not fill everyone's needs, but it spans a gap between simple "split / /, $string" and the other options that involve much larger and complex modules.
Also note that this is not a lexical analyser. Many people confuse tokenization with lexical analysis. A tokenizer merely splits its input into specific chunks, a lexical analyzer classifies those chunks. Sometimes these two steps are combined, but not here.
(5 + (100 * (20 - 35)) + 4)
The "tokenize" method without a $delimiter parameter would return the following comma separated list of tokens:
'(5', '+', '(100', '*', '(20', '-', '35))', '+', '4)'
However, if you were to pass the following set of delimiters "(, )" to "tokenize", you would get the following comma separated list of tokens:
'(', '5', '+', '(', '100', '*', '(', '20', '-', '35', ')', ')', '+', '4', ')'
We now can differentiate the parens from the numbers, and no globbing occurs. If you wanted to allow for optionally leaving out the whitespace in the expression, like this:
(5+(100*(20-35))+4)
as some languages do. Then you would give this delimiter "+*-()" to arrive at the same result.
If you decide that whitespace is significant in your string, then you need to specify that like this:
my $st = String::Tokenizer->new(
'this is a test with, (significant) whitespace',
',()',
String::Tokenizer->RETAIN_WHITESPACE
);
A call to "getTokens" on this instance would result in the following token set.
'this', ' ', 'is', ' ', 'a', ' ', 'test', ' ', 'with', ' ', '(', 'significant', ')', ' ', 'whitespace'
All running whitespace is grouped together into a single token, we make no attempt to split it into its individual parts.
A String::Tokenizer::Iterator instance is returned from the String::Tokenizer's "iterator" method and serves as yet another means of iterating through an array of tokens. The simplest way would be to call "getTokens" and just manipulate the array yourself, or push the array into another object. However, iterating through a set of tokens tends to get messy when done manually. So here I have provided the String::Tokenizer::Iterator to address those common token processing idioms. It is basically a bi-directional iterator which can look ahead, skip and be reset to the beginning.
NOTE: String::Tokenizer::Iterator is an inner class, which means that only String::Tokenizer objects can create an instance of it. That said, if String::Tokenizer::Iterator's "new" method is called from outside of the String::Tokenizer package, an exception is thrown.
Possibly compliment this expansion with compression as well, so for instance double quoted strings could be compressed into a single token.
None that I am aware of. Of course, if you find a bug, let me know, and I will be sure to fix it.
I use Devel::Cover to test the code coverage of my tests, below is the Devel::Cover report on this module's test suite.
------------------------ ------ ------ ------ ------ ------ ------ ------ File stmt branch cond sub pod time total ------------------------ ------ ------ ------ ------ ------ ------ ------ String/Tokenizer.pm 100.0 100.0 64.3 100.0 100.0 100.0 97.6 ------------------------ ------ ------ ------ ------ ------ ------ ------ Total 100.0 100.0 64.3 100.0 100.0 100.0 97.6 ------------------------ ------ ------ ------ ------ ------ ------ ------
The interface and workings of this module are based largely on the StringTokenizer class from the Java standard library.
Below is a short list of other modules that might be considered similar to this one. If this module does not suit your needs, you might look at one of these.
stevan little, <stevan@cpan.org>
Copyright 2004-2016 by Infinity Interactive, Inc.
<http://www.iinteractive.com>
This library is free software; you can redistribute it and/or modify it under the same terms as Perl itself.
| 2022-10-13 | perl v5.34.0 |