DOKK / manpages / debian 10 / libplucene-perl / Plucene::Analysis::Standard::StandardTokenizer.3pm.en
Plucene::Analysis::Standard::StandardTokenizer(3pm) User Contributed Perl Documentation Plucene::Analysis::Standard::StandardTokenizer(3pm)

Plucene::Analysis::Standard::StandardTokenizer - standard tokenizer

        # isa Plucene::Analysis::CharTokenizer

This is the standard tokenizer.

This should be a good tokenizer for most European-language documents.

The regular expression for tokenising.

Remove 's and .

2018-04-02 perl v5.26.1