uconv - convert data from one encoding to another
uconv [ -h, -?, --help ]
[ -V, --version ] [ -s,
--silent ] [ -v, --verbose
] [ -l, --list | -l,
--list-code code | --default-code
| -L, --list-transliterators ] [
--canon ] [ -x transliteration ] [
--to-callback callback | -c ] [
--from-callback callback | -i ] [ --callback
callback ] [ --fallback | --no-fallback ] [ -b,
--block-size size ] [ -f,
--from-code encoding ] [
-t, --to-code encoding ] [
--add-signature ] [ --remove-signature ] [ -o,
--output file ] [ file...
]
uconv converts, or transcodes, each given file (or
its standard input if no file is specified) from one encoding
to another. The transcoding is done using Unicode as a pivot encoding (i.e.
the data are first transcoded from their original encoding to Unicode, and
then from Unicode to the destination encoding).
If an encoding is not specified or is -, the default
encoding is used. Thus, calling uconv with no encoding
provides an easy way to validate and sanitize data files for further
consumption by tools requiring data in the default encoding.
When calling uconv, it is possible to specify callbacks
that are used to handle invalid characters in the input, or characters that
cannot be transcoded to the destination encoding. Some encodings, for
example, offer a default substitution character that can be used to
represent the occurrence of such characters in the input. Other callbacks
offer a useful visual representation of the invalid data.
uconv can also run the specified transliteration on
the transcoded data, in which case transliteration will happen as an
intermediate step, after the data have been transcoded to Unicode. The
transliteration can be either a list of semicolon-separated
transliterator names, or an arbitrarily complex set of rules in the ICU
transliteration rules format.
For transcoding purposes, uconv options are compatible with
those of iconv(1), making it easy to replace it in scripts. It is not
necessarily the case, however, that the encoding names used by uconv
and ICU are the same as the ones used by iconv(1). Also, options that
provide informational data, such as the -l, --list
one offered by some iconv(1) variants such as GNU's, produce
data in a slightly different and easier to parse format.
- -h, -?,
--help
- Print help about usage and exit.
- -V,
--version
- Print the version of uconv and exit.
- -s,
--silent
- Suppress messages during execution.
- -v,
--verbose
- Display extra informative messages during execution.
- -l,
--list
- List all the available encodings and exit.
- -l,
--list-code code
- List only the code encoding and exit. If code is not a
proper encoding, exit with an error.
- --default-code
- List only the name of the default encoding and exit.
- -L,
--list-transliterators
- List all the available transliterators and exit.
- --canon
- If used with -l, --list or
--default-code, the list of encodings is produced in a format
compatible with convrtrs.txt(5). If used with -L,
--list-transliterators, print only one
transliterator name per line.
- -x
transliteration
- Run the given transliteration on the transcoded Unicode data, and
use the transliterated data as input for the transcoding to the
destination encoding.
- --to-callback
callback
- Use callback to handle characters that cannot be transcoded to the
destination encoding. See section CALLBACKS for details on valid
callbacks.
- -c
- Omit invalid characters from the output. Same as --to-callback
skip.
- --from-callback
callback
- Use callback to handle characters that cannot be transcoded from
the original encoding. See section CALLBACKS for details on valid
callbacks.
- -i
- Ignore invalid sequences in the input. Same as --from-callback
skip.
- --callback
callback
- Use callback to handle both characters that cannot be transcoded
from the original encoding and characters that cannot be transcoded to the
destination encoding. See section CALLBACKS for details on valid
callbacks.
- --fallback
- Use the fallback mapping when transcoding from Unicode to the destination
encoding.
- --no-fallback
- Do not use the fallback mapping when transcoding from Unicode to the
destination encoding. This is the default.
- -b,
--block-size size
- Read input in blocks of size bytes at a time. The default block
size is 4096.
- -f,
--from-code encoding
- Set the original encoding of the data to encoding.
- -t,
--to-code encoding
- Transcode the data to encoding.
- --add-signature
- Add a U+FEFF Unicode signature character (BOM) if the output charset
supports it and does not add one anyway.
- --remove-signature
- Remove a U+FEFF Unicode signature character (BOM).
- -o,
--output file
- Write the transcoded data to file.
uconv supports specifying callbacks to handle invalid data.
Callbacks can be set for both directions of transcoding: from the original
encoding to Unicode, with the --from-callback option, and from
Unicode to the destination encoding, with the --to-callback
option.
The following is a list of valid callback names, along with
a description of their behavior. The list of callbacks actually supported by
uconv is displayed when it is called with -h,
--help.
- substitute
- Write the encoding's substitute sequence, or the Unicode replacement
character U+FFFD when transcoding to Unicode.
- skip
- Ignore the invalid data.
- stop
- Stop with an error when encountering invalid data. This is the default
callback.
- escape
- Same as escape-icu.
- escape-icu
- Replace the missing characters with a string of the format
%Uhhhh for plane 0 characters, and
%Uhhhh%Uhhhh for planes 1 and above characters, where
hhhh is the hexadecimal value of one of the UTF-16 code units
representing the character. Characters from planes 1 and above are written
as a pair of UTF-16 surrogate code units.
- escape-java
- Replace the missing characters with a string of the format
\uhhhh for plane 0 characters, and
\uhhhh\uhhhh for planes 1 and above characters, where
hhhh is the hexadecimal value of one of the UTF-16 code units
representing the character. Characters from planes 1 and above are written
as a pair of UTF-16 surrogate code units.
- escape-c
- Replace the missing characters with a string of the format
\uhhhh for plane 0 characters, and \Uhhhhhhhh
for planes 1 and above characters, where hhhh and hhhhhhhh
are the hexadecimal values of the Unicode codepoint.
- escape-xml
- Same as escape-xml-hex.
- escape-xml-hex
- Replace the missing characters with a string of the format
&#xhhhh;, where hhhh is the hexadecimal value of
the Unicode codepoint.
- escape-xml-dec
- Replace the missing characters with a string of the format
&#nnnn;, where nnnn is the decimal value of the
Unicode codepoint.
- escape-unicode
- Replace the missing characters with a string of the format
{U+hhhh}, where hhhh is the hexadecimal value of the
Unicode codepoint. That hexadecimal string is of variable length and can
use from 4 to 6 digits. This is the format universally used to denote a
Unicode codepoint in the literature, delimited by curly braces for easy
recognition of those substitutions in the output.
Convert data from a given encoding to the platform
encoding:
$ uconv -f encoding
Check if a file contains valid data for a given
encoding:
$ uconv -f encoding -c file
>/dev/null
Convert a UTF-8 file to a given encoding and ensure
that the resulting text is good for any version of HTML:
$ uconv -f utf-8 -t encoding \
--callback escape-xml-dec file
Display the names of the Unicode code points in a UTF-file:
$ uconv -f utf-8 -x any-name file
Print the name of a Unicode code point whose value is known
(U+30AB in this example):
$ echo '\u30ab' | uconv -x 'hex-any; any-name';
echo
{KATAKANA LETTER KA}{LINE FEED}
$
(The names are delimited by curly braces. Also, the name of the
line terminator is also displayed.)
Normalize UTF-8 data using Unicode NFKC, remove all control
characters, and map Katakana to Hiragana:
$ uconv -f utf-8 -t utf-8 \
-x '::nfkc; [:Cc:] >; ::katakana-hiragana;'
uconv does report errors as occurring at the first invalid
byte encountered. This may be confusing to users of GNU iconv(1),
which reports errors as occurring at the first byte of an invalid sequence.
For multi-byte character sets or encodings, this means that uconv
error positions may be at a later offset in the input stream than would be
the case with GNU iconv(1).
The reporting of error positions when a transliterator is used may
be inaccurate or unavailable, in which case uconv will report the
offset in the output stream at which the error occurred.
Jonas Utterstroem
Yves Arrouye
Copyright (C) 2000-2005 IBM, Inc. and others.