rheostream(2rheolef) | rheolef-7.0 | rheostream(2rheolef) |
irheostream, orheostream - large data streams
This class provides a stream interface for large data management. File decompresion is assumed using gzip and a recursive search in a directory list is provided for input.
is like
orheostream foo("NAME", "suffix");
However, if NAME does not end with `.suffix', then `.suffix' is automatically added. By default, compression is performed on the fly with gzip, adding an additional `.gz' suffix. The flush action is nicely handled in compression mode:
ofstream foo("NAME.suffix").
This feature allows intermediate results to be available during long computations. The compression can be deactivated while opening a file by an optional argument:
foo.flush();
An existing compressed file can be reopen in append mode: new results will be appended at the end of an existing file:
orheostream foo("NAME", "suffix", io::nogz);
orheostream foo("NAME", "suffix", io::app);
Conversely,
is like
irheostream foo("NAME","suffix");
However, we look at a search path environment variable RHEOPATH in order to find NAME while suffix is assumed. Moreover, gzip compressed files, ending with the `.gz' suffix is assumed, and decompression is done.
ifstream foo("NAME.suffix").
Finally, a set of useful functions are provided.
The following code:
will recursively look for a `results[.data[.gz]]' file in the directory mentioned by the RHEOPATH environment variable.
irheostream is("results", "data");
For instance, if you insert in our ".cshrc" something like:
the process will study the current directory `.', then, if neither `square.data.gz' nor `square.data' exits, it scan all subdirectory of the current directory. Then, if file is not founded, it start recusively in `/home/dupond' and then in `/usr/local/math/demo'.
setenv RHEOPATH ".:/home/dupont:/usr/local/math/demo"
File decompression is performed by using the gzip command, and data are pipe-lined directly in memory.
If the file start with `.' as `./square' or with a `/' as `/home/oscar/square', no search occurs and RHEOPATH environment variable is not used.
Also, if the environment variable RHEOPATH is not set, the default value is the current directory `.'.
For output stream:
file compression is assumed, and "newresults.data.gz" will be created.
orheostream os("newresults", "data");
File loading and storing are mentioned by a message, either:
or:
! load "./results.data.gz"
on the clog stream. By adding the following:
! file "./newresults.data.gz" created.
you turn off these messages
clog << noverbose;
class irheostream : public boost::iostreams::filtering_stream<boost::iostreams::input> { public:
irheostream() : boost::iostreams::filtering_stream<boost::iostreams::input>() {}
irheostream(const std::string& name, const std::string& suffix = std::string());
virtual ~irheostream();
void open (const std::string& name, const std::string& suffix = std::string());
void close(); protected:
std::ifstream _ifs; }; static const bool dont_gzip = false; class orheostream : public boost::iostreams::filtering_stream<boost::iostreams::output> { public:
orheostream() : boost::iostreams::filtering_stream<boost::iostreams::output>() {}
orheostream(const std::string& name, const std::string& suffix = std::string(),
io::mode_type mode = io::out);
virtual ~orheostream();
void open (const std::string& name, const std::string& suffix = std::string(),
io::mode_type mode = io::out);
void flush();
void close();
const std::string& filename() const { return _full_name; } protected:
void _open_internal (io::mode_type mode);
void _close_internal (); // data:
io::mode_type _mode;
std::string _full_name; }; std::string itos (std::string::size_type i); std::string ftos (const Float& x); // catch first occurrence of string in file bool scatch (std::istream& in, const std::string& ch, bool full_match = true); // has_suffix("toto.suffix", "suffix") -> true bool has_suffix (const std::string& name, const std::string& suffix); // "toto.suffix" --> "toto" std::string delete_suffix (const std::string& name, const std::string& suffix); // has_any_suffix("toto.any_suffix") -> true bool has_any_suffix (const std::string& name); // delete_any_suffix("toto.any_suffix") --> "toto" std::string delete_any_suffix (const std::string& name); // "/usr/local/dir/toto.suffix" --> "toto.suffix" std::string get_basename (const std::string& name); // "/usr/local/dir/toto.suffix" --> "/usr/local/dir" std::string get_dirname (const std::string& name); // "toto" --> "/usr/local/math/data/toto.suffix" std::string get_full_name_from_rheo_path (const std::string& rootname, const std::string& suffix); // "." + "../geodir" --> ".:../geodir" void append_dir_to_rheo_path (const std::string& dir); // "../geodir" + "." --> "../geodir:." void prepend_dir_to_rheo_path (const std::string& dir); bool file_exists (const std::string& filename); // string to float bool is_float (const std::string&); Float to_float (const std::string&); // in TMPDIR environment variable or "/tmp" by default std::string get_tmpdir();
Copyright (C) 2000-2018 Pierre Saramito <Pierre.Saramito@imag.fr> GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>. This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law.
rheolef-7.0 | rheolef-7.0 |