sluice - a tool to control data flow at a specified rate
sluice reads input and outputs a specified data rate. It has
various data rate controlling mechanisms that can be tuned for specific use
cases where necessary.
sluice options are as follow:
- -a
- append output to a file (used in conjunction with the -t 'tee' or -O
options). Instead of creating a new file or truncating an existing file,
this option appends data to the file.
- -c delay
- enables a constant delay time (in seconds) between writes. This option
adjusts the output buffer size to try and keep the data rate constant. The
output buffer size in this mode is initially set to the data rate ×
the delay.
This option is mutually exclusive to the -i option and implicitly enables
the -o overrun and -u underrun buffer management options to dynamically
re-size the read/write buffer to keep the data rate constant. By default
this adjusts the buffer based on the total amount of data transferred and
the time to write this (equivalent to the -s 0 turning mode). However, if
the -s shift value is greater than 0, then the new size is adjusted by the
previous size right shifted by the shift value.
- -d
- discard data, do not copy it to stdout. This makes sluice act as a data
sink.
- -D mode
- select the delay mode. The are various approaches to when to perform the
data rate delays. The default is to perform the read, then write and
finally the delay for each iteration. However, the -D option allows one to
select the delay mode as follows:
Mode |
Delay strategy |
Delay Duration |
0 |
Read, Write, Delay (default) |
1 × delay time |
1 |
Delay, Read, Write |
1 × delay time |
2 |
Read, Delay, Write |
1 × delay time |
3 |
Delay, Read, Delay, Write |
2 × 1/2 delay time |
4 |
Read, Delay, Write, Delay |
2 × 1/2 delay time |
5 |
Delay, Read, Delay, Write, Delay |
3 × 1/3 delay time |
Note that modes 3 and 4 perform two delays each comprising of 1/2 the delay
time and mode 5 performs 3 delays each comprising of 1/3 the delay time.
Modes 1, 3, 5 maybe considered as not entirely accurate in terms of the
total run duration. In these modes an extraneous delay occurs before the
final end-of-file empty read is performed.
- -e
- ignore read errors. The failed read is replaced by zeros.
- -f freq
- specify the frequency of -v verbose statistics updates. The default is 1/4
of a second. Note that sluice will try to emit updates close to the
requested frequency, however, if the read/write rate is less than the
frequency then the updates occur only at the read/write rate.
- -h
- show help
- -i size
- specify the read/write size in bytes. The K, M, G, T and P suffixes allow
one to specify size in Kilobytes, Megabytes, Gigabytes, Terabytes and
Petabytes respectively. This option is mutually exclusive to the -c
option.
In this mode, the delays between writes are used to control the data rate.
By default the delay is based on the total amount of data transferred and
the time taken to write this. This is equivalent to the -s 0 tuning mode.
However, if the -s shift value is greater than 0, then the new delay is
adjusted by the previous delay right shifted by the shift value.
A special hybrid rate control mode can be invoked by also using the -o
overflow and -u underflow options to also enable dynamic re-sizing of the
read/write buffer. By default this adjusts the buffer based on the total
amount of data transferred and the time to write this (equivalent to the
-s 0 turning mode). However, if the -s shift value is greater than 0, then
the new size is adjusted by the previous size right shifted by the shift
value.
- -I file
- read input from file rather than from stdin.
- -m size
- specify amount of data to process, the default size is in bytes, but the
K, M, G, T and P suffixes can specify size in Kilobytes, Megabytes,
Gigabytes, Terabytes and Petabytes respectively. If this size is less than
the write size, then the write size is truncated to be the -m size.
- -n
- no rate control. This is just a straight data copy much like cat and all
data rate controls cannot be used. Combined with the -v and -S options one
can observe the data rates of the copy.
- -o
- detect overrun and re-size read/write buffer size to try and stop overrun.
This will shrink the buffer each time consecutive overruns are detected.
See the -s option for details of the size re-adjustment mechanism.
- -O file
- send output to file, equivalent to -dt file
- -p
- enable verbose stats showing % progress and ETA information. This is only
valid using the -I or -m option and the if file size is non-zero. See the
-v option for more details.
- -P pidfile
- write the process ID of sluice in file pidfile. The file is removed when
sluice exits.
- -r rate
- specify the data rate in bytes per second. The K, M, G and T suffixes can
specify the rate in Kilobytes/sec, Megabytes/sec, Gigabytes/sec and
Terabytes/sec respectively. This option must always be provided except
when the -n option is used. A zero rate is equivalent to no rate control
(same as -n).
- -R
- do not read from stdin, instead read random data from /dev/urandom.
- -s shift
- modify the rate adjustment shift. This is a data rate tuning scaling
factor used by the -r, -c, -o and -u options.
For the -r option, the delay between each write is controlled by modifying
the previous delay by adding or subtracting the previous delay right
shifted by this shift value. The larger the shift value the longer it
takes to adjust up/down to the specified rate. The smaller the shift value
the quicker it takes to reach the optimal delay, however, this can result
in a highly fluctuating rates at the the beginning because the delay
varies by a large amount causing large overruns and underruns. A shift
value of 3 works well for most fast rates.
For the -c, -o and -u options, the size of the buffer is modified by adding
or subtracting the previous size shifted by the shift value. Again, a
shift value of 3 works well for most fast rates.
If the shift value is set to 0, then the shift rate adjustment tuning
mechanism is explicitly turned off and data rates are adjusted based on
the total amount of data transferred and the time to write this.
Small -s shift values of 1 and 2 can cause rapid oscillations before the
data rate damping fully kicks into action. The value of -s 0 (the default)
is recommended for accurate low-speed data transfers.
- -S
- print various performance and buffering statistics to stderr when end of
file is reached.
- -t file
- tee output to the specified file. Output is written to both stdout and to
the named file. By default, the file will be created if it does not exist
or re-written if it already exists. Use the -a option to append to an
existing file.
- -T t
- stop slice test after t seconds. One can also specify the units of time in
seconds, minutes, hours, days or years with the suffix s, m, h, d or
y.
- -u
- detect underrun and re-size read/write buffer size to try and stop
underrun. This will expand the buffer each time consecutive underruns are
detected. The buffer will not be expanded any more than 4MB in size. See
the -s option for details of the size re-adjustment mechanism.
- -v
- write verbose statistics to stderr. By default, this will display the
current data rate, the last data rate adjusment ('-' = underrun, size.
With the -p option, the progess statistics are displayed. This will display
the current data rate, total bytes transferred, duration, percentage
complete so far and the estimated time to completion. Note that the
estimation is available using the -I and -m options and if the file size
is non-zero.
- -V
- print version information to standard out and exit successfully.
- -w
- warn if a long burst of continuous data rate underrun occurs, the warning
is issued just once. To overcome the underrun increase the -i read/write
buffer size or use the -u option to auto-expand the read/write buffer. Too
many underruns implies that too small a buffer or not enough CPU is
available to keep up with the required data rate.
- -x size
- set pipe transfer size. If data is being piped into or out of sluice then
this option allows one to specify the pipe size. Larger pipe sizes provied
better throughput and less context switching; smaller pipe sizes are
useful for low bandwidth rates where latency needs to be kept low.
- -z
- do not read from stdin, instead generate a stream of zeros (equivalent to
reading from /dev/zero).
- SIGUSR1
SIGINFO
- Sending SIGUSR1 (or SIGINFO on BSD systems) will toggle the verbose data
rate mode on/off.
- SIGUSR2
- Toggle underrun/overrun (-u, -o) options on/off.
If neither -i or -c options are used, then sluice defaults to
using a write buffer size of 1/32 of the data rate and bounded between the
limits of 1 byte and 64MB. Sluice will try to keep the data rate steady by
adjusting the delay between writes. To tune this, see the -s option.
Read /dev/zero and write in 4K sizes at the rate of 1MB/sec to the
file 'example.dat'
cat /dev/zero | sluice -i 4K -r 1M > example.dat
Read 32MB from /dev/zero and write at the rate of 64K/sec to
stdout with feedback on duration and ETA on stderr using 4K buffer writes
and a tuning shift of 4.
cat /dev/zero | sluice -r 64K -vp -m 32M -i 4K -s 4
Generate a stream of zeros and write at a rate of 1MB/sec to a
fifo named 'myfifo' with underrun and overrun buffer management
sluice -z -u -o -r 1MB -O myfifo
Write random data at 5MB per second to the file 'myfile' doing a
write every 0.1 seconds
sluice -R -r 5M -c 0.1 > myfile
Write zeros to the file 'example-file' in 64K chunks and measure
write rate as a crude throughput test
sluice -nzSv -f 0.1 -i 64K > example-file
Read data from somehost.com on port 1234 at a rate of 2MB per
second and discard the data, e.g. this is a constant rate data sink.
nc somehost.com 1234 | sluice -d -r 2MB -i 8K
Sluice sets the exit status as follows:
Status |
Decription |
0 |
Exited successfully. |
1 |
Invalid or out of range option provided. |
2 |
File open error. |
3 |
Sleep error. |
4 |
Failed to get time of day. |
5 |
Signal handler setup error. |
6 |
Read error (file or stdin). |
7 |
Write error (file or stdout). |
8 |
Buffer allocation failed. |
Stopping and starting sluice using SIGSTOP and SIGCONT will
interfere with the internal buffering rate calculations causing sluice to
try to catch up and this may affect the short term data rate immediately
after the SIGCONT.
sluice was written by Colin Ian King
<colin.i.king@gmail.com> with testing feedback and help from Kamal
Mostafa.
This manual page was written by Colin Ian King for the Ubuntu
project (but may be used by others).
Copyright © 2014-2021 Canonical Ltd, Copyright ©
2021-2022 Colin Ian King
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR
PURPOSE.