DDPT(8) | DDPT | DDPT(8) |
ddpt - copies data between files and storage devices. Support for devices that understand the SCSI command set.
ddpt [bpt=BPT[,OBPC]] [bs=BS] [cdbsz={6|10|12|16|32}] [coe={0|1}] [coe_limit=CL] [conv=CONVS] [count=COUNT] [delay=MS[,W_MS]] [ibs=IBS] [id_usage=LIU] if=IFILE [iflag=FLAGS] [intio={0|1}] [iseek=SKIP] [ito=ITO] [list_id=LID] [obs=OBS] [of=OFILE] [of2=OFILE2] [oflag=FLAGS] [oseek=SEEK] [prio=PRIO] [protect=RDP[,WRP]] [retries=RETR] [rtf=RTF] [rtype=RTYPE] [seek=SEEK] [skip=SKIP] [status=STAT] [to=TO] [verbose=VERB] [--help] [--job=JF] [--odx] [--verbose] [--version] [--wscan] [--xcopy] [JF]
For comparison here is the synopsis for GNU's dd command:
dd [bs=BS] [cbs=CBS] [conv=CONVS] [count=COUNT] [ibs=IBS] [if=IFILE] [iflag=FLAGS] [obs=OBS] [of=OFILE] [oflag=FLAGS] [seek=SEEK] [skip=SKIP] [status=STAT] [--help] [--version]
Copies data between files or simply reads data from a file. This utility is specialized for "files" that are storage devices, especially those that can use the SCSI command sets (e.g. SATA and SAS disks). It can issue SCSI commands in pass-through ("pt") mode. Similar syntax and semantics to the Unix dd(1) command.
For comparison, the SYNOPSIS section above shows both the ddpt command line options followed by GNU's dd(1) command line options. Broadly speaking ddpt can be considered a super-set of dd. See the section on DD DIFFERENCES for significant differences between ddpt and dd.
This utility either does direct copies, based on read-write sequences, or offloaded copies. In an offloaded copy the data being copied does not necessarily pass through the memory of the the machine originating the copy operation; this can save a significant amount of time and lessen CPU usage.
When doing a direct copy, this utility breaks the copy into segments since computer RAM is typically a scarce resource. First it reads in BPT*IBS bytes from IFILE (or less if near the end of the copy) into a copy buffer. In the absence of the various options and conditions that bypass the write operation, the copy buffer is then written out to OFILE. The copy process continues working its way along IFILE and OFILE until either COUNT is exhausted, an end of file is detected, or an error occurs. If IBS and OBS are different, ddpt restricts the value of OBS such that the copy buffer is an integral number of output blocks (i.e. (((IBS * BPT) % OBS) == 0) ). In the following descriptions, "segment" refers to all or part of a copy buffer.
The term "pt device" is used for a pass-through device to which SCSI commands like READ(10), WRITE(10) or POPULATE TOKEN may be sent. A pt device may only be able to process SCSI commands in which case the "pt" flag is assumed. The ability to recognize such a pt only device may vary depending on the operating system (e.g. in Linux /dev/sg2 and /dev/bsg/3:0:1:0 are recognized). However if a device can process either normal UNIX read()/ write() calls or pass-through SCSI commands then the default is to use UNIX read()/write() calls. That default can be overridden by using the "pt" flag (e.g. "if=/dev/sdc iflag=pt"). When pt access is specified any partition information is ignored. So "if=/dev/sdc2 iflag=pt skip=3" will start at logical block address 3 of '/dev/sdc'. As a protection measure ddpt will only accept that if the force flag is also given (i.e. 'iflag=pt,force').
This utility supports two types of offloaded copies. Both are based on the EXTENDED COPY (XCOPY or xcopy) family of SCSI commands. The first uses the XCOPY(LID1) command to do a disk to disk copy. LID1 stands for List IDentifier length of 1 byte and the command is described in the SPC-4 drafts and the earlier SPC-3 and SPC-2 standards. Recent SPC-4 drafts have added the XCOPY(LID4) sub-family of copy offloaded commands. There is a subset of XCOPY(LID4), specialized for offloaded disk to disk copies, that is known by the market name: ODX. In the descriptions below "xcopy" refers to copies based on XCOPY(LID1) while "odx" refers to either full or partial ODX copies. See the XCOPY and ODX sections below for more information.
The dd-like options with the name=value syntax are listed first, sorted by name. Following that, options starting with "-" are listed.
When the count=COUNT option is not given (or COUNT is '-1') then an attempt is made to deduce COUNT as follows.
When both or either IFILE and OFILE are block devices, then the minimum size, expressed in units of input blocks, is used. When both or either IFILE and OFILE are pass-through devices, then the minimum size, expressed in units of input blocks, is used.
If a regular file is used as input, its size, expressed in units of input blocks (and rounded up if necessary) is used. Note that the rounding up of the deduced COUNT may result in a partial read of the last input block and a corresponding partial write to OFILE if it is a regular file. After a regular file to regular file copy the length of OFILE will be the same as IFILE unless OFILE existed and its length was already greater than that of IFILE. To get a copy like the standard Unix cp command, use oflag=trunc with ddpt.
The size of pt devices is deduced from the SCSI READ CAPACITY command. Block device sizes (or their partition sizes) are obtained from the operating system, if available.
If skip=SKIP or skip=SEEK are given and the COUNT is deduced (i.e. not explicitly given) then that size is scaled back so that the copy will not overrun the file or device.
If COUNT is not given and IFILE is a fifo (and stdin is treated as a fifo) then IFILE is read until an EOF is detected. If COUNT is not given and IFILE is a /dev/zero (or equivalent) then zeros are read until an error occurs (e.g. file system full).
If COUNT is not given and cannot be deduced then an error message is issued and no copy takes place.
One or more conversions can be given to the "conv=" option. If more than one is given, they should be comma separated. ddpt does not perform the traditional dd conversions (e.g. ASCII to EBCDIC). Recently added conversions inherited from GNU's dd overlap somewhat with the some of ddpt flags.
A list of flags and their meanings follow. The flag name is followed by one or two indications in square brackets. The first indication is either "[i]", "[o]" or "[io]" indicating this flag is active for the IFILE, OFILE or both the IFILE and the OFILE. The second indication contains some combination of "reg", "blk" "pt", "odx", or "xcopy". These indicate whether the flag applies to a regular file, a block device (accessed via Unix read() and write() commands, a pass-through device, an ODX offloaded copy or a XCOPY(LID1) offloaded copy respectively. Other special file types that are sometimes referred to are "fifo" and "tape".
This section describes XCOPY(LID1) support with this utility. For ODX support (XCOPY(LID4) subset) see the ODX section.
A device (logical unit (LU)) that supports XCOPY operations should set the 3PC field (3PC stands for Third Party Copy) in its standard INQUIRY response. That is not checked when this utility does an xcopy operation but if it fails, that is one thing that the user may want to check.
If the xcopy starts and fails while underway, then 'sg_copy_results -s' may be useful to view the copy status. It might also be used from a different process with the same I_T nexus (i.e. the same machine) to check status during an xcopy operation.
The pad and cat flags control the handling of residual data. As the data can be specified either in terms of source or target block size and both might have different block sizes residual data is likely to happen in these cases. If both block sizes are identical these bits have no effect as residual data will not occur.
If neither of these flags are set, the EXTENDED COPY command will be aborted with additional sense 'UNEXPECTED INEXACT SEGMENT'.
If only the cat flag is set the residual data will be retained and made available for subsequent segment descriptors. Residual data will be discarded for the last segment descriptor.
If the pad flag is set for the source descriptor only, any residual data for both source or destination will be discarded.
If the pad flag is set for the target descriptor only any residual source data will be handled as if the cat flag is set, but any residual destination data will be padded to make a whole block transfer.
If the pad flag is set for both source and target any residual source data will be discarded, and any residual destination data will be padded.
There is a web page discussing ddpt, XCOPY and ODX at http://sg.danny.cz/sg/ddpt_xcopy_odx.html
This section describes ODX support (an XCOPY(LID4) subset) for this utility. ODX descriptions use the following command name abbreviations: PT for the POPULATE TOKEN command, RRTI for the READ ROD TOKEN INFORMATION command, and WUT for the WRITE USING TOKEN command.
A device (logical unit (LU)) that supports ODX operations is required to set the 3PC field (3PC stands for Third Party Copy) in its standard INQUIRY response and support the Third Party Copy VPD page. If this utility generates errors noting the absence of these then the device in question probably does not support ODX.
There a four variants of ODX supported by ddpt:
full copy : ddpt --odx if=/dev/sg3 bs=512 of=/dev/sg4
zero output blocks : ddpt if=/dev/null rtype=zero bs=512 of=/dev/sg4
read to tokens : ddpt if=/dev/sg3 bs=512 skip=@gath.lst rtf=a.rt
write from tokens : ddpt rtf=a.rt bs=512 of=/dev/sg4
seek=@scat.lst
The full copy will call PT and WUT commands repeatedly until the copy is complete. More precisely the full copy will make the largest single call to PT allowed by the input's Third Party Copy VPD page (and, if given, allowed by the BPT argument in the bpt=BPT[,OBPC] option). Then one or more WUT calls are made to write out from the ROD created by the PT step. The largest single WUT call is constrained by the output's Third Party Copy VPD page (and, if given, allowed by the OBPC argument in the bpt=BPT[,OBPC] option). This sequence continues until the requested copy is complete.
The zero output blocks variant is a special case of the full copy in which only WUT calls are made. ODX defines a special ROD Token to zero blocks. That special ROD Token has a fixed pattern (shown in SBC-3) and does not need to be created by a PT command like normal ROD Tokens.
The read to tokens and the write from tokens variants are designed to be the read (input) and write (output) sides respectively of a network copy. Each can run on different machines by sending the RTF file from the machine doing the read to the machine doing the write. The read to tokens will make one or more PT calls and output the resulting ROD Tokens to the RTF file. RTF might be a regular file or a named pipe.
All four variants can have the immed flag set. Then the PT and/or WUT commands are issued with the IMMED bit set and the RRTI command is used to poll for completion. The delay between the polls is as suggested by the RRTI command (or if no suggestion is made, 500 milliseconds). Either iflag=immed, oflag=immed or both can be given but are only effective if the corresponding IFILE or OFILE sends a PT or WUT command.
Typically there is no need to give the list_id=LID option. If this option is not given then 257 is chosen. If that is busy then 258 is tried. That continues until a usable LID is found or 10 LIDs have been tried. In the latter case ddpt exits with status of 55 (operation in progress). If the user gives list_id=LID option and LID is busy then ddpt exits with exit status 55.
If the block size of the input and output are different (i.e. IBS is not equal to OBS) then one must be a multiple of the other. So an input block size of 512 bytes and an output block size of 4096 bytes (or vice versa) is acceptable.
The four ODX variants are distinguished as follows: if OFILE is a pass-through device, if=/dev/null (or equivalent) and rtype=zero then the zero output blocks variant is selected. If both IFILE and OFILE are pass-through devices and there is some indication of an ODX request (e.g. the --odx option), then the full copy variant is selected. The read to tokens and the write from token variants are indicated by the absence of either a of=OFILE or a if=IFILE option, respectively, plus the presence of a rtf=RTF option.
The helper utility ddptctl contains options to issue a single PT, RRTI, WUT or COPY OPERATION ABORT command. It can also issue a series of polling RRTI commands. It can decode information in ROD Tokens (which is not as informative as it should be) and print the number of blocks and block size of a disk, plus protection information if available. See ddptctl.
There is a web page discussing ddpt, XCOPY and ODX at http://sg.danny.cz/sg/ddpt_xcopy_odx.html
Bypassing writes of blocks full of zeros can save a lot of IO. However with regular files, bypassed writes at the end of the copy can lead to an OFILE which is shorter than it would have been without sparse writes. This can lead to integrity checking programs like md5sum and sha1sum generating different values.
This utility has two ways of handling this file length problem: writing the last block (even if it is full of zeros) or using the ftruncate system call. A third approach is to ignore the problem (i.e. leaving OFILE shorter). The ftruncate approach is used when "oflag=strunc" while the last block is written when "oflag=sparse". To ignore the file length issue use "oflag=sparse,sparse". Note that if OFILE's length is already correct or longer than required, no action is taken.
The support for sparse writing of regular files may depend on the OS, the file system and the settings of OFILE. POSIX makes few guarantees when the ftruncate system call is used to extend a file's length, as may occur when "oflag=strunc". Further, primitive file systems like VFAT may not accept sparse writes or simulate the effect by writing blocks of zeros. The latter approach will defeat any sparse writing performance gain.
This is a new storage feature often associated with Solid State Disks (SSDs) or disk arrays with "thin provisioning". In the ATA command set (ACS-2) the relevant command is DATA SET MANAGEMENT with the TRIM bit set. In the SCSI command set (SBC-3) it is either the UNMAP or WRITE SAME command. Note there is no TRIM command however the term is frequently used in the technical press.
Trim is a way of telling a storage device that blocks are no longer needed. Keeping the pool of unwritten blocks large is important for the write performance of SSDs and the thrifty use of real storage in thin provisioned arrays. Currently file systems in recent OSes may issue trims associated with file deletes. The trim option in ddpt may be useful when a partition or a whole SSD is to be "deleted". Note that ddpt is bypassing file systems in that it only offers trim on pass-through (pt) devices.
This utility issues SCSI commands to pt devices and for "trim" currently issues a SCSI WRITE SAME(16) command with the UNMAP bit set. If the pt device is a SSD with a ATA interface then recent versions of Linux will translate the SCSI WRITE SAME to the ATA DATA SET MANAGEMENT command with the TRIM bit set. The maximum size of each "trim" command sent is the size of the copy buffer (i.e. IBS * BPT bytes). And that maximum can be reduced with the OBPC argument of the "bpt=" option.
The trim can be used various ways. One way is a copy where the copy buffer (or some part of it) is checked for zeros as is done by the sparse oflag. When a zero segment is found, a trim "command" is sent to the OFILE. For example:
ddpt if=dsk.img bs=512 of=/dev/sdc oflag=pt,trim
The copy buffer is 64 KiB (since BPT and OBPC default to 128 when "bs=512") and it is checked for all zeros. If it is all zeros then a trim command is sent to the corresponding location of /dev/sdc which is accessed via the pt interface. If it is not all zeros then a SCSI WRITE command is sent. Another way is to trim all or part of a disk. To trim a whole disk (i.e. deleting all its data):
ddpt if=/dev/zero bs=512 of=/dev/sdc oflag=pt,trim
A third way is to "self-trim" which is to only trim those parts of a disk that contain segments full of zeros:
ddpt if=/dev/sdc skip=0x2300 bs=512 iflag=pt,self,trim count=0x1234f0
The "self" oflag automatically sets up the output side of the copy to send trim commands (if required) back the the same device (i.e. /dev/sdc). If this example was self-trimming a partition then the partition would start at LBA 0x2300 and be 0x1234f0 blocks long.
Some random product examples: the Intel X25-M G2 SSDs have trim with recent firmware and they do deterministic read zero after trim. The Seagate Pulsar SSD has an ATA interface which supports the deterministic reads of zero after the DATA SET MANAGEMENT command with the TRIM option.
dd defaults "if=" and "of=" to stdin and stdout respectively. This follows Unix filter conventions. However since dd and ddpt are often used to read binary data for timing purposes, having to supply "of=/dev/null" can be easily forgotten. Without it dd will typically spew binary data on the console. So ddpt has changed its defaults: the "if=IFILE" is now mandatory for direct copies and to read from stdin "if=-" can be used; "of=OFILE" remains optional but its default changes to "/dev/null" (or "NUL" in Windows). To send output to stdout ddpt accepts "of=-".
dd truncates OFILE unless "conv=notrunc" is given. When dd truncates, it truncates to zero length unless SEEK is greater than zero. ddpt does not truncate OFILE by default. If OFILE exists it will be overwritten. The overwrite starts at block zero unless SEEK or "oflag=append" is given. If OFILE is a regular file then "oflag=trunc" (or "conv=trunc") will truncate OFILE prior to the copy.
Numeric arguments to ddpt can be given in hexadecimal, either with a leading "0x" or "0X" or with a trailing "h". Note that dd accepts "0x123" but interprets it as "0 * 123" (i.e. zero). ddpt will also interpret "x" as multiplies unless the left operand is zero (e.g. "0x123"). So both dd and ddpt will interpret "skip=2x123" as "skip=246".
Terabyte size disks make it impractical to copy all the data into a buffer before writing it out. Therefore both dd and ddpt read a relatively small amount of data into a copy (or transfer) buffer then write it out to the destination, repeating this process until the COUNT is exhausted.
A major difference in ddpt is the addition of BPT to control the size of the copy buffer. With dd, IBS is the size of the copy buffer and the unit of SKIP and COUNT. With ddpt, IBS * BPT is the size of the copy buffer and IBS is the unit of SKIP and COUNT. This allows ddpt to have its IBS set to the logical block size of IFILE without unduly restricting the size of the copy buffer. And setting IBS (and OBS for OFILE) accurately is required when the pass-through interface is used since with the SCSI READ and WRITE commands the logical block size is implicit.
The way dd handles its copy buffer (outlined in SUSv4 description of dd) is relatively complex, especially when IBS and OBS are different sizes. The restriction that ddpt places on IBS and OBS ( i.e. (((IBS * BPT) % OBS) == 0) ) means that a single copy buffer can be used since its size is a multiple of both IBS and OBS. Being able to precisely define the copy buffer size in ddpt makes sparse writing, write sparing and trim operations simpler to define and the user to control.
ddpt does not support dd's "cbs=" option (conversion block size). If the "cbs=" option is given to ddpt then it is ignored.
ddpt adds two types of disk to disk, offloaded copies: XCOPY(LID1) first introduced in SPC-2 (standardized in 2001), and ODX which is a subset of XCOPY(LID4) first introduced in SPC-4 draft (revision 34, 2012).
This section is about protection information which is typically an extra 8 bytes associated with each logical block. Those 8 byte are divided into 3 fields: logical block guard (16 bit (2 byte) CRC), logical block application tag (2 bytes) and the logical block reference tag (4 bytes). The acronym DIF is sometimes used for protection information.
The feature to read and/or write protection information by using the protect=RDP[,WRP] option is currently experimental. It should be used with care and may not "play well" with some other features such as write sparing and sparse writing. It should be used to copy user data plus the associated protection information to or from a regular file. It could also be used for a device to device copy assuming the "pt" interface is used for both. Also only modern SCSI disks support protection information.
When RDP or WRP is greater than 0 then a copy with associated protection information is active. In this state IBS and OBS must be the same and equal to the logical block size of the device(s) formatted with protection information. If a SCSI disk with 512 byte logical block size has protection information then the actual number of bytes transferred for each logical block is typically 520 bytes. For such a disk BS=512 is required even when additional protection information is being transferred.
By default numeric arguments to options are assumed to be decimal. Almost all numeric arguments to options (e.g. COUNT in the count=COUNT option) may include one of these multiplicative suffixes: c C *1; w W *2; b B *512; k K KiB *1,024; KB *1,000; m M MiB *1,048,576; MB *1,000,000 . This pattern continues for "G", "T" and "P". The latter two suffixes can only be used for 64 bit values. Some numeric arguments are limited to 32 bit values (e.g. BSin the bs=BS option). Also a suffix of the form "x<n>" multiplies the leading number by <n>; however the combinations "0x" and "0X" are treated differently, see the next paragraph. These multiplicative suffixes are compatible with GNU's dd command (since 2002) which claims compliance with the SI and with IEC 60027-2 standards.
Alternatively numerical values can be given in hexadecimal indicated by either a leading "0x" or "0X", or by a trailing "h" or "H". When hex numbers are given, suffix multipliers cannot be used.
If a numeric argument is required to fit in 32 bits and is too large then an error is reported. Usually negative numbers are not permitted but "count=-1" is a special case and means "all available"; "verbose=-1" is another special case.
Copying data behind an Operating System's back can cause problems.
In the case of Linux, users should look at this link:
http://linux-mm.org/Drop_Caches
This command sequence may be useful:
sync; echo 3 > /proc/sys/vm/drop_caches
A partial write is a write to the OFILE of less than OBS bytes. This typically occurs at the end of a copy. dd can do partial writes. ddpt does partial writes to regular files and fifos (including stdout). However ddpt ignores partial writes when OFILE is a block device or a pt device. When ddpt ignores a partial write, it sends a warning to the console (stderr).
At the end of the copy two lines are reported to the console:
<in_full>+<in_partial> records in
<out_full>+<out_partial> records out
The "records in" line is the number of full input blocks (each of IBS bytes) that have been read plus the number of partial blocks ( usually less than IBS bytes) that have been read. Following the lead of dd when 'iflag=coe' is active a block that cannot be read (and has zeros substituted for its output) is regarded as a partial read. The "records out" line is the number of full output blocks (each of OBS bytes) that have been written plus the number of partial blocks (usually less than OBS bytes) that have been written.
Block devices (e.g. /dev/sda and /dev/hda) can be given for IFILE. If neither 'iflag=direct' nor 'iflag=pt' is given then normal block IO involving buffering and caching is performed. If 'iflag=direct' is given then the buffering and caching is bypassed (this is applicable to both SCSI devices and ATA disks). When 'iflag=pt' is given SCSI commands are sent to the device which bypasses most of the actions performed by the block layer. The same applies for block devices given for OFILE.
All informative, warning and error reports are sent to stderr so that dd's output file can be stdout and remain unpolluted. If no options are given, then no copying (nor reading) takes place and a brief message is sent to stderr inviting the user to invoke ddpt again but with '--help' option to get the usage message.
Disk partition information can often be found with fdisk(8) [the "-ul" argument is useful in this respect]. Also parted(8) can be used like this: 'parted /dev/sda unit s print' .
For pt devices this utility issues SCSI READ and WRITE (SBC) commands which are appropriate for disks and reading from CD/DVD/BD drives. Those commands are not formatted correctly for tape drives so ddpt cannot be used on tape drives via a pt device. If the largest block address of the requested transfer exceeds a 32 bit block number (i.e 0xffffffff) then a warning is issued and the pt device is accessed via SCSI READ(16) and WRITE(16) commands.
The attributes of a block device (e.g. partitions) are ignored when the pt flag is used. Hence the whole device is read (rather than just the second partition) by this invocation:
ddpt if=/dev/sdb2 iflag=pt of=t bs=512
Assuming /dev/sdb and /dev/sg2 refer to the same device, then after the following two invocations, the contents of the files "t", "tt" and "ttt" should be same:
ddpt if=/dev/sdb of=tt bs=512
ddpt if=/dev/sg2 of=ttt bs=512
The SCSI READ(32) and WRITE(32) commands are restricted to media that is formatted with protection type 2. This is a T10 restriction.
The signal handling has been borrowed from GNU's dd: SIGINT, SIGQUIT and SIGPIPE report the number of remaining blocks to be transferred and the records in + out counts; then they have their default action. SIGUSR1 (or SIGINFO) causes the same information to be output and the copy continues. All output caused by signals is sent to stderr.
Like GNU's dd, ddpt respects the signal disposition of "ignored" (SIG_IGN) set by the shell, script or other program that invokes ddpt. So in that case it will ignore such signals. Further dd ignores SIGUSR1 if the environment variable POSIXLY_CORRECT is set because POSIX defines dd will only act on SIGINFO (and Linux has no such signal); ddpt ignores the POSIXLY_CORRECT environment variable. As recommended by Susv3, ddpt does not expect the signal (blocking) mask to be blocking SIGUSR1 (SIGINFO), SIGINT or SIGPIPE on entry.
Unix system calls that do IO can be interrupted by signal processing, typically returning an EINTR error number. The dd utility (and many other Unix utilities) restart the IO operation that was interrupted. While this will work most of the time for disk IO it is problematic for tape drives because the implicit position pointer on the tape may have moved. So the default (i.e. "intio=0") in this utility is to mask those signals during IO operations and only check them prior to starting an IO operation. Most low level IO (e.g. using SCSI command to write to a disk) will timeout if there is a low level error. However NFS (the Network File System) will potentially wait for a long time (e.g. expecting a network problem will soon be fixed) and in this case using "intio=1" may be best.
There is support for copies to and from tape drives in Linux. Only the st driver device names can be used (e.g. /dev/st0 and /dev/nst2). Hence use of Linux pass-through device names (e.g. /dev/sg2) for tape drives is not supported. On Debian-based distributions, it is suggested that the mt-st package is installed as it provides a more fully-featured version of the "mt" tape control program.
Tape drives can operate in fixed- or variable-length block modes. In variable-block mode, each write to the tape writes a single block of that size. In fixed-block mode, each write to the tape must be a multiple of the previously-selected block size.
The block size/mode can be set with the mt command prior to
invoking ddpt. For example:
# mt -f /dev/nst0 setblk 0
sets variable-block mode, and
# mt -f /dev/nst0 setblk 32768
sets fixed-block mode with block size 32768 bytes.
Note that some tape drives support only fixed-block mode, and possibly even only one block size. (For example, QIC-150 tapes use a fixed block size of 512 bytes.) There may also be restrictions on the block size, e.g. it may have to be even.
When using ddpt to write to tape, if the final read from the input is less than OBS, it is padded to OBS bytes before writing to tape to ensure that all blocks of the tape file are the same length. Having a shorter final block would fail if the drive is in fixed-block mode, and could create interchange problems. It is common to expect all blocks in a file on tape to be the same length. However, to tell ddpt to not pad the final block, use 'oflag=nopad'.
The st tape driver normally writes a filemark when the file (e.g.
/dev/nst0) is closed. To not have the filemark written, use 'oflag=nofm'.
One use case for that might be if using ddpt several times in succession to
append more data to the same file on tape. In that case it is probably
desirable to write the filemark at the end of the sequence. So either omit
'oflag=nofm' on the last ddpt invocation, or manually write a filemark using
mt after ddpt exits:
# mt -f /dev/nst0 weof 1
For reading from an unknown tape where the block size(s) is not
known, read in variable-block mode specifying a large IBS. The st
driver returns a smaller amount of data if the size of the block read is
smaller. Thus a command like:
# ddpt if=/dev/nst0 of=output.bin bs=262144
should read the file from tape regardless of the block size used (assuming no
blocks are larger than 256KB). ddpt's verbose option will display what the
actual block size(s) is.
If the command line invocation of an xcopy does not explicitly (and unambiguously) indicate whether the XCOPY SCSI command should be sent to IFILE (i.e. the source) or OFILE (i.e. the destination) then a check is made for the presence of the XCOPY_TO_SRC and XCOPY_TO_DST environment variables. If either one exists (but not both) then it indicates where the SCSI XCOPY command will be sent. By default the XCOPY command is sent to OFILE.
The ODX write from tokens variant is very complex to implement if the amount of data held in each ROD is not known. The value should be found in the "number of bytes represented" field in the ROD Token but that is not well supported yet by vendors. So for such case that number can be appended as a big endian 8 byte integer following each ROD Token in the RTF file. The conv=rtf_len will cause that length to be appended. Specifying that option on each read to tokens and write from tokens invocation can be a nuisance. Setting the environment variable ODX_RTF_LEN will cause this utility to act as if the conv=rtf_len option has been given.
Sometimes the default block size of 512 can be a nuisance. This can be overridden by the value associated with the DDPT_DEF_BS environment variable. If the environment variable is not found, the value cannot be decoded or is zero or less, then the default block size remains at 512 bytes.
To aid scripts that call ddpt, the exit status is set to indicate success (0) or failure (1 or more). Note that some of the lower values correspond to the SCSI sense key values. The exit status values are:
The examples in this page use Linux device names. For suitable device names in other supported Operating Systems see this web page: http://sg.danny.cz/sg/device_name.html . The sg3_utils(8) man page in the sg3_utils package also covers device naming.
ddpt usage looks quite similar to dd:
ddpt if=/dev/sg0 of=t bs=512 count=1MB
This will copy 1 million 512 byte blocks from the device associated with /dev/sg0 (which should have 512 byte blocks) to a file called t. Assuming /dev/sda and /dev/sg0 are the same device then the above is equivalent to:
dd if=/dev/sda iflag=direct of=t bs=512 count=1000000
although dd's speed may improve if bs was larger and count was suitably reduced. The use of the 'iflag=direct' option bypasses the buffering and caching that is usually done on a block device.
The dd command's bs argument can be thought of as roughly equivalent to ddpt's bs*bpt . dd almost assumes buffering on a block device and will work as long as bs is a multiple of the actual logical block size. Since ddpt can work at a lower level in some cases the bs argument must be a disk's actual logical block size. Thus the bpt argument was introduced to make the copy more efficient. So these two invocations are roughly equivalent:
dd if=/dev/sda of=t bs=8k count=64
ddpt if=/dev/sda of=t bs=512 bpt=16 count=1k
In both cases the total number of bytes moved is bs*count . And that will be done by reading 8k (8192 bytes) into a buffer then writing out that buffer to the file t. The read write sequence continues until the count is complete or an error occurs.
The 'of2=' option can save time when the input would otherwise need to be read twice. For example, to copy data and take a md5sum of it without needing to re-read the data:
mkfifo fif
md5sum fif &
ddpt if=/dev/sg3 iflag=coe of=sg3.img oflag=sparse of2=fif bs=512
This will image /dev/sg3 (e.g. an unmounted disk) and place the contents in the (sparse) file sg3.img . Without re-reading the data it will also perform a md5sum calculation on the image.
Now we use sparse writing logic to get some idea of how many blocks on a disk are full of zeros. After a SCSI FORMAT UNIT command or an ATA SECURITY ERASE command a disk may be all zeros.
ddpt if=/dev/sdc bs=512 oflag=sparse
Since no "of=" option is given, output goes to /dev/null so nothing is actually written so the "records out" will be zero. However there will be a count of "records in" and "bypassed records out". If /dev/sdc is full of zeros then "records in" and "bypassed records out" will be the same. Since the "bpt=" option is not given it defaults to "bpt=128,128" so the copy buffer will be 64 KiB and the sparse check for zeros will be done with 64 KiB (128 block) granularity.
For examples of the trim and self,trim options see the section above on TRIM, UNMAP AND WRITE SAME.
Following is an example run on a Windows OS using the '--wscan' option which shows the available device names (e.g. PD1) and the associated volume name(s):
ddpt -w
PD0 [C] FUJITSU MHY2160BH 0000
PD1 [DF] WD 2500BEV External 1.05 WD-WXE90
CDROM0 [E] MATSHITA DVD/CDRW UJDA775 CB03
So, for example, volumes D: and F: reside on PhysicalDisk1 (abbreviated to "PD1") which is manufactured by WD (Western Digital).
Further examples can be found on this web page: http://sg.danny.cz/sg/ddpt.html . There is a text file containing examples called ddpt_examples.txt in the "doc" directory of this package's distribution tarball. The ddpt_examples.txt file contains some examples of using job files.
Written by Doug Gilbert
Report bugs to <dgilbert at interlog dot com>.
Copyright © 2008-2014 Douglas Gilbert
This software is distributed under the GPL version 2. There is NO warranty;
not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
This utility has a companion/helper utility called
ddptctl(8)
There is a web page discussing ddpt at http://sg.danny.cz/sg/ddpt.html
The lmbench package contains lmdd which is also interesting. For moving data to and from tapes see dt which is found at http://www.scsifaq.org/RMiller_Tools/index.html
To change mode parameters that effect a SCSI device's caching and error recovery see sdparm(sdparm)
To verify the data on the media or to verify it against some other copy of the data see sg_verify(sg3_utils)
To scan and repair disk partitions see TestDisk (testdisk).
Additional references: dd(1), open(2), flock(2), sg_xcopy,sg_copy_results, sg_dd(sg3_utils)
December 2014 | ddpt-0.95 |