Note: more information on submitting HTCondor jobs can be found
here: 2.5.
As of version 8.5.6, the condor_submitlanguage supports multi-line
values in commands. The syntax is the same as the configuration language
(see more details here: 3.3.5).
Each submit description file describes one or more clusters of
jobs to be placed in the HTCondor execution pool. All jobs in a cluster must
share the same executable, but they may have different input and output
files, and different program arguments. The submit description file is
generally the last command-line argument to condor_submit. If the submit
description file argument is omitted, condor_submitwill read the submit
description from standard input.
The submit description file must contain at least one
executablecommand and at least one queuecommand. All of the other commands
have default actions.
Note that a submit file that contains more than one executable
command will produce multiple clusters when submitted. This is not generally
recommended, and is not allowed for submit files that are run as DAG node
jobs by condor_dagman.
The commands which can appear in the submit description file are
numerous. They are listed here in alphabetical order by category.
BASIC COMMANDS
-
-
-
arguments = <argument_list>
-
- List of arguments to be supplied to the executable as part of the command
line.
-
- In the javauniverse, the first argument must be the name of the class
containing main.
-
- There are two permissible formats for specifying arguments, identified as
the old syntax and the new syntax. The old syntax supports white space
characters within arguments only in special circumstances; when used, the
command line arguments are represented in the job ClassAd attribute Args.
The new syntax supports uniform quoting of white space characters within
arguments; when used, the command line arguments are represented in the
job ClassAd attribute Arguments.
-
- Old Syntax
-
- In the old syntax, individual command line arguments are delimited
(separated) by space characters. To allow a double quote mark in an
argument, it is escaped with a backslash; that is, the two character
sequence \" becomes a single double quote mark within an
argument.
-
- Further interpretation of the argument string differs depending on the
operating system. On Windows, the entire argument string is passed
verbatim (other than the backslash in front of double quote marks) to the
Windows application. Most Windows applications will allow spaces within an
argument value by surrounding the argument with double quotes marks. In
all other cases, there is no further interpretation of the arguments.
-
- Example:
-
arguments = one \"two\" 'three'
-
- Produces in Unix vanilla universe:
-
argument 1: one
argument 2: "two"
argument 3: 'three'
-
- New Syntax
-
- Here are the rules for using the new syntax:
-
-
- 1. The entire string representing the command line arguments is surrounded
by double quote marks. This permits the white space characters of spaces
and tabs to potentially be embedded within a single argument. Putting the
double quote mark within the arguments is accomplished by escaping it with
another double quote mark.
-
-
- 2. The white space characters of spaces or tabs delimit arguments.
-
-
- 3. To embed white space characters of spaces or tabs within a single
argument, surround the entire argument with single quote marks.
-
-
- 4. To insert a literal single quote mark, escape it within an argument
already delimited by single quote marks by adding another single quote
mark.
-
-
- Example:
arguments = "3 simple arguments"
-
- Produces:
argument 1: 3
argument 2: simple
argument 3: arguments
-
- Another example:
arguments = "one 'two with spaces' 3"
-
- Produces:
argument 1: one
argument 2: two with spaces
argument 3: 3
-
- And yet another example:
arguments = "one ""two"" 'spacey
”quoted” argument'"
-
- Produces:
argument 1: one
argument 2: "two"
argument 3: spacey 'quoted' argument
-
- Notice that in the new syntax, the backslash has no special meaning. This
is for the convenience of Windows users.
-
-
environment = <parameter_list>
-
- List of environment variables.
-
- There are two different formats for specifying the environment variables:
the old format and the new format. The old format is retained for
backward-compatibility. It suffers from a platform-dependent syntax and
the inability to insert some special characters into the environment.
-
- The new syntax for specifying environment values:
-
-
- 1. Put double quote marks around the entire argument string. This
distinguishes the new syntax from the old. The old syntax does not have
double quote marks around it. Any literal double quote marks within the
string must be escaped by repeating the double quote mark.
-
-
- 2. Each environment entry has the form
-
<name>=<value>
-
-
- 3. Use white space (space or tab characters) to separate environment
entries.
-
-
- 4. To put any white space in an environment entry, surround the space and
as much of the surrounding entry as desired with single quote marks.
-
-
- 5. To insert a literal single quote mark, repeat the single quote mark
anywhere inside of a section surrounded by single quote marks.
-
-
- Example:
-
environment = "one=1 two=""2""
three='spacey ”quoted” value'"
-
- Produces the following environment entries:
-
one=1
two="2"
three=spacey 'quoted' value
-
- Under the old syntax, there are no double quote marks surrounding the
environment specification. Each environment entry remains of the form
<name>=<value>
-
- Under Unix, list multiple environment entries by separating them with a
semicolon (;). Under Windows, separate multiple entries with a vertical
bar (|). There is no way to insert a literal semicolon under Unix or a
literal vertical bar under Windows. Note that spaces are accepted, but
rarely desired, characters within parameter names and values, because they
are treated as literal characters, not separators or ignored white space.
Place spaces within the parameter list only if required.
-
- A Unix example:
-
environment = one=1;two=2;three="quotes have no 'special'
meaning"
-
- This produces the following:
-
one=1
two=2
three="quotes have no 'special' meaning"
-
- If the environment is set with the environmentcommand andgetenvis also set
to true, values specified with environmentoverride values in the
submitter's environment (regardless of the order of the environmentand
getenvcommands).
-
-
error = <pathname>
-
- A path and file name used by HTCondor to capture any error messages the
program would normally write to the screen (that is, this file becomes
stderr). A path is given with respect to the file system of the machine on
which the job is submitted. The file is written (by the job) in the remote
scratch directory of the machine where the job is executed. When the job
exits, the resulting file is transferred back to the machine where the job
was submitted, and the path is utilized for file placement. If not
specified, the default value of /dev/nullis used for submission to a Unix
machine. If not specified, error messages are ignored for submission to a
Windows machine. More than one job should not use the same error file,
since this will cause one job to overwrite the errors of another. The
error file and the output file should not be the same file as the outputs
will overwrite each other or be lost. For grid universe jobs, errormay be
a URL that the Globus tool globus_url_copyunderstands.
-
-
executable = <pathname>
-
- An optional path and a required file name of the executable file for this
job cluster. Only one executablecommand within a submit description file
is guaranteed to work properly. More than one often works.
-
- If no path or a relative path is used, then the executable file is
presumed to be relative to the current working directory of the user as
the condor_submit command is issued.
-
- If submitting into the standard universe, then the named executable must
have been re-linked with the HTCondor libraries (such as via the
condor_compilecommand). If submitting into the vanilla universe (the
default), then the named executable need not be re-linked and can be any
process which can run in the background (shell scripts work fine as well).
If submitting into the Java universe, then the argument must be a compiled
.classfile.
-
-
getenv = <True |False>
-
- If getenvis set to True, then condor_submit will copy all of the user's
current shell environment variables at the time of job submission into the
job ClassAd. The job will therefore execute with the same set of
environment variables that the user had at submit time. Defaults to
False.
-
- If the environment is set with the environmentcommand andgetenvis also set
to true, values specified with environmentoverride values in the
submitter's environment (regardless of the order of the environmentand
getenvcommands).
-
-
input = <pathname>
-
- HTCondor assumes that its jobs are long-running, and that the user will
not wait at the terminal for their completion. Because of this, the
standard files which normally access the terminal, (stdin, stdout, and
stderr), must refer to files. Thus, the file name specified with
inputshould contain any keyboard input the program requires (that is, this
file becomes stdin). A path is given with respect to the file system of
the machine on which the job is submitted. The file is transferred before
execution to the remote scratch directory of the machine where the job is
executed. If not specified, the default value of /dev/nullis used for
submission to a Unix machine. If not specified, input is ignored for
submission to a Windows machine. For grid universe jobs, inputmay be a URL
that the Globus tool globus_url_copyunderstands.
-
- Note that this command does notrefer to the command-line arguments of the
program. The command-line arguments are specified by the
argumentscommand.
-
-
log = <pathname>
-
- Use logto specify a file name where HTCondor will write a log file of what
is happening with this job cluster, called a job event log. For example,
HTCondor will place a log entry into this file when and where the job
begins running, when the job produces a checkpoint, or moves (migrates) to
another machine, and when the job completes. Most users find specifying a
logfile to be handy; its use is recommended. If no logentry is specified,
HTCondor does not create a log for this cluster. If a relative path is
specified, it is relative to the current working directory as the job is
submitted or the directory specified by submit command initialdiron the
submit machine.
-
-
log_xml = <True |False>
-
- If log_xmlis True, then the job event log file will be written in ClassAd
XML. If not specified, XML is not used. Note that the file is an XML
fragment; it is missing the file header and footer. Do not mix XML and
non-XML within a single file. If multiple jobs write to a single job event
log file, ensure that all of the jobs specify this option in the same
way.
-
-
notification = <Always |Complete |Error |Never>
-
- Owners of HTCondor jobs are notified by e-mail when certain events occur.
If defined by Always, the owner will be notified whenever the job produces
a checkpoint, as well as when the job completes. If defined by Complete,
the owner will be notified when the job terminates. If defined by Error,
the owner will only be notified if the job terminates abnormally, or if
the job is placed on hold because of a failure, and not by user request.
If defined by Never(the default), the owner will not receive e-mail,
regardless to what happens to the job. The HTCondor User's manual
documents statistics included in the e-mail.
-
-
notify_user = <email-address>
-
- Used to specify the e-mail address to use when HTCondor sends e-mail about
a job. If not specified, HTCondor defaults to using the e-mail address
defined by
job-owner@UID_DOMAIN
-
- where the configuration variable UID_DOMAINis specified by the HTCondor
site administrator. If UID_DOMAINhas not been specified, HTCondor sends
the e-mail to:
job-owner@submit-machine-name
-
-
output = <pathname>
-
- The outputfile captures any information the program would ordinarily write
to the screen (that is, this file becomes stdout). A path is given with
respect to the file system of the machine on which the job is submitted.
The file is written (by the job) in the remote scratch directory of the
machine where the job is executed. When the job exits, the resulting file
is transferred back to the machine where the job was submitted, and the
path is utilized for file placement. If not specified, the default value
of /dev/nullis used for submission to a Unix machine. If not specified,
output is ignored for submission to a Windows machine. Multiple jobs
should not use the same output file, since this will cause one job to
overwrite the output of another. The output file and the error file should
not be the same file as the outputs will overwrite each other or be lost.
For grid universe jobs, outputmay be a URL that the Globus tool
globus_url_copyunderstands.
-
- Note that if a program explicitly opens and writes to a file, that file
should notbe specified as the outputfile.
-
-
priority = <integer>
-
- An HTCondor job priority can be any integer, with 0 being the default.
Jobs with higher numerical priority will run before jobs with lower
numerical priority. Note that this priority is on a per user basis. One
user with many jobs may use this command to order his/her own jobs, and
this will have no effect on whether or not these jobs will run ahead of
another user's jobs.
-
- Note that the priority setting in an HTCondor submit file will be
overridden by condor_dagmanif the submit file is used for a node in a DAG,
and the priority of the node within the DAG is non-zero (see 2.10.9for
more details).
-
-
queue [<int expr>]
-
- Places zero or more copies of the job into the HTCondor queue.
-
queue
-
- [<int expr>] [<varname>] in[slice] <list of items>Places
zero or more copies of the job in the queue based on items in a <list
of items>
-
queue
-
- [<int expr>] [<varname>] matching[files | dirs] [slice]
<list of items with file globbing>] Places zero or more copies of
the job in the queue based on files that match a <list of items with
file globbing>
-
queue
-
- [<int expr>] [<list of varnames>] from[slice] <file
name> | <list of items>] Places zero or more copies of the job in
the queue based on lines from the submit file or from <file
name>
-
- The optional argument <int expr>specifies how many times to repeat
the job submission for a given set of arguments. It may be an integer or
an expression that evaluates to an integer, and it defaults to 1. All but
the first form of this command are various ways of specifying a list of
items. When these forms are used <int expr>jobs will be queued for
each item in the list. The in, matchingand fromkeyword indicates how the
list will be specified.
-
- * inThe list of items is an explicit comma and/or space separated <list
of items>. If the <list of items>begins with an open paren, and
the close paren is not on the same line as the open, then the list
continues until a line that begins with a close paren is read from the
submit file.
-
- * matchingEach item in the <list of items with file globbing>will be
matched against the names of files and directories relative to the current
directory, the set of matching names is the resulting list of items.
-
- * filesOnly filenames will matched.
-
- * dirsOnly directory names will be matched.
-
- * from<file name> | <list of items>Each line from <file
name>or <list of items>is a single item, this allows for multiple
variables to be set for each item. Lines from <file name>or <list
of items>will be split on comma and/or space until there are values for
each of the variables specified in <list of varnames>. The last
variable will contain the remainder of the line. When the <list of
items>form is used, the list continues until the first line that begins
with a close paren, and lines beginning with pound sign ('#') will be
skipped. When using the <file name>form, if the <file
name>ends with |, then it will be executed as a script whatever the
script writes to stdoutwill be the list of items. The optional argument
<varname>or <list of varnames>is the name or names of of
variables that will be set to the value of the current item when queuing
the job. If no <varname>is specified the variable ITEM will be used.
Leading and trailing whitespace be trimmed. The optional argument
<slice>is a python style slice selecting only some of the items in
the list of items. Negative step values are not supported.
-
- A submit file may contain more than one queuestatement, and if desired,
any commands may be placed between subsequent queuecommands, such as new
input, output, error, initialdir, or argumentscommands. This is handy when
submitting multiple runs into one cluster with one submit description
file.
-
-
universe = <vanilla |standard |scheduler |local |grid |java
|vm |parallel |docker>
-
- Specifies which HTCondor universe to use when running this job. The
HTCondor universe specifies an HTCondor execution environment.
-
- The vanillauniverse is the default (except where the configuration
variable DEFAULT_UNIVERSEdefines it otherwise), and is an execution
environment for jobs which do not use HTCondor's mechanisms for taking
checkpoints; these are ones that have not been linked with the HTCondor
libraries. Use the vanillauniverse to submit shell scripts to
HTCondor.
-
- The standarduniverse tells HTCondor that this job has been re-linked via
condor_compilewith the HTCondor libraries and therefore supports taking
checkpoints and remote system calls.
-
- The scheduleruniverse is for a job that is to run on the machine where the
job is submitted. This universe is intended for a job that acts as a
metascheduler and will not be preempted.
-
- The localuniverse is for a job that is to run on the machine where the job
is submitted. This universe runs the job immediately and will not preempt
the job.
-
- The griduniverse forwards the job to an external job management system.
Further specification of the griduniverse is done with the
grid_resourcecommand.
-
- The javauniverse is for programs written to the Java Virtual Machine.
-
- The vmuniverse facilitates the execution of a virtual machine.
-
- The paralleluniverse is for parallel jobs (e.g. MPI) that require multiple
machines in order to run.
-
- The dockeruniverse runs a docker container as an HTCondor job.
-
-
COMMANDS FOR MATCHMAKING
-
-
-
rank = <ClassAd Float Expression>
-
- A ClassAd Floating-Point expression that states how to rank machines which
have already met the requirements expression. Essentially, rank expresses
preference. A higher numeric value equals better rank. HTCondor will give
the job the machine with the highest rank. For example,
request_memory = max({60, Target.TotalSlotMemory})
rank = Memory
-
- asks HTCondor to find all available machines with more than 60 megabytes
of memory and give to the job the machine with the most amount of memory.
The HTCondor User's Manual contains complete information on the syntax and
available attributes that can be used in the ClassAd expression.
-
-
request_cpus = <num-cpus>
-
- A requested number of CPUs (cores). If not specified, the number requested
will be 1. If specified, the expression
&& (RequestCpus <= Target.Cpus)
-
- is appended to the requirementsexpression for the job.
-
- For pools that enable dynamic condor_startdprovisioning, specifies the
minimum number of CPUs requested for this job, resulting in a dynamic slot
being created with this many cores.
-
-
request_disk = <quantity>
-
- The requested amount of disk space in KiB requested for this job. If not
specified, it will be set to the job ClassAd attribute DiskUsage. The
expression
&& (RequestDisk <= Target.Disk)
-
- is appended to the requirementsexpression for the job.
-
- For pools that enable dynamic condor_startdprovisioning, a dynamic slot
will be created with at least this much disk space.
-
- Characters may be appended to a numerical value to indicate units. Kor
KBindicates KiB, numbers of bytes. Mor MBindicates MiB, numbers of bytes.
Gor GBindicates GiB, numbers of bytes. Tor TBindicates TiB, numbers of
bytes.
-
-
request_memory = <quantity>
-
- The required amount of memory in MiB that this job needs to avoid
excessive swapping. If not specified and the submit command vm_memoryis
specified, then the value specified for vm_memorydefines request_memory.
If neither request_memorynor vm_memoryis specified, the value is set by
the configuration variable JOB_DEFAULT_REQUESTMEMORY. The actual amount of
memory used by a job is represented by the job ClassAd attribute
MemoryUsage.
-
- For pools that enable dynamic condor_startdprovisioning, a dynamic slot
will be created with at least this much RAM.
-
- The expression
&& (RequestMemory <= Target.Memory)
-
- is appended to the requirementsexpression for the job.
-
- Characters may be appended to a numerical value to indicate units. Kor
KBindicates KiB, numbers of bytes. Mor MBindicates MiB, numbers of bytes.
Gor GBindicates GiB, numbers of bytes. Tor TBindicates TiB, numbers of
bytes.
-
-
request_<name> = <quantity>
-
- The required amount of the custom machine resource identified by
<name>that this job needs. The custom machine resource is defined in
the machine's configuration. Machines that have available GPUs will define
<name>to be GPUs.
-
-
requirements = <ClassAd Boolean Expression>
-
- The requirements command is a boolean ClassAd expression which uses C-like
operators. In order for any job in this cluster to run on a given machine,
this requirements expression must evaluate to true on the given
machine.
-
- For scheduler and local universe jobs, the requirements expression is
evaluated against the SchedulerClassAd which represents the the
condor_schedddaemon running on the submit machine, rather than a remote
machine. Like all commands in the submit description file, if multiple
requirements commands are present, all but the last one are ignored. By
default, condor_submit appends the following clauses to the requirements
expression:
-
- 1. Arch and OpSys are set equal to the Arch and OpSys of the submit
machine. In other words: unless you request otherwise, HTCondor will give
your job machines with the same architecture and operating system version
as the machine running condor_submit .
-
- 2. Cpus >=RequestCpus, if the job ClassAd attribute RequestCpusis
defined.
-
- 3. Disk >=RequestDisk, if the job ClassAd attribute RequestDiskis
defined. Otherwise, Disk >=DiskUsage is appended to the requirements.
The DiskUsageattribute is initialized to the size of the executable plus
the size of any files specified in a transfer_input_filescommand. It
exists to ensure there is enough disk space on the target machine for
HTCondor to copy over both the executable and needed input files. The
DiskUsageattribute represents the maximum amount of total disk space
required by the job in kilobytes. HTCondor automatically updates the
DiskUsageattribute approximately every 20 minutes while the job runs with
the amount of space being used by the job on the execute machine.
-
- 4. Memory >=RequestMemory, if the job ClassAd attribute RequestMemoryis
defined.
-
- 5. If Universe is set to Vanilla, FileSystemDomain is set equal to the
submit machine's FileSystemDomain. View the requirements of a job which
has already been submitted (along with everything else about the job
ClassAd) with the command condor_q -l; see the command reference for
condor_qon page . Also, see the HTCondor Users Manual for complete
information on the syntax and available attributes that can be used in the
ClassAd expression.
-
-
FILE TRANSFER COMMANDS
-
-
-
dont_encrypt_input_files = < file1,file2,file...
>
-
- A comma and/or space separated list of input files that are notto be
network encrypted when transferred with the file transfer mechanism.
Specification of files in this manner overrides configuration that would
use encryption. Each input file must also be in the list given by
transfer_input_files. When a path to an input file or directory is
specified, this specifies the path to the file on the submit side. A
single wild card character (*) may be used in each file name.
-
-
dont_encrypt_output_files = < file1,file2,file...
>
-
- A comma and/or space separated list of output files that are notto be
network encrypted when transferred back with the file transfer mechanism.
Specification of files in this manner overrides configuration that would
use encryption. The output file(s) must also either be in the list given
by transfer_output_filesor be discovered and to be transferred back with
the file transfer mechanism. When a path to an output file or directory is
specified, this specifies the path to the file on the execute side. A
single wild card character (*) may be used in each file name.
-
-
encrypt_execute_directory = <True |False>
-
- Defaults to False. If set to True, HTCondor will encrypt the contents of
the remote scratch directory of the machine where the job is executed.
This encryption is transparent to the job itself, but ensures that files
left behind on the local disk of the execute machine, perhaps due to a
system crash, will remain private. In addition, condor_submit will append
to the job's requirementsexpression
&& (TARGET.HasEncryptExecuteDirectory)
-
- to ensure the job is matched to a machine that is capable of encrypting
the contents of the execute directory. This support is limited to Windows
platforms that use the NTFS file system and Linux platforms with the
ecryptfs-utilspackage installed.
-
-
encrypt_input_files = < file1,file2,file... >
-
- A comma and/or space separated list of input files that are to be network
encrypted when transferred with the file transfer mechanism. Specification
of files in this manner overrides configuration that would notuse
encryption. Each input file must also be in the list given by
transfer_input_files. When a path to an input file or directory is
specified, this specifies the path to the file on the submit side. A
single wild card character (*) may be used in each file name. The method
of encryption utilized will be as agreed upon in security negotiation; if
that negotiation failed, then the file transfer mechanism must also fail
for files to be network encrypted.
-
-
encrypt_output_files = < file1,file2,file... >
-
- A comma and/or space separated list of output files that are to be network
encrypted when transferred back with the file transfer mechanism.
Specification of files in this manner overrides configuration that would
notuse encryption. The output file(s) must also either be in the list
given by transfer_output_filesor be discovered and to be transferred back
with the file transfer mechanism. When a path to an output file or
directory is specified, this specifies the path to the file on the execute
side. A single wild card character (*) may be used in each file name. The
method of encryption utilized will be as agreed upon in security
negotiation; if that negotiation failed, then the file transfer mechanism
must also fail for files to be network encrypted.
-
-
max_transfer_input_mb = <ClassAd Integer
Expression>
-
- This integer expression specifies the maximum allowed total size in MiB of
the input files that are transferred for a job. This expression does
notapply to grid universe, standard universe, or files transferred via
file transfer plug-ins. The expression may refer to attributes of the job.
The special value -1 indicates no limit. If not defined, the value set by
configuration variable MAX_TRANSFER_INPUT_MBis used. If the observed size
of all input files at submit time is larger than the limit, the job will
be immediately placed on hold with a HoldReasonCodevalue of 32. If the job
passes this initial test, but the size of the input files increases or the
limit decreases so that the limit is violated, the job will be placed on
hold at the time when the file transfer is attempted.
-
-
max_transfer_output_mb = <ClassAd Integer
Expression>
-
- This integer expression specifies the maximum allowed total size in MiB of
the output files that are transferred for a job. This expression does
notapply to grid universe, standard universe, or files transferred via
file transfer plug-ins. The expression may refer to attributes of the job.
The special value -1 indicates no limit. If not set, the value set by
configuration variable MAX_TRANSFER_OUTPUT_MBis used. If the total size of
the job's output files to be transferred is larger than the limit, the job
will be placed on hold with a HoldReasonCodevalue of 33. The output will
be transferred up to the point when the limit is hit, so some files may be
fully transferred, some partially, and some not at all.
-
-
output_destination = <destination-URL>
-
- When present, defines a URL that specifies both a plug-in and a
destination for the transfer of the entire output sandbox or a subset of
output files as specified by the submit command transfer_output_files. The
plug-in does the transfer of files, and no files are sent back to the
submit machine. The HTCondor Administrator's manual has full details.
-
-
should_transfer_files = <YES |NO |IF_NEEDED >
-
- The should_transfer_filessetting is used to define if HTCondor should
transfer files to and from the remote machine where the job runs. The file
transfer mechanism is used to run jobs which are not in the standard
universe (and can therefore use remote system calls for file access) on
machines which do not have a shared file system with the submit machine.
should_transfer_filesequal to YESwill cause HTCondor to always transfer
files for the job. NOdisables HTCondor's file transfer mechanism.
IF_NEEDEDwill not transfer files for the job if it is matched with a
resource in the same FileSystemDomainas the submit machine (and therefore,
on a machine with the same shared file system). If the job is matched with
a remote resource in a different FileSystemDomain, HTCondor will transfer
the necessary files.
-
- For more information about this and other settings related to transferring
files, see the HTCondor User's manual section on the file transfer
mechanism.
-
- Note that should_transfer_filesis not supported for jobs submitted to the
grid universe.
-
-
skip_filechecks = <True |False>
-
- When True, file permission checks for the submitted job are disabled. When
False, file permissions are checked; this is the behavior when this
command is not present in the submit description file. File permissions
are checked for read permissions on all input files, such as those defined
by commands inputand transfer_input_files, and for write permission to
output files, such as a log file defined by logand output files defined
with outputor transfer_output_files.
-
-
stream_error = <True |False>
-
- If True, then stderris streamed back to the machine from which the job was
submitted. If False, stderris stored locally and transferred back when the
job completes. This command is ignored if the job ClassAd attribute
TransferErris False. The default value is False. This command must be used
in conjunction with error, otherwise stderrwill sent to /dev/nullon Unix
machines and ignored on Windows machines.
-
-
stream_input = <True |False>
-
- If True, then stdinis streamed from the machine on which the job was
submitted. The default value is False. The command is only relevant for
jobs submitted to the vanilla or java universes, and it is ignored by the
grid universe. This command must be used in conjunction with input,
otherwise stdinwill be /dev/nullon Unix machines and ignored on Windows
machines.
-
-
stream_output = <True |False>
-
- If True, then stdoutis streamed back to the machine from which the job was
submitted. If False, stdoutis stored locally and transferred back when the
job completes. This command is ignored if the job ClassAd attribute
TransferOutis False. The default value is False. This command must be used
in conjunction with output, otherwise stdoutwill sent to /dev/nullon Unix
machines and ignored on Windows machines.
-
-
transfer_executable = <True |False>
-
- This command is applicable to jobs submitted to the grid and vanilla
universes. If transfer_executableis set to False, then HTCondor looks for
the executable on the remote machine, and does not transfer the executable
over. This is useful for an already pre-staged executable; HTCondor
behaves more like rsh. The default value is True.
-
-
transfer_input_files = < file1,file2,file... >
-
- A comma-delimited list of all the files and directories to be transferred
into the working directory for the job, before the job is started. By
default, the file specified in the executablecommand and any file
specified in the inputcommand (for example, stdin) are transferred.
-
- When a path to an input file or directory is specified, this specifies the
path to the file on the submit side. The file is placed in the job's
temporary scratch directory on the execute side, and it is named using the
base name of the original path. For example, /path/to/input_filebecomes
input_filein the job's scratch directory.
-
- A directory may be specified by appending the forward slash character (/)
as a trailing path separator. This syntax is used for both Windows and
Linux submit hosts. A directory example using a trailing path separator is
input_data/. When a directory is specified with the trailing path
separator, the contents of the directory are transferred, but the
directory itself is not transferred. It is as if each of the items within
the directory were listed in the transfer list. When there is no trailing
path separator, the directory is transferred, its contents are
transferred, and these contents are placed inside the transferred
directory.
-
- For grid universe jobs other than HTCondor-C, the transfer of directories
is not currently supported.
-
- Symbolic links to files are transferred as the files they point to.
Transfer of symbolic links to directories is not currently supported.
-
- For vanilla and vm universe jobs only, a file may be specified by giving a
URL, instead of a file name. The implementation for URL transfers requires
both configuration and available plug-in.
-
-
transfer_output_files = < file1,file2,file... >
-
- This command forms an explicit list of output files and directories to be
transferred back from the temporary working directory on the execute
machine to the submit machine. If there are multiple files, they must be
delimited with commas. Setting transfer_output_filesto the empty string (
"" ) means that no files are to be transferred.
-
- For HTCondor-C jobs and all other non-grid universe jobs, if
transfer_output_filesis not specified, HTCondor will automatically
transfer back all files in the job's temporary working directory which
have been modified or created by the job. Subdirectories are not scanned
for output, so if output from subdirectories is desired, the output list
must be explicitly specified. For grid universe jobs other than
HTCondor-C, desired output files must also be explicitly listed. Another
reason to explicitly list output files is for a job that creates many
files, and the user wants only a subset transferred back.
-
- For grid universe jobs other than with grid type condor, to have files
other than standard output and standard error transferred from the execute
machine back to the submit machine, do use transfer_output_files, listing
all files to be transferred. These files are found on the execute machine
in the working directory of the job.
-
- When a path to an output file or directory is specified, it specifies the
path to the file on the execute side. As a destination on the submit side,
the file is placed in the job's initial working directory, and it is named
using the base name of the original path. For example,
path/to/output_filebecomes output_filein the job's initial working
directory. The name and path of the file that is written on the submit
side may be modified by using transfer_output_remaps. Note that this remap
function only works with files but not with directories.
-
- A directory may be specified using a trailing path separator. An example
of a trailing path separator is the slash character on Unix platforms; a
directory example using a trailing path separator is input_data/. When a
directory is specified with a trailing path separator, the contents of the
directory are transferred, but the directory itself is not transferred. It
is as if each of the items within the directory were listed in the
transfer list. When there is no trailing path separator, the directory is
transferred, its contents are transferred, and these contents are placed
inside the transferred directory.
-
- For grid universe jobs other than HTCondor-C, the transfer of directories
is not currently supported.
-
- Symbolic links to files are transferred as the files they point to.
Transfer of symbolic links to directories is not currently supported.
-
-
transfer_output_remaps < “ name newname ; name2
newname2 ... ”>
-
- This specifies the name (and optionally path) to use when downloading
output files from the completed job. Normally, output files are
transferred back to the initial working directory with the same name they
had in the execution directory. This gives you the option to save them
with a different path or name. If you specify a relative path, the final
path will be relative to the job's initial working directory.
-
- namedescribes an output file name produced by your job, and
newnamedescribes the file name it should be downloaded to. Multiple remaps
can be specified by separating each with a semicolon. If you wish to remap
file names that contain equals signs or semicolons, these special
characters may be escaped with a backslash. You cannot specify directories
to be remapped.
-
-
when_to_transfer_output = < ON_EXIT |ON_EXIT_OR_EVICT
>
-
-
- Setting when_to_transfer_outputequal to ON_EXITwill cause HTCondor to
transfer the job's output files back to the submitting machine only when
the job completes (exits on its own).
-
- The ON_EXIT_OR_EVICToption is intended for fault tolerant jobs which
periodically save their own state and can restart where they left off. In
this case, files are spooled to the submit machine any time the job leaves
a remote site, either because it exited on its own, or was evicted by the
HTCondor system for any reason prior to job completion. The files spooled
back are placed in a directory defined by the value of the
SPOOLconfiguration variable. Any output files transferred back to the
submit machine are automatically sent back out again as input files if the
job restarts.
-
-
POLICY COMMANDS
-
-
-
max_retries = <integer>
-
- The maximum number of retries allowed for this job (must be non-negative).
If the job fails (does not exit with the success_exit_codeexit code) it
will be retried up to max_retriestimes (unless retries are ceased because
of the retry_untilcommand). If max_retriesis not defined, and either
retry_untilor success_exit_codeis, the value of
DEFAULT_JOB_MAX_RETRIESwill be used for the maximum number of
retries.
-
- The combination of the max_retries, retry_until, and
success_exit_codecommands causes an appropriate OnExitRemoveexpression to
be automatically generated. If retry command(s) and on_exit_removeare both
defined, the OnExitRemoveexpression will be generated by OR'ing the
expression specified in OnExitRemoveand the expression generated by the
retry commands.
-
-
retry_until <Integer | ClassAd Boolean
Expression>
-
- An integer value or boolean expression that prevents further retries from
taking place, even if max_retrieshave not been exhausted. If retry_untilis
an integer, the job exiting with that exit code will cause retries to
cease. If retry_untilis a ClassAd expression, the expression evaluating to
Truewill cause retries to cease.
-
-
success_exit_code = <integer>
-
- The exit code that is considered successful for this job. Defaults to 0 if
not defined.
-
- Note: non-zero values of success_exit_codeshould generally not be used for
DAG node jobs.At the present time, condor_dagmandoes not take into account
the value of success_exit_code. This means that, if success_exit_codeis
set to a non-zero value, condor_dagmanwill consider the job failed when it
actually succeeds. For single-proc DAG node jobs, this can be overcome by
using a POST script that takes into account the value of
success_exit_code(although this is not recommended). For multi-proc DAG
node jobs, there is currently no way to overcome this limitation.
-
-
hold = <True |False>
-
- If holdis set to True, then the submitted job will be placed into the Hold
state. Jobs in the Hold state will not run until released by
condor_release. Defaults to False.
-
-
keep_claim_idle = <integer>
-
- An integer number of seconds that a job requests the condor_scheddto wait
before releasing its claim after the job exits or after the job is
removed.
-
- The process by which the condor_scheddclaims a condor_startdis somewhat
time-consuming. To amortize this cost, the condor_scheddtries to reuse
claims to run subsequent jobs, after a job using a claim is done. However,
it can only do this if there is an idle job in the queue at the moment the
previous job completes. Sometimes, and especially for the node jobs when
using DAGMan, there is a subsequent job about to be submitted, but it has
not yet arrived in the queue when the previous job completes. As a result,
the condor_scheddreleases the claim, and the next job must wait an entire
negotiation cycle to start. When this submit command is defined with a
non-negative integer, when the job exits, the condor_scheddtries as usual
to reuse the claim. If it cannot, instead of releasing the claim, the
condor_scheddkeeps the claim until either the number of seconds given as a
parameter, or a new job which matches that claim arrives, whichever comes
first. The condor_startdin question will remain in the Claimed/Idle state,
and the original job will be "charged" (in terms of priority)
for the time in this state.
-
-
leave_in_queue = <ClassAd Boolean Expression>
-
- When the ClassAd Expression evaluates to True, the job is not removed from
the queue upon completion. This allows the user of a remotely spooled job
to retrieve output files in cases where HTCondor would have removed them
as part of the cleanup associated with completion. The job will only exit
the queue once it has been marked for removal (via condor_rm, for example)
and the leave_in_queueexpression has become False. leave_in_queuedefaults
to False.
-
- As an example, if the job is to be removed once the output is retrieved
with condor_transfer_data, then use
leave_in_queue = (JobStatus == 4) && ((StageOutFinish =?=
UNDEFINED) ||\
(StageOutFinish == 0))
-
-
next_job_start_delay = <ClassAd Boolean
Expression>
-
- This expression specifies the number of seconds to delay after starting up
this job before the next job is started. The maximum allowed delay is
specified by the HTCondor configuration variable MAX_NEXT_JOB_START_DELAY,
which defaults to 10 minutes. This command does not apply to scheduleror
localuniverse jobs.
-
- This command has been historically used to implement a form of job start
throttling from the job submitter's perspective. It was effective for the
case of multiple job submission where the transfer of extremely large
input data sets to the execute machine caused machine performance to
suffer. This command is no longer useful, as throttling should be
accomplished through configuration of the condor_schedddaemon.
-
-
on_exit_hold = <ClassAd Boolean Expression>
-
- The ClassAd expression is checked when the job exits, and if True, places
the job into the Hold state. If False(the default value when not defined),
then nothing happens and the on_exit_removeexpression is checked to
determine if that needs to be applied.
-
- For example: Suppose a job is known to run for a minimum of an hour. If
the job exits after less than an hour, the job should be placed on hold
and an e-mail notification sent, instead of being allowed to leave the
queue.
-
on_exit_hold = (time() - JobStartDate) < (60 * $(MINUTE))
-
- This expression places the job on hold if it exits for any reason before
running for an hour. An e-mail will be sent to the user explaining that
the job was placed on hold because this expression became True.
-
- periodic_*expressions take precedence over on_exit_*expressions, and
*_holdexpressions take precedence over a *_removeexpressions.
-
- Only job ClassAd attributes will be defined for use by this ClassAd
expression. This expression is available for the vanilla, java, parallel,
grid, local and scheduler universes. It is additionally available, when
submitted from a Unix machine, for the standard universe.
-
-
on_exit_hold_reason = <ClassAd String Expression>
-
- When the job is placed on hold due to the on_exit_holdexpression becoming
True, this expression is evaluated to set the value of HoldReasonin the
job ClassAd. If this expression is UNDEFINEDor produces an empty or
invalid string, a default description is used.
-
-
on_exit_hold_subcode = <ClassAd Integer
Expression>
-
- When the job is placed on hold due to the on_exit_holdexpression becoming
True, this expression is evaluated to set the value of HoldReasonSubCodein
the job ClassAd. The default subcode is 0. The HoldReasonCodewill be set
to 3, which indicates that the job went on hold due to a job policy
expression.
-
-
on_exit_remove = <ClassAd Boolean Expression>
-
- The ClassAd expression is checked when the job exits, and if True(the
default value when undefined), then it allows the job to leave the queue
normally. If False, then the job is placed back into the Idle state. If
the user job runs under the vanilla universe, then the job restarts from
the beginning. If the user job runs under the standard universe, then it
continues from where it left off, using the last checkpoint.
-
- For example, suppose a job occasionally segfaults, but chances are that
the job will finish successfully if the job is run again with the same
data. The on_exit_removeexpression can cause the job to run again with the
following command. Assume that the signal identifier for the segmentation
fault is 11 on the platform where the job will be running.
on_exit_remove = (ExitBySignal == False) || (ExitSignal != 11) This
expression lets the job leave the queue if the job was not killed by a
signal or if it was killed by a signal other than 11, representing
segmentation fault in this example. So, if the exited due to signal 11, it
will stay in the job queue. In any other case of the job exiting, the job
will leave the queue as it normally would have done.
-
- As another example, if the job should only leave the queue if it exited on
its own with status 0, this on_exit_removeexpression works well:
-
on_exit_remove = (ExitBySignal == False) && (ExitCode == 0) If the
job was killed by a signal or exited with a non-zero exit status, HTCondor
would leave the job in the queue to run again.
-
- periodic_*expressions take precedence over on_exit_*expressions, and
*_holdexpressions take precedence over a *_removeexpressions.
-
- Only job ClassAd attributes will be defined for use by this ClassAd
expression.
-
-
periodic_hold = <ClassAd Boolean Expression>
-
- This expression is checked periodically when the job is not in the Held
state. If it becomes True, the job will be placed on hold. If unspecified,
the default value is False.
-
- periodic_*expressions take precedence over on_exit_*expressions, and
*_holdexpressions take precedence over a *_removeexpressions.
-
- Only job ClassAd attributes will be defined for use by this ClassAd
expression. Note that, by default, this expression is only checked once
every 60 seconds. The period of these evaluations can be adjusted by
setting the PERIODIC_EXPR_INTERVAL, MAX_PERIODIC_EXPR_INTERVAL, and
PERIODIC_EXPR_TIMESLICEconfiguration macros.
-
-
periodic_hold_reason = <ClassAd String
Expression>
-
- When the job is placed on hold due to the periodic_holdexpression becoming
True, this expression is evaluated to set the value of HoldReasonin the
job ClassAd. If this expression is UNDEFINEDor produces an empty or
invalid string, a default description is used.
-
-
periodic_hold_subcode = <ClassAd Integer
Expression>
-
- When the job is placed on hold due to the periodic_holdexpression becoming
true, this expression is evaluated to set the value of HoldReasonSubCodein
the job ClassAd. The default subcode is 0. The HoldReasonCodewill be set
to 3, which indicates that the job went on hold due to a job policy
expression.
-
-
periodic_release = <ClassAd Boolean Expression>
-
- This expression is checked periodically when the job is in the Held state.
If the expression becomes True, the job will be released.
-
- Only job ClassAd attributes will be defined for use by this ClassAd
expression. Note that, by default, this expression is only checked once
every 60 seconds. The period of these evaluations can be adjusted by
setting the PERIODIC_EXPR_INTERVAL, MAX_PERIODIC_EXPR_INTERVAL, and
PERIODIC_EXPR_TIMESLICEconfiguration macros.
-
-
periodic_remove = <ClassAd Boolean Expression>
-
- This expression is checked periodically. If it becomes True, the job is
removed from the queue. If unspecified, the default value is False.
-
- See the Examples section of this manual page for an example of a
periodic_removeexpression.
-
- periodic_*expressions take precedence over on_exit_*expressions, and
*_holdexpressions take precedence over a *_removeexpressions. So, the
periodic_removeexpression takes precedent over the
on_exit_removeexpression, if the two describe conflicting actions.
-
- Only job ClassAd attributes will be defined for use by this ClassAd
expression. Note that, by default, this expression is only checked once
every 60 seconds. The period of these evaluations can be adjusted by
setting the PERIODIC_EXPR_INTERVAL, MAX_PERIODIC_EXPR_INTERVAL, and
PERIODIC_EXPR_TIMESLICEconfiguration macros.
-
-
COMMANDS SPECIFIC TO THE STANDARD UNIVERSE
-
-
-
allow_startup_script = <True |False>
-
- If True, a standard universe job will execute a script instead of
submitting the job, and the consistency check to see if the executable has
been linked using condor_compileis omitted. The executablecommand within
the submit description file specifies the name of the script. The script
is used to do preprocessing before the job is submitted. The shell script
ends with an execof the job executable, such that the process id of the
executable is the same as that of the shell script. Here is an example
script that gets a copy of a machine-specific executable before the
exec.
#! /bin/sh
# get the host name of the machine
$host=`uname -n`
# grab a standard universe executable designed specifically
# for this host
scp elsewhere@cs.wisc.edu:${host} executable
# The PID MUST stay the same, so exec the new standard universe process.
exec executable ${1+"$@"} If this command is not present (defined),
then the value defaults to false.
-
-
append_files = file1, file2, ...
-
-
- If your job attempts to access a file mentioned in this list, HTCondor
will force all writes to that file to be appended to the end. Furthermore,
condor_submit will not truncate it. This list uses the same syntax as
compress_files, shown above.
-
- This option may yield some surprising results. If several jobs attempt to
write to the same file, their output may be intermixed. If a job is
evicted from one or more machines during the course of its lifetime, such
an output file might contain several copies of the results. This option
should be only be used when you wish a certain file to be treated as a
running log instead of a precise result.
-
- This option only applies to standard-universe jobs.
-
-
buffer_files < “ name (size,block-size) ; name2
(size,block-size) ... ” >
-
-
buffer_size <bytes-in-buffer>
-
-
buffer_block_size <bytes-in-block>
-
- HTCondor keeps a buffer of recently-used data for each file a job
accesses. This buffer is used both to cache commonly-used data and to
consolidate small reads and writes into larger operations that get better
throughput. The default settings should produce reasonable results for
most programs.
-
- These options only apply to standard-universe jobs.
-
- If needed, you may set the buffer controls individually for each file
using the buffer_files option. For example, to set the buffer size to 1
MiB and the block size to 256 KiB for the file input.data, use this
command:
-
buffer_files = "input.data=(1000000,256000)"
-
- Alternatively, you may use these two options to set the default sizes for
all files used by your job:
-
buffer_size = 1000000
buffer_block_size = 256000
-
- If you do not set these, HTCondor will use the values given by these two
configuration file macros:
-
DEFAULT_IO_BUFFER_SIZE = 1000000
DEFAULT_IO_BUFFER_BLOCK_SIZE = 256000
-
- Finally, if no other settings are present, HTCondor will use a buffer of
512 KiB and a block size of 32 KiB.
-
-
compress_files = file1, file2, ...
-
-
- If your job attempts to access any of the files mentioned in this list,
HTCondor will automatically compress them (if writing) or decompress them
(if reading). The compress format is the same as used by GNU gzip.
-
- The files given in this list may be simple file names or complete paths
and may include as a wild card. For example, this list causes the file
/tmp/data.gz, any file named event.gz, and any file ending in .gzip to be
automatically compressed or decompressed as needed:
-
compress_files = /tmp/data.gz, event.gz, *.gzip
-
- Due to the nature of the compression format, compressed files must only be
accessed sequentially. Random access reading is allowed but is very slow,
while random access writing is simply not possible. This restriction may
be avoided by using both compress_files and fetch_files at the same time.
When this is done, a file is kept in the decompressed state at the
execution machine, but is compressed for transfer to its original
location.
-
- This option only applies to standard universe jobs.
-
-
fetch_files = file1, file2, ...
-
- If your job attempts to access a file mentioned in this list, HTCondor
will automatically copy the whole file to the executing machine, where it
can be accessed quickly. When your job closes the file, it will be copied
back to its original location. This list uses the same syntax as
compress_files, shown above.
-
- This option only applies to standard universe jobs.
-
-
file_remaps < “ name newname ; name2 newname2 ...
”>
-
-
- Directs HTCondor to use a new file name in place of an old one.
namedescribes a file name that your job may attempt to open, and
newnamedescribes the file name it should be replaced with. newnamemay
include an optional leading access specifier, local: or remote: . If left
unspecified, the default access specifier is remote: . Multiple remaps can
be specified by separating each with a semicolon.
-
- This option only applies to standard universe jobs.
-
- If you wish to remap file names that contain equals signs or semicolons,
these special characters may be escaped with a backslash.
-
-
- Example One:
-
- Suppose that your job reads a file named dataset.1. To instruct HTCondor
to force your job to read other.datasetinstead, add this to the submit
file:
file_remaps = "dataset.1=other.dataset"
-
-
- Example Two:
-
- Suppose that your run many jobs which all read in the same large file,
called very.big. If this file can be found in the same place on a local
disk in every machine in the pool, (say /bigdisk/bigfile,) you can
instruct HTCondor of this fact by remapping very.bigto /bigdisk/bigfileand
specifying that the file is to be read locally, which will be much faster
than reading over the network.
file_remaps = "very.big = local:/bigdisk/bigfile"
-
-
- Example Three:
-
- Several remaps can be applied at once by separating each with a
semicolon.
file_remaps = "very.big = local:/bigdisk/bigfile ; dataset.1
= other.dataset"
-
-
-
local_files = file1, file2, ...
-
-
- If your job attempts to access a file mentioned in this list, HTCondor
will cause it to be read or written at the execution machine. This is most
useful for temporary files not used for input or output. This list uses
the same syntax as compress_files, shown above.
-
local_files = /tmp/*
-
- This option only applies to standard universe jobs.
-
-
want_remote_io = <True |False>
-
- This option controls how a file is opened and manipulated in a standard
universe job. If this option is true, which is the default, then the
condor_shadowmakes all decisions about how each and every file should be
opened by the executing job. This entails a network round trip (or more)
from the job to the condor_shadowand back again for every single open()in
addition to other needed information about the file. If set to false, then
when the job queries the condor_shadowfor the first time about how to open
a file, the condor_shadowwill inform the job to automatically perform all
of its file manipulation on the local file system on the execute machine
and any file remapping will be ignored. This means that there mustbe a
shared file system (such as NFS or AFS) between the execute machine and
the submit machine and that ALLpaths that the job could open on the
execute machine must be valid. The ability of the standard universe job to
checkpoint, possibly to a checkpoint server, is not affected by this
attribute. However, when the job resumes it will be expecting the same
file system conditions that were present when the job checkpointed.
-
-
COMMANDS FOR THE GRID
-
-
-
batch_queue = <queuename>
-
- Used for pbs, lsf, and sgegrid universe jobs. Specifies the name of the
PBS/LSF/SGE job queue into which the job should be submitted. If not
specified, the default queue is used.
-
-
boinc_authenticator_file = <pathname>
-
- For grid type boincjobs, specifies a path and file name of the
authorization file that grants permission for HTCondor to use the BOINC
service. There is no default value when not specified.
-
-
cream_attributes = <name=value;...;name=value>
-
- Provides a list of attribute/value pairs to be set in a CREAM job
description of a grid universe job destined for the CREAM grid system. The
pairs are separated by semicolons, and written in New ClassAd syntax.
-
-
delegate_job_GSI_credentials_lifetime = <seconds>
-
- Specifies the maximum number of seconds for which delegated proxies should
be valid. The default behavior when this command is not specified is
determined by the configuration variable
DELEGATE_JOB_GSI_CREDENTIALS_LIFETIME, which defaults to one day. A value
of 0 indicates that the delegated proxy should be valid for as long as
allowed by the credential used to create the proxy. This setting currently
only applies to proxies delegated for non-grid jobs and for HTCondor-C
jobs. It does not currently apply to globus grid jobs, which always behave
as though this setting were 0. This variable has no effect if the
configuration variable DELEGATE_JOB_GSI_CREDENTIALSis False, because in
that case the job proxy is copied rather than delegated.
-
-
ec2_access_key_id = <pathname>
-
- For grid type ec2jobs, identifies the file containing the access key.
-
-
ec2_ami_id = <EC2 xMI ID>
-
- For grid type ec2jobs, identifies the machine image. Services compatible
with the EC2 Query API may refer to these with abbreviations other than
AMI, for example EMIis valid for Eucalyptus.
-
-
ec2_availability_zone = <zone name>
-
- For grid type ec2jobs, specifies the Availability Zone that the instance
should be run in. This command is optional, unless ec2_ebs_volumesis set.
As an example, one current zone is us-east-1b.
-
-
ec2_block_device_mapping =
<block-device>:<kernel-device>,<block-device>:<kernel-device>,
...
-
- For grid type ec2jobs, specifies the block device to kernel device
mapping. This command is optional.
-
-
ec2_ebs_volumes = <ebs name>:<device name>,<ebs
name>:<device name>,...
-
- For grid type ec2jobs, optionally specifies a list of Elastic Block Store
(EBS) volumes to be made available to the instance and the device names
they should have in the instance.
-
-
ec2_elastic_ip = <elastic IP address>
-
- For grid type ec2jobs, and optional specification of an Elastic IP address
that should be assigned to this instance.
-
-
ec2_iam_profile_arn = <IAM profile ARN>
-
- For grid type ec2jobs, an Amazon Resource Name (ARN) identifying which
Identity and Access Management (IAM) (instance) profile to associate with
the instance.
-
-
ec2_iam_profile_name= <IAM profile name>
-
- For grid type ec2jobs, a name identifying which Identity and Access
Management (IAM) (instance) profile to associate with the instance.
-
-
ec2_instance_type = <instance type>
-
- For grid type ec2jobs, identifies the instance type. Different services
may offer different instance types, so no default value is set.
-
-
ec2_keypair = <ssh key-pair name>
-
- For grid type ec2jobs, specifies the name of an SSH key-pair that is
already registered with the EC2 service. The associated private key can be
used to sshinto the virtual machine once it is running.
-
-
ec2_keypair_file = <pathname>
-
- For grid type ec2jobs, specifies the complete path and file name of a file
into which HTCondor will write an SSH key for use with ec2 jobs. The key
can be used to sshinto the virtual machine once it is running. If
ec2_keypairis specified for a job, ec2_keypair_fileis ignored.
-
-
ec2_parameter_names = ParameterName1, ParameterName2,
...
-
- For grid type ec2jobs, a space or comma separated list of the names of
additional parameters to pass when instantiating an instance.
-
-
ec2_parameter_<name> = <value>
-
- For grid type ec2jobs, specifies the value for the correspondingly named
(instance instantiation) parameter. <name>is the parameter name
specified in the submit command ec2_parameter_names, but with any periods
replaced by underscores.
-
-
ec2_secret_access_key = <pathname>
-
- For grid type ec2jobs, specifies the path and file name containing the
secret access key.
-
-
ec2_security_groups = group1, group2, ...
-
- For grid type ec2jobs, defines the list of EC2 security groups which
should be associated with the job.
-
-
ec2_security_ids = id1, id2, ...
-
- For grid type ec2jobs, defines the list of EC2 security group IDs which
should be associated with the job.
-
-
ec2_spot_price = <bid>
-
- For grid type ec2jobs, specifies the spot instance bid, which is the most
that the job submitter is willing to pay per hour to run this job.
-
-
ec2_tag_names = <name0,name1,name...>
-
- For grid type ec2jobs, specifies the case of tag names that will be
associated with the running instance. This is only necessary if a tag name
case matters. By default the list will be automatically generated.
-
-
ec2_tag_<name> = <value>
-
- For grid type ec2jobs, specifies a tag to be associated with the running
instance. The tag name will be lower-cased, use ec2_tag_namesto change the
case.
-
-
WantNameTag = <True |False>
-
- For grid type ec2jobs, a job may request that its 'name' tag be (not) set
by HTCondor. If the job does not otherwise specify any tags, not setting
its name tag will eliminate a call by the EC2 GAHP, improving
performance.
-
-
ec2_user_data = <data>
-
- For grid type ec2jobs, provides a block of data that can be accessed by
the virtual machine. If both ec2_user_dataand ec2_user_data_fileare
specified for a job, the two blocks of data are concatenated, with the
data from this ec2_user_datasubmit command occurring first.
-
-
ec2_user_data_file = <pathname>
-
- For grid type ec2jobs, specifies a path and file name whose contents can
be accessed by the virtual machine. If both ec2_user_dataand
ec2_user_data_fileare specified for a job, the two blocks of data are
concatenated, with the data from that ec2_user_datasubmit command
occurring first.
-
-
ec2_vpc_ip = <a.b.c.d>
-
- For grid type ec2jobs, that are part of a Virtual Private Cloud (VPC), an
optional specification of the IP address that this instance should have
within the VPC.
-
-
ec2_vpc_subnet = <subnet specification string>
-
- For grid type ec2jobs, an optional specification of the Virtual Private
Cloud (VPC) that this instance should be a part of.
-
-
gce_auth_file = <pathname>
-
- For grid type gcejobs, specifies a path and file name of the authorization
file that grants permission for HTCondor to use the Google account. There
is no default value when not specified.
-
-
gce_image = <image id>
-
- For grid type gcejobs, the identifier of the virtual machine image
representing the HTCondor job to be run. This virtual machine image must
already be register with GCE and reside in Google's Cloud Storage
service.
-
-
gce_json_file = <pathname>
-
- For grid type gcejobs, specifies the path and file name of a file that
contains JSON elements that should be added to the instance description
submitted to the GCE service.
-
-
gce_machine_type = <machine type>
-
- For grid type gcejobs, the long form of the URL that describes the machine
configuration that the virtual machine instance is to run on.
-
-
gce_metadata = <name=value,...,name=value>
-
- For grid type gcejobs, a comma separated list of name and value pairs that
define metadata for a virtual machine instance that is an HTCondor
job.
-
-
gce_metadata_file = <pathname>
-
- For grid type gcejobs, specifies a path and file name of the file that
contains metadata for a virtual machine instance that is an HTCondor job.
Within the file, each name and value pair is on its own line; so, the
pairs are separated by the newline character.
-
-
gce_preemptible = <True |False>
-
- For grid type gcejobs, specifies whether the virtual machine instance
should be preemptible. The default is for the instance to not be
preemptible.
-
-
globus_rematch = <ClassAd Boolean Expression>
-
- This expression is evaluated by the condor_gridmanagerwhenever:
-
- 1. the globus_resubmitexpression evaluates to True
-
- 2. the condor_gridmanagerdecides it needs to retry a submission (as when a
previous submission failed to commit) If globus_rematchevaluates to True,
then beforethe job is submitted again to globus, the
condor_gridmanagerwill request that the condor_schedddaemon renegotiate
with the matchmaker (the condor_negotiator). The result is this job will
be matched again.
-
-
globus_resubmit = <ClassAd Boolean Expression>
-
- The expression is evaluated by the condor_gridmanagereach time the
condor_gridmanagergets a job ad to manage. Therefore, the expression is
evaluated:
-
- 1. when a grid universe job is first submitted to HTCondor-G
-
- 2. when a grid universe job is released from the hold state
-
- 3. when HTCondor-G is restarted (specifically, whenever the
condor_gridmanageris restarted) If the expression evaluates to True, then
any previous submission to the grid universe will be forgotten and this
job will be submitted again as a fresh submission to the grid universe.
This may be useful if there is a desire to give up on a previous
submission and try again. Note that this may result in the same job
running more than once. Do not treat this operation lightly.
-
-
globus_rsl = <RSL-string>
-
- Used to provide any additional Globus RSL string attributes which are not
covered by other submit description file commands or job attributes. Used
for griduniversejobs, where the grid resource has a grid-type-stringof
gt2.
-
-
grid_resource = <grid-type-string>
<grid-specific-parameter-list>
-
- For each grid-type-stringvalue, there are further type-specific values
that must specified. This submit description file command allows each to
be given in a space-separated list. Allowable grid-type-stringvalues are
batch, condor, cream, ec2, gt2, gt5, lsf, nordugrid, pbs, sge, and
unicore. The HTCondor manual chapter on Grid Computing details the variety
of grid types.
-
- For a grid-type-stringof batch, the single parameter is the name of the
local batch system, and will be one of pbs, lsf, or sge.
-
- For a grid-type-stringof condor, the first parameter is the name of the
remote condor_schedddaemon. The second parameter is the name of the pool
to which the remote condor_schedddaemon belongs.
-
- For a grid-type-stringof cream, there are three parameters. The first
parameter is the web services address of the CREAM server. The second
parameter is the name of the batch system that sits behind the CREAM
server. The third parameter identifies a site-specific queue within the
batch system.
-
- For a grid-type-stringof ec2, one additional parameter specifies the EC2
URL.
-
- For a grid-type-stringof gt2, the single parameter is the name of the
pre-WS GRAM resource to be used.
-
- For a grid-type-stringof gt5, the single parameter is the name of the
pre-WS GRAM resource to be used, which is the same as for the
grid-type-stringof gt2.
-
- For a grid-type-stringof lsf, no additional parameters are used.
-
- For a grid-type-stringof nordugrid, the single parameter is the name of
the NorduGrid resource to be used.
-
- For a grid-type-stringof pbs, no additional parameters are used.
-
- For a grid-type-stringof sge, no additional parameters are used.
-
- For a grid-type-stringof unicore, the first parameter is the name of the
Unicore Usite to be used. The second parameter is the name of the Unicore
Vsite to be used.
-
-
keystore_alias = <name>
-
- A string to locate the certificate in a Java keystore file, as used for a
unicorejob.
-
-
keystore_file = <pathname>
-
- The complete path and file name of the Java keystore file containing the
certificate to be used for a unicorejob.
-
-
keystore_passphrase_file = <pathname>
-
- The complete path and file name to the file containing the passphrase
protecting a Java keystore file containing the certificate. Relevant for a
unicorejob.
-
-
MyProxyCredentialName = <symbolic name>
-
- The symbolic name that identifies a credential to the MyProxyserver. This
symbolic name is set as the credential is initially stored on the server
(using myproxy-init).
-
-
MyProxyHost = <host>:<port>
-
- The Internet address of the host that is the MyProxyserver. The hostmay be
specified by either a host name (as in head.example.com) or an IP address
(of the form 123.456.7.8). The portnumber is an integer.
-
-
MyProxyNewProxyLifetime = <number-of-minutes>
-
- The new lifetime (in minutes) of the proxy after it is refreshed.
-
-
MyProxyPassword = <password>
-
- The password needed to refresh a credential on the MyProxyserver. This
password is set when the user initially stores credentials on the server
(using myproxy-init). As an alternative to using MyProxyPasswordin the
submit description file, the password may be specified as a command line
argument to condor_submit with the -passwordargument.
-
-
MyProxyRefreshThreshold = <number-of-seconds>
-
- The time (in seconds) before the expiration of a proxy that the proxy
should be refreshed. For example, if MyProxyRefreshThresholdis set to the
value 600, the proxy will be refreshed 10 minutes before it expires.
-
-
MyProxyServerDN = <credential subject>
-
- A string that specifies the expected Distinguished Name (credential
subject, abbreviated DN) of the MyProxyserver. It must be specified when
the MyProxyserver DN does not follow the conventional naming scheme of a
host credential. This occurs, for example, when the MyProxyserver DN
begins with a user credential.
-
-
nordugrid_rsl = <RSL-string>
-
- Used to provide any additional RSL string attributes which are not covered
by regular submit description file parameters. Used when the universeis
grid, and the type of grid system is nordugrid.
-
-
transfer_error = <True |False>
-
- For jobs submitted to the grid universe only. If True, then the error
output (from stderr) from the job is transferred from the remote machine
back to the submit machine. The name of the file after transfer is given
by the errorcommand. If False, no transfer takes place (from the remote
machine to submit machine), and the name of the file is given by the
errorcommand. The default value is True.
-
-
transfer_input = <True |False>
-
- For jobs submitted to the grid universe only. If True, then the job input
(stdin) is transferred from the machine where the job was submitted to the
remote machine. The name of the file that is transferred is given by the
inputcommand. If False, then the job's input is taken from a pre-staged
file on the remote machine, and the name of the file is given by the
inputcommand. The default value is True.
-
- For transferring files other than stdin, see transfer_input_files.
-
-
transfer_output = <True |False>
-
- For jobs submitted to the grid universe only. If True, then the output
(from stdout) from the job is transferred from the remote machine back to
the submit machine. The name of the file after transfer is given by the
outputcommand. If False, no transfer takes place (from the remote machine
to submit machine), and the name of the file is given by the
outputcommand. The default value is True.
-
- For transferring files other than stdout, see transfer_output_files.
-
-
use_x509userproxy = <True |False>
-
- Set this command to Trueto indicate that the job requires an X.509 user
proxy. If x509userproxyis set, then that file is used for the proxy.
Otherwise, the proxy is looked for in the standard locations. If
x509userproxyis set or if the job is a grid universe job of grid type gt2,
gt5, cream, or nordugrid, then the value of use_x509userproxyis forced to
True. Defaults to False.
-
-
x509userproxy = <full-pathname>
-
- Used to override the default path name for X.509 user certificates. The
default location for X.509 proxies is the /tmpdirectory, which is
generally a local file system. Setting this value would allow HTCondor to
access the proxy in a shared file system (for example, AFS). HTCondor will
use the proxy specified in the submit description file first. If nothing
is specified in the submit description file, it will use the environment
variable X509_USER_PROXY. If that variable is not present, it will search
in the default location. Note that proxies are only valid for a limited
time. Condor_submit will not submit a job with an expired proxy, it will
return an error. Also, if the configuration parameter CRED_MIN_TIME_LEFT
is set to some number of seconds, and if the proxy will expire before that
many seconds, condor_submit will also refuse to submit the job. That is,
if CRED_MIN_TIME_LEFT is set to 60, condor_submit will refuse to submit a
job whose proxy will expire 60 seconds from the time of submission.
-
- x509userproxyis relevant when the universeis vanilla, or when the
universeis gridand the type of grid system is one of gt2, gt5, condor,
cream, or nordugrid. Defining a value causes the proxy to be delegated to
the execute machine. Further, VOMS attributes defined in the proxy will
appear in the job ClassAd.
-
-
COMMANDS FOR PARALLEL, JAVA, and SCHEDULER UNIVERSES
-
-
-
hold_kill_sig = <signal-number>
-
- For the scheduler universe only, signal-numberis the signal delivered to
the job when the job is put on hold with condor_hold. signal-numbermay be
either the platform-specific name or value of the signal. If this command
is not present, the value of kill_sigis used.
-
-
jar_files = <file_list>
-
- Specifies a list of additional JAR files to include when using the Java
universe. JAR files will be transferred along with the executable and
automatically added to the classpath.
-
-
java_vm_args = <argument_list>
-
- Specifies a list of additional arguments to the Java VM itself, When
HTCondor runs the Java program, these are the arguments that go before the
class name. This can be used to set VM-specific arguments like stack size,
garbage-collector arguments and initial property values.
-
-
machine_count = <max>
-
- For the parallel universe, a single value (max) is required. It is neither
a maximum or minimum, but the number of machines to be dedicated toward
running the job.
-
-
remove_kill_sig = <signal-number>
-
- For the scheduler universe only, signal-numberis the signal delivered to
the job when the job is removed with condor_rm. signal-numbermay be either
the platform-specific name or value of the signal. This example shows it
both ways for a Linux signal:
remove_kill_sig = SIGUSR1
remove_kill_sig = 10
-
- If this command is not present, the value of kill_sigis used.
-
-
COMMANDS FOR THE VM UNIVERSE
-
-
-
vm_disk = file1:device1:permission1,
file2:device2:permission2:format2, ...
-
- A list of comma separated disk files. Each disk file is specified by 4
colon separated fields. The first field is the path and file name of the
disk file. The second field specifies the device. The third field
specifies permissions, and the optional fourth field specifies the image
format. If a disk file will be transferred by HTCondor, then the first
field should just be the simple file name (no path information).
-
- An example that specifies two disk files:
vm_disk = /myxen/diskfile.img:sda1:w,/myxen/swap.img:sda2:w
-
-
vm_checkpoint = <True |False>
-
- A boolean value specifying whether or not to take checkpoints. If not
specified, the default value is False. In the current implementation,
setting both vm_checkpointand vm_networkingto Truedoes not yet work in all
cases. Networking cannot be used if a vm universe job uses a checkpoint in
order to continue execution after migration to another machine.
-
-
vm_macaddr = <MACAddr>
-
- Defines that MAC address that the virtual machine's network interface
should have, in the standard format of six groups of two hexadecimal
digits separated by colons.
-
-
vm_memory = <MBytes-of-memory>
-
- The amount of memory in MBytes that a vm universe job requires.
-
-
vm_networking = <True |False>
-
- Specifies whether to use networking or not. In the current implementation,
setting both vm_checkpointand vm_networkingto Truedoes not yet work in all
cases. Networking cannot be used if a vm universe job uses a checkpoint in
order to continue execution after migration to another machine.
-
-
vm_networking_type = <nat |bridge >
-
- When vm_networkingis True, this definition augments the job's requirements
to match only machines with the specified networking. If not specified,
then either networking type matches.
-
-
vm_no_output_vm = <True |False>
-
- When True, prevents HTCondor from transferring output files back to the
machine from which the vm universe job was submitted. If not specified,
the default value is False.
-
-
vm_type = <vmware |xen |kvm>
-
- Specifies the underlying virtual machine software that this job
expects.
-
-
vmware_dir = <pathname>
-
- The complete path and name of the directory where VMware-specific files
and applications such as the VMDK (Virtual Machine Disk Format) and VMX
(Virtual Machine Configuration) reside. This command is optional; when not
specified, all relevant VMware image files are to be listed using
transfer_input_files.
-
-
vmware_should_transfer_files = <True |False>
-
- Specifies whether HTCondor will transfer VMware-specific files located as
specified by vmware_dirto the execute machine (True) or rely on access
through a shared file system (False). Omission of this required command
(for VMware vm universe jobs) results in an error message from
condor_submit , and the job will not be submitted.
-
-
vmware_snapshot_disk = <True |False>
-
- When True, causes HTCondor to utilize a VMware snapshot disk for new or
modified files. If not specified, the default value is True.
-
-
xen_initrd = <image-file>
-
- When xen_kernelgives a file name for the kernel image to use, this
optional command may specify a path to a ramdisk (initrd) image file. If
the image file will be transferred by HTCondor, then the value should just
be the simple file name (no path information).
-
-
xen_kernel = <included |path-to-kernel>
-
- A value of includedspecifies that the kernel is included in the disk file.
If not one of these values, then the value is a path and file name of the
kernel to be used. If a kernel file will be transferred by HTCondor, then
the value should just be the simple file name (no path information).
-
-
xen_kernel_params = <string>
-
- A string that is appended to the Xen kernel command line.
-
-
xen_root = <string>
-
- A string that is appended to the Xen kernel command line to specify the
root device. This string is required when xen_kernelgives a path to a
kernel. Omission for this required case results in an error message during
submission.
-
-
COMMANDS FOR THE DOCKER UNIVERSE
-
-
-
docker_image = < image-name >
-
- Defines the name of the Docker image that is the basis for the docker
container.
-
-
ADVANCED COMMANDS
-
-
-
accounting_group = <accounting-group-name>
-
- Causes jobs to negotiate under the given accounting group. This value is
advertised in the job ClassAd as AcctGroup. The HTCondor Administrator's
manual contains more information about accounting groups.
-
-
accounting_group_user =
<accounting-group-user-name>
-
- Sets the user name associated with the accounting group name for resource
usage accounting purposes. If not set, defaults to the value of the job
ClassAd attribute Owner. This value is advertised in the job ClassAd as
AcctGroupUser. If an accounting group has not been set with the
accounting_groupcommand, this command is ignored.
-
-
concurrency_limits = <string-list>
-
- A list of resources that this job needs. The resources are presumed to
have concurrency limits placed upon them, thereby limiting the number of
concurrent jobs in execution which need the named resource. Commas and
space characters delimit the items in the list. Each item in the list is a
string that identifies the limit, or it is a ClassAd expression that
evaluates to a string, and it is evaluated in the context of machine
ClassAd being considered as a match. Each item in the list also may
specify a numerical value identifying the integer number of resources
required for the job. The syntax follows the resource name by a colon
character ( : ) and the numerical value. Details on concurrency limits are
in the HTCondor Administrator's manual.
-
-
concurrency_limits_expr = <ClassAd String
Expression>
-
- A ClassAd expression that represents the list of resources that this job
needs after evaluation. The ClassAd expression may specify machine ClassAd
attributes that are evaluated against a matched machine. After evaluation,
the list sets concurrency_limits.
-
-
copy_to_spool = <True |False>
-
- If copy_to_spoolis True, then condor_submit copies the executable to the
local spool directory before running it on a remote host. As copying can
be quite time consuming and unnecessary, the default value is Falsefor all
job universes other than the standard universe. When False, condor_submit
does not copy the executable to a local spool directory. The default is
Truein standard universe, because resuming execution from a checkpoint can
only be guaranteed to work using precisely the same executable that
created the checkpoint.
-
-
coresize = <size>
-
- Should the user's program abort and produce a core file, coresizespecifies
the maximum size in bytes of the core file which the user wishes to keep.
If coresizeis not specified in the command file, the system's user
resource limit coredumpsizeis used (note that coredumpsizeis not an
HTCondor parameter - it is an operating system parameter that can be
viewed with the limitor ulimitcommand on Unix and in the Registry on
Windows). A value of -1 results in no limits being applied to the core
file size. If HTCondor is running as root, a coresizesetting greater than
the system coredumpsizelimit will override the system setting; if HTCondor
is notrunning as root, the system coredumpsizelimit will override
coresize.
-
-
cron_day_of_month = <Cron-evaluated Day>
-
- The set of days of the month for which a deferral time applies. The
HTCondor User's manual section on Time Scheduling for Job Execution has
further details.
-
-
cron_day_of_week = <Cron-evaluated Day>
-
- The set of days of the week for which a deferral time applies. The
HTCondor User's manual section on Time Scheduling for Job Execution has
further details.
-
-
cron_hour = <Cron-evaluated Hour>
-
- The set of hours of the day for which a deferral time applies. The
HTCondor User's manual section on Time Scheduling for Job Execution has
further details.
-
-
cron_minute = <Cron-evaluated Minute>
-
- The set of minutes within an hour for which a deferral time applies. The
HTCondor User's manual section on Time Scheduling for Job Execution has
further details.
-
-
cron_month = <Cron-evaluated Month>
-
- The set of months within a year for which a deferral time applies. The
HTCondor User's manual section on Time Scheduling for Job Execution has
further details.
-
-
cron_prep_time = <ClassAd Integer Expression>
-
- Analogous to deferral_prep_time. The number of seconds prior to a job's
deferral time that the job may be matched and sent to an execution
machine.
-
-
cron_window = <ClassAd Integer Expression>
-
- Analogous to the submit command deferral_window. It allows cron jobs that
miss their deferral time to begin execution.
-
- The HTCondor User's manual section on Time Scheduling for Job Execution
has further details.
-
-
dagman_log = <pathname>
-
- DAGMan inserts this command to specify an event log that it watches to
maintain the state of the DAG. If the logcommand is not specified in the
submit file, DAGMan uses the logcommand to specify the event log.
-
-
deferral_prep_time = <ClassAd Integer Expression>
-
- The number of seconds prior to a job's deferral time that the job may be
matched and sent to an execution machine.
-
- The HTCondor User's manual section on Time Scheduling for Job Execution
has further details.
-
-
deferral_time = <ClassAd Integer Expression>
-
- Allows a job to specify the time at which its execution is to begin,
instead of beginning execution as soon as it arrives at the execution
machine. The deferral time is an expression that evaluates to a Unix Epoch
timestamp (the number of seconds elapsed since 00:00:00 on January 1,
1970, Coordinated Universal Time). Deferral time is evaluated with respect
to the execution machine. This option delays the start of execution, but
not the matching and claiming of a machine for the job. If the job is not
available and ready to begin execution at the deferral time, it has missed
its deferral time. A job that misses its deferral time will be put on hold
in the queue.
-
- The HTCondor User's manual section on Time Scheduling for Job Execution
has further details.
-
- Due to implementation details, a deferral time may not be used for
scheduler universe jobs.
-
-
deferral_window = <ClassAd Integer Expression>
-
- The deferral window is used in conjunction with the deferral_timecommand
to allow jobs that miss their deferral time to begin execution.
-
- The HTCondor User's manual section on Time Scheduling for Job Execution
has further details.
-
-
description = <string>
-
- A string that sets the value of the job ClassAd attribute JobDescription.
When set, tools which display the executable such as condor_qwill instead
use this string.
-
-
email_attributes = <list-of-job-ad-attributes>
-
- A comma-separated list of attributes from the job ClassAd. These
attributes and their values will be included in the e-mail notification of
job completion.
-
-
image_size = <size>
-
- Advice to HTCondor specifying the maximum virtual image size to which the
job will grow during its execution. HTCondor will then execute the job
only on machines which have enough resources, (such as virtual memory), to
support executing the job. If not specified, HTCondor will automatically
make a (reasonably accurate) estimate about the job's size and adjust this
estimate as the program runs. If specified and underestimated, the job may
crash due to the inability to acquire more address space; for example, if
malloc()fails. If the image size is overestimated, HTCondor may have
difficulty finding machines which have the required resources. sizeis
specified in KiB. For example, for an image size of 8 MiB, sizeshould be
8000.
-
-
initialdir = <directory-path>
-
- Used to give jobs a directory with respect to file input and output. Also
provides a directory (on the machine from which the job is submitted) for
the job event log, when a full path is not specified.
-
- For vanilla universe jobs where there is a shared file system, it is the
current working directory on the machine where the job is executed.
-
- For vanilla or grid universe jobs where file transfer mechanisms are
utilized (there is nota shared file system), it is the directory on the
machine from which the job is submitted where the input files come from,
and where the job's output files go to.
-
- For standard universe jobs, it is the directory on the machine from which
the job is submitted where the condor_shadowdaemon runs; the current
working directory for file input and output accomplished through remote
system calls.
-
- For scheduler universe jobs, it is the directory on the machine from which
the job is submitted where the job runs; the current working directory for
file input and output with respect to relative path names.
-
- Note that the path to the executable is notrelative to initialdir; if it
is a relative path, it is relative to the directory in which the
condor_submit command is run.
-
-
job_ad_information_attrs = <attribute-list>
-
- A comma-separated list of job ClassAd attribute names. The named
attributes and their values are written to the job event log whenever any
event is being written to the log. This implements the same thing as the
configuration variable EVENT_LOG_INFORMATION_ATTRS(see page ), but it
applies to the job event log, instead of the system event log.
-
-
JobBatchName = <batch_name>
-
- Set the batch name for this submit. The batch name is displayed by
condor_q-batch. It is intended for use by users to give meaningful names
to their jobs and to influence how condor_qgroups jobs for display. This
value in a submit file can be overridden by specifying the
-batch-nameargument on the condor_submit command line.
-
-
job_lease_duration = <number-of-seconds>
-
- For vanilla, parallel, VM, and java universe jobs only, the duration in
seconds of a job lease. The default value is 2,400, or forty minutes. If a
job lease is not desired, the value can be explicitly set to 0 to disable
the job lease semantics. The value can also be a ClassAd expression that
evaluates to an integer. The HTCondor User's manual section on Special
Environment Considerations has further details.
-
-
job_machine_attrs = <attr1, attr2, ...>
-
- A comma and/or space separated list of machine attribute names that should
be recorded in the job ClassAd in addition to the ones specified by the
condor_schedddaemon's system configuration variable
SYSTEM_JOB_MACHINE_ATTRS. When there are multiple run attempts, history of
machine attributes from previous run attempts may be kept. The number of
run attempts to store may be extended beyond the system-specified history
length by using the submit file command job_machine_attrs_history_length.
A machine attribute named Xwill be inserted into the job ClassAd as an
attribute named MachineAttrX0. The previous value of this attribute will
be named MachineAttrX1, the previous to that will be named MachineAttrX2,
and so on, up to the specified history length. A history of length 1 means
that only MachineAttrX0will be recorded. The value recorded in the job
ClassAd is the evaluation of the machine attribute in the context of the
job ClassAd when the condor_schedddaemon initiates the start up of the
job. If the evaluation results in an Undefinedor Errorresult, the value
recorded in the job ad will be Undefinedor Error, respectively.
-
-
want_graceful_removal = <boolean expression>
-
- When True, this causes a graceful shutdown of the job when the job is
removed or put on hold, giving it time to clean up or save state.
Otherwise, the job is abruptly killed. The default is false.
-
-
kill_sig = <signal-number>
-
- When HTCondor needs to kick a job off of a machine, it will send the job
the signal specified by signal-number. signal-numberneeds to be an integer
which represents a valid signal on the execution machine. For jobs
submitted to the standard universe, the default value is the number for
SIGTSTP which tells the HTCondor libraries to initiate a checkpoint of the
process. For jobs submitted to other universes, the default value, when
not defined, is SIGTERM , which is the standard way to terminate a program
in Unix.
-
-
kill_sig_timeout = <seconds>
-
- This submit command should no longer be used as of HTCondor version 7.7.3;
use job_max_vacate_timeinstead. If job_max_vacate_timeis not defined, this
defines the number of seconds that HTCondor should wait following the
sending of the kill signal defined by kill_sigand forcibly killing the
job. The actual amount of time between sending the signal and forcibly
killing the job is the smallest of this value and the configuration
variable KILLING_TIMEOUT, as defined on the execute machine.
-
-
load_profile = <True |False>
-
- When True, loads the account profile of the dedicated run account for
Windows jobs. May not be used with run_as_owner.
-
-
match_list_length = <integer value>
-
- Defaults to the value zero (0). When match_list_lengthis defined with an
integer value greater than zero (0), attributes are inserted into the job
ClassAd. The maximum number of attributes defined is given by the integer
value. The job ClassAds introduced are given as
LastMatchName0 = "most-recent-Name"
LastMatchName1 = "next-most-recent-Name"
-
- The value for each introduced ClassAd is given by the value of the
Nameattribute from the machine ClassAd of a previous execution (match). As
a job is matched, the definitions for these attributes will roll, with
LastMatchName1 becoming LastMatchName2 , LastMatchName0 becoming
LastMatchName1 , and LastMatchName0 being set by the most recent value of
the Nameattribute.
-
- An intended use of these job attributes is in the requirements expression.
The requirements can allow a job to prefer a match with either the same or
a different resource than a previous match.
-
-
job_max_vacate_time = <integer expression>
-
- An integer-valued expression (in seconds) that may be used to adjust the
time given to an evicted job for gracefully shutting down. If the job's
setting is less than the machine's, the job's is used. If the job's
setting is larger than the machine's, the result depends on whether the
job has any excess retirement time. If the job has more retirement time
left than the machine's max vacate time setting, then retirement time will
be converted into vacating time, up to the amount requested by the
job.
-
- Setting this expression does not affect the job's resource requirements or
preferences. For a job to only run on a machine with a minimum
MachineMaxVacateTime, or to preferentially run on such machines,
explicitly specify this in the requirements and/or rank expressions.
-
-
max_job_retirement_time = <integer expression>
-
- An integer-valued expression (in seconds) that does nothing unless the
machine that runs the job has been configured to provide retirement time.
Retirement time is a grace period given to a job to finish when a resource
claim is about to be preempted. The default behavior in many cases is to
take as much retirement time as the machine offers, so this command will
rarely appear in a submit description file.
-
- When a resource claim is to be preempted, this expression in the submit
file specifies the maximum run time of the job (in seconds, since the job
started). This expression has no effect, if it is greater than the maximum
retirement time provided by the machine policy. If the resource claim is
notpreempted, this expression and the machine retirement policy are
irrelevant. If the resource claim ispreempted the job will be allowed to
run until the retirement time expires, at which point it is hard-killed.
The job will be soft-killed when it is getting close to the end of
retirement in order to give it time to gracefully shut down. The amount of
lead-time for soft-killing is determined by the maximum vacating time
granted to the job.
-
- Standard universe jobs and any jobs running with nice_userpriority have a
default max_job_retirement_timeof 0, so no retirement time is utilized by
default. In all other cases, no default value is provided, so the maximum
amount of retirement time is utilized by default.
-
- Setting this expression does not affect the job's resource requirements or
preferences. For a job to only run on a machine with a minimum
MaxJobRetirementTime, or to preferentially run on such machines,
explicitly specify this in the requirements and/or rank expressions.
-
-
nice_user = <True |False>
-
- Normally, when a machine becomes available to HTCondor, HTCondor decides
which job to run based upon user and job priorities. Setting
nice_userequal to Truetells HTCondor not to use your regular user
priority, but that this job should have last priority among all users and
all jobs. So jobs submitted in this fashion run only on machines which no
other non-nice_user job wants -- a true bottom-feeder job! This is very
handy if a user has some jobs they wish to run, but do not wish to use
resources that could instead be used to run other people's HTCondor jobs.
Jobs submitted in this fashion have "nice-user."prepended to the
owner name when viewed from condor_qor condor_userprio. The default value
is False.
-
-
noop_job = <ClassAd Boolean Expression>
-
- When this boolean expression is True, the job is immediately removed from
the queue, and HTCondor makes no attempt at running the job. The log file
for the job will show a job submitted event and a job terminated event,
along with an exit code of 0, unless the user specifies a different signal
or exit code.
-
-
noop_job_exit_code = <return value>
-
- When noop_jobis in the submit description file and evaluates to True, this
command allows the job to specify the return value as shown in the job's
log file job terminated event. If not specified, the job will show as
having terminated with status 0. This overrides any value specified with
noop_job_exit_signal.
-
-
noop_job_exit_signal = <signal number>
-
- When noop_jobis in the submit description file and evaluates to True, this
command allows the job to specify the signal number that the job's log
event will show the job having terminated with.
-
-
remote_initialdir = <directory-path>
-
- The path specifies the directory in which the job is to be executed on the
remote machine. This is currently supported in all universes except for
the standard universe.
-
-
rendezvousdir = <directory-path>
-
- Used to specify the shared file system directory to be used for file
system authentication when submitting to a remote scheduler. Should be a
path to a preexisting directory.
-
-
run_as_owner = <True |False>
-
- A boolean value that causes the job to be run under the login of the
submitter, if supported by the joint configuration of the submit and
execute machines. On Unix platforms, this defaults to True, and on Windows
platforms, it defaults to False. May not be used with load_profile. See
the HTCondor manual Platform-Specific Information chapter for
administrative details on configuring Windows to support this option.
-
-
stack_size = <size in bytes>
-
- This command applies only to Linux platform jobs that are not standard
universe jobs. An integer number of bytes, representing the amount of
stack space to be allocated for the job. This value replaces the default
allocation of stack space, which is unlimited in size.
-
-
submit_event_notes = <note>
-
- A string that is appended to the submit event in the job's log file. For
DAGMan jobs, the string DAG Node:and the node's name is automatically
defined for submit_event_notes, causing the logged submit event to
identify the DAG node job submitted.
-
-
+<attribute> = <value>
-
- A line that begins with a '+' (plus) character instructs condor_submit to
insert the given attributeinto the job ClassAd with the given value. Note
that setting an attribute should not be used in place of one of the
specific commands listed above. Often, the command name does not directly
correspond to an attribute name; furthermore, many submit commands result
in actions more complex than simply setting an attribute or attributes.
See for a list of HTCondor job attributes.
-
-
PRE AND POST SCRIPTS IMPLEMENTED WITH SPECIALLY-NAMED
ATTRIBUTES
-
-
-
+PreCmd = <executable>
-
- A vanilla universe job may specify that a script is to be run on the
execute machine before the job, and this is called a prescript. Definition
of this specifically-named attribute causes the script, identified by path
and file name, to be executed. The prescript could prepare or initialize
the job. Note that this definition of a prescript is different from the
PRE script described in DAGMan. The prescript is not automatically
transferred with the job, as the main executable is, so it must be entered
into the transfer_input_fileslist, when file transfer is enabled.
-
-
+PreArgs = <argument_list>
-
- Defines command line arguments for the prescript, presuming the Old
argument syntax.
-
-
+PreArguments = <argument_list>
-
- Defines command line arguments for the prescript, presuming the New
argument syntax. An exception to the syntax is that double quotes must be
escaped with a backslash instead of another double quote.
-
- Note that if both +PreArgs and +PreArguments are specified, the
+PreArguments value is used and the +PreArgs value is ignored.
-
-
+PreEnv = <environment_vars>
-
- Defines the environment for the prescript, presuming the Old environment
syntax.
-
-
+PreEnvironment = <environment_vars>
-
- Defines the environment for the prescript, presuming the New environment
syntax.
-
- Note that if both +PreEnv and +PreEnvironment are specified, the
+PreEnvironment value is used and the +PreEnv value is ignored.
-
-
+PostCmd = <executable>
-
- A vanilla universe job may specify that a script is to be run on the
execute machine after the job exits, and this is called a postscript.
Definition of this specifically-named attribute causes the script,
identified by path and file name, to be executed. The postscript is run if
the job exits, but not if the job is evicted. Note that this definition of
a postscript is different from the POST script described in DAGMan. The
postscript is not automatically transferred with the job, as the main
executable is, so it must be entered into the transfer_input_fileslist,
when file transfer is enabled.
-
-
+PostArgs = <argument_list>
-
- Defines command line arguments for the postscript, presuming the Old
argument syntax.
-
-
+PostArguments = <argument_list>
-
- Defines command line arguments for the postscript, presuming the New
argument syntax. An exception to the syntax is that double quotes must be
escaped with a backslash instead of another double quote mark.
-
- Note that if both +PostArgs and +PostArguments are specified, the
+PostArguments value is used and the +PostArgs value is ignored.
-
-
+PostEnv = <environment_vars>
-
- Defines the environment for the postscript, presuming the Old environment
syntax.
-
-
+PostEnvironment = <environment_vars>
-
- Defines the environment for the postscript, presuming the New environment
syntax.
-
- Note that if both +PostEnv and +PostEnvironment are specified, the
+PostEnvironment value is used and the +PostEnv value is ignored.
-
-
If any of the prescript or postscript values are not enclosed in
double quotes, they are silently ignored.
Below is an example of the use of starter pre and post
scripts:
+PreCmd = "my_pre"
+PreArgs = "pre\"arg1 prea'rg2"
+PreEnv = "one=1;two=\"2\""
+PostCmd = "my_post"
+PostArguments = "post\"arg1 'post”ar g2'"
+PostEnvironment = "one=1 two=\"2\""
For this example PreArgsgenerates a first argument of
pre"a1" and a second argument of pre'a2 . PostArgumentsgenerates a
first argument of post"a1 and a second argument of post'a 2 .
MACROS AND COMMENTS
In addition to commands, the submit description file can contain
macros and comments.
Macros
-
- Parameterless macros in the form of $(macro_name:default initial value)may
be used anywhere in HTCondor submit description files to provide textual
substitution at submit time. Macros can be defined by lines in the form
of
<macro_name> = <string>
-
- Two pre-defined macros are supplied by the submit description file parser.
The $(Cluster)or $(ClusterId)macro supplies the value of the ClusterIdjob
ClassAd attribute, and the $(Process)or $(ProcId)macro supplies the value
of the ProcIdjob ClassAd attribute. These macros are intended to aid in
the specification of input/output files, arguments, etc., for clusters
with lots of jobs, and/or could be used to supply an HTCondor process with
its own cluster and process numbers on the command line.
-
- The $(Node)macro is defined for parallel universe jobs, and is especially
relevant for MPI applications. It is a unique value assigned for the
duration of the job that essentially identifies the machine (slot) on
which a program is executing. Values assigned start at 0 and increase
monotonically. The values are assigned as the parallel job is about to
start.
-
- Recursive definition of macros is permitted. An example of a construction
that works is the following:
foo = bar
foo = snap $(foo)
-
- As a result, foo = snap bar.
-
- Note that both left- and right- recursion works, so
foo = bar
foo = $(foo) snap
-
- has as its result foo = bar snap.
-
- The construction
foo = $(foo) bar
-
- by itself will notwork, as it does not have an initial base case. Mutually
recursive constructions such as:
B = bar
C = $(B)
B = $(C) boo
-
- will notwork, and will fill memory with expansions.
-
- A default value may be specified, for use if the macro has no definition.
Consider the example
D = $(E:24)
-
- Where Eis not defined within the submit description file, the default
value 24 is used, resulting in
D = 24
-
- This is of limited value, as the scope of macro substitution is the submit
description file. Thus, either the macro is or is not defined within the
submit description file. If the macro is defined, then the default value
is useless. If the macro is not defined, then there is no point in using
it in a submit command.
-
- To use the dollar sign character ( $ ) as a literal, without macro
expansion, use
$(DOLLAR)
-
- In addition to the normal macro, there is also a special kind of macro
called a substitution macrothat allows the substitution of a machine
ClassAd attribute value defined on the resource machine itself (gotten
after a match to the machine has been made) into specific commands within
the submit description file. The substitution macro is of the form:
$$(attribute)
-
- As this form of the substitution macro is only evaluated within the
context of the machine ClassAd, use of a scope resolution prefix TARGET.or
MY.is not allowed.
-
- A common use of this form of the substitution macro is for the
heterogeneous submission of an executable:
executable = povray.$$(OpSys).$$(Arch)
-
- Values for the OpSysand Archattributes are substituted at match time for
any given resource. This example allows HTCondor to automatically choose
the correct executable for the matched machine.
-
- An extension to the syntax of the substitution macro provides an
alternative string to use if the machine attribute within the substitution
macro is undefined. The syntax appears as:
$$(attribute:string_if_attribute_undefined)
-
- An example using this extended syntax provides a path name to a required
input file. Since the file can be placed in different locations on
different machines, the file's path name is given as an argument to the
program.
arguments = $$(input_file_path:/usr/foo)
-
- On the machine, if the attribute input_file_pathis not defined, then the
path /usr/foois used instead.
-
- A further extension to the syntax of the substitution macro allows the
evaluation of a ClassAd expression to define the value. In this form, the
expression may refer to machine attributes by prefacing them with the
TARGET.scope resolution prefix. To place a ClassAd expression into the
substitution macro, square brackets are added to delimit the expression.
The syntax appears as:
$$([ClassAd expression])
-
- An example of a job that uses this syntax may be one that wants to know
how much memory it can use. The application cannot detect this itself, as
it would potentially use all of the memory on a multi-slot machine. So the
job determines the memory per slot, reducing it by 10% to account for
miscellaneous overhead, and passes this as a command line argument to the
application. In the submit description file will be
arguments = --memory $$([TARGET.Memory * 0.9])
-
- To insert two dollar sign characters ( $$ ) as literals into a ClassAd
string, use
$$(DOLLARDOLLAR)
-
- The environment macro, $ENV, allows the evaluation of an environment
variable to be used in setting a submit description file command. The
syntax used is
$ENV(variable)
-
- An example submit description file command that uses this functionality
evaluates the submitter's home directory in order to set the path and file
name of a log file:
log = $ENV(HOME)/jobs/logfile
-
- The environment variable is evaluated when the submit description file is
processed.
-
- The $RANDOM_CHOICE macro allows a random choice to be made from a given
list of parameters at submission time. For an expression, if some
randomness needs to be generated, the macro may appear as
$RANDOM_CHOICE(0,1,2,3,4,5,6)
-
- When evaluated, one of the parameters values will be chosen.
-
-
Comments
-
- Blank lines and lines beginning with a pound sign ('#') character are
ignored by the submit description file parser.
-
-