download_files_in_parallel#
- astropy.utils.data.download_files_in_parallel(urls, cache='update', show_progress=True, timeout=None, sources=None, multiprocessing_start_method=None, pkgname='astropy')[source]#
Download multiple files in parallel from the given URLs.
Blocks until all files have downloaded. The result is a list of local file paths corresponding to the given urls.
The results will be stored in the cache under the values in
urls
even if they are obtained from some other location viasources
. Seedownload_file
for details.- Parameters:
- urls
list
ofstr
The URLs to retrieve.
- cachebool or “update”, optional
Whether to use the cache (default is
True
). If “update”, always download the remote URLs to see if new data is available and store the result in cache.Changed in version 4.0: The default was changed to
"update"
and setting it toFalse
will print a Warning and set it to"update"
again, because the function will not work properly without cache. UsingTrue
will work as expected.Changed in version 3.0: The default was changed to
True
and setting it toFalse
will print a Warning and set it toTrue
again, because the function will not work properly without cache.- show_progressbool, optional
Whether to display a progress bar during the download (default is
True
)- timeout
float
, optional Timeout for each individual requests in seconds (default is the configurable
astropy.utils.data.Conf.remote_timeout
).- sources
dict
, optional If provided, for each URL a list of URLs to try to obtain the file from. The result will be stored under the original URL. For any URL in this dictionary, the original URL will not be tried unless it is in this list; this is to prevent long waits for a primary server that is known to be inaccessible at the moment.
- multiprocessing_start_method
str
, optional Useful primarily for testing; if in doubt leave it as the default. When using multiprocessing, certain anomalies occur when starting processes with the “spawn” method (the only option on Windows); other anomalies occur with the “fork” method (the default on Linux).
- pkgname
str
, optional The package name to use to locate the download cache. i.e. for
pkgname='astropy'
the default cache location is~/.astropy/cache
.
- urls
- Returns:
Notes
If a URL is unreachable, the downloading will grind to a halt and the exception will propagate upward, but an unpredictable number of files will have been successfully downloaded and will remain in the cache.