Downloads

WEAVE data products can be very large. A test request with a smaller set of search results may help towards understanding how long a larger request might take to complete. We also suggest checking your internet bandwidth before starting to download a very large file, once it is ready to be downloaded.

In order to request data to download from the WEAVE Archive System, it is first necessary to perform a search, from which a job will be queued on the server. Files cannot be downloaded or listed directly, since each target may have many associated files that the server needs to collect and generate, depending on the download request.

It is important to keep in mind that downloads will be generated on all of the search results, so it highly recommended to set some filters to limit the size of the data set you will be downloading.

The download button will appear once you have performed a search and your search has generated a result. Depending on whether the search has been performed on an iDR or an iDRC, the download button will allow you to choose between a Regular or a Subscription download.

Download button on search interface

"Regular" download type

This is the default download method and the one users are likely to use most often. Clicking on the download button will open a dialogue box, from which you can select various options and set a custom name for the download.

If the results obtained are Single Fibre type, it will be possible to choose between downloading only the specific Target fibres files or the file for the complete OB.

Selecting files types and pipeline products

By clicking on Downloads icon in the main menu, you can see the list of generated downloads and the status of each one.

The default download is a links.out file with the URLs of all of the files contained in the job. This file can be used with a download manager software or in your custom software if applicable.

In order to download files using conventional methods, we provide several custom commands that use traditional command-line clients (rsync, wget, curl, etc.) accessible from the detail view of each job.

• Status
  • EXECUTED: job is complete and data can be downloaded
  • EXECUTING: server is collecting data – please wait
  • QUEUED: pending processing
  • ERROR: there was a problem – please check the detailed view
• Size: total download size
• Started/Finished: server processing times
• Query: computed query based on parameters the user has provided for the search
• Output: List, CSV
• Files: total number of files
• Targets: total number of targets
• Download links: options for downloading files through different common clients. Copy-paste any into your terminal, making sure you have the chosen client already installed.

"Subscriptions" download type. [DEPRECATED]

This is another download option, designed exclusively for iDRC. Unlike an iDR, the iDRC data cycle is not closed, so new data may appear daily. In this case, and to avoid needing to frequently check whether new data have appeared, a download subscription can be generated. Every subscription job is automatically re-scheduled on the WEAVE Archive System every day to gather new data that match the search parameters.

To set up a subscription download, simply select Subscription from the drop-down menu (only applicable if you are browsing through iDRC data). The rest of the process is exactly the same as for the Regular download type.

Subscription job dialog

• Status
  • Scheduled to refresh: job is complete – data can be downloaded and the next refresh is scheduled
  • Scheduled to delete: job will be deleted on the next refresh
  • Paused: job is completed – data can be downloaded but automatic scheduling is disabled
  • ERROR: there was a problem – please check the detailed view
• Size: total download size
• Started: server processing time
• Last update: last time the job was refreshed
• Expiration: Days remaining to keep the job active
• Query: computed query based on parameters the user has provided for the search
• Output: job progress
• Files: total number of files
• Targets: total number of targets
• RSYNC: command for downloading files through rsync protocol. Copy-paste into your terminal, making sure you have the rsync client already installed.