Skip to main content
NASA Advanced Supercomputing (NAS) Division World-class modeling, simulation, and supercomputing in support of the NASA mission

Self-Healing Independent File Transfer (Shift)

The Self-Healing Independent File Transfer (Shift) framework provides high performance and resilience for remote transfers of very large datasets to and from high-performance computing (HPC) facilities, as well as local transfers between filesystems within the facilities, by utilizing a variety of techniques including:

  • End-to-end integrity via cryptographic hashes
  • Throttling of transfers to prevent resource exhaustion
  • Balancing transfers across resources based on load and availability
  • Parallelization of transfers across multiple source and destination hosts for increased redundancy and performance

In addition, Shift is designed to accommodate the diverse heterogeneous environments of a widespread user base with minimal assumptions about operating environments. Shift is unique in its ability to provide advanced reliability and automatic single and multi-file parallelization to any stock command-line transfer application while being easy to deploy by both individual users and organizations.

Advanced Features

Shift includes the following advanced features:

  • Automatic parallelization of transfers
  • Local and remote tar creation and extraction
  • Synchronization based on modification times and checksums (similar to rsync)
  • Automatic file integrity verification and correction
  • Ability to restart transfers
  • Automatic retrieval of files from tape storage (DMF-managed filesystems)
  • Ability to check status of current transfers

Shift has been in active production at the NAS facility since it was deployed in March 2012.

For more information about using Shift at NAS, see Shift File Transfer Overview in the High-End Computing Capability (HECC) Knowledge Base.