Large files and resumable upload

2019-08-27 10:47

Our storage service is based on OpenStack Swift and there is a limitation on the size of a singe file, which can be uploaded.  By default it is 5GB. This, however, does NOT limit the size of the file which can be downloaded.

In other words, in order to upload files larger than 5GB, you can use the so-called "segmentation". It will allow you to split the larger file in smaller chunks, for example to 1GB, and in this way you can upload file which is larger than the limitation.

What storage backend is doing afterwards, is to:

  • split the file in the designated segment size
  • upload all segments in parallel
  • create manifest file which allows the file to be downloaded/accessed as one, without the client noticing it is segmented

The quickest way to try out this feature is to use the swift tool(for more information on the parameters, please check Swiftclient):

swift --os-username YOUR_USERNAME --os-tenant-name YOUR_TENANT --os-password YOUR_PASSWORD --os-auth-url upload YOUR_CONTAINER -S 1073741824 --segment-container YOUR_CONTAINER large_file

The above command will upload the "large_file" in segments of 1GB. Of course, you can specify any size of the segments(not bigger than 5GB)

Our FTP/SFTP service automatically performs segmentation for larger files.

Resumable upload

Unfortunately OpenStack's Swift does not support resumable upload at the moment.

If you are experiencing upload problems due to poor network connectivity or any other issues you can use the segmenation mentioned above to split your files in smaller chunks which will be uploaded in parallel and should significantly improve the process.

For more information on the middleware and different methods how you can use it, please check the following article: