Upload Large Files with Python-Swiftclient

Our storage service is based on OpenStack Swift. Swift has a limit on the size of a single uploaded file. By default, this is 5GB. Nevertheless, the download size of a single file is virtually unlimited with the concept of segmentation. 

To upload files larger than 5GB, you can use segmentation. It allows you to split them into smaller segments. For example, if the size of a file is 6GB, you can split it into two segments of 3GB. 


What OpenStack Swift storage backend is doing afterward, is to:


1. Split the file in the designated segment size.

2. Upload all segments in parallel.

3. Create a manifest file that allows the file to be downloaded/accessed as one, without noticing the segmentation.


The quickest way to try out this feature is to use the python-swiftclient. For more information go to Upload with Python-Swiftclient.


Please review the example below:

swift upload YOUR_STORAGE_NAME --segment-size 3221225472 LARGE_FILE --os-username YOUR_USERNAME --os-tenant-name YOUR_TENANT --os-password YOUR_FTP_PASSWORD --os-auth-url https://auth.files.us01.cloud.servers.com:5000/v3 --auth-version 3

IMPORTANT! Use-S option to specify the segment size in bytes to use when splitting a large file (e.g. 1 Gigabyte = 1073741824 Bytes, 3 Gigabytes = 3221225472 Bytes) 

The above command will upload the “LARGE_FILE” in segments of 3GB. 


Where

--os-username

Is the first part is your Openstack Swift username (–os-username) and the second part is your Openstack Swift tenant-name (–os-tenant-name) (e.g 1011131.1011131).

Due to the requirements of the Object Storage service (swift) architecture, your FTP username contains two authentication parts separated by dot (.) character.

In the provided example the Openstack Swift username (–os-username) is 1011131


--os-tenant-name

Is the second part of your FTP username is (–os-tenant-name).

In the provided example the Openstack Swift tenant-name (–os-tenant-name) is 1011131


--os-password

Is the FTP password, which can be obtained through the Universal CDN Control Panel.


--os-auth-url

Is the Swift authorization URL.


Below are the authorization URLs that should be used:

https://auth.files.nl01.cloud.servers.com:5000/v3 --auth-version 3 – to connect to your Europe container.

https://auth.files.us01.cloud.servers.com:5000/v3 --auth-version 3 – to connect to your North American container.


Our FTP/SFTP service automatically performs segmentation for larger files.



Upload with Python-Swiftclient

Upload Large Files with Python-Swiftclient

Upload Through OpenStack Swift API

Upload with Rclone