With the Rclone command-line program you can manage your Universal CDN Storage containers. It is used to sync, copy, upload files and directories to and from storage containers.
Important! For uploading files larger than 5GB, please refer to Upload Large Files with Python-Swiftclient.
To configure a new remote that allows access to your Universal CDN Storage container on Rclone please follow the steps outlined below:
1. Install Rclone by following the article at https://rclone.org/install/
2. Run the following command in the terminal window after installing the Rclone tool:
rclone config
$ rclone config
No remotes found - make a new one
n) New remote
s) Set configuration password
q) Quit config
n/s/q>
3. Type n and hit enter to add a new remote.
4. Choose a remote name. For this tutorial, we have chosen “new_remote” as a name for the remote.
name> new_remote
5. Type of storage to configure.
Type of storage to configure.
Enter a string value. Press Enter for the default ("").
Choose a number from below, or type in your own valueSelect the following option (28):
28 / OpenStack Swift (Rackspace Cloud Files, Memset Memstore, OVH)
\ "swift"
6. On the next step, type 1 and hit enter.
Get swift credentials from environment variables in standard OpenStack form.
Enter a boolean value (true or false). Press Enter for the default ("false").
Choose a number from below, or type in your own value
1 / Enter swift credentials in the next step
\ "false"7. User name to log in (OS_USERNAME). Enter your username which is the first part of the FTP username (e.g. 1011131).
User name to log in (OS_USERNAME).
Enter a string value. Press Enter for the default ("").
user> 1011131
8. API key or password (OS_PASSWORD). Enter the FTP password of your CDN storage.
API key or password (OS_PASSWORD).
Enter a string value. Press Enter for the default ("").
key> PRwvGMGbnKfabFto
9. Authentication URL for server (OS_AUTH_URL). Enter authentication URL which is:
https://auth.files.nl01.cloud.servers.com:5000/v3 – if your storage is located in Europe.
https://auth.files.us01.cloud.servers.com:5000/v3 – if your storage is located in North America.
auth> https://auth.files.us01.cloud.servers.com:5000/v3
10. User ID to log in – optional – most swift systems use user and leave this blank (v3 auth) (OS_USER_ID). Leave this field blank and press Enter for the default.
11. User domain – Please enter “default“ explicitly as a user domain name and hit enter.
domain = default
12. Tenant name – optional for v1 auth, required otherwise. Enter your tenant name which second part of the FTP username. For example, if your FTP username is 1012231.1011131, the part after the period is your tenant name – 1011131.
Enter a string value. Press Enter for the default ("").
tenant> 1011131
13. Tenant ID – optional for v1 auth, this or tenant required otherwise (OS_TENANT_ID). Leave this field blank and press Enter for the default.
14. Tenant domain – Please enter “default“ explicitly as a tenant name and hit enter.
tenant_domain = default
15. Region name – optional (OS_REGION_NAME). Leave this field blank and press Enter for the default.
16. Storage URL – optional (OS_STORAGE_URL). Leave this field blank and press Enter for the default.
17. Auth Token from alternate authentication – optional (OS_AUTH_TOKEN). Leave this field blank and press Enter for the default.
18. Application Credential ID (OS_APPLICATION_CREDENTIAL_ID). Leave this field blank and press Enter for the default.
19. Application Credential Name (OS_APPLICATION_CREDENTIAL_NAME). Leave this field blank and press Enter for the default.
20. Application Credential Secret (OS_APPLICATION_CREDENTIAL_SECRET). Leave this field blank and press Enter for the default.
21. AuthVersion – optional – set to (1,2,3) if your auth URL has no version (ST_AUTH_VERSION). eave this field blank and press Enter for the default.
22. Endpoint type to choose from the service catalogue (OS_ENDPOINT_TYPE). Choose option 1 and hit enter key.
endpoint_type> 123. The storage policy to use when creating a new container. Choose option 1 and hit enter key.
storage_policy> 124. Edit advanced config?. Please type n for No and hit enter.
n) No (default)25. Finally, you will see the Remote config. Type y for Yes and hit enter.
y) Yes this is OK (default)
Overall, your rclone config file should be similar to the following one:
[new_remote]
type = swift
env_auth = false
user = 1011131
key = KGwvJMGjmyqabFau
auth = https://auth.files.us01.cloud.servers.com:5000/v3
tenant = 1011131
endpoint_type = public
domain = default
tenant_domain = defaultIMPORTANT! Please note that the storage containers must be created only from the Universal CDN Control Panel. Storage containers created not from the Control Panel will be rejected by the CDN system.
We will provide a few examples of how to copy or sync files from your local machine to a remote storage container, or vice versa, or from one storage container to another.
To copy the files from one storage container to another, use the following command:
rclone copy source:sourcepath dest:destpath
$ rclone copy 'new_remote:test1' other_remote:test2Where:
– new_remote is the remote name (name> remote)
– other_remote is the remote name of your other remote added (name> remote)
– test1 is the name of your storage container which data is used to configure new_remote
– test2 is the name of your storage container which data is used to configure other_remote
Note that you need to configure separate remotes in Rclone. In the example, we have configured a new_remote linked with the test1 storage, and other_remote linked with the test2 storage container.
To copy the files from a storage container to your local directory, use the following command:
rclone copy source:sourcepath dest:destpath
$ rclone copy 'new_remote:test1' /path/to/your/directoryWhere:
– new_remote is your remote name (name> new_remote)
– test1 is the name of your container.
To copy the files from your local directory to a storage container, use the following command:
rclone copy source:local dest:destpath
$ rclone copy 'path/to/your/directory' new_remote:test1Where:
– new_remote is your remote name (name> new_remote)
– test1 is the name of your container.
To sync the files from one storage container to another, use the following command:
rclone sync source:sourcepath dest:destpath
$ rclone sync 'new_remote:test1' new_remote:test2Where:
– new_remote is the remote name (name> new_remote)
– test1 and test2 are the names of your containers
To sync the files from a directory on your local machine to your storage container, use the following command:
rclone sync source:sourcepath dest:destpath
$ rclone sync 'new_remote:test1' /path/to/your/directoryWhere:
– new_remote is your remote name (name> new_remote)
– test1 is the name of your container.
To copy the files from one storage container to your local machine, use the following command:
rclone sync source:sourcepath dest:destpath
$ rclone sync 'path/to/your/directory' new_remote:test1Where:
– new_remote is your remote name (name> remote)
– test1 is the name of your container.
The tutorial is based on the rclone v1.55.1. For more information please visit rclone.org.
Upload with Python-Swiftclient