ERDA backup

There are three different ways to create backup on ERDA: automatic backupserver backup to ERDA with scripting, scheduled tasks, and data transfer

We will walk through all three ways here, so you can pick the one most suited to your needs.

Automatic backup to ERDA

You can combine Avancered data access with Duplicati, a free, open source backup programme you can use to automatically save a backup of your PC on ERDA.

Duplicati saves backup files in a special wrapped format, which means that you can use it to save special files and files with exotic names ERDA doesn't accept. This way, you can easily encrypt your backup as well, so no one else can read it, even if they manage to access it. 

Duplicati comes in both a user friendly graphic version, and a version with command-line, which is better for making backups of servers and certain NAS-solutions. We will walk through the graphic version here.

There are two parts to Duplicati set-up:

1) configuration on ERDA through Duplicati Setup

2) loading the resultant Duplicati configurations on your PC.

Part 1: configuration on ERDA

Part 1:

Go to ERDA Setup and click the Duplicati header.

Enter the name of your backup in the Backup field, for instance the name of the PC or folder you want to back up.

The rest of the fields are voluntary. Remember that SFTP, or the alternative you pick in the Protocol field, must be set up for login with password in ERDA Setup. We shows how you set up passwords for the different protocols here: link.

Click Save Duplicati Settings. At the bottom, you will see links to a .json file for each of the backups you picked. Fetch the file by right clicking on the link and picking Save as.

Part 2: loading Duplicati configurations

To complete the second half of the setup, you must import the .json file in Duplicati.

Download and install the latest 2.x version of Duplicati, which has ready wrapped versions for a range of popular platforms. Here, we use the file that ends in -x86.msi, and which fits with 32-bit Windows. If you're using 64-bit Windows, download the one ending in -x64.msi, and .dmg if you're using Mac. Double-click on the package, that fits your system, and follow the install wizard. On Windows, Update should have the necessary .Net 4.5+ library, but on other platforms you may have to install the corresponding Mono 3.0+ libraries.

When Duplicati has finished installing, it will automatically open overview. Pick Add backup, Import a file, and click Next. Click Browse in the Configuration file field to find the saved .json file. Click Import.

You will now go through the five phases of the Duplicati backup process.

1: General

Before clicking Next here, you can choose to set up encryption. We will show the process without encryption.

You should consider your need for encryption yourself, weighing the burden it is for your PC to encrypt and deencrypt data. If you work with confideltial data, it is always a good idea to encrypt your data, and it may also be demanded by law. For further information, see Danish Data Protection Agency.

Click Next

2: Destination

Here you will see the ERDA settings pre-entered from the configuration file.

Enter your password, with the selected protocol. If you're using standard settings, you'll need to use your ERDA WebDAVS password. 

Click Test connection to ensure that connection and login are set up correctly.

Click Next.

3: Source data

Here you have to pick which folders to include in backup on ERDA.

Basically, Duplicati runs as your normal user profile, so it doesn't automatically have access to all the system files and -facilities needed to shadow-copy open files. Therefore, you may get warnings or errors if, for instance, you try to backup folders outside your user area or of running application data. Either avoid including such folders in your backup, or run Duplicati as privileged user.

Click Next

4: Schedule

Here, you set up the frequency of the automatic backup. You don't have to pick anything under Allow days, since a blank field here has the same effect as picking all. Consider correcting Keep backup to, for instance, 3 years or similar.

Click Save and Next.

5: Options

If you didn't pick encryption you will now be asked whether you're certain. Either click Continue without encryption or Cancel to go back and set up encryption.

Backup is now ready to begin. Click Run now

When backup is complete, you can recreate files and folders with Restore from the Duplicati menu or by opening backup status and picking Restore files… and Duplicati will guide you through the process.

You can find further information in the guide.

Server backup to ERDA with scripting

Some research groups have their own NAS /server for their data. If you want to back up one of those on ERDA without using Duplicati tyou can script the setup of ERDA backup archives.

The easiest way to do so is through a project. Pick a group (if you don't have one yet, follow our guide to Projects (VGrids)). Create a subfolder in one of your folders, for instance, backup.

We assume that, for reasons of security, you prefer to avoid saving your ERDA password on the PC performing backup. To avoid this you can either use SFTP/SSHFS with an ssh-key or create a share link with read- and write-access (as we describe under Share Links). If you're using share links you can access the folder via WebDAVS/SFTP access to link-sharing, that is with the share link ID as user name and password. In this way access to the subfolder is limited and the risk of abuse reduced.

At this point, you can use a command like sftp or lftp to upload files from the storage server. You can even upload ERDA as a network drive with e.g. sshfs and use rsync or  similar if you only want to upload changes to the files.

Set up a workflow rule in your group to integrate the creation of a backup archive in the backup process.

Open ERDA Settings. Under Site Collaboration Links pick advanced in stead of default to access workflows.

Now, on the project overview you'll see a Workflows-pillar. Click Open link and fill out the form with a new workflows-rule.

Actually, you're creating a backup archive of the subfolder every time you upload or save an X.create-backup file to your backup folder.

Scheduled Tasks

ERDA can perform tasks on your behalf through Scheduled Tasks. Here you can plan recurring tasks as well as unique ones. 

The Scheduled Tasks app is not automatically part of the ERDA front page, so click Add, pick the app, and click Save

Using Scheduled Tasks lets you set up ERDA to create a new backup archive of, for instance, the uploads folder every day at 4:38. You can either save to the folder manually, or you can use Server Backup to ERDA with scripting to upload data continually.

The Createbackup command uses a name with three automatic variables like +SHCEDDAY+ to name the archive after the date of running.

You can keep an eye on running and past Scheduled Tasks in View Logs. Read more about available variables and commands at the bottom of Manage Tasks under Schedule Tasks

Data transfer

If you use other storage systems or measuring instruments online, it can be useful to be bale to im- or export data fro ERDA in an effective manner. Users who are used to handling larger datasets may need to download such systems and instruments, regardless of whether they are dedicated Linux/UNIX servers or imaging sites like SLS, ESRF, where you first record data and then access it remorely.

ERDA usupports low level data im- and export in the background via different effective transfer protocols, such as SFTP/RSYNC/HTTPS and WebDAVS. 

Go to Home and click Data Transfer

On the transfers page you will see two headers: Manage Data Transfers and Manage Transfer Keys. The first is for setting up and surveilling transfers, the other is for managing transfer keys (you can use digitale keys in stead of passwords to add security to remote login). Here we will show how you create an ssh-key and use it for importing from a server  to which you have SFTP-access.

If you already have access through login you won't need an ssh-key, but can select Login with password in stead of Login with key when you create the transfer.

Enter the name and click Generate Key.

Now you should be able to see the public part of the key along with brief instructions. Briefly stated, you insert the key where you want to use it for login. 
The ssh-key's private part is solely saved on an unreachable space in ERDA, so that no one else can access it.

Enter the public part on the PC from or to which you want to transfer files, and follow the guide. You can now create a background transfer to or from your ERDA folder. Configuer the transfer to import the contents of the build/cirrular folder on the remote server and save it to the ERDA dataimports/ folder.

Click Request transfer to queue the transfer. You can keep an eye on the transfer in the overview, via the status-buttons, and in the destination folder by clicking on the local component icon. There are many other ways to combine transfers, and overall they follow the same method. 

Click on the status files icon for more information if the transfer fails. It can be quite the task to troubleshoot login and transfers, so contact us if you need help.


To get help with 2-factor authentication and mouting, contact your local IT-support

Find your local IT support

Terms of Use