Write errors when backup to FTP target

[Tompa]

Hi,

I'm trying to use the new backup feature to store the results on a remote FTP server, but I can see in the restore section that it couldn't write the files to the target. I've tried 2 different ftp-servers as targets with the same result. It seems some files has been written so permissions seems to be in order?

Is there any features in particilar that's required on the server end for this to work?

/Tompa

[pa]

Hi Tompa,

Can you provide the settings for the backup-job and also the error message from the restore-page?

/PA

[Tompa]

The backup job settings are:

• Target: Remote FTP
  Host: tomalars.vndv.com
  Destination dir: bubba2
  Remote user/pass: Corresponding credentials
  
  Run every week, and all days selected.
  Time of day: 04:00
  
  Fullbackups to keep: 5
  Time until invalid: 1 week
  
  No encryption settings

Here's a picture of the restore screen:


It looks almost as if the ftp-host is called without the dots as separator?

If I logon to the FTP after the job has been run it contains a folder structure with the following files:

Code: Select all

bubba2
   |---test
        |----duplicity-full.20090806T203116Z.vol1.difftar.gz (54 032 bytes)

[pa]

Hi Tompa,

Regarding the dots, it is a minor bug in the UI, they are not shown.

It is a bit hard to read, but the date on the "restore page" seems off?

Unfortunately the backup-log are not available through the UI (at least not yet).
Do you have ssh access to bubba?
If you do, can you run the backup-job "by hand", which will give us a bit more information.
It will also print the name of the log file that is created for the job.

Run as root:

Code: Select all

/usr/lib/web-admin/backup.pl backup admin test

where "test" is the name of your backup-job.

/PA

[Tompa]

Yeah, sorry about the blurry picture, but the date is actually correct.

I tried the manual execution and here is the output log:

Code: Select all

NOTICE 1
. NcFTP version is 3.2.2

NOTICE 1
. Reading globbing filelist /home/admin/.backup/test/excludeglob.list

NOTICE 1
. Reading globbing filelist /home/admin/.backup/test/includeglob.list

INFO 1
. Running 'ncftp -u tomalars tomalars.vndv.com' (attempt #1)

INFO 1
. ftp command: 'set yes-i-know-about-NcFTPd yes'

INFO 1
. ftp command: 'set autosave-bookmark-changes no'

INFO 1
. ftp command: 'set confirm-close no'

INFO 1
. ftp command: 'type binary'

INFO 1
. ftp command: 'set passive yes'

INFO 1
. ftp command: 'mkdir bubba2/test'

INFO 1
. ftp command: 'cd bubba2/test'

INFO 1
. ftp command: 'ls -1'

INFO 1
. ftp command: 'quit'

NOTICE 1
. Last full backup date: none

NOTICE 1
. Last full backup is too old, forcing full backup

INFO 1
. Using temporary directory /tmp/duplicity-LIEdsJ-tempdir

INFO 1
. Using temporary directory /home/admin/.backup/test/archives/duplicity-HGy6WJ-tempdir

INFO 1
. AsyncScheduler: instantiating at concurrency 0

INFO 4 '.'
. Generating delta - new file: .

INFO 4 'admin'
. Generating delta - new file: admin

INFO 4 'admin/.backup'
. Generating delta - new file: admin/.backup

INFO 4 'admin/.backup/test'
. Generating delta - new file: admin/.backup/test

INFO 4 'admin/.backup/test/excludeglob.list'
. Generating delta - new file: admin/.backup/test/excludeglob.list

INFO 4 'admin/.backup/test/fileinfo'
. Generating delta - new file: admin/.backup/test/fileinfo

INFO 4 'admin/.backup/test/include_childglob.list'
. Generating delta - new file: admin/.backup/test/include_childglob.list

INFO 4 'admin/.backup/test/includeglob.list'
. Generating delta - new file: admin/.backup/test/includeglob.list

INFO 4 'admin/.backup/test/jobdata'
. Generating delta - new file: admin/.backup/test/jobdata

INFO 4 'tomalars'
. Generating delta - new file: tomalars

INFO 4 'tomalars/Ekonomi'
. Generating delta - new file: tomalars/Ekonomi

INFO 4 'tomalars/Ekonomi/Gemensam ekonomi 2009.xlsx'
. Generating delta - new file: tomalars/Ekonomi/Gemensam ekonomi 2009.xlsx

INFO 1
. AsyncScheduler: running task synchronously (asynchronicity disabled)

INFO 1
. Running 'ncftp -u tomalars tomalars.vndv.com' (attempt #1)

INFO 1
. ftp command: 'set yes-i-know-about-NcFTPd yes'

INFO 1
. ftp command: 'set autosave-bookmark-changes no'

INFO 1
. ftp command: 'set confirm-close no'

INFO 1
. ftp command: 'type binary'

INFO 1
. ftp command: 'set passive yes'

INFO 1
. ftp command: 'mkdir bubba2/test'

INFO 1
. ftp command: 'cd bubba2/test'

INFO 1
. ftp command: 'put -z /tmp/duplicity-LIEdsJ-tempdir/mktemp-ujGbU7-2 duplicity-full.20090807T073316Z.vol1.difftar.gz'

INFO 1
. Timeout waiting for response

WARNING 1
. Running 'ncftp -u tomalars tomalars.vndv.com' failed (attempt #1)

INFO 1
. Running 'ncftp -u tomalars tomalars.vndv.com' (attempt #2)

INFO 1
. Timeout waiting for response

WARNING 1
. Running 'ncftp -u tomalars tomalars.vndv.com' failed (attempt #2)

WARNING 1
. Giving up trying to execute 'ncftp -u tomalars tomalars.vndv.com' after 2 attempts

ERROR 30 BackendException
. Traceback (most recent call last):
.   File "/usr/bin/duplicity", line 589, in ?
.     with_tempdir(main)
.   File "/usr/bin/duplicity", line 582, in with_tempdir
.     fn()
.   File "/usr/bin/duplicity", line 562, in main
.     full_backup(col_stats)
.   File "/usr/bin/duplicity", line 235, in full_backup
.     bytes_written = write_multivol("full", tarblock_iter, globals.backend)
.   File "/usr/bin/duplicity", line 162, in write_multivol
.     (tdp, dest_filename)))
.   File "/usr/lib/python2.4/site-packages/duplicity/asyncscheduler.py", line 148, in schedule_task
.     return self.__run_synchronously(fn, params)
.   File "/usr/lib/python2.4/site-packages/duplicity/asyncscheduler.py", line 174, in __run_synchronously
.     ret = fn(*params)
.   File "/usr/bin/duplicity", line 161, in <lambda>
.     async_waiters.append(io_scheduler.schedule_task(lambda tdp, dest_filename: put(tdp, dest_filename),
.   File "/usr/bin/duplicity", line 115, in put
.     backend.put(tdp, dest_filename)
.   File "/usr/lib/python2.4/site-packages/duplicity/backends/ftpbackend.py", line 183, in put
.     res = self.run_ftp_command_list(commands)
.   File "/usr/lib/python2.4/site-packages/duplicity/backends/ftpbackend.py", line 178, in run_ftp_command_list
.     raise BackendException("Error running '%s'" % self.commandline)
. BackendException: Error running 'ncftp -u tomalars tomalars.vndv.com'
. 

[pa]

Can you try to log into your ftp server using ncftp and upload the file?

Code: Select all

ncftp -u tomalars tomalars.vndv.com

then uploding the file:

Code: Select all

put -z /tmp/duplicity-LIEdsJ-tempdir/mktemp-ujGbU7-2 duplicity-full.20090807T073316Z.vol1.difftar.gz

If the sourcefile does not exist just create a dummy file:

Code: Select all

mkdir /tmp/duplicity-LIEdsJ-tempdir
dd if=/dev/zero of=/tmp/duplicity-LIEdsJ-tempdir/mktemp-ujGbU7-2 bs=1M count=5

There is something that goes wrong in the upload process since there is a timeout waiting for the response.
This might help us to get more information on what is happening on the serverside.

/PA

[Tompa]

Hi again and thanks for your help.

Those commands seems to work as they should, though it took quite a long time to upload the 5mb file (transfer speed 10kb/s), maybe thats the reason for the timeout?

I will try the same backup job using another ftp target and check the logs accordingly.

/Tompa

[RandomUsername]

I'm having a similar problem.

Uploading files manually using ncftp works fine as per pa's suggestion above.

Running a backup from the web interface or the command line results in this error:

Code: Select all

. Running 'ncftp -u xxxxxxx xxxxxx.onlinestoragesolution.net' failed (attempt #2)
. Giving up trying to execute 'ncftp -u xxxxxx xxxxxx.onlinestoragesolution.net' after 2 attempts

Although a tar file does get uploaded it's not big enough to be my entire backup set.

The interesting thing about the above error is that the ncftp line doesn't have a password parameter. Is this the reason it's not working?

Here's the backup job file (which does have a password in it):

Code: Select all

local_user = admin
jobname = nightlybackup
target_protocol = FTP
target_host = xxxxxx.onlinestoragesolution.net
target_path = 
target_user = xxxxxx
target_FTPpasswd = xxxxxx
monthweek = disabled
nbr_fullbackups = 1
full_expiretime = 1D
encrypt = on
GPG_key = xxxxxx

I'd appreciate an urgent response to this as I don't like not having my stuff backed up.

[asparak]

Can you insert the -p parameter into the line and test?

[RandomUsername]

Can you insert the -p parameter into the line and test?

Hi asparak,

I'm not sure where you mean.

If I use ncftp and then enter my username and password it works fine.

If I use

Code: Select all

/usr/lib/web-admin/backup.pl backup admin nightlybackup

it generates the error

Code: Select all

. Running 'ncftp -u xxxxxxx xxxxxx.onlinestoragesolution.net' failed (attempt #2)
. Giving up trying to execute 'ncftp -u xxxxxx xxxxxx.onlinestoragesolution.net' after 2 attempts

Which suggests that even though the jobdata file has a password in it the backup.pl code isn't recognising that fact.

[pa]

The password shall not be listed in this entry in the log file so that is correct.

It seems as if the backup can not find your target host.
Can you post a couple of lines prior to this:

Code: Select all

 Running 'ncftp -u xxxxxxx xxxxxx.onlinestoragesolution.net' failed (attempt #2)

/PA

[RandomUsername]

Hi pa,

Here's the full output when running /usr/lib/web-admin/backup.pl backup admin test

Code: Select all

Queueing job: test for user: admin
New job found for admin test
Running queued job: test for user: admin 
RUN NOW: admin test 
Logging to /var/log/backup/admin/test/2009-10-28-15:15:51.log
Running ncftpput
/usr/bin/ncftpput -t 5 -r 1 -u 'xxxxxx' -p 'xxxxxx -m xxxxxx.onlinestoragesolution.net  /tmp/backup.READMEStarting backup
Backup done.
Errors/warnings encountered during backup.
Errors found:
Errors/warnings encountered during backup.
. Running 'ncftp -u xxxxxx xxxxxx.onlinestoragesolution.net' failed (attempt #1)
. Running 'ncftp -u xxxxxx xxxxxx.onlinestoragesolution.net' failed (attempt #2)
. Giving up trying to execute 'ncftp -u xxxxxx xxxxxx.onlinestoragesolution.net' after 2 attempts

Info to: /home/admin/.backup/test/fileinfo/2009-10-28-15:18:23.err.info 

The file /home/admin/.backup/test/fileinfo/2009-10-28-15:18:23.err.info gives the same information.

Some data is getting uploaded though because there's a 2.7 MB tarball sitting on the ftp server (the total size of what I'm trying to backup is more like 12 MB and they're jpgs so won't compress that much). If I download the tar it seems to contain a 2.7 GB file called duplicity-full.20091028T15155Z.vol1.difftar which won't extract or open.

Thanks.

[RandomUsername]

This afternoon I remembered I've got a bit of webspace with ftp access so I decided to try to backup to that so I could at least eliminate the ftp server from the equation. The results were exactly the same.

[RandomUsername]

Hi pa,

I'd really like some feedback here. It's features like this that made me buy a Bubba over a (cheaper) dumb NAS box. It'd be really great if the advertised features actually worked!

I've abandoned a Mozy account that my old Windows file server backed up to. I've only got 30 days to get an online backup running on my Bubba before I lose the stuff backed up with Mozy and I have no backups at all.

Thanks.

[pa]

Hi,

I just ran a test with your jobfile (and the username you send me separately) and that worked as far as connecting. Of course the backup did not work since I could not log in...

To me it seems as if there is a problem with resolving the name, or a timeout when connecting.
Can you run:

Code: Select all

host xxxxx.onlinestoragesolution.net

and provide the ouput?

Also, when you run ncftp by hand:

Code: Select all

ncftp -u username xxxxx.onlinestoragesolution.net

how long does it take to connect?

/PA