duplicity backup fails with obscure error message Ubuntu 22.04

I think that /tmp filename is a file that duplicity proceeds to throw away. It copies files there, tries to unpack them, then cleans up after itself (regardless of success). So that error message would be a lot more useful if it indicated what the actual source file it had copied from the storage location was named (like duplicity-vol1.gz or whatever).

The error message makes it sound like the file it is trying to unpack is corrupted. It's possible that a backup file got interrupted/cutoff during writing/transfer?

I'd like a log file, please. This is going to be big and I only need a portion. Please run duplicity with -v9, redirect to a log file, then run:

$ head -n 200 < log > log.head
$ tail -n 200 < log > log.tail

Attach, do NOT copy/paste, log.head and log.tail to this bug report. See green button below.

Kenneth:

First, thank you!

I don't run duplicity via the command line. I run it using the Ubuntu
GUI app "Backups" which I believe is just a gui-name for deja-dup. In
that setup I have a saved Preferences which governs how the Backup is
done, which files/directories to back up, which to exclude, where to
send the backup, etc.

Trying duplicity -v9 fails with
$ duplicity -v9
Command line error: Expected 2 args, got 0
Enter 'duplicity --help' for help screen.

Evidently, the deja-dup app is saving this information in a config file
somewhere and then building a command line to send to duplicity. I
don't see how to invoke duplicity so as to use these config files, whose
location I don't know either.

So, next step is to try to invoke the debugging via a deja-dup command line.

I try this (per deja-dup --help):

$deja-dup --gtk-debug=9 > log

and it starts the GUI, producing the same failure message; however
nothing is appended to the log file, so evidently --gtk-debug=9 in
deja-dup is not equivalent to duplicity -v9.

Finally, using the Google, I find that the way to turn logging on for
deja-dup is

DEJA_DUP_DEBUG=1 deja-dup

Redirecting this to log does indeed produce a massive file.
Hopefully, this is what you are looking for. If not please let me know
how to
1) run duplicity so that my deja-dup preferences are used or
2) run deja-dup to produce the input you want.

log head and tail will be attached.

Thank you for responding.

I'm convinced the error is a truncated .gz file, but duplicity is not telling us which one. So, let's go to the source directly.

On your remote machine:

$ cd /home/scohen/Taormina
$ for f in *.gz; do echo "--- $f ---"; tar tzf $f; done

This will list the index of all the *.gz files and failure should indicate which one is truncated. That file will be the one with '--- filename ---' prior to the error.

Then we can figure out what to do next.

Your command gave too much output. After a few attempts, I modified it to this, which produced useful data:

$ for f in *.gz; do echo "--- $f ---" >>output; tar tzf $f >/dev/null 2>>output; done

This confirms that your hypothesis is correct. The file with the error is
duplicity-full-signatures.20211007T135951Z.sigtar.gz

Good catch on the script.

Looks like you've run afoul of two bugs, one in Samba and one in duplicity:

1. https://bugs.launchpad.net/duplicity/+bug/385495 - no solution yet.

2. Google "samba large file size". Looks like 2GB is the limit. I suspect the file was written on the last backup and samba simply truncated it at 2GB. The only option suggested in what I read was to mount the drive as in https://www.cyberciti.biz/faq/howto-access-files-greater-than-2gb-samba, and samba will accept the LFS. There may be a way to do it in config.

I would suggest mounting the drive with LFS enabled and rerunning the backup after running
"duplicity cleanup --force <target>".

Hmmmm. Many questions.

The networked machine is an old desktop running a very old version of Ubuntu. There is no Windows involved here, and no mount so I'm not understanding where Samba comes into the picture. As I said in the original statement of the bug, I am connecting via the SSH protocol, so I am not entirely sure where the question of mounting a drive comes in. If I do mount a drive from the other machine, would this solve my problem and enable backups to finish, and if so, what protocol should I then set up duplicity to use? Or would another of the available duplicity protocols work better for me?

Even more puzzling, a directory listing of the troublesome file shows it is < 2GB in size by a factor of 16 or so:

-rw-rw-r-- 1 scohen scohen 262180864 Oct 7 2021 duplicity-full-signatures.20211007T135951Z.sigtar.gz

I suppose truncation might have occurred to a size well < 2GB.

Also, why is this file so big? In looking over the earlier output, I appear to be backing up a lot of old junk that no longer requires backup or even existence. Would eliminating some of it help me?

Or, since I have no need of earlier backups, might I be well served by simply erasing everything in the Taormina directory of the backup drive and starting over?

Thanks.

Perhaps I should also add that my installation of Ubuntu 22.04 dates from last week when it had just come out. The timestamps on the backup files on the remote machine date from last summer/fall, when I was running 20.x or 21.x. Might a bug have been there that is no longer there?

Might I ask why the admins have marked this bug as "Invalid" and unassigned to anyone? Does this mean that no one will look at it anymore? I feel that this action should be explained.

That said, I have taken the step of eliminating previous backups and now the system works again and the information Kenneth provided was helpful in leading me there, although I do question the Samba theory for reasons stated above.

I feel that at minimum duplicity should issue a more informative error message, indicating perhaps the name of the file that actually has the truncation problem.

Whoops! You're right. I'm marking this as 'confirmed' and will fix the error message to the correct filename.

I have one idea, based on my experience yesterday.

I renamed the /home/scohen/Taormina directory and then created a new directory of that name on the remote drive. A quick check of disk space informed me that I would not have enough room for a full backup on the remote drive. So I looked more closely and saw that there were a lot of very old backups under another directory that I could easily do without on the drive, and deleted those, ran the backup successfully.

My idea is this: what happens if there is not enough room to complete the backup on the intended drive? What error message does duplicity give in that case? I have no memory of any particular error message on 2021/10/07, though I do remember a problem, and being unable to solve it then, I simply gave up on backups for awhile, which I hoped to get back to with 22.04.

My idea is that this situation could lead to a truncated backup file, and if the error message were not clear, confusion on the part of the user. If a backup fails due to lack of disk space on the destination, a specific error message to that effect would be useful. I don't know if that's what happened here but it could be.

All systems run a bit strange if disk space runs out, most not too gracefully. duplicity will estimate the space needed on the local system to run duplicity, but cannot check the remote. It will fail with whatever status the remote returns, which you should then correct if possible. I don't know what caused your truncation, but the error message should now be clearer and have the responsible filename in it.

Yeah, I see that SSH doesn't have a specific disk-full error message.