Gabriel Filion [Fri, 30 Apr 2010 05:53:13 +0000 (01:53 -0400)]
code clarity: one-letter var carried for too long
In split-cmd.py, the "w" variable is first seen on line 55 and is kept
around until line 96. Variables that are sparsely used in a medium
distance in the code should have a name that carries more sense when
read on its own.
Change "w" into "pack_writer" to better identify the purpose of the
variable.
Jon Dowland [Wed, 28 Apr 2010 14:41:05 +0000 (15:41 +0100)]
adjust .md files to make lexgrog happy
the whatis(1) tool cannot parse the bup manpages, because there
are two words before the '-' separator. This patch joins the words
using another '-', in the same fashion as git, to overcome this
limitation.
Before:
$ whatis bup fuse
bup (1) - Backup program using rolling checksums and git file fo...
fuse: nothing appropriate.
$ whatis bup-fuse
bup-fuse: nothing appropriate.
After:
$ whatis bup-fuse
bup-fuse (1) - mount a bup repository as a filesystem
Gabriel Filion [Wed, 28 Apr 2010 17:12:05 +0000 (13:12 -0400)]
Documentation: some placeholders are lost
Some pieces of text in the documentation files use the <...> syntax to
mark named placeholders. However, the conversion done by pandoc from
Markdown to makefile makes some of these placeholders disappear.
The affected elements are those that contain only characters that could
be valid for an e-mail address or a URL, but are not supposed to be one
of both. Also, elements inside `...`-style code blocks are unaffected.
Fix this situation by escaping the < and > characters where the tags
disappear.
Jon Dowland [Wed, 28 Apr 2010 13:50:30 +0000 (14:50 +0100)]
add -o/--allow-other to bup-fuse
Setting the fuse option allow_other will fail if user_allow_other
is not set in fuse.conf. Add toggle -o/--allow-other to bup-fuse
(disabled by default).
cmd/version, etc: fix version number detection stuff.
Gabriel Filion pointed out that bup's version number (which we added to the
man pages automatically) was not detected when you used a bup tarball
generated from 'git archive' (or more likely, you let GitHub call 'git
archive' for you). That makes sense, since our version detection was based
on having a .git directory around, which the tarball doesn't.
Instead, let's create a .gitattributes file and have it auto-substitute some
version information during 'git archive'. Of course, if we actually *do*
have a .git directory, continue to use that.
While we're here, add a new 'bup version' command and alias "bup --version"
and "bup -V" to call it, since those options are pretty standard.
vfs: take advantage of bup chunking to make file seeking faster.
If you have a huge file, you can now seek around inside it (eg. in 'bup
fuse') without having to read its entire contents. Calculating the file
size is also really fast now.
This makes a bup fuse-mounted filesystem much more useful for real-time
access. For example, I was able to connect to an sqlite3 database and have
it work at a reasonable speed. (Obviously, since 'bup fuse' is written in
python and doesn't currently support threading, the speed could still be
improved, but at least it's no longer godawful.)
git.CatPipe: more resilience against weird errors.
Notably, MemoryErrors thrown because the file we're trying to load into
memory is too big to load all at once. Now the MemoryError gets thrown, but
the main program is potentially able to recover from it because CatPipe at
least doesn't get into an inconsistent state.
Also we can recover nicely if some lamer kills our git-cat-file subprocess.
The AutoFlushIter we were using for this purpose turns out to not have been
good enough, and it's never been used anywhere but in CatPipe, so I've
revised it further and renamed it to git.AbortableIter.
cmd/save: when a file is chunked, mangle its name from * to *.bup
Files that are already named *.bup are renamed to *.bup.bupl, so that we can
just always drop either .bup or .bupl from a filename if it's there, and the
result will be the original filename.
Also updated lib/bup/vfs.py to demangle the names appropriately, and treat
git trees named *.bup as real chunked files (ie. by joining them back
together).
Also change main.py to search around in appropriate places for the installed
library files. By default, if your bup is in /usr/bin/bup, it'll look in
/usr/lib/bup. (It drops two words off the end of the filename and adds
/lib/bup to the end.)
This also makes the Debian packager at
http://git.debian.org/collab-maint/bup
actually produce a usable package.
It's annoying when your log messages come out before stdout messages do.
But it's equally annoying (and inefficient) to have to flush every time you
print something. This seems like a nice compromise.
Get rid of a sha-related DeprecationWarning in python 2.6.
hashlib is only available in python 2.5 or higher, but the 'sha' module
produces a DeprecationWarning in python 2.6 or higher. We want to support
python 2.4 and above without any stupid warnings, so let's try using
hashlib. If it fails, switch to the old sha module.
Avery Pennarun [Sun, 21 Mar 2010 20:48:23 +0000 (16:48 -0400)]
server: only suggest a max of one pack per receive-objects cycle.
Since the client only handles one at a time and forgets the others anyway,
suggesting others is a bit of a waste of time... and because of the cheating
way we figure out which index to suggest when using a midx, suggesting packs
is more expensive than it should be anyway.
The "correct" fix in the long term will be to make the client accept
multiple suggestions at once, plus make midx files a little smarter about
figuring out which pack is the one that needs to be suggested. But in the
meantime, this makes things a little nicer: there are fewer confusing log
messages from the server, and a lot less disk grinding related to looking
into which pack to suggest, followed by finding out that we've already
suggested that pack anyway.
Avery Pennarun [Sun, 21 Mar 2010 04:41:52 +0000 (00:41 -0400)]
rbackup-cmd: we can now backup a *remote* machine to a *local* server.
The -r option to split and save allowed you to backup from a local machine
to a remote server, but that doesn't always work; sometimes the machine you
want to backup is out on the Internet, and the backup repo is safe behind a
firewall. In that case, you can ssh *out* from the secure backup machine to
the public server, but not vice versa, and you were out of luck. Some
people have apparently been doing this:
ssh publicserver tar -c / | bup split -n publicserver
(ie. running tar remotely, piped to a local bup split) but that isn't
efficient, because it sends *all* the data from the remote server over the
network before deduplicating it locally. Now you can do instead:
bup rbackup publicserver index -vux /
bup rbackup publicserver save -n publicserver /
And get all the usual advantages of 'bup save -r', except the server runs
locally and the client runs remotely.
Avery Pennarun [Sun, 21 Mar 2010 03:03:53 +0000 (23:03 -0400)]
client: Extract 'bup server' connection code into its own module.
The screwball function we use to let us run 'bup xxx' on a remote server
after correctly setting the PATH variable is about to become useful for more
than just 'bup server'.
Avery Pennarun [Sun, 21 Mar 2010 04:36:31 +0000 (00:36 -0400)]
options: allow user to specify an alternative to getopt.gnu_getopt.
The most likely alternative is getopt.getopt, which doesn't rearrange
arguments. That would mean "-a foo -p" is considered as the option "-a"
followed by the non-option arguments ['foo', '-p'].
The non-gnu behaviour is annoying most of the time, but can be useful when
you're receiving command lines that you want to pass verbatim to someone
else.
Avery Pennarun [Sun, 21 Mar 2010 05:47:24 +0000 (01:47 -0400)]
save/index/drecurse: correct handling for fifos and nonexistent paths.
When indexing a fifo, you can try to open it (for security reasons) but it
has to be O_NDELAY just in case the fifo doesn't have anyone on the other
end; otherwise indexing can freeze.
In index.reduce_paths(), we weren't reporting ENOENT for reasons I can no
longer remember, but I think they must have been wrong. Obviously if
someone specifies a nonexistent path on the command line, we should barf
rather than silently not back it up.
Avery Pennarun [Sun, 21 Mar 2010 04:34:21 +0000 (00:34 -0400)]
main.py: don't leak a file descriptor.
subprocess.Popen() is a little weird about when it closes the file
descriptors you give it. In this case, we have to dup() it because if
stderr=2 (the default) and stdout=2 (because fix_stderr), it'll close fd 2.
But if we dup it first, it *won't* close the dup, because stdout!=stderr.
So we have to dup it, but then we have to close it ourselves.
This was apparently harmless (it just resulted in an extra fd#3 getting
passed around to subprocesses as a clone of fd#2) but it was still wrong.
Lukasz Kosewski [Mon, 15 Mar 2010 03:20:08 +0000 (23:20 -0400)]
cmd/index-cmd.py: How it pains me to have to explicitly close() stuff
If we don't explicitly close() the wr reader object while running
update-index, the corresponding writer object won't be able to unlink
its temporary file under Cygwin.
Lukasz Kosewski [Mon, 15 Mar 2010 03:17:42 +0000 (23:17 -0400)]
lib/bup/index.py: mmap.mmap() objects need to be closed() for Win32.
Not *entirely* sure why this is the case, but it appears through some
refcounting weirdness, just setting the mmap variables to None in
index.Readers doesn't cause the mmap to be freed under Cygwin, though
I can't find any reason why this would be the case.
Naturally, this caused all sort of pain when we attempt to unlink
an mmaped file created while running bup index --check -u.
Fix the issue by explicitly .close()ing the mmap in Reader.close().
Avery Pennarun [Sun, 14 Mar 2010 06:59:45 +0000 (01:59 -0500)]
PackIdxList.refresh(): remember to exclude old midx files.
Previously, if you called refresh(), it would fail to consider
the contents of already-loaded .midx files as already-loaded. That means
it would load all the constituent .idx files, so you'd actually lose all the
advantages of the .midx after the first refresh().
Thus, the midx optimization mainly worked before you filled up your first
pack (about 1GB of data saved) or until you got an index suggestion. This
explains why backups would slow down significantly after running for a
while.
Also, get rid of the stupid forget_packs option; just automatically prune
the packs that aren't relevant after the refresh. This avoids the
possibility of weird behaviour if you set forget_packs incorrectly (which we
did).
Avery Pennarun [Sun, 14 Mar 2010 07:50:05 +0000 (03:50 -0400)]
bup.client: fix freeze when suggest-index after finishing a full pack.
It was just rare enough to be hard to find: if you write an entire pack full
of stuff (1GB or more) and *then* trigger a suggest-index, the client would
freeze because it would send a send-index command without actually
suspending the receive-pack first.
The whole Client/PackWriter separation is pretty gross, so it's not terribly
surprising this would happen.
Add a unit test to detect this case if it ever happens in the future, for
what it's worth.
Avery Pennarun [Sun, 14 Mar 2010 05:55:08 +0000 (00:55 -0500)]
main: even more fixes for signal handling.
If the child doesn't die after the first SIGINT and the user presses ctrl-c
one more time, the main bup process would die instead of forwarding it on to
the child. That's no good; we actually have to loop forwarding signals
until the child is really good and dead.
And if the child refuses to die, well, he's the one with the bug, not
main.py. So main.py should stay alive too in the name of not losing track
of things.
Avery Pennarun [Sat, 13 Mar 2010 01:40:46 +0000 (20:40 -0500)]
cmd/{index,save}: handle ctrl-c without printing a big exception trace.
It's not very exciting to look at a whole stack trace just because someone
hit ctrl-c, especially since that's designed to work fine. Trim it down in
that case.
Avery Pennarun [Fri, 12 Mar 2010 23:46:40 +0000 (18:46 -0500)]
git.PackWriter: avoid pack corruption if interrupted by a signal.
PackWriter tries to "finish" a half-written pack in its destructor if
interrupted. To do this, it flushes the stream, seeks back to the beginning
to update the sha1sum and object count, then runs git-index-pack on it to
create the .idx file.
However, sometimes if you were unlucky, you'd interrupt PackWriter partway
through writing an object to the pack. If only half an object exists at the
end, it would have the wrong header and thus come out as corrupt when
index-pack would run.
Since our objects are meant to be small anyway, just make sure we write
everything all in one file.write() operation. The files themselves are
buffered, so this wouldn't survive a surprise termination of the whole
unix process, but we wouldn't run index-pack in that case anyway, so it
doesn't matter.
Now when I press ctrl-c in 'bup save', it consistently writes the half-saved
objects as it should.
Avery Pennarun [Fri, 12 Mar 2010 23:05:54 +0000 (18:05 -0500)]
Correctly pass along SIGINT to child processes.
Ever since we introduced bup newliner, signal handling has been a little
screwy. The problem is that ctrl-c is passed to *all* processes in the
process group, not just the parent, so everybody would start terminating at
the same time, with very messy results.
Two results were particularly annoying: git.PackWriter()'s destructor
wouldn't always get called (so half-finished packs would be lost instead of
kept so we don't need to backup the same stuff next time) and bup-newliner
would exit, so the stdout/stderr of a process that *did* try to clean up
would be lost, usually resulting in EPIPE, which killed the proces while
attempting to clean up.
The fix is simple: when starting a long-running subprocess, give it its own
session by calling os.setsid(). That way ctrl-c is only sent to the
toplevel 'bup' process, who can forward it as it should.
Next, fix bup's signal forwarding to actually forward the same signal as it
received, instead of always using SIGTERM.
Avery Pennarun [Fri, 12 Mar 2010 22:18:20 +0000 (17:18 -0500)]
hashsplit: use posix_fadvise(DONTNEED) when available.
When reading through large disk images to back them up, we'll only end up
reading the data once, but it still takes up space in the kernel's disk
cache. If you're backing up a whole disk full of stuff, that's bad news for
anything else running on your system, which will rapidly have its stuff
dumped out of cache to store a bunch of stuff bup will never look at again.
The posix_fadvise() call actually lets us tell the kernel we won't be using
this data anymore, thus greatly reducing our hit on the disk cache.
Theoretically it improves things, anyway. I haven't been able to come up
with a really scientific way to test it, since of course *bup's* performance
is expected to be the same either way (we're only throwing away stuff we're
done using). It really does throw things out of cache, though, so the rest
follows logically at least.
projects / packages / b / bup.git / log