From cbdefb5ac43a1a34e71121a7a6a6434f0b8aa1cf Mon Sep 17 00:00:00 2001 From: Mark A Rada Date: Thu, 6 Aug 2009 10:28:25 -0400 Subject: [PATCH] gitweb: add support for XZ compressed snapshots The XZ compression format uses the LZMA2 compression algorithm, which often yields higher compression ratios than both GZip and BZip2 at the cost of using more CPU time and RAM. XZ is the slowest for compression, but still much faster than BZip2 for decompression, almost comparable to GZip (see benchmarks below). Some simple benchmarks show the pros and cons of using XZ compression; starting with an already tarball'd archive of the repos listed below. Memory usage seemed to be consistent for any given algorithm at their respective default compression levels. CPU: AMD Sempron 3400+ (1 core @ 1.8GHz with 256K L2 cache) Virtual Memory Usage GZip: 4152K BZip2: 13352K XZ: 102M Linux 2.6 series (f5886c7f96f2542382d3a983c5f13e03d7fc5259) 349M gzip 23.70s user 0.47s system 99% cpu 24.227 total 76M gunzip 3.74s user 0.74s system 94% cpu 4.741 total bzip2 130.96s user 0.53s system 99% cpu 2:11.97 total 59M bunzip2 31.05s user 1.02s system 99% cpu 32.355 total xz 448.78s user 0.91s system 99% cpu 7:31.28 total 51M unxz 7.67s user 0.80s system 98% cpu 8.607 total Git (0a53e9ddeaddad63ad106860237bbf53411d11a7) 11M gzip 0.77s user 0.03s system 99% cpu 0.792 total 2.5M gunzip 0.12s user 0.02s system 98% cpu 0.142 total bzip2 3.42s user 0.02s system 99% cpu 3.454 total 2.1M bunzip2 0.95s user 0.03s system 99% cpu 0.984 total xz 12.88s user 0.14s system 98% cpu 13.239 total 1.9M unxz 0.27s user 0.03s system 99% cpu 0.298 total XZ (669413bb2db954bbfde3c4542fddbbab53891eb4) 1.8M gzip 0.12s user 0.00s system 95% cpu 0.132 total 442K gunzip 0.02s user 0.00s system 97% cpu 0.027 total bzip2 1.28s user 0.01s system 99% cpu 1.298 total 363K bunzip2 0.15s user 0.01s system 100% cpu 0.157 total xz 1.62s user 0.03s system 99% cpu 1.652 total 347K unxz 0.05s user 0.00s system 99% cpu 0.058 total From a time and memory perspective, nothing compares to GZip, but if given an average upload speed of 20KB/s, it would take ~400 seconds longer to transfer the BZip2'd kernel snapshot than the XZ snapshot; the transfer time difference is even greater between GZip and XZ. The real time savings are relatively the same for all test cases, but less dramatic for smaller repositories. XZ decompresses ~1.8-2 times slower than GZip, and ~2.7-3.75 times faster than BZip2; XZ gets relatively faster as snapshots get larger. However, XZ takes relatively longer to compress as snapshots get larger. The downside for XZ'd snapshots is the large CPU and memory load put on the server to generate the compressed snapshot, though XZ will eventually have threading support, and the real clock time for making XZ'd snapshots would decrease if the server had a beefy multi-core CPU. XZ compression is disabled by default to allow upgrades to take place without any surprises, as the CPU and memory requirements will be an issue for high load or lightweight servers. Also, the XZ format is still new (format declared stable ~6 months ago), and there have been no "stable" releases of the utils yet. Signed-off-by: Mark Rada Signed-off-by: Junio C Hamano --- gitweb/gitweb.perl | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/gitweb/gitweb.perl b/gitweb/gitweb.perl index a0cdf31666..84659f5672 100755 --- a/gitweb/gitweb.perl +++ b/gitweb/gitweb.perl @@ -177,6 +177,14 @@ BEGIN 'format' => 'tar', 'compressor' => ['bzip2']}, + 'txz' => { + 'display' => 'tar.xz', + 'type' => 'application/x-xz', + 'suffix' => '.tar.xz', + 'format' => 'tar', + 'compressor' => ['xz'], + 'disabled' => 1}, + 'zip' => { 'display' => 'zip', 'type' => 'application/x-zip', @@ -189,6 +197,7 @@ BEGIN our %known_snapshot_format_aliases = ( 'gzip' => 'tgz', 'bzip2' => 'tbz2', + 'xz' => 'txz', # backward compatibility: legacy gitweb config support 'x-gzip' => undef, 'gz' => undef,