1
0
Fork 0
mirror of https://github.com/git/git.git synced 2024-10-31 14:27:54 +01:00
Commit graph

231 commits

Author SHA1 Message Date
Junio C Hamano
8c3c7b2adb pack-objects: remove redundant and wrong call to deflateEnd()
We somehow called deflateEnd() on a stream that we have called
deflateEnd() on already.

In fact, the second deflateEnd() has always been returning
Z_STREAM_ERROR.  We just never checked the error return from
that particular deflateEnd().

The first one returns 0 for success.  We might want to tighten
the check even more to check that.

Noticed by Marco.

Signed-off-by: Junio C Hamano <gitster@pobox.com>
2008-01-10 23:27:39 -08:00
Jim Meyering
872c930dcb Don't access line[-1] for a zero-length "line" from fgets.
A NUL byte at beginning of file, or just after a newline
would provoke an invalid buf[-1] access in a few places.

* builtin-grep.c (cmd_grep): Don't access buf[-1].
* builtin-pack-objects.c (get_object_list): Likewise.
* builtin-rev-list.c (read_revisions_from_stdin): Likewise.
* bundle.c (read_bundle_header): Likewise.
* server-info.c (read_pack_info_file): Likewise.
* transport.c (insert_packed_refs): Likewise.

Signed-off-by: Jim Meyering <meyering@redhat.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2008-01-04 12:28:58 -08:00
Johannes Sixt
68e6a4f80d Plug a resource leak in threaded pack-objects code.
A mutex and a condition variable is allocated for each thread and torn
down when the thread terminates. However, for certain workloads it can
happen that some threads are actually not started at all. In this case
we would leak the mutex and condition variable. Now we allocate them only
for those threads that are actually started.

Signed-off-by: Johannes Sixt <johannes.sixt@telecom.at>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2007-12-17 16:08:40 -08:00
Johannes Sixt
50f22ada52 threaded pack-objects: Use condition variables for thread communication.
In the threaded pack-objects code the main thread and the worker threads
must mutually signal that they have assigned a new pack of work or have
completed their work, respectively. Previously, the code used mutexes that
were locked in one thread and unlocked from a different thread, which is
bogus (and happens to work on Linux).

Here we rectify the implementation by using condition variables: There is
one condition variable on which the main thread waits until a thread
requests new work; and each worker thread has its own condition variable
on which it waits until it is assigned new work or signaled to terminate.

As a cleanup, the worker threads are spawned only after the initial work
packages have been assigned.

Signed-off-by: Johannes Sixt <johannes.sixt@telecom.at>
Acked-by: Nicolas Pitre <nico@cam.org>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2007-12-16 19:26:12 -08:00
Nicolas Pitre
eb9688ff65 pack-objects: more threaded load balancing fix with often changed paths
The code that splits the object list amongst work threads tries to do so
on "path" boundaries not to prevent good delta matches.  However, in
some cases, a few paths may largely dominate the hash distribution and
it is not possible to have good load balancing without ignoring those
boundaries.

Signed-off-by: Nicolas Pitre <nico@cam.org>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2007-12-10 17:10:16 -08:00
Nicolas Pitre
384b32c09b pack-objects: fix threaded load balancing
The current method consists of a master thread serving chunks of objects
to work threads when they're done with their previous chunk.  The issue
is to determine the best chunk size: making it too large creates poor
load balancing, while making it too small has a negative effect on pack
size because of the increased number of chunk boundaries and poor delta
window utilization.

This patch implements a completely different approach by initially
splitting the work in large chunks uniformly amongst all threads, and
whenever a thread is done then it steals half of the remaining work from
another thread with the largest amount of unprocessed objects.

This has the advantage of greatly reducing the number of chunk boundaries
with an almost perfect load balancing.

Signed-off-by: Nicolas Pitre <nico@cam.org>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2007-12-08 03:38:36 -08:00
Nicolas Pitre
b904166ccb pack-objects: reverse the delta search sort list
It is currently sorted and then walked backward.  Not only this doesn't
feel natural for my poor brain, but it would make the next patch less
obvious as well.

So reverse the sort order, and reverse the list walking direction,
which effectively produce the exact same end result as before.

Also bring the relevant comment nearer the actual code and adjust it
accordingly, with minor additional clarifications.

Signed-off-by: Nicolas Pitre <nico@cam.org>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2007-12-08 03:38:35 -08:00
Nicolas Pitre
b7a28f7827 pack-objects: fix delta cache size accounting
The wrong value was substracted from delta_cache_size when replacing
a cached delta, as trg_entry->delta_size was used after the old size
had been replaced by the new size.

Noticed by Linus.

Signed-off-by: Nicolas Pitre <nico@cam.org>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2007-12-08 03:38:35 -08:00
Linus Torvalds
4d1012c370 Fix rev-list when showing objects involving submodules
The function mark_tree_uninteresting() assumed that the tree entries
are blob when they are not trees.  This is not so.  Since we do
not traverse into submodules (yet), the gitlinks should be ignored.

In general, we should try to start moving away from using the
"S_ISLNK()" like things for internal git state. It was a mistake to
just assume the numbers all were same across all systems in the first
place.  This implementation converts to the "object_type", and then
uses a case statement.

Noticed by Ilari on IRC.
Test script taken from an earlier version by Dscho.

Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2007-11-14 03:44:22 -08:00
Junio C Hamano
e091653951 Merge branch 'np/pack'
* np/pack:
  pack-objects: get rid of an ugly cast
  make the pack index version configurable

Conflicts:

	builtin-pack-objects.c
2007-11-04 01:11:17 -07:00
Nicolas Pitre
79814f425c pack-objects: get rid of an ugly cast
... when calling write_idx_file().

Signed-off-by: Nicolas Pitre <nico@cam.org>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2007-11-02 01:41:04 -07:00
Nicolas Pitre
4d00bda2aa make the pack index version configurable
It is a good idea to use pack index version 2 all the time since it has
proper protection against propagation of certain pack corruptions when
repacking which is not possible with index version 1, as demonstrated
in test t5302.

Hence this config option.

The default is still pack index version 1.

Signed-off-by: Nicolas Pitre <nico@cam.org>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2007-11-02 01:32:02 -07:00
Nicolas Pitre
2a128d63dc add throughput display to git-push
This one triggers only when git-pack-objects is called with
--all-progress and --stdout which is the combination used by
git-push.

Signed-off-by: Nicolas Pitre <nico@cam.org>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2007-10-30 16:08:40 -07:00
Nicolas Pitre
4d4fcc5451 relax usage of the progress API
Since it is now OK to pass a null pointer to display_progress() and
stop_progress() resulting in a no-op, then we can simplify the code
and remove a bunch of lines by not making those calls conditional all
the time.

Signed-off-by: Nicolas Pitre <nico@cam.org>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2007-10-30 16:08:40 -07:00
Nicolas Pitre
dc6a0757c4 make struct progress an opaque type
This allows for better management of progress "object" existence,
as well as making the progress display implementation more independent
from its callers.

Signed-off-by: Nicolas Pitre <nico@cam.org>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2007-10-30 16:08:40 -07:00
Shawn O. Pearce
9c60a966f2 Change 'Deltifying objects' to 'Compressing objects'
Recently I was referred to the Grammar Police as the git-pack-objects
progress message 'Deltifying %u objects' is considered to be not
proper English to at least some small but vocal segment of the
English speaking population.  Techncially we are applying delta
compression to these objects at this stage, so the new term is
slightly more acceptable to the Grammar Police but is also just
as correct.

Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
2007-10-18 23:34:09 -04:00
Nicolas Pitre
4049b9cfc0 fix const issues with some functions
Two functions, namely write_idx_file() and open_pack_file(), currently
return a const pointer.  However that pointer is either a copy of the
first argument, or set to a malloc'd buffer when that first argument
is null.  In the later case it is wrong to qualify that pointer as const
since ownership of the buffer is transferred to the caller to dispose of,
and obviously the free() function is not meant to be passed const
pointers.

Making the return pointer not const causes a warning when the first
argument is returned since that argument is also marked const.

The correct thing to do is therefore to remove the const qualifiers,
avoiding the need for ugly casts only to silence some warnings.

Signed-off-by: Nicolas Pitre <nico@cam.org>
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
2007-10-17 02:54:57 -04:00
Nicolas Pitre
7ba502c47b pack-objects.c: fix some global variable abuse and memory leaks
To keep things well layered, sha1close() now returns the file descriptor
when it doesn't close the file.

An ugly cast was added to the return of write_idx_file() to avoid a
warning.  A proper fix will come separately.

Signed-off-by: Nicolas Pitre <nico@cam.org>
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
2007-10-17 02:54:56 -04:00
Nicolas Pitre
2f8b89472c pack-objects: no delta possible with only one object in the list
... so don't even try in that case, and save another useless line of
progress display.

Signed-off-by: Nicolas Pitre <nico@cam.org>
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
2007-10-17 02:54:56 -04:00
Nicolas Pitre
42e18fbf5f more compact progress display
Each progress can be on a single line instead of two.

[sp: Changed "Checking files out" to "Checking out files" at
     Johannes Sixt's suggestion as it better explains the
	 action that is taking place]

Signed-off-by: Nicolas Pitre <nico@cam.org>
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
2007-10-17 02:54:55 -04:00
Junio C Hamano
0341091a9e Merge branch 'jc/autogc'
* jc/autogc:
  git-gc --auto: run "repack -A -d -l" as necessary.
  git-gc --auto: restructure the way "repack" command line is built.
  git-gc --auto: protect ourselves from accumulated cruft
  git-gc --auto: add documentation.
  git-gc --auto: move threshold check to need_to_gc() function.
  repack -A -d: use --keep-unreachable when repacking
  pack-objects --keep-unreachable
  Export matches_pack_name() and fix its return value
  Invoke "git gc --auto" from commit, merge, am and rebase.
  Implement git gc --auto
2007-10-03 03:05:32 -07:00
Junio C Hamano
08cdfb1337 pack-objects --keep-unreachable
This new option is meant to be used in conjunction with the
options "git repack -a -d" usually invokes the underlying
pack-objects with.  When this option is given, objects unreachable
from the refs in packs named with --unpacked= option are added
to the resulting pack, in addition to the reachable objects that
are not in packs marked with *.keep files.

Signed-off-by: Junio C Hamano <gitster@pobox.com>
2007-09-17 12:25:26 -07:00
Junio C Hamano
e1ef867328 builtin-pack-objects.c: avoid bogus gcc warnings
These empty statement marcos can solicit bogus "statement with no effect"
warnings; squelch them.

Signed-off-by: Junio C Hamano <gitster@pobox.com>
2007-09-14 22:30:20 -07:00
Nicolas Pitre
3c70183918 threaded delta search: proper locking for cache accounting
Signed-off-by: Nicolas Pitre <nico@cam.org>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2007-09-12 12:56:09 -07:00
Nicolas Pitre
693b86fffb threaded delta search: add pack.threads config variable
Signed-off-by: Nicolas Pitre <nico@cam.org>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2007-09-10 10:50:21 -07:00
Nicolas Pitre
b81d9af71e fix threaded delta search locking
Found by Jeff King.

Signed-off-by: Nicolas Pitre <nico@cam.org>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2007-09-10 10:49:11 -07:00
Nicolas Pitre
367f4a4343 threaded delta search: specify number of threads at run time
This adds a --threads=<n> parameter to 'git pack-objects' with
documentation.

Signed-off-by: Nicolas Pitre <nico@cam.org>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2007-09-09 22:26:06 -07:00
Nicolas Pitre
59921b4b3f threaded delta search: better chunck split point
Try to keep object with the same name hash together.

Suggested by Martin Koegler.

Signed-off-by: Nicolas Pitre <nico@cam.org>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2007-09-09 22:25:43 -07:00
Nicolas Pitre
c2a33679a7 threaded delta search: refine work allocation
With this, each thread get repeatedly assigned the next available chunk of
objects to process until the whole list is done.  The idea is to have
reasonably small chunks so that all CPUs remain busy with a minimum
number of threads for as long as there is data to process.

Signed-off-by: Nicolas Pitre <nico@cam.org>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2007-09-09 22:25:33 -07:00
Nicolas Pitre
8ecce684a3 basic threaded delta search
this is still rough, hence it is disabled by default.  You need to compile
with "make THREADED_DELTA_SEARCH=1 ..." at the moment.

Threading is done on different portions of the object list to be
deltified. This is currently done by spliting the list into n parts and
then a thread is spawned for each of them.  A better method would consist
of spliting the list into more smaller parts and have the n threads
pick the next part available.

Signed-off-by: Nicolas Pitre <nico@cam.org>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2007-09-06 00:01:45 -07:00
Nicolas Pitre
e334977dfa rearrange delta search progress reporting
This is to help threadification of the delta search code, with a bonus
consistency check.

Signed-off-by: Nicolas Pitre <nico@cam.org>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2007-09-06 00:01:44 -07:00
Nicolas Pitre
ef0316fcd9 localize window memory usage accounting
This is to help threadification of delta searching.

Signed-off-by: Nicolas Pitre <nico@cam.org>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2007-09-05 23:49:28 -07:00
Nicolas Pitre
75d3985319 straighten the list of objects to deltify
Not all objects are subject to deltification, so avoid carrying those
along, and provide the real count to progress display.

Signed-off-by: Nicolas Pitre <nico@cam.org>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2007-09-05 23:49:10 -07:00
Junio C Hamano
776398709d Keep last used delta base in the delta window
This is based on Martin Koegler's idea to keep the object that
was successfully used as the base of the delta when it is about
to fall off the edge of the window.  Instead of doing so only
for the objects at the edge of the window, this makes the window
a lru eviction mechanism.  If an entry is used as a base, it is
moved to the last of the queue to be evicted.

This is a quick-and-dirty implementation, as it keeps the original
implementation of the data structure used for the window.  This
originally was done as an array, not as an array of pointers,
because it was meant to be used as a cyclic FIFO buffer and a
plain array avoids an extra pointer indirection, while its FIFOness
eant that we are not "moving" the entries like this patch does.

The runtime from three versions were comparable.  It seems to
make the resulting chain even shorter, which can only be good.

(stock "master") 15782196 bytes
chain length = 1: 2972 objects
chain length = 2: 2651 objects
chain length = 3: 2369 objects
chain length = 4: 2121 objects
chain length = 5: 1877 objects
...
chain length = 46: 490 objects
chain length = 47: 515 objects
chain length = 48: 527 objects
chain length = 49: 570 objects
chain length = 50: 408 objects

(with your patch) 15745736 bytes (0.23% smaller)
chain length = 1: 3137 objects
chain length = 2: 2688 objects
chain length = 3: 2322 objects
chain length = 4: 2146 objects
chain length = 5: 1824 objects
...
chain length = 46: 503 objects
chain length = 47: 509 objects
chain length = 48: 536 objects
chain length = 49: 588 objects
chain length = 50: 357 objects

(with this patch) 15612086 bytes (1.08% smaller)
chain length = 1: 4831 objects
chain length = 2: 3811 objects
chain length = 3: 2964 objects
chain length = 4: 2352 objects
chain length = 5: 1944 objects
...
chain length = 46: 327 objects
chain length = 47: 353 objects
chain length = 48: 304 objects
chain length = 49: 298 objects
chain length = 50: 135 objects

[jc: this is with code simplification follow-up from Nico]

Signed-off-by: Junio C Hamano <gitster@pobox.com>
2007-09-01 23:53:47 -07:00
Nicolas Pitre
9e2d57a04a fix same sized delta logic
The code favoring shallower deltas when size is equal was triggered
only when previous delta was also cached.  There should be no relation
between cached deltas and same sized deltas.

Signed-off-by: Nicolas Pitre <nico@cam.org>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2007-08-30 19:18:31 -07:00
Junio C Hamano
2e3404c324 pack-objects: check return value from read_sha1_file()
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2007-08-25 08:33:47 -07:00
Linus Torvalds
1211be6bed Make thin-pack generation subproject aware.
When a thin pack wants to send a tree object at "sub/dir", and
the commit that is common between the sender and the receiver
that is used as the base object has a subproject at that path,
we should not try to use the data at "sub/dir" of the base tree
as a tree object.  It is not a tree to begin with, and more
importantly, the commit object there does not have to even
exist.

Signed-off-by: Junio C Hamano <gitster@pobox.com>
2007-08-19 11:44:47 -07:00
Nicolas Pitre
d250626cbb pack-objects: remove bogus arguments to delta_cacheable()
Not only are they unused, but the order in the function declaration
and the actual usage don't match.

Signed-off-by: Nicolas Pitre <nico@cam.org>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2007-08-15 21:39:07 -07:00
Luiz Fernando N. Capitulino
7647b17f1d Use xmkstemp() instead of mkstemp()
xmkstemp() performs error checking and prints a standard error message when
an error occur.

Signed-off-by: Luiz Fernando N. Capitulino <lcapitulino@mandriva.com.br>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2007-08-14 22:20:26 -07:00
Nicolas Pitre
7d7baa5e15 Pack-objects: properly initialize the depth value
Commit 5a235b5e was missing this little detail.  Otherwise your pack
will explode.

Problem noted by Brian Downing.

Signed-off-by: Nicolas Pitre <nico@cam.org>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2007-07-12 20:11:41 -07:00
Nicolas Pitre
5a235b5ed0 reduce git-pack-objects memory usage a little more
The delta depth doesn't have to be stored in the global object array
structure since it is only used during the deltification pass.

Signed-off-by: Nicolas Pitre <nico@cam.org>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2007-07-12 14:36:09 -07:00
Brian Downing
a97773ce7e Add pack-objects window memory usage limit
This adds an option (--window-memory=N) and configuration variable
(pack.windowMemory = N) to limit the memory size of the pack-objects
delta search window.  This works by removing the oldest unpacked objects
whenever the total size goes above the limit.  It will always leave
at least one object, though, so as not to completely eliminate the
possibility of computing deltas.

This is an extra limit on top of the normal window size (--window=N);
the window will not dynamically grow above the fixed number of entries
specified to fill the memory limit.

With this, repacking a repository with a mix of large and small objects
is possible even with a very large window.

Cleaner and correct circular buffer handling courtesy of Nicolas Pitre.

Signed-off-by: Brian Downing <bdowning@lavos.net>
Acked-by: Nicolas Pitre <nico@cam.org>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2007-07-12 14:32:35 -07:00
Brian Downing
a1dab41af4 Don't try to delta if target is much smaller than source
Add a new try_delta heuristic.  Don't bother trying to make a delta if
the target object size is much smaller (currently 1/32) than the source,
as it's very likely not going to get a match.  Even if it does, you will
have to read at least 32x the size of the new file to reassemble it,
which isn't such a good deal.  This leads to a considerable performance
improvement when deltifying a mix of small and large files with a very
large window, because you don't have to wait for the large files to
percolate out of the window before things start going fast again.

Signed-off-by: Brian Downing <bdowning@lavos.net>
Acked-by: Nicolas Pitre <nico@cam.org>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2007-07-12 14:32:34 -07:00
Nicolas Pitre
c83f032e09 apply delta depth bias to already deltified objects
We already apply a bias on the initial delta attempt with max_size being
a function of the base object depth.  This has the effect of favoring
shallower deltas even if deeper deltas could be smaller, and therefore
creating a wider delta tree (see commits 4e8da195 and c3b06a69).

This principle should also be applied to all delta attempts for the same
object and not only the first attempt.  With this the criteria for the
best delta is not only its size but also its depth, so that a shallower
delta might be selected even if it is larger than a deeper one.  Even if
some deltas get larger, they allow for wider delta trees making the
depth limit less quickly reached and therefore better deltas can be
subsequently found, keeping the resulting pack size even smaller.
Runtime access to the pack should also benefit from shallower deltas.

Testing on different repositories showed slighter faster repacks,
smaller resulting packs, and a much nicer curve for delta depth
distribution with no more peak at the maximum depth level.
Improvements are even more significant with smaller depth limits.

Signed-off-by: Nicolas Pitre <nico@cam.org>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2007-07-12 14:18:14 -07:00
Brian Downing
848d732c10 pack-objects: Prefer shallower deltas if the size is equal
Change "try_delta" so that if it finds a delta that has the same size
but shallower depth than the existing delta, it will prefer the
shallower one.  This makes certain delta trees vastly less deep.

Signed-off-by: Brian Downing <bdowning@lavos.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2007-07-08 23:48:17 -07:00
Junio C Hamano
a6080a0a44 War on whitespace
This uses "git-apply --whitespace=strip" to fix whitespace errors that have
crept in to our source files over time.  There are a few files that need
to have trailing whitespaces (most notably, test vectors).  The results
still passes the test, and build result in Documentation/ area is unchanged.

Signed-off-by: Junio C Hamano <gitster@pobox.com>
2007-06-07 00:04:01 -07:00
Geert Bosch
aa7e44bf57 Unify write_index_file functions
This patch unifies the write_index_file functions in
builtin-pack-objects.c and index-pack.c.  As the name
"index" is overloaded in git, move in the direction of
using "idx" and "pack idx" when refering to the pack index.
There should be no change in functionality.

Signed-off-by: Geert Bosch <bosch@gnat.com>
Acked-by: Nicolas Pitre <nico@cam.org>
Signed-off-by: Junio C Hamano <junkio@cox.net>
2007-06-02 13:14:18 -07:00
Nicolas Pitre
5476a8adcc fix repack with --max-pack-size
Two issues here:

1) git-repack -a --max-pack-size=10 on the GIT repo dies pretty quick.
   There is a lot of confusion about deltas that were suposed to be
   reused from another pack but that get stored undeltified due to pack
   limit and object size doesn't match entry->size anymore.  This test
   is not really worth the complexity for determining when it is valid
   so get rid of it.

2) If pack limit is reached, the object buffer is freed, including when
   it comes from a cached delta data.  In practice the object will be
   stored in a subsequent pack undeltified, but let's make sure no
   pointer to freed data subsists by clearing entry->delta_data.

I also reorganized that code a bit to make it more readable.

Signed-off-by: Nicolas Pitre <nico@cam.org>
Signed-off-by: Junio C Hamano <junkio@cox.net>
2007-05-30 23:32:42 -07:00
Martin Koegler
e3dfddb377 builtin-pack-object: cache small deltas
Signed-off-by: Martin Koegler <mkoegler@auto.tuwien.ac.at>
Signed-off-by: Junio C Hamano <junkio@cox.net>
2007-05-29 01:24:42 -07:00
Martin Koegler
074b2eea29 git-pack-objects: cache small deltas between big objects
Creating deltas between big blobs is a CPU and memory intensive task.
In the writing phase, all (not reused) deltas are redone.

This patch adds support for caching deltas from the deltifing phase, so
that that the writing phase is faster.

The caching is limited to small deltas to avoid increasing memory usage very much.
The implemented limit is (memory needed to create the delta)/1024.

Signed-off-by: Martin Koegler <mkoegler@auto.tuwien.ac.at>
Signed-off-by: Junio C Hamano <junkio@cox.net>
2007-05-29 01:24:42 -07:00
Martin Koegler
a588d88aaf builtin-pack-objects: don't fail, if delta is not possible
If builtin-pack-objects runs out of memory while finding
the best deltas, it bails out with an error.

If the delta index creation fails (because there is not enough memory),
we can downgrade the error message to a warning and continue with the
next object.

Signed-off-by: Martin Koegler <mkoegler@auto.tuwien.ac.at>
Signed-off-by: Junio C Hamano <junkio@cox.net>
2007-05-29 01:24:42 -07:00
Junio C Hamano
a77a33a51d Merge branch 'dh/repack' (early part)
* 'dh/repack' (early part):
  Ensure git-repack -a -d --max-pack-size=N deletes correct packs
  pack-objects: clarification & option checks for --max-pack-size
  git-repack --max-pack-size: add option parsing to enable feature
  git-repack --max-pack-size: split packs as asked by write_{object,one}()
  git-repack --max-pack-size: write_{object,one}() respect pack limit
  git-repack --max-pack-size: new file statics and code restructuring
  Alter sha1close() 3rd argument to request flush only
2007-05-29 01:16:28 -07:00
Dana How
01c12a2312 pack-objects: clarification & option checks for --max-pack-size
Explain the special code for detecting a corner-case error,
and complain about --stdout & --max-pack-size being used together.

Signed-off-by: Dana L. How <danahow@gmail.com>
Signed-off-by: Junio C Hamano <junkio@cox.net>
2007-05-23 11:09:48 -07:00
Junio C Hamano
6315d52a84 builtin-pack-objects: remove unnecessary code for no-delta
As we do not consider objects marked as "no-delta" early, there
is no point to check if the other objects already in the delta
window are marked as such -- "no-delta" objects will not enter
the window to begin with.

Pointed out by Nico.

Signed-off-by: Junio C Hamano <junkio@cox.net>
2007-05-22 23:04:49 -07:00
Junio C Hamano
a74db82e15 Teach "delta" attribute to pack-objects.
This teaches pack-objects to use .gitattributes mechanism so
that the user can specify certain blobs are not worth spending
CPU cycles to attempt deltification.

The name of the attrbute is "delta", and when it is set to
false, like this:

	== .gitattributes ==
	*.jpg	-delta

they are always stored in the plain-compressed base object
representation.

Signed-off-by: Junio C Hamano <junkio@cox.net>
2007-05-21 23:58:20 -07:00
Junio C Hamano
bc32fed551 pack-objects: pass fullname down to add_object_entry()
Instead of giving a hash for grouping, pass fullname to add_object_entry().
I want to add "do not try deltifying this object" bit to object_entry based on
the settings in .gitattributes, and hashing the name down too early would
interfere with that plan.

Signed-off-by: Junio C Hamano <junkio@cox.net>
2007-05-21 23:58:20 -07:00
Dana L. How
6b94b1a09a git-repack --max-pack-size: add option parsing to enable feature
Add --max-pack-size parsing and usage messages.
Upgrade git-repack.sh to handle multiple packfile names,
and build packfiles in GIT_OBJECT_DIRECTORY not GIT_DIR.
Update documentation.

Signed-off-by: Dana L. How <danahow@gmail.com>
Signed-off-by: Junio C Hamano <junkio@cox.net>
2007-05-20 21:55:26 -07:00
Dana L. How
ebe27b137c git-repack --max-pack-size: split packs as asked by write_{object,one}()
Rewrite write_pack_file() to break to a new packfile
whenever write_object/write_one request it,  and
correct the header's object count in the previous packfile.
Change write_index_file() to write an index
for just the objects in the most recent packfile.

Signed-off-by: Dana L. How <danahow@gmail.com>
Signed-off-by: Junio C Hamano <junkio@cox.net>
2007-05-20 21:55:26 -07:00
Dana L. How
17b08f2cd0 git-repack --max-pack-size: write_{object,one}() respect pack limit
With --max-pack-size,  generate the appropriate write limit
for each object and check against it before each group of writes.
Update delta usability rules to handle base being in a previously-
written pack.  Inline sha1write_compress() so we know the
exact size of the written data when it needs to be compressed.
Detect and return write "failure".

Signed-off-by: Dana L. How <danahow@gmail.com>
Signed-off-by: Junio C Hamano <junkio@cox.net>
2007-05-20 21:55:26 -07:00
Dana L. How
d01fb92f8d git-repack --max-pack-size: new file statics and code restructuring
Add "pack_size_limit", the limit specified by --max-pack-size,
"written_list", the list of objects written to the current pack,
and "nr_written", the number of objects in written_list.
Put "base_name" at file scope again and add forward declarations.
Move write_index_file() call from cnd_pack_objects() to
write_pack_file() since only the latter will know how
many times to call write_index_file().

Signed-off-by: Dana L. How <danahow@gmail.com>
Signed-off-by: Junio C Hamano <junkio@cox.net>
2007-05-20 21:55:26 -07:00
Junio C Hamano
45bde46bfb Merge branch 'dh/pack'
* dh/pack:
  Custom compression levels for objects and packs
2007-05-20 02:19:19 -07:00
Dana How
960ccca680 Custom compression levels for objects and packs
Add config variables pack.compression and core.loosecompression ,
and switch --compression=level to pack-objects.

Loose objects will be compressed using core.loosecompression if set,
else core.compression if set, else Z_BEST_SPEED.
Packed objects will be compressed using --compression=level if seen,
else pack.compression if set, else core.compression if set,
else Z_DEFAULT_COMPRESSION.  This is the "pack compression level".

Loose objects added to a pack undeltified will be recompressed
to the pack compression level if it is unequal to the current
loose compression level by the preceding rules,  or if the loose
object was written while core.legacyheaders = true.  Newly
deltified loose objects are always compressed to the current
pack compression level.

Previously packed objects added to a pack are recompressed
to the current pack compression level exactly when their
deltification status changes,  since the previous pack data
cannot be reused.

In either case,  the --no-reuse-object switch from the first
patch below will always force recompression to the current pack
compression level,  instead of assuming the pack compression level
hasn't changed and pack data can be reused when possible.

This applies on top of the following patches from Nicolas Pitre:
[PATCH] allow for undeltified objects not to be reused
[PATCH] make "repack -f" imply "pack-objects --no-reuse-object"

Signed-off-by: Dana L. How <danahow@gmail.com>
Signed-off-by: Junio C Hamano <junkio@cox.net>
2007-05-10 15:23:09 -07:00
Nicolas Pitre
726f852b0e deprecate the new loose object header format
Now that we encourage and actively preserve objects in a packed form
more agressively than we did at the time the new loose object format and
core.legacyheaders were introduced, that extra loose object format
doesn't appear to be worth it anymore.

Because the packing of loose objects has to go through the delta match
loop anyway, and since most of them should end up being deltified in
most cases, there is really little advantage to have this parallel loose
object format as the CPU savings it might provide is rather lost in the
noise in the end.

This patch gets rid of core.legacyheaders, preserve the legacy format as
the only writable loose object format and deprecate the other one to
keep things simpler.

Signed-off-by: Nicolas Pitre <nico@cam.org>
Signed-off-by: Junio C Hamano <junkio@cox.net>
2007-05-10 15:22:33 -07:00
Nicolas Pitre
fa736f72b0 allow for undeltified objects not to be reused
Currently non deltified object data is always reused when possible.
This means that any change to core.compression has no effect on those
objects as they don't get recompressed when repacking them.

Let's add a --no-reuse-object flag to git-repack in order to force
recompression of all objects when desired.

Signed-off-by: Nicolas Pitre <nico@cam.org>
Signed-off-by: Junio C Hamano <junkio@cox.net>
2007-05-10 15:22:33 -07:00
Theodore Ts'o
618e613a70 Increase pack.depth default to 50
Signed-off-by: "Theodore Ts'o" <tytso@mit.edu>
Signed-off-by: Junio C Hamano <junkio@cox.net>
2007-05-08 22:47:17 -07:00
Theodore Ts'o
842aaf9323 Add pack.depth option to git-pack-objects.
Signed-off-by: "Theodore Ts'o" <tytso@mit.edu>
Signed-off-by: Junio C Hamano <junkio@cox.net>
2007-05-08 22:47:17 -07:00
Alex Riesen
3082acfa7c Use GIT_OBJECT_DIR for temporary files of pack-objects
Signed-off-by: Alex Riesen <raa.lkml@gmail.com>
Signed-off-by: Junio C Hamano <junkio@cox.net>
2007-05-07 15:45:24 -07:00
Nicolas Pitre
13aaf14825 make progress "title" part of the common progress interface
If the progress bar ends up in a box, better provide a title for it too.

Signed-off-by: Nicolas Pitre <nico@cam.org>
Signed-off-by: Junio C Hamano <junkio@cox.net>
2007-04-22 22:18:05 -07:00
Nicolas Pitre
96a02f8f6d common progress display support
Instead of having this code duplicated in multiple places, let's have
a common interface for progress display.  If someday someone wishes to
display a cheezy progress bar instead then only one file will have to
be changed.

Note: I left merge-recursive.c out since it has a strange notion of
progress as it apparently increase the expected total number as it goes.
Someone with more intimate knowledge of what that is supposed to mean
might look at converting it to the common progress interface.

Signed-off-by: Nicolas Pitre <nico@cam.org>
Signed-off-by: Junio C Hamano <junkio@cox.net>
2007-04-22 22:18:05 -07:00
Junio C Hamano
d83c9af5c6 pack-objects: make generated packfile read-only
Signed-off-by: Junio C Hamano <junkio@cox.net>
2007-04-22 19:00:16 -07:00
Junio C Hamano
b6b32ccb92 Fix 'quickfix' on pack-objects.
The earlier quickfix forced world-readable permission bits.  This
updates it to honor umask and core.sharedrepository settings.

Signed-off-by: Junio C Hamano <junkio@cox.net>
2007-04-22 12:28:34 -07:00
Junio C Hamano
aef5aedd85 pack-objects: quickfix for permission modes.
mkstemp() often creates the file in 0600 which means the
resulting packfile is not readable by anybody other than the
repository owner.  Force 0644 for now, even though this is not
strictly correct.

Signed-off-by: Junio C Hamano <junkio@cox.net>
2007-04-22 11:49:35 -07:00
Nicolas Pitre
e4d58311ba pack-objects: remove obsolete comments
The sorted-by-sha ans sorted-by-type arrays are no more.

Signed-off-by: Nicolas Pitre <nico@cam.org>
Signed-off-by: Junio C Hamano <junkio@cox.net>
2007-04-19 20:37:41 -07:00
Nicolas Pitre
5c49c11686 pack-objects: better check_object() performances
With large amount of objects, check_object() is really trashing the pack
sliding map and the filesystem cache.  It has a completely random access
pattern especially with old objects where delta replay jumps back and
forth all over the pack.

This patch improves things by:

 1) sorting objects by their offset in pack before calling check_object()
    so the pack access pattern is linear;

 2) recording the object type at add_object_entry() time since it is
    already known in most cases;

 3) recording the pack offset even for preferred_base objects;

 4) avoid calling sha1_object_info() if all possible.

This limits pack accesses to the bare minimum and makes them perfectly
linear.

In the process check_object() was made more clear (to me at least).

Note: I thought about walking the sorted_by_offset list backward in
get_object_details() so if a pack happens to be larger than the available
file cache, then the cache would have been populated with useful data from
the beginning of the pack already when find_deltas() is called.  Strangely,
testing (on Linux) showed absolutely no performance difference.

Signed-off-by: Nicolas Pitre <nico@cam.org>
Signed-off-by: Junio C Hamano <junkio@cox.net>
2007-04-16 17:43:31 -07:00
Nicolas Pitre
a3fbf4dfe1 pack-objects: make in_pack_header_size a variable of its own
It currently aliases delta_size on the principle that reused deltas won't
go through the whole delta matching loop hence delta_size was unused.
This is not true if given delta doesn't find its base in the pack though.
But we need that information even for whole object data reuse.

Well in short the current state looks awful and is prone to bugs.  It just
works fine now because try_delta() tests trg_entry->delta before using
trg_entry->delta_size, but that is a bit subtle and I was wondering for a
while why things just worked fine... even if I'm guilty of having
introduced this abomination myself in the first place.

Let's do the sensible thing instead with no ambiguity, which is to have
a separate variable for in_pack_header_size.  This might even help future
optimizations.

While at it, let's reorder some struct object_entry members so they all
align well with their own width, regardless of the architecture or the
size of off_t.  Some memory saving is to be expected with this alone.

Signed-off-by: Nicolas Pitre <nico@cam.org>
Signed-off-by: Junio C Hamano <junkio@cox.net>
2007-04-16 17:43:31 -07:00
Nicolas Pitre
81a216a5d6 pack-objects: get rid of create_final_object_list()
Because we don't have to know the SHA1 h(hence the name) of the pack
up front anymore, let's get rid of yet another global sorted object list
and sort them only in write_index_file(), then compute the object list
SHA1 on the fly.

This has the advantage of saving another chunk of memory, and the sorted
list SHA1 won't be computed needlessly on servers during a fetch.

Of course the cunning plan is also to make write_index_file() much like
the function with the same name in index-pack.c for an eventual easy
sharing.

Signed-off-by: Nicolas Pitre <nico@cam.org>
Signed-off-by: Junio C Hamano <junkio@cox.net>
2007-04-16 17:43:31 -07:00
Nicolas Pitre
f7ae6a930a pack-objects: get rid of reuse_cached_pack
This capability is practically never useful, and therefore never tested,
because it is fairly unlikely that the requested pack will be already
available.  Furthermore it is of little gain over the ability to reuse
existing pack data.

In fact the ability to change delta type on the fly when reusing delta
data is a nice thing that has almost no cost and allows greater backward
compatibility with a client's capabilities than if the client is blindly
sent a whole pack without any discrimination.

And this "feature" is simply in the way of other cleanups.
Let's get rid of it.

Signed-off-by: Nicolas Pitre <nico@cam.org>
Signed-off-by: Junio C Hamano <junkio@cox.net>
2007-04-16 17:43:31 -07:00
Nicolas Pitre
9668cf59a8 pack-objects: clean up list sorting
Get rid of sort_comparator() as it impose a run time double indirect
function call for little compile time type checking gain.

Also get rid of create_sorted_list() as it only has one user which would
as well be just fine doing its sorting locally.  Eventually the list of
deltifiable objects might be shorter than the whole object list.

Signed-off-by: Nicolas Pitre <nico@cam.org>
Signed-off-by: Junio C Hamano <junkio@cox.net>
2007-04-16 17:43:31 -07:00
Nicolas Pitre
898b14cedc pack-objects: rework check_delta_limit usage
Objects that have delta "children" from pack data reuse must consider the
depth of their deepest child when they try to deltify themselves for those
children not to become too deep.

However, in the context of a "thin" pack, the delta children depth was
skipped entirely on the presumption that the pack was always going to be
exploded on the receiving end, hence the delta length wasn't an issue.

Now that we keep received packs as is and reuse pack data when repacking,
those packs do contain delta chains that are longer than expected. Worse,
those delta chain may even grow longer when the pack is further repacked
into another thin pack for a subsequent transmission.

So this patch restores strict delta length even for thin packs, and it
moves check_delta_limit() usage directly in the delta loop where it is
needed.  This way the delta_limit can be removed from struct object_entry
as well.  Oh and the initial value was wrong too.

The  progress_interval() function was moved to a more logical location in
the process.

Signed-off-by: Nicolas Pitre <nico@cam.org>
Signed-off-by: Junio C Hamano <junkio@cox.net>
2007-04-16 17:43:31 -07:00
Nicolas Pitre
adcc70950e pack-objects: equal objects in size should delta against newer objects
Before finding best delta combinations, we sort objects by name hash,
then by size, then by their position in memory.  Then we walk the list
backwards to test delta candidates.

We hope that a bigger size usually means a newer objects.  But a bigger
address in memory does not mean a newer object.  So the last comparison
must be reversed.

Signed-off-by: Nicolas Pitre <nico@cam.org>
Signed-off-by: Junio C Hamano <junkio@cox.net>
2007-04-16 17:43:30 -07:00
Nicolas Pitre
8a5a8d6c97 pack-objects: optimize preferred base handling a bit
Let's avoid some cycles when there is no base to test against, and avoid
unnecessary object lookups.

Signed-off-by: Nicolas Pitre <nico@cam.org>
Signed-off-by: Junio C Hamano <junkio@cox.net>
2007-04-16 17:43:30 -07:00
Nicolas Pitre
29b734e478 clean up add_object_entry()
This function used to call locate_object_entry_hash() _twice_ per added
object while only once should suffice. Let's reorganize that code a bit.

Signed-off-by: Nicolas Pitre <nico@cam.org>
Signed-off-by: Junio C Hamano <junkio@cox.net>
2007-04-11 19:35:25 -07:00
Nicolas Pitre
91ecbeca48 validate reused pack data with CRC when possible
This replaces the inflate validation with a CRC validation when reusing
data from a pack which uses index version 2.  That makes repacking much
safer against corruptions, and it should be a bit faster too.

Signed-off-by: Nicolas Pitre <nico@cam.org>
Signed-off-by: Junio C Hamano <junkio@cox.net>
2007-04-10 12:48:14 -07:00
Nicolas Pitre
4ba7d71153 allow forcing index v2 and 64-bit offset treshold
This is necessary for testing the new capabilities in some automated
way without having an actual 4GB+ pack.

Signed-off-by: Nicolas Pitre <nico@cam.org>
Signed-off-by: Junio C Hamano <junkio@cox.net>
2007-04-10 12:48:14 -07:00
Nicolas Pitre
c553ca25bd pack-objects: learn about pack index version 2
Pack index version 2 goes as follows:

 - 8 bytes of header with signature and version.

 - 256 entries of 4-byte first-level fan-out table.

 - Table of sorted 20-byte SHA1 records for each object in pack.

 - Table of 4-byte CRC32 entries for raw pack object data.

 - Table of 4-byte offset entries for objects in the pack if offset is
   representable with 31 bits or less, otherwise it is an index in the next
   table with top bit set.

 - Table of 8-byte offset entries indexed from previous table for offsets
   which are 32 bits or more (optional).

 - 20-byte SHA1 checksum of sorted object names.

 - 20-byte SHA1 checksum of the above.

The object SHA1 table is all contiguous so future pack format that would
contain this table directly won't require big changes to the code. It is
also tighter for slightly better cache locality when looking up entries.

Support for large packs exceeding 31 bits in size won't impose an index
size bloat for packs within that range that don't need a 64-bit offset.
And because newer objects which are likely to be the most frequently used
are located at the beginning of the pack, they won't pay the 64-bit offset
lookup at run time either even if the pack is large.

Right now an index version 2 is created only when the biggest offset in a
pack reaches 31 bits.  It might be a good idea to always use index version
2 eventually to benefit from the CRC32 it contains when reusing pack data
while repacking.

[jc: with the "oops" fix to keep track of the last offset correctly]

Signed-off-by: Nicolas Pitre <nico@cam.org>
Signed-off-by: Junio C Hamano <junkio@cox.net>
2007-04-10 12:48:14 -07:00
Nicolas Pitre
78d1e84fe5 compute a CRC32 for each object as stored in a pack
The most important optimization for performance when repacking is the
ability to reuse data from a previous pack as is and bypass any delta
or even SHA1 computation by simply copying the raw data from one pack
to another directly.

The problem with  this is that any data corruption within a copied object
would go unnoticed and the new (repacked) pack would be self-consistent
with its own checksum despite containing a corrupted object.  This is a
real issue that already happened at least once in the past.

In some attempt to prevent this, we validate the copied data by inflating
it and making sure no error is signaled by zlib.  But this is still not
perfect as a significant portion of a pack content is made of object
headers and references to delta base objects which are not deflated and
therefore not validated when repacking actually making the pack data reuse
still not as safe as it could be.

Of course a full SHA1 validation could be performed, but that implies
full data inflating and delta replaying which is extremely costly, which
cost the data reuse optimization was designed to avoid in the first place.

So the best solution to this is simply to store a CRC32 of the raw pack
data for each object in the pack index.  This way any object in a pack can
be validated before being copied as is in another pack, including header
and any other non deflated data.

Why CRC32 instead of a faster checksum like Adler32?  Quoting Wikipedia:

   Jonathan Stone discovered in 2001 that Adler-32 has a weakness for very
   short messages. He wrote "Briefly, the problem is that, for very short
   packets, Adler32 is guaranteed to give poor coverage of the available
   bits. Don't take my word for it, ask Mark Adler. :-)" The problem is
   that sum A does not wrap for short messages. The maximum value of A for
   a 128-byte message is 32640, which is below the value 65521 used by the
   modulo operation. An extended explanation can be found in RFC 3309,
   which mandates the use of CRC32 instead of Adler-32 for SCTP, the
   Stream Control Transmission Protocol.

In the context of a GIT pack, we have lots of small objects, especially
deltas, which are likely to be quite small and in a size range for which
Adler32 is dimed not to be sufficient.  Another advantage of CRC32 is the
possibility for recovery from certain types of small corruptions like
single bit errors which are the most probable type of corruptions.

OK what this patch does is to compute the CRC32 of each object written to
a pack within pack-objects.  It is not written to the index yet and it is
obviously not validated when reusing pack data yet either.

Signed-off-by: Nicolas Pitre <nico@cam.org>
Signed-off-by: Junio C Hamano <junkio@cox.net>
2007-04-10 12:48:14 -07:00
Nicolas Pitre
d7dd02231f add overflow tests on pack offset variables
Change a few size and offset variables to more appropriate type, then
add overflow tests on those offsets.  This prevents any bad data to be
generated/processed if off_t happens to not be large enough to handle
some big packs.

Better be safe than sorry.

Signed-off-by: Nicolas Pitre <nico@cam.org>
Signed-off-by: Junio C Hamano <junkio@cox.net>
2007-04-10 12:48:14 -07:00
Nicolas Pitre
8723f21626 make overflow test on delta base offset work regardless of variable size
This patch introduces the MSB() macro to obtain the desired number of
most significant bits from a given variable independently of the variable
type.

It is then used to better implement the overflow test on the OBJ_OFS_DELTA
base offset variable with the property of always working correctly
regardless of the type/size of that variable.

Signed-off-by: Nicolas Pitre <nico@cam.org>
Signed-off-by: Junio C Hamano <junkio@cox.net>
2007-04-10 12:48:14 -07:00
Nicolas Pitre
57059091fa get rid of num_packed_objects()
The coming index format change doesn't allow for the number of objects
to be determined from the size of the index file directly.  Instead, Let's
initialize a field in the packed_git structure with the object count when
the index is validated since the count is always known at that point.

While at it let's reorder some struct packed_git fields to avoid padding
due to needed 64-bit alignment for some of them.

Signed-off-by: Nicolas Pitre <nico@cam.org>
Signed-off-by: Junio C Hamano <junkio@cox.net>
2007-04-10 12:48:14 -07:00
Nicolas Pitre
d72308e01c clean up and optimize nth_packed_object_sha1() usage
Let's avoid the open coded pack index reference in pack-object and use
nth_packed_object_sha1() instead.  This will help encapsulating index
format differences in one place.

And while at it there is no reason to copy SHA1's over and over while a
direct pointer to it in the index will do just fine.

Signed-off-by: Nicolas Pitre <nico@cam.org>
Acked-by: Shawn O. Pearce <spearce@spearce.org>
Signed-off-by: Junio C Hamano <junkio@cox.net>
2007-04-05 14:59:47 -07:00
Linus Torvalds
6fda5e5180 Initialize tree descriptors with a helper function rather than by hand.
This removes slightly more lines than it adds, but the real reason for
doing this is that future optimizations will require more setup of the
tree descriptor, and so we want to do it in one place.

Also renamed the "desc.buf" field to "desc.buffer" just to trigger
compiler errors for old-style manual initializations, making sure I
didn't miss anything.

Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
Signed-off-by: Junio C Hamano <junkio@cox.net>
2007-03-21 10:21:57 -07:00
Linus Torvalds
a8c40471ab Remove "pathlen" from "struct name_entry"
Since we have the "tree_entry_len()" helper function these days, and
don't need to do a full strlen(), there's no point in saving the path
length - it's just redundant information.

Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
Signed-off-by: Junio C Hamano <junkio@cox.net>
2007-03-21 10:21:56 -07:00
Nicolas Pitre
4287307833 [PATCH] clean up pack index handling a bit
Especially with the new index format to come, it is more appropriate
to encapsulate more into check_packed_git_idx() and assume less of the
index format in struct packed_git.

To that effect, the index_base is renamed to index_data with void * type
so it is not used directly but other pointers initialized with it. This
allows for a couple pointer cast removal, as well as providing a better
generic name to grep for when adding support for new index versions or
formats.

And index_data is declared const too while at it.

Signed-off-by: Nicolas Pitre <nico@cam.org>
Signed-off-by: Junio C Hamano <junkio@cox.net>
2007-03-16 21:27:36 -07:00
Shawn O. Pearce
6777a59fcd Use off_t in pack-objects/fast-import when we mean an offset
Always use an off_t value in pack-objects anytime we are dealing
with an offset to some data within a packfile.

Also fixed a minor uintmax_t that was incorrectly defined before.

Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Signed-off-by: Junio C Hamano <junkio@cox.net>
2007-03-07 11:06:33 -08:00
Shawn O. Pearce
7cadf491c6 Use uint32_t for pack-objects counters.
As we technically try to support up to a maximum of 2**32-1 objects
in a single packfile we should act like it and use unsigned 32 bit
integers for all of our object counts and progress output.

This change does not modify everything in pack-objects that probably
needs to change to fully support the maximum of 2**32-1 objects.
I'm intentionally breaking the improvements into slightly smaller
commits to make them easier to follow.

No logic change should be occuring here, with the exception that
some comparsions will now work properly when the number of objects
exceeds 2**31-1.

Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Signed-off-by: Junio C Hamano <junkio@cox.net>
2007-03-07 11:02:38 -08:00
Nicolas Pitre
21666f1aae convert object type handling from a string to a number
We currently have two parallel notation for dealing with object types
in the code: a string and a numerical value.  One of them is obviously
redundent, and the most used one requires more stack space and a bunch
of strcmp() all over the place.

This is an initial step for the removal of the version using a char array
found in object reading code paths.  The patch is unfortunately large but
there is no sane way to split it in smaller parts without breaking the
system.

Signed-off-by: Nicolas Pitre <nico@cam.org>
Signed-off-by: Junio C Hamano <junkio@cox.net>
2007-02-27 01:34:21 -08:00
Nicolas Pitre
df8436622f formalize typename(), and add its reverse type_from_string()
Sometime typename() is used, sometimes type_names[] is accessed directly.
Let's enforce typename() all the time which allows for validating the
type.

Also let's add a function to go from a name to a type and use it instead
of manual memcpy() when appropriate.

Signed-off-by: Nicolas Pitre <nico@cam.org>
Signed-off-by: Junio C Hamano <junkio@cox.net>
2007-02-27 01:34:21 -08:00
Junio C Hamano
2c7ca1fcf1 Merge branch 'maint'
* maint:
  Add Release Notes to prepare for 1.5.0.2
  Allow arbitrary number of arguments to git-pack-objects
  rerere: do not deal with symlinks.
  rerere: do not skip two conflicted paths next to each other.
  Don't modify CREDITS-FILE if it hasn't changed.
2007-02-25 11:08:47 -08:00
Roland Dreier
ffa84ffb77 Allow arbitrary number of arguments to git-pack-objects
If a repository ever gets in a situation where there are too many
packs (more than 60 or so), perhaps because of frequent use of
git-fetch -k or incremental git-repack, then it becomes impossible to
fully repack the repository with git-repack -a.  That command just
dies with the cryptic message

    fatal: too many internal rev-list options

This message comes from git-pack-objects, which is passed one command
line option like --unpacked=pack-<SHA1>.pack for each pack file to be
repacked.  However, the current code has a static limit of 64 command
line arguments and just aborts if more arguments are passed to it.

Fix this by dynamically allocating the array of command line
arguments, and doubling the size each time it overflows.

Signed-off-by: Roland Dreier <roland@digitalvampire.org>
Signed-off-by: Junio C Hamano <junkio@cox.net>
2007-02-25 10:50:12 -08:00
Junio C Hamano
599065a3bb prefixcmp(): fix-up mechanical conversion.
Previous step converted use of strncmp() with literal string
mechanically even when the result is only used as a boolean:

    if (!strncmp("foo", arg, 3)) ==> if (!(-prefixcmp(arg, "foo")))

This step manually cleans them up to read:

    if (!prefixcmp(arg, "foo"))

Signed-off-by: Junio C Hamano <junkio@cox.net>
2007-02-20 22:03:15 -08:00
Junio C Hamano
cc44c7655f Mechanical conversion to use prefixcmp()
This mechanically converts strncmp() to use prefixcmp(), but only when
the parameters match specific patterns, so that they can be verified
easily.  Leftover from this will be fixed in a separate step, including
idiotic conversions like

    if (!strncmp("foo", arg, 3))

  =>

    if (!(-prefixcmp(arg, "foo")))

This was done by using this script in px.perl

   #!/usr/bin/perl -i.bak -p
   if (/strncmp\(([^,]+), "([^\\"]*)", (\d+)\)/ && (length($2) == $3)) {
           s|strncmp\(([^,]+), "([^\\"]*)", (\d+)\)|prefixcmp($1, "$2")|;
   }
   if (/strncmp\("([^\\"]*)", ([^,]+), (\d+)\)/ && (length($1) == $3)) {
           s|strncmp\("([^\\"]*)", ([^,]+), (\d+)\)|(-prefixcmp($2, "$1"))|;
   }

and running:

   $ git grep -l strncmp -- '*.c' | xargs perl px.perl

Signed-off-by: Junio C Hamano <junkio@cox.net>
2007-02-20 22:03:15 -08:00
Junio C Hamano
b18b00a661 Use fixed-size integers for .idx file I/O
This attempts to finish what Simon started in the previous commit.

Signed-off-by: Junio C Hamano <junkio@cox.net>
2007-01-18 14:11:50 -08:00
Junio C Hamano
9c18df1907 pack-objects: fix use of use_pack().
The code calls use_pack() to make that the variably encoded
offset fits in the mmap'ed window, but it forgot that the
operation gives the pointer to the beginning of the asked
region.

Signed-off-by: Junio C Hamano <junkio@cox.net>
2006-12-29 11:36:45 -08:00
Shawn O. Pearce
f5b1b5a07e Fix random segfaults in pack-objects.
Junio noticed that 'non-trivial' pushes were failing if executed
using the sliding window mmap changes.  This was somewhat difficult
to track down as the failure was appearing randomly.

It turns out this was a failure caused by the delta base reference
(either ref or offset format) spanning over the end of a mmap window.

The error in pack-objects was we were not recalling use_pack
after the object header was unpacked, and therefore we did not
get the promise of at least 20 bytes in the buffer for the delta
base parsing.  This would case later memcmp() calls to walk into
unassigned address space at the end of the window.

The reason Junio and I had hard time tracking this down in current
Git repositories is we were both probably packing with offset deltas,
which minimized the odds of the delta base reference spanning over
the end of the mmap window.  Stepping back and repacking with
version 1.3.3 (which only supported reference deltas) increased
the likelyhood of seeing the bug.

The correct technique (as used in sha1_file.c) is to invoke
use_pack() after unpack_object_header_gently to ensure we have
enough data available for the delta base decoding.

Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Signed-off-by: Junio C Hamano <junkio@cox.net>
2006-12-29 11:36:45 -08:00
Shawn O. Pearce
079afb18fe Loop over pack_windows when inflating/accessing data.
When multiple mmaps start getting used for all pack file access it
is not possible to get all data associated with a specific object
in one contiguous memory region.  This limitation prevents simply
passing a single address and length to SHA1_Update or to inflate.

Instead we need to loop until we have processed all data of interest.

As we loop over the data we are always interested in reusing the same
window 'cursor', as the prior window will no longer be of any use
to us.  This allows the use_pack() call to automatically decrement
the use count of the prior window before setting up access for us
to the next window.

Within each loop we need to make use of the available length output
parameter of use_pack() to tell us how many bytes are available in
the current memory region, as we cannot tell otherwise.

Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Signed-off-by: Junio C Hamano <junkio@cox.net>
2006-12-29 11:36:44 -08:00
Shawn O. Pearce
03e79c88aa Replace use_packed_git with window cursors.
Part of the implementation concept of the sliding mmap window for
pack access is to permit multiple windows per pack to be mapped
independently.  Since the inuse_cnt is associated with the mmap and
not with the file, this value is in struct pack_window and needs to
be incremented/decremented for each pack_window accessed by any code.

To faciliate that implementation we need to replace all uses of
use_packed_git() and unuse_packed_git() with a different API that
follows struct pack_window objects rather than struct packed_git.

The way this works is when we need to start accessing a pack for
the first time we should setup a new window 'cursor' by declaring
a local and setting it to NULL:

  struct pack_windows *w_curs = NULL;

To obtain the memory region which contains a specific section of
the pack file we invoke use_pack(), supplying the address of our
current window cursor:

  unsigned int len;
  unsigned char *addr = use_pack(p, &w_curs, offset, &len);

the returned address `addr` will be the first byte at `offset`
within the pack file.  The optional variable len will also be
updated with the number of bytes remaining following the address.

Multiple calls to use_pack() with the same window cursor will
update the window cursor, moving it from one window to another
when necessary.  In this way each window cursor variable maintains
only one struct pack_window inuse at a time.

Finally before exiting the scope which originally declared the window
cursor we must invoke unuse_pack() to unuse the current window (which
may be different from the one that was first obtained from use_pack):

  unuse_pack(&w_curs);

This implementation is still not complete with regards to multiple
windows, as only one window per pack file is supported right now.

Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Signed-off-by: Junio C Hamano <junkio@cox.net>
2006-12-29 11:36:44 -08:00
Shawn O. Pearce
c41ee586dc Refactor packed_git to prepare for sliding mmap windows.
The idea behind the sliding mmap window pack reader implementation
is to have multiple mmap regions active against the same pack file,
thereby allowing the process to mmap in only the active/hot sections
of the pack and reduce overall virtual address space usage.

To implement this we need to refactor the mmap related data
(pack_base, pack_use_cnt) out of struct packed_git and move them
into a new struct pack_window.

We are refactoring the code to support a single struct pack_window
per packfile, thereby emulating the prior behavior of mmap'ing the
entire pack file.

Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Signed-off-by: Junio C Hamano <junkio@cox.net>
2006-12-29 11:36:44 -08:00
Junio C Hamano
63049292e0 Teach git-repack to preserve objects referred to by reflog entries.
This adds a new option --reflog to pack-objects and revision
machinery; do not bother documenting it for now, since this is
only useful for local repacking.

When the option is passed, objects reachable from reflog entries
are marked as interesting while computing the set of objects to
pack.

Signed-off-by: Junio C Hamano <junkio@cox.net>
2006-12-20 17:22:10 -08:00
Junio C Hamano
85023577a8 simplify inclusion of system header files.
This is a mechanical clean-up of the way *.c files include
system header files.

 (1) sources under compat/, platform sha-1 implementations, and
     xdelta code are exempt from the following rules;

 (2) the first #include must be "git-compat-util.h" or one of
     our own header file that includes it first (e.g. config.h,
     builtin.h, pkt-line.h);

 (3) system headers that are included in "git-compat-util.h"
     need not be included in individual C source files.

 (4) "git-compat-util.h" does not have to include subsystem
     specific header files (e.g. expat.h).

Signed-off-by: Junio C Hamano <junkio@cox.net>
2006-12-20 09:51:35 -08:00
Nicolas Pitre
67c08ce14f pack-objects: remove redundent status information
The final 'nr_result' and 'written' values must always be the same
otherwise we're in deep trouble.  So let's remove a redundent report.

And for paranoia sake let's make sure those two variables are actually
equal after all objects are written (one never knows).

Signed-off-by: Nicolas Pitre <nico@cam.org>
Signed-off-by: Junio C Hamano <junkio@cox.net>
2006-11-29 19:06:53 -08:00
Junio C Hamano
e9195b584f pack-objects: tweak "do not even attempt delta" heuristics
The heuristics to give up deltification when both the source and the
target are both in the same pack affects negatively when we are
repacking the subset of objects in the existing pack.  This caused
any incremental updates to use suboptimal packs.  Tweak the heuristics
to avoid this problem.

Signed-off-by: Junio C Hamano <junkio@cox.net>
2006-11-17 00:09:52 -08:00
Nicolas Pitre
231f240b63 git-pack-objects progress flag documentation and cleanup
This adds documentation for --progress and --all-progress, remove a
duplicate --progress handling and make usage string more readable.

Signed-off-by: Nicolas Pitre <nico@cam.org>
Signed-off-by: Junio C Hamano <junkio@cox.net>
2006-11-07 16:05:37 -08:00
Nicolas Pitre
fa438a2eb1 make git-push a bit more verbose
Currently git-push displays progress status for the local packing of
objects to send, but nothing once it starts to push it over the
connection.  Having progress status in that later case is especially
nice when pushing lots of objects over a slow network link.

Signed-off-by: Nicolas Pitre <nico@cam.org>
Signed-off-by: Junio C Hamano <junkio@cox.net>
2006-11-01 15:13:10 -08:00
Junio C Hamano
63fba759bc pack-objects: document --delta-base-offset option
Signed-off-by: Junio C Hamano <junkio@cox.net>
2006-10-10 01:06:20 -07:00
Nicolas Pitre
a270069699 allow delta data reuse even if base object is a preferred base
Signed-off-by: Nicolas Pitre <nico@cam.org>
Signed-off-by: Junio C Hamano <junkio@cox.net>
2006-09-27 13:32:54 -07:00
Nicolas Pitre
f130446920 zap a debug remnant
Signed-off-by: Nicolas Pitre <nico@cam.org>
Signed-off-by: Junio C Hamano <junkio@cox.net>
2006-09-27 12:35:19 -07:00
Nicolas Pitre
780e6e735b make pack data reuse compatible with both delta types
This is the missing part to git-pack-objects allowing it to reuse delta
data to/from any of the two delta types.  It can reuse delta from any
type, and it outputs base offsets when --allow-delta-base-offset is
provided and the base is also included in the pack.  Otherwise it
outputs base sha1 references just like it always did.

Signed-off-by: Nicolas Pitre <nico@cam.org>
Signed-off-by: Junio C Hamano <junkio@cox.net>
2006-09-27 00:12:00 -07:00
Nicolas Pitre
be6b19145f make git-pack-objects able to create deltas with offset to base
This is enabled with --delta-base-offset only, and doesn't work with
pack data reuse yet.

The idea is to allow for the fetch protocol to use an extension flag
to notify the remote end that --delta-base-offset can be used with
git-pack-objects. Eventually git-repack will always provide this flag.

With this, all delta base objects are now pushed before deltas that depend
on them.  This is a requirements for OBJ_OFS_DELTA.  This is not a
requirement for OBJ_REF_DELTA but always doing so makes the code simpler.

Signed-off-by: Nicolas Pitre <nico@cam.org>
Signed-off-by: Junio C Hamano <junkio@cox.net>
2006-09-27 00:12:00 -07:00
Nicolas Pitre
eb32d236df introduce delta objects with offset to base
This adds a new object, namely OBJ_OFS_DELTA, renames OBJ_DELTA to
OBJ_REF_DELTA to better make the distinction between those two delta
objects, and adds support for the handling of those new delta objects
in sha1_file.c only.

The OBJ_OFS_DELTA contains a relative offset from the delta object's
position in a pack instead of the 20-byte SHA1 reference to identify
the base object.  Since the base is likely to be not so far away, the
relative offset is more likely to have a smaller encoding on average
than an absolute offset.  And for those delta objects the base must
always be stored first because there is no way to know the distance of
later objects when streaming a pack.  Hence this relative offset is
always meant to be negative.

The offset encoding is slightly denser than the one used for object
size -- credits to <linux@horizon.com> (whoever this is) for bringing
it to my attention.

This allows for pack size reduction between 3.2% (Linux-2.6) to over 5%
(linux-historic).  Runtime pack access should be faster too since delta
replay does skip a search in the pack index for each delta in a chain.

Signed-off-by: Nicolas Pitre <nico@cam.org>
Signed-off-by: Junio C Hamano <junkio@cox.net>
2006-09-27 00:11:59 -07:00
Nicolas Pitre
43057304c0 many cleanups to sha1_file.c
Those cleanups are mainly to set the table for the support of deltas
with base objects referenced by offsets instead of sha1.  This means
that many pack lookup functions are converted to take a pack/offset
tuple instead of a sha1.

This eliminates many struct pack_entry usages since this structure
carried redundent information in many cases, and it increased stack
footprint needlessly for a couple recursively called functions that used
to declare a local copy of it for every recursion loop.

In the process, packed_object_info_detail() has been reorganized as well
so to look much saner and more amenable to deltas with offset support.

Finally the appropriate adjustments have been made to functions that
depend on the above changes.  But there is no functionality changes yet
simply some code refactoring at this point.

Signed-off-by: Nicolas Pitre <nico@cam.org>
Signed-off-by: Junio C Hamano <junkio@cox.net>
2006-09-23 01:51:33 -07:00
Junio C Hamano
4321134cd8 pack-objects: document --revs, --unpacked and --all.
Signed-off-by: Junio C Hamano <junkio@cox.net>
2006-09-12 22:59:15 -07:00
Junio C Hamano
8d1d8f83b5 pack-objects: further work on internal rev-list logic.
This teaches the internal rev-list logic to understand options
that are needed for pack handling: --all, --unpacked, and --thin.

It also moves two functions from builtin-rev-list to list-objects
so that the two programs can share more code.

Signed-off-by: Junio C Hamano <junkio@cox.net>
2006-09-07 02:46:02 -07:00
Junio C Hamano
b5d97e6b0a pack-objects: run rev-list equivalent internally.
Instead of piping the rev-list output from its standard input,
you can say:

	pack-objects --all --unpacked --revs pack

and feed the rev parameters you would otherwise give the
rev-list on its command line from the standard input.
In other words:

	echo 'master..next' | pack-objects --revs pack

and

	rev-list --objects master..next | pack-objects pack

are equivalent.

Signed-off-by: Junio C Hamano <junkio@cox.net>
2006-09-07 02:46:01 -07:00
Junio C Hamano
72518e9c26 more lightweight revalidation while reusing deflated stream in packing
When copying from an existing pack and when copying from a loose
object with new style header, the code makes sure that the piece
we are going to copy out inflates well and inflate() consumes
the data in full while doing so.

The check to see if the xdelta really apply is quite expensive
as you described, because you would need to have the image of
the base object which can be represented as a delta against
something else.

Signed-off-by: Junio C Hamano <junkio@cox.net>
2006-09-03 21:09:18 -07:00
Junio C Hamano
7042dbf7a1 pack-objects: fix thinko in revalidate code
When revalidating an entry from an existing pack entry->size and
entry->type are not necessarily the size of the final object
when the entry is deltified, but for base objects they must
match.

Signed-off-by: Junio C Hamano <junkio@cox.net>
2006-09-03 14:44:46 -07:00
Junio C Hamano
df6d61017a pack-objects: re-validate data we copy from elsewhere.
When reusing data from an existing pack and from a new style
loose objects, we used to just copy it staight into the
resulting pack.  Instead make sure they are not corrupt, but
do so only when we are not streaming to stdout, in which case
the receiving end will do the validation either by unpacking
the stream or by constructing the .idx file.

Signed-off-by: Junio C Hamano <junkio@cox.net>
2006-09-02 17:08:10 -07:00
Shawn Pearce
e702496e43 Convert memcpy(a,b,20) to hashcpy(a,b).
This abstracts away the size of the hash values when copying them
from memory location to memory location, much as the introduction
of hashcmp abstracted away hash value comparsion.

A few call sites were using char* rather than unsigned char* so
I added the cast rather than open hashcpy to be void*.  This is a
reasonable tradeoff as most call sites already use unsigned char*
and the existing hashcmp is also declared to be unsigned char*.

[jc: Splitted the patch to "master" part, to be followed by a
 patch for merge-recursive.c which is not in "master" yet.

 Fixed the cast in the latter hunk to combine-diff.c which was
 wrong in the original.

 Also converted ones left-over in combine-diff.c, diff-lib.c and
 upload-pack.c ]

Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Signed-off-by: Junio C Hamano <junkio@cox.net>
2006-08-23 13:53:10 -07:00
David Rientjes
a89fccd281 Do not use memcmp(sha1_1, sha1_2, 20) with hardcoded length.
Introduces global inline:

	hashcmp(const unsigned char *sha1, const unsigned char *sha2)

Uses memcmp for comparison and returns the result based on the length of
the hash name (a future runtime decision).

Acked-by: Alex Riesen <raa.lkml@gmail.com>
Signed-off-by: David Rientjes <rientjes@google.com>
Signed-off-by: Junio C Hamano <junkio@cox.net>
2006-08-17 14:23:53 -07:00
David Rientjes
96f1e58f52 remove unnecessary initializations
[jc: I needed to hand merge the changes to the updated codebase,
 so the result needs to be checked.]

Signed-off-by: David Rientjes <rientjes@google.com>
Signed-off-by: Junio C Hamano <junkio@cox.net>
2006-08-15 21:22:20 -07:00
Junio C Hamano
647377c4c9 Merge branch 'jc/pack-objects' 2006-08-12 19:33:16 -07:00
Matthias Kestenholz
5d4a600335 Make git-pack-objects a builtin
Signed-off-by: Matthias Kestenholz <matthias@spinlock.ch>
Signed-off-by: Junio C Hamano <junkio@cox.net>
2006-08-03 23:15:11 -07:00
Renamed from pack-objects.c (Browse further)