When we are fetching from a repository that is on a local
filesystem, first check if we have all the objects that we are
going to fetch available locally, by not just checking the tips
of what we are fetching, but with a full reachability analysis
to our existing refs. In such a case, we do not have to run
git-fetch-pack which would send many needless objects. This is
especially true when the other repository is an alternate of the
current repository (e.g. perhaps the repository was created by
running "git clone -l -s" from there).
The useless objects transferred used to be discarded when they
were expanded by git-unpack-objects called from git-fetch-pack,
but recent git-fetch-pack prefers to keep the data it receives
from the other end without exploding them into loose objects,
resulting in a pack full of duplicated data when fetching from
your own alternate.
This also uses fetch--tool pick-rref on dumb transport side to
remove a shell loop to do the same.
Signed-off-by: Junio C Hamano <junkio@cox.net>
This script helper takes list of fully qualified refnames and
results from ls-remote and grabs only the lines for the named
refs from the latter.
Signed-off-by: Junio C Hamano <junkio@cox.net>
We have fairly extensive coverage of read-tree 3-way machinery,
and many Porcelain-ish tests use git-merge front-end tests, but
we did not have good basic test for merge-recursive, which made
it very hard to hack on it.
I used this during the recent work to teach D/F conflicts to
merge-recursive.
Signed-off-by: Junio C Hamano <junkio@cox.net>
When a path D that originally was blob in the ancestor was
modified on our branch while it was removed on the other branch,
we keep stages 1 and 2, and leave our version in the working
tree. If the other branch created a path D/F, however, that
path can cleanly be resolved in the index (after all, the
ancestor nor we do not have it and only the other side added),
but it cannot be checked out. The issue is the same when the
other branch had D and we had renamed it to D/F, or the ancestor
had D/F instead of D (so there are four combinations).
Do not stop the merge, but leave both D and D/F paths in the
index so that the user can clear things up.
Signed-off-by: Junio C Hamano <junkio@cox.net>
When update-trees::threeway_merge() decides that a path that
exists in the current index (and HEAD) is to be removed, it
leaves a stage 0 entry whose mode bits are set to 0. The code
mistook it as "this stage wants the blob here", and proceeded
to call update_file_flags() which ended up trying to put the
mode=0 entry in the index, got very confused, and ended up
barfing with "do not know what to do with 000000".
Since threeway_merge() does not handle case #10 (one side
removes while the other side does not do anything), this is not
a problem while we refuse to merge branches that have D/F
conflicts, but when we start resolving them, we would need to be
able to remove cache entries, and at that point it starts to
matter.
Signed-off-by: Junio C Hamano <junkio@cox.net>
This fixes three buglets in threeway_merge() regarding D/F
conflict entries.
* After finishing with path D and handling path D/F, some stages
have D/F conflict entry which are obviously non-NULL. For the
purpose of determining if the path D/F is missing in the
ancestor, they should not be taken into account.
* D/F conflict entry is a marker to say "this stage does _not_
have the path", so do not send them to keep_entry().
Signed-off-by: Junio C Hamano <junkio@cox.net>
Case #10 is not handled with unpack-trees.c:threeway_merge()
internally, unless under the agressive rule, and it is not a
bug. As the test expects, ND (one side did not do anything,
other side deleted) case was meant to be handled by the caller's
policy (e.g. git-merge-one-file or git-merge-recursive).
Signed-off-by: Junio C Hamano <junkio@cox.net>
Some oneline descriptions are just too long. In shortlog, it looks much
nicer when they are wrapped. Since print_wrapped_text() is UTF-8 aware,
it also works with those descriptions.
[jc: with minimum fixes]
Signed-off-by: Johannes Schindelin <Johannes.Schindelin@gmx.de>
Signed-off-by: Junio C Hamano <junkio@cox.net>
This replaces the inflate validation with a CRC validation when reusing
data from a pack which uses index version 2. That makes repacking much
safer against corruptions, and it should be a bit faster too.
Signed-off-by: Nicolas Pitre <nico@cam.org>
Signed-off-by: Junio C Hamano <junkio@cox.net>
This is necessary for testing the new capabilities in some automated
way without having an actual 4GB+ pack.
Signed-off-by: Nicolas Pitre <nico@cam.org>
Signed-off-by: Junio C Hamano <junkio@cox.net>
Initially the conversion was made using nth_packed_object_sha1() which
made this file completely index version agnostic. Unfortunately the
overhead was quite significant so I went back to raw index walking but
with selectable base and step values which brought back similar
performances as the original.
Signed-off-by: Nicolas Pitre <nico@cam.org>
Signed-off-by: Junio C Hamano <junkio@cox.net>
When index v2 is encountered, the CRC32 of each object is also displayed
in parenthesis at the end of the line.
Signed-off-by: Nicolas Pitre <nico@cam.org>
Signed-off-by: Junio C Hamano <junkio@cox.net>
With this patch, packs larger than 4GB are usable, even on a 32-bit machine
(at least on Linux). If off_t is not large enough to deal with a large
pack then die() is called instead of attempting to use the pack and
producing garbage.
This was tested with a 8GB pack specially created for the occasion on
a 32-bit machine.
Signed-off-by: Nicolas Pitre <nico@cam.org>
Signed-off-by: Junio C Hamano <junkio@cox.net>
Like previous patch but for index-pack.
[ There is quite some code duplication between pack-objects and index-pack
for generating a pack index (and fast-import as well I suppose). This
should be reworked into a common function eventually. But not now. ]
Signed-off-by: Nicolas Pitre <nico@cam.org>
Signed-off-by: Junio C Hamano <junkio@cox.net>
Pack index version 2 goes as follows:
- 8 bytes of header with signature and version.
- 256 entries of 4-byte first-level fan-out table.
- Table of sorted 20-byte SHA1 records for each object in pack.
- Table of 4-byte CRC32 entries for raw pack object data.
- Table of 4-byte offset entries for objects in the pack if offset is
representable with 31 bits or less, otherwise it is an index in the next
table with top bit set.
- Table of 8-byte offset entries indexed from previous table for offsets
which are 32 bits or more (optional).
- 20-byte SHA1 checksum of sorted object names.
- 20-byte SHA1 checksum of the above.
The object SHA1 table is all contiguous so future pack format that would
contain this table directly won't require big changes to the code. It is
also tighter for slightly better cache locality when looking up entries.
Support for large packs exceeding 31 bits in size won't impose an index
size bloat for packs within that range that don't need a 64-bit offset.
And because newer objects which are likely to be the most frequently used
are located at the beginning of the pack, they won't pay the 64-bit offset
lookup at run time either even if the pack is large.
Right now an index version 2 is created only when the biggest offset in a
pack reaches 31 bits. It might be a good idea to always use index version
2 eventually to benefit from the CRC32 it contains when reusing pack data
while repacking.
[jc: with the "oops" fix to keep track of the last offset correctly]
Signed-off-by: Nicolas Pitre <nico@cam.org>
Signed-off-by: Junio C Hamano <junkio@cox.net>
The most important optimization for performance when repacking is the
ability to reuse data from a previous pack as is and bypass any delta
or even SHA1 computation by simply copying the raw data from one pack
to another directly.
The problem with this is that any data corruption within a copied object
would go unnoticed and the new (repacked) pack would be self-consistent
with its own checksum despite containing a corrupted object. This is a
real issue that already happened at least once in the past.
In some attempt to prevent this, we validate the copied data by inflating
it and making sure no error is signaled by zlib. But this is still not
perfect as a significant portion of a pack content is made of object
headers and references to delta base objects which are not deflated and
therefore not validated when repacking actually making the pack data reuse
still not as safe as it could be.
Of course a full SHA1 validation could be performed, but that implies
full data inflating and delta replaying which is extremely costly, which
cost the data reuse optimization was designed to avoid in the first place.
So the best solution to this is simply to store a CRC32 of the raw pack
data for each object in the pack index. This way any object in a pack can
be validated before being copied as is in another pack, including header
and any other non deflated data.
Why CRC32 instead of a faster checksum like Adler32? Quoting Wikipedia:
Jonathan Stone discovered in 2001 that Adler-32 has a weakness for very
short messages. He wrote "Briefly, the problem is that, for very short
packets, Adler32 is guaranteed to give poor coverage of the available
bits. Don't take my word for it, ask Mark Adler. :-)" The problem is
that sum A does not wrap for short messages. The maximum value of A for
a 128-byte message is 32640, which is below the value 65521 used by the
modulo operation. An extended explanation can be found in RFC 3309,
which mandates the use of CRC32 instead of Adler-32 for SCTP, the
Stream Control Transmission Protocol.
In the context of a GIT pack, we have lots of small objects, especially
deltas, which are likely to be quite small and in a size range for which
Adler32 is dimed not to be sufficient. Another advantage of CRC32 is the
possibility for recovery from certain types of small corruptions like
single bit errors which are the most probable type of corruptions.
OK what this patch does is to compute the CRC32 of each object written to
a pack within pack-objects. It is not written to the index yet and it is
obviously not validated when reusing pack data yet either.
Signed-off-by: Nicolas Pitre <nico@cam.org>
Signed-off-by: Junio C Hamano <junkio@cox.net>
Change a few size and offset variables to more appropriate type, then
add overflow tests on those offsets. This prevents any bad data to be
generated/processed if off_t happens to not be large enough to handle
some big packs.
Better be safe than sorry.
Signed-off-by: Nicolas Pitre <nico@cam.org>
Signed-off-by: Junio C Hamano <junkio@cox.net>
This patch introduces the MSB() macro to obtain the desired number of
most significant bits from a given variable independently of the variable
type.
It is then used to better implement the overflow test on the OBJ_OFS_DELTA
base offset variable with the property of always working correctly
regardless of the type/size of that variable.
Signed-off-by: Nicolas Pitre <nico@cam.org>
Signed-off-by: Junio C Hamano <junkio@cox.net>
The coming index format change doesn't allow for the number of objects
to be determined from the size of the index file directly. Instead, Let's
initialize a field in the packed_git structure with the object count when
the index is validated since the count is always known at that point.
While at it let's reorder some struct packed_git fields to avoid padding
due to needed 64-bit alignment for some of them.
Signed-off-by: Nicolas Pitre <nico@cam.org>
Signed-off-by: Junio C Hamano <junkio@cox.net>
This just makes sure that when we do a read_directory(), we check
that the filename fits in the buffer we allocated (with a bit of
slop)
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
Signed-off-by: Junio C Hamano <junkio@cox.net>
The diff helpers used to do the magic mode canonicalization and all the
other special mode handling by hand ("trust executable bit" and "has
symlink support" handling).
That's bogus. Use "ce_mode_from_stat()" that does this all for us.
This is also going to be required when we add support for links to other
git repositories.
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
Signed-off-by: Junio C Hamano <junkio@cox.net>
Use the comment in the code to document the --exclude-existing
function to git-show-ref.
Signed-off-by: Julian Phillips <julian@quantumfyre.co.uk>
Signed-off-by: Junio C Hamano <junkio@cox.net>
Unlike 'patch --fuzz=NUM', which specifies the number of lines allowed
to mismatch, 'git-apply -CNUM' requests the match of NUM lines of
context. Omitting -C requests full context match, and that's what
should be used for cvsexportcommit -p.
Signed-off-by: Junio C Hamano <junkio@cox.net>
As noted by Junio, --format=tar should be assumed if no format
was specified.
Signed-off-by: Rene Scharfe <rene.scharfe@lsrfire.ath.cx>
Signed-off-by: Junio C Hamano <junkio@cox.net>
* jc/merge-subtree:
A new merge stragety 'subtree'.
It is safe to merge this early as this is a feature that user
explicitly needs to ask for and would not trigger otherwise. A
known issue with the current implementation is that the subtree
matching heuristics is very stupid. It could run ls-tree twice
and try to count intersection.
Giving it wider audience would help it to get improved by
motivated volunteers.
Signed-off-by: Junio C Hamano <junkio@cox.net>
* maint:
Add Documentation/cmd-list.made to .gitignore
git-svn: fix log command to avoid infinite loop on long commit messages
git-svn: dcommit/rebase confused by patches with git-svn-id: lines
git-svn: bail out on incorrect command-line options
This bug has been around since the the conversion to use the
Git.pm library back in October or November. Eventually I'd like
"git rev-list/log" to have the option to not truncate overly
long messages.
Signed-off-by: Eric Wong <normalperson@yhbt.net>
Signed-off-by: Junio C Hamano <junkio@cox.net>
When patches are merged from another git-svn managed branch,
they will have the git-svn-id: metadata line in them (generated
by git-format-patch).
When doing rebase or dcommit via git-svn, this would cause
git-svn to find the wrong upstream branch. We now verify
that the commit is consistent with the value in the .rev_db
file.
Signed-off-by: Eric Wong <normalperson@yhbt.net>
Signed-off-by: Junio C Hamano <junkio@cox.net>
"git svn log" is the only command that needs the pass-through
option in Getopt::Long; otherwise we will bail out and let the
user know something is wrong.
Also, avoid printing out unaccepted mixed-case options (that
are reserved for the command-line) such as --useSvmProps
in the usage() function.
Signed-off-by: Eric Wong <normalperson@yhbt.net>
Signed-off-by: Junio C Hamano <junkio@cox.net>
Documents the new configuration variables and the variable
substitution mechanism.
Signed-off-by: Frank Lichtenheld <frank@lichtenheld.de>
Signed-off-by: Junio C Hamano <junkio@cox.net>
* 'jc/read-tree-df' (early part):
Fix switching to a branch with D/F when current branch has file D.
Fix twoway_merge that passed d/f conflict marker to merged_entry().
Fix read-tree --prefix=dir/.
unpack-trees: get rid of *indpos parameter.
unpack_trees.c: pass unpack_trees_options structure to keep_entry() as well.
add_cache_entry(): removal of file foo does not conflict with foo/bar
Don't include the scheme name in gitcvs.dbdriver, it is
always 'dbi' anyway.
Don't allow ':' in driver names nor ';' in database names for
sanity reasons.
Signed-off-by: Frank Lichtenheld <frank@lichtenheld.de>
Signed-off-by: Junio C Hamano <junkio@cox.net>
Fix a typo: s/Not/Note/
Some formating fixes: Use ` ` syntax for all filenames and
' ' syntax for all commandline switches.
Signed-off-by: Frank Lichtenheld <frank@lichtenheld.de>
Signed-off-by: Junio C Hamano <junkio@cox.net>
This merge strategy largely piggy-backs on git-merge-recursive.
When merging trees A and B, if B corresponds to a subtree of A,
B is first adjusted to match the tree structure of A, instead of
reading the trees at the same level. This adjustment is also
done to the common ancestor tree.
If you are pulling updates from git-gui repository into git.git
repository, the root level of the former corresponds to git-gui/
subdirectory of the latter. The tree object of git-gui's toplevel
is wrapped in a fake tree object, whose sole entry has name 'git-gui'
and records object name of the true tree, before being used by
the 3-way merge code.
If you are merging the other way, only the git-gui/ subtree of
git.git is extracted and merged into git-gui's toplevel.
The detection of corresponding subtree is done by comparing the
pathnames and types in the toplevel of the tree.
Heuristics galore! That's the git way ;-).
Signed-off-by: Junio C Hamano <junkio@cox.net>
When pushing into multiple repositories with git push, via
multiple URL in .git/remotes/$shorthand or multiple url
variables in [remote "$shorthand"] section, we used to stop upon
the first failure. Continue the operation and report the
failure at the end.
Signed-off-by: Junio C Hamano <junkio@cox.net>
This came up on #git when somebody was getting 'unable to create
./objects/tmp_oXXXX' but sweared he had write permission to that
directory. It turned out that the repository URL was changed
and he was accessing a repository he does not have a write
permission anymore.
I am not sure how much this would have helped somebody who
believed he was accessing location when the permission of that
location was changed while he was looking the other way, though.
But giving more information on the error path would be better,
and the next change would be helped with this as well.
Signed-off-by: Junio C Hamano <junkio@cox.net>
* jc/index-output:
git-read-tree --index-output=<file>
_GIT_INDEX_OUTPUT: allow plumbing to output to an alternative index file.
Conflicts:
builtin-apply.c
* cc/bisect:
git-bisect: allow bisecting with only one bad commit.
t6030: add a bit more tests to git-bisect
git-bisect: modernization
Documentation: bisect: "start" accepts one bad and many good commits
Bisect: teach "bisect start" to optionally use one bad and many good revs.
* maint:
Documentation: tighten dependency for git.{html,txt}
Makefile: iconv() on Darwin has the old interface
t5300-pack-object.sh: portability issue using /usr/bin/stat
t3200-branch.sh: small language nit
usermanual.txt: some capitalization nits
Make builtin-branch.c handle the git config file
rename_ref(): only print a warning when config-file update fails
Distinguish branches by more than case in tests.
Avoid composing too long "References" header.
cvsimport: Improve formating consistency
cvsimport: Reorder options in documentation for better understanding
cvsimport: Improve usage error reporting
cvsimport: Improve documentation of CVSROOT and CVS module determination
cvsimport: sync usage lines with existing options
Conflicts:
Documentation/Makefile
Every time _any_ documentation page changed, cmds-*.txt files
were regenerated, which caused git.{html,txt} to be remade. Try
not to update cmds-*.txt files if their new contents match the
old ones.
Signed-off-by: Junio C Hamano <junkio@cox.net>
