My prior version of git-describe ran very slowly on even reasonably
sized projects like git.git and linux.git as it tended to identify
a large number of possible tags and then needed to generate the
revision list for each of those tags to sort them and select the
best tag to describe the input commit.
All we really need is the number of commits in the input revision
which are not in the tag. We can generate these counts during
the revision walking and tag matching loop by assigning a color to
each tag and coloring the commits as we walk them. This limits us
to identifying no more than 26 possible tags, as there is limited
space available within the flags field of struct commit.
The limitation of 26 possible tags is hopefully not going to be a
problem in real usage, as most projects won't create 26 maintenance
releases and merge them back into a development trunk after the
development trunk was tagged with a release candidate tag. If that
does occur git-describe will start to revert to its old behavior of
using the newer maintenance release tag to describe the development
trunk, rather than the development trunk's own tag. The suggested
workaround would be to retag the development trunk's tip.
However since even 26 possible tags can take a while to generate a
description for on some projects I'm defaulting the limit to 10 but
offering the user --candidates to increase the number of possible
matches if they need a more accurate result. I specifically chose
10 for the default as it seems unlikely projects will have more
than 10 maintenance releases merged into a development trunk before
retagging the development trunk, and it seems to perform about the
same on linux.git as v1.4.4.4 git-describe.
A large amount of debugging information was also added during
the development of this change, so I've left it in to be toggled
on with --debug. It may be useful to the end user to help them
understand why git-describe took one particular tag over another.
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Signed-off-by: Junio C Hamano <junkio@cox.net>
If a project has a really huge number of tags (such as several
thousand tags) then we are likely to have nearly a hundred tags in
some buckets. Scanning those buckets as linked lists could take
a large amount of time if done repeatedly during history traversal.
Since we are searching for a unique commit SHA1 we can sort all
tags by commit SHA1 and perform a binary search within the bucket.
Once we identify a particular tag as matching this commit we walk
backwards within the bucket matches to make sure we pick up the
highest priority tag for that commit, as the binary search may
have landed us in the middle of a set of tags which point at the
same commit.
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Signed-off-by: Junio C Hamano <junkio@cox.net>
If a project has a very large number of tags then git-describe
will spend a good part of its time looping over the tags testing
them one at a time to determine if it matches a given commit.
For 10 tags this is not a big deal, but for hundreds of tags the
time could become considerable if we don't find an exact match for
the input commit and we need to walk back along the history chain.
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Signed-off-by: Junio C Hamano <junkio@cox.net>
Several people have suggested that its always better to describe
a commit using an annotated tag, and to only use a lightweight tag
if absolutely no annotated tag matches the input commit.
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Signed-off-by: Junio C Hamano <junkio@cox.net>
1) talk about "git merge" instead of "git pull ."
2) suggest "git repo-config" instead of directly editing config files
3) echo "URL: blah" > .git/remotes/foo is obsolete and should be
"git repo-config remote.foo.url blah"
4) support for partial URL prefix has been removed (see commit
ea560e6d64) so drop mention of it.
Signed-off-by: Nicolas Pitre <nico@cam.org>
Signed-off-by: Junio C Hamano <junkio@cox.net>
Now that the sha1_file.c library routines use the sliding mmap
routines to perform efficient access to portions of a packfile
I can remove that code from fast-import.c and just invoke it.
One benefit is we now have reloading support for any packfile which
uses OBJ_OFS_DELTA. Another is we have significantly less code
to maintain.
This code reuse change *requires* that fast-import generate only
an OBJ_OFS_DELTA format packfile, as there is absolutely no index
available to perform OBJ_REF_DELTA lookup in while unpacking
an object. This is probably reasonable to require as the delta
offsets result in smaller packfiles and are faster to unpack,
as no index searching is required. Its also only a temporary
requirement as users could always repack without offsets before
making the import available to older versions of Git.
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
We say "this shows only the most often used ones"; so instead of
teaching --max-number=<n> form, list -<n> form which is much
easier to type.
Signed-off-by: Junio C Hamano <junkio@cox.net>
Now that we are showing the output messages for verbosity levels
<5 after all actions have been performed (due to the progress meter
running during the actions) it can be confusing to see messages in
the present tense when the user is looking at a '100% done' message
right above them. Converting the messages to past tense will appear
more correct in this case, and shouldn't affect a developer who is
debugging the application and running it at a verbosity level >=5.
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Signed-off-by: Junio C Hamano <junkio@cox.net>
Because large merges on slow systems can take up to a minute to
execute we should try to keep the user entertained with a progress
meter to let them know how far we have progressed through the
current merge.
The progress meter considers each entry in the in-memory index to
be a unit, which means a single recursive merge will double the
number of units in the progress meter. Files which are unmerged
after the 3-way tree merge are also considered a unit within the
progress meter.
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Signed-off-by: Junio C Hamano <junkio@cox.net>
Buffering all message output until a merge invocation is complete is
necessary to prevent intereferring with a progress meter that would
indicate the number of files completely merged, and how many remain.
This change does not introduce a progress meter, but merely lays
the groundwork to buffer the output.
To aid debugging output buffering is only enabled if verbosity
is lower than 5. When using verbosity levels above 5 the user is
probably debugging the merge program itself and does not want to
see the output delayed, especially if they are stepping through
portions of the code in a debugger.
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Signed-off-by: Junio C Hamano <junkio@cox.net>
Junio C Hamano <junkio@cox.net> writes:
>
> I think the output from merge-recursive can be categorized into 5
> verbosity levels:
>
> 1. "CONFLICT", "Rename", "Adding here instead due to D/F conflict"
> (outermost)
>
> 2. "Auto-merged successfully" (outermost)
>
> 3. The first "Merging X with Y".
>
> 4. outermost "Merging:\ntitle1\ntitle2".
>
> 5. outermost "found N common ancestors\nancestor1\nancestor2\n..."
> and anything from inner merge.
>
> I would prefer the default verbosity level to be 2 (that is, show
> both 1 and 2).
and this change makes it so. I think level 3 is probably pointless
as its only one line of output above level 2, but I can see how some
users may want to view it but not view the slightly more verbose
output of level 4.
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Signed-off-by: Junio C Hamano <junkio@cox.net>
Because the output_indent always matches the call_depth value
there is no reason to pass around the call_depth to the merge
function during each recursive invocation.
This is a simple refactoring that will make the code easier to
follow later on as I start to add output verbosity controls.
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Signed-off-by: Junio C Hamano <junkio@cox.net>
* jc/int:
More tests in t3901.
Consistent message encoding while reusing log from an existing commit.
t3901: test "format-patch | am" pipe with i18n
Use log output encoding in --pretty=email headers.
* jc/subdir:
Allow whole-tree operations to be started from a subdirectory
Use cd_to_toplevel in scripts that implement it by hand.
Define cd_to_toplevel shell function in git-sh-setup
Originally I introduced read_or_die for the purpose of reading
the pack header and trailer, and I was too lazy to print proper
error messages.
Linus Torvalds <torvalds@osdl.org>:
> For a read error, at the very least you have to say WHICH FILE
> couldn't be read, because it's usually a matter of some file just
> being too short, not some system-wide problem.
and of course Linus is right. Make it so.
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Signed-off-by: Junio C Hamano <junkio@cox.net>
If the user doesn't have SVN::Core installed or working then the
SVN tests properly turn themselves off. But the user doesn't need
to know that SVN::Core isn't loadable as a Perl module. Unless of
course they are trying to debug the test, so lets relegate the Perl
failures to --verbose only.
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Signed-off-by: Junio C Hamano <junkio@cox.net>
I'm bringing master in early so that the OBJ_OFS_DELTA implementation
is available as part of the topic. This way git-fast-import can
learn about this new slightly smaller and faster packfile format,
and can generate them directly rather than needing to have them be
repacked with git-pack-objects.
Due to the API changes in master during the period of development
of git-fast-import, a few minor tweaks to fast-import.c are needed
to produce a working merge. I've done them here as part of the
merge to ensure bisection always works.
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Some importers may want to create a branch long before they actually
commit to it, or in some cases they may never commit to the branch
but they still need the ref to be created in the repository after
the import is complete.
This extends the 'reset ' command to automatically create a new
branch if the supplied reference isn't already known as a branch.
While I'm at it I also modified the syntax of the reset command
to terminate with an empty line, like commit and tag operate.
This just makes the command set more consistent.
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Some importers are able to determine when branch merges occurred
within their source data. In these cases they will want to supply
the correct commits to fast-import so that a proper merge commit
will exist in Git. This is now supported by supplying a 'merge '
command after the commit message and optional from command.
A merge is not actually performed by fast-import, its assumed that
the frontend performed any sort of merging activity already and
that fast-import should simply be storing its result.
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Apparently we did not copy the blob SHA1 into the stack variable
'sha1' when a mark is used to refer to a prior blob. This code
was not previously tested as the Mozilla CVS -> git-fast-import
program always fed us full SHA1s for modified blobs and did not
use the mark feature there.
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
The new tree delta implementation caused blob SHA1s to be used
instead of a tree SHA1 when a tree was written out. This really
only appeared to happen when converting an existing file to a tree,
but may have been possible in some other situations.
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Since most commits and tag objects are around the same size and we
only generate one at a time we can reuse the same buffer rather than
xmalloc'ing and free'ing the buffer every time we generate a commit.
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
We only ever generate at most two tree streams at a time. Since most
trees are around the same size we can simply recycle the buffers from
one tree generation to the next rather than constantly xmalloc'ing
and free'ing them. This should perform slightly better when handling
a large number of trees as malloc has less work to do.
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
We now store for every tree entry two modes and two sha1 values;
the base (aka "version 0") and the current/new (aka "version 1").
When we generate a tree object we also regenerate the prior version
object and use that as our base object for a delta. This strategy
saves a significant amount of memory as we can continue to use the
atom pool for file/directory names and only increases each tree
entry by an additional 24 bytes of memory.
Branches should automatically delta against their ancestor tree,
unless the ancestor tree is already at the delta chain limit.
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Sometimes an import frontend may need to work with a temporary branch
which will actually contain many different branches over the life
of the import. This is especially useful when the frontend needs
to create a tag from a set of file versions which are otherwise
never a commit.
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
When generating a very large pack file (for example close to 1 GB
in size) it may be impossible for the kernel to find a contiguous
free range within a 32 bit address space for the mapping to be
located at. This is especially problematic on large imports where
there is a lot of malloc activity occuring within the same process
and the malloc'd regions may straddle the previously mapped regions,
thereby creating large holes in the address space.
So instead we map only 128 MB of the pack at any given time.
This will likely increase the number of times the file gets mapped
(with additional system time required to update the page tables
more frequently) but will allow the program to handle packs up to
4 GB in size.
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
fast-import was encounting a GPF when it ran out of free tree_entry
objects but didn't know this was the cause because the last
tree_entry wasn't terminated with a NULL pointer. The missing NULL
pointer occurred when we allocated additional entries via xmalloc
but didn't set the last tree_entry's "next" pointer to NULL.
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
This option can be used to have a record of every commit, the mark
(if supplied) and branch name of the commit recorded into a log file
when the commit is generated. This log can be useful to verify the
results of an import as the commits can be compared to some source
repository matching commits through the mark value.
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
The marks table can be used by the frontend to load any commit after
the import and compare it to whatever data the frontend knows about
that commit. If the mark idnums can be easily correlated to some
reference source then its relatively trivial to compare the GIT
tree to the reference to verify the accuracy of the import.
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
cvs2svn has three phases: begin_commit, middle_commit, end_commit.
The ancester is computed in the middle_commit phase. So its easier
to generate a stream if the from command appears after the commit
message itself but before the file change commands.
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Jon Smirl was finding it difficult to alter cvs2svn to generate
branch commands prior to the first commit of the same branch.
This change moves the 'from' command to be an optional parameter of
the 'commit' command, thereby allowing a new branch to be defined
at the moment it gets used to create the first commit on that branch.
This change makes it impossible to create a branch with no commits
on it as at least one commit is needed to register the branch.
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Some architectures (e.g. SPARC) would require that we access pointers
only on pointer-sized alignments. So ensure the pool allocator
rounds out non-pointer sized allocations to the next pointer so we
don't generate bad memory addresses. This could have occurred if
we had previously allocated an atom whose string was not a whole
multiple of the pointer size, for example.
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Tree reloading allows fast-import to swap out the least-recently used
branch by simply deallocating the data structures from memory that
were associated with that branch. Later if the branch becomes active
again it can lazily recreate those structures on demand by reloading
the necessary trees from the pack file it originally wrote them to.
The reloading process is implemented by mmap'ing the pack into
memory and using a much tighter variant of the pack reading code
contained in sha1_file.c. This was a blatent copy from sha1_file.c
but the unpacking functions were significantly simplified and are
actually now in a form that should make it easier to map only the
necessary regions of a pack rather than the entire file.
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Tags received from the frontend are generated in memory in a simple
linked list in the order that the tag commands were sent by the
frontend. If multiple different tag objects for the same tag name
get generated the last one sent by the frontend will be the one
that gets written out at termination. Multiple tag objects for
the same name will cause all older tags of the same name to be lost.
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
If the branch load count exceeds the number of branches created then
the frontend is causing fast-import to page branches into and out of
memory due to the way its ordering its commits. Performance can
likely be increased if the frontend were to alter its commit
sequence such that it stays on one branch before switching to another
branch, then never returns to the prior branch.
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Marks are now saved when the mark directive gets used by the frontend
and may be used in place of a SHA1 expression to locate a previous
SHA1 which fast-import may have generated. This is particularly
useful with commits where the frontend does not (easily) have the
ability to compute the SHA1 for an arbitrary commit but needs it
to generate a branch or tag from that commit.
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
The following command line options are now accepted before the
pack name:
--objects=n # replaces the object count after the pack name
--depth=n # delta chain depth to use (default is 10)
--active-branches=n # maximum number of branches to keep in memory
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Growing a tree caused all subtrees to be deallocated and put back
into the free list yet those subtree's contents were still actively
in use. Consequently they were doled out again and got stomped
on elsewhere. Releasing a tree is now performed in two parts,
either releasing only the content array or releasing the content
array and recursively releasing the subtree(s).
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
If a frontend is smart enough to import a symlink then we should
let them do so. We'll assume that they were smart enough to first
generate a blob to hold the link target, as that's how symlinks
get represented in GIT.
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Frontend clients can now send a text stream to fast-import rather
than a binary stream. This should facilitate developing frontend
software as the data stream is easier to view, manipulate and debug
my hand and Mark-I eyeball.
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
