* np/pack:
add the capability for index-pack to read from a stream
index-pack: compare only the first 20-bytes of the key.
git-repack: repo.usedeltabaseoffset
pack-objects: document --delta-base-offset option
allow delta data reuse even if base object is a preferred base
zap a debug remnant
let the GIT native protocol use offsets to delta base when possible
make pack data reuse compatible with both delta types
make git-pack-objects able to create deltas with offset to base
teach git-index-pack about deltas with offset to base
teach git-unpack-objects about deltas with offset to base
introduce delta objects with offset to base
Most callers of write_sha1_file_prepare() are only interested in the
resulting hash but don't care about the returned file name or the header.
This patch adds a simple wrapper named hash_sha1_file() which does just
that, and converts potential callers.
Signed-off-by: Rene Scharfe <rene.scharfe@lsrfire.ath.cx>
Signed-off-by: Junio C Hamano <junkio@cox.net>
This adds a new object, namely OBJ_OFS_DELTA, renames OBJ_DELTA to
OBJ_REF_DELTA to better make the distinction between those two delta
objects, and adds support for the handling of those new delta objects
in sha1_file.c only.
The OBJ_OFS_DELTA contains a relative offset from the delta object's
position in a pack instead of the 20-byte SHA1 reference to identify
the base object. Since the base is likely to be not so far away, the
relative offset is more likely to have a smaller encoding on average
than an absolute offset. And for those delta objects the base must
always be stored first because there is no way to know the distance of
later objects when streaming a pack. Hence this relative offset is
always meant to be negative.
The offset encoding is slightly denser than the one used for object
size -- credits to <linux@horizon.com> (whoever this is) for bringing
it to my attention.
This allows for pack size reduction between 3.2% (Linux-2.6) to over 5%
(linux-historic). Runtime pack access should be faster too since delta
replay does skip a search in the pack index for each delta in a chain.
Signed-off-by: Nicolas Pitre <nico@cam.org>
Signed-off-by: Junio C Hamano <junkio@cox.net>
Those cleanups are mainly to set the table for the support of deltas
with base objects referenced by offsets instead of sha1. This means
that many pack lookup functions are converted to take a pack/offset
tuple instead of a sha1.
This eliminates many struct pack_entry usages since this structure
carried redundent information in many cases, and it increased stack
footprint needlessly for a couple recursively called functions that used
to declare a local copy of it for every recursion loop.
In the process, packed_object_info_detail() has been reorganized as well
so to look much saner and more amenable to deltas with offset support.
Finally the appropriate adjustments have been made to functions that
depend on the above changes. But there is no functionality changes yet
simply some code refactoring at this point.
Signed-off-by: Nicolas Pitre <nico@cam.org>
Signed-off-by: Junio C Hamano <junkio@cox.net>
If receive.denyNonFastforwards is set to true, git-receive-pack will deny
non fast-forwards, i.e. forced updates. Most notably, a push to a repository
which has that flag set will fail.
As a first user, 'git-init-db --shared' sets this flag, since in a shared
setup, you are most unlikely to want forced pushes to succeed.
Signed-off-by: Johannes Schindelin <Johannes.Schindelin@gmx.de>
Signed-off-by: Junio C Hamano <junkio@cox.net>
builtin-mailinfo.c has its own hexval implementaiton but it can
share the table-lookup one recently implemented in sha1_file.c
Signed-off-by: Junio C Hamano <junkio@cox.net>
* jc/pack:
pack-objects: document --revs, --unpacked and --all.
pack-objects --unpacked=<existing pack> option.
pack-objects: further work on internal rev-list logic.
pack-objects: run rev-list equivalent internally.
Separate object listing routines out of rev-list
git_connect() can return 0 if we use git protocol for example.
Users of this function don't know and don't care if a process
had been created or not, and to avoid them to check it before
calling finish_connect() this patch allows finish_connect() to
take a null pid. And in that case return 0.
[jc: updated function signature of git_connect() with a comment on
its return value. ]
Signed-off-by: Franck Bui-Huu <vagabon.xyz@gmail.com>
Signed-off-by: Junio C Hamano <junkio@cox.net>
Incremental repack without -a essentially boils down to:
rev-list --objects --unpacked --all |
pack-objects $new_pack
which picks up all loose objects that are still live and creates
a new pack.
This implements --unpacked=<existing pack> option to tell the
revision walking machinery to pretend as if objects in such a
pack are unpacked for the purpose of object listing. With this,
we could say:
rev-list --objects --unpacked=$active_pack --all |
pack-objects $new_pack
instead, to mean "all live loose objects but pretend as if
objects that are in this pack are also unpacked". The newly
created pack would be perfect for updating $active_pack by
replacing it.
Since pack-objects now knows how to do the rev-list's work
itself internally, you can also write the above example by:
pack-objects --unpacked=$active_pack --all $new_pack </dev/null
Signed-off-by: Junio C Hamano <junkio@cox.net>
When copying from an existing pack and when copying from a loose
object with new style header, the code makes sure that the piece
we are going to copy out inflates well and inflate() consumes
the data in full while doing so.
The check to see if the xdelta really apply is quite expensive
as you described, because you would need to have the image of
the base object which can be represented as a delta against
something else.
Signed-off-by: Junio C Hamano <junkio@cox.net>
If GIT_TRACE is set to an absolute path (starting with a
'/' character), we interpret this as a file path and we
trace into it.
Also if GIT_TRACE is set to an integer value greater than
1 and lower than 10, we interpret this as an open fd value
and we trace into it.
Note that this behavior is not compatible with the
previous one.
We also trace whole messages using one write(2) call to
make sure messages from processes do net get mixed up in
the middle.
This patch makes it possible to get trace information when
running "make test".
Signed-off-by: Christian Couder <chriscool@tuxfamily.org>
Signed-off-by: Junio C Hamano <junkio@cox.net>
* js/c-merge-recursive: (21 commits)
discard_cache(): discard index, even if no file was mmap()ed
merge-recur: do not die unnecessarily
merge-recur: try to merge older merge bases first
merge-recur: if there is no common ancestor, fake empty one
merge-recur: do not setenv("GIT_INDEX_FILE")
merge-recur: do not call git-write-tree
merge-recursive: fix rename handling
.gitignore: git-merge-recur is a built file.
merge-recur: virtual commits shall never be parsed
merge-recur: use the unpack_trees() interface instead of exec()ing read-tree
merge-recur: fix thinko in unique_path()
Makefile: git-merge-recur depends on xdiff libraries.
merge-recur: Explain why sha_eq() and struct stage_data cannot go
merge-recur: Cleanup last mixedCase variables...
merge-recur: Fix compiler warning with -pedantic
merge-recur: Remove dead code
merge-recur: Get rid of debug code
merge-recur: Convert variable names to lower_case
Cumulative update of merge-recursive in C
recur vs recursive: help testing without touching too many stuff.
...
This is an evil merge that removes TEST script from the toplevel.
I noticed that I was looking at the kernel gitweb output at some point
rather than just do "git log", simply because I liked seeing the
simplified date-format, ie the "5 days ago" rather than a full date.
This adds infrastructure to do that for "git log" too. It does NOT add the
actual flag to enable it, though, so right now this patch is a no-op, but
it should now be easy to add a command line flag (and possibly a config
file option) to just turn on the "relative" date format.
The exact cut-off points when it switches from days to weeks etc are
totally arbitrary, but are picked somewhat to avoid the "1 weeks ago"
thing (by making it show "10 days ago" rather than "1 week", or "70
minutes ago" rather than "1 hour ago").
[jc: with minor fix and tweak around "month" and "week" area.]
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
Signed-off-by: Junio C Hamano <junkio@cox.net>
This abstracts away the size of the hash values when copying them
from memory location to memory location, much as the introduction
of hashcmp abstracted away hash value comparsion.
A few call sites were using char* rather than unsigned char* so
I added the cast rather than open hashcpy to be void*. This is a
reasonable tradeoff as most call sites already use unsigned char*
and the existing hashcmp is also declared to be unsigned char*.
[jc: Splitted the patch to "master" part, to be followed by a
patch for merge-recursive.c which is not in "master" yet.
Fixed the cast in the latter hunk to combine-diff.c which was
wrong in the original.
Also converted ones left-over in combine-diff.c, diff-lib.c and
upload-pack.c ]
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Signed-off-by: Junio C Hamano <junkio@cox.net>
The little helper write_or_die() won't come back with bad news about
full disks or broken pipes. It either succeeds or terminates the
program, making additional error handling unnecessary.
This patch adds the new function and uses it to replace two similar
ones (the one in tar-tree originally has been copied from cat-file
btw.). I chose to add the fd parameter which both lacked to make
write_or_die() just as flexible as write() and thus suitable for
lib-ification.
There is a regression: error messages emitted by this function don't
show the program name, while the replaced two functions did. That's
acceptable, I think; a lot of other functions do the same.
Signed-off-by: Rene Scharfe <rene.scharfe@lsrfire.ath.cx>
Signed-off-by: Junio C Hamano <junkio@cox.net>
Introduces global inline:
hashcmp(const unsigned char *sha1, const unsigned char *sha2)
Uses memcmp for comparison and returns the result based on the length of
the hash name (a future runtime decision).
Acked-by: Alex Riesen <raa.lkml@gmail.com>
Signed-off-by: David Rientjes <rientjes@google.com>
Signed-off-by: Junio C Hamano <junkio@cox.net>
Replace sha1 comparisons to null_sha1 with a global inline (which previously an
unused static inline in builtin-apply.c)
[jc: with a fix from Jonas Fonseca.]
Signed-off-by: David Rientjes <rientjes@google.com>
Signed-off-by: Junio C Hamano <junkio@cox.net>
Most of the callers except the one in refs.c use the function to
update the index file. Among the index writers, everybody
except write-tree dies if they cannot open it for writing.
This gives the function an extra argument, to tell it to die
when it cannot create a new file as the lockfile.
The only caller that does not have to die is write-tree, because
updating the index for the cache-tree part is optional and not
being able to do so does not affect the correctness. I think we
do not have to be so careful and make the failure into die() the
same way as other callers, but that would be a different patch.
Signed-off-by: Junio C Hamano <junkio@cox.net>
enable/disable colored output when the pager is in use
Signed-off-by: Matthias Lederhofer <matled@gmx.net>
Signed-off-by: Junio C Hamano <junkio@cox.net>
* js/read-tree: (107 commits)
read-tree: move merge functions to the library
read-trees: refactor the unpack_trees() part
tar-tree: illustrate an obscure feature better
git.c: allow alias expansion without a git directory
setup_git_directory_gently: do not barf when GIT_DIR is given.
Build on Debian GNU/kFreeBSD
Call setup_git_directory() much earlier
Call setup_git_directory() early
Display an error from update-ref if target ref name is invalid.
Fix http-fetch
t4103: fix binary patch application test.
git-apply -R: binary patches are irreversible for now.
Teach git-apply about '-R'
Makefile: ssh-pull.o depends on ssh-fetch.c
log and diff family: honor config even from subdirectories
git-reset: detect update-ref error and report it.
lost-found: use fsck-objects --full
Teach git-http-fetch the --stdin switch
Teach git-local-fetch the --stdin switch
Make pull() support fetching multiple targets at once
...
This also moves add_file_to_index() to read-cache.c. Oh, and while
touching builtin-add.c, it also removes a duplicate git_config() call.
Signed-off-by: Johannes Schindelin <Johannes.Schindelin@gmx.de>
Signed-off-by: Junio C Hamano <junkio@cox.net>
This backports the pieces that are not uncooked from the merge-recursive
WIP we have seen earlier, to be used in git-mv rewritten in C.
Signed-off-by: Junio C Hamano <junkio@cox.net>
This exposes map_sha1_file() interface to mmap a loose object file,
and legacy_loose_object() function, split from unpack_sha1_header().
They will be used in the next patch to reuse the deflated data from
new-style loose object files when generating packs.
Signed-off-by: Junio C Hamano <junkio@cox.net>
The pack-file format is slightly different from the traditional git
object format, in that it has a much denser binary header encoding.
The traditional format uses an ASCII string with type and length
information, which is somewhat wasteful.
A new object format starts with uncompressed binary header
followed by compressed payload -- this will allow us later to
copy the payload straight to packfiles.
Obviously they cannot be read by older versions of git, so for
now new object files are created with the traditional format.
core.legacyheaders configuration item, when set to false makes
the code write in new format for people to experiment with.
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
Signed-off-by: Junio C Hamano <junkio@cox.net>
This is just an update for people being interested. Alex and me were
busy with that project for a few days now. While it has progressed nicely,
there are quite a couple TODOs in merge-recursive.c, just search for "TODO".
For impatient people: yes, it passes all the tests, and yes, according
to the evil test Alex did, it is faster than the Python script.
But no, it is not yet finished. Biggest points are:
- there are still three external calls
- in the end, it should not be necessary to write the index more than once
(just before exiting)
- a lot of things can be refactored to make the code easier and shorter
BTW we cannot just plug in git-merge-tree yet, because git-merge-tree
does not handle renames at all.
This patch is meant for testing, and as such,
- it compile the program to git-merge-recur
- it adjusts the scripts and tests to use git-merge-recur instead of
git-merge-recursive
- it provides "TEST", a script to execute the tests regarding -recursive
- it inlines the changes to read-cache.c (read_cache_from(), discard_cache()
and refresh_cache_entry())
Brought to you by Alex Riesen and Dscho
Signed-off-by: Junio C Hamano <junkio@cox.net>
Nobody else uses them, and I'm going to start changing them.
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
Signed-off-by: Junio C Hamano <junkio@cox.net>
This allows you to say:
git -p diff v2.6.16-rc5..
and the command pipes the output of any git command to your pager.
[jc: this resurrects a month old RFC patch with improvement
suggested by Linus to call it --paginate instead of --less.]
Signed-off-by: Junio C Hamano <junkio@cox.net>
This makes git-peek-remote able to basically do everything that
git-ls-remote does (but obviously just for the native protocol, so no
http[s]: or rsync: support).
The default behaviour is the same, but you can now give a mixture of
"--refs", "--tags" and "--heads" flags, where "--refs" forces
git-peek-remote to only show real refs (ie none of the fakey tag lookups,
but also not the special pseudo-refs like HEAD and MERGE_HEAD).
The "--tags" and "--heads" flags respectively limit the output to just
regular tags and heads, of course.
You can still also ask to limit them by name too.
You can combine the flags, so
git peek-remote --refs --tags .
will show all local _true_ tags, without the generated tag lookups
(compare the output without the "--refs" flag).
And "--tags --heads" will show both tags and heads, but will avoid (for
example) any special refs outside of the standard locations.
I'm also planning on adding a "--ignore-local" flag that allows us to ask
it to ignore any refs that we already have in the local tree, but that's
an independent thing.
All this is obviously gearing up to making "git fetch" cheaper.
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
Signed-off-by: Junio C Hamano <junkio@cox.net>
With the change in default, "git add ." on kernel dir is about
twice as fast as before, with only minimal (0.5%) change in
object size. The speed difference is even more noticeable
when committing large files, which is now up to 8 times faster.
The configurability is through setting core.compression = [-1..9]
which maps to the zlib constants; -1 is the default, 0 is no
compression, and 1..9 are various speed/size tradeoffs, 9
being slowest.
Signed-off-by: Joachim B Haga (cjhaga@fys.uio.no)
Acked-by: Linus Torvalds <torvalds@osdl.org>
Signed-off-by: Junio C Hamano <junkio@cox.net>
This lets you use something like this in your $GIT_DIR/config
file.
[diff]
color = auto
[diff.color]
new = blue
old = yellow
frag = reverse
When diff.color is set to "auto", colored diff is enabled when
the standard output is the terminal. Other choices are "always",
and "never". Usual boolean true/false can also be used.
The colormap entries can specify colors for the following slots:
plain - lines that appear in both old and new file (context)
meta - diff --git header and extended git diff headers
frag - @@ -n,m +l,k @@ lines (hunk header)
old - lines deleted from old file
new - lines added to new file
The following color names can be used:
normal, bold, dim, l, blink, reverse, reset,
black, red, green, yellow, blue, magenta, cyan,
white
Signed-off-by: Junio C Hamano <junkio@cox.net>
This cleans up the use of safe_strncpy() even more. Since it has the
same semantics as strlcpy() use this name instead. Also move the
definition from inside path.c to its own file compat/strlcpy.c, and use
it conditionally at compile time, since some platforms already has
strlcpy(). It's included in the same way as compat/setenv.c.
Signed-off-by: Peter Eriksen <s022018@student.dtu.dk>
Signed-off-by: Junio C Hamano <junkio@cox.net>
This implements a protocol extension between fetch-pack and
upload-pack to allow stderr stream from upload-pack (primarily
used for the progress bar display) to be passed back.
Signed-off-by: Junio C Hamano <junkio@cox.net>
This creates a simple specialized object allocator for basic
objects.
This avoids wasting space with malloc overhead (metadata and
extra alignment), since the specialized allocator knows the
alignment, and that objects, once allocated, are never freed.
It also allows us to track some basic statistics about object
allocations. For example, for the mozilla import, it shows
object usage as follows:
blobs: 627629 (14710 kB)
trees: 1119035 (34969 kB)
commits: 196423 (8440 kB)
tags: 1336 (46 kB)
and the simpler allocator shaves off about 2.5% off the memory
footprint off a "git-rev-list --all --objects", and is a bit
faster too.
[ Side note: this concludes the series of "save memory in object storage".
The thing is, there simply isn't much more to be saved on the objects.
Doing "git-rev-list --all --objects" on the mozilla archive has a final
total RSS of 131498 pages for me: that's about 513MB. Of that, the
object overhead is now just 56MB, the rest is going somewhere else (put
another way: the fact that this patch shaves off 2.5% of the total
memory overhead, considering that objects are now not much more than 10%
of the total shows how big the wasted space really was: this makes
object allocations much more memory- and time-efficient).
I haven't looked at where the rest is, but I suspect the bulk of it is
just the pack-file loading. It may be that we should pack the tree
objects separately from the blob objects: for git-rev-list --objects, we
don't actually ever need to even look at the blobs, but since trees and
blobs are interspersed in the pack-file, we end up not being dense in
the tree accesses, so we end up looking at more pages than we strictly
need to.
So with a 535MB pack-file, it's entirely possible - even likely - that
most of the remaining RSS is just the mmap of the pack-file itself. We
don't need to map in _all_ of it, but we do end up mapping a fair
amount. ]
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
Signed-off-by: Junio C Hamano <junkio@cox.net>
This enhances core.sharedrepository to have additionally
specify that read and exec permissions to be given to others as
well. It is useful when serving a repository via gitweb and
git-daemon that runs as a user outside the project group.
The configuration item can take the following values:
[core]
sharedrepository ; the same as "group"
sharedrepository = true ; ditto
sharedrepository = 1 ; ditto
sharedrepository = group ; allow rwx to group
sharedrepository = all ; allow rwx to group, allow rx to other
sharedrepository = umask ; not shared - use umask
It also extends "git init-db" to take "--shared=all" and friends
from the command line.
Signed-off-by: Junio C Hamano <junkio@cox.net>
The framework to create lockfiles that are removed at exit is
first used to reliably write the index file, but it is
applicable to other things, so stop calling it "cache_file".
This also rewords a few remaining error message that called the
index file "cache file".
Signed-off-by: Junio C Hamano <junkio@cox.net>
* sp/reflog:
fetch.c: do not pass uninitialized lock to unlock_ref().
Test that git-branch -l works.
Verify git-commit provides a reflog message.
Enable ref log creation in git checkout -b.
Create/delete branch ref logs.
Include ref log detail in commit, reset, etc.
Change order of -m option to update-ref.
Correct force_write bug in refs.c
Change 'master@noon' syntax to 'master@{noon}'.
Log ref updates made by fetch.
Force writing ref if it doesn't exist.
Added logs/ directory to repository layout.
General ref log reading improvements.
Fix ref log parsing so it works properly.
Support 'master@2 hours ago' syntax
Log ref updates to logs/refs/<ref>
Convert update-ref to use ref_lock API.
Improve abstraction of ref lock/write.
* jc/cache-tree: (26 commits)
builtin-rm: squelch compiler warnings.
git-write-tree writes garbage on sparc64
Fix crash when reading the empty tree
fsck-objects: do not segfault on missing tree in cache-tree
cache-tree: a bit more debugging support.
read-tree: invalidate cache-tree entry when a new index entry is added.
Fix test-dump-cache-tree in one-tree disappeared case.
fsck-objects: mark objects reachable from cache-tree
cache-tree: replace a sscanf() by two strtol() calls
cache-tree.c: typefix
test-dump-cache-tree: validate the cached data as well.
cache_tree_update: give an option to update cache-tree only.
read-tree: teach 1-way merege and plain read to prime cache-tree.
read-tree: teach 1 and 2 way merges about cache-tree.
update-index: when --unresolve, smudge the relevant cache-tree entries.
test-dump-cache-tree: report number of subtrees.
cache-tree: sort the subtree entries.
Teach fsck-objects about cache-tree.
index: make the index file format extensible.
cache-tree: protect against "git prune".
...
Conflicts:
Makefile, builtin.h, git.c: resolved the same way as in next.
* master: (90 commits)
fetch.c: remove an unused variable and dead code.
Clean up sha1 file writing
Builtin git-cat-file
builtin format-patch: squelch content-type for 7-bit ASCII
CMIT_FMT_EMAIL: Q-encode Subject: and display-name part of From: fields.
add more informative error messages to git-mktag
remove the artificial restriction tagsize < 8kb
git-rebase: use canonical A..B syntax to format-patch
git-format-patch: now built-in.
fmt-patch: Support --attach
fmt-patch: understand old <his> notation
Teach fmt-patch about --keep-subject
Teach fmt-patch about --numbered
fmt-patch: implement -o <dir>
fmt-patch: output file names to stdout
Teach fmt-patch to write individual files.
built-in tar-tree and remote tar-tree
Builtin git-diff-files, git-diff-index, git-diff-stages, and git-diff-tree.
Builtin git-show-branch.
Builtin git-apply.
...
This makes "git format-patch" a built-in.
* js/fmt-patch:
git-rebase: use canonical A..B syntax to format-patch
git-format-patch: now built-in.
fmt-patch: Support --attach
fmt-patch: understand old <his> notation
Teach fmt-patch about --keep-subject
Teach fmt-patch about --numbered
fmt-patch: implement -o <dir>
fmt-patch: output file names to stdout
Teach fmt-patch to write individual files.
Use RFC2822 dates from "git fmt-patch".
git-fmt-patch: thinkofix to show [PATCH] properly.
rename internal format-patch wip
Minor tweak on subject line in --pretty=email
Tentative built-in format-patch.
This makes 'git add' and 'git rm' built-ins.
* lt/dirwalk:
Add builtin "git rm" command
Move pathspec matching from builtin-add.c into dir.c
Prevent bogus paths from being added to the index.
builtin-add: fix unmatched pathspec warnings.
Remove old "git-add.sh" remnants
builtin-add: warn on unmatched pathspecs
Do "git add" as a builtin
Clean up git-ls-file directory walking library interface
libify git-ls-files directory traversal
