Enables threading in index-pack to resolve base data in parallel.
By Nguyễn Thái Ngọc Duy (3) and Ramsay Jones (1)
* nd/threaded-index-pack:
index-pack: disable threading if NO_PREAD is defined
index-pack: support multithreaded delta resolving
index-pack: restructure pack processing into three main functions
compat/win32/pthread.h: Add an pthread_key_delete() implementation
Your build platform may support hardlinks but you may prefer not to use
them, e.g. when installing to DESTDIR to make a tarball and untarring on
a filesystem that has poor support for hardlinks.
The Makefile in git-gui project may need to learn to honor the same
setting; it unconditionally creates git-citool by hardlinking git-gui.
* jc/install-no-hardlinks:
Makefile: NO_INSTALL_HARDLINKS
This puts delta resolving on each base on a separate thread, one base
cache per thread. Per-thread data is grouped in struct thread_local.
When running with nr_threads == 1, no pthreads calls are made. The
system essentially runs in non-thread mode.
An experiment on a Xeon 24 core machine with git.git shows that
performance does not increase proportional to the number of cores. So
by default, we use maximum 3 cores. Some numbers with --threads from 1
to 16:
1..4
real 0m8.003s 0m5.307s 0m4.321s 0m3.830s
user 0m7.720s 0m8.009s 0m8.133s 0m8.305s
sys 0m0.224s 0m0.372s 0m0.360s 0m0.360s
5..8
real 0m3.727s 0m3.604s 0m3.332s 0m3.369s
user 0m9.361s 0m9.817s 0m9.525s 0m9.769s
sys 0m0.584s 0m0.624s 0m0.540s 0m0.560s
9..12
real 0m3.036s 0m3.139s 0m3.177s 0m2.961s
user 0m8.977s 0m10.205s 0m9.737s 0m10.073s
sys 0m0.596s 0m0.680s 0m0.684s 0m0.680s
13..16
real 0m2.985s 0m2.894s 0m2.975s 0m2.971s
user 0m9.825s 0m10.573s 0m10.833s 0m11.361s
sys 0m0.788s 0m0.732s 0m0.904s 0m1.016s
On an Intel dual core and linux-2.6.git
1..4
real 2m37.789s 2m7.963s 2m0.920s 1m58.213s
user 2m28.415s 2m52.325s 2m50.176s 2m41.187s
sys 0m7.808s 0m11.181s 0m11.224s 0m10.731s
Thanks Ramsay Jones for troubleshooting and support on MinGW platform.
Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Your filesystem may support hardlinks, but you may choose not to use them
when installing git-foo builtins and favor symblic links or copies for
whatever reason.
The installation procedure of git-gui/ directory is not touched with this
patch and git-citool still ends up being a hardlink to git-gui, but it
needs to be addressed separately.
Signed-off-by: Junio C Hamano <gitster@pobox.com>
We tend to keep long lists sorted (extensions are not taken into
account), which helps spot a name easily by eye. Rearrange a few
items so these lists remain sorted.
Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
A couple of commands learn --column option to produce columnar output.
By Nguyễn Thái Ngọc Duy (9) and Zbigniew Jędrzejewski-Szmek (1)
* nd/columns:
tag: add --column
column: support piping stdout to external git-column process
status: add --column
branch: add --column
help: reuse print_columns() for help -a
column: add dense layout support
t9002: work around shells that are unable to set COLUMNS to 1
column: add columnar layout
Stop starting pager recursively
Add column layout skeleton and git-column
Use word-at-a-time comparison to find end of line or NUL (end of buffer),
borrowed from the linux-kernel discussion.
By Thomas Rast
* tr/xdiff-fast-hash:
xdiff: choose XDL_FAST_HASH code on sizeof(long) instead of __WORDSIZE
xdiff: load full words in the inner loop of xdl_hash_record
More message strings marked for i18n.
By Nguyễn Thái Ngọc Duy (10) and Jonathan Nieder (1)
* nd/i18n:
help: replace underlining "help -a" headers using hyphens with a blank line
i18n: bundle: mark strings for translation
i18n: index-pack: mark strings for translation
i18n: apply: update say_patch_name to give translators complete sentence
i18n: apply: mark strings for translation
i18n: remote: mark strings for translation
i18n: make warn_dangling_symref() automatically append \n
i18n: help: mark strings for translation
i18n: mark relative dates for translation
strbuf: convenience format functions with \n automatically appended
Makefile: feed all header files to xgettext
Trivially shrinks the on-disk size of the index file to save both I/O and
checksum overhead.
The topic should give a solid base to build on further updates, with the
code refactoring in its earlier parts, and the backward compatibility
mechanism in its later parts.
* jc/index-v4:
index-v4: document the entry format
unpack-trees: preserve the index file version of original
update-index: upgrade/downgrade on-disk index version
read-cache.c: write prefix-compressed names in the index
read-cache.c: read prefix-compressed names in index on-disk version v4
read-cache.c: move code to copy incore to ondisk cache to a helper function
read-cache.c: move code to copy ondisk to incore cache to a helper function
read-cache.c: report the header version we do not understand
read-cache.c: make create_from_disk() report number of bytes it consumed
read-cache.c: allow unaligned mapping of the index file
cache.h: hide on-disk index details
varint: make it available outside the context of pack
A column option string consists of many token separated by either
a space or a comma. A token belongs to one of three groups:
- enabling: always, never and auto
- layout mode: currently plain (which does not layout at all)
- other future tuning flags
git-column can be used to pipe output to from a command that wants
column layout, but not to mess with its own output code. Simpler output
code can be changed to use column layout code directly.
Thanks-to: Ramsay Jones <ramsay@ramsay1.demon.co.uk>
Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
This patch also marks most common commands' synopsis for translation
so that "git help" gives a friendly listing.
Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
"git push --recurse-submodules" learns to optionally look into the
histories of submodules bound to the superproject and push them out.
By Heiko Voigt
* hv/submodule-recurse-push:
push: teach --recurse-submodules the on-demand option
Refactor submodule push check to use string list instead of integer
Teach revision walking machinery to walk multiple times sequencially
Setting up a revision traversal with many starting points was inefficient
as these were placed in a date-order priority queue one-by-one.
By René Scharfe (3) and Junio C Hamano (1)
* rs/commit-list-sort-in-batch:
mergesort: rename it to llist_mergesort()
revision: insert unsorted, then sort in prepare_revision_walk()
commit: use mergesort() in commit_list_sort_by_date()
add mergesort() for linked lists
Make it easier for distros to document custom pager and editor they
used when building their binary releases in "git var" documentation.
By Jonathan Nieder
* jn/debian-customizes-default-editor:
var doc: advertise current DEFAULT_PAGER and DEFAULT_EDITOR settings
var doc: default editor and pager are configurable at build time
Translation markers may be present in header files too. Make sure we
don't miss any.
Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
This adds a generic bottom-up mergesort implementation for singly linked
lists. It was inspired by Simon Tatham's webpage on the topic[1], but
not so much by his implementation -- for no good reason, really, just a
case of NIH.
[1] http://www.chiark.greenend.org.uk/~sgtatham/algorithms/listsort.html
Signed-off-by: Rene Scharfe <rene.scharfe@lsrfire.ath.cx>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Document the default pager and editor chosen at compile time in the
git-var(1) manpage so users curious about what command _this_ copy of
git will fall back to when EDITOR, VISUAL, and PAGER are unset can
find the answer quickly.
In builds leaving those settings uncustomized, this patch makes the
manpage continue to say "usually vi" and "usually less" so the
formatted documentation is usable for a wide audience including users
of custom builds that change those settings. If you would like your
copy of the docs to be less noncommittal, you will need to set
DEFAULT_PAGER=less and DEFAULT_EDITOR=vi explicitly.
Suggested-by: Junio C Hamano <gitster@pobox.com>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Redo the hashing loop in xdl_hash_record in a way that loads an entire
'long' at a time, using masking tricks to see when and where we found
the terminating '\n'.
I stole inspiration and code from the posts by Linus Torvalds around
https://lkml.org/lkml/2012/3/2/452https://lkml.org/lkml/2012/3/5/6
His method reads the buffers in sizeof(long) increments, and may thus
overrun it by at most sizeof(long)-1 bytes before it sees the final
newline (or hits the buffer length check). I considered padding out
all buffers by a suitable amount to "catch" the overrun, but
* this does not work for mmap()'d buffers: if you map 4096+8 bytes
from a 4096 byte file, accessing the last 8 bytes results in a
SIGBUS on my machine; and
* it would also be extremely ugly because it intrudes deep into the
unpacking machinery.
So I adapted it to not read beyond the buffer at all. Instead, it
reads the final partial word byte-by-byte and strings it together.
Then it can use the same logic as before to finish the hashing.
So far we enable this only on x86_64, where it provides nice speedup
for diff-related work:
Test origin/next tr/xdiff-fast-hash
-----------------------------------------------------------------------------
4000.1: log -3000 (baseline) 0.07(0.05+0.02) 0.08(0.06+0.02) +14.3%
4000.2: log --raw -3000 (tree-only) 0.37(0.33+0.04) 0.37(0.32+0.04) +0.0%
4000.3: log -p -3000 (Myers) 1.75(1.65+0.09) 1.60(1.49+0.10) -8.6%
4000.4: log -p -3000 --histogram 1.73(1.62+0.09) 1.58(1.49+0.08) -8.7%
4000.5: log -p -3000 --patience 2.11(2.00+0.10) 1.94(1.80+0.11) -8.1%
Perhaps other platforms could also benefit. However it does NOT work
on big-endian systems!
[jc: minimum style and compilation fixes]
Signed-off-by: Thomas Rast <trast@student.ethz.ch>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Move git-p4 out of contrib/fast-import into the main code base,
aside other foreign SCM tools.
Signed-off-by: Pete Wyckoff <pw@padd.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
During the testing of the 1.7.10 rc series on Solaris for OpenCSW, it
was discovered that t7006-pager was failing due to finding a bad "sh"
in PATH after a call to execvp("sh", ...). This call was setup by
run_command.c:prepare_shell_cmd.
The PATH in use at the time saw /opt/csw/bin given precedence to
traditional Solaris paths such as /usr/bin and /usr/xpg4/bin. A
package named schilyutils (Joerg Schilling's utilities) was installed
on the build system and it delivered a modified version of the
traditional Solaris /usr/bin/sh as /opt/csw/bin/sh. This version of
sh suffers from many of the same problems as /usr/bin/sh.
The command-specific pager test failed due to the broken "sh" handling
^ as a pipe character. It tried to fork two processes when it
encountered "sed s/^/foo:/" as the pager command. This problem was
entirely dependent on the PATH of the user at runtime.
Possible fixes for this issue are:
1. Use the standard system() or popen() which both launch a POSIX
shell on Solaris as long as _POSIX_SOURCE is defined.
2. The git wrapper could prepend SANE_TOOL_PATH to PATH thus forcing
all unqualified commands run to use the known good tools on the
system.
3. The run_command.c:prepare_shell_command() could use the same
SHELL_PATH that is in the #! line of all all scripts and not rely
on PATH to find the sh to run.
Option 1 would preclude opening a bidirectional pipe to a filter
script and would also break git for Windows as cmd.exe is spawned from
system() (cf. v1.7.5-rc0~144^2, "alias: use run_command api to execute
aliases, 2011-01-07).
Option 2 is not friendly to users as it would negate their ability to
use tools of their choice in many cases. Alternately, injecting
SANE_TOOL_PATH such that it takes precedence over /bin and /usr/bin
(and anything with lower precedence than those paths) as
git-sh-setup.sh does would not solve the problem either as the user
environment could still allow a bad sh to be found. (Many OpenCSW
users will have /opt/csw/bin leading their PATH and some subset would
have schilyutils installed.)
Option 3 allows us to use a known good shell while still honouring the
users' PATH for the utilities being run. Thus, it solves the problem
while not negatively impacting either users or git's ability to run
external commands in convenient ways. Essentially, the shell is a
special case of tool that should not rely on SANE_TOOL_PATH and must
be called explicitly.
With this patch applied, any code path leading to
run_command.c:prepare_shell_cmd can count on using the same sane shell
that all shell scripts in the git suite use. Both the build system
and run_command.c will default this shell to /bin/sh unless
overridden.
Signed-off-by: Ben Walton <bwalton@artsci.utoronto.ca>
Reviewed-by: Jonathan Nieder <jrnieder@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Previously it was not possible to iterate revisions twice using the
revision walking api. We add a reset_revision_walk() which clears the
used flags. This allows us to do multiple sequencial revision walks.
We add the appropriate calls to the existing submodule machinery doing
revision walks. This is done to avoid surprises if future code wants to
call these functions more than once during the processes lifetime.
Signed-off-by: Heiko Voigt <hvoigt@hvoigt.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
* maint:
Document accumulated fixes since 1.7.9.2
Git 1.7.8.5
grep -P: Fix matching ^ and $
am: don't infloop for an empty input file
rebase -m: only call "notes copy" when rewritten exists and is non-empty
git-p4: remove bash-ism in t9800
git-p4: remove bash-ism in t9809
git-p4: fix submit regression with clientSpec and subdir clone
git-p4: set useClientSpec variable on initial clone
Makefile: add thread-utils.h to LIB_H
Conflicts:
RelNotes
t/t9809-git-p4-client-view.sh
Starting with commit v1.7.8-165-g0579f91, grep.h includes
thread-utils.h, so the latter has to be added to LIB_H.
Signed-off-by: Dmitry V. Levin <ldv@altlinux.org>
Acked-by: Thomas Rast <trast@student.ethz.ch>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
This introduces a performance testing framework under t/perf/. It
tries to be as close to the test-lib.sh infrastructure as possible,
and thus should be easy to get used to for git developers.
The following points were considered for the implementation:
1. You usually want to compare arbitrary revisions/build trees against
each other. They may not have the performance test under
consideration, or even the perf-lib.sh infrastructure.
To cope with this, the 'run' script lets you specify arbitrary
build dirs and revisions. It even automatically builds the revisions
if it doesn't have them at hand yet.
2. Usually you would not want to run all tests. It would take too
long anyway. The 'run' script lets you specify which tests to run;
or you can also do it manually. There is a Makefile for
discoverability and 'make clean', but it is not meant for
real-world use.
3. Creating test repos from scratch in every test is extremely
time-consuming, and shipping or downloading such large/weird repos
is out of the question.
We leave this decision to the user. Two different sizes of test
repos can be configured, and the scripts just copy one or more of
those (using hardlinks for the object store). By default it tries
to use the build tree's git.git repository.
This is fairly fast and versatile. Using a copy instead of a clone
preserves many properties that the user may want to test for, such
as lots of loose objects, unpacked refs, etc.
Signed-off-by: Thomas Rast <trast@student.ethz.ch>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
The file 'po/git.pot' is generated using the command 'make pot'
against git v1.7.9-209-gb6b3b (Update draft release notes to 1.7.10).
Since po/git.pot is tracked, remove the entry from .gitignore, and
not delete the file again when doing 'make distclean'.
Signed-off-by: Jiang Xin <worldhello.net@gmail.com>
Ever since the very first commit to git.git we've been setting CC to
"gcc". Presumably this is behavior that Linus copied from the Linux
Makefile.
However unlike Linux Git is written in ANSI C and supports a multitude
of compilers, including Clang, Sun Studio, xlc etc. On my Linux box
"cc" is a symlink to clang, and on a Solaris box I have access to "cc"
is Sun Studio's CC.
Both of these are perfectly capable of compiling Git, and it's
annoying to have to specify CC=cc on the command-line when compiling
Git when that's the default behavior of most other portable programs.
So change the default to "cc". Users who want to compile with GCC can
still add "CC=gcc" to the make(1) command-line, but those users who
don't have GCC as their "cc" will see expected behavior, and as a
bonus we'll be more likely to smoke out new compilation warnings from
our distributors since they'll me using a more varied set of compilers
by default.
Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
On some systems, the function locale_charset() may not be exported from
libiconv but is available from libcharset, and we need -lcharset when
linking.
Introduce a make variable CHARSET_LIB that can be set to -lcharsetlib
on such systems. Also autodetect this in the configure script by first
looking for the symbol in libiconv, and then libcharset.
Signed-off-by: Дилян Палаузов <dilyan.palauzov@aegee.org>
$X is appended to binary names for Windows builds (ie. git.exe).
Pollution from the environment can inadvertently trigger this behaviour,
resulting in 'git' turning into 'gitwhatever' without warning.
Signed-off-by: Michael Palimaka <kensington@astralcloak.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
It is necessary to write the else branch as a nested conditional. Also,
write the conditions with parentheses because we use them throughout the
Makefile.
Signed-off-by: Johannes Sixt <j6t@kdbg.org>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
* jn/svn-fe: (36 commits)
vcs-svn: suppress a -Wtype-limits warning
vcs-svn: allow import of > 4GiB files
vcs-svn: rename check_overflow arguments for clarity
vcs-svn/svndiff.c: squelch false "unused" warning from gcc
vcs-svn: reset first_commit_done in fast_export_init
vcs-svn: do not initialize report_buffer twice
vcs-svn: avoid hangs from corrupt deltas
vcs-svn: guard against overflow when computing preimage length
vcs-svn: cap number of bytes read from sliding view
test-svn-fe: split off "test-svn-fe -d" into a separate function
vcs-svn: implement text-delta handling
vcs-svn: let deltas use data from preimage
vcs-svn: let deltas use data from postimage
vcs-svn: verify that deltas consume all inline data
vcs-svn: implement copyfrom_data delta instruction
vcs-svn: read instructions from deltas
vcs-svn: read inline data from deltas
vcs-svn: read the preimage when applying deltas
vcs-svn: parse svndiff0 window header
vcs-svn: skeleton of an svn delta parser
...
There was a number of problems I ran into when trying the
profile-directed optimizations added by Andi Kleen in git commit
7ddc2710b9. (This was using gcc 4.4 found on many enterprise
distros.)
1) The -fprofile-generate and -fprofile-use commands are incompatible
with ccache; the code ends up looking in the wrong place for the gcda
files based on the ccache object names.
2) If the makefile notices that CFLAGS are different, it will rebuild
all of the binaries. Hence the recipe originally specified by the
INSTALL file ("make profile-all" followed by "make install") doesn't
work. It will appear to work, but the binaries will end up getting
built with no optimization.
This patch fixes this by using an explicit set of options passed via
the PROFILE variable then using this to directly manipulate CFLAGS and
EXTLIBS.
The developer can run "make PROFILE=BUILD all ; sudo make
PROFILE=BUILD install" automatically run a two-pass build with the
test suite run in between as the sample workload for the purpose of
recording profiling information to do the profile-directed
optimization.
Alternatively, the profiling version of binaries can be built using:
make PROFILE=GEN PROFILE_DIR=/var/cache/profile all
make PROFILE=GEN install
and then after git has been used for a while, the optimized version of
the binary can be built as follows:
make PROFILE=USE PROFILE_DIR=/var/cache/profile all
make PROFILE=USE install
Signed-off-by: "Theodore Ts'o" <tytso@mit.edu>
Cc: Andi Kleen <ak@linux.intel.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
* ar/i18n-no-gettext:
i18n: Do not force USE_GETTEXT_SCHEME=fallthrough on NO_GETTEXT
i18n: Make NO_GETTEXT imply fallthrough scheme in shell l10n
add a Makefile switch to avoid gettext translation in shell scripts
git-sh-i18n: restructure the logic to compute gettext.sh scheme
This simplifies svn-fe a great deal and fulfills a longstanding wish:
support for dumps with deltas in them, and incremental imports.
The cost is that commandline usage of the svn-fe tool becomes a little
more complicated since it no longer keeps state itself but instead reads
blobs back from fast-import in order to copy them between revisions and
apply deltas to them.
Also removes a couple of custom data structures and replaces them with
strbufs like other parts of Git.
* 'svn-fe' of git://repo.or.cz/git/jrn: (32 commits)
vcs-svn: reset first_commit_done in fast_export_init
vcs-svn: do not initialize report_buffer twice
vcs-svn: avoid hangs from corrupt deltas
vcs-svn: guard against overflow when computing preimage length
vcs-svn: cap number of bytes read from sliding view
test-svn-fe: split off "test-svn-fe -d" into a separate function
vcs-svn: implement text-delta handling
vcs-svn: let deltas use data from preimage
vcs-svn: let deltas use data from postimage
vcs-svn: verify that deltas consume all inline data
vcs-svn: implement copyfrom_data delta instruction
vcs-svn: read instructions from deltas
vcs-svn: read inline data from deltas
vcs-svn: read the preimage when applying deltas
vcs-svn: parse svndiff0 window header
vcs-svn: skeleton of an svn delta parser
vcs-svn: make buffer_read_binary API more convenient
vcs-svn: learn to maintain a sliding view of a file
Makefile: list one vcs-svn/xdiff object or header per line
vcs-svn: avoid using ls command twice
...
Conflicts:
Makefile
contrib/svn-fe/svn-fe.txt
It should merely be the default used when the builder does not say
anything about USE_GETTEXT_SCHEME.
Even with NO_GETTEXT, USE_GETTEXT_SCHEME=gnu may be a way to avoid
possibly slower emulation in our shell scripts.
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Some systems have gettext.sh (GNU gettext) installed, but it is either
broken or misconfigured in such a way so its output is not usable. In
case the users of these systems are unable or not interested in fixing
them, setting the new Makefile switch should help:
make USE_GETTEXT_SCHEME=fallthrough
This will replace the translation routines with fallthrough versions,
that does not use gettext from the platform.
Signed-off-by: Alex Riesen <raa.lkml@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
* jk/git-prompt:
contrib: add credential helper for OS X Keychain
Makefile: OS X has /dev/tty
Makefile: linux has /dev/tty
credential: use git_prompt instead of git_getpass
prompt: use git_terminal_prompt
add generic terminal prompt function
refactor git_getpass into generic prompt function
move git_getpass to its own source file
imap-send: don't check return value of git_getpass
imap-send: avoid buffer overflow
Conflicts:
Makefile
* tr/cache-tree:
reset: update cache-tree data when appropriate
commit: write cache-tree data when writing index anyway
Refactor cache_tree_update idiom from commit
Test the current state of the cache-tree optimization
Add test-scrap-cache-tree
* jk/credentials:
t: add test harness for external credential helpers
credentials: add "store" helper
strbuf: add strbuf_add*_urlencode
Makefile: unix sockets may not available on some platforms
credentials: add "cache" helper
docs: end-user documentation for the credential subsystem
credential: make relevance of http path configurable
credential: add credential.*.username
credential: apply helper config
http: use credential API to get passwords
credential: add function for parsing url components
introduce credentials API
t5550: fix typo
test-lib: add test_config_global variant
Conflicts:
strbuf.c
* jc/stream-to-pack:
bulk-checkin: replace fast-import based implementation
csum-file: introduce sha1file_checkpoint
finish_tmp_packfile(): a helper function
create_tmp_packfile(): a helper function
write_pack_header(): a helper function
Conflicts:
pack.h
Therefore we can turn on our custom prompt function instead
of relying on getpass.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
When we need to prompt the user for input interactively, we
want to access their terminal directly. We can't rely on
stdio because it may be connected to pipes or files, rather
than the terminal. Instead, we use "getpass()", because it
abstracts the idea of prompting and reading from the
terminal. However, it has some problems:
1. It never echoes the typed characters, which makes it OK
for passwords but annoying for other input (like usernames).
2. Some implementations of getpass() have an extremely
small input buffer (e.g., Solaris 8 is reported to
support only 8 characters).
3. Some implementations of getpass() will fall back to
reading from stdin (e.g., glibc). We explicitly don't
want this, because our stdin may be connected to a pipe
speaking a particular protocol, and reading will
disrupt the protocol flow (e.g., the remote-curl
helper).
4. Some implementations of getpass() turn off signals, so
that hitting "^C" on the terminal does not break out of
the password prompt. This can be a mild annoyance.
Instead, let's provide an abstract "git_terminal_prompt"
function that addresses these concerns. This patch includes
an implementation based on /dev/tty, enabled by setting
HAVE_DEV_TTY. The fallback is to use getpass() as before.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
This is like "cache", except that we actually put the
credentials on disk. This can be terribly insecure, of
course, but we do what we can to protect them by filesystem
permissions, and we warn the user in the documentation.
This is not unlike using .netrc to store entries, but it's a
little more user-friendly. Instead of putting credentials in
place ahead of time, we transparently store them after
prompting the user for them once.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
This is currently in connect.c, but really has nothing to
do with the git protocol itself. Let's make a new source
file all about prompting the user, which will make it
cleaner to refactor.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Introduce a configuration option NO_UNIX_SOCKETS to exclude code that
depends on Unix sockets and use it in MSVC and MinGW builds.
Notice that unix-socket.h was missing from LIB_H before; fix that, too.
Signed-off-by: Johannes Sixt <j6t@kdbg.org>
Helped-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
If you access repositories over smart-http using http
authentication, then it can be annoying to have git ask you
for your password repeatedly. We cache credentials in
memory, of course, but git is composed of many small
programs. Having to input your password for each one can be
frustrating.
This patch introduces a credential helper that will cache
passwords in memory for a short period of time.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
There are a few places in git that need to get a username
and password credential from the user; the most notable one
is HTTP authentication for smart-http pushing.
Right now the only choices for providing credentials are to
put them plaintext into your ~/.netrc, or to have git prompt
you (either on the terminal or via an askpass program). The
former is not very secure, and the latter is not very
convenient.
Unfortunately, there is no "always best" solution for
password management. The details will depend on the tradeoff
you want between security and convenience, as well as how
git can integrate with other security systems (e.g., many
operating systems provide a keychain or password wallet for
single sign-on).
This patch provides an abstract notion of credentials as a
data item, and provides three basic operations:
- fill (i.e., acquire from external storage or from the
user)
- approve (mark a credential as "working" for further
storage)
- reject (mark a credential as "not working", so it can
be removed from storage)
These operations can be backed by external helper processes
that interact with system- or user-specific secure storage.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
* jc/pull-signed-tag:
commit-tree: teach -m/-F options to read logs from elsewhere
commit-tree: update the command line parsing
commit: teach --amend to carry forward extra headers
merge: force edit and no-ff mode when merging a tag object
commit: copy merged signed tags to headers of merge commit
merge: record tag objects without peeling in MERGE_HEAD
merge: make usage of commit->util more extensible
fmt-merge-msg: Add contents of merged tag in the merge message
fmt-merge-msg: package options into a structure
fmt-merge-msg: avoid early returns
refs DWIMmery: use the same rule for both "git fetch" and others
fetch: allow "git fetch $there v1.0" to fetch a tag
merge: notice local merging of tags and keep it unwrapped
fetch: do not store peeled tag object names in FETCH_HEAD
Split GPG interface into its own helper library
Conflicts:
builtin/fmt-merge-msg.c
builtin/merge.c
* jc/request-pull-show-head-4:
request-pull: use the annotated tag contents
fmt-merge-msg.c: Fix an "dubious one-bit signed bitfield" sparse error
environment.c: Fix an sparse "symbol not declared" warning
builtin/log.c: Fix an "Using plain integer as NULL pointer" warning
fmt-merge-msg: use branch.$name.description
request-pull: use the branch description
request-pull: state what commit to expect
request-pull: modernize style
branch: teach --edit-description option
format-patch: use branch description in cover letter
branch: add read_branch_desc() helper function
Conflicts:
builtin/branch.c
A simple utility that invalidates all existing cache-tree data. We
need this for tests. (We don't need a tool to rebuild the cache-tree
data; git read-tree HEAD works for that.)
Signed-off-by: Thomas Rast <trast@student.ethz.ch>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Change the skeleton implementation of i18n in Git to one that can show
localized strings to users for our C, Shell and Perl programs using
either GNU libintl or the Solaris gettext implementation.
This new internationalization support is enabled by default. If
gettext isn't available, or if Git is compiled with
NO_GETTEXT=YesPlease, Git falls back on its current behavior of
showing interface messages in English. When using the autoconf script
we'll auto-detect if the gettext libraries are installed and act
appropriately.
This change is somewhat large because as well as adding a C, Shell and
Perl i18n interface we're adding a lot of tests for them, and for
those tests to work we need a skeleton PO file to actually test
translations. A minimal Icelandic translation is included for this
purpose. Icelandic includes multi-byte characters which makes it easy
to test various edge cases, and it's a language I happen to
understand.
The rest of the commit message goes into detail about various
sub-parts of this commit.
= Installation
Gettext .mo files will be installed and looked for in the standard
$(prefix)/share/locale path. GIT_TEXTDOMAINDIR can also be set to
override that, but that's only intended to be used to test Git itself.
= Perl
Perl code that's to be localized should use the new Git::I18n
module. It imports a __ function into the caller's package by default.
Instead of using the high level Locale::TextDomain interface I've
opted to use the low-level (equivalent to the C interface)
Locale::Messages module, which Locale::TextDomain itself uses.
Locale::TextDomain does a lot of redundant work we don't need, and
some of it would potentially introduce bugs. It tries to set the
$TEXTDOMAIN based on package of the caller, and has its own
hardcoded paths where it'll search for messages.
I found it easier just to completely avoid it rather than try to
circumvent its behavior. In any case, this is an issue wholly
internal Git::I18N. Its guts can be changed later if that's deemed
necessary.
See <AANLkTilYD_NyIZMyj9dHtVk-ylVBfvyxpCC7982LWnVd@mail.gmail.com> for
a further elaboration on this topic.
= Shell
Shell code that's to be localized should use the git-sh-i18n
library. It's basically just a wrapper for the system's gettext.sh.
If gettext.sh isn't available we'll fall back on gettext(1) if it's
available. The latter is available without the former on Solaris,
which has its own non-GNU gettext implementation. We also need to
emulate eval_gettext() there.
If neither are present we'll use a dumb printf(1) fall-through
wrapper.
= About libcharset.h and langinfo.h
We use libcharset to query the character set of the current locale if
it's available. I.e. we'll use it instead of nl_langinfo if
HAVE_LIBCHARSET_H is set.
The GNU gettext manual recommends using langinfo.h's
nl_langinfo(CODESET) to acquire the current character set, but on
systems that have libcharset.h's locale_charset() using the latter is
either saner, or the only option on those systems.
GNU and Solaris have a nl_langinfo(CODESET), FreeBSD can use either,
but MinGW and some others need to use libcharset.h's locale_charset()
instead.
=Credits
This patch is based on work by Jeff Epler <jepler@unpythonic.net> who
did the initial Makefile / C work, and a lot of comments from the Git
mailing list, including Jonathan Nieder, Jakub Narebski, Johannes
Sixt, Erik Faye-Lund, Peter Krefting, Junio C Hamano, Thomas Rast and
others.
[jc: squashed a small Makefile fix from Ramsay]
Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Signed-off-by: Ramsay Jones <ramsay@ramsay1.demon.co.uk>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
This extends the earlier approach to stream a large file directly from the
filesystem to its own packfile, and allows "git add" to send large files
directly into a single pack. Older code used to spawn fast-import, but the
new bulk-checkin API replaces it.
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Now that the COMPUTE_HEADER_DEPENDENCIES feature is turned on
automatically for compilers that support it (see v1.7.8-rc0~142^2~1,
2011-08-18), there is no easy way to force it off. For example,
setting COMPUTE_HEADER_DEPENDENCIES to the empty string in config.mak
just tells the makefile to treat it as undefined and run a test
command to see if the -MMD option is supported.
So allow setting COMPUTE_HEADER_DEPENDENCIES=no to explicitly force
the feature off. The new semantics:
- "yes" means to explicitly enable the feature
- "no" means to disable it
- "auto" means to autodetect
The default is still "auto". Any value other than these three will
cause the build to error out with a descriptive message so typos and
stale settings in config.mak don't result in mysterious behavior.
Makefile:1278: *** please set COMPUTE_HEADER_DEPENDENCIES to
yes, no, or auto (not "1"). Stop.
So now when someone using a compiler without -MMD support reports
trouble building git, you can reproduce it by running "make
COMPUTE_HEADER_DEPENDENCIES=no".
Suggested-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
Improved-by: Junio C Hamano <gitster@pobox.com>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
Tested-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
When the streaming filter API was introduced in v1.7.7-rc0~60^2~7
(2011-05-20), we forgot to add its header to LIB_H. Most translation
units depend on streaming.h via cache.h.
v1.7.5-rc0~48 (Fix sparse warnings, 2011-03-22) introduced undeclared
dependencies by url.o on url.h and thread-utils.o on thread-utils.h.
Noticed by make CHECK_HEADER_DEPENDENCIES=1.
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
This mostly moves existing code from builtin/tag.c (for signing)
and builtin/verify-tag.c (for verifying) to a new gpg-interface.c
file to provide a more generic library interface.
- sign_buffer() takes a payload strbuf, a signature strbuf, and a signing
key, runs "gpg" to produce a detached signature for the payload, and
appends it to the signature strbuf. The contents of a signed tag that
concatenates the payload and the detached signature can be produced by
giving the same strbuf as payload and signature strbuf.
- verify_signed_buffer() takes a payload and a detached signature as
<ptr, len> pairs, and runs "gpg --verify" to see if the payload matches
the signature. It can optionally capture the output from GPG to allow
the callers to pretty-print it in a way more suitable for their
contexts.
"verify-tag" (aka "tag -v") used to save the whole tag contents as if it
is a detached signature, and fed gpg the payload part of the tag. It
relied on gpg to fail when the given tag is not signed but just is
annotated. The updated run_gpg_verify() function detects the lack of
detached signature in the input, and errors out without bothering "gpg".
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Since systems that omit strtoumax() will likely omit strtomax() too, and
likewise for strtoull() and strtoll(), we arrange for the make variables
NO_STRTOUMAX and NO_STRTOULL to cover both the signed and unsigned
functions, and define compatibility implementations for them.
Signed-off-by: Nick Alcock <nix@esperi.org.uk>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Earlier we moved this header file in the code but forgot to
update the Makefile that refers to it.
Signed-off-by: Erik Faye-Lund <kusmabite@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Both XSI and upstream Gnulib versions expects to find poll.h at
the root of some include path, not inside the sys-folder.
This helps us when upgrading Gnulib.
Signed-off-by: Erik Faye-Lund <kusmabite@gmail.com>
Acked-by: Johannes Sixt <j6t@kdbg.org>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
* jk/argv-array:
run_hook: use argv_array API
checkout: use argv_array API
bisect: use argv_array API
quote: provide sq_dequote_to_argv_array
refactor argv_array into generic code
quote.h: fix bogus comment
add sha1_array API docs
The [ce]tags and cscope targets used to run "find" looking for any paths
that match '*.[chS]' to feed the list of source files to downstream xargs.
Use "git ls-files" if it is already available to us, and otherwise use a
tighter "find" expression that does not list directories and does not go
into our .git directory.
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Ever since mergetool--lib was split into multiple files in
v1.7.7-rc0~3^2~1 (2011-08-18), the Makefile takes care to reset umask
and use tar --no-owner when installing merge tool definitions to
$(gitexecdir)/mergetools/. Unfortunately it does not take into
account the possibility that the permission bits of the files being
copied might already be wrong.
Rather than fixing the "tar" incantation and making it even more
complicated, let's just use the "install" utility. This only means
losing the ability to install executables and subdirectories of
mergetools/, which wasn't used.
Noticed by installing from a copy of git checked out with umask 002.
Compare v1.6.0.3~81^2 (Fix permission bits on sources checked out with
an overtight umask, 2008-08-21).
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
This teaches "merge --log" and fmt-merge-msg to use branch description
information when merging a local topic branch into the mainline. The
description goes between the branch name label and the list of commit
titles.
The refactoring to share the common configuration parsing between
merge and fmt-merge-msg needs to be made into a separate patch.
Signed-off-by: Junio C Hamano <gitster@pobox.com>
* jk/argv-array:
run_hook: use argv_array API
checkout: use argv_array API
bisect: use argv_array API
quote: provide sq_dequote_to_argv_array
refactor argv_array into generic code
quote.h: fix bogus comment
add sha1_array API docs
* jc/receive-verify:
receive-pack: check connectivity before concluding "git push"
check_everything_connected(): libify
check_everything_connected(): refactor to use an iterator
fetch: verify we have everything we need before updating our ref
Conflicts:
builtin/fetch.c
* rr/revert-cherry-pick-continue:
builtin/revert.c: make commit_list_append() static
revert: Propagate errors upwards from do_pick_commit
revert: Introduce --continue to continue the operation
revert: Don't implicitly stomp pending sequencer operation
revert: Remove sequencer state when no commits are pending
reset: Make reset remove the sequencer state
revert: Introduce --reset to remove sequencer state
revert: Make pick_commits functionally act on a commit list
revert: Save command-line options for continuing operation
revert: Save data for continuing after conflict resolution
revert: Don't create invalid replay_opts in parse_args
revert: Separate cmdline parsing from functional code
revert: Introduce struct to keep command-line options
revert: Eliminate global "commit" variable
revert: Rename no_replay to record_origin
revert: Don't check lone argument in get_encoding
revert: Simplify and inline add_message_to_msg
config: Introduce functions to write non-standard file
advice: Introduce error_resolve_conflict
The g+s bit on directories to make group ownership inherited is a
SysVism --- BSD and most of its descendants do not need it since they
do the sane thing by default without g+s. In fact, on some
filesystems (but not all --- tmpfs works this way but UFS does not),
the kernel of FreeBSD does not even allow non-root users to set setgid
bit on directories and produces errors when one tries:
$ git init --shared dir
fatal: Could not make /tmp/dir/.git/refs writable by group
Since the setgid bit would only mean "do what you were going to do
already", it's better to avoid setting it. Accordingly, ever since
v1.5.5-rc0~59^2 (Do not use GUID on dir in git init --share=all on
FreeBSD, 2008-03-05), git on true FreeBSD has done exactly that. Set
DIR_HAS_BSD_GROUP_SEMANTICS in the makefile for GNU/kFreeBSD, too, so
machines that use glibc with the kernel of FreeBSD get the same fix.
This fixes t0001-init.sh and t1301-shared-repo.sh on GNU/kFreeBSD
when running tests with --root pointing to a directory that uses
tmpfs.
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
The submodule code recently grew generic code to build a
dynamic argv array. Many other parts of the code can reuse
this, too, so let's make it generically available.
There are two enhancements not found in the original code:
1. We now handle the NULL-termination invariant properly,
even when no strings have been pushed (before, you
could have an empty, NULL argv). This was not a problem
for the submodule code, which always pushed at least
one argument, but was not sufficiently safe for
generic code.
2. There is a formatted variant of the "push" function.
This is a convenience function which was not needed by
the submodule code, but will make it easier to port
other users to the new code.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Extract the helper function and the type definition of the iterator
function it uses out of builtin/fetch.c into a separate source and a
header file.
Signed-off-by: Junio C Hamano <gitster@pobox.com>
* fk/use-kwset-pickaxe-grep-f:
obstack: Fix portability issues
Use kwset in grep
Use kwset in pickaxe
Adapt the kwset code to Git
Add string search routines from GNU grep
Add obstack.[ch] from EGLIBC 2.10
The Makefile enables CHECK_HEADER_DEPENDENCIES when the
compiler supports generating header dependencies.
Make the check use the same flags as the invocation
to avoid a false positive when user-configured compiler
flags contain incompatible options.
For example, without this patch, trying to build universal
binaries on a Mac using CFLAGS='-arch i386 -arch x86_64'
produces:
gcc-4.2: -E, -S, -save-temps and -M options are
not allowed with multiple -arch flags
While at it, remove "sh -c" in the command passed to $(shell);
at this point in the Makefile, SHELL has already been set to
a sensible shell and it is better not to override that.
Signed-off-by: David Aguilar <davvid@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
* da/difftool-mergtool-refactor:
mergetools/meld: Use '--output' when available
mergetool--lib: Refactor tools into separate files
mergetool--lib: Make style consistent with git
difftool--helper: Make style consistent with git
