1
0
Fork 0
mirror of https://github.com/git/git.git synced 2024-10-31 22:37:54 +01:00
Commit graph

41529 commits

Author SHA1 Message Date
Stefan Beller
c553c72eed run-command: add an asynchronous parallel child processor
This allows to run external commands in parallel with ordered output
on stderr.

If we run external commands in parallel we cannot pipe the output directly
to the our stdout/err as it would mix up. So each process's output will
flow through a pipe, which we buffer. One subprocess can be directly
piped to out stdout/err for a low latency feedback to the user.

Example:
Let's assume we have 5 submodules A,B,C,D,E and each fetch takes a
different amount of time as the different submodules vary in size, then
the output of fetches in sequential order might look like this:

 time -->
 output: |---A---| |-B-| |-------C-------| |-D-| |-E-|

When we schedule these submodules into maximal two parallel processes,
a schedule and sample output over time may look like this:

process 1: |---A---| |-D-| |-E-|

process 2: |-B-| |-------C-------|

output:    |---A---|B|---C-------|DE

So A will be perceived as it would run normally in the single child
version. As B has finished by the time A is done, we can dump its whole
progress buffer on stderr, such that it looks like it finished in no
time. Once that is done, C is determined to be the visible child and
its progress will be reported in real time.

So this way of output is really good for human consumption, as it only
changes the timing, not the actual output.

For machine consumption the output needs to be prepared in the tasks,
by either having a prefix per line or per block to indicate whose tasks
output is displayed, because the output order may not follow the
original sequential ordering:

 |----A----| |--B--| |-C-|

will be scheduled to be all parallel:

process 1: |----A----|
process 2: |--B--|
process 3: |-C-|
output:    |----A----|CB

This happens because C finished before B did, so it will be queued for
output before B.

To detect when a child has finished executing, we check interleaved
with other actions (such as checking the liveliness of children or
starting new processes) whether the stderr pipe still exists. Once a
child closed its stderr stream, we assume it is terminating very soon,
and use `finish_command()` from the single external process execution
interface to collect the exit status.

By maintaining the strong assumption of stderr being open until the
very end of a child process, we can avoid other hassle such as an
implementation using `waitpid(-1)`, which is not implemented in Windows.

Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2015-12-16 12:06:08 -08:00
Stefan Beller
bfb6b53c05 sigchain: add command to pop all common signals
The new method removes all common signal handlers that were installed
by sigchain_push.

CC: Jeff King <peff@peff.net>
Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2015-12-16 12:06:08 -08:00
Stefan Beller
b4e04fb66e strbuf: add strbuf_read_once to read without blocking
The new call will read from a file descriptor into a strbuf once. The
underlying call xread is just run once. xread only reattempts
reading in case of EINTR, which makes it suitable to use for a
nonblocking read.

Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2015-12-16 12:06:08 -08:00
Stefan Beller
1079c4be0b xread: poll on non blocking fds
The man page of read(2) says:

  EAGAIN The file descriptor fd refers to a file other than a socket
	 and has been marked nonblocking (O_NONBLOCK), and the read
	 would block.

  EAGAIN or EWOULDBLOCK
	 The file descriptor fd refers to a socket and has been marked
	 nonblocking (O_NONBLOCK), and the read would block.  POSIX.1-2001
	 allows either error to be returned for this case, and does not
	 require these constants to have the same value, so a portable
	 application should check for both possibilities.

If we get an EAGAIN or EWOULDBLOCK the fd must have set O_NONBLOCK.
As the intent of xread is to read as much as possible either until the
fd is EOF or an actual error occurs, we can ease the feeder of the fd
by not spinning the whole time, but rather wait for it politely by not
busy waiting.

We should not care if the call to poll failed, as we're in an infinite
loop and can only get out with the correct read().

Signed-off-by: Stefan Beller <sbeller@google.com>
Acked-by: Johannes Sixt <j6t@kdbg.org>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2015-12-16 12:06:08 -08:00
Jonathan Nieder
fbf71645d1 submodule.c: write "Fetching submodule <foo>" to stderr
The "Pushing submodule <foo>" progress output correctly goes to
stderr, but "Fetching submodule <foo>" is going to stdout by
mistake.  Fix it to write to stderr.

Noticed while trying to implement a parallel submodule fetch.  When
this particular output line went to a different file descriptor, it
was buffered separately, resulting in wrongly interleaved output if
we copied it to the terminal naively.

Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2015-12-16 12:06:08 -08:00
Junio C Hamano
4b9ab0ee01 Update release notes to 2.7
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2015-12-11 11:20:23 -08:00
Junio C Hamano
52b2e6be99 Merge branch 'maint'
* maint:
  Prepare for 2.6.5
2015-12-11 11:19:43 -08:00
Junio C Hamano
49e863b02a Prepare for 2.6.5
This back-merges hopefully the last batch of trivially correct fixes
to the 2.6.x maintenance track from the master branch.

Signed-off-by: Junio C Hamano <gitster@pobox.com>
2015-12-11 11:17:47 -08:00
Junio C Hamano
58e3dd21f6 Merge branch 'sn/null-pointer-arith-in-mark-tree-uninteresting' into maint
mark_tree_uninteresting() has code to handle the case where it gets
passed a NULL pointer in its 'tree' parameter, but the function had
'object = &tree->object' assignment before checking if tree is
NULL.  This gives a compiler an excuse to declare that tree will
never be NULL and apply a wrong optimization.  Avoid it.

* sn/null-pointer-arith-in-mark-tree-uninteresting:
  revision.c: fix possible null pointer arithmetic
2015-12-11 11:14:38 -08:00
Junio C Hamano
abca668a93 Merge branch 'sg/lock-file-commit-error' into maint
Cosmetic improvement to lock-file error messages.

* sg/lock-file-commit-error:
  Make error message after failing commit_lock_file() less confusing
2015-12-11 11:14:18 -08:00
Junio C Hamano
76058817e8 Merge branch 'cb/t3404-shellquote' into maint
* cb/t3404-shellquote:
  t3404: fix quoting of redirect for some versions of bash
2015-12-11 11:14:18 -08:00
Junio C Hamano
17e5bcab71 Merge branch 'sb/doc-submodule-sync-recursive' into maint
* sb/doc-submodule-sync-recursive:
  document submodule sync --recursive
2015-12-11 11:14:17 -08:00
Junio C Hamano
63b3db71d8 Merge branch 'nd/doc-check-ref-format-typo' into maint
* nd/doc-check-ref-format-typo:
  git-check-ref-format.txt: typo, s/avoids/avoid/
2015-12-11 11:14:15 -08:00
Junio C Hamano
288fe0cfb6 Merge branch 'rs/show-branch-argv-array' into maint
Code simplification.

* rs/show-branch-argv-array:
  show-branch: use argv_array for default arguments
2015-12-11 11:14:14 -08:00
Junio C Hamano
0af22d6fff Merge branch 'rs/pop-commit' into maint
Code simplification.

* rs/pop-commit:
  use pop_commit() for consuming the first entry of a struct commit_list
2015-12-11 11:14:13 -08:00
Junio C Hamano
8c0a546670 Merge branch 'as/subtree-with-spaces' into maint
Update "git subtree" (in contrib/) so that it can take whitespaces
in the pathnames, not only in the in-tree pathname but the name of
the directory that the repository is in.

* as/subtree-with-spaces:
  contrib/subtree: respect spaces in a repository path
  t7900-subtree: test the "space in a subdirectory name" case
2015-12-11 11:14:11 -08:00
Junio C Hamano
4cb5488fa6 Merge branch 'jk/test-lint-forbid-when-finished-in-subshell' into maint
Because "test_when_finished" in our test framework queues the
clean-up tasks to be done in a shell variable, it should not be
used inside a subshell.  Add a mechanism to allow 'bash' to catch
such uses, and fix the ones that were found.

* jk/test-lint-forbid-when-finished-in-subshell:
  test-lib-functions: detect test_when_finished in subshell
  t7800: don't use test_config in a subshell
  test-lib-functions: support "test_config -C <dir> ..."
  t5801: don't use test_when_finished in a subshell
  t7610: don't use test_config in a subshell
2015-12-11 11:14:10 -08:00
Junio C Hamano
782ca8c44e Merge branch 'sn/null-pointer-arith-in-mark-tree-uninteresting'
mark_tree_uninteresting() has code to handle the case where it gets
passed a NULL pointer in its 'tree' parameter, but the function had
'object = &tree->object' assignment before checking if tree is
NULL.  This gives a compiler an excuse to declare that tree will
never be NULL and apply a wrong optimization.  Avoid it.

* sn/null-pointer-arith-in-mark-tree-uninteresting:
  revision.c: fix possible null pointer arithmetic
2015-12-11 10:41:01 -08:00
Junio C Hamano
fa41b05253 Merge branch 'sb/doc-submodule-sync-recursive'
* sb/doc-submodule-sync-recursive:
  document submodule sync --recursive
2015-12-11 10:41:00 -08:00
Junio C Hamano
c87eec9784 Merge branch 'cb/t3404-shellquote'
* cb/t3404-shellquote:
  t3404: fix quoting of redirect for some versions of bash
2015-12-11 10:40:58 -08:00
Junio C Hamano
e0048d3e0d Merge branch 'sg/lock-file-commit-error'
Cosmetic improvement to lock-file error messages.

* sg/lock-file-commit-error:
  Make error message after failing commit_lock_file() less confusing
2015-12-11 10:40:55 -08:00
Junio C Hamano
7d722536dd Git 2.7-rc0
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2015-12-10 12:59:37 -08:00
Junio C Hamano
39e07f77b6 Sync with maint
* maint:
  Documentation/git-update-index: add missing opts to synopsis
2015-12-10 12:45:17 -08:00
Junio C Hamano
86c95ac5d2 Update release notes to 2.7
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2015-12-10 12:43:52 -08:00
Junio C Hamano
56d63d0eab Merge branch 'nd/doc-check-ref-format-typo'
* nd/doc-check-ref-format-typo:
  git-check-ref-format.txt: typo, s/avoids/avoid/
2015-12-10 12:36:15 -08:00
Junio C Hamano
844a9ce472 Merge branch 'bc/object-id'
More transition from "unsigned char[40]" to "struct object_id".

This needed a few merge fixups, but is mostly disentangled from other
topics.

* bc/object-id:
  remote: convert functions to struct object_id
  Remove get_object_hash.
  Convert struct object to object_id
  Add several uses of get_object_hash.
  object: introduce get_object_hash macro.
  ref_newer: convert to use struct object_id
  push_refs_with_export: convert to struct object_id
  get_remote_heads: convert to struct object_id
  parse_fetch: convert to use struct object_id
  add_sought_entry_mem: convert to struct object_id
  Convert struct ref to use object_id.
  sha1_file: introduce has_object_file helper.
2015-12-10 12:36:13 -08:00
Junio C Hamano
b12a966eff Merge branch 'dt/fsck-verify-pack-error'
The exit code of git-fsck didnot reflect some types of errors found
in packed objects, which has been corrected.

* dt/fsck-verify-pack-error:
  verify_pack: do not ignore return value of verification function
2015-12-10 12:36:12 -08:00
Junio C Hamano
9eb2449c35 Merge branch 'ls/travis-yaml'
The necessary infrastructure to build topics using the free Travis
CI has been added. Developers forking from this topic (and enabling
Travis) can do their own builds, and we can turn on auto-builds for
git/git (including build-status for pull requests that people
open).

* ls/travis-yaml:
  Add Travis CI support
2015-12-10 12:36:12 -08:00
Christian Couder
bc49712789 Documentation/git-update-index: add missing opts to synopsis
Split index related options should appear in the 'SYNOPSIS'
section.

These options are already documented in the 'OPTIONS' section.

Signed-off-by: Christian Couder <chriscool@tuxfamily.org>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2015-12-10 12:21:26 -08:00
Junio C Hamano
545299f822 Merge branch 'ep/ident-with-getaddrinfo'
A build without NO_IPv6 used to use gethostbyname() when guessing
user's hostname, instead of getaddrinfo() that is used in other
codepaths in such a build.

* ep/ident-with-getaddrinfo:
  ident.c: add support for IPv6
2015-12-08 14:14:50 -08:00
Junio C Hamano
2b597f3307 Merge branch 'ls/test-must-fail-sigpipe'
Fix some racy client/server tests by treating SIGPIPE the same as a
normal non-zero exit.

* ls/test-must-fail-sigpipe:
  add "ok=sigpipe" to test_must_fail and use it to fix flaky tests
  implement test_might_fail using a refactored test_must_fail
2015-12-08 14:14:49 -08:00
Junio C Hamano
b1cda70fff Merge branch 'dt/refs-backend-pre-vtable'
Code preparation for pluggable ref backends.

* dt/refs-backend-pre-vtable:
  refs: break out ref conflict checks
  files_log_ref_write: new function
  initdb: make safe_create_dir public
  refs: split filesystem-based refs code into a new file
  refs/refs-internal.h: new header file
  refname_is_safe(): improve docstring
  pack_if_possible_fn(): use ref_type() instead of is_per_worktree_ref()
  copy_msg(): rename to copy_reflog_msg()
  verify_refname_available(): new function
  verify_refname_available(): rename function
2015-12-08 14:14:49 -08:00
Junio C Hamano
fa5f2398e5 Sync with 2.6.4 2015-12-08 14:13:52 -08:00
Junio C Hamano
bdfc6b364a Git 2.6.4
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2015-12-08 14:12:45 -08:00
Junio C Hamano
4d0069f277 Merge branch 'rs/status-detached-head-memcmp' into maint
Fix some string-matching corner cases when digging in the reflog for
"git status".

* rs/status-detached-head-memcmp:
  wt-status: correct and simplify check for detached HEAD
2015-12-08 14:11:32 -08:00
Junio C Hamano
b5d2d8eef0 Merge branch 'ad/sha1-update-chunked' into maint
Apple's common crypto implementation of SHA1_Update() does not take
more than 4GB at a time, and we now have a compile-time workaround
for it.

* ad/sha1-update-chunked:
  sha1: allow limiting the size of the data passed to SHA1_Update()
  sha1: provide another level of indirection for the SHA-1 functions
2015-12-08 14:05:03 -08:00
Junio C Hamano
e6ed5a438c Merge branch 'sg/bash-prompt-dirty-orphan' into maint
Produce correct "dirty" marker for shell prompts, even when we
are on an orphan or an unborn branch.

* sg/bash-prompt-dirty-orphan:
  bash prompt: indicate dirty index even on orphan branches
  bash prompt: remove a redundant 'git diff' option
  bash prompt: test dirty index and worktree while on an orphan branch
2015-12-08 14:05:02 -08:00
Junio C Hamano
22386ad6ee Merge branch 'jk/interpret-trailers-outside-a-repository' into maint
Allow "git interpret-trailers" to run outside of a Git repository.

* jk/interpret-trailers-outside-a-repository:
  interpret-trailers: allow running outside a repository
2015-12-08 14:05:02 -08:00
Junio C Hamano
697bd2871c Merge branch 'jk/rebase-no-autostash' into maint
There was no way to defeat a configured rebase.autostash variable
from the command line, as "git rebase --no-autostash" was missing.

* jk/rebase-no-autostash:
  Documentation/git-rebase: fix --no-autostash formatting
  rebase: support --no-autostash
2015-12-08 14:05:01 -08:00
Stefan Naewe
a2678df335 revision.c: fix possible null pointer arithmetic
mark_tree_uninteresting() dereferences a tree pointer before
checking if the pointer is valid. Fix that by doing the check first.

Signed-off-by: Stefan Naewe <stefan.naewe@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2015-12-07 12:32:02 -08:00
Charles Bailey
7966230b7d t3404: fix quoting of redirect for some versions of bash
As CodingGuidelines says, some versions of bash errors out when
$variable substitution is used as the target for redirection without
being quoted (even though POSIX may not require such a quote).

Signed-off-by: Charles Bailey <cbailey32@bloomberg.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2015-12-04 14:05:18 -08:00
Nguyễn Thái Ngọc Duy
56a8aea0c6 git-check-ref-format.txt: typo, s/avoids/avoid/
Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2015-12-04 13:09:48 -08:00
Stefan Beller
e7220c40b1 document submodule sync --recursive
The git-submodule(1) is inconsistent. In the synopsis, it says:

       git submodule [--quiet] sync [--recursive] [--] [<path>...]

The description of the sync does not mention --recursive, and the
description of --recursive says that it is only available for foreach,
update and status.

The option was introduced (82f49f294c, Teach --recursive to submodule
sync, 2012-10-26) a while ago, so let's document it, too.

Reported-by: Per Cederqvist <cederp@opera.com>
Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2015-12-04 13:08:05 -08:00
Junio C Hamano
e5da8655b2 Sync with maint
* maint:
  Prepare for 2.6.4
2015-12-04 11:39:56 -08:00
Junio C Hamano
9a8c740225 Prepare for 2.6.4
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2015-12-04 11:39:19 -08:00
Junio C Hamano
aa0b4c31d9 Merge branch 'da/difftool' into maint
The code to prepare the working tree side of temporary directory
for the "dir-diff" feature forgot that symbolic links need not be
copied (or symlinked) to the temporary area, as the code already
special cases and overwrites them.  Besides, it was wrong to try
computing the object name of the target of symbolic link, which may
not even exist or may be a directory.

* da/difftool:
  difftool: ignore symbolic links in use_wt_file
2015-12-04 11:34:24 -08:00
Junio C Hamano
b50ceab48f Merge branch 'dk/gc-idx-wo-pack' into maint
Having a leftover .idx file without corresponding .pack file in
the repository hurts performance; "git gc" learned to prune them.

We may want to do the same for .bitmap (and notice but not prune
.keep) without corresponding .pack, but that can be a separate
topic.

* dk/gc-idx-wo-pack:
  gc: remove garbage .idx files from pack dir
  t5304: test cleaning pack garbage
  prepare_packed_git(): refactor garbage reporting in pack directory
2015-12-04 11:33:08 -08:00
Junio C Hamano
9ed86a5d4e RelNotes update for 2.7
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2015-12-04 11:31:28 -08:00
Junio C Hamano
80c17cac36 Merge branch 'sg/bash-prompt-dirty-orphan'
Produce correct "dirty" marker for shell prompts, even when we
are on an orphan or an unborn branch.

* sg/bash-prompt-dirty-orphan:
  bash prompt: indicate dirty index even on orphan branches
  bash prompt: remove a redundant 'git diff' option
  bash prompt: test dirty index and worktree while on an orphan branch
2015-12-04 11:19:12 -08:00
Junio C Hamano
3a5b6eeceb Merge branch 'np/credential-cache-sighup'
Workaround for using credential-cache with emacs.

* np/credential-cache-sighup:
  credential-cache: new option to ignore sighup
2015-12-04 11:19:11 -08:00