When running large git grep (ie: git grep regexp $(git rev-list --all)), glibc error sometimes occur:
*** glibc detected *** git: double free or corruption (!prev): 0x00000000010abdf0 ***
According to gdb the problem originate from release_delta_cash (sha1_file.c:1703)
free(ent->data);
>From my analysis it seems that git grep threads do acquire lock before calling read_sha1_file but not before calling
read_object_with_reference who ends up calling read_sha1_file too.
Adding the lock around read_object_with_reference seems to fix the issue for me.
I've ran git grep about a dozen time and seen no more error while
it usually happened half the time before.
Signed-off-by: Nicolas Morey-Chaisemartin <nicolas@morey-chaisemartin.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Make git grep use threads when it is available.
The results below are best of five runs in the Linux repository (on a
box with two cores).
With the patch:
git grep qwerty
1.58user 0.55system 0:01.16elapsed 183%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+800outputs (0major+5774minor)pagefaults 0swaps
Without:
git grep qwerty
1.59user 0.43system 0:02.02elapsed 100%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+800outputs (0major+3716minor)pagefaults 0swaps
And with a pattern with quite a few matches:
With the patch:
$ /usr/bin/time git grep void
5.61user 0.56system 0:03.44elapsed 179%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+800outputs (0major+5587minor)pagefaults 0swaps
Without:
$ /usr/bin/time git grep void
5.36user 0.51system 0:05.87elapsed 100%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+800outputs (0major+3693minor)pagefaults 0swaps
In either case we gain about 40% by the threading.
Signed-off-by: Fredrik Kuivinen <frekui@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Teach "git grep" a new "-q" option to report the presense of a match via
its exit status without showing any output, similar to how "grep -q"
works. Internally "grep" engine already knew this "status-only" mode of
operation because it needed to grep inside log message to filter commits
when called from the "git log" machinery, and this patch only exposes it
to the command line tool.
A somewhat unfair benchmark in the Linux kernel directory shows a dramatic
improvement:
(with patch)
$ time ../git.git/git grep -q linux HEAD ; echo $?
real 0m0.030s
user 0m0.004s
sys 0m0.004s
0
(without patch)
$ time git grep linux HEAD >/dev/null; echo $?
real 0m4.432s
user 0m4.272s
sys 0m0.076s
0
This is "somewhat unfair" because I knew a file with such a string comes
very early in the tree traversal (namely, ".gitignore").
Signed-off-by: Junio C Hamano <gitster@pobox.com>
* jc/grep-lookahead:
grep --no-index: allow use of "git grep" outside a git repository
grep: prepare to run outside of a work tree
grep: rip out pessimization to use fixmatch()
grep: rip out support for external grep
grep: optimize built-in grep by skipping lines that do not hit
Conflicts:
builtin-grep.c
t/t7002-grep.sh
Internally "git grep" runs regexec(3) that expects its input string
to be NUL terminated. When searching inside blob data, read_sha1_file()
automatically gives such a buffer, but builtin-grep.c forgot to put
the NUL at the end, even though it allocated enough space for it.
Signed-off-by: Jim Meyering <meyering@redhat.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Just like some people wanted diff features that are not found in
other people's diff implementations outside of a git repository
and added --no-index mode to the command, this adds --no-index mode
to the "git grep" command.
Also, inside a git repository, --no-index mode allows you to grep
in untracked (but not ignored) files.
Signed-off-by: Junio C Hamano <gitster@pobox.com>
This moves the call to setup_git_directory() for running "grep" from
the "git" wrapper to the implementation of the "grep" subcommand. A
new variable "use_index" is always true at this stage in the series,
and when it is on, we require that we are in a directory that is under
git control. To make sure we die the same way, we make a second call
into setup_git_directory() when we detect this situation.
Signed-off-by: Junio C Hamano <gitster@pobox.com>
* nd/sparse: (25 commits)
t7002: test for not using external grep on skip-worktree paths
t7002: set test prerequisite "external-grep" if supported
grep: do not do external grep on skip-worktree entries
commit: correctly respect skip-worktree bit
ie_match_stat(): do not ignore skip-worktree bit with CE_MATCH_IGNORE_VALID
tests: rename duplicate t1009
sparse checkout: inhibit empty worktree
Add tests for sparse checkout
read-tree: add --no-sparse-checkout to disable sparse checkout support
unpack-trees(): ignore worktree check outside checkout area
unpack_trees(): apply $GIT_DIR/info/sparse-checkout to the final index
unpack-trees(): "enable" sparse checkout and load $GIT_DIR/info/sparse-checkout
unpack-trees.c: generalize verify_* functions
unpack-trees(): add CE_WT_REMOVE to remove on worktree alone
Introduce "sparse checkout"
dir.c: export excluded_1() and add_excludes_from_file_1()
excluded_1(): support exclude files in index
unpack-trees(): carry skip-worktree bit over in merged_entry()
Read .gitignore from index if it is skip-worktree
Avoid writing to buffer in add_excludes_from_file_1()
...
Conflicts:
.gitignore
Documentation/config.txt
Documentation/git-update-index.txt
Makefile
entry.c
t/t7002-grep.sh
The -L (--files-without-match) option is supposed to show paths that
produced no matches. When running the internal grep on work tree files,
however, we had an optimization to just return on zero-sized files,
without doing anything.
This optimization doesn't matter too much in practice (a tracked empty
file must be rare, or there is something wrong with your project); to
produce results consistent with GNU grep, we should stop the optimization
and show empty files as not having the given pattern.
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Skip-worktree entries are not on disk. We cannot use external grep in such
cases.
Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
While we're at it, also unset GREP_COLOR and GREP_COLORS in case colouring
is not enabled, to be on the safe side. The presence of these variables
alone is not sufficient to trigger coloured output with GNU grep, but
other implementations may behave differently.
Signed-off-by: Rene Scharfe <rene.scharfe@lsrfire.ath.cx>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
"git grep" currently an error when you combine the -F and -i flags.
This isn't in line with how GNU grep handles it.
This patch allows the simultaneous use of those flags.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Brian Collins <bricollins@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Clarification: the following description only talks about "git
grep -h" without any other options and arguments.
Such a change cannot be breaking backward compatibility. "grep
-h" cannot be asking for suppressing filenames, as there is no
match pattern specified.
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
"git grep" would segfault if its -f option was used because it would
try to use an uninitialized strbuf, so initialize the strbuf.
Thanks to Johannes Sixt <j.sixt@viscovery.net> for the help with the
test cases.
Signed-off-by: Matt Kraai <kraai@ftbfs.org>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
* cb/maint-1.6.3-grep-relative-up:
grep: accept relative paths outside current working directory
grep: fix exit status if external_grep() punts
Conflicts:
t/t7002-grep.sh
"git grep" would barf at relative paths pointing outside the current
working directory (or subdirectories thereof). Use quote_path_relative(),
which can handle such cases just fine.
[jc: added tests.]
Signed-off-by: Clemens Buchacher <drizzd@aon.at>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
If external_grep() is called and punts, grep_cache() mistakenly reported a
hit, even if there were none. The bug can be triggered by calling "git
grep --no-color" from a subdirectory.
Signed-off-by: Clemens Buchacher <drizzd@aon.at>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
grep: turn on --cached for files that is marked skip-worktree
ls-files: do not check for deleted file that is marked skip-worktree
update-index: ignore update request if it's skip-worktree, while still allows removing
diff*: skip worktree version
Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
It is useful to grep directories non-recursively, e.g. when one wants to
look for all files in the toplevel directory, but not in any subdirectory,
or in Documentation/, but not in Documentation/technical/.
This patch adds support for --max-depth <depth> option to git-grep. If it is
given, git-grep descends at most <depth> levels of directories below paths
specified on the command line.
Note that if path specified on command line contains wildcards, this option
makes no sense, e.g.
$ git grep -l --max-depth 0 GNU -- 'contrib/*'
(note the quotes) will search all files in contrib/, even in
subdirectories, because '*' matches all files.
Documentation updates, bash-completion and simple test cases are also
provided.
Signed-off-by: Michał Kiedrowicz <michal.kiedrowicz@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
* rs/grep-p:
grep: simplify -p output
grep -p: support user defined regular expressions
grep: add option -p/--show-function
grep: handle pre context lines on demand
grep: print context hunk marks between files
grep: move context hunk mark handling into show_line()
userdiff: add xdiff_clear_find_func()
It was found a bit too loud to show == separators between the function
headers.
Signed-off-by: Rene Scharfe <rene.scharfe@lsrfire.ath.cx>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Respect the userdiff attributes and config settings when looking for
lines with function definitions in git grep -p.
Signed-off-by: Rene Scharfe <rene.scharfe@lsrfire.ath.cx>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
The new option -p instructs git grep to print the previous function
definition as a context line, similar to diff -p. Such context lines
are marked with an equal sign instead of a dash. This option
complements the existing context options -A, -B, -C.
Function definitions are detected using the same heuristic that diff
uses. User defined regular expressions are not supported, yet.
Signed-off-by: Rene Scharfe <rene.scharfe@lsrfire.ath.cx>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Print a hunk mark before matches from a new file are shown, in addition
to the current behaviour of printing them if lines have been skipped.
The result is easier to read, as (presumably unrelated) matches from
different files are separated by a hunk mark. GNU grep does the same.
Signed-off-by: Rene Scharfe <rene.scharfe@lsrfire.ath.cx>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Change calls to die(..., strerror(errno)) to use the new die_errno().
In the process, also make slight style adjustments: at least state
_something_ about the function that failed (instead of just printing
the pathname), and put paths in single quotes.
Signed-off-by: Thomas Rast <trast@student.ethz.ch>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
To give OPT_FILENAME the prefix, we pass the prefix to parse_options()
which passes the prefix to parse_options_start() which sets the prefix
member of parse_opts_ctx accordingly. If there isn't a prefix in the
calling context, passing NULL will suffice.
Signed-off-by: Stephen Boyd <bebarino@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Suggested by Stephen Boyd: make the callback functions used for option
parsing static.
Signed-off-by: Rene Scharfe <rene.scharfe@lsrfire.ath.cx>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Convert git-grep to parseopt.
The bitfields in struct grep_opt are converted to full ints,
increasing its size. This shouldn't be a problem as there is only a
single instance in memory.
Signed-off-by: Rene Scharfe <rene.scharfe@lsrfire.ath.cx>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Replace the only global variable in builtin-grep.c, builtin_grep, by a
local one and a function parameter with reversed meaning.
Signed-off-by: Rene Scharfe <rene.scharfe@lsrfire.ath.cx>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
color.grep and color.grep.* is the official and documented way to
highlight grep matches. Comparable options like diff.color.* and
status.color.* exist for backward compatibility reasons only and are not
documented any more.
Signed-off-by: Markus Heidelberg <markus.heidelberg@web.de>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
As far as I know, not all grep programs support coloring, so we should
rely on builtin grep. If you want external grep, set
color.grep.external to empty string.
Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Add the config variable color.grep.external, which can be used to
switch on coloring of external greps. To enable auto coloring with
GNU grep, one needs to set color.grep.external to --color=always to
defeat the pager started by git grep. The value of the config
variable will be passed to the external grep only if it would
colorize internal grep's output, so automatic terminal detected
works. The default is to not pass any option, because the external
grep command could be a program without color support.
Also set the environment variables GREP_COLOR and GREP_COLORS to
pass the configured color for matches to the external grep. This
works with GNU grep; other variables could be added as needed.
Signed-off-by: Rene Scharfe <rene.scharfe@lsrfire.ath.cx>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Coloring matches makes them easier to spot in the output.
Add two options and two parameters: color.grep (to turn coloring on
or off), color.grep.match (to set the color of matches), --color
and --no-color (to turn coloring on or off, respectively).
The output of external greps is not changed.
This patch is based on earlier ones by Nguyễn Thái Ngọc Duy and
Thiago Alves.
Signed-off-by: Rene Scharfe <rene.scharfe@lsrfire.ath.cx>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
"Assume unchanged" bit means "please pretend that I have never touched
this file", so if user removes the file, we should not care.
This patch teaches "git grep" to use cache version in such
situations. External grep case has not been fixed yet. But given that
on the platform that CE_VALID bit may be used like Windows, external
grep is not available anyway, I would wait for people to raise their
hands before touching it.
Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Here's a trivial patch that adds "-z" and "--null" options to "git
grep". It was discussed on the mailing-list that git's "-z"
convention should be used instead of GNU grep's "-Z".
So things like 'git grep -l -z "$FOO" | xargs -0 sed -i "s/$FOO/$BOO/"'
do work now.
Signed-off-by: Raphael Zimmerer <killekulla@rdrz.de>
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
This is a mechanical conversion of all '*.c' files with:
s/((?:die|error|warning)\("git)-(\S+:)/$1 $2/;
The result was manually inspected and no false positive was found.
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Unless used with --cached or grepping on a tree, "git grep" will
search on working directory, so set up worktree properly
Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
