* 'svn-fe' of git://repo.or.cz/git/jrn: (31 commits)
fast-import: make code "-Wpointer-arith" clean
vcs-svn: teach line_buffer about temporary files
vcs-svn: allow input from file descriptor
vcs-svn: allow character-oriented input
vcs-svn: add binary-safe read function
t0081 (line-buffer): add buffering tests
vcs-svn: tweak test-line-buffer to not assume line-oriented input
tests: give vcs-svn/line_buffer its own test script
vcs-svn: make test-line-buffer input format more flexible
vcs-svn: teach line_buffer to handle multiple input files
vcs-svn: collect line_buffer data in a struct
vcs-svn: replace buffer_read_string memory pool with a strbuf
vcs-svn: eliminate global byte_buffer
fast-import: add 'ls' command
vcs-svn: Allow change nodes for root of tree (/)
vcs-svn: Implement Prop-delta handling
vcs-svn: Sharpen parsing of property lines
vcs-svn: Split off function for handling of individual properties
vcs-svn: Make source easier to read on small screens
vcs-svn: More dump format sanity checks
...
The dereference() function to peel a tree-ish and find the underlying
tree expects arithmetic to (void *) to work on byte addresses. We
should be reading the text of objects through a char * anyway.
Noticed-by: Junio C Hamano <gitster@pobox.com>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
Rather than using 'errno == EMFILE' after a failed open() call
to indicate the process is out of file descriptors and an LRU
pack window should be closed, place a hard upper limit on the
number of open packs based on the actual rlimit of the process.
By using a hard upper limit that is below the rlimit of the current
process it is not necessary to check for EMFILE on every single
fd-allocating system call. Instead reserving 25 file descriptors
makes it safe to assume the system call won't fail due to being over
the filedescriptor limit. Here 25 is chosen as a WAG, but considers
3 for stdin/stdout/stderr, and at least a few for other Git code
to operate on temporary files. An additional 20 is reserved as it
is not known what the C library needs to perform other services on
Git's behalf, such as nsswitch or name resolution.
This fixes a case where running `git gc --auto` in a repository
with more than 1024 packs (but an rlimit of 1024 open fds) fails
due to the temporary output file not being able to allocate a
file descriptor. The output file is opened by pack-objects after
object enumeration and delta compression are done, both of which
have already opened all of the packs and fully populated the file
descriptor table.
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Signed-off-by: Sebastian Schuberth <sschuberth@gmail.com>
Tested-by: Chris Packham <judge.packham@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Signed-off-by: Sebastian Schuberth <sschuberth@gmail.com>
Tested-by: Chris Packham <judge.packham@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
According to blog post "FindBin, __FILE__, $0 and PSGI woes"
http://bulknews.typepad.com/blog/2011/02/findbin-__file__-0-and-psgi-woes.html
by Tatsuhiko Miyagawa, using 'if (__FILE__ eq $0)' in .psgi code
(to check if script was run from command line), is not supposed to work
since Plack 0.9971.
Replace it with one of proposed solutions; while at it return $app
explicitely, rather than implicitely by being a last expression.
This affects 'plackup' web server.
While at it cleanup whitespace.
Signed-off-by: Jakub Narebski <jnareb@gmail.com>
Acked-by: Eric Wong <normalperson@yhbt.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
* mg/placeholders-are-lowercase:
Make <identifier> lowercase in Documentation
Make <identifier> lowercase as per CodingGuidelines
Make <identifier> lowercase as per CodingGuidelines
Make <identifier> lowercase as per CodingGuidelines
CodingGuidelines: downcase placeholders in usage messages
* pw/p4:
git-p4: support clone --bare
git-p4: decode p4 wildcard characters
git-p4: better message for "git-p4 sync" when not cloned
git-p4: reinterpret confusing p4 message
git-p4: accommodate new move/delete type in p4
git-p4: add missing newline in initial import message
git-p4: fix key error for p4 problem
git-p4: test script
* uk/checkout-ambiguous-ref:
Rename t2019 with typo "amiguous" that meant "ambiguous"
checkout: rearrange update_refs_for_switch for clarity
checkout: introduce --detach synonym for "git checkout foo^{commit}"
checkout: split off a function to peel away branchname arg
checkout: fix bug with ambiguous refs
Conflicts:
builtin/checkout.c
* hv/mingw-fs-funnies:
mingw_rmdir: set errno=ENOTEMPTY when appropriate
mingw: add fallback for rmdir in case directory is in use
mingw: make failures to unlink or move raise a question
mingw: work around irregular failures of unlink on windows
mingw: move unlink wrapper to mingw.c
* en/object-list-with-pathspec:
Add testcases showing how pathspecs are handled with rev-list --objects
Make rev-list --objects work together with pathspecs
* nd/struct-pathspec: (22 commits)
t6004: add pathspec globbing test for log family
t7810: overlapping pathspecs and depth limit
grep: drop pathspec_matches() in favor of tree_entry_interesting()
grep: use writable strbuf from caller for grep_tree()
grep: use match_pathspec_depth() for cache/worktree grepping
grep: convert to use struct pathspec
Convert ce_path_match() to use match_pathspec_depth()
Convert ce_path_match() to use struct pathspec
struct rev_info: convert prune_data to struct pathspec
pathspec: add match_pathspec_depth()
tree_entry_interesting(): optimize wildcard matching when base is matched
tree_entry_interesting(): support wildcard matching
tree_entry_interesting(): fix depth limit with overlapping pathspecs
tree_entry_interesting(): support depth limit
tree_entry_interesting(): refactor into separate smaller functions
diff-tree: convert base+baselen to writable strbuf
glossary: define pathspec
Move tree_entry_interesting() to tree-walk.c and export it
tree_entry_interesting(): remove dependency on struct diff_options
Convert struct diff_options to use struct pathspec
...
The most important issue is that after unsetting `i18n.commitencoding'
config variable t9500 no longer will use author and comitter name
containing ISO-8859-1 characters, which are invalid UTF-8 characters.
Besides it is good practice in general to clean up the state in tests.
Signed-off-by: Jakub Narebski <jnareb@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Currently, it sounds as if "notice removal of files" distinguishes "-A"
from "-u", and there is no mention of the worktree.
But both notice the removal, and only "-A" adds changes from untracked
files. Say so.
Signed-off-by: Michael J Gruber <git@drmicha.warpmail.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
If the CVS server is down, this reduced the git-cvsimport output from:
ssh: connect to host ijbswa.cvs.sourceforge.net port 22: Connection refused
Use of uninitialized value $rep in scalar chomp at /usr/local/libexec/git-core/git-cvsimport line 369.
Use of uninitialized value $rep in substitution (s///) at /usr/local/libexec/git-core/git-cvsimport line 370.
Expected Valid-requests from server, but got: <unknown>
to the less noisy:
ssh: connect to host ijbswa.cvs.sourceforge.net port 22: Connection refused
Failed to read from server at /usr/local/libexec/git-core/git-cvsimport line 370.
In this case a silent exit() instead of the die() would probably do,
but I assume that there could be cases where the connection attempt
succeeds, but reading from the server fails for other reasons.
Signed-off-by: Fabian Keil <fk@fabiankeil.de>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
This bases on the original work by Robin Rosenberg.
Signed-off-by: Robin Rosenberg <robin.rosenberg@dewire.com>
Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
* git://github.com/gitster/git:
vcs-svn: Allow change nodes for root of tree (/)
vcs-svn: Implement Prop-delta handling
vcs-svn: Sharpen parsing of property lines
vcs-svn: Split off function for handling of individual properties
vcs-svn: Make source easier to read on small screens
vcs-svn: More dump format sanity checks
vcs-svn: Reject path nodes without Node-action
vcs-svn: Delay read of per-path properties
vcs-svn: Combine repo_replace and repo_modify functions
vcs-svn: Replace = Delete + Add
vcs-svn: handle_node: Handle deletion case early
vcs-svn: Use mark to indicate nodes with included text
vcs-svn: Unclutter handle_node by introducing have_props var
vcs-svn: Eliminate node_ctx.mark global
vcs-svn: Eliminate node_ctx.srcRev global
vcs-svn: Check for errors from open()
vcs-svn: Allow simple v3 dumps (no deltas yet)
Conflicts:
t/t9010-svn-fe.sh
vcs-svn/svndump.c
It can sometimes be useful to write information temporarily to file,
to read back later. These functions allow a program to use the
line_buffer facilities when doing so.
It works like this:
1. find a unique filename with buffer_tmpfile_init.
2. rewind with buffer_tmpfile_rewind. This returns a stdio
handle for writing.
3. when finished writing, declare so with
buffer_tmpfile_prepare_to_read. The return value indicates
how many bytes were written.
4. read whatever portion of the file is needed.
5. if finished, remove the temporary file with buffer_deinit.
otherwise, go back to step 2,
The svn support would use this to buffer the postimage from delta
application until the length is known and fast-import can receive
the resulting blob.
Based-on-patch-by: David Barr <david.barr@cordelta.com>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
buffer_read_char can be used in place of buffer_read_string(1) to
avoid consuming valuable static buffer space. The delta applier will
use this to read variable-length integers one byte at a time.
Underneath, it is fgetc, wrapped so the line_buffer library can
maintain its role as gatekeeper of input.
Later it might be worth checking if fgetc_unlocked is faster ---
most line_buffer functions are not thread-safe anyway.
Helpd-by: David Barr <david.barr@cordelta.com>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
buffer_read_string works well for non line-oriented input except for
one problem: it does not tell the caller how many bytes were actually
written. This means that unless one is very careful about checking
for errors (and eof) the calling program cannot tell the difference
between the string "foo" followed by an early end of file and the
string "foo\0bar\0baz".
So introduce a variant that reports the length, too, a thinner wrapper
around strbuf_fread. Its result is written to a strbuf so the caller
does not need to keep track of the number of bytes read.
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
POSIX makes the behavior of read(2) from a pipe fairly clear: a read
from an empty pipe will block until there is data available and any
other read will not block, prefering to return a partial result.
Likewise, fread(3) and fgets(3) are clearly specified to act as
though implemented by calling fgetc(3) in a simple loop. But the
buffering behavior of fgetc is less clear.
Luckily, no sane platform is going to implement fgetc by calling the
equivalent of read(2) more than once. fgetc has to be able to
return without filling its buffer to preserve errno when errors are
encountered anyway. So let's assume the simpler behavior (trust) but
add some tests to catch insane platforms that violate that when they
come (verify).
First check that fread can handle a 0-length read from an empty fifo.
Because open(O_RDONLY) blocks until the writing end is open, open the
writing end of the fifo in advance in a subshell.
Next try short inputs from a pipe that is not filled all the way.
Lastly (two tests) try very large inputs from a pipe that will not fit
in the relevant buffers. The first of these tests reads a little
more than 8192 bytes, which is BUFSIZ (the size of stdio's buffers)
on this Linux machine. The second reads a little over 64 KiB (the
pipe capacity on Linux) and is not run unless requested by setting
the GIT_REMOTE_SVN_TEST_BIG_FILES environment variable.
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
Do not expect an implicit newline after each input record.
Use a separate command to exercise buffer_skip_bytes.
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
Split the line_buffer test into small pieces and move it to its
own file as preparation for adding more tests.
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
Imitate the input format of test-obj-pool to support arbitrary
sequences of commands rather than alternating read/copy. This should
make it easier to add tests that exercise other line_buffer functions.
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
Collect the line_buffer state in a newly public line_buffer struct.
Callers can use multiple line_buffers to manage input from multiple
files at a time.
svn-fe's delta applier will use this to stream a delta from svnrdump
and the preimage it applies to from fast-import at the same time.
The tests don't take advantage of the new features, but I think that's
okay. It is easier to find lingering examples of nonreentrant code by
searching for "static" in line_buffer.c.
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
Prepare for the line_buffer lib to support input from multiple files,
by collecting global state in a struct that can be easily passed
around.
No API change yet.
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
obj_pool is inherently global and does not use the standard growing
factor alloc_nr, which makes it feel out of place in the git codebase.
Plus it is overkill for this application: all that is needed is a
buffer that can grow between requests to accomodate larger strings.
Use a strbuf instead.
As a side effect, this improves the error handling: allocation
failures will result in a clean exit instead of segfaults. It would
be nice to add a test case (using ulimit or failmalloc) but that can
wait for another day.
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
The data stored in byte_buffer[] is always either discarded or
written to stdout immediately. No need for it to persist between
function calls.
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
Lazy fast-import frontend authors that want to rely on the backend to
keep track of the content of the imported trees _almost_ have what
they need in the 'cat-blob' command (v1.7.4-rc0~30^2~3, 2010-11-28).
But it is not quite enough, since
(1) cat-blob can be used to retrieve the content of files, but
not their mode, and
(2) using cat-blob requires the frontend to keep track of a name
(mark number or object id) for each blob to be retrieved
Introduce an 'ls' command to complement cat-blob and take care of the
remaining needs. The 'ls' command finds what is at a given path
within a given tree-ish (tag, commit, or tree):
'ls' SP <dataref> SP <path> LF
or in fast-import's active commit:
'ls' SP <path> LF
The response is a single line sent through the cat-blob channel,
imitating ls-tree output. So for example:
FE> ls :1 Documentation
gfi> 040000 tree 9e6c2b599341d28a2a375f8207507e0a2a627fe9 Documentation
FE> ls 9e6c2b599341d28a2a375f8207507e0a2a627fe9 git-fast-import.txt
gfi> 100644 blob 4f92954396e3f0f97e75b6838a5635b583708870 git-fast-import.txt
FE> ls :1 RelNotes
gfi> 120000 blob b942e49944 RelNotes
FE> cat-blob b942e49944
gfi> b942e49944 blob 32
gfi> Documentation/RelNotes/1.7.4.txt
The most interesting parts of the reply are the first word, which is
a 6-digit octal mode (regular file, executable, symlink, directory,
or submodule), and the part from the second space to the tab, which is
a <dataref> that can be used in later cat-blob, ls, and filemodify (M)
commands to refer to the content (blob, tree, or commit) at that path.
If there is nothing there, the response is "missing some/path".
The intent is for this command to be used to read files from the
active commit, so a frontend can apply patches to them, and to copy
files and directories from previous revisions.
For example, proposed updates to svn-fe use this command in place of
its internal representation of the repository directory structure.
This simplifies the frontend a great deal and means support for
resuming an import in a separate fast-import run (i.e., incremental
import) is basically free.
Signed-off-by: David Barr <david.barr@cordelta.com>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
Improved-by: Junio C Hamano <gitster@pobox.com>
Improved-by: Sverre Rabbelier <srabbelier@gmail.com>
In a variable-args function, the code for writing into a strbuf is
non-trivial. We ended up cutting and pasting it in several places
because there was no vprintf-style function for strbufs (which in turn
was held up by a lack of va_copy).
Now that we have a fallback va_copy, we can add strbuf_vaddf, the
strbuf equivalent of vsprintf. And we can clean up the cut and paste
mess.
Signed-off-by: Jeff King <peff@peff.net>
Improved-by: Christian Couder <christian.couder@gmail.com>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
va_copy is C99. We have avoided using va_copy many times in the past,
which has led to a bunch of cut-and-paste. From everything I found
searching the web, implementations have historically either provided
va_copy or just let your code assume that simple assignment of worked.
So my guess is that this will be sufficient, though we won't really
know for sure until somebody reports a problem.
Signed-off-by: Jeff King <peff@peff.net>
Improved-by: Erik Faye-Lund <kusmabite@gmail.com>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
