mirrors/git
1
0
Fork 0
mirror of https://github.com/git/git.git synced 2024-11-09 02:33:11 +01:00
Code Activity
git/utf8.h

59 lines
1.9 KiB
C
Raw Normal View History

commit-tree: encourage UTF-8 commit messages. Introduce is_utf() to check if a text looks like it is encoded in UTF-8, utf8_width() to count display width, and implements print_wrapped_text() using them. git-commit-tree warns if the commit message does not minimally conform to the UTF-8 encoding when i18n.commitencoding is either unset, or set to "utf-8". Signed-off-by: Junio C Hamano <junkio@cox.net>
2006-12-22 22:06:08 +01:00
#ifndef GIT_UTF8_H
#define GIT_UTF8_H
utf8: pick_one_utf8_char() utf8_width() function was doing two different things. To pick a valid character from UTF-8 stream, and compute the display width of that character. This splits the former to a separate function pick_one_utf8_char(). Signed-off-by: Junio C Hamano <gitster@pobox.com>
2008-01-07 04:02:22 +01:00
typedef unsigned int ucs_char_t; /* assuming 32bit int */
pretty: support %>> that steal trailing spaces This is pretty useful in `%<(100)%s%Cred%>(20)% an' where %s does not use up all 100 columns and %an needs more than 20 columns. By replacing %>(20) with %>>(20), %an can steal spaces from %s. %>> understands escape sequences, so %Cred does not stop it from stealing spaces in %<(100). Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2013-04-19 01:08:52 +02:00
size_t display_mode_esc_sequence_len(const char *s);
utf8_width(): allow non NUL-terminated input The original interface assumed that the input string is always terminated with a NUL, but that wasn't too useful. Signed-off-by: Junio C Hamano <gitster@pobox.com>
2008-01-02 10:49:58 +01:00
int utf8_width(const char **start, size_t *remainder_p);
utf8.c: add utf8_strnwidth() with the ability to skip ansi sequences Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2013-04-19 01:08:45 +02:00
int utf8_strnwidth(const char *string, int len, int skip_ansi);
utf8: add utf8_strwidth() I'm about to use this pattern more than once, so make it a common function. Signed-off-by: Geoffrey Thomas <geofft@mit.edu> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2009-01-30 10:41:28 +01:00
int utf8_strwidth(const char *string);
commit-tree: encourage UTF-8 commit messages. Introduce is_utf() to check if a text looks like it is encoded in UTF-8, utf8_width() to count display width, and implements print_wrapped_text() using them. git-commit-tree warns if the commit message does not minimally conform to the UTF-8 encoding when i18n.commitencoding is either unset, or set to "utf-8". Signed-off-by: Junio C Hamano <junkio@cox.net>
2006-12-22 22:06:08 +01:00
int is_utf8(const char *text);
commit-tree: cope with different ways "utf-8" can be spelled. People can spell config.commitencoding differently from what we internally have ("utf-8") to mean UTF-8. Try to accept them and treat them equally. Signed-off-by: Junio C Hamano <junkio@cox.net>
2006-12-30 21:20:43 +01:00
int is_encoding_utf8(const char *name);
reencode_string(): introduce and use same_encoding() Callers of reencode_string() that re-encodes a string from one encoding to another all used ad-hoc way to bypass the case where the input and the output encodings are the same. Some did strcmp(), some did strcasecmp(), yet some others when converting to UTF-8 used is_encoding_utf8(). Introduce same_encoding() helper function to make these callers use the same logic. Notably, is_encoding_utf8() has a work-around for common misconfiguration to use "utf8" to name UTF-8 encoding, which does not match "UTF-8" hence strcasecmp() would not consider the same. Make use of it in this helper function. Signed-off-by: Junio C Hamano <gitster@pobox.com>
2012-10-19 07:41:56 +02:00
int same_encoding(const char *, const char *);
add missing "format" function attributes For most of our functions that take printf-like formats, we use gcc's __attribute__((format)) to get compiler warnings when the functions are misused. Let's give a few more functions the same protection. In most cases, the annotations do not uncover any actual bugs; the only code change needed is that we passed a size_t to transfer_debug, which expected an int. Since we expect the passed-in value to be a relatively small buffer size (and cast a similar value to int directly below), we can just cast away the problem. Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2013-07-10 02:18:40 +02:00
__attribute__((format (printf, 2, 3)))
Add utf8_fprintf helper that returns correct number of columns Since command usages can be translated, they may include utf-8 encoded strings, and the output in console may not align well any more. This is because strlen() is different from strwidth() on utf-8 strings. A wrapper utf8_fprintf() can help to return the correct number of columns required. Signed-off-by: Jiang Xin <worldhello.net@gmail.com> Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com> Reviewed-by: Torsten Bögershausen <tboegi@web.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2013-02-09 07:31:09 +01:00
int utf8_fprintf(FILE *, const char *, ...);
commit-tree: cope with different ways "utf-8" can be spelled. People can spell config.commitencoding differently from what we internally have ("utf-8") to mean UTF-8. Try to accept them and treat them equally. Signed-off-by: Junio C Hamano <junkio@cox.net>
2006-12-30 21:20:43 +01:00
utf8-bom: introduce skip_utf8_bom() helper With the recent change to ignore the UTF8 BOM at the beginning of .gitignore files, we now have two codepaths that do such a skipping (the other one is for reading the configuration files). Introduce utf8_bom[] constant string and skip_utf8_bom() helper and teach .gitignore code how to use it. Signed-off-by: Junio C Hamano <gitster@pobox.com>
2015-04-16 19:45:29 +02:00
extern const char utf8_bom[];
extern int skip_utf8_bom(char **, size_t);
strbuf_add_wrapped*(): Remove unused return value Since shortlog isn't using the return value anymore (see previous commit), the functions can be changed to void. Signed-off-by: Steffen Prohaska <prohaska@zib.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2012-12-11 06:59:22 +01:00
void strbuf_add_wrapped_text(struct strbuf *buf,
Add strbuf_add_wrapped_text() to utf8.[ch] The newly added function can rewrap text according to a given first-line indent, other-indent and text width. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
2008-11-10 18:47:00 +01:00
const char *text, int indent, int indent2, int width);
strbuf_add_wrapped*(): Remove unused return value Since shortlog isn't using the return value anymore (see previous commit), the functions can be changed to void. Signed-off-by: Steffen Prohaska <prohaska@zib.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2012-12-11 06:59:22 +01:00
void strbuf_add_wrapped_bytes(struct strbuf *buf, const char *data, int len,
strbuf: add fixed-length version of add_wrapped_text The function strbuf_add_wrapped_text takes a NUL-terminated string. This makes it annoying to wrap strings we have as a pointer and a length. Refactoring strbuf_add_wrapped_text and all of its sub-functions to handle fixed-length strings turned out to be really ugly. So this implementation is lame; it just strdups the text and operates on the NUL-terminated version. This should be fine as the strings we are wrapping are generally pretty short. If it becomes a problem, we can optimize later. Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2011-02-23 10:50:19 +01:00
int indent, int indent2, int width);
pretty: support truncating in %>, %< and %>< %>(N,trunc) truncates the right part after N columns and replace the last two letters with "..". ltrunc does the same on the left. mtrunc cuts the middle out. Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2013-04-19 01:08:51 +02:00
void strbuf_utf8_replace(struct strbuf *sb, int pos, int width,
const char *subst);
commit-tree: encourage UTF-8 commit messages. Introduce is_utf() to check if a text looks like it is encoded in UTF-8, utf8_width() to count display width, and implements print_wrapped_text() using them. git-commit-tree warns if the commit message does not minimally conform to the UTF-8 encoding when i18n.commitencoding is either unset, or set to "utf-8". Signed-off-by: Junio C Hamano <junkio@cox.net>
2006-12-22 22:06:08 +01:00
Move encoding conversion routine out of mailinfo to utf8.c This moves the body of convert_to_utf8() routine used in mailinfo to the utf8.c i18n library. Signed-off-by: Junio C Hamano <junkio@cox.net>
2006-12-24 08:36:55 +01:00
#ifndef NO_ICONV
utf8.c: add reencode_string_len() that can handle NULs in string Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2013-04-19 01:08:46 +02:00
char *reencode_string_iconv(const char *in, size_t insz,
iconv_t conv, int *outsz);
char *reencode_string_len(const char *in, int insz,
const char *out_encoding,
const char *in_encoding,
int *outsz);
Move encoding conversion routine out of mailinfo to utf8.c This moves the body of convert_to_utf8() routine used in mailinfo to the utf8.c i18n library. Signed-off-by: Junio C Hamano <junkio@cox.net>
2006-12-24 08:36:55 +01:00
#else
utf8: NO_ICONV: silence uninitialized variable warning The last argument of reencode_string_len() is an 'int *' which is assigned the length of the converted string. When NO_ICONV is defined, however, reencode_string_len() is stubbed out by the macro: #define reencode_string_len(a,b,c,d,e) NULL which never assigns a value to the final argument. When called like this: int n; char *s = reencode_string_len(..., &n); if (s) do_something(s, n); some compilers complain that 'n' is used uninitialized within the conditional. Signed-off-by: Eric Sunshine <sunshine@sunshineco.com> Reviewed-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2015-06-05 08:42:16 +02:00
static inline char *reencode_string_len(const char *a, int b,
const char *c, const char *d, int *e)
{ if (e) *e = 0; return NULL; }
Move encoding conversion routine out of mailinfo to utf8.c This moves the body of convert_to_utf8() routine used in mailinfo to the utf8.c i18n library. Signed-off-by: Junio C Hamano <junkio@cox.net>
2006-12-24 08:36:55 +01:00
#endif
utf8.c: add reencode_string_len() that can handle NULs in string Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2013-04-19 01:08:46 +02:00
static inline char *reencode_string(const char *in,
const char *out_encoding,
const char *in_encoding)
{
return reencode_string_len(in, strlen(in),
out_encoding, in_encoding,
NULL);
}
format-patch: RFC 2047 says multi-octet character may not be split Even though an earlier attempt (bafc478..41dd00bad) cleaned up RFC 2047 encoding, pretty.c::add_rfc2047() still decides where to split the output line by going through the input one byte at a time, and potentially splits a character in the middle. A subject line may end up showing like this: ".... fö?? bar". (instead of ".... föö bar".) if split incorrectly. RFC 2047, section 5 (3) explicitly forbids such beaviour Each 'encoded-word' MUST represent an integral number of characters. A multi-octet character may not be split across adjacent 'encoded- word's. that means that e.g. for Subject: .... föö bar encoding Subject: =?UTF-8?q?....=20f=C3=B6=C3=B6?= =?UTF-8?q?=20bar?= is correct, and Subject: =?UTF-8?q?....=20f=C3=B6=C3?= <-- NOTE ö is broken here =?UTF-8?q?=B6=20bar?= is not, because "ö" character UTF-8 encoding C3 B6 is split here across adjacent encoded words. To fix the problem, make the loop grab one _character_ at a time and determine its output length to see where to break the output line. Note that this version only knows about UTF-8, but the logic to grab one character is abstracted out in mbs_chrlen() function to make it possible to extend it to other encodings with the help of iconv in the future. Signed-off-by: Kirill Smelkov <kirr@mns.spb.ru> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2013-03-07 11:55:07 +01:00
int mbs_chrlen(const char **text, size_t *remainder_p, const char *encoding);
utf8: add is_hfs_dotgit() helper We do not allow paths with a ".git" component to be added to the index, as that would mean repository contents could overwrite our repository files. However, asking "is this path the same as .git" is not as simple as strcmp() on some filesystems. HFS+'s case-folding does more than just fold uppercase into lowercase (which we already handle with strcasecmp). It may also skip past certain "ignored" Unicode code points, so that (for example) ".gi\u200ct" is mapped ot ".git". The full list of folds can be found in the tables at: https://www.opensource.apple.com/source/xnu/xnu-1504.15.3/bsd/hfs/hfscommon/Unicode/UCStringCompareData.h Implementing a full "is this path the same as that path" comparison would require us importing the whole set of tables. However, what we want to do is much simpler: we only care about checking ".git". We know that 'G' is the only thing that folds to 'g', and so on, so we really only need to deal with the set of ignored code points, which is much smaller. Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2014-12-15 23:56:59 +01:00
/*
* Returns true if the the path would match ".git" after HFS case-folding.
* The path should be NUL-terminated, but we will match variants of both ".git\0"
* and ".git/..." (but _not_ ".../.git"). This makes it suitable for both fsck
* and verify_path().
*/
int is_hfs_dotgit(const char *path);
commit-tree: encourage UTF-8 commit messages. Introduce is_utf() to check if a text looks like it is encoded in UTF-8, utf8_width() to count display width, and implements print_wrapped_text() using them. git-commit-tree warns if the commit message does not minimally conform to the UTF-8 encoding when i18n.commitencoding is either unset, or set to "utf-8". Signed-off-by: Junio C Hamano <junkio@cox.net>
2006-12-22 22:06:08 +01:00
#endif
Copy permalink