Quellcode durchsuchen

[Fix] Fix normalization of non-alphabet based languages

rspamd-1.9
Vsevolod Stakhov vor 4 Jahren
Ursprung
Commit
44776f99b8
2 geänderte Dateien mit 4 neuen und 7 gelöschten Zeilen
  1. 2
    1
      src/libstat/backends/redis_backend.c
  2. 2
    6
      src/libstat/tokenizers/tokenizers.c

+ 2
- 1
src/libstat/backends/redis_backend.c Datei anzeigen

"HSET %b_tokens %b %b", "HSET %b_tokens %b %b",
prefix, (size_t) prefix_len, prefix, (size_t) prefix_len,
n0, (size_t) l0, n0, (size_t) l0,
tok->t1->stemmed.begin, tok->t1->stemmed.len);
tok->t1->stemmed.begin,
tok->t1->stemmed.len);
} }
} }
else { else {

+ 2
- 6
src/libstat/tokenizers/tokenizers.c Datei anzeigen

} }
#endif #endif


if (cat == U_UPPERCASE_LETTER ||
cat == U_LOWERCASE_LETTER ||
cat == U_DECIMAL_DIGIT_NUMBER ||
if ((cat >= U_UPPERCASE_LETTER && cat <= U_OTHER_NUMBER) ||
cat == U_CONNECTOR_PUNCTUATION || cat == U_CONNECTOR_PUNCTUATION ||
cat == U_MATH_SYMBOL || cat == U_MATH_SYMBOL ||
cat == U_CURRENCY_SYMBOL ||
cat == U_INITIAL_PUNCTUATION ||
cat == U_FINAL_PUNCTUATION) {
cat == U_CURRENCY_SYMBOL) {
*d++ = u_tolower (t); *d++ = u_tolower (t);
} }
} }

Laden…
Abbrechen
Speichern