BreakIterator_CTL in the non CharacterIteratorMode::SKIPCELL mode did
not handle UTF-16 surrogate pairs at all, causing backspace to delete
lone surrogates which is really bad. Just copied the corresponding code
from BreakIterator_Unicode.

Additionally, BreakIterator_th was not correctly skipping non-Thai text
and always treating one character as Thai.

Change-Id: Ia379327e042ff602fc19a485c4cbd1a3683f9230
Reviewed-on: https://gerrit.libreoffice.org/54631Tested-by: Jenkins <ci@libreoffice.org>
Reviewed-by: Eike Rathke <erack@redhat.com>

Korean words are composed of Hangul and are separated
by space or newline. This patch improves line breaking
function in CJK break iterator so that it does not
break Korean words in the middle. It now breaks at the
first character of the last Korean word.

Change-Id: I91b20733c0c5ec4755bf68eb0d7c14c42c1f3556
Reviewed-on: https://gerrit.libreoffice.org/42987Tested-by: Jenkins <ci@libreoffice.org>
Reviewed-by: Eike Rathke <erack@redhat.com>

Change-Id: I23368c3ce6d29c7b2e758e209e5a8315e82a2818
Reviewed-on: https://gerrit.libreoffice.org/40051Tested-by: Jenkins <ci@libreoffice.org>
Reviewed-by: Noel Grandin <noel.grandin@collabora.co.uk>

Change-Id: Ia9b20a8ca95684cbeb21e3425972c43ba50df3cd
Reviewed-on: https://gerrit.libreoffice.org/39187Tested-by: Jenkins <ci@libreoffice.org>
Reviewed-by: Noel Grandin <noel.grandin@collabora.co.uk>

Change-Id: Ia4e02589d2fe79a27b83200a0e7a528a2c806519
Reviewed-on: https://gerrit.libreoffice.org/38508Tested-by: Jenkins <ci@libreoffice.org>
Reviewed-by: Noel Grandin <noel.grandin@collabora.co.uk>

Change-Id: I7307cc96eac5868ed26e8ace1fc3c1a93e1bfec4

We are classifying characters in the “Combining Diacritical Marks”
Unicode block with ScriptType::LATIN, but these are combining marks and
can combine with any script and should have been ScriptType::WEAK. Just
removing them from the range in scriptList does the trick as we will
fallback to getting the script classification based on the Unicode
script property.

Change-Id: I3577f4b03360a1c8e094a207f01b6bbb6abbaf30
Reviewed-on: https://gerrit.libreoffice.org/35811Tested-by: Jenkins <ci@libreoffice.org>
Reviewed-by: Caolán McNamara <caolanm@redhat.com>
Tested-by: Caolán McNamara <caolanm@redhat.com>

Change-Id: I10a8298e5379fa93a5d3616202a7802c0eda1cbb

Change-Id: I1cf7b7d20a0b567c7363c5a9abc5bf1195b57262

Change-Id: I2ebe54af7b769189e248b1a3af55ee1b6a66174a
Reviewed-on: https://gerrit.libreoffice.org/29399Reviewed-by: Miklos Vajna <vmiklos@collabora.co.uk>
Tested-by: Jenkins <ci@libreoffice.org>

Change-Id: If4bc7dd5af49cca85f474e817cc3cc358c2b48c2

Change-Id: I76f09a09fd6c3b114ba74737d4a1ba5dad0fd28f

Change-Id: I11822c50fa66d038a3d6f38054ab35c2e613f077

Change-Id: I2ea407acd763ef2d7dae2d3b8f32525523ac8274

Enabling the ICU Breakiterator causes Khmer spelling checkers to be worthless
as well as wrecks havoc on various minority languages that use the Khmer
script. See this https://bugs.documentfoundation.org/show_bug.cgi?id=52020
this: https://bugs.freedesktop.org/show_bug.cgi?id=59448 and this:
https://bugs.documentfoundation.org/show_bug.cgi?id=59447&redirected_from=fdo

Change-Id: I84529db9cca72ffe1108d2e12c1885f63a48f89d

with the variadic variants.

Change-Id: I0381de7fde198df74556355984bdaba2ecdedd4b
Reviewed-on: https://gerrit.libreoffice.org/17768Reviewed-by: Noel Grandin <noelgrandin@gmail.com>
Tested-by: Noel Grandin <noelgrandin@gmail.com>

Change-Id: I7ef1dc1d42237ef101731b363359801d9e1e9db6

Change-Id: Ic91ddc695e9f3c7d8ba4d2bb67c8c84de2a22d5c

...which had already been conditional on U_ICU_VERSION_MAJOR_NUM > 51, so
452a06b7 "Activate unit test for Lao
breakiterator support" should merely have extended that #if to the function
declaration/definition in response to loplugin:unreffun, not unconditionally
enabled it.

Change-Id: Ia1ce4e99b4a960b1e70cae0a61b0d526b0353d9a

Change-Id: I26abf6d0e102712f76977e0763c4a672434a1325

In f24fa8ef added unit test wasn't
activated. Activating it now reveals that the test is failing. Add
TODOs for the failing asserts.

Change-Id: I6c9f36be3aa84a5d5092d2a3f576af39a15b935e

Change-Id: I6ef4ae530b7fb4e615100803ae6e3972d9b4545f

Change-Id: I6f2fabd9248b1b385439ff5c074342029fb660e0

Change-Id: I283dddaa4bd8baf05b90ce5f81d43b785021a3c4

Change-Id: I770c1b3e5164cb486b5a5c2b1259f713914a1bae

...mostly done with a rewriting Clang plugin, with just some manual tweaking
necessary to fix poor macro usage.

Change-Id: I71fa20213e86be10de332ece0aa273239df7b61a

Issue:
OUString uses UTF-16, so for a Unicode surrogate character there are 2
values stored, not just 1.
So we are getting assert failure in "rtl_uString_iterateCodePoints" method.

erAck: Underlying cause was that the dictionary breakiterator misused UTF-16 positions as Unicode code point positions.

Change-Id: I923485f56c2d879b63687adaea2b489a3479991c
Reviewed-on: https://gerrit.libreoffice.org/6955Reviewed-by: Eike Rathke <erack@redhat.com>
Tested-by: Eike Rathke <erack@redhat.com>

This breaks windows build with localized versions of MSVC.

Change-Id: I23c46830f96ae661eced88352476e7ae61fbcc2a
Reviewed-on: https://gerrit.libreoffice.org/6847Reviewed-by: Eike Rathke <erack@redhat.com>
Tested-by: Eike Rathke <erack@redhat.com>

Convert code like:
   aOStringBuf.append( RTL_CONSTASCII_STRINGPARAM( " is missing )") );
to:
   aOStringBuf.append( " is missing )" );
which compiles down to the same code.

Change-Id: I3d8ed0cbf96a881686524a167412d5f303c06b71

change code like
   aStr = OUString("xxxx");
to
   aStr = "xxxx";

Change-Id: Ib981a5cc735677ec5dba76ef9279a107d22e99d4

Change-Id: I56c1190c93333636981acf2dd271515170a8a904

Change-Id: I6da721dc25394dfee12e3028aefbf0546d1be984
Reviewed-on: https://gerrit.libreoffice.org/6669Reviewed-by: Caolán McNamara <caolanm@redhat.com>
Tested-by: Caolán McNamara <caolanm@redhat.com>

Change-Id: Ifb220af71857ddacd64e8204fb6d3e4aad8eef71

This also means that this code now gets bounds checked in debug builds.

Change-Id: Id777f85eaee6a737bbcb84625e6e110abe0e0f27

Change-Id: I56f08d58d8d8a0e397412580451c90f9605bcb46

Change-Id: Ic1240114d667fb7797afae4847427cc889f3cb48

commit 30c303 "Make charmap.cxx compile with icu >= 4.4." was incomplete
and had wrong version checks. After ICU 4.8 (4.8.1.1) the next version
of ICU was 49 (49.1) so U_ICU_VERSION_MAJOR_NUM contains two digets (49),
earlier that it was just one digit (4). The correct header to include to
do version checks is unicode/uversion.h. USCRIPT_MANDAEAN is the old
alias of USCRIPT_MANDAIC (same numeric value). U_JG_FARSI_YEH is only
available since ICU 4.4. Note that on older icu versions (4.2.1) the
200B (ZWSP) Zero Width Space breakiterator testcase fails (others
succeed).

Change-Id: If73c1402239a28546077437e9382f0bd38642bad
Reviewed-on: https://gerrit.libreoffice.org/4139Reviewed-by: Luboš Luňák <l.lunak@suse.cz>
Tested-by: Luboš Luňák <l.lunak@suse.cz>

Mechanical removal of usage together with OUString ctor, done
by compiler plugin.

Change-Id: I554227f76df0dac620b1b46fca32516f78b462c5

Modules sal, salhelper, cppu, cppuhelper, codemaker (selectively) and odk
have kept them, in order not to break external API (the automatic using declaration
is LO-internal).

Change-Id: I588fc9e0c45b914f824f91c0376980621d730f09