Text.tokenize add unicode support ,fix Bug

[HonestManXin]

When the string argument of function Text.tokenize's length is 1, then the tokens list will be a empty list.

[titan-cla]

Hi @HonestManXin, thanks for your contribution!

In order for us to evaluate and accept your PR, we ask that you sign a contribution license agreement. It's all electronic and will take just minutes.

[titan-cla]

You did it @HonestManXin!

Thank you for signing the Contribution License Agreement.

[graben1437]

Hi,
This is a nice suggested fix.
I pulled this into my https://github.com/graben1437/titan1withtp3.1.git build but am wondering if you can also provide/suggest a new test case that "breaks" with the old code but works with the new code ?

[HonestManXin]

I just found this Java Character API Documentation，

The methods that only accept a char value cannot support supplementary characters. They treat char values from the surrogate ranges as undefined characters. For example, Character.isLetter('\uD840') returns false, even though this specific value if followed by any low-surrogate value in a string would represent a letter.
The methods that accept an int value support all Unicode characters, including supplementary characters. For example, Character.isLetter(0x2F81A) returns true because the code point value represents a letter (a CJK ideograph).

And I'm so sorry, this still have some problems, the proper implementation would be below.

public static List<String> tokenize(String str) {
        ArrayList<String> tokens = new ArrayList<String>();
        int previous = 0;
        int codePoint;
        for (int p = 0; p < str.length(); p += Character.charCount(codePoint)) {
            codePoint = str.codePointAt(p);
            if (!Character.isLetterOrDigit(codePoint)) {
                if (p > previous + MIN_TOKEN_LENGTH) tokens.add(str.substring(previous, p));
                previous = p + Character.charCount(codePoint);
            }
        }
        if (previous + MIN_TOKEN_LENGTH <= str.length()) tokens.add(str.substring(previous, str.length()));
        return tokens;
    }