5. String Classes and Collation

Installing Character Data Tables

The following methods can only be executed by SystemUser. After commit, the installed tables apply to all logins for the repository. Any indexes or collections that depend on string collation must be rebuilt after installing new tables. Installed tables are not affected by upgrade or conversion. The tables distributed with GemStone are based on Unicode version 5.1; note that this is an older version of the Unicode standard.

• To install a 0...255 character table in GemStone legacy collate order:

Character installCharTables: (PassiveObject fromServerTextFile:

		'$GEMSONE/examples/CharTableDefault.tab') activate.

• To install the full Unicode character table in GemStone legacy collate order:

Character activateCharTablesFromFile:

	'$GEMSTONE/examples/CharTableUnicode510.dat'.

• To install the full Unicode character data table in Unicode DUCET collate order:

Character activateCharTablesFromFile:

	'$GEMSTONE/examples/CharTableUnicode410.dat'

• To reset to the default internal character table:

Globals removeKey: #CharacterDataTables.

• To disable loading of installed character tables on session login, set the following environment variable:

export GS_DISABLE_CHARACTER_TABLE_LOAD=TRUE

Strings

These classes are traditional strings.

String
Strings hold Characters with codepoints in the range 0..255.

DoubleByteString
DoubleByteStrings are required when one or more Characters in a string needs more than one byte of storage. DoubleByteStrings hold Characters with codepoints in the range 0...16rFFFF (64K).

QuadByteString
QuadByteStrings are required when one or more Characters in a string needs more than two bytes of storage. QuadByteStrings hold Characters with codepoints in the range 0...16r10FFFF.

While traditional Strings normally hold human-readable text characters, this is not a requirement. Generally, raw byte data would be held in an instance of ByteArray, but it may be more convenient to use a String. In particular, there are cases when an instance of String will hold raw UTF-8 encoded bytes.

Unicode Strings

For strings that require locale-specific collation, specialized subclasses of the String classes are provided. These classes rely on the open-source ICU libraries to provide comparison and sorting behavior.

Unicode7
A subclass of String, limited to holding Characters with codepoints in the range 0..127 that are represented in 7 bits.

Unicode16
A subclass of DoubleByteString, holding Characters with codepoints in the range 0...16rFFFF (64K), excluding the range 16rD800-16rDFFF. This range is reserved for surrogates that allow encoding into UTF-16.

Unicode32
A subclass of QuadByteString, holding Characters with codepoints in the range 0..16r10FFFF. Again, this excludes the range 16rD800-16rDFFF.

Unicode strings should not hold raw byte data.

Symbol

Class Symbol is a subclass of String. Each Symbol with a unique set of Characters is guaranteed to have only one canonical instance in GemStone. Symbols are created by a special process, the SymbolGem, to ensure this uniqueness.

Like Strings, symbols may also contain Characters with values that require more than a byte of storage, and will convert into DoubleByteSymbols or QuadByteSymbols as needed. GemStone Smalltalk uses symbols internally to represent variable names and selectors. All symbols may be viewed by all users. Private information should be maintained in Strings, not in Symbols.

Symbols, DoubleByteSymbols, and QuadByteSymbols are restricted to 1024 or fewer characters.

You can “create” symbols using asSymbol or withAll: method. If the Symbol was created previously as part of the GemStone kernel, by another user, or by yourself, you will get the existing Symbol. A new symbol is only created if it has not been previously defined. Existing Symbols cannot be modified.

Since Symbols are canonical, the class of a Symbol always depends on the contents. While you can create a DoubleByteString with only characters in the range of String, you cannot create a DoubleByteSymbol that does not contain at least one character in the DoubleByte range, and the same is true for QuadByteString.

Symbols that have no references from anywhere in the system will eventually be garbage collected, if the system is configured to do so. See the System Administration Guide for more information on symbol garbage collection.

ByteArray

ByteArray is a specialized collection that is restricted to holding Integers between 0 and 255 (inclusive). While ByteArray is not a kind of String, the contents may be interpreted as a String.

Instances of ByteArray can be creating using literal syntax #[]. For example:

#[ 1 2 3 4 ]

Utf8

Utf8 is a subclass of ByteArray. It is not a kind of String, but may easily be converted back and forth from a traditional or Unicode String. A Utf8 holds the UTF-8 encoded bytes created by sending encodeAsUTF8 to a string, or by reading encoded data from a GsFile using contentsAsUTF8. Utf8 instances should not be directly created or edited.

'šamas' encodeAsUTF8

anUtf8( 197, 161, 97, 109, 97, 115)

Instances of Utf8 can be read from and written to instance of GsFile, which cannot directly handle characters with codePoints over 256.

Creating Strings

You have already seen strings created as literals. Strings created as literals are invariant; they cannot be modified after they are created.

In addition to creating strings literally, you can use the inherited instance creation methods, such as new: and withAll:. For example:

| myString |

myString := String withAll: #($a $z $u $r $e).

myString

azure

To create a string that can be modified later, you can use withAll: with a literal String:

| myString |

myString := String withAll: 'azure'.

myString at: 1 put: $A.

myString

Azure

Concatenating Strings

A string responds to the comma operator by returning a new string in which the argument to the comma has been appended to the string’s original contents. For example:

'String ' , 'con' , 'catenation'

String concatenation

Although this technique is handy when you need to build a small string, it’s not very efficient. In the last example, GemStone Smalltalk creates a String object for the first literal, 'String'. The #, message returns a new instance of String containing 'String con', and the second #, message creates a third string.

When you need to build a longer string, you’ll find it more efficient to use addAll: or add: (they’re the same for class String), which modifies the original string. Note that you cannot start with a literal string, since a literal string is invariant.

For example:

| resultString |

resultString := String new. 

resultString add: 'String ';

             add: 'con';

             add: 'catenation'. 

resultString

%

String concatenation

Converting between String classes and encodings

To convert between UTF-8 encoded bytes and the various kinds of string classes, there are a number of methods:

• Instances of Symbols and traditional Strings can be converted to the lowest-storage type of Unicode string using asUnicodeString.
• Instances of Symbols and Unicode strings can be converted to the lowest-storage type of traditional strings using asString.
• A traditional String that is compose of raw UTF-8 encoded bytes can be decoded to a Unicode string using decodeFromUTF8ToUnicode, or to another traditional String with decoded bytes, using decodeFromUTF8ToString.
• To convert from a ByteArray or Utf8 to a Unicode string, use decodeToUnicode, or to convert to a traditional String, use decodeToString.
• Instances of ByteArray and Utf8 may be converted to a traditional String without decoding by using bytesIntoString.
• All kinds of Strings can be encoded to an instance of Utf8 by using encodeAsUTF8.

String Transformations

CharacterCollection and its subclasses define messages that let you perform various conversions.

Strings can be converted in case:

• asUppercase creates a new instance with all uppercase letters
• asLowercase creates a new instance with all lowercase letters
• asTitlecase creates a new instance with the first letter of each word capitalized, the remaining letters lowercase.
• asFoldcase returns a new instance in “fold case”, which is case-free for comparison, and usually is similar to the lowercase.

For example:

'abcde' asUppercase

ABCDE

You can remove leading and/or trailing whitespace separators using methods such as trimSeparators. There are a number of variants; see the image for details.

For example:

'  abcde  ' trimSeparators 

'abcde'

Strings can be split using the subStrings: method, which allows you to specify one or more characters to use as markers.

For example, to split a text into lines with /:

'owa/tagu/siam' subStrings: '/'

anArray( 'owa', 'tagu', 'siam')

Strings can be converted to numbers and other types of objects as well. Note that not all Strings can be converted to all kinds of other objects —if the String does not contain the representation of a number, for example, it’s meaningless to convert it to an Integer, so this will return an error.

For example:

'15' asFloat

15.0

Equality and Identity

Traditional strings are equal to each other if they contain the exact same Characters in the same case; equality is case-sensitive.

Unicode strings compared using = follow the ICU library comparison rules for equality, which are similar, although any non-whitespace control characters (such as null) are ignored for the comparison.

Traditional strings and Unicode strings cannot be compared to each other for equality using =. Any comparison involving a Unicode string require a collator. To compare traditional and Unicode strings in any combination, use compareTo:collator:. Specifying nil for the collator uses the default collator.

Strings can be compared for case-insensitive equality using the methods isEquivalent: or equalsNoCase:.

| nonlitString1 nonlitString2 |

nonlitString1 := String withAll: #($a $b $c).

nonlitString2 := String withAll: #($a $b $c).

(nonlitString1 == nonlitString2)

false

| litString1 litString2 |

litString1 := 'abc'.

litString2 := 'abc'.

(litString1 == litString2)

true

Searching and Pattern matching

CharacterCollection and its subclasses define methods that can tell you whether a string contains a particular sequence of characters and, if so, where the sequence begins. This search can be case sensitive, case insensitive, and may include wild cards.

Below are some common methods; see the image for further methods.

'weimaraner' matchPattern: #('w' $* 'r')

true

'weimaraner' findPattern: #('r' $* 'r') startingAt: 1

6

'w*r' matchPattern: #('weimaraner')

false

'weimaraner' findPattern: #('w*r') startingAt: 1

0

IcuLocale

Instances of IcuLocale represent a specific language, country, and language variant. The available IcuLocales are in the shared library and can be listed using IcuLocale class >> availableLocales.

A default instance of IcuLocale is instantiated on first reference, and stored in session state. The default IcuLocale is based on the operating system locale setting for the gem.

To set a specific default IcuLocale, use the method IcuLocale class >> default:. This sets the default locale for the session executing this code. While the instance of IcuLocale can be made persistent, the default IcuLocale does not persist from session to session.

IcuLocale class >> getUS is an example of instantiating an IcuLocale for the language English in the country US.

To determine what IcuLocale is currently in use, use the method IcuLocale >> default.

topaz 1> printit

IcuLocale default

%

a IcuLocale

  name                en_US

IcuCollator

An IcuCollator encapsulates the rules involved in collation for a specific IcuLocale. A default instance of IcuCollator is instantiated on first reference, based on the default IcuLocale, and stored in session state.

When comparing instances of Unicode String classes, the comparison always uses an IcuCollator, using the method compareTo:collator:. If an IcuCollator is not specified, such as when Unicode String classes are compared using >, the IcuCollator default is used; which in turn uses IcuLocale default.

You can also create an instance of IcuCollator for a specific locale, if you need to use specific collation rules other than the default. You can do this using IcuCollator class methods forLocale: anIcuLocale or forLocaleNamed: aString. For example, to create an IcuCollator for the German language as spoken in Germany:

IcuCollator forLocaleNamed: 'de_DE' 

The actual string comparison is done by the ICU libraries, and follows the ICU comparison rules for that locale. Collation rules are similar in most western languages, but there are differences in specific languages.

For example, in the Hungarian language, ’cs’ is considered a single letter, so words that start with ’cs’ are sorted together and follow other words beginning with ’c’. The following example sets up a collection that is sorted according to Hungarian rules:

| hungarianWords collator |

collator := IcuCollator forLocaleNamed: 'hu_HU'.

hungarianWords := IcuSortedCollection newUsingCollator: collator.

hungarianWords 

	add: 'csak' asUnicodeString; 

	add: 'cukor' asUnicodeString; 

	add: 'comb' asUnicodeString.

hungarianWords 

a IcuSortedCollection

  sortBlock           a ExecBlock2

  collator            a IcuCollator

  #1 comb

  #2 cukor

  #3 csak

Customizing Sort

IcuCollator includes a number of attributes that can be used to customize the sort. These attributes work within the specific language rules of the associated IcuLocale.

Keep in mind that while the default values and the descriptions listed in Table 5.2 apply to most locales, particularly with non-Western scripts, the defaults may be different in different locales, and the attribute may have different behaviors.

See the ICU site, particularly the pages under http://userguide.icu-project.org/collation, for more precise descriptions and more detailed documentation.

Strength allows degrees of sort, to consider or not consider things like accent characters and case when performing the sort. The default strength is TERTIARY for most locales (the main exception being Japanese). The following are the sort strengths:

• PRIMARY sorts by primary differences, ignoring secondary and later differences. The base letter represents a primary difference, so for example 'a' and 'b'.
• SECONDARY sorts by primary and secondary differences, ignoring tertiary and later differences. An example of a secondary difference is diacritical differences on the same base letter, for example 'o' and 'ó'.
• TERTIARY sorts by primary, then secondary, then tertiary differences. Uppercase vs. lowercase is a tertiary differences. TERTIARY is the default sort order for most locales.
• QUATERNARY is used in Japanese, where it distinguishes between Japanese Katakana and Hiragana, and can be used to break ties among separator characters when alternateHandling is true.
• IDENTICAL sorts by the specific character, by codepoints in the NFD (Normalization Form Canonical Decomposition) form. There is a performance impact with this strength.

The default sort strength is TERTIARY. As an example, when two strings are compared using TERTIARY strength, characters in the strings are compared first by the base character, ignoring any case or diacritical marks. If the base characters are the same, they are compared by diacritical mark, ignoring case. If both base characters and diacritical marks are the same, then case is considered. Note that unlike GemStone’s Strings or ASCII ordering, the default sorts places lowercase before uppercase.

Keep in mind that with lower sort strengths, when a factor such as case is not used, the relative position in the results of similar strings is not deterministic; the strings compare as the same, and so their position will depend on the order of the input.

By using the IcuCollator sort attributes, you have a great deal of control over your specific sorting.

For example, using the alternative handling example, you can sort strings that include spaces, dashes and other punctuation without considering the punctuation characters when doing the comparison:

| blues collator|

collator := IcuCollator forLocale: IcuLocale default.

collator alternateHandling: true.

blues := IcuSortedCollection newUsingCollator: collator.

blues add: (Unicode7 withAll: 'blue berry').

blues add: (Unicode7 withAll: 'blue moon').

blues add: (Unicode7 withAll: 'bluebird').

blues add: (Unicode7 withAll: 'blue bird').

blues add: (Unicode7 withAll: 'blue-bird').

blues add: (Unicode7 withAll: 'bluetooth').

blues

%

a IcuSortedCollection

  sortBlock           a ExecBlock2

  collator            a IcuCollator

  #1 blue berry

  #2 bluebird

  #3 blue bird

  #4 blue-bird

  #5 blue moon

  #6 bluetooth

IcuSortedCollection

An IcuSortedCollection is a specialized subclass of SortedCollection for which you do not set the sortBlock. An IcuSortedCollection may only hold instances of subclasses of CharacterCollection. It is associated with a IcuCollator, which in turn is associated with an IcuLocale, and the sorting behavior is specific to the configuration of these instances. IcuSortedCollections rely on the open-source ICU libraries to perform the comparisons and produce correctly collated results.

Using IcuCollator is recommended if you will have sorted collections containing Unicode strings. This avoids lookup failures if a different collator is used to lookup than was used to sort the elements in the collection.