5. String Classes and Collation

Symbol

A symbol is similar to a string, but each symbol with a unique set of Characters is guaranteed to have only one canonical instance in GemStone. Symbols are created by a special process, the SymbolGem, to ensure this uniqueness. Creating a new symbol will return an existing symbol, if one exists; a new symbol is only created if it has not been previously defined. Existing symbols cannot be modified.

Like strings, symbols may also contain Characters with values that require more than a byte of storage, and will convert from class Symbol into DoubleByteSymbols or QuadByteSymbols as needed. Since symbols are canonical, the class of a symbol always depends on the contents. While you can create a DoubleByteString with only characters in the range of String, you cannot create a DoubleByteSymbol that does not contain at least one character in the DoubleByte range, and the same is true for QuadByteString.

All symbols may be viewed by all users. Private information should be maintained in strings, not in symbols.

Symbols, DoubleByteSymbols, and QuadByteSymbols are restricted to 1024 or fewer characters.

Symbols that have no references from anywhere in the system may eventually be garbage collected, if the system is configured to do so. See the System Administration Guide for more information on symbol garbage collection.

Symbols, like strings, collate using character-based tables in Legacy String Comparison Mode and using ICU string-based collation in Unicode Comparison Mode. As a result, they cannot be compared to Unicode strings in Legacy String Comparison Mode.

Symbol equality comparisons with = compare identity, while other comparisons (> >= < <= ) use the ICU string-based collation. GemStone symbols that contain the same printing characters but different non-printing characters will return false for = but compare true for >= and <=.

The literal form of a Symbol is specified using a leading #. The body of the symbol may additionally include single quotes. This is optional for symbols that are legal identifiers and keywords, but required for symbols that start with a number, include punctuation/spaces, etc. For example:

#'22 skidoo'

#fooBar

ByteArray

ByteArray is a specialized collection that is restricted to holding Integers between 0 and 255 (inclusive). While ByteArray is not a kind of String, the contents may be interpreted as a String.

Instances of ByteArray can be creating using literal syntax #[]. For example:

#[ 1 2 3 4 ]

Utf8

Utf8 is a subclass of ByteArray. It is not a kind of String, but may easily be converted back and forth from a traditional or Unicode string. A Utf8 holds the UTF-8 encoded bytes created by sending encodeAsUTF8 to a string, or by reading encoded data from a GsFile using contentsAsUTF8. Utf8 instances should not be directly created or edited.

'šamas' encodeAsUTF8

anUtf8( 197, 161, 97, 109, 97, 115)

Instances of Utf8 can be read from and written to instance of GsFile, which cannot directly handle characters with codePoints over 256. FileSystem by default reads and writes in UTF-8 encoding, transparently encoding and decoding to string instances.

Utf16

Utf16, like Utf8 is a subclass of ByteArray that is not a String, but may be converted back and forth from a traditional or Unicode string. A Utf8 holds UTF-16 encoded bytes corresponding to a traditional or Unicode String.

Creating Strings

Strings created as literals, that is, in text encased in single quotes, are invariant; they cannot be modified after they are created.

In addition to creating strings as literals, you can use the inherited instance creation methods, such as new: and withAll:. For example:

String withAll: #($a $z $u $r $e).

azure

Concatenating Strings

A string responds to the comma operator by returning a new string in which the argument to the comma has been appended to the string’s original contents. For example:

'String ' , 'con' , 'catenation'

String concatenation

Although this technique is handy, it’s not very efficient; each #, message send creates a new instance of String, so this example creates three Strings, returning the final one.

To build a string efficiently, by appending onto the original object, you can use add:, which modifies the original string. Note that you cannot start with a literal string, since a literal string is invariant.

For example:

| resultString |

resultString := String new. 

resultString add: 'String ';

             add: 'con';

             add: 'catenation'. 

resultString

%

String concatenation

Converting between String classes and encodings

To convert between UTF-8 encoded bytes and the various kinds of string classes, there are a number of methods:

• Instances of Symbols and Traditional strings can be converted to the lowest-storage type of Unicode string using asUnicodeString.
• Instances of Symbols and Unicode strings can be converted to the lowest-storage type of Traditional strings using asString.
• A Traditional string that is composed of raw UTF-8 encoded bytes can be decoded to a Unicode string using decodeFromUTF8ToUnicode, or to another Traditional string with decoded bytes, using decodeFromUTF8ToString.
• A Traditional string can be encoded into a String containing the raw UTF-8 encoded bytes, using encodeAsUTF8IntoString.
• To convert from a ByteArray containing UTF-8 or from a Utf8 to a Unicode string, use decodeFromUTF8ToUnicode, or to convert to a Traditional string, use decodeFromUTF8ToString.
• Instances of ByteArray and Utf8 may be converted to a Traditional string without decoding by using bytesIntoString.
• All kinds of strings can be encoded to an instance of Utf8 by using encodeAsUTF8.

String Transformations

CharacterCollection and its subclasses define messages that let you perform various conversions.

Strings can be converted in case:

• asUppercase creates a new instance with all uppercase letters
• asLowercase creates a new instance with all lowercase letters
• asTitlecase creates a new instance with the first letter of each word capitalized, the remaining letters lowercase.
• asFoldcase returns a new instance in “fold case”, which is case-free for comparison, and usually is similar to the lowercase.

For example:

'abcde' asUppercase

ABCDE

You can remove leading and/or trailing whitespace separators using methods such as trimSeparators. There are a number of variants; see the image for details.

For example:

'  abcde  ' trimSeparators 

'abcde'

Strings can be split using the subStrings: method, which allows you to specify one or more characters to use as markers.

For example, to split a text into lines with /:

'owa/tagu/siam' subStrings: '/'

anArray( 'owa', 'tagu', 'siam')

Strings can be converted to numbers and other types of objects as well. For example:

'15' asFloat

15.0

Note that not all Strings can be converted to all kinds of other objects; if the String does not contain the representation of a number, for example, it’s meaningless to convert it to an Integer, so this will return an error.

Equality and Identity

Traditional strings are equal to each other if they contain the exact same Characters in the same case; equality is case-sensitive.

Unicode strings compared using = follow the ICU library comparison rules for equality, which are similar, although any non-whitespace control characters (such as null) are ignored for the comparison.

As mentioned above, Traditional strings and Unicode strings cannot be compared to each other for equality using =, when the repository is in Legacy String Comparison Mode. To compare traditional and Unicode strings in any combination, use compareTo:collator:, specifying nil for the collator to indicate the default collator.

Strings can be compared for case-insensitive equality using the methods isEquivalent: or equalsNoCase:.

| nonlitString1 nonlitString2 |

nonlitString1 := String withAll: #($a $b $c).

nonlitString2 := String withAll: #($a $b $c).

(nonlitString1 == nonlitString2)

false

| litString1 litString2 |

litString1 := 'abc'.

litString2 := 'abc'.

(litString1 == litString2)

true

Searching and Pattern matching

CharacterCollection and its subclasses define methods that can tell you whether a string contains a particular sequence of characters and, if so, where the sequence begins. This search can be case sensitive, case insensitive, and may include wild cards.

Below are some common methods; see the image for further methods.

'weimaraner' matchPattern: #('w' $* 'r')

true

'weimaraner' findPattern: #('r' $* 'r') startingAt: 1

6

'w*r' matchPattern: #('weimaraner')

false

'weimaraner' findPattern: #('w*r') startingAt: 1

0

StringConfiguration

The Comparison Mode is controlled by the Global #StringConfiguration. By default, StringConfiguration is set to String, and the repository is therefore in Legacy String Comparison Mode.

To enable Unicode Comparison Mode, as SystemUser, execute:

StringConfiguration enableUnicodeComparisonMode

This returns the previous setting for Unicode Comparison Mode. Note that this commits, but the current session is not affected; the new mode will take effect for all subsequent logins.

To enable Legacy String Comparison Mode, as SystemUser, execute:

StringConfiguration disableUnicodeComparisonMode

Again, note that this operation commits, but the change does not affect the current session; the new mode will take effect for all subsequent logins.

To verify the mode in this repository, execute:

StringConfiguration isInUnicodeComparisonMode

Auto-conversion

When you create or update a kind of Traditional or Unicode string with a Character that requires more bits than the specific class of string can hold, it is transparently auto-converted to the appropriate class.

For example, if you add if you add the Euro character (code point 8364) to an instance of String, which can only hold codePoints up to 255, it will auto-convert to an instance of DoubleByteString. Likewise, if you add an Yen symbol (codePoint 177) to an instance of Unicode7, which can only hold codePoints up to 127, it is auto-converted to an instance of Unicode16.

When the repository is in Unicode Comparison Mode, an instance of String that would otherwise auto-convert to an instance of DoubleByteString is converted to an instance of Unicode16, for improved comparison and collation since comparisons use the ICU libraries.

IcuLocale

Instances of IcuLocale represent a specific language, country, and language variant. The available IcuLocales are in the shared library and can be listed using IcuLocale class >> availableLocales.

A default instance of IcuLocale is instantiated on first reference, and stored in session state. The default IcuLocale is based on the operating system locale setting for the gem. The default IcuLocale affects collation, so some care should be taken in configuring the operating system locale for the gem processes. In applications with distributed locales, it may be safer to set a default IcuLocale on login, using UserProfile >> loginHook: (see the System Administration Guide).

To set a specific default IcuLocale, use the method IcuLocale class >> default:. This sets the default locale for the session executing this code. While the instance of IcuLocale can be made persistent, the default IcuLocale does not persist from session to session.

To determine what IcuLocale is currently in use, use the method IcuLocale >> default.

IcuLocale default

IcuLocale en_US

IcuCollator

An IcuCollator encapsulates the rules involved in collation for a specific IcuLocale. A default instance of IcuCollator is instantiated on first reference, based on the default IcuLocale, and stored in session state.

When comparing instances of Unicode string classes, the comparison always uses an IcuCollator, using the method compareTo:collator:. If an IcuCollator is not specified, such as when Unicode string classes are compared using >, the IcuCollator default is used; which in turn uses IcuLocale default.

You can also create an instance of IcuCollator for a specific locale, if you need to use specific collation rules other than the default. You can do this using IcuCollator class methods forLocale: anIcuLocale or forLocaleNamed: aString. For example, to create an IcuCollator for the German language as used in Germany:

IcuCollator forLocaleNamed: 'de_DE' 

The actual string comparison is done by the ICU libraries, and follows the ICU comparison rules for that locale. Collation rules are similar in most western languages, but there are differences in specific languages.

For example, in the Hungarian language, ’cs’ is considered a single letter, so words that start with ’cs’ are sorted together and follow other words beginning with ’c’. The following example sets up a collection that is sorted according to Hungarian rules:

| hungarianWords collator |

collator := IcuCollator forLocaleNamed: 'hu_HU'.

hungarianWords := IcuSortedCollection newUsingCollator: collator.

hungarianWords 

	add: 'csak' asUnicodeString; 

	add: 'cukor' asUnicodeString; 

	add: 'comb' asUnicodeString.

hungarianWords 

a IcuSortedCollection

  sortBlock           a ExecBlock2

  collator            a IcuCollator

  #1 comb

  #2 cukor

  #3 csak

Customizing Sort

IcuCollator includes a number of attributes that can be used to customize the sort. These attributes work within the specific language rules of the associated IcuLocale.

Keep in mind that while the default values and the descriptions listed in Table 5.2 apply to most locales, particularly with non-Western scripts, the defaults may be different in different locales, and the attribute may have different behaviors.

See the ICU site, particularly the pages under http://userguide.icu-project.org/collation, for more precise descriptions and more detailed documentation.

Strength allows degrees of sort, to consider or not consider things like accent characters and case when performing the sort. The default strength is TERTIARY for most locales (the main exception being Japanese). The following are the sort strengths:

• PRIMARY sorts by primary differences, ignoring secondary and later differences. The base letter represents a primary difference, so for example 'a' and 'b'.
• SECONDARY sorts by primary and secondary differences, ignoring tertiary and later differences. An example of a secondary difference is diacritical differences on the same base letter, for example 'o' and 'ó'.
• TERTIARY sorts by primary, then secondary, then tertiary differences. Uppercase vs. lowercase is a tertiary differences. TERTIARY is the default sort order for most locales.
• QUATERNARY is used in Japanese, where it distinguishes between Japanese Katakana and Hiragana, and can be used to break ties among separator characters when alternateHandling is true.
• IDENTICAL sorts by the specific character, by codepoints in the NFD (Normalization Form Canonical Decomposition) form. There is a performance impact with this strength.

The default sort strength is TERTIARY. As an example, when two strings are compared using TERTIARY strength, characters in the strings are compared first by the base character, ignoring any case or diacritical marks. If the base characters are the same, they are compared by diacritical mark, ignoring case. If both base characters and diacritical marks are the same, then case is considered. Note that unlike GemStone’s Strings or ASCII ordering, the default sorts places lowercase before uppercase.

Keep in mind that with lower sort strengths, when a factor such as case is not used, the relative position in the results of similar strings is not deterministic; the strings compare as the same, and so their position will depend on the order of the input.

By using the IcuCollator sort attributes, you have a great deal of control over your specific sorting.

For example, using the alternative handling example, you can sort strings that include spaces, dashes and other punctuation without considering the punctuation characters when doing the comparison:

| blues collator|

collator := IcuCollator forLocale: IcuLocale default.

collator alternateHandling: true.

blues := IcuSortedCollection newUsingCollator: collator.

blues add: (Unicode7 withAll: 'blue berry').

blues add: (Unicode7 withAll: 'blue moon').

blues add: (Unicode7 withAll: 'bluebird').

blues add: (Unicode7 withAll: 'blue bird').

blues add: (Unicode7 withAll: 'blue-bird').

blues add: (Unicode7 withAll: 'bluetooth').

blues

%

a IcuSortedCollection

  sortBlock           a ExecBlock2

  collator            a IcuCollator

  #1 blue berry

  #2 bluebird

  #3 blue bird

  #4 blue-bird

  #5 blue moon

  #6 bluetooth

IcuSortedCollection

An IcuSortedCollection is a specialized subclass of SortedCollection for which you do not set the sortBlock. An IcuSortedCollection may only hold instances of subclasses of CharacterCollection. It is associated with a IcuCollator, which in turn is associated with an IcuLocale, and the sorting behavior is specific to the configuration of these instances. IcuSortedCollections rely on the open-source ICU libraries to perform the comparisons and produce correctly collated results.

Using IcuSortedCollection is recommended if you will have sorted collections containing Unicode strings. This avoids lookup failures if a different collator is used to lookup than was used to sort the elements in the collection.

ICU and Unicode versioning

The Unicode Consortium periodically releases new versions of the Unicode Standard, with (usually minor) changes in collation and the addition of new characters. The ICU organization then periodically releases new versions of their libraries reflecting these changes in the standard. Major GemStone releases include the latest version of the ICU libraries.

The indexing structures depend on collation encodings from ICU that may change between versions, even if the collation changes would not otherwise affect the application. So even in cases where the Unicode differences are minor, the ICU library version loaded in an application must match the ICU version used to build indexes.

To accommodate the (generally) low value of upgrading to a new ICU library, and the potentially high cost of rebuilding structures in your application that depend on collation, GemStone preserves the existing ICU library version over upgrade.

IcuLibraryVersion

The version of the ICU library that is used in a repository is stored under (Globals at: #IcuLibraryVersion). This is a string, which must correspond to one of the versions of the ICU libraries in the product distribution. When a session logs in, it will select the ICU shared libraries to load based on the IcuLibraryVersion value.

As with StringConfiguration, IcuLibraryVersion is a global, repository-wide setting that can be only changed by SystemUser, to avoid the risk of lookup failures and incorrect query results. It should be managed as part of System Administration, not by individual developers on a shared repository.

Globals at: #IcuLibraryVersion put: newVersionString