1. GemStone/S 64 Bit 3.4 Release Notes

New keyfiles required

New keyfiles are required for version 3.4. To obtain a new keyfile for GemStone/S v3.4, write to keyfiles@gemtalksystems.com. In your request, include your license information, platform and any updates to contact information. It may be helpful to include your current keyfile with the request.

Contact GemTalk Technical Support if you have issues or questions.

Platforms for Version 3.4

GemStone/S 64 Bit version 3.4 is supported on the following platforms:

• Red Hat Enterprise Linux Server 6.7 and 7.1,
  Ubuntu 14.04 and 16.04, and SUSE Linux Enterprise 12, on x86
• Solaris 10 and 11.3 on x86
• AIX 6.1 on POWER6 and POWER7 and AIX 7.1 on POWER8
• OS X 10.11.2 (El Capitan) with Darwin 15.2.0 kernel, and
  OS X 10.12.6 (Sierra), with Darwin 16.7.0 kernel, on x86 (Mac is supported for development only)

Note that Solaris on SPARC is no longer a supported platform, although distributions are available for development or debugging.

For more information and detailed requirements for each supported platforms, please refer to the GemStone/S 64 Bit v3.4 Installation Guide for that platform.

GemBuilder for Smalltalk (GBS) Versions

GemStone/S 64 Bit version 3.4 requires GBS version 8.3 or later for VisualWorks Smalltalk, or version 5.4.4 or later for VA Smalltalk.

The following versions of GBS are supported with GemStone/S 64 Bit version 3.4:

VSD Version

The GemStone/S 64 Bit v3.4 distribution includes VSD version 5.3.1; The previous version of GemStone/S 64 Bit, v3.3.6, included VSD v5.3. Changes between these versions are minor, and include:

• updated zlib libraries.
• added definitions for new cache statistics.
• a bug fix in reading empty timestamps in statmonitor data records.

See the Release Notes for VSD v5.3.1, or the VSD home page.

Other Products

The current versions of GemBuilder for Java (GBJ) and GemConnect can be used with GemStone/S 64 Bit v3.4.

Documentation Changes

All documentation has been revised for this release, with modifications to incorporate new and changed features, as well as corrections and improvements.

In addition to the maintenance changes, the following improvements have been made:

• The GemBuilder for C manual is now is now available in .html as well as .pdf, and has been extensively updated for clarity and organization. The first chapter has been split into an introductory and a programming chapter. Compile and link information is by platform rather than by operation.
• Chapters 1-4 of the System Administration Guide have been extensively reorganized, and material moved to two added chapters: an introductory chapter and a separate chapter for the NetLDI and interprocess connectivity. Some material has been moved to the expanded NRS appendix.
• The Programming Guide chapters on Collections, Strings, and Indexing have been extensively revised for clarity and organization, as well as including new material. More information is provided on behavior for repositories in Unicode Comparison Mode.

Updated library versions

• The version of OpenSSL has been updated to v1.1.0f.
• The version of Zlib has been updated to 1.2.11.
• ICU version 58.2 is included (in addition to the older ICU versions), and used for new repositories.
• The version of OpenLDAP has been updated to 2.4.45
• The version of LZ4 has been updated to 1.7.5

Library distribution change

Previously, the GemStone distribution included separate shared libraries for OpenSSL and LDAP, and (in 3.4 alpha versions) Kerberos. These libraries are now combined into a single file, libfloss (floss stands for Free/Libre Open Source Software).

This does not apply to Windows clients; there are no changes in distribution of Windows client libraries, other than version numbers.

The ICU libraries are not included in the libfloss library, since GemStone may load different versions of the ICU library.

The ICU libraries are now combined into a single file, rather than in three separate files as in previous releases. This only applies for the latest version of ICU, v58.2. The v3.4 distribution includes ICU v51.2 (for 3.2.x) and 54.1 (for 3.3.x), which continue to be distributed as three separate files.

Updated ICU shared library loading

GemStone/S 64 Bit v3.4 includes libraries for ICU versions 58.2, 54.1 and 51.2. Control over the ICU version that is used is managed similarly to the simplified process introduced in v3.3.5.

On AIX, is it no longer required to perform the manuals steps to ensure the correct libraries are loaded; ICU library loading is handled on AIX as it is on other platforms.

No environment variable or configuration parameter is required. The global variable (Globals at: #IcuLibraryVersion) controls the version of the shared library that is loaded. During upgrade, no special processing is needed; the code determines your previous version and sets (Globals at: #IcuLibraryVersion) appropriately.

However, there are a large number of possible upgrade paths. If you have created persistent structures depending on collation using Unicode strings, or any strings if the application in Unicode Comparison Mode, in v3.1.x or 3.0, you may need to rebuild/resort these structures. Contact GemTalk Technical Support if you think this case may apply.

The following ICU versions will be used:

• (Globals at: #IcuLibraryVersion), if set, will override the version set during upgrade.
• New applications (originating in v3.4) will use ICU v58.2, as will applications upgraded from 2.x or 3.0.x directly to 3.4.
• (Globals at: #IcuLibraryVersion) is automatically set during upgrade to v3.3.1 or later. Applications upgraded from 3.3.x to 3.4 will use ICU v54.1 or 51.2, depending on the previous upgrade.
• Applications upgraded directly from v3.2.x to 3.4 will use ICU v51.2.
• Applications upgraded directly from v3.1.x will use ICU v58.2, and require rebuild/resort of ICU-dependent structures.

During the startstone step of upgrade, information regarding the handling of the IcuLibraryVersion, and related processing, is written to the SymbolGem log (stoneName_PIDsymbolgem.log). This upgrade SymbolGem log is not deleted as is normally the default for SymbolGem logs.

Updated ZoneInfo TimeZone database

GemStone includes the zoneinfo TimeZone database. The has been update to version 2017d.

Updated SSL example certificates

GemStone distribution includes OpenSSL example certificates under:

$GEMSTONE/examples/openssl

These are used in examples for secure backup and GsSecureSocket example methods.

All the example certificates have been replaced. The directory structure includes an expanded directory structure as produced by openSSL.

There are now multiple client server example certificates, private keys, and password files. These certificates have been updated with improved security:

• 4096 bit RSA keys; previously they were 1024 bit.
• sha256 hashing; previously they were sha1.
• Add strong 32 character random passwords for private keys; previously they were short, predictable passwords.
• keys without passphrases are no longer distributed.

$GEMSTONE/examples/openssl/config/openssl.cnf

create_ca.sh

create_new_cert.sh

Web/Community Edition license distribution on non-GLASS platforms

The GLASS/Seaside open source project and the Web Edition/Community Edition licenses are only supported for Linux and Macintosh. The free license distributed under the $GEMSTONE/seaside directory is no longer included with the AIX and Solaris distributions.

Option to update the keyfile without stopping the stone

Applications that do not have scheduled downtime may not easily be able to install a new keyfile with, for example, an updated Sunset date.

Methods have been added to allow a new keyfile to be applied without the need to shutdown and restart the stone.

System class >> applyKeyfileNamed: keyfilenameOrNil
Causes the stone to read the keyfile keyfilenameOrNil. If nil, the stone will use the same file name used when the stone was started.

The new keyfile permissions apply only as long as the stone is running. To permanently use the new keyfile, modify the KEYFILE option in the stone configuration file.

Returns true upon success. In this case, the keyfile was read and one or more changes were made to the system. Returns false if the keyfile was successfully read but contained no meaningful changes from the current configuration.

Raises an exception if the keyfile could not be read, has expired, or if applying it would modify parameters which may not be changed at runtime.

System class >> getKeyfileAttributes
Returns a SymbolDictionary which contains attributes of the current keyfile. See the image method for specific keys.

Primitives optimized

A number of primitives, including often-used ones like at:, have been optimized for better performance.

TLS/SSL now used over remote connection from GCI client to NetLDI

TLS/SSL is now enabled for the GCI client to NetLDI connection if the GCI is on a different machine than the NetLDI.

Note that this effects the handling of client library version mismatch errors; on a version mismatch between 3.4 and an older version, the error reported will be an SSL error or similar handshake error, rather than a version mismatch.

Transport Layer Security (TLS) is the successor to Secure Sockets Layer (SSL). GemStone continues to use the term "SSL" in most cases; SSL is commonly used to refer to both, and is more widely known.

More efficient transaction logging

A number of changes have been made to reduce the volume of tranlog data related to updates to collections, and new primitives have been added to support this. Specific optimizations are to:

• Operations on btrees that support indexes
• Add to and remove from IdentityBag and IdentitySet
• Inserts into small SequenceableCollections

The incremental logging of objects has been reduced from 1/16 to 1/32 of an object.

To reduce the space required for commits, the new configuration option GEM_COMPRESS_TRANLOG_RECORDS allows you to write lz4 compressed tranlog records. This is true by default; the transaction logs generated in v3.4 are significantly smaller than those generated without compression.

The new configuration option STN_GROUP_COMMITS allows you to group commits into a tranlog write.

The logsender in a hot standby system can transmit the record-level compressed transaction logs, and can now also read manually file-level compressed tranlogs (see bugs here).

Multithreaded mid-cache page servers

Similarly to how page servers were multithreaded in previous releases, the mid-cache page servers are now multithreaded. Each client gem on the mid cache host will have a thread in a single mid-level cache page server.

Listening ports in for mid-cache page servers come from the pool specified by STN_PGSVR_PORT_RANGE.

Commit Record disposal without disk I/O

Using the existing configuration STN_COMMIT_RECORD_QUEUE_SIZE, commit record pages may be cached in memory, avoiding the need for disk I/O to dispose. However, bitmaps that are too big to compress onto the commit records still require disk access.

To avoid this, the configuration parameter STN_COMMIT_RECORD_BM_CACHING has been added. This is true by default and allows reduced I/O during commit record disposal, most important when the commit record backlog is high or there is a lot of page preemption in the shared cache.

This setting does reduce the maximum commit rate. If your commit record backlog is usually small, and you need the maximum commit performance, set this to false.

Using multiple sizes of large memory pages on Linux

Newer Linux systems may be able to support both 2GB and 1GB large pages. Previously, it was required that the desired GemStone large page size be set as the default large page size for the system.

Now, GemStone can be configured to use either size of pages without resetting the OS default page size, using a new configuration parameter SHR_PAGE_CACHE_LARGE_MEMORY_PAGE_SIZE_MB.

This only applies to Linux. On AIX, GemStone only supports 16MB large pages (in addition to normal page sizes), so this configuration parameter does not apply. Other platforms do not require special large page configuration.

See SHR_PAGE_CACHE_LARGE_MEMORY_PAGE_SIZE_MB for details.

Single sign-on using Kerberos

GemStone now supports Single Sign-On (SSO) using Kerberos. With SSO, logins to GemStone do not require a GemStone password; GemStone uses a current Kerberos ticket to authenticate the GemStone userId.

The NetLDI can also be configured to use Kerberos to authenticate the Unix userId, when authentications are required (that is, when the NetLDI is not in guest mode).

There are a number of ways to configure one or more UserIds for single sign-on authentication, including by group membership.

A new class, UserProfileGroup, replaces the old use of Strings as groups, and group manipulation can now be done more logically.

For details on all these changes, see Single sign-on with Kerberos.

Secure backups

Using secure backups, you can create a backup file that is digitally signed and optionally encrypted. Creating the backup requires one or more RSA keypairs, for signing and encrypting; DSA is not supported. One or up to eight public keys is used for encryption; subsequently restoring the backup requires any one of the matching private keys.

Secure backups are created and restored using specific protocol. These methods support the same features as regular backups; multiple files, compression, etc. Secure backups are created with the extension ’.sdbf’.

copydbf has added options to report and extract information from a secure backup.

For details on secure backups, see Secure Backups.

Cache warming the working set of data pages

Cache warming of data pages traditionally loaded pages by pageId, irrespective of whether the pages with lower pageIds were useful or not; this meant it was of limited value if the cache was not large enough to hold the entire repository. Now, you can configure your system to reload the specific set of data pages that was previously in the cache. See Cache warming a working set.

In addition, there is an improved way to configure cache warming to run on startup, and startstone and waitstone can wait for cache warming to complete.

Backup, restore, copydbf, and statmonitor now support LZ4

GemStone now supports LZ4 as well as gzip for backup and other compression operations. LZ4 has been used for GemStone interprocess compression for several releases, and provides much better compression performance. LZ4-compressed files have the extension .lz4 while gzip-compressed files have the .gz extension. The API has been modified to allow specification of the compression format.

See Backup and restore support lz4 compression, copydbf supports lz4 compression, and statmonitor supports lz4 compression.

GsBitmaps

GsBitmaps are an abstraction for the old hidden set internal bitmap structures that were accessible via System class API. The public hidden set API is now "legacy" (with the exception of a few operations that have a distinct API), and may be deprecated in a future release.

GsBitmap provides a collection-like API, but cannot include specials or temporary objects and cannot be committed. Like hidden sets, GsBitmaps are in C heap and do not use temporary object memory.

A number of operations that previously put results in hidden sets now return instances of GsBitmap.

For details, see Bitmap based operations with new class GsBitmap.

New Reduced-Conflict Collection classes

A number of new RC classes have been added that use replay to resolve certain kinds of conflict, rather than relying on session-based subcollections. These classes are more efficient, and do not require maintenance to avoid inadvertently retaining references to obsolete objects; but may avoid conflict for only a subset of collection operations.

If multiple sessions have resolvable conflicts at a rapid rate, it is possible for replay-based conflict resolution to have performance issues, since replay for the same conflict may need to be performed multiple times. In these cases, a session based subcollection structure may be more efficient.

The new reduced-conflict collection classes include:

• RcArray
• GsPipe and RcPipe, with FIFO queue semantics similar to RcQueue
• RcIdentitySet
• RcLowMaintenanceIdentityBag, with semantics of RcIdentityBag but with a replay rather than session-subcollection based implementation, and intended to eventually replace RcIdentityBag.

Detailed descriptions of these classes start here.

Indexing changes

The indexing system has an additional internal structure that can significantly improve performance for queries. Both the new btreePlusIndex and the traditional legacyIndex internal infrastructures are available.

By default, upgraded applications will continue to use legacyIndex. It is simple to rebuild using btreePlusIndexes, but performance testing should be done, and there are element type restrictions required to get the best performance using optimizedComparison.

The type of indexing infrastructure is configured using GsIndexOptions, which has an expanded and clarified default and combination semantics. You may now set a system or session default for GsIndexOptions.

The new replay-based RC UnoderedCollections, RcIdentitySet and RcLowMaintenanceIdentityBag, only support indexes of type btreePlusIndex, not legacyIndex.

Queries can now be run on instances of collections that do not support indexes, such as Array.

Indexing changes are described under Indexing changes.

Classes no longer in Globals

The class GsMethod has been moved to ObsoleteClasses.

Replaced methods

System class >> _sharedCounter:incrementBy:

System class >> _sharedCounter:setValue:

Methods deprecated in v3.4

Repository >> auditPageOrderOopFileWithId:

Repository >> closePageOrderOopFileWithId:

Repository >> numberOfObjectsInPageOrderOopFileWithId:

Repository >> openPageOrderOopFile:

Repository >> readObjectsFromFileWithId:startingAt:upTo:into:

Repository >> _pagesWithPercentFree:doScavenge:

Repository >> _validateSegmentId:

System class >> fastFindObjectsLargerThan:limit:

System class >> findObjectsLargerThan:limit:

UserProfileSet >> membersOfGroup:

Removed Deprecated Methods

IcuCollator >> hiraganaQuarternary:

ProfMonitor >> spyOn:

Repository >> setArchiveLogDirectories:

Repository >> setArchiveLogDirectory:

UnorderedCollection >> createEqualityIndexOn:withLastElementClass:commitInterval:

UnorderedCollection >> createIdentityIndexOn:commitInterval:

UnorderedCollection >> createEqualityIndexOn:commitInterval:

UnorderedCollection >> createEqualityIndexOn:

UnorderedCollection >> createRcEqualityIndexOn:

Behavior >> addInstVar:withConstraint:

Behavior >> allConstraints

Behavior >> constraintOfInstVar:

Behavior >> constraintOn:

Behavior >> elementConstraint

Behavior >> instVar:constrainTo:

Behavior >> varyingConstraint

Behavior >> varyingConstraint:

Dictionary >> constraint

UnorderedCollection >> getLastElementConstraintOnPath:

Removed Ruby Methods

These are replace by more general environment-based methods, see Support for multiple execution environments.

Behavior >> isRubySingletonClass

Behavior >> rubyPrimaryCopy

Behavior >> rubySuperclass:

GsNMethod >> isRubyBridgeMethod

GsNMethod >> rubyMethodProtection

GsNMethod >> rubyOptArgsBits

GsNMethod >> rubyPrivate

GsNMethod >> rubyProtected

Module >> rubyHierarchy:

Module >> rubySuperclass:

Removed Private Methods

The following private methods have been removed