1. GemStone/S 64 Bit 3.5.2 Release Notes

Platforms for Version 3.5.2

GemStone/S 64 Bit version 3.5.2 is supported on the following platforms:

• Red Hat Enterprise Linux Server 6.9, 7.4, and 7.6,
  Ubuntu 16.04 and 18.04, and SUSE Linux Enterprise 12, on x86.
  Testing has been done on some earlier Red Hat Enterprise Linux Server 6.x and 7.x versions with the latest security patches.
• Solaris 10 and 11.4 on x86
• AIX 6.1, 7.1, and 7.2
• OS X 10.14.3 (Mojave) with Darwin 18.2.0 kernel, and OSX 10.15.1 (Catalina) with Darwin 19.0.0 kernel, on x86
  (Mac is supported for development only)

For more information and detailed requirements for each supported platforms, please refer to the GemStone/S 64 Bit v3.5.1 Installation Guide for that platform.

GemBuilder for Smalltalk (GBS) Versions

GemStone/S 64 Bit version 3.5.2 requires GBS version 8.4 or later for VisualWorks Smalltalk, or version 5.4.5 or later for VA Smalltalk.

The following versions of GBS are supported with GemStone/S 64 Bit version 3.5.2:

VSD Version

The GemStone/S 64 Bit v3.5.2 distribution includes VSD v5.5. The previous version of GemStone/S, v3.5.1, included VSD v5.4.

VSD version 5.5 is a new release including a number of new features and bug fixes. For details on the changes, see the Release Notes for VSD v5.5.

VSD versions are not tied to GemStone server versions: both older and newer versions of VSD can be used to read statmonitor files generated by both older and newer versions of GemStone/S and GemStone/S 64 Bit.

GemStone revision numbering

GemStone internal development has moved from SVN to Git. Build numbers within the GemStone product (such as log file headers) that previously showed the 5-digit revision, now include the Git SHA; for completeness the entire 40 characters are included.

For example:

VERSION: 3.5.2, Thu Mar  5 11:59:57 2020 

COMMIT: 2020-03-03T14:50:13-08:00 77c22dd6aba1790789b6ce1096f881f20d5553d5

Updated library versions

• The version of openssl has been updated to 1.1.1g
• The version of Kerberos has been updated to 1.18.2
• The version of openldap has updated to 2.4.50

With the new version of VSD, the TCL library versions have also been updated.

Support for UUIDs

Universally Unique Identifiers are 128-bit numbers that, while not relying on a registry, provide practically unique numbers.

UUIDs are provided by the new class GsUuidV4. GsUuidV4 is an implementation of a version 4 UUID as specified in RFC 4122, A Universally Unique IDentifier (UUID) URN Namespace. UUIDs are generated randomly using the secure OpenSSL random number generator.

Instances of GsUuidV4 are invariant and therefore cannot be modified.

To create an instance of UUID:

GsUuidV4 class >> new

GsUuidV4 class >> fromString: aUuidString
aUuidString must be a valid UUID version 4 string in the following format:
  xxxxxxxx-xxxx-4xxx-Vxxx-xxxxxxxxxxxx
where x is any valid lower-case hex digit and V is one of 8, 9, a or b.

See the image for additional protocol.

Replacing Hidden sets with GsBitmap

The GsBitmap class was introduced in v3.4, as an improved API for accessing Hidden Set functionality. This replaces the public API for hidden sets which was added in v3.1, since the older private hidden set API was increasing in use.

In this release, while these three APIs remain unchanged (other than improvements in GsBitmap listed below), the functionality is now implemented with GsBitmap. The public and private hidden set API ultimately invoke the GsBitmap API.

A number of references within GemStone code to the System hidden set API have been updated in this release.

If you are using the System hidden set API methods it is recommended that you update your code to use GsBitmap.

#Indexing

#Conversion

#WeakReferences

#ObjInventory

#Customer1

#Customer2

#Customer3

#Customer4

#Customer5

Added GsBitmap Methods

The following methods have been added to GsBitmap.

GsBitmap >> enumerateAsOopsWithLimit: maxResultSize startingAfter: anOop
Returns an Array containing the first maxResultSize bits as oops starting with the first one after the specified oop. To start the enumeration use the SmallInteger zero as anOop. To get the next batch use the last element of the array returned. This enumeration is non destructive, so no values are cleared. If the GsBitmap contains fewer than maxResultSize elements after the specified object or the maxResultSize has the value 0, then it returns all of the elements after the specified object.

GsBitmap >> enumerateWithLimit: maxResultSize startingAfter: anObject
Returns an Array containing the first maxResultSize bits as objects starting with the first one after the specified object. To start the enumeration use the SmallInteger zero as anObject. To get the next batch use the last element of the array returned. This enumeration is non destructive, so no values are cleared. If the GsBitmap contains fewer than maxResultSize elements or the maxResultSize has the value 0, then it returns all of the elements after the specified object.

The equivalent private methods (with a leading underscore) have been removed.

GsSecureSockets support for SNI

Server Name Indication (SNI) provides a way for a client to provide a hostname along with the other connection parameters. This allows the server to present multiple certificates in response to connections on the same IP address/port.

getServerNameIndication
Answer a string indicating the SNI name assigned to the receiver, or nil if no SNI name has been assigned.

setServerNameIndication: aString
Sets the SNI name for the receiver to be aString. This method is only valid for client sockets which have not yet securely connected to a peer.

Conveniences for using X509 keys with passphrases

The GemStone distribution examples provide private keys that have passphrases, which are also provided as files. For convenience, creating instances of GsTlsPrivateKey, and making and restoring secure backups, now have methods that allow you to provide the passphrase as a filename.

newFromPemFile: fileNameString withPassPhraseFile: aPassphraseFile

newFromPemString: aPemString withPassPhraseFile: aPassphraseFile

GsFile added methods

The following method has been added, allowing to read a file and return the contents in a single operation.

GsFile class >> getContentsOfServerFile: filePathAndName
Return the contents of the named file in a single operation. Use caution when reading a large file as doing so may cause an out of memory error.

A convenience method to get access to stdin for the server has also bee added:

GsFile class >> stdinServer
Return an instance of the receiver that is set up to read the standard input of the server process.

Improvements to thread specification for repository scans

Repository scan functions, such as backup, restore, and finding instances, calculate the number of threads to use based on the number of cores and extents. These calculations may not be optional if, for example, there is one very large extent.

For any backup and restore method, the default number of threads can be explicitly specified by executing, before the backup and restore, the following:

SessionTemps current at: #GsOverrideNumThreads put: numThreads 

For convenience, the following methods have been added to control the number of threads:

Repository >> getDefaultNumThreads
Returns the default number of threads to use for multi threaded operations

Repository >> setDefaultNumThreads: numThreads
Sets default number of threads to use for multi threaded operations

Repository >> setDefaultNumThreads: numThreads during: aBlock
Sets default number of threads to use for the execution of aBlock containing a multi threaded operation and then resets it after the operation completes.

The following convenience methods have been added, which are similar to existing methods, but include an additional numThreads argument with the number of threads to use for the operation:

Repository >> allInstances: inputArg threads: numThreads

Repository >> allObjectsInObjectSecurityPolicies: inputArg threads: numThreads

Repository >> allObjectsLargerThan: aSize threads: numThreads

Repository >> allReferences: inputArg threads: numThreads

Repository >> fullBackupGzCompressedTo: fileNameOrArrayOfNames threads: numThreads

Repository >> fullBackupLz4CompressedTo: fileNameOrArrayOfNames threads: numThreads

Repository >> fullBackupTo: fileNameOrArrayOfNames threads: numThreads

Repository >> markForCollectionWithMaxThreads: numThreads

Repository >> objectAuditWithMaxThreads: numThreads

Repository >> restoreFromBackups: fileNameOrArrayOfNames threads: numThreads

Programmatic access to shared cache configuration

Methods have been added to allow you to collection information on the shared page cache configuration. These methods return values for the shared cache to which the Gem is attached, which may be the Stone’s cache, a remote cache, or a mid-level cache. This is similar to information that can be found in the shared page cache monitor log.

The following information is provided by these methods:

The following methods provided access:

System class >> getSharedCacheAttributes
Returns a SymbolDictionary which contains attributes of the shared page cache. Returns nil if the current session is a solo session, which does not connect to a Stone.

System class >> sharedCacheAttributesReport
Returns a string describing attributes of the shared page cache. Returns nil if the current session is a solo session.

For example,

topaz 1> exec System sharedCacheAttributesReport %

Attributes Report For Shared Cache 'devstone~64623bb08b19f712':

cacheLocked -> false

cacheName -> devstone~64623bb08b19f712

isRemoteCache -> false

isRemoteMidCache -> false

largeMemoryPagePolicy -> 0

largeMemoryPageSizeMb -> 0

largeMemoryPagesUsed -> 0

numFrames -> 4736

numProcesses -> 52

numSharedCounters -> 1900

physicalSizeInBytes -> 127926272

spinLockCount -> 5000

targetFreeFrameCount -> 585

Improvements to results for fileSizeReport in GB

Starting in 3.5.1, the method System >> fileSizeReport reported in units of GB rather than MB, for repositories with a size over 2 GB. Units of GB were too coarse for repositories that were only somewhat over this size limit. Now, all GB sizes are reported rounded to tenths of a GB.

Added String compare method

The method String/MultibyteString >> codePointCompareTo: has been added. This provides a character-by-character codePoint-based comparison, returning -1, 0, or 1.

Fix for bug 48343 (reference to JISCharacter) backed out

In v3.5.1, a reference to the class JISCharacter, which is now in Obsolete Classes, was removed. However, this reference is required to avoid potential issues with legacy indexes using JISCharacter (#48530). The fix for bug 48343 has been backed out, and the reference to JISCharacter has been restored.

If you need to recompile kernel code, you should include ObsoleteClasses in the SymbolList of SystemUser during recompile.

Handling of objects that do not exist by objectAudit

When the objectAudit scan reports objects that do not exist—that is, a valid object contains a reference to an OOP that is on the list of free OOPs, rather than pointing to an actual object—this is now handled immediately, to avoid logical corruption of the repository.

The objectAudit removes these OOPs from the free list immediately, to avoid them being reused for another new object by another session, and immediately checkpoints. If the checkpoint fails, or if any of the invalid referenced OOPs has already been reused, the stone will shutdown to avoid any commits for new values for these OOPs.

Script to verify secure backup renamed

The script that is used to verify a secure backup, verify_backup_with_openssl.sh, has been renamed to omit the trailing .sh; it is now verify_backup_with_openssl.

This script also now supports the -h argument to print usage information.

Functionally, the script is unchanged.

Configuration Parameter-Related Change

Statmonitor and Cache Statistics Changes

Further details for failures in network handshake

To diagnose errors during login, the NetSAccept/NetSConnect handshake now prints additional diagnostic information. Details of failure in NetSAccept/NetSConnect handshake are automatically included in the error message of the GciErrSType coming back to the GCI on login failure (and thus printed by topaz or delivered to GBS).

Removed private/example methods

The Ruby-support classes Regexp and RegexpError have been removed, and are now in ObsoleteClasses. A number of methods that reference this class have been updated.

The following methods have been removed:

GsBitmap >> _enumerateAsOopsWithLimit:startingAfter:

GsBitmap >> _enumerateWithLimit:startingAfter:

GsSecureSocket class >> httpsClientExampleForHost:certificateDirectory:

Object >> _isRegexp

Object >> _isRubyHash

Performance issue in AIO Pgsvr

The AIO pgsvr made several calls to HostGetNanoTime() within its loop. These calls put load on the CPU and impacted performance. (#48503)

objectAudit did not run to completion for some cases of corruption

When SystemRepository objectAudit encountered a corrupted data page that could not be read, it stopped, rather than completing the audit and reporting that as an error. (#48518)

Collection Issues

XFS file system errors for 32-bit processes

Some GemStone operations could error when run within 32-bit processes (such as 32-bit topaz RPC, or a 32-bit GCI application), on 64-bit XFS file system. Specifically, GsFile client file directory operations from 32-bit client processes for 32-bit client processes resulted in EOVERFLOW. (#48462, #48601)

Error on upgrading from version 3.3.x

When upgrading from v3.3.x to v3.5.x, a Message Not Understood error occurred on #_selectedPrimeGreaterThan:. (#48477)

Periodic 30-second hangs with small STN_MAX_SESSIONS

When STN_MAX_SESSIONS was explicitly set to a small value (such as 10), there was likely to be a periodic hang for as long as 30 seconds, after each 10 to 20 commits, while tranlog AIO buffers are recycled. (#48520

Garbage Collection Issues

Backup and restore issues

Page cache fault in remote cache

It is possible for a remote cache to become corrupted with bad lifetimes of pages, such that Gems could not login. This resulted in messages in the stone log of the form "STOP_ZOMBIE stoppedSessionTimeLimit", and "Warning: PageSetLostOt didn't find session" in the page manager log. (#48631)

Issues related to Repository low space

GsExternalSession executeString: crash on nil argument

Passing nil to GsExternalSession>>executeString: or GsExternalSession>> forkString: caused the Gem to SEGV. (#48644)

Negative bitShift: for negative LargeInteger is off by one

If a negative LargeInteger is sent bitShift: with a negative argument, the result is computed based on the LargeInteger internal representation, which is sign-magnitude. This is off by one (one larger than the expected result), compared to other Smalltalk dialects and what would be inferred from the behavior of SmallIntegers. (#48602)

The ANSI specification says that #bitShift with a negative receiver has undefined results, however, the Smalltalk convention is that bitShift: should behave as if the representation was twos-complement, regardless of the internal representation, to be consistent with SmallInteger. SmallIntegers are generally represented as twos-complement.

Adjustments to hot standby debug logging output

The hotstandby processes, logreceiver and logsender, print some information to their log files. Starting in v3.4.2, to reduce clutter in the logs, some of these output statements are only printed when debugging is enabled. Debugging is enabled using the startlogreceiver/startlogsender utility argument -d. Some log messages now printed only in debug mode were still prefixed with [Info]. In addition, flush records are printed to the log in regular as well as debug mode. (#48484)]

gslist -x does not include exe entry for -n process on -m host

When the gslist options -x, -m, and -n are all specified, the resulting report does not include the line providing the exe=<executable> information. (#48489)

gslist -p returned invalid PID when process is not running/status killed

gslist -p returns the processId only of one or more GemStone processes. This information is from GemStone locks files, and thus the former PID returned for processes with status killed. In the case of that PID being reused, this is incorrect. Using this information, such as to kill process, may cause unexpected errors outside of GemStone. (#48734)

GsFile remove file did not handle filenames with * or ?

GsFile removeServerFile: and removeClientFile: did and do not support wild card removes; however, they did not remove files that contained the literal character * or ?. Now, files with names containing these characters can be removed by the removeServerFile: and removeClientFile: methods. (#48533)

Repeated DEBUGGEM/logout sequences can hang process

The DEBUGGEM command allows you to connect a second process to a running Gem for debugging. If multiple repeated DEBUGGEM/logout commands are sent from the debugging process to the running target gem, the target may have stopped accepting connections and may have hung. (#48515)

Statmonitor could fail to add correct file extension

When specifying a file name for a statistics file using the -f or -F command, if the file pattern included .lz4 or .out in a non-terminal location, the correct extension was not appended, resulting in a misleading file name. (#48531)

Risk of SEGV on Stone shutdown after local network disruption

If the local network connections are disrupted by changes in the OS configuration, on shutdown the Stone may SEGV. (#48564)

Attempt to cluster in ClusterBucket over 31 crashed gem

Attempting to cluster into a ClusterBucket with ID greater than 31 caused the Gem process to SEGV. (#48507)

copydbf with compression to directory did not compress

When the destination for a copydbf is a directory rather than a file name, a compression argument (-Z, -z, -C) also included was not applied. (#48540)

GEM_KEYRING_DIRS did not accept a list

The configuration parameter GEM_KEYRING_DIRS only accepted a single directory, not a list of directories as documented. (#48537)

Irrelevant socket listen errors in stone log on startup

When the Stone starts up with listening on the wild address (::), it attempted to listen on loopback 127.0.0.1 as well, which failed as a duplicate. This resulted in listen failure errors in the Stone log, although everything was working correctly. (#48544)