1. Release Notes for 3.7.1

Platforms for Version 3.7.1

GemStone/S 64 Bit version 3.7.1 is supported on the following platforms:

• Red Hat-compatible Linux 7.9, 8.9, and 9.3, and Ubuntu 20.04 and 22.04, on x86; and
  Ubuntu 20.04 on ARM
  GemStone is tested on a mixture of Red Hat, CentOS, and Rocky; these are all considered fully certified platforms. Any reference to Red Hat applies to any Red Hat-compatible distribution.
• AIX 7.1 and 7.2
• macOS 14.3 (Sonoma) and 11.7.10 (Big Sur) on x86 and Apple silicon (ARM)

Note that GemStone/S 64 Bit v3.7 and later on Linux are built using compilation features that are not available on older CPUs (more than about 10 years old); v3.7.1 will not run on these CPUs, regardless of the Linux OS version.

For more information and detailed requirements for each supported platforms, please refer to the GemStone/S 64 Bit v3.7 Installation Guide for that platform.

GemBuilder for Smalltalk (GBS) Versions

The following versions of GBS are supported with GemStone/S 64 Bit version 3.7.1:

VSD Version

The GemStone/S 64 Bit v3.7.1 distribution includes VSD version 5.6.1. This is the same version of VSD that was included with the previous release, v3.7.

VSD 5.6.1 is included with the GemStone distribution, and can also be downloaded as a separate product from https://gemtalksystems.com/vsd/

Updated Library Versions

The version of openssl has been updated to 3.0.13

The version of libssh has been updated to 0.10.6

The version of openldap has been updated to 2.6.6

The version of zlib has been updated to 1.3

The version of MIT Kerberos has been updated to 1.21.2

The version of cJSON has been updated to 1.7.17

The version of curl has been updated to 8.5.0

The version of zoneinfo has been updated to 2023d

The version of prometheus-cpp has been updated to 1.2.0

The version of AWS has been updated; azure-sdk-cpp to version 1.10.3, and aws-sdk-cpp to version 1.11.243

Distribution changes

Linux on ARM now supported for production use

Previously, Linux on ARM was supported only for development. Sufficient testing has been done that Linux on ARM is now certified for production use.

Native code on Mac

As in v3.7, native code is not available for Mac on silicon (ARM).

Recent Mac versions may allocate virtual addresses for temporary object memory more than 32bits offset away from code for libgcilnk.so, resulting in native code errors. To avoid such issues, native code is disabled by default for Mac on x86 (GEM_NATIVE_CODE_ENABLED = 0).

Preliminary Support for Raspberry Pi for Development

Some testing has been done on Raspberry Pi, and this platform is now supported for development only, on Raspberry Pi 5 with 8GB memory, running Debian GNU/Linux 12 (bookworm).

Tranlogging Changes

Some additional incremental (rather than complete) tranlogging has been added for operations on small objects.

ScavengablePages are now logged with STN_TRAN_LOG_DEBUG_LEVEL=1.

Extent pregrow on ZFS file systems

In earlier releases, extent pregrow on ZFS file systems was very slow, since the OS call to allocate disk (fallocate()) does not work, and the backup of appending blocks of zeros is not performant, and also is inappropriate for a COW (copy on write) file system. In this release, the preallocation is done using ftruncate().

Note that with ZFS or any COW (Copy On Write) file system, preallocation does not guarantee that writes to the file will not run out of space. Extent pregrow may not provide improved performance nor reserve disk space exclusively.

Configurable location of services.dat lookup file

The location and handling of services.dat is now configurable.$GEMSTONE/sys/services.dat is a services lookup file that is used:

• by the Stone, to looking administrative services that are only invoked by the Stone, such as starting up the GcGems.
• by the NetLDI, to lookup gemnetobject and other login service scripts, which are not used by the Stone
• by both the Stone and the NetLDI, to lookup services that support remote Gems.

With 3.7.1, there are changes in the way this file is used, and the requirements for the Stone to lookup administrative scripts and for the NetLDI to lookup scripts for login.

As in previous releases, if $GEMSTONE/sys/services.dat is present, the services (runreclaimgem, etc.) may have different names and be located in different directories, and the lookups in $GEMSTONE/sys/services.dat will be used for all Stone service lookups; missing service names or incorrect paths will cause Stone startup to fail.

Now, however, the Stone no longer requires $GEMSTONE/sys/services.dat to be present, as long as all the services (scripts) that would have been looked up in that file, are present with their distribution names in $GEMSTONE/sys/.

Likewise, the NetLDI does not require $GEMSTONE/sys/services.dat to be present, provided the services files (i.e. gemnetobject and other scripts) are present with their distribution names in $GEMSTONE/sys/.

If $GEMSTONE/sys/services.dat is present, this will be used for lookups.

NetLDI can also be provided with the path to a custom services lookup file, using the new startnetldi -Q argument. When provided, the services lookup file specified with the -Q argument will be used for all NetLDI lookups, and the $GEMSTONE/sys/services.dat will not be read by the NetLDI, although it will still be used by the Stone. In this case, the NetLDI will not use services with their distribution names that are present in $GEMSTONE/sys/, but for which there are no mappings in the -Q argument services lookup file.

startnetldi now requires port to match GEMSTONE_NRS_ALL

GEMSTONE_NRS_ALL is commonly used to define the listening port for a NetLDI; in this case, startnetldi can be started without specifying a name or port, and it will be started on the port specified in GEMSTONE_NRS_ALL.

Previously, if you invoked startnetldi including the name or port, (e.g., startnetldi 54321 or startnetldi gs64ldi), and the port number specified or the port mapped from the text name does not match the port specified in GEMSTONE_NRS_ALL, it would start up the NetLDI on the startnetldi argument port, and print a warning about the mismatch. Now, this configuration is a error, and the NetLDI will not start.

This avoids certain corner cases in which there may be confusion in accessing the correct NetLDI, such as external sessions.

To start up a NetLDI with a name or port other than the port specified in GEMSTONE_NRS_ALL, you should unset GEMSTONE_NRS_ALL, or redefine it to provide the intended port number.

Solo logins temporary files renamed

Solo logins create a temporary file in /tmp, previously named gemRO_pid_extent1.dbf; this may be left behind on certain kinds of crashes. To make this more identifiable, the file is now named gemSolo_pid_extent1.dbf.

Upgrade for customer-defined special classes

With customer-defined special classes, added in v3.7, you rename and customize one of a number of kernel classes, Special56Bit0..Special56Bit15. The upgrade process for these classes does not delete all methods on these classes, as normally happens during upgrade.

The Money class, an example of customer-defined special classes, has minor refinements in v3.7.1.

Simplified finding reference path methods

The following methods have been added, which invoke search methods on GsSingleRefPathFinder, and suppress printing of some intermediate search details.

Object >> findReferencePath

Object >> findReferencePathString

These methods allow you to find a reference path to an object, by executing a expression such as:

(Object objectForOop: 97418241) findReferencePath

Added Repository analysis methods

The following methods have been added:

Repository >> countInstances: anArray numThreads: maxThreads objMaxSize: aSize
Scans the entire Repository and generates an array of instance counts of the classes contained in anArray, counting only objects with size <= aSize. Does not include in-memory objects.

Repository >> listInstances: anArrayOfClasses toDirectory: aString numThreads: maxThreads objMaxSize: aSize
List instances of specified classes, listing only objects with size <= aSize. Writes files named className-instances.bm files in the directory specified by aString.

Repository >> listReferencesFromBmFiles: anArray numThreads: numThreads toDirectory: aDirectoryString
Reads bitmap files of instances whose paths are listed in anArray. Writes files, one per thread, in the directory specified by aDirectoryString. Files are named references-N.txt, where N is thread number, containing one line per object containing a reference, with the format

<objId> <1-based iv offset>:<objId of value> <ivOffset>:<valueId>

Excludes references to OOP_EMPTY_INVARIANT_ARRAY, OOP_EMPTY_INVARIANT_STRING, and OOP_EMPTY_SYMBOL, which have oops 233217, 233473, and 233729. Also excludes references from instances of VariableContext, GsNMethod, and GsProcess.

Returns a report of classes and number of instances of each class containing references to objects specified by anArray.

Backup and Restore Changes

...

[01/12/2024 14:35:27.416 PST] ProgressCount =   116961902 (22%)

[01/12/2024 14:35:30.503 PST] ProgressCount =   193092488 (36%)

[01/12/2024 14:35:33.591 PST] ProgressCount =   265600707 (50%)

[01/12/2024 14:35:36.679 PST] ProgressCount =   336629872 (63%)

...

Hot Standby-related changes and bug fixes

Avoid running out of stack while handling an AlmostOutOfStack error

The AlmostOutOfStack error is sent when the process has a small amount of stack space remaining, allowing handling of that error rather than terminating the process. However, if the handler code uses further stack space, it may run out of stack space and shut down.

Now, the amount of stack headroom set aside between the limit at which AlmostOutOfStack is sent, and the hard limit on stack size, is now scalable and configurable. By default, this is 25% of the stack depth; the stack depth defaults to 1000.

GEM_SMALLTALK_STACK_ERROR_PERCENT
After an AlmostOutOfStack or AlmostOutOfStackError is signaled, and if an error handler (on:do:) is handling the notification or error, execution of the error handler runs in a yellow zone area of the stack. If the stack grows beyond the yellow stone a not-trappable AlmostOutOfStackError is signalled to the GCI.The value of this parameter is computed as a percentage of GEM_MAX_SMALLTALK_STACK_DEPTH.
Default: 25 min: 10 max: 100

Utility Changes

copydbf, updatesecuredbf, and startstone includes changes to support multiple encryption keys; see Backup and Restore Changes

GCI Changes

The following thread-safe functions have been added:

(BoolType) GciTsAddOopsToNsc(

    GciSession sess,

    OopType theObject,

    const OopType *theOops,

    int numOops,

    GciErrSType *err );

(int) GciTsFetchNamedOops(

    GciSession sess,

    OopType theObject,

    int64 startIndex,

    OopType *theOops,

    int numOops,

    GciErrSType *err );

(int) GciTsFetchVaryingOops(

    GciSession sess,

    OopType theObject,

    int64 startIndex,

    OopType *theOops,

    int numOops,

    GciErrSType *err );

(BoolType) GciTsStoreIdxOops(

    GciSession sess,

    OopType theObject,

    int64 startIndex,

    const OopType *theOops,

    int numOops,

    GciErrSType *err );

(BoolType) GciTsStoreNamedOops(

    GciSession sess,

    OopType theObject,

    int64 startIndex,

    const OopType *theOops,

    int numOops,

    GciErrSType *err,

    BoolType overlay = FALSE );

Instance-variable specific dbTransient feature

The dbTransient feature can now be applied to instance variables, as well as entire classes.

The GemStone class creation includes the option dbTransient, which allows you to create classes in which the data (instance variable contents) is not written to the extents. Now, you can also specify only specific instance variables of a class to be dbTransient; this allows you to have an instance that includes both transient data and persistent data.

When faulted in from disk, an object’s dbTransient instance variable values will be nil.To preserve the in-memory state of the instVar, ensure the object is kept reachable in memory, such as from SessionTemps current, or from the Smalltalk stack.

The value on disk of a dbTransient instVar is always nil. Flush for commit writes nil to disk, and does not change the in-memory value.

Only the first 60 of the instance variables of a class (including inherited instance variables) may be dbTransient. If aSymbol specifies an instVar beyond 60 (i.e. instVarAt: 61 or subsequent instVar), an Error will be signalled.

dbTransient only applies to named instance variables. For an indexable class with named instance variables, if an instance’s total size is greater than 2034 (self size + self class instSize) then this feature is disabled; the per-instVar dbTransient attribute will always behave as false on that instance.

Instance variables that are dbTransient may not participate in any index on an UnorderedCollection; errors will be signaled if any element of a path for an Index evaluates to a dbTransient instance variable.

The changes specified by an execution of Class >> instVarDbTransient:value: will not take effect completely until after this session commits, and will only be completely in effect in sessions which login after such commit. For instances committed prior to setting dbTransient attribute to true, that instance variable, may be non-nil on disk, but that non-nil value will not be visible to Smalltalk execution in sessions that login after the attribute change. listReferences and markForCollection will see the non-nil value; the next commit that changes some other instance variable in the instance will set that instVar on disk to nil.

The following methods have been added:

Class >> dbTransientInstVarNames
Returns an Array (possibly empty) of instance variable names that are dbTransient on a per-instVar basis.

Class >> dbTransientMask
Returns a SmallInteger

Class >> instVarDbTransient: aSymbol value: aBoolean
Change the per-instVar dbTransient attribute of instVar specified by aSymbol to aBoolean. The change will only been completely in effect for sessions that login after this change is committed.

TreeDictionary and TreeSet added

The TreeDictionary and TreeSet classes have been added, along with a number of supporting Ht* classes. TreeDictionary and TreeSet provide improvements in performance and disk space over the functionally equivalent classes KeyValueDictionary and Set.

Internally, TreeDictionary is structured as a tree which is sorted based on the hash of the dictionary's keys. Its leaf nodes are page-sized hash tables, and its internal nodes are a B+tree variant that allows duplicate keys. An improved hash algorithm allows much more even hash distribution. Several of the Ht class instances that the TreeDictionary or TreeSet uses are DbTransient, avoiding creating garbage during split operations.

TreeDictionary implements all of the required AbstractDictionary protocol, and can be used in place of Dictionary or KeyValueDictionary.

TreeSet likewise implements most Set behavior; however, it is does not currently support the operators #+. #-, or #*, and indexing over a Tree is not is not currently allowed.

TreeDictionary performance is comparable to or better than KeyValueDictionary, and performs faster for additions. Since TreeDictionary does not have to perform "grow" operations that rebuild the entire collection, it avoids the periodic multi-second delays when a threshold is reached on a particular add operation.

TreeDictionary requires moderately less storage, and it grows and shrinks incrementally in increments of a few pages. Unlike KeyValueDictionary, which does not reduce its internal table size, TreeDictionary can shrink down to its original size if all the elements are removed.

Now possible to step into accessor methods

Previously, the debugger would not step into methods that consisted of only a simple instance variable access; this was intentional, for implementation reasons. To improve the flow of debugging, this limitation has been removed.

ProfMonitor object creation profiling enhancements and bug fixes

Avoiding performance impact of process host statistics

In v3.7, the way GemStone collects host process stats on Linux was updated to get additional and more accurate statistics; this however has a significantly greater impact on performance than the equivalent query in earlier releases.

New methods have been added that collect only the less expensive statistics, reporting the smaps-based statistics as 0.

System class >> hostEasyStatisticsForMyProcess
Same as hostStatisticsForMyProcess on all platforms except Linux. On Linux, same as hostStatisticsForMyProcess except certain statistics which are expensive to collect are not computed and have zeros stored in their place.

System class >> hostEasyStatisticsForProcess: anInteger
Same as hostStatisticsForProcess: on all platforms except Linux. On Linux, same as hostStatisticsForProcess: except certain statistics which are expensive to collect are not computed and have zeros stored in their place.

With these methods, the following host process statistics are collected:

UserTime

SysTime

MinorFaults

MajFlt

PercentCpuUsed

PageTablesMemoryKB

ThreadCount

VolCSW

IVolCSW

The following host process statistics are not collected, and returned as 0:

HeapKBytes

PrivateDirty

PrivateClean

PSSSize

Swap

SwapPss

MemMapRegions

FileSystem changes and bug fixes

GsTsExternalSession forkAndDetachBlock:

Normally, processes started by GsTsExternalSession are limited to executing while the GsTsExternalSession is logged in. This limits its value for starting background processes and running scripts.

The following methods have been added to support this use case:

GsTsExternalSession >> forkAndDetachBlock:

GsTsExternalSession >> forkAndDetachBlock:withArguments:

GsTsExternalSession >> forkAndDetachString:

These methods perform an non-blocking execute of the given string or block. If the operation returns, it must be followed by GsTsExternalSession >> nbResult. Execution will continue running after nbLogout of this session, but until logout, it will respond to softBreak or hardBreak from the external session.

Note that the argument should include error handling and printing, so that there is enough information in the Gem log to allow problems to be debugged.

GsTestCase added methods

The following methods have been added to GsTestCase:

should: aBlock raise: anObject description: descString details: detailsString

should: aBlock raise: anObject details: aString

should: aBlock raiseClasses: expClass numbers: expectedErrorNumbers description: descString

IndexManager new methods

The following indexing methods have been added:

IndexManager >> allIndexesReport
Provide a report on all indexes in the image.

IndexManager >> auditIndexes
Perform an auditIndexes on all indexes in the system.

Reporting tranlog positions

The following methods have been added. They return a position as a ScaledDecimal, in which the integer portion is the transaction log number and the fractional digits indicate the record id within the tranlog.

lastCheckpointPosition
Returns a ScaledDecimal representation of the last checkpoint position within the tranlogs.

lastTranlogPosition
Returns a ScaledDecimal representation of the tranlog position.

Other added and moved methods

Removed methods

The following methods have been removed:

GsFileIn >> classMethod:

GsFileIn >> method:

Removed by redesign of method parsing to avoid some corner conditions

GsTestResult class >> excludeInSoloSession:
No longer required.

PositionableStreamPortable >> originalContents
this was a cover method for PositionableStreamPortable>>collection

Class class >> classVariablesAssociationClass:

GsNMethod class >> dynamicLiteralVariables:

CanonStringDict >> become:

KeyValueDictionary >> become:

Behavior >> _checkCompileResult:source:

BinaryFloat class >> _finishFromStream:signFound:factor:integerPart:

BinaryFloat class >> _finishFromStream:signFound:factor:integerPart:fractionalPart:exponent:

Number class >> _finishFromStream:signFound:factor:integerPart:fractionalPart:exponent:exponentPart:

Support for up to 24 encryption keys

In previous releases, encrypted backups supported encryption with up to 8 keys, while extents and tranlogs only supported encryption with a single key.

In v3.7.1, extents, tranlogs, and backups all support up to 24 encryption keys. These can be certs, public keys, or a mix of certs and keys. In practice, the limit for extents and tranlogs that are encrypted using certs (rather than public keys) is 21 keys, due to space constraints within the file; certs use more space than public keys.

The private key corresponding to any encryption key that is in the set of keys on an extent, tranlog, or backup can be used to decrypt or read the dbf.

There are a number of changes associated with this:

• copydbf has been updated to allow the initial encryption to include multiple keys; see copydbf argument changes.
• upgradesecuredbf can add and remove multiple keys; the arguments have changed; see updatesecuredbf argument changes.
• The Stone can read extents as long as any one of the extent’s encryption keys matches the Stone’s startup private key.

startstone has been updated to allow you to specify additional keys, which are only used for reading encrypted tranlogs. The Stone can read transaction logs as long as any of the tranlog’s encryption keys matches the Stone’s startup private key or any of the additional private keys; see startstone argument changes.

• Tranlogs are written by the Stone are encrypted using all of the encryption keys for the first extent. Any one of these keys can be used by the Stone to read the tranlogs.

Path requirements and -K

Previously, the filename for the public cert and private key must have been specified without the path, the private key’s passphrase required a full path, and the -K argument was always required. In v3.7.1, this has been made much more flexible. With some exceptions, you may specify the full path for cert, key, and passphrase files, and omit the -K argument, or specify only the filenames and include -K to specify the directories containing these files.

The exceptions are that -K is required for encrypted backup files:

• updatesecuredbf, for a secure backup file, always requires the -K argument.
• copydbf -V (verify digital signature), on a secure backup file, always requires the -K argument to the directory containing the signing key/passphrase.

Utility Changes related to encrypted DBFs

[Info]: Updating secure extent: pathToExtent

   1 Public key successfully removed.

   1 Public key successfully added.