1. Release Notes for 3.6.4

Platforms for Version 3.6.4

GemStone/S 64 Bit version 3.6.4 is supported on the following platforms:

• Red Hat Enterprise Linux Server, CentOS Linux, and Rocky Linux 7.9 and 8.5; and
  Ubuntu 18.04 and 20.04
  GemStone performs testing on a mixture of Red Hat, CentOS, and Rocky servers; these are all fully certified platforms. Any reference to Red Hat applies to all these distributions.
• Solaris 10 on x86
• AIX 7.1 and 7.2
• OSX 11.1 (Big Sur) with Darwin 20.2.0 kernel on x86, and OSX 10.15.6 (Catalina) with Darwin 19.6.0 kernel; and OSX 11.6 (Big Sur) with Darwin 20.6.0 kernel on Apple M1. v3.6.4 is known to work on MacOS Monterey.
  (Mac is supported for development only)

For more information and detailed requirements for each supported platforms, please refer to the GemStone/S 64 Bit Installation Guide for that platform.

GemBuilder for Smalltalk (GBS) Versions

The following versions of GBS can be used with GemStone/S 64 Bit version 3.6.4:

VSD Version

The GemStone/S 64 Bit v3.6.4 distribution includes VSD version 5.5.3. The previous version of GemStone/S 64 Bit, v3.6.3, included VSD v5.5.2. VSD 5.5.3 includes improved searching, bug fixes, and updates to topaz online help.

Note that in GemStone/S 64 Bit v3.6 and later, statmonitor writes additional information to the statmonitor file. As a result, statmonitor files from v3.6 and later cannot be read by versions of VSD earlier than v5.5. VSD 5.5.3 can read statmonitor files generated in older versions of GemStone/S 64, 32-bit GemStone, and GBS, as well as those generated by GemStone/S 64 Bit v3.6 and later.

VSD v5.5.3 is included with the GemStone distribution, and can also be downloaded as a separate product. For details or to download, go to https://gemtalksystems.com/vsd/.

Open Source Library Versions

The version of OpenSSL has been updated to 1.1.1n.

Distribution Changes

Support for Rowan upgrades

This release includes upgrade support for applications running certain earlier versions of GemStone 3.6.x with Rowan versions 1.2.x and 2.2.x, that are upgrading to v3.6.4.

Banner notice when running with SELinux

SELinux is a RedHat mode that applies additional security restrictions. When running in this mode, the process log banners now will include a notice; for example,

| MEMORY: 64187 MB , getauxval(AT_SECURE) = 2 |

Additional information in stone log messages

Stone log messages now includes additional information, such as the IP addresses of Gem sessions, to help with diagnosing problems.

Print GCI trace records on protocol error

To aid in analysis of GCI protocol errors, recent GCI call history is now automatically printed to the gem log when a GCI protocol error occurs. GCI call trace records are maintained in a wraparound buffer, and the most recent 100 records will be printed to the log.

Rolling over Gem log files

For long-lived Gems, it can be helpful to roll over the log file periodically. This can now be done using the following new method.

System class >> startNewGemLog: aFileName
Close the existing log file for this Gem and start a new log with the name aFileName. aFileName must be a file name, not a path ( '/' is not allowed); this file name is appended to the result of System gemLogPath to produce the new log file. If a file named aFileName already exists in this location, it will be opened for append. If the file cannot be created or written to, this method returns an error and the existing log file will continue to be used. Has no effect and signals a Warning in a topaz -l or other linked GCI application.

GsHostProcess changes

GsHostProcess >> stderrPath

GsHostProcess >> stderrPath: aStringOrUtf8 

GsHostProcess >> stdinPath

GsHostProcess >> stdinPath: aStringOrUtf8 

GsHostProcess >> stdoutPath

GsHostProcess >> stdoutPath: aStringOrUtf8

GsSingleRefPathFinder disallowed in remote sessions

It is now disallowed to find reference paths using GsSingleRefPathFinder in sessions that are remote from the Stone (on a remote cache).

GsObjectInventory profiling methods now wait up to 60 seconds for the GcLock

GsObjectInventory scans previously only waited for 3 seconds for another session using the GcLock to finish and release the lock. Now, these will wait for 60 seconds before timing out.

Private methods handling days from 1970 removed

The following private methods were removed

Date >> _asDaysFrom1970

Date class >> _newFromDays1970:

GsFileIn now ignores login, logout

GsFileIn allows programmatic filein of topaz-formatted files. GsFileIn now ignores login and logout commands that are encountered in the input file.

Error changes

The Smalltalk symbol #errIllegalRamSelf has been added for the error ERR_illegalRamSelf #2156, with the error message 'Self is not a ram oop, method needs recompile.'

ProfMonitor object creation tree report format

This report was unreadable with deep trees. The indents have been reduced, and indents more than 25 deep are all printed with the same indent level.

External Session Changes

GsExternalSession and GciLegacyExteranlSession previously hardcoded errors to GciError or GciLegacyError; now, methods invoke a new method GsExternalSession >> gciErrorClass, to return the class of Error.

The following method was added:

GsTsExternalSession >> isReadReady
Return true if the session is ready to read a result or error.

OtherPassword split with ReadOtherUserProfile

Finding out any information about another user’s UserProfile, such as the lastLoginTime or authenticationScheme, previously required OtherPassword privilege. This privilege also allowed modifying another users’s UserProfile.

To allow reading another user’s UserProfile, but not allow modification, a new privilege, ReadOtherUserProfile, has been added. A UserProfile with this privilege can read specific information for other UserProfiles, but cannot make changes.

A UserProfile with OtherPassword has implicitly also ReadOtherUserProfile, so there is no need to grant this additional privilege to users with OtherPassword.

The following methods can be executed by a UserProfile with OtherPassword, ReadOtherUserProfile, or both privileges:

UserProfile >> activeUserIdLimit

UserProfile >> authenticationScheme

UserProfile >> hasLoginLogging

UserProfile >> isReadOnly

UserProfile >> lastLoginTime

UserProfile >> lastPasswordChange

UserProfile >> ldapBaseDn

UserProfile >> ldapSearchFilterDn

UserProfile >> loginsAllowedBeforeExpiration

UserProfile >> passwordAgeLimit

UserProfile >> passwordAgeWarning

UserProfile >> staleAccountAgeLimit

UserProfile >> userIdAlias

UserProfile >> passwordNeverExpires

Other Added Methods

Array >> copyNotNilFrom: startIndex to: stopIndex
Return an Array containing the non-nil elements that are within the specified offsets of the receiver.

Array >> indexOfNotNil: startOffset to: endOffset
If startOffset <= endOffset, returns the first offset within in the specified range of a non-nil element of the receiver. If startOffset > endOffset, returns the last offset within the specified range of a non-nil element of the receiver. Returns zero if all are nil. Intended for use only on Arrays of size <= 2000; performance on larger arrays will be slow.

GsFile class >> stderrServer
Returns an instance of the receiver which can be used to write to the standard error of the current Gem process, or nil if an error occurs.

GsTestCase >> assert: objA includes: objB 

GsTestCase >> assert: objA includesIdentical: objB 

GsTestCase >> assert: objA isKindOfClass: objB 

GsTestCase >> deny: objA includes: objB 

GsTestCase >> deny: objA includesIdentical: objB   

GsTestCase >> isArm64 

Topaz changes

topaz 1> exec 'abc' %

abc

[39747073 size:3  String] abc

Cache statistics changes

Script Types

SuperDoit scripts come in two variants:

• stone
  executes the script as a normal login to a running Stone. Login details are provided in a .topazini, which may be provided on the command line using the -I argument, or in one of the standard .topazini locations: the current working directory, or the home directory of the user executing the script.

All the details required for a login to a Stone in your specific environment must be available, in the .topazini, $GEMSTONE_NRS_ALL, and/or other environment variables.

By convention, these scripts end in .stone.

• solo
  executes the script as a solo session. Commits, and operations that require a stone, are not allowed, and solo scripts cannot login RPC. Solo scripts by default use $GEMSTONE/bin/extent0.rowan.dbf for their solo extent.

Solo scripts do not automatically look for a standard .topazini file. Solo scripts use a solo-specific topaz initialization file, located within the superDoit project, which logs in as SystemUser. You may use the -I argument to specify a different topaz initialization file.

By convention, solo scripts end in .solo.

Script structure

A SuperDoit script consists of a number of sections, each one started by a keyword, and ending with %. All sections are optional.

The Smalltalk code that the script executed is in a section with the keyword doit. Note that while this looks similar to topaz syntax, it is a keyword defining the code execution section; topaz syntax is not understood in solo and stone scripts.

A minimal SuperDoit script might be, for example,

#!/usr/bin/env superdoit_solo

doit

Time now asString

%

In addition to a doit section, there are number of sections that you may include depending on your script’s requirements. Most commonly, you will want to include sections for script command line options, in the options section, and usage information in a usage section.

An example of a script with the options, usage, and doit sections:

#!/usr/bin/env superdoit_stone

# Print the contents of DbfHistory with a byte limit

options

{SuperDoitOptionalOptionWithRequiredArg 

	  long: 'bytes' short: 'b'}

%

usage

$basename [-h] [-D] [-b <number>] [-- [-I <topazini>]]

Return DbfHistory for the stone specified by <topazini>, or

if not specified, the default .topazini for the environment.

The optional -b argument specifies to return only the first

<number> bytes of the DbfHistory.

%

doit

self bytes isNil

   	ifTrue: [DbfHistory] 

	   ifFalse: [DbfHistory copyFrom: 1 to: 

		(self bytes asInteger min: DbfHistory size)]

%

This script show how access to the argument is handled: self bytes returns the contents of the command line argument specified using -b or --bytes.

Also note that the usage here is simplified; the arguments --help, --Debug, and --bytes may also be used on the command line. See the section under Arguments.

Finally, the argument to topazini that is specifically mentioned in the usage is only one of a number of arguments to topaz that can be passed to the SuperDoit script and forwarded to topaz. These arguments, if used, are separated from the script arguments by --. See the section Arguments to Topaz.

Arguments

Command line arguments may be defined using name-value pairs, names without values, or as positional arguments.

The options section defines the named and name-value command line arguments; positional arguments are accessed in the doit section, sending self positionalArgument to return the array of zero or more positional arguments that were included on the command line.

Named and name-value pairs can be specified either using getopts syntax -Character value (referred to as "short" in SuperDoit), or --String=value ("long").

The long version is always supported; you may also specify a short version when defining the script arguments.

Arguments with values may be required or optional, and optional arguments may have a default value specified. See the specifications on Defining Script Arguments.

	SuperDoitOptionalOptionWithNoArg long: 

	SuperDoitOptionalOptionWithNoArg long:short: 

	SuperDoitOptionalOptionWithRequiredArg long: 

	SuperDoitOptionalOptionWithRequiredArg long:default: 

	SuperDoitOptionalOptionWithRequiredArg long:short:

	SuperDoitOptionalOptionWithRequiredArg long:short:default: 

	SuperDoitRequiredOptionWithRequiredArg long:

	SuperDoitRequiredOptionWithRequiredArg long:short: 

options

{SuperDoitOptionalOptionWithRequiredArg 

	  long: 'bytes' short: 'b'}

%

getDbfHistory.stone -b 100 -- -I .topazini

simple.solo -- -C GEM_SOLO_EXTENT=$GEMSTONE/bin/extent0.dbf

Usage

The usage section includes a section of text that will be displayed when the script is invoked with -h or --help.

usage

$basename [-h] [-D] [-b <number>] [-- [-I <topazini>]] 

Return DbfHistory for the stone specified by <topazini>, or

if not specified, the default .topazini for the environment.

The optional -b argument specifies to return only the first

<number> bytes of the DbfHistory.

%

The usage section is optional. If it is not included in the script, when -h or -help is invoked, the default help with -h/--help/-D/--debug options is displayed.

Coding in the doit section

The doit section consists of GemStone Smalltalk code. Within the doit, self corresponds to the instance of SuperDoitExecution. This provides access to the script arguments and environment, and set of convenience helper methods.

Available methods include (but are not limited to):

executionStoneName
Returns the name of the running stone; for solo, this will be gs64stone.

globalNamed: globalName
Return a global variable with the given name.

globalNamed: globalName ifAbsent: notfoundblock
Return a global variable with the given name, executing the notfoundblock if it does not exist.

positionalArgs
Returns an array containing all positional arguments from the command line.

stderr
The stderr that the script writes to.

stdout
The stdout that the script writes to.

noResult
At the end of the script, this allows you to return with no messages on the command line.

exit: errmsg withStatus: anInteger
Exit the script, writing errmsg to stderr and with the OS process exit status set to anInteger.

exitWithStatus: anInteger
Exits the script, with the OS process exit status set to anInteger.

isSolo
Return true if in a solo session.

log: anObject
Write the argument in STON format to stdout.

logErrorMessage: aString
Write the given string on stderr.

logMessage: aString
Write the given string on stdout.

commandLine
Returns the command line for the script.

Examples and Templates

CByteArray support for copy

Previously, #copy was disallowed for instances of kinds of CByteArray. This is now supported.

In addition to copy semantics, as implemented by Object >> copy, the following method has been added:

CByteArray >> shallowCopy
Returns a copy of the receiver which shares the receiver's instance variables. If self autoRelease == true, the result's C memory will be a copy of the C memory of the receiver, with result autoRelease == true. Otherwise the result will contain a reference to the C memory of the receiver, with result autoRelease == false.

CHeader support for variable arguments

While the FFI supported C functions with variable arguments in previous releases, now the instance methods generated by CHeader include a separate keyword, varArgs:, allowing you to pass in variable arguments. The argument to varArgs: should be an Array in which there are two entries for each variable argument: the first is the type symbol, and the second is the value.

The earlier syntax is still valid; FFI wrapper classes generated in earlier versions of GemStone do not need to be regenerated after upgrade to v3.6.4, and message sends to the previously generated methods do not need to be updated.

Improved access to errno

Previously, to fetch error information, the method CCallout class >> errno and errno: accessed the errno information from the GsProcess. There was a risk that two interleaved FFI calls (within a single GsProcess) could access this, and set/retrieve incorrect values.

Support for C functions with struct type arguments or results

The FFI now supports functions with C structs that are passed by value for arguments, and functions that return a C struct by value. Structs passed by address is not the same as a struct passed by value; these were previously supported by CPointers.

To support these calls, the class CCalloutStructs has been added, and is generated by CHeader wrapper methods when the source C header files includes struct arguments or return values.

A struct passed by value is represented as a kind of CByteArray, which must be of the correct size.

It is recommended to use CHeader>>wrapper* methods to generate the instances of CCalloutStructs and subclasses of CByteArray, to ensure that the CCalloutStructs and CByteArray are correctly constructed.

Limitations:

• Use of CCalloutStructs is supported only on Linux and MacOS on x86_64 and ARM/Apple M1 systems. Instances of CCalloutStructs generated by parsing the header files are platform-independent, but can only be used to make C calls on supported platforms.
• If the C function takes variable arguments, the variable argument values may not be structures passed by value.
• Structs by value are not supported as arguments nor as results, if any other argument to the C function is a C double or C float. However, structs that are passed as argument or results may themselves have fields of type double or float.

Example of struct return and variable arguments

The following example shows both a hand-crafted and a CHeader wrapper construction for a call that returns a struct. This example is based on the open source RabbitMQ® messaging library. Note that for simplicity, the login example uses (CPointer newNull) for the argument that requires an established connection from an earlier call.

| wrapper |

wrapper := (CHeader path: '/usr/include/amqp.h')

	wrapperForTypeNamed: 'amqp_rpc_reply_t'.

UserGlobals at: wrapper name put: wrapper.

| theResult  myCO fArgs varArgs |

theResult := amqp_rpc_reply_t new.

myCO := CCalloutStructs 

	library: (CLibrary named: '$GEMSTONE/testlib/testmq')

	name: 'amqp_login'

  	result:  #struct 

  	args: #( #'ptr' #'const char*' #'int32' #'int32' #'int32' #'int32' )

	varArgsAfter: 6

	structSizes: { (theResult size) . nil . nil . nil . nil . nil . nil ). 

fArgs := { (CPointer newNull) . '/'  . 0 . 131072 . 0 . 0  }.

varArgs := #( #'const char*' 'guest' #'const char*' 'guest' ).

myCO 

	callWith: (fArgs, varArgs) 

	structResult: theResult 

	errno: nil.

theResult 

| class wrapper |

wrapper := (CHeader path: '/usr/include/amqp.h')

	wrapperNamed: 'MyAmq'

	forLibraryAt: '$GEMSTONE/testlib/testmq'

	select: [:each | each name = 'amqp_login'].

wrapper initializeFunctions.

UserGlobals at: wrapper name put: wrapper.

| theResult |

theResult := MyAmq new

	amqp_login_: (amqp_rpc_reply_t new)

	_: (CPointer newNull)	_: '/'	_: 0 	_: 131072 _: 0 _: 0 	 

	varArgs: #( #'const char*' 'guest' #'const char*' 'guest' ).

theResult

Other Image Changes related to FFI

CByteArray >> _signed:at:with:

CDeclaration >> baseAccessorForOffset:

CDeclaration >> baseStructByteSize

CDeclaration >> baseUnionByteSize

CDeclaration >> byteSizeRounded

CHeader >> initializeFunctionsCodeForFunctions:libraryPathExpressionString:

Risk of SPC corruption for allInstances and related repository scan operations

Repository scan operations that invoke primitive 1030 to return results as GsBitmap instances (allInstances*, allObjects*, allReferences*; see the bugnote for a complete list), has a timing bug in acquiring the GcLock. The Gem executing this primitive could get invalid page views, which may result in cache corruption. (#49945)

Reclaim issues

Symbol garbage collection may collect a referenced Symbol

A code path was found in which a referenced Symbol could be garbage collected. (#49886)

Gem crash recovery may cause Shrpcmon to SEGV

When a Gem crashes, the Shrpcmon performs recovery to clean up spin locks. It was possible for the address into the process table to be out of range, causing the Shrpcmon to crash. (#49988)

File descriptor leak in Gem

On login, a number of file descriptors are needed. One of these file descriptors was not released on logout. There was also a file descriptor that was opened that is not necessary. (#49983)

objectAudit issue in DataPageCheck not handled well

If objectAudit detects a problem with DataPageCheck, it some recent versions it may have produced large error sets, or caused the audit to crash. (#49830)

pageaudit may fail if Stone startup takes too long

The pageaudit utility starts a temporary Stone, and if the Stone startup process takes too long, the pageaudit may timeout and fail with an error. (#49875)

GsDevKit upgrade bugs fixed

There were bugs in the code to upgrade GsDevKit (issues 15, 16, 21 and 23 on the github project). These fixes are included in the distribution in $GEMSTONE/examples/seaside/bin/GsDevKit_upgrade.gs. (#49795, 48510)

Cache Statistics Issues

totalExtentGrows

timeInExtentGrows

avgTimePerExtentGrow

.topazini could silently clear -X arguments for X509 login

By default, topaz automatically reads a existing .topazini file on startup. If the .topazini sets any of the standard login parameters (gemstone, username, etc.), it clears any X509 login parameters, including those passed in via the topaz command line. This design is intended to avoid mixing sets of login parameters

Now, a warning is printed if command line -X arguments are cleared by a .topazini setting. (#49761)

Time comparison incorrect results for arguments in same millisecond range

The comparison methods > < = converted the operands to milliseconds, and this returned incorrect results in cases where the operands’s milliseconds values matched; incorrectly returning true for = and incorrectly returning false for < or >. (#49798)

During tranlog restore, existing file without permissions not reported

If tranlog restore encountered a transaction log that it did not have permission to read, it stopped restore, as if restore was complete, without reporting a permission error. (#49837)

Login possible during restore

It was possible to login during a restore from backup, which caused the restore to fail. (#49958)

Commit issues during suspendCommitsForFailover

When commits are suspended for failover (System suspendCommitsForFailover), an attempt to commit will error. The commit attempt is not aborted, and an attempt to abort or resumeCommits causes the session to lose its connection to the stone. (#49943)

If the SymbolGem needs to commit, this will fail; the Stone will continue to attempt to restart the SymbolGem which will continue to fail to commit. (#49944).

SecurityErrors during upgrade in some GsObjectSecurityPolicies configurations

When upgrading from a version earlier than 3.5, if the GemStone base classes in the originating repository have an unusual distribution of classes and methods over the GsObjectSecurityPolicies, there may be security errors in upgradeImage, when CodeLibrarian performs the recompile of methods. (#49807)

ScaledDecimal literal parsing issues

Cannot activate a passivated structure containing an instance of Utf8

The contents of an instance of Utf8 cannot be updated using standard methods, and so attempting to activate a passivated Utf8 failed. (#49864)

GCI and FFI related Issues

After failed commit due to RC conflicts, commit allowed without abort

If a commit fails due to RC conflicts, an abort should be done before a subsequent attempt to commit. This abort was not required, so a commit was allowed, which could bypass correct transactional behavior, since the commit was based on the view with partial updates from the RC replay. (#49772)

gslist prints unnecessary extra Warning if no servers running

If no servers are running and gslist can produce no results, it printed an extra Warning line, in addition to reporting No GemStone Servers. (#49754)