1. GemStone/S 64 Bit 3.5.7 Release Notes 
Overview
GemStone/S 64 Bit™ 3.5.7 is a new version of the GemStone/S 64 Bit object server, add the Object Read Tracking feature and including a number of other feature enhancements and bug fixes.fixes.
These release notes describe changes between the previous version of GemStone/S 64 Bit, version 3.5.6, and version 3.5.7. If you are upgrading from a version prior to 3.5.6, review the release notes for each intermediate release to see the full set of changes.
The Installation Guide has not been updated for this release. For installation, upgrade and conversion instructions, use the Installation Guide for version 3.5.6 for the Mac, or 3.5.3 for other platforms.
Supported Platforms
Platforms for Version 3.5.7
GemStone/S 64 Bit version 3.5.7 is supported on the following platforms:
-
Red Hat Enterprise Linux Server and CentOS Linux 6.10, 7.9, and 8.3; Ubuntu 16.04, 18.04, and 20.04
GemStone performs testing on a mixture of Red Hat and CentOS servers; both are fully certified platforms. Any reference to Red Hat applies to both distributions. - Solaris 10 on x86
- AIX 7.1 and 7.2
-
MacOS 11.1 (Big Sur) with Darwin 20.2.0 kernel, and MacOS 10.15.6 (Catalina) with Darwin 19.6.0 kernel, on x86
Mac is supported for development only.
For more information, please refer to the GemStone/S 64 Bit Installation Guide for that platform.
GemBuilder for Smalltalk (GBS) Versions
GemStone/S 64 Bit version 3.5.7 requires GBS version 8.4 or later for VisualWorks Smalltalk, or version 5.4.5 or later for VAST Platform (VA Smalltalk).
The following versions of GBS are supported with GemStone/S 64 Bit version 3.5.7.
GBS version 8.5
GBS version 8.4
GBS version 5.4.6
GBS version 5.4.5
Windows 7 reached end of life in January of 2020. While GemTalk continues to test on and support GemStone clients on Windows 7, it is no longer considered fully certified.
For more details on supported GBS and client Smalltalk platforms and requirements, see the GemBuilder for Smalltalk Installation Guide for that version of GBS.
VSD Version
The GemStone/S 64 Bit v3.5.7 distribution includes VSD version 5.5.2. The previous version of GemStone/S 64 Bit, v3.5.6, included VSD v5.5.1.
VSD version 5.5.2 includes several minor bug fixes. For details on the changes, see the Release Notes for VSD v5.5.2.
VSD 5.5.2 is included with the GemStone distribution, and can also be downloaded as a separate product from https://gemtalksystems.com/vsd/
Changes in this release
Object Read Tracking
Object Read Tracking enables GemStone to record the time and userId when specific objects in the repository are read. Records of read operations for all session are written to comma-delimited (CSV) files by the Stone.
Read tracking is enabled for a repository using the new stone configuration parameter STN_OBJECT_READ_LOG_ENABLED.
Objects for which reads are tracked are associated with specific GsObjectSecurityPolicies; each object in that GsObjectSecurityPolicy will have reads tracked. Objects can also be added to the tracking log dynamically.
By default, reads by all UserProfiles (other than special system users, listed here) are tracked. Individual UserProfiles can be configured to not have reads tracked, so for example batch jobs can run without tracking; and UserProfiles with appropriate privileges can dynamically disable and re-enabled object read tracking.
Configuring objects to be tracked
A new instance variable, trackReads has been added to GsObjectSecurityPolicy; by default, this is false. When this is set to true, all objects associated with that security policy have reads tracked.
The following methods have been added:
GsObjectSecurityPolicy >> trackReads
Returns a Boolean, true if object read logging is enabled for objects in this security policy.
GsObjectSecurityPolicy >> trackReads: aBoolean
If aBoolean is true enables object read logging for objects in this security policy, false disables it. Signals an Error if any element of AllUsers has a userId incompatible with writing to a .csv file.
Before any read tracking can be recorded, the application configure the objects to be tracked by doing the following:
- Define a security policy with trackReads set to true
- Determine which application objects require read tracking
- Associate these objects with the read tracking security policy.
In addition, specific individual objects can be tracked within a specific session by using the following method:
Object >> trackRead
If receiver is not a special object, and (System objectReadLogEnabled == true) and ((System myUserProfile _hasPrivilegeName: #DisableObjectReadLogging) == false), add the receiver to the sessions read tracking buffer. Returns true if the object was added to the read tracking buffer, false otherwise.
This addition to the read tracking buffer is independent of whether (self objectSecurityPolicy trackReads == true) and is independent of whether this method has already been sent to the receiver.
Enable and disable Read Tracking
Read tracking must be enabled by setting the configuration parameters STN_OBJECT_READ_LOG_ENABLED and STN_OBJECT_READ_LOG_DIRECTORIES before the Stone is started up.
If a session logs in as a user with the DisableObjectReadLogging privilege, then read tracking is not enabled for that particular user.
If a session logs in as a user without the DisableObjectReadLogging, objects that have object read logging configured are written to the object read logs.
If the user additionally has the DynamicDisableObjectReadLogging privilege, the user can disable read tracking temporarily within that session by executing
System setObjectReadTracking: false
When object read tracking is dynamically disabled, a record is written to the object read tracking log.
To determine if the current session has read logging currently enabled, send:
System objectReadLogEnabled
The following methods have been added:
System class >> objectReadLogEnabled
Answer a Boolean indicating if the Stone was started with the configuration parameter STN_OBJECT_READ_LOG_ENABLED set to true. Note that the actual state of object read logging will depend on the privileges of the current session’s userProfile and if object reads tracking was dynamically disabled; see System class >> setObjectReadTracking:.
System class >> setObjectReadTracking: aBoolean
Dynamically disable or re-enable object read tracking for the current session, and return the previous status of object read tracking. If setObjectReadTracking: successfully disables or re-enables read tracking, a record is written to the Object read tracking log.
When aBoolean is false, then if object read logging is not enabled for the Stone, or if the current session’s UserProfile has the DisableObjectReadLogging privilege, or if setObjectReadTracking: false was previously invoked to disable object read tracking, then this method has no effect, and returns false. If the current user does not have DynamicDisableObjectReadLogging privilege, than an exception is raised.
A true value for aBoolean only has effect if setObjectReadTracking: false was previously successfully executed.
This is expected to be used in constructions such as:
| prev | [ prev := System setObjectReadTracking: false.
query Execute ] ensure:
[ System setObjectReadTracking: prev ]
System class >> objectReadTrackingEnabled
Returns true if object read tracking is currently being performed for this session, for any objects with tracking configured via GsObjectSecurityPolicy >> trackReads or Object >> trackRead. It returns false if no object read logging will occur. Specifically, this method will:
- return false if Object read tracking is not enabled for the Stone
- return false if the current session’s UserProfile has the DisableObjectReadLogging privilege
- return false if the session has dynamically disabled Object Read Tracking; that is, System setObjectReadTracking:false must not have been executed and be still in effect.
Read Tracking
Tracking occurs when an object is faulted into the temporary object memory of a session. Only one read is recorded per object per session. Once an object has been tracked as read, if the object is read again, no further tracking events are recorded for that object in that session.
The session sends tracking records to the Stone:
- At each transaction boundary (commit or abort)
- When the gem's tracking buffer is full. The buffer is about 100K bytes.
- When the gem explicitly calls System class >> flushObjectReadBuffer
- At session logout.
Each read record in the buffer has the timestamp of the first object recorded in the buffer (this avoids excessive calls to fetch exact timestamps from the OS). There is a final record in the buffer, indicating the end of the buffer, which also has an exact timestamp.
Limitation in Read Tracking
If the session terminates with a fatal error or is killed with SIGTERM, the last tracking buffer may be lost. This creates a way for a user to intentionally (as well as inadvertently) read an object without this read action being recorded in the log.
Object Read Log File
The object read log location is specified using a new stone configuration option, STN_OBJECT_READ_LOG_DIRECTORIES. This specifies a list of one or more directories in which to write object read log files. The Stone will not start up if STN_OBJECT_READ_LOG_ENABLED is true and STN_OBJECT_READ_LOG_DIRECTORIES does not contain at least one valid, writable directory.
The object read log file name will be composed as follows:
ObjectReadLogDir/stoneName-ObjectReadLog_timestamp[.current].csv
- ObjectReadLogDir is the directory for object read logs, one of the directories specified by STN_OBJECT_READ_LOG_DIRECTORIES in the stone configuration file.
- stoneName is the short name of the stone (without the NRS).
- timestamp is UTC (GMT) time when the file was created, in the format YYYY-MM-DD-hh:mm:ss.nnn, where nnn is the 3 digit milliseconds of the time.
- [.current] is a suffix that appears only on the file that is currently being written by the Stone. Files that are not actively being written to do not have this suffix.
The maximum size of the object read log file in bytes is specified using the new stone configuration parameter STN_OBJECT_READ_LOG_MAX_FILE_SIZE. When the maximum file size is reached, the current object read log file will be closed and renamed to remove the .current suffix, after which a new log file will be opened.
If a buffer of records from a Gem needs to be written, but will not fit into the current object read log file without exceeding the file maximum size, a new log is started. The contents of a Gem’s buffer of read records is not split over two object read log files. Each Gem’s buffer of records is written contiguously, before another Gem’s buffer.
When stone starts, if a single .current.csv file is found within the directories specified by STN_OBJECT_READ_LOG_DIRECTORIES, and if this file is smaller than the maximum file size, it will be reopened for writing. Otherwise a new object read log file will be started.
When the local system time crosses midnight UTC, the current object read log is closed and a new one is started.
The following methods have been added:
System class >> currentObjectReadLogFile
Answer a string containing the current object read log file which the stone is writing to, or nil if the object read logging feature is disabled.
System class >> startNewObjectReadLog
Requests Stone close the current object read log file and start a new one. Blocks until the operation has completed. Returns true on success, false if the object read logging feature is disabled, and raises an exception on error. Requires FileControl privilege.
System class >> flushObjectReadBuffer
Forces the gem to immediately send any buffered object read records to the stone.
File Management
Applications must manage storage and archiving of object read log files, to ensure the file system containing these logs does not become full.
It is recommended that the log files be compressed using gzip or lz4 data compression, and that archiving scripts run using CRON so that object read log files are automatically archived.
If a write to the object read log file fails, the following actions will be taken in sequence:
1. Log a message to the stone log regarding the write failure.
3. If the write still fails, open a new object read log file in the next directory.
4. Repeat item 2 until all object read log directories have been tried.
5. Log a message to the stone log indicating all object read log space is full.
6. Suspend (block) the session and retry opening a new object read log file periodically.
7. Terminate the session if the writes cannot be completed within 5 minutes or the blocked session causes a commit record backlog, whichever occurs first.
Errors opening, renaming and writing the object read log file are recorded in the Stone log.
Added Configuration File Options
STN_OBJECT_READ_LOG_ENABLED
A Boolean (true or false) indicating if object read logging is enabled.
Runtime equivalent: #StnObjectReadLogEnabled (read-only at runtime).
STN_OBJECT_READ_LOG_DIRECTORIES
A list of directories where object read log files can be written. Must contain at least 1 element if STN_OBJECT_READ_LOG_ENABLED is true. The Stone will not start if STN_OBJECT_READ_LOG_ENABLED is true and this configuration parameter is not set to at least one valid directory.
UserProfiles exempt from read tracking
UserProfiles may be given a new privilege, DisableObjectReadLogging.
UserProfiles with this privilege do not have their object reads tracked. To allow an automated batch job to operated without generating excessive read tracking, execute that job as a UserProfile with the DisableObjectReadLogging privilege.
To ensure the UserProfiles can be clearly written to the csv file, creation of new UserProfiles requires that the userId contain only compatible characters; see UserProfile userId permissible Character limitations.
Added Privileges
DisableObjectReadLogging
UserProfiles with this privilege never have their object reads tracked. SystemUser, DataCurator, GcUser, SymbolUser, HostAgentUser and CodeLibrarianUser always have this privilege.
Existing and new users without this privilege will have reads tracked, if object read tracking is enabled for the repository and has not been dynamically disabled.
DynamicDisableObjectReadLogging
UserProfiles with this privilege may dynamically disable their read tracking using System setObjectReadTracking: false. Users without this privilege cannot disable read tracking. Has no effect for users with DisableObjectReadLogging, who never have read tracking enabled.
Added Cache Statistics
The following cache statistics related to object read tracking have been added:
NumInReadTrackingQueue (Stn)
Number of sessions waiting for data to be written to the Read Tracking Log.
ObjectsReadTracked (Gem)
Number of object faults for which a record was added to the Read Tracking Log.
ReadTrackingFileSize (Stn)
Size in bytes of the currently open Read Tracking Log.
ReadTrackingServiceCount (Stn)
Number of Read Tracking buffers processed by stone.
Object Read Log File Format
The object read log is in CSV file format, as specified by RFC 4180.
The first line of each file will contain column headers.
There are three record types: O, F, and R.
The first field of a record is a single letter, either O, F, or R, indicating the type.
- An O record includes the details on a specific read record.
- An F record marks the end of a group of O records that were in the same buffer composed by a gem.
- An R record is generated when System class >> setObjectReadTracking: is used, indicating dynamic disable or re-enable of object tracking.
The fields for each record are:
O, TimeGMT, UserName, UserProfileOop, GemProcessId,
GemHostName, GemHostIpAddress, GemClientIpAddress,
ObjectReadOop, ObjectReadClassOop, ObjectReadClassName
R, TimeGMT, UserName, UserProfileOop, GemProcessId,
GemHostName, GemHostIpAddress, GemClientIpAddress, empty,
empty, ReadTrackState
F, TimeGMT, UserName, UserProfileOop, GemProcessId
The following fields are defined:
TimeGMT (Integer)
Timestamp in GMT of the object read operation, updated when a gem composes the first O record of a session, when an R record is written, and when the F record is composed by a gem to flush a buffer.
UserName (String)
The GemStone user name of the user performing the read, UTF8 encoded and enclosed in double quotes. This is obtained from the userId instance variable of the session’s UserProfile object.
UserProfileOop (Integer)
The OOP of the user profile.
GemProcessId (String)
The process ID of the session.
GemHostName (String)
The name of the host on which the gem is running, the result of calling hostname(). This field is enclosed in double quotes.
GemHostIpAddress (String)
The IP Address of the gem’s end of the GCI client to gem connection; the result of calling getsockname() on the socket for the GCI client to gem connection. Does not do a reverse DNS lookup. For linked sessions, such as topaz -l, this field is empty.
GemClientIpAddress (String)
The IP Address of the gem’s client’s host, the result of calling getpeername() on the socket which is the gem's end of the GCI client to gem connection. For linked sessions, such as topaz -l, this field is set to gcilnk.
ObjectReadOop (Integer)
The OOP of the object read by the user.
ObjectReadClassOop (Integer)
The OOP of the class of the object read.
ObjectReadClassName (String)
The name of the class, UTF8 encoded.
gslist -m contacting remote NetLDI running with authentication
gslist -m hostNameOrIp contacts the NetLDI running on the remote host, to query for information on running processes on hostNameOrIp. If the NetLDI on the remote host is running with authentication (startnetldi -a argument), this could cause issues with Unix/LDAP account disable. If the NetLDI additionally was running in secure mode (startnetldi -s argument), gslist -m could not be used.
NetLDI -a attempted authentication with empty password
When a gslist -m request was made, it synthesized an empty password for the request to the remote NetLDI. If that remote NetLDI was running with authentication for new processes only (using startnetldi -a but not -s), the NetLDI attempted to authenticate using PAM, which would fail. This did not cause the gslist query to fail, since authentication for gslist is not required in this mode, and results were returned.
However, if this call was made often enough, these failures could result in the underlying account being locked as a result of too many failed authentication requests. (#49591)
gslist -m not usable with remote NetLDI in secure mode
When gslist -m contacts the NetLDI on a remote host, it uses the userID of the user executing the gslist, and an empty host password. If the remote NetLDI is running in secure mode (startnetldi -s argument), then gslist -m would fail. (#49594)
The following option has been added to gslist:
-a userid Unix userId used for contacting remote netldi
with -m. If -m also specified, prompts for a Unix password.
Has no effect unless -m is specified, required if remote netldi was started with -s.
Do not use if remote netldi started with -k (for Kerberos).
On a system where the NetLDI is running with authentication (the startnetldi -a but not -s), using -a with gslist -m to a remote host is optional. If the NetLDI is running in secure mode (startnetldi -a and -s options), the gslist -a option is required with -m. The gslist -a argument has no effect running on the local host.
When gslist includes the -m and -a arguments, gslist prompts the user, on the command line, for the password for the remote host.
Note that single sign on using Kerberos legitimately uses an empty password. This mode is not affected by changes in this release; the -a option should not be used in a Single sign-on system.
Multi-threaded warnings regarding cache slots use now include operation
MT operations that request more cache slots than are available print a warning message to stdout and the Stone log. These messages now include the operations, such as mfc or listinstances.
Number rounding/truncation
When a number is rounded using roundTo: or truncateTo:, it returns an object of the same class as the receiver, per the ANSI standard. However when rounding, for example, a float to an integer, it may be more useful to return an objects of the class of the argument.
New methods have been added to round/truncate and return an object of the class of the argument. These have been added to AbstractFraction, Float, Integer, and ScaledDecimal.
roundAndCoerceTo: aNumber
Returns the multiple of aNumber that is nearest in value to the receiver.
truncateAndCoerceTo: aNumber
Returns the multiple of aNumber that is closest to the receiver, on the same side of the receiver as zero is located. Returns the receiver if the receiver is a multiple of aNumber.
UserProfile userId permissible Character limitations
The UserIds of UserProfiles are limited to Strings containing Characters with values < 256.
Newly added UserProfiles are now required to also have all Characters within the String be alphanumeric, _ (underscore, code point 95), - (hyphen, code point 45), . (dot, code point 46), or space (code point 32).
Existing UserIds with other characters (this is expected to be unlikely), will continue to behave as before; however, if such UserIds exist in GemStone, sending GsObjectSecurityPolicy >> trackReads: will error.
Bugs Fixed
The following bugs in v3.5.6 are fixed in this version:
Incorrect results from Object >> passivate
Passivation of complex objects produced incorrect results in a few cases.(#49566)
If an instance of ScaledDecimal, Fraction, or FixedPoint has an instance variable containing a LargeInteger, and if that instance of ScaledDecimal, Fraction or FixedPoint appears more than once in the object graph, the passivation of the second reference is incorrect, and the reactivated object will contain an incorrect value/class.
When an Object's closure contains ExecBlocks (such as SortedCollections, which contain a sortBlock), the source string for the block is included in the passivated form. If in-memory GC occurs while passivating, the source string may be flushed and not included in the passivated form, which creates an error on activation.
Tranlog restore may encounter illegal store error
When restoring a transaction log into a restored database or a hot standby, there is a risk that the replay of a selective abort in the tranlog, under certain conditions may cause an illegal store error, which terminates the restore process. (#49578)
Upgrade from 3.2.x did not convert simple sortBlocks
The upgrade process for 3.2.x to 3.3 and later includes the postconv step, which recompiles the sortBlocks of SortedCollections. This recompile failed due to an overly strict block structure comparison. (#49634)
objectAudit/pageaudit do not correctly handle some kinds of error state
There are possible cases of internal corruption, that objectAudit and pageaudit do not handle correctly. objectAudit may fail to complete (#49619), and pageaudit may SEGV (#49617).
Shared Page Cache could have run out of slots
With a large database and a default shared page cache size, it was possible for an operation such as MFC to run out of PCEs (page cache entries), causing the Stone to shut down. (#49582)
GsFile position incorrect for r+ file mode with atEnd
If a GsFile was open for read/write, mode r+ (for example, GsFile class >> openUpdateOnServer:), then after invoking atEnd, the file position is incorrect; GsFile >> position will return 0. Subsequent writes will be in the correct location.
However, if GsFile >> position: is sent, followed by atEnd, a subsequent write may not be in the correct location. (#49568)
Failure in OS call from GsFile write primitives potential to SEGV
Within the GsFile write primitives, if an OS write call returned a negative value due to an unexpected write error, the code did not immediately return. Further execution could cause a SEGV. (#49540)
Page read error message unhelpful
The Error message for a failed read (4009) did not provide information about the page or extent, and was not specific about if a pageserver was involved. (#49613)
kernel.yama.ptrace_scope=1 disallowed stacks with NetLDI in authentication mode
When kernel.yama.ptrace_scope=1, gdb is disallowed from attaching to a process. This means that GemStone’s pstack utility, which is a wrapper for gdb, cannot be used to get stack traces. For systems in which the NetLDI is running in guest mode, on kill -USR1 pid or on fatal error, the GemStone process can still print stacks to the log file.
However, on systems with the NetLDI is in authentication mode, the real and effective UserIDs of a process are not the same. This prevented the GemStone process itself from getting a stack, so no useful stacks could be printed to the process log. (#49592)