1. Release Notes for 3.6.5


GemStone/S 64 Bitâ„¢ 3.6.5 is a new version of the GemStone/S 64 Bit object server. Version 3.6.5 adds support for monitoring via Prometheus, as well as fixing several significant bugs. We recommend everyone using or planning to use GemStone/S 64 Bit upgrade to this new version.

These Release Notes include changes between the previous version of GemStone/S 64 Bit, v3.6.4, and v3.6.5. If you are upgrading from a version prior to 3.6.4, review the release notes for each intermediate release to see the full set of changes.

The Installation Guide has not been updated for this release. For installation, upgrade and conversion instructions, use the Installation Guide for version 3.6.2.

Supported Platforms

Platforms for Version 3.6.5

GemStone/S 64 Bit version 3.6.5 is supported on the following platforms:

  • Red Hat Enterprise Linux Server, CentOS Linux, and Rocky Linux 7.9 and 8.5;
    Ubuntu 18.04 and 20.04
    GemStone performs testing on a mixture of Red Hat, CentOS, and Rocky servers; these are all fully certified platforms. Any reference to Red Hat applies to all these distributions.
  • Solaris 10 on x86
  • AIX 7.1 and 7.2
  • OSX 11.1 (Big Sur) with Darwin 20.2.0 kernel on x86, and OSX 10.15.6 (Catalina) with Darwin 19.6.0 kernel; and OSX 11.6 (Big Sur) with Darwin 20.6.0 kernel on Apple M1. v3.6.5 is known to work on MacOS Monterey.
    (Mac is supported for development only)

For more information and detailed requirements for each supported platforms, please refer to the GemStone/S 64 Bit Installation Guide for that platform.

GemBuilder for Smalltalk (GBS) Versions

The following versions of GBS can be used with GemStone/S 64 Bit version 3.6.5:

GBS/VW version 8.6


32-bit and 64-bit

  • Windows 10
  • RedHat ES 7.9 and 8.5; Ubuntu 18.04 and 20.04
GBS/VA version 5.4.6

VAST Platform

VAST Platform

VA Smalltalk

  • Windows Server 2016 and Windows 10
  • Windows Server 2016 and Windows 10
  • Windows Server 2016 and Windows 10

For more details on GBS and client Smalltalk platforms and requirements, see the GemBuilder for Smalltalk Installation Guide for that version of GBS.

VSD Version

The GemStone/S 64 Bit v3.6.5 distribution includes VSD version 5.5.4. The previous version of GemStone/S 64 Bit, v3.6.4, included VSD v5.5.3. VSD version 5.5.4 includes a fix for a rare but serious bug. For details on the changes, see the Release Notes for VSD v5.5.4.

VSD v5.5.4 is included with the GemStone distribution, and can also be downloaded as a separate product. For details or to download, go to https://gemtalksystems.com/vsd/.

Open Source Library Versions

The version of OpenSSL has been updated to 1.1.1p.

GemStone now includes additional libraries:

cJSON v1.7.15

prometheus-cpp v0.13.0

Support for cache monitoring via Prometheus

On Linux, it is now possible to start up a process that can be queried from the Prometheus monitoring software for GemStone cache statistics values.

This feature is provided by the new statprom executable. This is only available on the GemStone Linux distribution for x86_64.

A statprom process allows Prometheus to monitor one shared page cache, which may be the Stone’s cache or a remote cache. To monitor both a local and remote cache, you must run multiple statprom instances.

When statprom is started, it takes an argument configuration file that is customized for the cache and monitoring requirements; this includes the port number, the Stone name, and specific statistics to report. This configuration file is in JSON format.

While statprom can report any GemStone statistic originating in the shared page cache, host system statistics are not available.


Prometheus is an open-source systems monitoring and alerting toolkit, that collects and stores metrics (numeric data) as time series, along with key-value tags. Prometheus is widely used, and the Prometheus Github project has a active developer and user community.

In addition to Prometheus, Grafana can be installed and used for live monitoring of Prometheus data; Grafana provides out of the box support for Prometheus, and no additional configuration is needed to collect the GemStone data from Prometheus.

statprom is built on the open-source github project Prometheus Client Library for Modern C++ (jupp0r.github.io/prometheus-cpp). This imbeds a web server from the CivetWeb project (civetweb.github.io/civetweb), which handles http requests from Prometheus. statprom in turn uses the GemStone C Statistics Interface (GCSI) to access cache statistics. The GCSI attaches the process to the shared page cache as read-only; it does not create a gem session, and therefore has no view of the repository.

Configuring Statprom and Prometheus

Configure statprom

The statprom process must be started with a configuration file that specifies the stone or cache name, the port to expect queries from prometheus, and the specific cache statistics that can be returned to prometheus. The statprom configuration file is in JSON format and requires specific keys to be present in a specific structure.

For details on the file format, see Configuration file JSON format. An example configuration file is included in the distribution, $GEMSTONE/examples/Prometheus/stats.json.

The statprom configuration file allows you to monitor the Stone, the Shared Page Cache Monitor, or Gems. Any GemStone cache-based statistic of these process can be accessed; however, host process statistics cannot be monitored directly.

Additional configuration requirements for monitoring Gems

The Stone and Shared Page Cache Monitor are singletons within a cache, and thus straightforward to monitor. However, a GemStone system contains multiple gems with different purposes and monitoring requirements. Prometheus does not handle multiple instances, so there is additional configuration required for monitoring Gems.

When multiple Gems match the criteria, the statistics values for all of them are added together. The monitoring criteria should be designed carefully so that either a single Gem process is identified, or if there is a chance that multiple Gem processes will be matched, that the specific statistic being monitored can logically be summed over all the matched Gems.

There are two options for filtering on the Gem you wish to monitor:

  • ProcessName

Cache statistics data includes an entry for a String ProcessName, which is set by the System; for example, Gem or TopazL. This can be set to a specific value in your application in several ways: using the -u option on the topaz command line, executing System cacheName: gemName, or set cachename gemName in topaz (note that topaz set cachename takes effect on the next login, and does not affect any current sessions).

statprom matches the Gem ProcessName using regular expressions, e.g. MyGemName*.

  • GemKind

Starting with this version, cache statistics data includes an entry for an integer GemKind. This is 0 by default, and negative for system Gems. This can be set using System setGemKind: anInt. Negative values are reserved for use by GemTalk.

statprom matches the Gem processes’ GemKind, using a high and low value, forming an inclusive range.

Statprom usage

statprom -f cfgFile [-c] [-d] [-r]
statprom -h | -v
-c	Check the JSON file (-f argument) for errors and exit.
	Requires -f.
-d	Enable printing debug output to stderr.
-f <cfgFile>
	Specifies a configuration file in JSON format which
	determines which processes and statistics are collected.
-h	Print this help screen and exit.
-r	Retry if the shared page cache not running. If the cache
	connection is lost, sleep and attempt to reattach.
	Without -r the process exits if the cache connection is
	lost or not present at startup time.
-v	Print the program version and exit.

Configure Prometheus

Prometheus in turn must be started with a configuration file that specifies the node on which statprom is running and the configured port.

For example, the prometheus configuration file may include an additional entry such as:

- job_name: 'gemstone'
      - targets: ['nodename.gemtalksystems.com:9985']


Once Prometheus is started with its updated configuration file, and statprom has been started with its configuration file, Prometheus will starting querying and recording information for the specified statistics.

Prometheus will retrieve and store values with the name provided in the statprom configuration file. In addition to data tags such as the job name, the name of the Stone is provided as a tag with the key StoneName.

Configuration file JSON format

A sample JSON file, that could be used as an argument to statprom, is included in $GEMSTONE/examples/Prometheus/stats.json.

This provides an example of the main features of statprom configuration.

“http” Section
  • listen_addresses
    This argument povides informatoin on the port that statprom will listen on for connections from Prometheus. It is an Array of strings representing port numbers and interfaces/addresses. Both IPv4 and IPv6 addresses are supported. The default port number for statprom is 9995. The address format is specific to the civetweb webserver, see: https://civetweb.github.io/civetweb/UserManual.html

For example:

"http" : { 
   "listen_addresses" : ["[::]:9985","9985"] 

specifies that either IVp6 or IVP4 connection is accepted on port 9985 on localhost.

Note that statprom does not support ssl connections at this time.

“gemstone” Section
  • cache_name
    a String which represents the name of the remote shared page cache to monitor or NULL. Only relevant when monitoring a remote shared page cache; this must be NULL when monitoring the Stone’s cache. The cache name may be obtained from gslist. The cache name is derived from a GemStone-generated hostId and should not change over the life of a host.
  • stone_name
    a String which represents the name of the Stone to monitor or NULL. Only relevant when monitoring a primary shared page cache; should be NULL when monitoring a remote cache.
  • sample_interval
    an Integer representing the sample interval, in seconds.

For example:

"gemstone" : { 
   "cache_name" : null, 
   "stone_name" : "gs64stone", 
   "sample_interval": 60 
“metrics” Section

Three types of metrics are supported: monitor (shared page cache monitor), stone, and gem. You do not need to include all three types.

For each type, there should be an array of specific metrics to be monitored for that type. The JSON metrics objects have the following members:

  • vsd_name (required)
    a String which represents the vsd name of the statistic to monitor. Must exactly match the name of the statistic. It is an error if the statistic name does not exist for the given metric type.
  • metric_name (required)
    a String which represents the name of the statistic in Prometheus. All values of metric_name in the configuration file must be unique. This is a limitation of Prometheus which does not allow duplicate names.
  • metric_type (required)
    a String representing the Prometheus type for the stat. Accepted values are:
  • Counter – values that only increase and never decrease
  • Gauge – values that may either increase or decrease
  • Histogram – values that will be plotted as a histogram.
  • metric_help (required)
    a String which describes the function of the statistic.
  • metric_units (required)
    a String which describes the measurement units of the statistic.

For example,

"metrics" : { 
   "stone" : [ 
      "vsd_name" : "CommitRecordCount", 
      "metric_name" : "gemstone_stone_commit_records", 
      "metric_type" : "Gauge", 
      "metric_help" : "Number of commit records.", 
      "metric_units" : "Commit Records" 

In addition to the above object members, entries for Gems require a filter criteria; either cache_name_regex, or both gem_kind_min and gem_kind_max, or all three.

  • cache_name_regex
    a String which represents a regex expression used to match against the cache names of all gems in the cache.
  • gem_kind_min
    an Integer indicating the minimum value of the GemKind statistic used to match against gems in the cache. Requires gem_kind_max be specified.
  • gem_kind_max
    an Integer indicating the maximum value of the GemKind statistic used to match against gems in the cache. Must be greater than or equal to gem_kind_min. Requires gem_kind_min be specified.

All Gems in the cache that match the filter criteria have their statistics values added together for return to Prometheus. You must be careful to ensure that either a unique Gem can be confidently matched to the filter criteria, or that the particular statistic values are meaningful if added together for multiple Gems.

If you have multiple Gems with statistics that need to be separately reported, you will need separate entries, each with a unique Prometheus statistic name, for each Gem/statistic.

Gem matching examples

Since Gems matching the filter criteria have values summed, you can make use of it, for example, if tasks are divided over multiple Gems.

For example:

"gem" : [ 
  "cache_name_regex" : "Widget.*", 
  "vsd_name" : "SessionStat01", 
  "metric_name" : "gemstone_gem_widgets_produced", 
  "metric_type" : "Counter",  
  "metric_help" : "Number of widgets produced.", 
  "metric_units" : "Widgets" 

With this example, if there are multiple Gems with names that match Widget, the values will be summed. E.g. if there are the following Gems in the cache:

Gem Widget1, SessionStat01 == 6 
Gem Widget2, SessionStat01 == 8 

In this case, Prometheus will show the aggregate value of 14 (6 + 8) for the statistic gemstone_gem_widgets_produced.

Note that this monitor example would also match Gems named WidgetLogging, WidgetDefects, and so on; you must be aware of your application’s Gem cacheName conventions.

If you do not wish to sum the values, you must be able to identify the specific Gem or Gems for reporting, and create specific entries for each individual value. In the above example, if you wished to monitor Widget1’s 6 and Widget2’s 8 separately, rather than summed, you could create the following entries:

"gem" : [ 
  "cache_name_regex" : "Widget1", 
  "vsd_name" : "SessionStat01", 
  "metric_name" : "gemstone_gem_widgets1_produced", 
  "metric_type" : "Counter",  
  "metric_help" : "Number of widgets produced by Widget1.", 
  "metric_units" : "Widgets" 
  "cache_name_regex" : "Widget2", 
  "vsd_name" : "SessionStat01", 
  "metric_name" : "gemstone_gem_widgets2_produced", 
  "metric_type" : "Counter",  
  "metric_help" : "Number of widgets produced by Widget2.", 
  "metric_units" : "Widgets" 

Validating JSON configuration file

After editing your .json file, you may use statprom to verify that it is valid for use with statprom, using the -c option. With this option, the configuration file is validated, but statprom is not started.

statprom -c -f promfile.json

Other Changes and Bug Fixes

New GemKind statistic

A new attribute has been added to distinguish Gems in cache statistics, GemKind. This is an integer that can be used instead of or along with the Process’s cacheName to identify specific Gems or sets of Gems for monitoring by Prometheus, as well as other situations in which specific-purpose Gems need to be distinguished.

By default, GemKind is 0, and can be set in Smalltalk code to a positive integer. It is set to a negative value by GemStone internal operations such as markForCollection; negative values are reserved for use by GemTalk.

See the use of GemKind by statprom for Prometheus monitoring here.

GemKind (Gem)
A signed integer indicating the specific type of the Gem. 0 be default, set to a negative value for system Gems; negative values are reserved for GemStone internal use.

The following method has been added:

System >> setGemKind: aSmalInt
Set the GemKind cache stat to be aSmallInt. The default GemKind is zero. Customers are free to use any positive value for this statistic. Negative values are reserved for internal use, and set the GemKind for System processes such as reclaim and markForCollection.

Other added Cache Statistics

ReposAtMaxSize (Stn)
A boolean, true if all extents are at maximum configured size or all have failed grow due to disk full

TranState (Stn)
Value of TranState._state:
0 - startup
1 - pageAudit
2 - recovery
3, 4 - restore from log
5 - normal

Added GCSI functions


(int) GcsiCacheIsAlive(

Check the status of the cache.

Function returns:

1 - cache is alive and we are attached.
0 - cache has died and/or we are not or no longer attached.


Search the cache for gems with a gemKind stat value within a given range. Return cache stats for matching gems or a count of how many gems meet the criteria.

(int) GcsiAllGemStatsForGemKindRange(
    int gemKindMin,
    int gemKindMax,
    GcsiResultSType *result,
    int *resultSize


gemKindMin Minimum value of gemKind to match to.

gemKindMax Maximum value of gemKind to match to; must be equal or greater than gemKindMin.

result Address of an array of kind GcsiResultSType where statistics will be stored. NULL means return only a count of how many gems match.

resultSize Pointer to an integer that indicates the size of the result in elements (not bytes). On return, indicates the number of gems whose gemKind is in the given range, inclusive. If result is not NULL, this is also the number of valid elements in the result array.

Function returns:

0 - success

Otherwise an error code. See gcsierr.ht for a list of error codes.


Available on Linux only.

Uses a regular expression to search against the cache name of every gem connected to the shared page cache and return stats for those that match.

(int) GcsiAllGemStatsForRegexName(
    const char *regexString,
    BoolType ignoreCase,
    GcsiResultSType *result,
    int *resultSize,
    char *errBuf,
    size_t errBufSize


regexString A valid regular expression string. This string will be used to match against the cache name of every gem attached to the shared page cache.

ignoreCase TRUE causes the regex search to ignore case.

result Address of an array of kind GcsiResultSType where statistics will be stored. NULL means return only a count of how many gems match.

resultSize Pointer to an integer that indicates the size of the result in elements (not bytes). On return, indicates the number of gems whose cache name match. If result is not NULL, this is also the number of valid elements in the result array.

errBuf Buffer which will contain a string describing an error if regexString is invalid.

errBufSize Size of errBuf in bytes.

Function returns:

0 - success

Otherwise an error code. See gcsierr.ht for a list of error codes and examine errBufMsg for an error message.

Bugs Fixed

Performance of markForCollection for long object chains

In versions 3.5.x and 3.6.x, when the object environment includes very long chains of objects (a depth of multiple hundreds), the performance of markForCollection and other repository wide mark/sweep operations, such as epoch and write set union sweep, was substantially worse than in 3.3.x. (#50025)

In a hot standby system, shutdown of slave Stone may hang

When a logreceiver is connected to a slave stone, and the slave stone is executing continousRestore, if stopstone is attempted on the slave stone, the stone will fail to shutdown and stopstone will hang. (#50013)

Multi-file encrypted backup fails when signing key is DSA

When creating an encrypted programmatic backup, if the backup writes to multiple files and the signing key is DSA rather than RSA, the backup will fail, or crash, depending on the version. (#47207)

Reclaim may be deferred when extents are full but could grow

When running with no size limits on extents, and no shortage of disk space, the Reclaim GcGem may defer reclaim due to the free space being below the threshold, rather than the extents being grown. (#49993)

Slow stone restart after unexpected shutdown during reclaim

When the Stone is restarted after an unexpected shutdown and has to recover, and there are many unreclaimed dead objects, the reclaim does not correctly reclaim priority pages, making the restart unnecessarily slow. (#49998)

Warning message during login

On login, it was possible to encounter warnings of the form:

[Warning]: Unable to find /proc/23456/oom_score_adj or /proc/23456/oom_adj

This release includes code changes to address this issue. (#49990)

DEBUGGEM messages sent to stderr

In v3.6.4, DEBUGGEM messages (which are reported when GEM_LISTEN_FOR_DEBUG= true), were printed to stderr as well as stdout. In v3.6.5, messages are not sent to stderr. (#50009)