2. Changes in Administration
This chapter describes changes and new features that apply to the administration of a GemStone Repository, and changes in the GemStone environment, configuration, and tools, including:
2.1 Encrypted Extents and Transaction Logs
GemStone provides user and object level security to protect your application data from unauthorized access via applications or Smalltalk code execution. However, the repository extents contain all repository objects, and the tranlogs record recent changes; these may include readable data, and are normally protected by disk permissions from unauthorized access.
With v3.6, you may also configure your system to keep the extents and transaction logs encrypted on disk at all times.
Encrypted Extents
With encrypted extent files, the on-disk extents are encrypted at all times. Transaction logs written by a Stone using encrypted extents are always written in encrypted form, using the same keypair as the Stone.
To use encrypted extents, you must manually encrypt the repository extent files prior to starting the Stone, using a public key or X509 certificate. The startstone command takes additional arguments, specifying the private key. If the repository has more than one extent, all extents must be encrypted using the same key. Only one keypair can be associated with encrypted extents.
The encryption step is done using a public key; to start the stone on these extents, the private key corresponding to the public key (and the passphrase for the private key, if configured) are required. Encryption is done using AES-XTS symmetric encryption with a key length of 128 or 256 bits.
In an encrypted extent, all data in the extent is encrypted, except for a small number of records with metadata information. This unencrypted information also allows copydbf -i to report information without needing the key.
Once the Stone has started, the private key is not needed again until the next time the Stone is restarted. Pages from the extents are decrypted before reading into the cache, and encrypted on write back to disk. Since the Stone may need restart at any time if an unexpected outage occurs, the private key must be kept readily available.
Internal handling for other sessions
The Stone uses the private key argument provided by startstone to decrypt the session key stored in each of the extents. Each extent has its own unique session key. This is stored in private stone memory, and so is not accessible to other sessions on the shared page cache. Processes that read or write to the extents need access to this key.
For gems, which may be on remote hosts over unsecured networks, a random key is generated and used to encrypt the keys, which are provided in shared memory for local gems, or over an encrypted SSL socket for remote gems.
The first gem to attach to a remote shared page cache stores the keys in the encrypted extent key table in the remote cache, under spin lock control.
Transaction Logs
Transaction logs written by a Stone using encrypted extents are written in encrypted form, using the same keypair as the extents. When the Stone is started, if recovery is needed (that is, on unclean shutdown), it will be able to recover from transaction logs since they were encrypted using the same key as was used to start the Stone. Special care is needed when updating the encryption key of the Stone on an extent that has not been cleanly shutdown.
If the encryption key of the extents have changed, on startup after a clean shutdown, the Stone will start a new transaction log that is automatically encrypted with the new keys. In this case, you must take care to keep the older keypairs, or update the encryption keys for the older tranlogs, if you want to be able to restore these older tranlogs into an older backup.
Restart and Recovery
If the stone unexpectedly shuts down, the private key is needed to restart the stone. Automatic recovery from transaction logs requires that all required transaction logs for recovery be encrypted using the same key as the stone. If necessary, the transaction logs may have their keys updated using updatesecuredbf.
Programmatic Backups
Making a programmatic backup of a repository running with encrypted extents does not automatically create an encrypted backup. You must use the backup creation methods that specifically create encrypted backups.
- the standard methods Repository >> fullBackupTo:... make unencrypted backups, regardless of whether the running Stone’s extents are encrypted or not.
- the secure backup methods Repository >> secureFullBackupTo:... make encrypted or unencrypted/signed backups, using the key/s provided to that method, regardless of whether the running Stone’s extents are encrypted or not.
Extent Copy Backup
An extent copy backup of a repository with encrypted extents is, of course, encrypted.
Ensure that the private key remains available in order to decrypt or restore. If the certs and keys need to be updated, you can use updatesecuredbf on the extent copy backups, but you will need to use the -O option, since the extents were not cleanly shutdown.
Solo logins
Solo login sessions, sessions that use an extent file directly to login, without starting a Stone process, are disallowed with encrypted extents.
Generating keys
The following process can be used to generate your own public/private key pairs. The example uses $GEMSTONE/bin/openssl, but you may use any openssl executable.
unix> openssl genrsa -out privateKeyName 2048
Generating RSA private key, 2048 bit long modulus (2 primes)
....................+++++
.............+++++
e is 65537 (0x010001)
2. Extract the public key from the private key:
unix> openssl rsa -in privateKeyName -pubout -out publicKeyName
writing RSA key
3. Optional but strongly recommended: protect the private key with a passphrase.
unix> openssl rsa -aes256 -in privateKeyName -out privatePPKeyName
writing RSA key
Enter PEM pass phrase: <enter a passphrase>
Verifying - Enter PEM pass phrase: <enter passphrase again>
Then delete the unprotected private key:
unix> rm privateKeyName
Example Setting up Encrypted Extents
The following example shows the steps to encrypt a two-extent repository with a key from the GemStone distribution. Note that this does not provide security; you must generate your own private key meeting your own security requirements.
Step 1. Stop the Stone cleanly. While it is possible to encrypt an extent that is not cleanly shutdown, you would also need to encrypt the tranlogs needed for recovery.
unix> stopstone stoneName administrativeUser password
Step 2. Make encrypted copies of your extents using copydbf.
unix> copydbf $GEMSTONE/data/extent0.dbf
$GEMSTONE/data/secextent0.sdbf -e server_1_servercert.pem
-s 256 -K $GEMSTONE/examples/openssl/certs
unix> copydbf $GEMSTONE/data/extent1.dbf
$GEMSTONE/data/secextent1.sdbf -e server_1_servercert.pem
-s 256 -K $GEMSTONE/examples/openssl/certs
Step 3. Edit the configuration file used by the Stone to specify the new extent names.
DBF_EXTENT_NAMES = $GEMSTONE/data/secextent0.sdbf,
$GEMSTONE/data/secextent1.sdbf;
Step 4. Execute startstone using the private key.
unix> startstone -D server_1_serverkey.pem
-J $GEMSTONE/allcerts/server_1_server_passwd.txt
-K $GEMSTONE/examples/openssl/certs -K $GEMSTONE/examples/openssl/private gs64stone
On startup, the Stone will immediately start a new transaction log that is encrypted with the key server_1_servercert.pem.
Using the stone, and stopping the stone, now proceed as with any GemStone installation.
Encrypted Extents with Backup and Restore
How to make and restore the various kinds of backups, including encrypted backups, is described in the System Administration Guide, Chapter 11.
Making a backup in a repository configured with secure extents is no different than making a backup with ordinary extents. From a repository configured with encrypted extents, you may make:
- Online or offline extent snapshot backups, which remain encrypted with the same keys as the working system extents.
- Ordinary programmatic backups, which are not encrypted. An ordinary backup from a repository using encrypted extents is not automatically encrypted.
- Secure programmatic backups, which are signed and may either be encrypted or not encrypted. Encryption keys are provided as argument to the secure backup methods, so the backup may be encrypted using the same keypair as the working system’s extents, or an entirely different keypair.
While restoring a programmatic fullbackup or an extent snapshot back into a repository with extent encryption is not substantially different from restoring into a repository with regular unencrypted extents, if the encryption keypairs that are used to encrypt the extents has changed, this will impact restoring transaction logs.
Restoring Backups
When restoring a programmatic fullbackup to a system that is using encrypted extents, you will need to start with a clean, encrypted extent. This requires an additional step to use copydbf to encrypt a clean, empty extent.
Alternately, you can restore into an unencrypted extent, and then shut down the restored stone, encrypt the extent files, and restart.
Of course, all startstone invocations will require the appropriate key for the encrypted extents of the Stone you are restoring.
The restore process is similar to the processes described in the System Administration Guide, Chapter 11.
Restoring transaction logs with no changes in encryption keys
When using encrypted extents, the transaction logs are also encrypted with the same keypair as the Stone. As long as all required transaction logs for restore are encrypted with the same keypair as the newly restored Stone (or not encrypted), then restoring transaction logs into a Stone with encrypted extents follows the same process as restoring into a Stone with regular (non-encrypted) extents.
Restoring transaction logs with changes in encryption keys
When using encrypted extents, over time you may change the keypair used to encrypt the extents (using updatesecuredbf or copydbf twice to decrypt and re-encrypt). When the stone is restarted with the new key, it will create a new transaction log that is encrypted using the new keypair. Since the full series of transaction logs will be encrypted with multiple keys, special handling is required to allow all the transaction logs to be restored.
- Use updatesecuredbf to update the encryption key for every transaction log so it is using the current Stone’s encryption key. Any tranlogs that do not have the Stone’s encryption key cannot be read, and restore will stop there.
- Use the method Repository >> setTranlogPrivateKeysForRestore: to provide the stone with the private key for all of the transaction logs that you will be restoring. Any transaction logs using keys that are not provided to the stone cannot be read, and restore will stop there.
- Always make backups immediately after changing the encryption key, and set up your systems for backup and restore, such that you will never need to restore older encrypted repositories. For example, you may decrypt the older backups and transaction logs and archive these as insecure files using your own secure storage mechanisms.
Update transaction log keys
The following examples update tranlogs to the new Stone’s key, server_3_servercert.pem.
In this example tranlog1.sdbf was encrypted with server_1_servercert.pem, and tranlog2.sdbf was encrypted with server_2_servercert.pem.
unix> updatesecuredbf tranlog1.sdbf -e server_3_servercert.pem
-D server_1_serverkey.pem
-J $GEMSTONE/examples/openssl/certs/server_1_server_passwd.txt
-K $GEMSTONE/examples/openssl/certs
-K $GEMSTONE/examples/openssl/private
unix> updatesecuredbf tranlog2.sdbf -e server_3_servercert.pem
-D server_2_serverkey.pem
-J $GEMSTONE/examples/openssl/certs/server_2_server_passwd.txt
-K $GEMSTONE/examples/openssl/certs
-K $GEMSTONE/examples/openssl/private
Providing private keys to the Stone in Smalltalk code
Rather than updating all the keys for existing transaction logs, you may pass into the stone all the private keys for all the transaction logs that you will be restoring, using the method Repository >> setTranlogPrivateKeysForRestore: and creating instance of GsTlsPrivateKey.
Repository >> setTranlogPrivateKeysForRestore: arrayOfGsTlsPrivateKey
Specifies a list of instances of GsTlsPrivateKey representing private keys, that will be used to read encrypted tranlogs during a restoreFromArchiveLogs operation. The private key used to start the stone does not need to be included.
During a restoreFromArchiveLogs operation, the stone searches arrayOfGsTlsPrivateKey for a match to the public key stored in the encrypted tranlog. An error is raised if no match is found, and the restore operation fails.
GsTlsPrivateKey class >> newFromPemFile: pemFilename
Create an instance of GsTlsPrivateKey for the private key in the given filename. This key must not require a passphrase.
GsTlsPrivateKey class >> newFromPemFile: pemFilename withPassphraseFile: passphraseFilename
Create an instance of GsTlsPrivateKey for the private key in the given filename. The key in this file is assumed to require a passphrase, which is in the given passphrase file.
There are other instance creation methods on GsTlsPrivateKey; refer to the image for other options.
Once GsTlsPrivateKeys are provided to the Stone, this list of keys is saved in the Stone, so you may execute this expression multiple times with new arguments, without losing key information.
Example 2.1 Setting tranlog keys for restore
topaz 1> run
SystemRepository setTranlogPrivateKeysForRestore: {
(GsTlsPrivateKey newFromPemFile:
'$GEMSTONE/examples/openssl/private/server_1_serverkey.pem'
withPassphraseFile:
'$GEMSTONE/examples/openssl/private/server_1_server_passwd.txt')
.
(GsTlsPrivateKey newFromPemFile:
'$GEMSTONE/examples/openssl/private/server_2_serverkey.pem'
withPassphraseFile:
'$GEMSTONE/examples/openssl/private/server_2_server_passwd.txt')
}
%
Page Audit
To run pageaudit on a system with encrypted extents, you must include the same private key information as when starting a stone.
For example, when the extent files for a repository are encrypted using the key server_2_servercert.pem, then the following invocation will allow pageaudit to execute:
unix> pageaudit -D server_2_serverkey.pem
-J $GEMSTONE/examples/openssl/private/server_2_server_passwd.txt
-K $GEMSTONE/examples/openssl/private
Hot Standby
You may use encrypted extents within a warm or hot standby system. The logsender and logreceiver do not require any special configuration; they pass along the records without decrypting them.
The slave stone must be started with the same encryption key as the master stone, so that the process restoring the tranlogs is able to decrypt them. If the encryption key on the master is changed, you should also update the encryption key on the slave’s extents.
Alternatively, you can provide the new key to the slave system using setTranlogPrivateKeysForRestore:; however, this is not retained over shutdown/restart of the slave system, since it is in restore mode. After a restart of the slave system you will need to execute setTranlogPrivateKeysForRestore: again.
printlogs/searchlogs
Tranlog analysis works with encrypted transaction logs; see printlogs and searchlogs.
Programmatic Information about encryption
Methods have been added to System class to allow you to fetch information about encryption status.
System class >> stoneCommandLineArgs
Returns a String representing the command line arguments used to start the stone.
System class >> repositoryEncryptionScheme
Answers a symbol describing the algorithm used to encrypt the repository (extents and tranlogs) or #none if the repository is not encrypted.
System class >> repositoryIsEncrypted
Answers a boolean indicating if the repository is encrypted.
System class >> repositoryPrivateKey
If the repository is encrypted and the current user is SystemUser, returns an instance of GsTlsPrivateKey which represents the private key used to decrypt the repository. Returns nil if the repository is not encrypted. Raises an exception if this method is invoked by any user except SystemUser.
System class >> repositoryPublicKey
If the repository is encrypted, returns an instance of GsTlsPublicKey or GsX509Certificate which represents the public key used to encrypt the repository. Returns nil if the repository is not encrypted.
Modifying Encrypted Files
The utilities copydbf and updatesecuredbf allow you to perform a number of operations on encrypted extents, transaction logs, and backup files.
- copy the file without changing encryption state. This does not require cert files.
- encrypt an extent or transaction log file (backups cannot be encrypted using copydbf)
- decrypt an extent, transaction log, or backup to a regular file
- change the encryption key of an extent or transaction log, or one of the encryption keys of an encrypted backup, to a different key.
Note that it is possible to create secure backups that are signed, but not encrypted. The signing key for these backups cannot be changed, and these backups cannot be converted into encrypted nor into regular files.
Examples of modifying encrypted files
Keys are not needed to make a copy of an encrypted dbf. copydbf on an encrypted file, without using decryption keys, results in a file that is encrypted with the same key as the source file.
To encrypt an extent
This example encrypts the distribution extent into secextent0.sdbf.
unix> copydbf $GEMSTONE/bin/extent0.dbf
$GEMSTONE/data/secextent0.sdbf -e server_2_servercert.pem
-s 256 -K $GEMSTONE/examples/openssl/certs
If you have multiple extents, you must encrypt all extents with the same key, to be able to start a Stone.
The same process can be used to encrypt an extent snapshot backup. Unencrypted programmatic fullbackups cannot be encrypted.
You may also encrypt an unencrypted transaction log, although normally encrypted transaction logs are created by a Stone using encrypted extents.
To start a stone on an encrypted extent
When the Stone’s configuration file specified extent files that are encrypted using the key server_2_servercert.pem, then the following invocation will start the stone on those extents.
unix> startstone -D server_2_serverkey.pem
-J $GEMSTONE/examples/openssl/private/server_2_server_passwd.txt
-K $GEMSTONE/examples/openssl/private gs64stone
To decrypt an extent or extent snapshot backup
This example decrypts secextent0.sdbf, which was encrypted with server_2_servercert.pem, into decrypextent0.dbf.
unix> copydbf secextent0.sdbf decrypextent0.dbf
-D server_2_serverkey.pem
-J $GEMSTONE/examples/openssl/private/server_2_server_passwd.txt
-K $GEMSTONE/examples/openssl/certs -K $GEMSTONE/examples/openssl/private
If you have multiple extents, you must decrypt all extents to be able to start a Stone.
To decrypt an extent or extent snapshot backup
This example decrypts sectranlog1.sdbf into decryptranlog1.dbf.
unix> copydbf sectranlog1.sdbf decryptranlog1.dbf
-D server_2_serverkey.pem
-J $GEMSTONE/examples/openssl/private/server_2_server_passwd.txt
-K $GEMSTONE/examples/openssl/certs -K $GEMSTONE/examples/openssl/private
Decrypted transaction logs can be restored without special handling, including in a series with other transaction logs that are encrypted.
To change the key of an encrypted extent or extent snapshot backup
This example updates the original key server_2... to server_3... The modification is done to secextent0.dbf in place; we recommend making a copy before using updatesecuredbf.
The stone using this extent must have been shut down cleanly.
unix> updatesecuredbf secextent0.sdbf
-e server_3_servercert.pem -D server_2_serverkey.pem
-J $GEMSTONE/examples/openssl/private/server_2_server_passwd.txt
-K $GEMSTONE/examples/openssl/certs -K $GEMSTONE/examples/openssl/private
If you have multiple extents, you must update the keys for all extents to the same key to be able to start a Stone.
To change the key of an encrypted transaction log
This example updates the original key server_2... to server_3... The modification is done to sectranlog1.dbf in place; we recommend making a copy before using updatesecuredbf.
unix> updatesecuredbf sectranlog1.sdbf
-e server_3_servercert.pem -D server_2_serverkey.pem
-J $GEMSTONE/examples/openssl/private/server_2_server_passwd.txt
-K $GEMSTONE/examples/openssl/certs -K $GEMSTONE/examples/openssl/private
Transaction logs are normally part of a series that are retained to allow restore. Before a transaction log can be restored, it must either be using the same key as the extents for the Stone into which it is being restored, or the keys must be supplied using setTranlogPrivateKeysForRestore:.
To change the encryption key of an encrypted fullbackup
Encrypted programmatic fullbackups are always signed; you will need to supply the signing key (and passphrase, if required), as well as the previous and new encryption keys.
Encrypted backups may be encrypted with up to 8 different public keys, and the private key corresponding to any of these public keys can be used to decrypt. An invocation updatesecuredbf changes a single one of these public key encryptions, leaving the others unchanged.
unix> updatesecuredbf encryptedBackup1.sdbf
-e backup_encrypt_2_clientcert.pem
-J $GEMSTONE/examples/openssl/private/backup_encrypt_1_clientcert_passwd.txt -D backup_encrypt_1_clientcert.pem
-S backup_sign_3_clientkey.pem
-T $GEMSTONE/allcerts/backup_sign_3_client_passwd.txt
-K $GEMSTONE/examples/openssl/certs
-K $GEMSTONE/examples/openssl/private
For a multi-file fullbackup, you must update they keys for all backup files to allow them to be used for restore.
To decrypt a fullbackup
This example decrypts encryptedBackup1.sdbf into Backup.dbf, using the private encryption key backup_encrypt_2_clientkey.pem.
unix> copydbf encryptedBackup1.sdbf Backup.dbf
-D backup_encrypt_2_clientkey.pem
-J $GEMSTONE/examples/openssl/private/backup_encrypt_2_client_passwd.txt -K $GEMSTONE/examples/openssl/certs -K $GEMSTONE/examples/openssl/private
For a multi-file fullbackup, you should decrypt all backup files to allow them to be used for restore.
Encrypting and Decrypting GsTls* keys
The following methods allow some ability to encrypt and decrypt GsTlsPublicKey, GsX509Certificate, and GsTlsPrivateKey.
GsTlsPublicKey >> encrypt: aByteObj
Encrypts aByteObj using the receiver which must be an RSA public key or X509 certificate object. This method supports RSA keys only. Encryption is performed using the RSA-OAEP padding algorithm.
Only small amounts of data may be encrypted using asymmetric keys. The maximum allowed size of aByteObj is dependent on the number of bits in the receiver's public key. For 2048 bit keys, the maximum size is 214 bytes. For 4096 bit keys, the maximum size is 470 bytes.
aByteObj must be an instance of a byte class, typically ByteArray. Byte objects with character sizes greater than 1 are not supported.
GsX509Certificate >> encrypt: aByteObj
Encrypts aByteObj using the receiver which must be an RSA public key or X509 certificate object. This method supports RSA keys only. Encryption is performed using the RSA-OAEP padding algorithm.
Only small amounts of data may be encrypted using asymmetric keys. The maximum allowed size of aByteObj is dependent on the number of bits in the receiver's public key. For 2048 bit keys, the maximum size is 214 bytes. For 4096 bit keys, the maximum size is 470 bytes.
GsTlsPrivateKey >> decrypt: aByteArray
Decrypts aByteArray, which was previously encrypted with the receiver's matching public key. Only RSA keys are supported for encrypting and decrypting.
Examples are provided in the following methods:
2.2 Changes in Users and ObjectSecurityPolicies
CodeLibrarianUser no longer needed
The CodeLibrarianUser, and the associated CodeLibrarianObjectSecurityPolicy, is no longer needed in 3.6, with the changes in the way GemStone code is stored and built which bring GemStone further into alignment with Rowan code management.
This UserProfile, and the object security policy, are removed in new repositories. Upgraded repositories will contain CodeLibrarianUser in AllDeletedUsers, and the object security policy will remain but is not used.
DataCurator no longer has permission to modify Globals
In previous versions of GemStone, the Globals SymbolDictionary had the DataCuratorObjectSecurityPolicy. As of v3.6, the Globals object has the SystemUserObjectSecurityPolicy; so, only SystemUser is able to add, remove, or modify entries in Globals.
While necessary to close the security hole, this change means that some operations that were previously possible, such as adding or removing keys from Globals, including installing Portable or Legacy Streams, will now encounter a security error, and should be performed by SystemUser rather than DataCurator.
SystemUser no longer subject to privilege checks
SystemUser is intended to be a super-user, without restrictions to its powers. While methods that removed privileges from SystemUser were disallowed, there were ways to modify the SystemUser’s privileges, which left the SystemUser without the ability to perform the privileged tasks. (#48905)
Now, SystemUser is not subject to SecurityErrors when performing privileged tasks, regardless of its permissions.
SystemUser now exposes sub-dictionaries
The SystemUser’s SymbolList now includes SymbolDictionaries that were previously only referenced from keys within Globals and thus were less visible. This includes ObsoleteClasses, GsCompilerClasses, the Streams hierarchies, and new Rowan Kernel SymbolDictionary.
Changes in handling of transient symbolList
Previously, each session always had a transient symbolList, which was a copy of the symbolList for that UserProfile, and the transient symbolList was used for symbol resolution. Now, by default the persistent symbolList is used, making it easier for applications that do not need to distinguish between transient and persistent symbolLists. Querying for or assigning a transient symbol list, creates a copy of the persistent symbolList and provides the previous behavior.
In particular, previously UserProfiles without CodeModification privilege could not easily pick up changes in the persistent symbolList.
The new handling of the transient symbolList is described in more detail under Change in handing of transient symbol list.
Whitelist for performOnServer:
Accessing OS functions using performOnServer: (and the related GsHostProcess functionality) is very useful, and for some applications it may be necessary for users that should not have general OS command access, to be able to execute specific OS commands.
This can be set up via a per-user command whitelist. To allow a user with NoPerformOnServer privilege to execute a specific command, the full path (starting with /) to the command is put on a whitelist for that specific user. The full path must then be specified on the performOnServer: or GsHostProcess method argument; the paths and commands must match exactly for the execution to be permitted.
There are no checks or restrictions on arguments to the commands; only the commands themselves are examined and allowed (or not).
The following methods add one or more commands to the whitelist:
UserProfile >> addPerformOnServerCommand: fullPathToCommand
UserProfile >> addPerformOnServerCommands: arrayOfCommands
To report the existing whitelist:
UserProfile >> performOnServerCommands
To remove commands from the whitelist:
UserProfile >> removeAllPerformOnServerCommands
UserProfile >> removePerformOnServerCommand: aString
UserProfile >> removePerformOnServerCommands: arrayOfString
The requirement to use full paths in the whitelist and in the argument to performOnServer: applies to operating system commands such as ls. While users without NoPerformOnServer can pass commands such as 'pwd' and 'ls' to performOnServer:, and rely on the path lookup, users with NoPerformOnServer must configure the whitelist to include the full path to the specific executable that is allowed, such as '/bin/pwd' and '/bin/ls', and pass the same full path to performOnServer:.
Example
For example, as DataCurator or another privileged user, update a non-privileged user ReadingUser by executing:
(AllUsers userWithId: 'ReadingUser') addPrivilege: 'NoPerformOnServer'; addPerformOnServerCommand: '/usr/bin/git'.
System commitTransaction.
Now, as ReadingUser, the following will succeed:
System performOnServer: '/usr/bin/git log'
But ReadingUser cannot perform other OS commands, such as ls or cp. Each individual executable to be allowed must be specifically entered in the whitelist.
2.3 Changes in Backup, Restore and Hot Standby
Tranlog version numbers following restore
The way tranlog version numbers are handled during commitRestore has changed in this release.
Sending commitRestore following restoring a fullbackup and transaction logs normally creates a tranlog with the number that is one higher than the last transaction index in the backup/restored tranlogs (this number is reported by copydbf -i on backups and transaction logs).
In previous releases, if a tranlog with this name already existed, it would instead choose the number equivalent to the next unused tranlog filename. Starting in v3.6, it will now always write to a filename with a number one higher than the restored repository; it is an error if a tranlog with that number exists, and the commitRestore will fail.
Generally, you should use care in managing transaction logs locations to avoid confusion in the sequence, or risking fork-in-time scenarios.
Hot Standby Failover Changes
In a hot standby system, there is a master Stone that continuously updates a slave Stone. On a planned failover, the master Stone must complete the update of the slave Stone; and then, usually, the slave Stone is made into the master, and the former master becomes the slave.
This process has been improved in v3.6.
Rather than using System class >> suspendCommitsForFailover, a new method, System class >> failOverToSlave, has been added.
The old method suspendCommitsForFailover prepared for a failover to the slave system, but did not prepare the master to become the new slave. While the process to failover to the slave itself is not different, the additional steps to setup the old-master to act as the new-slave are much simplified.
The new method failOverToSlave, when executed on the master Stone, performs the same steps as suspendCommitsForFailover, and in addition puts the master stone into restore from logs mode. In this mode, the master slave will not write any more tranlog entries, and is ready to be the new slave Stone.
When failOverToSlave is used on the master system, then once the previous-slave Stone executes commitRestore, the old slave and old master are effectively reversed. The logreceiver and logsender are started on the new slave and new master, respectively; the old master/new slave executes continuousRestoreFromArchiveLogs:. The old slave system is now the master, and the old master system is now the slave.
Added methods
Repository >> failOverStatus
To use on a master system to determine the restoreStatus of a slave system. A logreceiver periodically transmits restoreStatus to the logsender, and the logsender stores that status into the primary shared cache. The failOverStatus methods fetch that restore status from the primary cache and report on it, thus allowing a session on a primary stone to observe the restore status of the slave system.
topaz 1> exec SystemRepository failOverStatus %
This repository last checkpoint file 2 record 4155
Other repository restored to checkpoint file 2 record 4155
Repository >> failOverToSlave
For use on the master system of a hotstandby pair. Suspends commits, starts a checkpoint, and waits for completion of the checkpoint. Then puts this repository into restore from logs mode.
New Planned Failover Process in Detail
Step 1. Confirm that all tranlog records have been transmitted to the slave system, and that the slave system has restored all transaction log records.
On the master system, before failing over, execute
SystemRepository failOverStatus
and verify that the master and slave are on the same log and record.
Step 2. On the master system, execute:
SystemRepository failOverToSlave
which will suspend commits, and perform a checkpoint, and enter restore from logs mode. Commits are disallowed. Do not shut down the master Stone.
Executing this method sets up the master system to automatically become a slave after the previous slave, new master has performed the commitRestore, and the logsender and logreceiver are set up on the different nodes.
Step 3. Stop the logsender process on the master system using the stoplogsender utility command.
stoplogsender -P 57222
Step 4. On the slave system, stop the logreceiver process using the stoplogreceiver utility command.
stoplogreceiver -P 57222
Step 5. On the slave system, send the message Repository >> stopContinuousRestore to exit continuous restore mode, then Repository >> commitRestore to terminate the restore process and enable logins. The slave system is now ready for use as the new master.
Step 6. If you have changes in the configuration parameters for the slave system, you should reset these to the values that you want for the master, in the configuration file or files used by the former-slave system. Depending on the specific parameters, you may need to restart the former-slave system for these changes to take effect.
You will also need to enable any automated processes that you normally run on the master, such as monitoring, on the former-slave system.
Step 7. Client applications will have to reconnect to the former-slave system, which now becomes the primary (master) system.
Note that, when using failOverToSlave in v3.6, it is no longer necessary to make a backup of the new master and restore it into the new slave. All that is needed to complete the setup of the old master as the new slave, is to setup the logsender/logreceiver and put the new slave into continuous restore mode.
Step 8. Start the logsender process on the former slave, new master system using the startlogsender utility.
startlogsender -P 57222 -A masterListeningAddress -s masterStone
Step 9. Start the logreceiver process on the former master, new slave system using the startlogreceiver utility.
startlogreceiver -P 57222 -A masterNode -T tranlogDirectories
-s slaveStone
Step 10. Put the former master, now slave Stone into continuous restore mode. To do this, log in and execute Repository >> continuousRestoreFromArchiveLogs:, passing in the list of directories.
After entering continuous restore mode, this method will exit and you can log out. The hot standby is now operational in the reverse direction.
2.4 Other Changes related to administrative operations
GsObjectInventory changes
Improved thread handling
GsObjectInventory profiling methods (other than fastProfileObjects*) previously used two threads to perform the scan. Now, you can control the number of threads using the standard way to control threads in a repository scan; that is, by executing
SystemRepository setDefaultNumThreads: numThreads
before executing the profiling method.
In addition, the following method has been added:
profileObjectsIn: aCollectionOrGsBitmap threads: numThreads
Same as profileObjectsIn: except that it allows specifying the number of threads to use for the scan.
Removed methods profileObjectsInBm: and profileObjectsInFile:
The methods GsObjectInventory >> profileObjectsInBm: and GsObjectInventory >> profileObjectsInFile: were unnecessarily added in v3.5.3, and have been removed.
GsObjectInventory >> profileObjectsIn:* now accepts a GsBitmap directly. To analyze objects in a file, read the objects from the file into a GsBitmap first.
Deprecated findAllReferencePaths feature removed
The methods Repository >> findAllReferencePathsToObjects*, were intended to find multiple reference paths to objects within a single scan. These were deprecated and unsupported as of v3.4, since they failed to scale well in practice, and have been removed in this release.
System currentSession* includes logsender, logreceiver
The report output by System class >> currentSessionsReport, System class >> currentSessionNames, and related methods, now includes a logsender or logreceiver process that is attached (that is, a logsender or logreceiver that was started using the -s key).
2.5 Changes in Errors
AlmostOutOfMemoryError
In addition to the class AlmostOutOfMemory/6013, a new exception class, AlmostOutOfMemoryError/6022, has been added.
- AlmostOutOfMemory is a Notification that is enabled by default, although you must enable signalling or set a handler block before any action is taken.
- AlmostOutOfMemoryError is an Error that is disabled by default, but once enabled does not require additional actions.
Note that the threshold setting is shared between these two classes.
AlmostOutOfStackError
In addition to the class AlmostOutOfStack/2059, a new exception class, AlmostOutOfStackError/2519, has been added.
ExitClientError
The class ExitClientError/3004 has been added.
ExitClientError allows you to exit the client from within GemStone Smalltalk code. See Exiting the Client from Smalltalk for more details.
UncontinuableError
The Exception class UncontinuableError ERR_EXC_RETURN_DISALLOWED/2758 has been added. This is used when return disallowed, a return from on:do: would cross frame of C primitive, user action, or FFI call.
2.6 Changes in Utilities
copydbf
copydbf has been rewritten in this release, separating the codebase from other utilities. In addition to the new features added to support encrypted extents, the semantics of copy has been made more intuitive, and there are a number of other changes.
The output printed by a copydbf command has changed significantly. The output is written to stderr rather than stdout, includes additional information, and some of the information has been rearranged within the output.
Secure extent support options added
The following options have been added to copydbf to support working with secure extents. See the section Encrypted Extents and Transaction Logs for details.
-D privKey
Use privKey to decrypt the source extent, backup or tranlog.
-e cert
Encrypt the destination extent or tranlog with the given public key or cert.
-F
Use fsync(2) to flush all data to the destination disk before copydbf exits.
-j passphrase
Use the given passphrase text to read the private key specified by -D.
-J pfFile
Use passphrase contained in file pfFile to read the private key specified by -D.
-O
Some operations on secure extents require a clean shutdown. Specifying this option overrides that requirement.
-s keyLength
Specifies the encryption key length in bits. keyLength may be 128, meaning AES-XTR-128, or 256, for AES-XTR-256.
- copydbf does not support creating files that are both encrypted and compressed. Compressing an encrypted file has little benefit since the encrypted bytes appear random.
- copydbf on encrypted files, or operations to encrypt or decrypt dbf files, cannot be combined with changes to the byte ordering. -l, -m, and -p commands are disallowed in these cases. All encrypted files are written in the byte order of the machine that the command is executing on.
- copydbf -i does not require encryption keys to provide the basic information about an encrypted extent. If you pass in the private key and keyring, the it will verify that the given key can be used to decrypt the extent.
Previous, non-working -s has new meaning
The -s argument to copydbf previously preallocated size for the new copy of the database file. This is not useful with modern systems. The -s has been repurposed to specify the encryption key size of a secure backup.
copydbf appends zip/secure extensions as needed
When copydbf is used to copy a file to a target filename, the correct extension (.gz, .lz4, .dbf, or .sdbf) is appended to the target filename, if needed according to the format of the destination file. It is recommended to omit the extension from the target filename.
Changes in file naming with copydbf to directory target
When the copydbf copy destination is an existing directory rather than a filename, and the -f newbasename argument is not specified, then copydbf composes the name of the target file.
In earlier versions, the new file was named tranlog, backup, or extent, followed by the file’s id, ending with .dbf. Now, the base name of the original file is preserved, with the appropriate extension added if needed.
copydbf semantics change for zip status
In previous releases, a copydbf operation wrote the destination file using the compression specified in the copydbf command, with the understanding that if no compression flag was included, the target would not be compressed. The compression state of the originating file was not considered; all state in the destination was as specified in the command line.
With v3.6, this has changed, and the compression flags are considered as transformations to apply to the originating file. In 3.6:
Required to use -u to copydbf a compressed file into raw partition
You may not put a compressed file onto a raw partition. With the new semantics, in order to copydbf a compressed file onto a raw partition, you must include the -u flag to explicitly decompress during copy.
copydbf output to stderr vs. stdout
copydbf status output (details about the copied file) now goes to stderr, rather than stdout.
This allows you to pipe the copydbf copy operation output to stdout. Support for this is added either by specifying stdout as the copydbf target name, or by using the added -a option. That is, either of these operations:
unix> copydbf extent0.dbf stdout > thecopy1.dbf
unix> copydbf extent0.dbf -a > thecopy2.dbf
copydbf no longer accepts NRS for remote access, -n removed
The arguments to copydbf should be filenames, not NRS, and copydbf of remote files requires that the file system be mounted. The -n option to specify a NetLDI has been removed from copydbf.
netldidebug utility added
A new utility has been added, which allows you to modify the debug state of a running NetLDI (that is, the state resulting from omitting or including -d with startnetldi), without the need to shut down and restart the NetLDI.
netldidebug netldiNameOrPort enabledebug | extradebug | disabledebug
netldiNameOrPort -- the name or port of a running NetLDI
enabledebug -- enable debugging (equivalent to startnetldi -d)
extradebug -- enable extra debugging, for argument expansion
printlogs and searchlogs
The printlogs and searchlogs utilities have been updated to work with encrypted tranlogs; you must include the private key, the passphrase if needed, and the keyring dir, using the following keyword arguments:
keyring privKeyPath privkey keyFileName privkeypf keyPassphrase privkeypffn fileContainingPassphrase
Only one of privkeypf and privkeypffn are needed.
unix> printlogs keyring $GEMSTONE/examples/openssl/private/ privkey server_2_serverkey.pem
privkeypffn $GEMSTONE/examples/openssl/private/server_2_server_passwd.txt tranlog4.sdbf
startlogsender, startlogreceiver
startlogreceiver -D argument for additional logging
The startlogreceiver utility now accept the -D argument. This provides logging similar to the -d argument, but also prints commands sent between the logsender and logreceiver. The logreceiver’s log file will be much larger.
startlogsender -u option removed
startlogsender no longer accepts the -u option; logs records are always compressed for transmission.
startlogsender option to specify tranlog prefix
The -g tranlogFilePrefix argument has been added to startlogsender. This is only needed if not using the -s option to specify the stone name, and if the master stone’s transaction logs are not named at the default, 'tranlog' (that is, a non-default STN_TRAN_LOG_PREFIX). If using the -s argument, the argument to -g is ignored, and the Stone’s setting is used.
startnetldi
Added option to configure threads
The NetLDI is now multithreaded, with a default of 10 threads, or 1 for a X509-Secured NetLDI or a NetLDI running as root.
To control the number of threads, the -t option has been added to startnetldi.
-t numberOfThreads
The number of threads that netldi will use, default 10, min 1, max 500. If -S specified, or netldi running as root, only uses one thread.
Resolving hostnames using reverse DNS lookup
Previously, the NetLDI would by default lookup IP address to fetch the host names of peers, using reverse DNS lookup; this could be disabled using the -N argument.
These names are rarely printed, and configuration issues in the reverse lookup could cause timeouts. Now, by default, there is no lookup of IP address. The -N argument remains but has no effect.
To enable printing of hosts by name, the new option -H has been added.
-H Do reverse DNS lookup and print peer hostnames in addition to IP address in logs.
The IP addresses are now printed in the NetLDI log when the NetLDI is in debug mode.
startstone
Resolving hostnames using reverse DNS lookup
Previously, the Stone would by default lookup IP address to fetch the host names of peers, using reverse DNS lookup. Since this is subject to issues if DNS is not configured correctly, this has been changed, and by default, there is no lookup of IP address.
To enable printing of hosts by name, the new option -H has been added.
-H Do reverse DNS lookup and print peer hostnames in addition to IP address in logs.
startstone and pageaudit secure database options added
The following have been added to startstone and pageaudit:
-D privateKeyFile
Specifies the private key file used to access the encrypted database. The file must be in PEM format. If the private key has a passphrase, then either -j or -J are also required. The file name must not include a path. Use -K to specify the path to the private key file.
-j passphrase
Specifies the passphrase for the private key specified by -D.
-J pffile
Specifies a file containing the passphrase for the private key specified by -D.
-K dir
Specifies one or more keyring directories. Directories may be separated by colons and/or -K may be specified multiple times.
See the section Encrypted Extents and Transaction Logs for details.
updatesecuredbf utility added
The updatesecuredbf utility allows you to modify the encryption key for an encrypted extent file, transaction log, or backup file. updatesecuredbf can only be used with already encrypted dbf files; use copydbf to encrypt or decrypt a dbf file.
For more information, see Modifying Encrypted Files, Examples of modifying encrypted files, and the System Administration Guide chapter 12 and Appendix B.
2.7 Topaz changes
Prompt change when input nested files
The input command allows you to input a series of topaz commands from an file. These files may themselves invoke input on other files.
To make it easier to see the level of file that is being input, the topaz prompt now includes a + (plus sign) for each level of file input nesting, up to 20 (the limit for nesting input files).
topaz 1> input aFile
topaz 1 +> <commands in aFile>
topaz 1 +> input nestedFile
topaz 1 ++> <commands in nestedFile>
topaz 1 +> <further commands in aFile>
Added commands
TFILE
This command allows you to input a tonel format class definition or class extension file. The methods within the file are all compiled. The Class specified in the file must already exist (i.e. class definition execution is not attempted).
DISPLAY/OMIT ALLOOPS
The command display/omit classoops was not correctly named, since it also enabled display of the oops of specials; it has been replaced by the new subcommands display/omit alloops. The command under the legacy name is still available.
DISPLAY/OMIT DECIMALBYTES
display decimalbytes is similar to display bytes, but displays the bytes of the contents as decimal rather than hexadecimal. Executing omit bytes or omit decimalbytes disables printing of any numeric byte contents.
SET SINGLECOLUMN ON/OFF
When stacks are printed, methods that have more than eight method arguments and temporary variables print their variables in four columns. With this new command, you can disable this feature, and display each variable on a single line.
set singlecolumn is off by default, providing the behavior in previous releases.
Changes in commands
STRINGS / STRINGSIC searches class comments
The strings and stringsic commands now include string search matches in class comments, as well as in class and instance method source code.
ALLSTACKS limit removed
The allstacks command previously was limited to 1000 frames per process. This limit has been removed.
QUIT / EXIT with file input
The quit or exit command is now ignored when reading a file using input, when stdin is a tty.
If topaz stdin is redirected to a file and the file does not end with a quit or exit command, topaz considers EOF on stdin to be an error, that contributes to non-zero exit status of topaz.
2.8 Configuration Parameter Changes
Added Configuration Parameters
GemDebuggerActive
GemDebuggerActive is a runtime-only configuration option for the Gem, enabled by:
System gemConfigurationAt:#GemDebuggerActive put: true
This controls debugging of exceptions signaled from within user actions.
When true, exceptions for which no handler is found are delivered with a complete stack, including the frames of the user action. When true, an AbstractException >> return disallowed from within a user action, will be signaled as not trappable so the debugger in the GCI client can debug a complete stack.
For more details, see Handling Errors in user actions.
StnReverseDns
Runtime-only configuration parameter for the Stone. If false, the IP addresses of other nodes are reported in the Stone log as IP addresses only. If true, the Stone uses reverse DNS lookup to get the node’s name, and reports both IP address and node name.
Default: true if Stone started with -H, false otherwise
Changed Configuration Parameter
STN_NUM_LOCAL_AIO_SERVERS
The configuration parameter STN_NUM_LOCAL_AIO_SERVERS now configures the number of AIO threads in the Stone, rather than the AIO page servers; the name has been retained for simplicity.
Historically, this was limited to 1 per extent. With modern hardware, this restriction is no longer valid. The default has been changed to -1, which allows the system to calculate the number of AIO page server threads.
- If the repository size is > 100GB, and the host has at least 20 cores, the number of threads is the larger of 5 or number of extents
- if the repository size is > 10GB and host has at least 8 cores, the number of threads is the larger of 2 or number of extents
- otherwise, 1 per extent up to a maximum of 4
2.9 Changes in Statmonitor and Cache Statistics
Statmonitor command line details
With 3.6, statmonitor writes an additional line to the statistics data file. This is displayed in VSD, in the Main Window > File > Info dialog.
This feature required changes to VSD as well as GemStone; the changes to support this additional line in the data file were made in the previously released VSD v5.5.
VSD 5.5 or later is required to read statmonitor data files written by GemStone/S v3.6.
Default process cache names
There are some changes in the default names for Gems in v3.6.
- Previously, Topaz Linked and RPC sessions both displayed with the default name 'Topaz'. Now, Gems from RPC logins from Topaz, by default, are named 'TopazR', and Gems from Linked topaz logins are now named 'TopazL'. These names can be changed using -u or set cachename.
- Other applications, such as GBS, previously wrote the name as 'GemNN', where NN was the cache slot index. Now, the default name is 'Gem'.
Topaz linked sessions distinguished in statmonitor data
When a topaz linked process logged out, and logged in again, it was previously treated as a single continuous session. However, this produced incorrect or misleading statistics, since values that were reset over logout and login were treated as continuous. (#48650)
Now, each topaz linked login through subsequent logout will be recorded as a separate session in statmonitor data.
FailedCommitCount includes commits that succeeded on retry
If a commit fails, but is retried and succeeds, previously this was counted as a failed commit and included in the cache statistic FailedCommitCount, although from the point of view of the end-user this was a successful commit. Now, commits that succeed on retry are not included in FailedCommitCount (#48387)
New related cache statistics include:
FailedAttemptedCommitCount (Gem)
The number of attempts to commit or continue a transaction which failed.
LastFailedCommitReasonCode (Gem)
A reason code identifying the cause of the most recent failed commit:
1 = Reduce Conflict (RC) Failure
Changed Cache Statistics
PagesWaitingForRemovalInStoneCount has been removed.
It is replaced by the following, which track the Page Manager thread’s work queue:
PagesWaitingForPagemgr (Stone)
The number of pages in the Stone in the Page Manager's work list.
PagesWaitingForRemoval (Stone)
The number of pages in the Stone that are waiting to be removed from the Shared Page Cache by the Page Manager.
CodeCacheSizeBytes has been renamed to CodeCacheSizeKBytes, and is now in units of KB.