12. Encrypted Extents and Transaction Logs
This chapter describes using encrypted extent files and tranlogs, how to manage these files, and some considerations when using encrypted extents and transaction logs with backup and restore.
12.1 Overview
GemStone provides user and object level security to protect your application data from unauthorized access via applications or Smalltalk code execution. These objects also have persistent state on disk, which, depending on the specifics of your repository, may be vulnerable. The repository extents and backup files contain all repository objects, and the tranlogs record recent changes. The objects on disk are stored for efficiency of access and the contents is likely to include readable data. These disk files are normally protected by disk permissions from unauthorized access.
For applications with high security requirements, the default protections may not be sufficient. The extents, transaction logs and backup files can be configured to be written in encrypted form.
Encryption of extents, transaction logs, and backups (as well as signing of backup files), is done using OpenSSL libraries. OpenSSL is a robust, widely used toolkit for the Transport Layer Security (TLS) and Secure Sockets Layer (SSL) protocols, and contains support for general-purpose cryptography. The GemStone distribution includes OpenSSL executables and libraries. For many OpenSSL functions, you may use your own OpenSSL installation if you prefer.
A full explanation of SSL key algorithms and the choice of which to use, and the details of generating keys, is outside the scope of this document, and SSL and security algorithms are under active development. You should review your application security requirements with the latest security recommendations and ensure that your security is sufficiently strong. Using encryption or signing with GemStone example keys from the distribution, for example, does not provide any security. You are responsible for integrating your own security procedures with the tools that GemStone provides, and verifying that your system has the appropriate security precautions for your requirements.
|
For applications that are not limited to internal, secure networks, security is a critical. The usage in these examples is NOT sufficient to provide complete security. |
When using encrypted extents, transaction logs backups you must put processes in place to avoid the dbf files being so secure that you yourself cannot read them.
- Ensure that you encrypt all files that are part of an extent, backup, or tranlog sequence with the same key.
- Retain older keys if you may need to use an older backups or tranlogs; or, update the keys of archived backups and tranlogs before discarding a key.
- Prior to encryption any dbf file, make a copy and verify that the encrypted dbfs are usable, before deleting the unencrypted version.
12.2 Encrypted Extents
With encrypted extent files, the on-disk extents are encrypted at all times. Transaction logs written by a Stone using encrypted extents are always written in encrypted form, using the same keypair as the Stone.
To use encrypted extents, you must manually encrypt the repository extent files prior to starting the Stone, using a public key or X.509 certificate. The startstone command takes additional arguments, specifying the private key. If the repository has more than one extent, all extents must be encrypted using the same key. Only one keypair can be associated with encrypted extents.
The encryption step is done using a public key; to start the stone on these extents, the private key corresponding to the public key (and the passphrase for the private key, if configured) are required. Encryption is done using AES-XTS symmetric encryption with a key length of 128 or 256 bits.
In an encrypted extent, all data in the extent is encrypted, except for a small number of records with metadata information. This unencrypted information also allows copydbf -i to report information without needing the key.
Once the Stone has started, the private key is not needed again until the next time the Stone is restarted. Pages from the extents are decrypted before reading into the cache, and encrypted on write back to disk. Since the Stone may need restart at any time if an unexpected outage occurs, the private key must be keep readily available.
Internal handling for other sessions
The Stone uses the private key argument provided by startstone to decrypt the session key stored in each of the extents. Each extent has its own unique session key. This is stored in private stone memory, and so is not accessible to other sessions on the shared page cache. Processes that read or write to the extents need access to this key.
For gems, which may be on remote hosts over unsecured networks, a random key is generated and used to encrypt the keys, which are provided in shared memory for local gems, or over an encrypted SSL socket for remote gems.
The first gem to attach to a remote shared page cache stores the keys in the encrypted extent key table in the remote cache, under spin lock control.
Transaction Logs
Transaction logs written by a Stone using encrypted extents are written in encrypted form, using the same keypair as the extents. When the Stone is started, if recovery is needed (that is, on unclean shutdown), it will be able to recover from transaction logs since they were encrypted using the same key as was used to start the Stone.
If the encryption key of the extents have changed, on startup after a clean shutdown, the Stone will start a new transaction log that is automatically encrypted with the new keys. In this case, you must take care to keep the older keypairs, or update the encryption keys for the older tranlogs, if you want to be able to restore these older tranlogs into an older backup.
Note that the encryption key/s of a secure backup are entirely distinct from the encryption keys of the transaction logs. For restore, the tranlog encryption key will need to match the encryption key used to start the stone (or the Stone may be provided with the list of older transaction log keys). Once the secure backup is restored, the encryption key that was used to encrypt the backup is no longer relevant.
Restart and Recovery
If the stone unexpectedly shuts down, the private key is needed to restart the stone. Automatic recovery from transaction logs requires that all required transaction logs for recovery be encrypted using the same key as the stone. If necessary, the transaction logs may have their keys updated using updatesecuredbf.
Programmatic Backups
Making a programmatic backup of a repository running with encrypted extents does not automatically create an encrypted backup.
Extent Copy Backup
An extent copy backup of a repository with encrypted extents is, of course, encrypted.
Ensure that the private key remains available in order to decrypt or restore. If the certs and keys need to be updated, you can use updatesecuredbf on the extent copy backups, but you will need to use the -O option, since the extents were not cleanly shutdown.
Example Setting up Encrypted Extents
The following example shows the steps to encrypt a two-extent repository with a key from the GemStone distribution. Note that this does not provide security; you must generate your own private key meeting your own security requirements.
Step 1. Stop the Stone cleanly. While it is possible to encrypt an extent that is not cleanly shutdown, you would also need to encrypt the tranlogs needed for recovery.
unix> stopstone stoneName administrativeUser password
Step 2. Make encrypted copies of your extents using copydbf.
unix> copydbf $GEMSTONE/data/extent0.dbf
$GEMSTONE/data/secextent0.sdbf -e server_1_servercert.pem
-s 256 -K $GEMSTONE/examples/openssl/certs
unix> copydbf $GEMSTONE/data/extent1.dbf
$GEMSTONE/data/secextent1.sdbf -e server_1_servercert.pem
-s 256 -K $GEMSTONE/examples/openssl/certs
Step 3. Edit the configuration file used by the Stone to specify the new extent names.
DBF_EXTENT_NAMES = $GEMSTONE/data/secextent0.sdbf,
$GEMSTONE/data/secextent1.sdbf;
Step 4. Execute startstone using the private key.
unix> startstone -D server_1_serverkey.pem
-J $GEMSTONE/allcerts/server_1_server_passwd.txt
-K $GEMSTONE/examples/openssl/certs -K $GEMSTONE/examples/openssl/private gs64stone
On startup, the Stone will immediately start a new transaction log that is encrypted with the key server_1_servercert.pem.
Using the stone, and stopping the stone, now proceed as with any GemStone installation.
12.3 Encrypted Extents with Backup and Restore
How to make and restore the various kinds of backups, including encrypted backups, is described in the System Administration Guide, Chapter 11. Making a backup in a repository configured with secure extents is no different than making a backup with ordinary extents, and restoring a backup is much the same.
However, if the encryption key for the Stone changes during the course of operation, some transaction logs may be written with different encryption keys. You must either update the transaction logs’s encryption, or provide the older keys to the Stone, before these transaction logs can be replayed.
Restoring Backups
When restoring a programmatic fullbackup to a system that is using encrypted extents, you will need to start with a clean, encrypted extent (in order to continue using encrypted extents). This requires an additional step: use copydbf to encrypt a clean, empty extent.
Alternately, you can restore into an unencrypted extent, and then shut down the restored stone, encrypt the extent files, and restart.
All startstone invocations will require the appropriate key for the encrypted extents of the Stone you are restoring, regardless of any encryption in the fullbackup.
The restore process is similar to the processes described in Chapter 11, under:
Restoring transaction logs
No changes in encryption keys
When using encrypted extents, the transaction logs are also encrypted with the same keypair as the Stone. As long as all required transaction logs for restore are encrypted with the same keypair as the newly restored Stone (or not encrypted), then restoring transaction logs into a Stone with encrypted extents follows the same process as restoring into a Stone with regular (non-encrypted) extents.
Changes in encryption keys
When using encrypted extents, over time you may change the keypair used to encrypt the extents (using updatesecuredbf or copydbf twice to decrypt and re-encrypt). When the stone is restarted with the new key, it will create a new transaction log that is encrypted using the new keypair. Since the full series of transaction logs will be encrypted with multiple keys, special handling is required to allow all the transaction logs to be restored.
- Use updatesecuredbf to update the encryption key for every transaction log so it is using the current Stone’s encryption key. Any tranlogs that do not have the Stone’s encryption key cannot be read, and restore will stop there.
- Use the method Repository >> setTranlogPrivateKeysForRestore: to provide the stone with the private key for all of the transaction logs that you will be restoring. Any transaction logs using keys that are not provided to the stone cannot be read, and restore will stop there.
- Always make backups immediately after changing the encryption key, and set up your systems for backup and restore, such that you will never need to restore older encrypted repositories. For example, you may decrypt the older backups and transaction logs and archive these as insecure files using your own secure storage mechanisms.
Update transaction log keys
The following examples updates a tranlog that was previously encrypted with server_1_servercert.pem to the new key, server_3_servercert.pem. Both keys must be provided so the tranlog can be read using the old key, and re-encrypted with the new key.
unix> updatesecuredbf tranlog1.sdbf -e server_3_servercert.pem
-D server_1_serverkey.pem
-J $GEMSTONE/examples/openssl/certs/server_1_server_passwd.txt
-K $GEMSTONE/examples/openssl/certs
-K $GEMSTONE/examples/openssl/private
Providing private keys to the Stone in Smalltalk code
Rather than updating all the keys for existing transaction logs, you may pass into the stone all the private keys for all the transaction logs that you will be restoring, using the method Repository >> setTranlogPrivateKeysForRestore: and creating instance of GsTlsPrivateKey.
Repository >> setTranlogPrivateKeysForRestore: arrayOfGsTlsPrivateKey
Specifies a list of instances of GsTlsPrivateKey representing private keys, that will be used to read encrypted tranlogs during a restoreFromArchiveLogs operation. The private key used to start the stone does not need to be included.
During a restoreFromArchiveLogs operation, the stone searches arrayOfGsTlsPrivateKey for a match to the public key stored in the encrypted tranlog. An error is raised if no match is found, and the restore operation fails.
GsTlsPrivateKey class >> newFromPemFile: pemFilename
Create an instance of GsTlsPrivateKey for the private key in the given filename. This key must not require a passphrase.
GsTlsPrivateKey class >> newFromPemFile: pemFilename withPassphraseFile: passPhraseFilename
Create an instance of GsTlsPrivateKey for the private key in the given filename. The key in this file is assumed to require a passphrase, which is in the given passphrase file.
There are other instance creation methods on GsTlsPrivateKey; refer to the image for other options.
Once GsTlsPrivateKeys are provided to the Stone, this list of keys is saved in the Stone, so you may execute this expression multiple times with new arguments, without losing key information.
Example 12.1 Setting tranlog keys for restore
topaz 1> run
SystemRepository setTranlogPrivateKeysForRestore: {
(GsTlsPrivateKey newFromPemFile:
'$GEMSTONE/examples/openssl/private/server_1_serverkey.pem'
withPassphraseFile:
'$GEMSTONE/examples/openssl/private/server_1_server_passwd.txt')
.
(GsTlsPrivateKey newFromPemFile:
'$GEMSTONE/examples/openssl/private/server_2_serverkey.pem'
withPassphraseFile:
'$GEMSTONE/examples/openssl/private/server_2_server_passwd.txt')
}
%
Page Audit
To run pageaudit on a system with encrypted extents, you must include the same private key information as when starting a stone.
For example, when the extent files for a repository are encrypted using the key server_2_servercert.pem, then the following invocation will allow pageaudit to execute:
unix> pageaudit -D server_2_serverkey.pem
-J $GEMSTONE/examples/openssl/private/server_2_server_passwd.txt
-K $GEMSTONE/examples/openssl/private
Hot Standby
You may use encrypted extents within a warm or hot standby system. The logsender and logreceiver do not require any special configuration; they pass along the records without decrypting them.
The slave stone must be started with the same encryption key as the master stone, so that the process restoring the tranlogs is able to decrypt them. If the encryption key on the master is changed, you should also update the encryption key on the slave’s extents.
Alternatively, if the encryption key on the master system has changed, you can provide the new key to the slave system using setTranlogPrivateKeysForRestore:. However, this is not retained over shutdown/restart of the slave system, since it is in restore mode. After a restart of the slave system you will need to execute setTranlogPrivateKeysForRestore: again.
12.4 Modifying Encrypted Files
The utilities copydbf and updatesecuredbf allow you to perform a number of operations on encrypted extents, transaction logs, and backup files.
- copy the file without changing encryption state
- encrypt an extent or transaction log file (backups cannot be encrypted using copydbf)
- decrypt an extent, transaction log, or backup to a regular file
- change the encryption key of an extent or transaction log, or one of the encryption keys of an encrypted backup, to a different key.
Note that it is possible to create secure backups that are signed, but not encrypted. The signing key for these backups cannot be changed, and these backups cannot be converted into encrypted nor into regular files.
Creating extent or transaction log files that are both encrypted and compressed is not supported. Manually compressing an encrypted dbf file is unlikely to provide much improvement on disk space use, since the encryption makes the bytes appear random and thus not compressible.
Examples
Keys are not needed to make a copy of an encrypted dbf. copydbf on an encrypted file, without using decryption keys, results in a file that is encrypted with the same key as the source file.
To encrypt an extent
This example encrypts the distribution extent into secextent0.sdbf.
unix> copydbf $GEMSTONE/bin/extent0.dbf
$GEMSTONE/data/secextent0.sdbf -e server_2_servercert.pem
-s 256 -K $GEMSTONE/examples/openssl/certs
If you have multiple extents, you must encrypt all extents with the same key, to be able to start a Stone.
The same process can be used to encrypt an extent snapshot backup. Unencrypted programmatic fullbackups cannot be encrypted using copydbf.
You may also encrypt an unencrypted transaction log, although normally encrypted transaction logs are created by a Stone using encrypted extents.
To start a stone on an encrypted extent
When the Stone’s configuration file specified extent files that are encrypted using the key server_2_servercert.pem, then the following invocation will start the stone on those extents.
unix> startstone -D server_2_serverkey.pem
-J $GEMSTONE/examples/openssl/private/server_2_server_passwd.txt
-K $GEMSTONE/examples/openssl/private gs64stone
To decrypt an extent or extent snapshot backup
This example decrypts secextent0.sdbf, which was encrypted with server_2_servercert.pem, into decrypextent0.dbf.
unix> copydbf secextent0.sdbf decrypextent0.dbf
-D server_2_serverkey.pem
-J $GEMSTONE/examples/openssl/private/server_2_server_passwd.txt
-K $GEMSTONE/examples/openssl/certs -K $GEMSTONE/examples/openssl/private
If you have multiple extents, you must decrypt all extents to be able to start a Stone.
To decrypt an extent or extent snapshot backup
This example decrypts sectranlog1.sdbf into decryptranlog1.dbf.
unix> copydbf sectranlog1.sdbf decryptranlog1.dbf
-D server_2_serverkey.pem
-J $GEMSTONE/examples/openssl/private/server_2_server_passwd.txt
-K $GEMSTONE/examples/openssl/certs -K $GEMSTONE/examples/openssl/private
Decrypted transaction logs can be restored without special handling, including in a series with other transaction logs that are encrypted.
To change the key of an encrypted extent or extent snapshot backup
This example updates the original key server_2... to server_3... The modification is done to secextent0.dbf in place; we recommend making a copy before using updatesecuredbf.
The stone using this extent must have been shut down cleanly.
unix> updatesecuredbf secextent0.sdbf
-e server_3_servercert.pem -D server_2_serverkey.pem
-J $GEMSTONE/examples/openssl/private/server_2_server_passwd.txt
-K $GEMSTONE/examples/openssl/certs -K $GEMSTONE/examples/openssl/private
If you have multiple extents, you must update the keys for all extents to the same key to be able to start a Stone.
To change the key of an encrypted transaction log
This example updates the original key server_2... to server_3... The modification is done to sectranlog1.dbf in place; we recommend making a copy before using updatesecuredbf.
unix> updatesecuredbf sectranlog1.sdbf
-e server_3_servercert.pem -D server_2_serverkey.pem
-J $GEMSTONE/examples/openssl/private/server_2_server_passwd.txt
-K $GEMSTONE/examples/openssl/certs -K $GEMSTONE/examples/openssl/private
Transaction logs are normally part of a series that are retained to allow restore. Before a transaction log can be restored, it must either be using the same key as the extents for the Stone into which it is being restored, or the keys must be supplied using setTranlogPrivateKeysForRestore:.
To change the encryption key of an encrypted fullbackup
Encrypted programmatic fullBackups are always signed; you will need to supply the signing key (and passphrase, if required), as well as the previous and new encryption keys.
Encrypted backups may be encrypted with up to 8 different public keys, and the private key corresponding to any of these public keys can be used to decrypt. An invocation updatesecuredbf changes a single one of these public key encryptions, leaving the others unchanged.
unix> updatesecuredbf encryptedBackup1.sdbf
-e backup_encrypt_2_clientcert.pem
-J $GEMSTONE/examples/openssl/private/backup_encrypt_1_clientcert_passwd.txt -D backup_encrypt_1_clientcert.pem
-S backup_sign_3_clientkey.pem
-T $GEMSTONE/allcerts/backup_sign_3_client_passwd.txt
-K $GEMSTONE/examples/openssl/certs
-K $GEMSTONE/examples/openssl/private
For a multi-file fullbackup, you must update they keys for all backup files to allow them to be used for restore.
To decrypt a fullbackup
This example decrypts encryptedBackup1.sdbf into Backup.dbf, using the private encryption key backup_encrypt_2_clientkey.pem.
unix> copydbf encryptedBackup1.sdbf Backup.dbf
-D backup_encrypt_2_clientkey.pem
-J $GEMSTONE/examples/openssl/private/backup_encrypt_2_client_passwd.txt -K $GEMSTONE/examples/openssl/certs -K $GEMSTONE/examples/openssl/private
For a multi-file fullbackup, you should decrypt all backup files to allow them to be used for restore.