3. Changes in GemStone Smalltalk
This chapter describes changes and new features important for programmers using GemStone Smalltalk, including:
3.1 Changes in Classes, compilation, and code management
Code Management
This version includes preliminary work on the packaging and management of GemStone source code, including methods that support the tonel code format and the STON serialization format. This code is primarily within the CodeLibrarian’s UserGlobals, but some support methods have been added to the image.
These new methods appear in the image with categories and/or method selectors that reference "tonel", "ston", or "metacello". All such methods are subject to change in future versions and should be used with caution.
Added private methods for Tonel support
Methods are in the category *tonel-gemstonecommon-core-private include some methods integrated from a Pharo-compatible code base, which have functionality that duplicates existing GemStone methods. These private methods may be removed in a future release without prior deprecation.
Compiling methods with reserved selectors
It is now allowed for sessions with CompilePrimitives privilege (as well as CodeModification) to compile methods with reserved selectors (see the Programming Guide for the list of reserved selectors). These selectors are still optimized and new definitions do not affect compiled code, but can be accessed, for example, via perform:.
Added Behavior compile method
The following method has been added:
Behavior >> compileMethod: sourceString
Similar to compileMethod:dictionaries:category:, it compiles the given sourceString into the receiver class, using the current session’s symbol list and into the category '(as yet unclassified)'.
Added class creation options
#selfCanBeSpecial
The subclass creation methods options: keyword may now include the option #selfCanBeSpecial. This is used in environments other than 0 with a modified class hierarchy, for classes in which self can be a special object, to avoid incorrect code being generated.
This option is not inherited from a superclass, but must be present in a superclass to be able to specify it for a subclass.
Class byte size options
The following options: arguments are now accepted for Byte-format classes:
#'2byteWords' #'4byteWords' #'8byteWords' #'signed2byteWords' #'signed4byteWords' #'signed8byteWords'
These options allows specification of the word size of a byte format class, and allows underlying swizzling support for multi-byte word subclasses (such as DoubleByteString and QuadByteString).
Multi-byte byte-format classes
Creating custom multi-byte classes that support automatic byte swizzling, and therefor are portable between big and little endian machines, can now be done by subclassing server classes.
The following classes have been added:
Uint16Array, Uint32Array, and Uint64Array
Int64Array, Int32Array, and Int16Array
These classes are defined using the new byte subclass creation options such as #'2byteWords', etc., and provide the additional methods that are needed to support byte swizzling. You may subclass these classes to create custom portable byte-format classes.
Changes in handling some class options
The class options #dbTransient #instancesNonPersistent and #instancesInvariant are mutually exclusive, and redefining a class when there are only changes in adding these options updates the existing class and does not create a new class version.
Previously, redefining a class with a different one of these options would error as the new option was added while the old one remained. Now, redefining with a different one of these three options will remove any of the other options so the operation succeeds.
Changes in handling Dynamic Instance Variables
Dynamic instance variables are per-instance instance variables, added using statements such as anObject dynamicInstVarAt: key put: value.
GBS 8.4 Inspector display
In GemBuilder for Smalltalk v8.4, dynamic instance variables are now visible in inspectors.
Handling during migration
Previously, dynamic instance variables were ignored during migration. Now, dynamic instance variables are added to the new instance during migration. As with any migration customization, override the method migrateFrom:instVarMap: in the new class version, to control instance variable mapping.
Added methods
Object >> copyDynamicInstVarsFrom: anObject
Copy all dynamic instVar name/value pairs from anObject to the receiver.
Object >> copyDynamicInstVars: aCollectionOfSymbols from: anObject
Copy the dynamic instance variables with names in aCollectionOfSymbols from anObject to the receiver.
Object >> dynamicInstVarPairs
Returns the dynamic instance variables of the receiver in the form of an Array of name/value pairs.
Class information when there are multiple versions of a class
The following methods have been added:
Class >> versionNumber
Returns the number of the receiver within its classHistory.
Class >> versionedName
Prints the name followed by [versionNumber], for classes that are not the last version in the classHistory.
Class >> classHistoryAt:
Equivalent to classHistory at:, returning the instance of class at the given position within the classHistory.
Source code with control characters
GemStone source code may include literals for characters such as tab and lf, which may cause difficulties with source code management tools. To avoid such problems, GemStone code now accesses a set of literals using Class variables. The source code in $GEMSTONE/upgrade no longer includes these control characters in method sources.
- Character added class variables Backspace Cr Esc Lf NewPage Tab.
- Stream added class variables Lf Tab.
- AppendStream added class variables CrLf CrTab, as well as inherited Lf and Tab.
- WriteStream, including both WriteStreamLegacy and WriteStreamPortable, added class variables Cr CrLf CrTab, as well as inherited Lf and Tab.
Pragmas have been made environment aware
Pragmas were previously stored in a class’s extraDict attribute; they have been moved to be attributes of the GsNMethod itself. This allows Pragmas to be used per environment. Existing uses of Pragmas are unaffected.
The method Behavior >> setPragmas:forMethod:, which installed pragmas, is deprecated.
The method Behavior >> pragmasForMethod:env:, which fetches pragmas, has been added.
Behavior methods for other environments
Code in environments other than the environment with envId 0 can be filed out now using methods in Behavior. The following methods have been added, which add an environmentId argument to similar existing methods:
Behavior >> fileOutCategory:on:environmentId:
Behavior >> fileOutClassDefinitionOn:environmentId:
Behavior >> fileOutClassOn:environmentId:
Behavior >> fileOutMethodRemovalOn:name:environmentId:
Behavior >> fileOutMethodsOn:environmentId:
Behavior >> fileOutPostMethodsOn:environmentId:
Behavior >> fileOutPreMethodsOn:environmentId:
Behavior >> removeAllMethods:
3.2 Transaction conflicts and object locks
Detailed information on transaction conflicts
When two sessions commit changes that conflict with each other, the second session to commit gets a commit failure. Executing the method System class >> transactionConflicts reports the nature of the commit (Write-Write, for example) and the objects that were in conflict. However, it was previously impossible to determine any information about the commit that succeeded in making changes and was the root cause of the commit conflict.
It is now possible to configure your system to allow detailed reporting on the commit with which your commit conflicted.
First, you must enable tracking, by setting the runtime-only configuration parameter GemCommitConflictDetails to true. For example,
System gemConfigurationAt: #GemCommitConflictDetails put: true
This must be done before the commit transaction, since the work to collect the conflict information must be done at commit failure.
Then, once the commit fails, execute the new method System class >> detailedConflictReportString, which returns a string containing information about the conflicting other commit. For example,
System detailedConflictReportString
%
Commit failed , failure
Attempt to commit at: 2019-05-14 15:39:55.959
1 Write-Write Conflicts
( 12200193(a SymbolDictionary))
1 commits by other sessions
session 7 at 2019-05-14 15:39:45.607 userId DataCurator
( 12200193(a SymbolDictionary))
Commit conflict tracking requires extra work to be done in the gem for every failed commit, and an iteration over commit records; so it is only recommended that this be turned on for debugging commit conflicts.
Other added methods on transaction conflicts
The following methods have also been added:
System class >> conflictReportString
Return a String describing the result of System class >> transactionConflicts.
System class >> conflictReportString: conflicts
The conflicts argument is Array returned by System class >> transactionConflicts. The report is limited to the first 100 oops in each category of conflict.
Object locking changes
Added lock report key #deferredUnlock
Reports on the kinds of object locks in the system, such as sessionLocks and systemLocks, now return #deferredUnlock, in addition to #read and #write.
Deferred unlocks are objects for which the unlock request was received by stone while another session was holding the commit token; they will be unlocked as soon as the commit token is released.
New lock status method
The following method has been added:
System class >> systemLocksDetailedReport
Returns a String describing all of the locked objects. This is an expensive method, it makes a call to stone for each element of the result of System currentSessions. For each object, the oop and the class are printed.
session 5(
(1 writeLocks: 11074305(a SymbolDictionary)))
session 6( (1 readLocks: 240641(a Repository))
(2 writeLocks: 979969(a UserProfile) 11074561(a SymbolDictionary)))
3.3 ClassOrganizer
Filtering deprecated methods
The following instance methods have been added to ClassOrganizer:
includeDeprecatedMethodsInReports
includeDeprecatedMethodsInReports: aBoolean
By default, deprecated methods are included. By setting the value to false, deprecated methods are omitted from results that are lists of methods.
topaz 1> run
ClassOrganizer new implementorsOfReport: #concurrencyMode
%
System class >> concurrencyMode
topaz 1> run
ClassOrganizer new includeDeprecatedMethodsInReports: false;
implementorsOfReport: #concurrencyMode
%
(Omitted 1 deprecated methods)
topaz 1>
Include method oops in reports
The following instance methods have been added to ClassOrganizer:
includeMethodOops
includeMethodOops: aBoolean
By default, the OOPs of methods are not included. By setting the value to true, the oops are printed for methods, for results that are lists of methods.
topaz 1>
ClassOrganizer new includeMethodOops: true;
implementorsOfReport: #today
%
Date class >> today 23101185
DateTime class >> today 5877505
topaz 1>
referencesToLiteral: shortcomings
ClassOrganizer >> referencesToLiteral: returns methods that contain references to literals such as literal strings, numbers, and nil, and the offset of the reference within the source. This was failing to return the offsets (#47829), and, for symbols that were lookup references, returning references to symbol association value rather than the symbol itself. (#47102)
3.4 Parsing JSON
GemStone supports object serialization using JSON (JavaScript Object Notation), via methods asJson and printJsonOn:.
In v3.5, the class JsonParser has been added, which is subclassed from PetiteParser classes (PP*Parser). JsonParser supports parsing a JSON string to create an object structure.
Note that asJson writes the object structure and instance variables as dictionaries and Arrays; unlike passivation, the class itself is not preserved. When the object is parsed from the JSON into an object, more processing may be required by the sender to restore the classes of the original objects.
Date today asJson
'{"year":2018,"dayOfYear":304}'
JsonParser new parse: '{"year":2018,"dayOfYear":304}'
aDictionary( 'dayOfYear'->304, 'year'->2018)
3.5 Image level filein/GsFileIn added
The GsFileIn class has been added, to support image-based filein of code in GemStone fileout format. This implementation is not complete and is subject to change in future releases.
To use, send one of the from* methods. This also performs the file-in.
GsFileIn fromServerPath: 'mySourceCode.gs'
Only a subset of topaz commands is supported:
doit, printit , run, nbrun
category:, category
method, method:, classmethod, classmethod:
removeallmethods, removeallclassmethods
commit, abort
input
send
env N
set compile_env N, set class, set class: (other set commands are ignored)
The following are read but ignored:
expectvalue, expecterror, errorcount
iferr, iferr_list, iferr_clear
fileout, output,
fileformat
display, omit
level, limit, remark, status, time
list
3.6 Changes in Numerics
Linux change affecting floating point #inexactResult exceptions
In Linux’s libc v2.24 and later, the inexact exception has been removed from the arithmetic functions floor, ceiling, round and truncate. This impacts GemStone executing on the newly supported Ubuntu 18.04 distribution, which includes libc 2.27. Ubuntu 16.04, Red Hat and SUSE include older versions of libc.
This means that, if floating point exceptions are enabled, GemStone operations such as 9.75 ceiling would signal an #inexactResult exception if running on Ubuntu 16.04 and do not signal any exception on Ubuntu 18.04.
For consistency of results across Linux and other platforms, operations such as floor, ceiling, round and truncate that invoke the OS calls will now ignore any #inexact exception, whether or not the underlying OS call signals inexact.
3.7 Other Changes and Enhancements
Session Signals have additional argument
InterSessionSignal and the deprecated System >> signalFromGemStoneSession now include a fourth element, the number of pending signals.
The following methods have been added:
InterSessionSignal >> fromSession:signal:message:numPending:
InterSessionSignal >> numPending
PetiteParser updated
The PetiteParser classes in GemStone have been updated with the code from PetitParser-JanKurs.290.mcz.
GsFile nextPutAsUtf8: replaced by nextPutAllUtf8:
The method GsFile >> nextPutAllUtf8: has been added, and the equivalent method GsFile >> nextPutAsUtf8: will be deprecated.
This is compatible with other methods on AppendableStream and AppendableString.
GsBitmap hiddenSetSpecifiers no longer include named Customer sets
GsBitmap >> hiddenSetSpecifiers previously included #CustomerSet1 through 5. Since individual GsBitmaps do not need identifiers, unlike the System hidden sets, static identifiers are unnecessary.
Also, #SaveDepMapChangedObjs and #SaveWrittenObjs have been removed, and #PreviousWsUnion has been added.
Added Socket methods
The following methods have been added:
GsSocket >> setCloseOnGc: aBoolean
If aBoolean is true the receiver's underlying socket will be closed when the in-memory state of the receiver is garbage collected.
GsSocket class hostIsLocalhost: hostNameOrIpAddrString
Returns true if the host that hostNameOrIpAddrString resolves to is equivalent to localhost.
GsSignallingSocket >>> acceptTimeoutMs: timeoutMs
errorOnTimeout: errOnTimeoutBool
Returns a socket created for a new connection, or signal an Error if there was an error. If errOnTimeoutBool == true, timeout signal an Error, otherwise returns false on timeout. The result is an instance of speciesForAccept, and has non-blocking state equal to non-blocking state of the receiver.
GsSocket >> read: maxBytes into: byteObj startingAt: index maxWait: timeoutMs
Reads up to the given number of bytes into the given byte object (for example, a String). The first byte read will go into the position indicated by index. Returns the number of bytes read, 0 for EOF, nil if an error occurs, or false if the receiver is not ready to read within timeoutMs. If no data is available for reading the current GsProcess is suspended for up to timeoutMs until data arrives.
GsSignalingSocket >> read: maxBytes into: byteObj startingAt: index maxWait: timeoutMs
Reads up to the given number of bytes into the given byte object. The first byte read will go into the position indicated by index. Returns the number of bytes read, 0 for EOF, or signals an error. If no data is available for reading the current GsProcess is suspended for up to timeoutMs until data arrives.
GsSecureSocket >> secureAcceptTimeoutMs: timeoutMs
errorOnTimeout: aBoolean
This is similar to the existing method GsSecureSocket >> secureAcceptTimeoutMs:, but provides the ability to return false rather than raise an exception.
Added System Class methods
System class >> currentSessionsReport
Return a string describing the current sessions. For example,
2 GcUser reclaimgcgem 17447
3 GcUser admingcgem 17449
4 SymbolUser symbolgem 17451
5 DataCurator gem 662 on benton
6 DataCurator gem 26928 on benton
System class >> stoneStartupId
Returns an Integer, a 64bit random number created in stone startup.
System class >> gemVersionReportString
Returns a String with keys and values of gemVersionReport, one pair per line.
System class >> stoneVersionReportString
Returns a String with keys and values of stoneVersionReport, one pair per line.
System class >> cacheStatusCount
Returns a positive SmallInteger value. The value maintained in the stone process increments whenever a remote cache starts up, shuts down, or becomes a midCache.
System class >> oldestCommitRecordAgeMilliseconds
Return a SmallInteger representing the age of the oldest commit record in milliseconds. This value represents the elapsed time since the commit that created the oldest commit record.
Added methods for GsExternalSession
Login/logout messages
By default when using GsExternalSessions, both login and logout write a log message using GsFile >> gciLogServer:, and additional user logging can be invoked using log:; all these messages can be suppressed using suppressLogging.
Now, GsExternaleSessions can be configured to suppress these messages without affecting user logging using log:. The following methods have been added:
GsExternalSession >> quiet
Disables messages on login and logout
As of the previous version v3.4, GCI login/logout messages are no longer included in GsExternalSession output.
Added methods for Squeak support
The following methods are implemented on Object and inherited/overridden by other classes:
Object >> squeakBasicAt: anIndex
For an OOP format receiver, return the value at anIndex. For a byte format receiver, return the unsigned Integer value of the word specified by anIndex, using Class _bytesPerWord as the size of a word.
Object >> squeakBasicAt: anIndex put: aValue
For an OOP format receiver, set the value at anIndex. For a byte format receiver, set the value at the word offset anIndex (calculating the offset using Class _bytesPerWord) to aValue.
Object >> squeakBasicSize
For an OOP format receiver, return the total number of instance variables. For a byte format receiver, return the size in words using Class _bytesPerWord as the size of a word.
Behavior >> squeakBasicNew: anInteger
Return an instance of the receiver, which must be indexable. If the receiver is OOP format, anInteger specifies the size; if byte format, the size in words, calculating the size using Class _bytesPerWord.
Changes to Strings and ByteArrays
Converting from ByteArray containing UTF-8 to decoded string
The following methods have been added:
ByteArray >> decodeFromUTF8ToString
The contents are bytes in UTF-8 encoding; decode and return as a String, DoubleByteString or QuadByteString.
ByteArray >> decodeFromUTF8ToUnicode
The contents are bytes in UTF-8 encoding; decode and return as Unicode7, Unicode16 or Unicode32.
As in recent 3.4.x versions, Utf8 class >> withAll: no longer accepts a ByteArray argument.
Support for Base64 encoding
Methods have been added to convert Strings and ByteArrays to Base64 strings.
Base64 is a binary-to-text encoding schemes that translates data into radix 64 representation. Base64 originates from a MIME content transfer encoding.
GemStone’s implementation allows conversion to include the output as 64-character lines or as a single line.
CharacterCollection >> asBase64StringOnOneLine: aBoolean
ByteArray >> asBase64StringOnOneLine: aBoolean
Return a String which represents the receiver represented in base64 format. If aBoolean is true, the resulting is one long line which does not contain newline characters. If aBoolean is false, newline characters are inserted such that each line does not exceed 64 characters.
CharacterCollection >> asBase64String
ByteArray >> asBase64String
Return a one-line String which represents the receiver represented in base64 format.
ByteArray class >> fromBase64String: aString
Creates an instance by decoding aString, which must be an instance of String or another single-byte character collection class. The argument must be in base64 format. The argument may contain newline and other whitespace characters, which are ignored.
Support for Hex String decoding
A hex string is a character collection that contains characters in the ranges 0 - 9, a-f, and A-F. Hex strings can be created using asHexString methods.
The following methods have been added to convert such hex strings into ByteArrays, in which two-characters substrings of the hex string are converted to an integer element. The argument hex string may additionally include either the C prefix of 0x or 0X in the first two characters, or the Smalltalk hex prefix of 16r as the first three characters.
If the hex string has an odd number of characters, then padding is done, either to the left or right; the default is to left pad.
ByteArray class >> fromHexString: aHexString leftPadded: padLeft
Create a new instance containing the byte values contained in aHexString. If aHexString contains an odd number of hex digits, then the result object is padded according to the padLeft argument; if true, the result is left-padded, meaning the first four bits of the first byte in the result object will be zero. If padLeft is false, the result is right-padded, meaning the last four bits of the last byte in the result object will be zero.
ByteArray class >> fromHexString: aHexString
Create a new instance containing the byte values contained in aHexString. If aHexString contains an odd number of hex digits, then the result object is left-padded.
ByteArray fromHexString: '201' leftPadded: true
aByteArray( 2, 1)
ByteArray fromHexString: '201' leftPadded: false
aByteArray( 32, 16)
Bitwise access to ByteArrays
The following methods have been added:
ByteArray >> bitAtZ: zeroBasedOffset
Returns a bit of the receiver, as if the receiver is an Array of bytes. The expression ((zeroBasedOffset bitShift: -3)+1) defines the byte addressed in the receiver. Within that byte,
bitAt:((zeroBasedOffset bitAnd: 7) + 1) is returned.
Returns 0 or 1.
ByteArray >> bitAtZ: zeroBasedOffset put: aBit
Stores a bit into the receiver, as if the receiver is an Array of bytes. The expression
((zeroBasedOffset bitShift: -3)+1) defines the byte addressed in the receiver. Within that byte,
bitAt:((zeroBasedOffset bitAnd: 7) + 1) is modified.
aBit must be either 0, 1, true, or false.
Other Added methods
Time >> asStringMs
Returns a String that expresses the receiver in local time in the format HH:MM:SS.sss, where sss are milliseconds.
DateAndTime >> asStringMs
Returns a String that expresses the receiver in local time in the format YYYY-MM-DD HH:MM:SS.sss, where sss are milliseconds.
GsHostProcess >> killChild
Terminate a GsHostProcess process that is running.
GsHostProcess >> killChild: timeout
Wait for the specified timeout and if the GsHostProcess child process is still running, terminate it.
3.8 Deprecations and removed methods
Cypress Classes moved to CodeLibrarian’s UserGlobals
Cypress classes with names Cypress* were moved to CodeLibrarian’s UserGlobals; see the information under Code Management.
Deprecated method
The method AutoComplete >> strings:cluster: has been deprecated; replace with AutoComplete>>strings:.
The method Behavior >> setPragmas:forMethod: has been deprecated. Pragmas are now generated into GsNMethod's debugInfo during compile.
Removed private methods
The following private methods were removed:
Behavior >> _primCompileMethod:symbolList:category:oldLitVars:
intoMethodDict:intoCategories:intoPragmas:environmentId:
Behavior >> _primitiveCompileMethod:symbolList:category:
oldLitVars:intoMethodDict:intoCategories:intoPragmas:
environmentId:
Class >> _validateOptions:withFormat:
GsHostProcess >> _waitChild
IndexManager class >> _loadHiddenSet:
IndexManager class >> _loadHiddenSet:fromFile:
Object >> __topazPerform0:withArguments:
Object >> _writeCypressJsonOn:
Object class >> _loadHiddenSet:fromFilesIn:
Object class >> _loadHiddenSet:fromFilesIn:withPattern:
PPEndOfFileParser >> acceptsEpsilon
PPEndOfFileParser >> exampleOn:
PPEndOfFileParser >> isNullable
PPEndOfFileParser >> nonEmpty
PPParser >> updateContext:
PPPluggableParser >> acceptsEpsilon
Repository >> _doListInstancesFrom:with:includeMemory:
Repository >> _fullBackupTo:MBytes:compressKind:bufSize:
Repository >> _listInstancesInMemory:
Repository:bufSize:privateDecryptionKey:passphrase:
Repository >> _primSecureFullBackupTo:MBytes:compressKind:
bufSize:encryptKind:publicKeyCerts:signatureHashKind:
signingKey:signingKeyPassphrase:
Repository >> _restoreBackups:scavPercentFree:bufSize:
SortedCollection class >> _loadHiddenSet:
SortedCollection class >> _loadHiddenSet:fromFile:
SymbolDictionary class >> _listClassesIn:
System class >> _conflictsReport:
System class >> _disallowSubsequentCommits
System class >> _remoteCachesListStartingAt:midLevelOnly:
System class >> _startGcCacheWarmer
Upgrade support methods changed to use GsBitmaps
During upgrade and conversion from older versions, sorted collections must be resorted and indexes rebuilt. The methods that support this have been adjusted to use GsBitmaps rather than Hidden Sets. The following methods have been specifically replaced:
IndexManager class >> rebuildAllCharCollIndexes:hiddenSetForErrors:
SortedCollection class >> convertAll:usingCache:hiddenSetForErrors:
SortedCollection class >> convertInstancesFromFilesForGem:of:
hsIdx:hsErrIdx:bmDirectoryPath:conversion:
SortedCollection class >> resortAll:hiddenSetForErrors:
SortedCollection class >> writeFailedConversionObjsToFileForGem:
hiddenSetForErrors:
The added methods that replace them are:
IndexManager class >> rebuildAllCharCollIndexes:
SortedCollection class >> convertAll:usingCache:bmForErrors:
SortedCollection class >> convertInstancesFromFilesForGem:of:
bm:errBm:bmDirectoryPath:conversion:
SortedCollection class >> resortAll:
SortedCollection class >> writeFailedConversionObjsToFileForGe
m:bmForErrors:
Other methods that interact with these methods have internal changes.
3.9 Encryption Changes
Support for SHA-3
The following methods have been added to CharacterCollection and ByteArray.
Support for HMAC (Hash-based message authentication codes)
Methods have been added to compute the HMAC for md5, sha1, sha256, sha512, sha3-224, sha3-256, sha3-384, and sha3-512. This can be computed for Strings or ByteArrays, which are converted to a base64 String to compute. The argument secretKeyString must be a single byte object: a String, ByteArray or Unicode7.
The following instance methods have been added to ByteArray and CharacterCollection.
asMd5HmacWithKey: secretKeyString
asMd5HmacStringWithKey: secretKeyString
asSha1HmacWithKey: secretKeyString
asSha1HmacStringWithKey: secretKeyString
asSha256HmacWithKey: secretKeyString
asSha256HmacStringWithKey: secretKeyString
asSha512HmacWithKey: secretKeyString
asSha512HmacStringWithKey: secretKeyString
asSha3_224HmacWithKey: secretKeyString
asSha3_224HmacStringWithKey: secretKeyString
asSha3_256HmacWithKey: secretKeyString
asSha3_256HmacStringWithKey: secretKeyString
asSha3_384HmacWithKey: secretKeyString
asSha3_384HmacStringWithKey: secretKeyString
asSha3_512HmacWithKey: secretKeyString
asSha3_512HmacStringWithKey: secretKeyString
Support for Symmetric-Key Encryption
Symmetric key encryption uses the same key to perform the encryption and the decryption (in contrast to asymmetric-key encryption, which use separate public and private keys).
AES encryption/decryption (Advanced Encryption Standard) is a block symmetric cipher, while ChaCha20 is a stream symmetric cipher.
- CBC is an acronym for cipher block chaining.
- OCB is an acronym for Offset Cookbook Mode.
- GCM is an acronym for Galois Counter Mode.
OCB, GCM and Poly1305 are Authenticated Encryption with Associated Data (AEAD) modes. AEAD provides data authenticity, confidentiality, and integrity.
AEAD also supports Additional Authenticated Data (AAD). AAD is not encrypted and therefore not confidential, but its authenticity and integrity are guaranteed. If AAD is used, it is not included in the encrypted payload, but must be provided in order to decrypt the data. The additional data is optional, so the argument for this may be nil, in which cases it is not needed for decryption.
AES is performed using the OpenSSL open source package and the AES specification, available at: http://csrc.nist.gov/publications/fips/fips197/fips-197.pdf.
Added methods
Methods have been added to CharacterCollection and ByteArray that allow you to sign and verify the signature of the receiver.
AlgorithmEncryptWithNNNBitKey: aKey salt: aSalt into: destObjOrNil tag: tag extraData: eData
AlgorithmDecryptWithNNNBitKey: aKey salt: aSalt into: destObjOrNil tag: tag extraData: eData
Encrypts/Decrypt the receiver using NNN bits and the algorithm Algorithm.
- 128. In this case aKey must be a ByteArray of size 16.
- 192. In this case aKey must be a ByteArray of size 24.
- 256. In this case aKey must be a ByteArray of size 32.
aesGcmEncryptWith128BitKey:salt:into:tag:extraData:
aesOcbDecryptWith256BitKey:salt:into:tag:extraData:
chacha20Poly1305DecryptWithKey:salt:into:tag:extraData:
destObjOrNil must be nil or an instance of a non-invariant byte object. If destObjOrNil is nil, the result of the operation will be placed into a new instance of ByteArray (encryption) or String (decryption); otherwise the result will be placed into the given byte object starting at offset 1.
The size of destObjOrNil will be modified to correctly contain all encrypted or decrypted data, and may differ from the size of the receiver due to the automatic addition or removal of padding by the cipher algorithm.
aSalt must be a ByteArray of size 12. aKey must be a ByteArray with the appropriate size for the method. The same key and salt must be used to decrypt as were used to encrypt.
During AEAD encryption, a tag is generated which is used during decryption to ensure data integrity. The tag data will be stored into the aTag argument, which must an instance of a byte object. eData must be nil or a byte object with a character size of one containing additional data to be used in generating the tag value.
For AEAD decryption, the tag argument aTag must be a byte object containing the tag bytes returned during encryption. eData must be a byte object containing the same bytes provided during encryption, or nil if no byte object was provided.
When encrypting a receiver that has a character size greater than one, data is placed into big-endian byte order before encryption.
When decrypting into a destObjOrNil object that has a character size greater than one, data is converted to big-endian byte order after decryption.
Example
| myKey mySalt myTag encoded decoded |
myKey := ByteArray withRandomBytes: 32.
mySalt := ByteArray withRandomBytes: 12.
myTag := ByteArray new.
encoded := 'abc' aesOcbEncryptWith256BitKey: myKey
salt: mySalt into: nil tag: myTag extraData: 'def'.
decoded := encoded aesOcbDecryptWith256BitKey: myKey
salt: mySalt into: nil tag: myTag extraData: 'def'.
3.10 GsTlsCredentials
The classes GsTlsCredential with subclasses GsTlsPrivateKey, GsTlsPublicKey, and GsX509Certificate have been added. These classes encapsulate TLS (SSL) private keys, public keys, and X509 certificates, respectively. Instances contain a hidden reference to C pointer to the OpenSSL representation of the TLS object.
Instances of GsTlsPublicKey and GsTlsPrivateKey are used for digital signing and digital envelopes, as well as in X509-Secured external sessions.
GsX509CertificateChain, a subclass of Array, has also been added to support X509-Secured logins using the new classes GemStoneX509Parameters and GsX509ExternalSession.
Creating a GsTlsCredentials
Instances are created by reading the PEM from a file or PEM-format string. The following new class methods can be used to instantiate any subclass of GsTlsCredential:
newFromPemFile:
newFromPemString:
Fetch public key from private key or certificate
To determine key type and extract a public key from a private key or from an X509 certificate, the following methods are available:
GsX509Certificate >> asPublicKey
Return the Public Key associated with the receiver.
GsTlsPrivateKey >> asPublicKey
Answers a new object which represents the public key for the receiver. The class of the new instance is the result of the message #speciesForPublicKey, which returns GsTlsPublicKey.
GsTlsKeyPublicKey class fromCertificate: aGsX509Certificate
Extract the public key from the argument aGsX509Certificate and return a new instance of the receiver.
Verifying public/private key pairs
The following method can be used to verify that two keys or X509 certificates form a matching public/private key pair:
matches: anotherKey
Return true if the receiver and anotherKey match each other as a valid public-private key pair, false otherwise. If the receiver is a public key, anotherKey is expected to be a private key, and vice versa.
If the receiver or anotherKey is an instance of GsX509Certificate, the public key is extracted from the certificate and used for comparison.
RSA and DSA key pairs match if both keys use the same modulus. Elliptic curve key pairs match if both keys use the same curve and the same point on that curve.
Querying a GsTlsCredential for its algorithm
OpenSSL 1.1.1 supports a number of encryption and signing algorithms. To query an instance of a kind of GsTlsCredential for the algorithm, the following methods are available:
algorithm
Answers a Symbol indicating the type of high-level PKI (Public Key Infrastructure) algorithm the receiver uses. The high-level PKI algorithms supported are:
#RSA - Rivest-Shamir-Adleman
#DSA - Data Signature Algorithm
#EC - Elliptic Curve Cryptography
All high-level algorithm have various sub-types. Use the sslAlgorithm method to obtain information about the specific PKI algorithm of the receiver.
sslAlgorithm
Answers a Symbol indicating the SSL type of PKI algorithm the receiver uses. See the image comment for details. If the receiver is an instance of GsX509Certificate, the result is the algorithm of its public key.
description
Answers a String obtained from SSL describing the receiver. The contents and format of the string vary depending on the receiver's class.
3.11 Support for Digital Signatures
Methods have been added to CharacterCollection and ByteArray that allow you to sign and verify the signature of the receiver.
signWithAlgorithmAndKeyTypePaddingPrivateKey: aGsTlsPrivateKey into: aByteArrayOrNil
Hashes the receiver using Algorithm and signs the resulting hash with the KeyType and Padding. Returns a ByteArray containing the resulting signature.
verifyWithAlgorithmAndKeyTypePaddingPublicKey: aGsTlsPublicKey signature: aByteArray
Hashes the receiver using Algorithm and verifies the resulting hash using the KeyType and Padding. Returns true if the signature is correct. Otherwise raises an exception.
Where the following AlgorithmAndKeyTypePadding are available:
signWithSha3_256AndRsaPssPrivateKey:into:
verifyWithSha3_256AndRsaPublicKey:signature:
signWithSha512AndDsaPrivateKey:into:
verifyWithSha512AndDsaPublicKey:signature:
GsTlsCredential Added Methods relating to Digital Signatures
Instances of the class GsTlsCredential and its subclasses, GsTlsPublicKey, GsTlsPrivateKey, and GsX509Certificate, have the following added methods.
supportsDigitalSignatures
Answers a Boolean indicating if the receiver supports digital signatures. The type of support offered, either signing or verifying, depends upon the specific class of the receiver.
canCreateDigitalSignatures
Returns true if the receiver is an instance of a GsTlsPrivateKey, false for GsTlsPublicKey and GsX509Certificate.
canVerifyDigitalSignatures
Returns true if the receiver is an instance of a GsTlsPrivateKey, false for GsTlsPublicKey and GsX509Certificate.
3.12 Support for Digital Envelopes
GemStone has added support for GsDigitalEnvelopes, which allow both encryption and signing in a single operation.
A GsDigitalEnvelope has the following security features:
- Confidentiality -- the message is encrypted using a randomly generated AES encryption session key and initialization vector. The session key is then encrypted with the provided public encryption key.
- Integrity -- authenticated encryption guarantees the cipher text has not been altered. The digital signature guarantees the encrypted key, initialization vector and tag have not been altered.
- Authentication -- the receiver of the envelope is assured that the sender signed the envelope with the private key matching the public key used to successfully verify the signature.
NOTE: In order to guarantee authentication, the receiver must confirm that the public verification key actually belongs to the sender. Normally this confirmation is done by verifying that an X509 certificate containing the sender's public key has been signed by a reputable certificate authority.
GsDigitalEnvelope does NOT do this public key/certificate signature verification; it is up to the envelope’s recipient to ensure that the public key or X509 certificate used to verify the signature is trustworthy.
Creating the GsDigitalEnvelope
Digital Envelopes are created using the following methods:
GsDigitalEnvelope class >> encryptMessage: messageBytes
withPublicEncryptionKey: publicEncryptionKey
cipherId: cipherOpCode
withPrivateSigningKey: privateSigningKey
GsDigitalEnvelope class >> encryptMessage: messageBytes
withPublicEncryptionKeys: arrayOfPublicEncryptionKey
cipherId: cipherOpCode
withPrivateSigningKey: privateSigningKey
The message to be encrypted may be any kind of String or ByteArray. The name of the class is stored in the envelope so when the envelope is opened, an instance of the same kind of object is returned. Internally, the string is converted into big-endian form, but this is transparent to the user. It is an error if the class of object that was encrypted into the envelope is not resolvable at the destination.
publicEncryptionKey and privateSigningKey
The message is encrypted using a public key, and signed using a private key. The envelope can only be "opened" (decrypted) by a recipient with the private key that matches the public encryption key, and the public signing key.
Providing multiple encryption keys allows a message to encrypted by multiple public keys into multiple envelopes in one operation. Each envelope can be opened by the single matching private key, and the public signing key.
The encryption and signing keys can be instances of GsTlsPublicKey or GsX509Certificate.
publicEncryptionKey or the elements in arrayOfPublicEncryptionKey must be RSA.
privateSigningKey may be any of the following:
For signing algorithms that require a message digest algorithm, the SHA2-256 message digest is used. Signatures generated with RSA keys are padded using the RSA_PKCS1_PSS_PADDING scheme
The cipherId: argument accepts a numeric code specifying the AEAD (Authenticated Encryption with Additional Data) of the cipher; legal values are in the following table.
Ciphers corresponding to 6 and 10 are considered the most secure and are recommended for most applications.
Using the GsDigitalEnvelope
Support has also been added to allow GsDigitalEnvelopes to be passivated into string form, for ease of transmission to the destination.
Once a GsDigitalEnvelope object is created, it can be converted to a passive object (String) for transmission to the recipient. Decrypting ("opening") the given envelope requires the private key matching the argument public key, and the public signing key matching the private key used to sign the envelope. The recipient activates the GsDigitalEnvelope object, and then opens it using the matching keys.
A detailed example is provided in class methods in GsDigitalEnvelope. The example follows the following form:
On the originating server
envelope := GsDigitalEnvelope
encryptMessage: messageText
withPublicKey: <gsTlsPubEncrKey>
cipherId: 10
withPrivateSigningKey: <gsTlsPrivSigKey>.
"envelope serialized to a string for sending"
stream := WriteStream on: String new.
PassiveObject passivate: envelope toStream: stream.
passiveString := stream contents.