16. System Sets
This chapter describes some internal structures that provide specialized behavior, different from the way normal objects are handled with respect to storage, visibility to other sessions, and transactional behavior.
These structures are intended for use by experienced GemStone programmers.
16.1 Hidden Sets
Hidden sets are internal GemStone structures that are used to hold objects in the form of OOPs. They are implemented as bitmaps, an efficient way to transfer large collections of objects. Hidden sets use heap memory, not temporary object cache memory, and the objects in the hidden set are not loaded in memory, so hidden sets can be very useful when working with very large collections.
Several repository-wide operations, such as listInstancesToHiddenSet:, write the results to a hidden set; this allows operations that may return very large result sets to complete, and the results to be enumerated, without exceeding memory limits.
Many hidden sets are used internally, but there are a number of hidden sets that are provided for customer use. The specific hidden sets and their purposes are documented in the image method System class >> HiddenSetSpecifiers. Hidden sets number 41 through 45 are designated for use by customers for their applications.
Hidden sets are ordered in OOP order. You can load hidden sets from data that is organized in any order, such as files containing oops sorted in page order, but that ordering will be lost in the hidden set. Any OOP can only appear once in a hidden set; so, like an IdentitySet, identical objects can only appear once, but equal objects can both be included.
Special objects that are encoded within the OOP cannot be stored in Hidden sets. Attempting to add objects such as SmallIntegers, SmallDoubles, Characters and Booleans to a Hidden set will result in an error.
For example, the method listInstancesToHiddenSet: puts the results of a listInstances operation in Hidden Set 1. The following code shows the call to this method, and how to use hidden set protocol to migrate each object:
Example 16.1
topaz 1> run
SystemRepository listInstancesToHiddenSet: MyClass.
[(System hiddenSetSize: 1) > 0]
whileTrue:
[ | resultBatch |
resultBatch := System hiddenSetEnumerate: 1 limit: 1024.
resultBatch do: [:aMyClass |
aMyClass migrate].
System commitTransaction.
].
%
Methods to work with Hidden Sets
Add
You can add a single object or an array of objects to a hidden set using the methods:
System Class >> add: anObject toHiddenSet: hiddenSetSpecifier
System Class >> addAll: anArray toHiddenSet: hiddenSetSpecifier
To add all objects in one hidden set to another hidden set, use:
System class >> addHiddenSet: hiddenSet1 to: hiddenSet2
Remove
You can remove a single object or an array of objects from a hidden set using the methods:
System Class >> remove: anObject fromHiddenSet: hiddenSetSpecifier
System Class >> removeFirst: count fromHiddenSet:hiddenSetSpecifier
System Class >> removeAll: anArray fromHiddenSet:hiddenSetSpecifier
System Class >> removeContentsOfHiddenSet: hiddenSet1
fromHiddenSet: hiddenSet2
System Class >> truncateHiddenSet: hiddenSetSpecifier toSize:newSize
Any objects to be removed that are not in the hidden set are ignored. For more details, see the method comments in the image.
To reinitialize the hidden set, removing all objects, use the following:
System Class >> hiddenSetReinit: hiddenSetSpecifier
Testing
To determine how large the hidden set is, use the method:
System Class >> hiddenSetSize: hiddenSetSpecifier
To determine if a specific object is in the hidden set, use:
System Class >> testIf: anObject isInHiddenSet: hiddenSetSpecifier
Set operations
To compute the union or difference of two hidden sets, and place the results in a third hidden set, use the following methods:
System Class >> computeUnionOfHiddenSet: hiddenSet1 and:
hiddenSet2 into: hiddenSet3
System Class >> computeDifferenceOfHiddenSet: hiddenSet1
and: hiddenSet2 into: hiddenSet3
Enumerating
The following method allows you to perform an operation on all elements in a given hidden set. This method removes up to 2000 elements at a time from the hidden set to operate on, so avoids memory issues for very large hidden sets, and returns a count of the total number of elements that were in the hidden set.
System class >> hiddenSetEnumerate: hiddenSetSpecifier do: aBlock
Retrieving Contents
Retrieving the contents of hidden sets is done through the following methods. These methods return a chunk of the contents of the hidden set as objects or as OOPs. These objects are removed from the hidden set.You can then perform whatever operations you need on each object in this chunk, before fetching another chunk. This way, very large collections of objects can be operated on.
System Class >> hiddenSetEnumerate: hiddenSetSpecifier
limit: maxResultSize
This method returns the first maxResultSize objects in the hidden set. If there are not that many objects in the hidden set, the result may be smaller than maxResultSize. If maxResultSize is 0, all objects are returned (similar to hiddenSetAsArray:).
System Class >> hiddenSetEnumerateAsInts: hiddenSetSpecifier
limit: maxResultSize
This method is the same as hiddenSetEnumerate:limit:, except the OOPs of the objects are returned, rather than the objects.
To create an Array containing all objects in the hidden set, use the following method.
System Class >> hiddenSetAsArray: hiddenSetSpecifier
Some care should be taken not to use this with very large hidden sets. The objects in the resulting array, unlike the objects in the hidden set, are in temporary object memory. If the hidden set is too large it may cause the session to run out of memory.
16.2 SessionTemps and access to Session State
Most data that you will work with in GemStone is either temporary or persistent. While most temporary data is only retained for as long as the method is executing, or until the session updates its commit record by committing or aborting, you may sometimes want data that is not persistent and not shared, so does not risk transaction conflicts, but remains unaffected by transaction status.
Session-specific data of this kind can be put into SessionTemps. SessionTemps current provides access to a kind of SymbolDictionary; elements in the SessionTemps dictionary remain until the session logs out or exits, are not affected by commit or abort, and are not visible outside of the session.
For example, if you wish to open a log file and leave it open:
SessionTemps at: #Log put: (GsFile openAppend: 'myFile.log')
Actual code, of course, would do more error checking. To write to the file, use code similar to this:
(SessionTemps at: #Log) nextPutAll: 'a message for the log file'.
Objects in SessionTemps use temporary object memory, and the objects cannot be removed from memory by in-memory garbage collection. While there is no limit on how much data can be stored in SessionTemps, if your session reaches the memory limit and exits, that data will be lost.
SessionState
SessionTemps uses a slot in the internal Session State structure, which is primarily provided for use by the kernel. Access to customer-available SessionState slots is provided primarily for legacy uses, but may be useful depending on application requirements.
SessionState is accessed by integer index, with slots 1 to 1994 available for use. The SessionState array is variable size, and will grow as needed.
The following methods can be used to read and update SessionState:
System >> sessionStateAt: anIndex
System >> sessionStateAt: anIndex put: anObject
System >> sessionStateSize
16.3 Shared Counters
There are two types of Shared Counters available; AppStat Shared Counters and Persistent Shared Counters.
AppStat Shared Counters provide a way for sessions on the same shared cache to read and update a set of counters. These counters are stored in the shared cache and are not persistent across cache restart. They are not visible to sessions on different shared page caches - that is, a session on the stone’s cache and a session on a remote cache cannot access the same Shared Counters. Values are also not recoverable from tranlogs.
Persistent shared counters are stored in the repository, and are visible to all sessions on all shared caches. On repository recovery or restore, the values of persistent shared caches are restored.
AppStat Shared Counters
Shared counters allow multiple sessions on the same SPC to read and update a common counter value.
Shared counters are indexed from 0 to (System numSharedCounters - 1), which is set by the configuration parameter SHR_PAGE_CACHE_NUM_SHARED_COUNTERS. The default value for SHR_PAGE_CACHE_NUM_SHARED_COUNTERS is 1900. Each counter is protected by a unique spinlock. The index of the first counter is 0.
Shared counters may be set to any signed 64 bit integer value, in the range:
-263 (-9223372036854775808) to 263 - 1 (9223372036854775807)
If you increment or decrement so that the result would be outside the range of a signed 64-bit integer, the value will be set to the minimum or maximum; directly setting an out of range value will result in an error.
Shared counters are transient, that is, they do not persist across cache restart.
Shared counter values are recorded by statmonitor when using the -n option and recorded as AppStats.
The following methods may be used to read and update shared counters. For details, see the method comments in the image.
System class >> numSharedCounters
System class >> sharedCounter: index
System class >> sharedCounter: index setValue: value
System class >> sharedCounter: index incrementBy: amount
System class >> sharedCounter: index decrementBy: amount
System class >> sharedCounter: index decrementBy: amount withFloor: floorValue
System class >> sharedCounterFetchValuesFrom: firstCounter to: lastCounter
Persistent Shared Counters
Persistent shared counters allow all sessions in a repository to read and update a set of counters. Persistent shared counters are globally visible to all sessions on all shared page caches.
There are 1536 persistent shared counters, numbered from 1 to 1536. The index of the first counter is 1.
Persistent shared counters may be set to any signed 64 bit integer value, in the range:
-263 (-9223372036854775808) to 263 - 1 (9223372036854775807)
No limit checks are done when incrementing or decrementing a counter. If you increment or decrement so that the result would be outside the range of a signed 64-bit integer, the value will “rollover” and the overflow bits will be lost. Directly setting an out of range value will result in an error.
Values of all persistent shared counters are stored in the repository and in tranlog records. They are persistent through Stone restart, and recovered on Stone crash, restore from backup, and restore from tranlog.
Persistent shared counters are independent of database transactions. Updates to counters are visible immediately and not affected by aborts.
Each update to a persistent shared counter causes a roundtrip to the Stone; but reading the value is handled by the gem (and the page server, if remote), and does not cause a roundtrip to the stone.
The following methods may be used to read and update persistent shared counters. For details, see the method comments in the image.
System class >> numberOfPersistentSharedCounters
System class >> persistentCounterAt: index put: value
System class >> persistentCounterAt: index
System class >> persistentCounterAt: index incrementBy: amount
System class >> persistentCounterAt: index decrementBy: amount