14. Managing Gem Memory
Executing your application code will naturally require access to previously committed objects in the repository. These objects are faulted into the Gem session’s memory to be examined or updated. When new objects are created, they must reside in memory while they are being modified.
GemStone automatically garbage-collects temporary objects that are no longer referenced, and clears out the space used by persistent, committed objects, when memory is needed. However, the memory available for any session is finite. If you need to create large temporaries or modify many objects within a transaction, you may need to tune your application to increase the available memory, or to use memory more efficiently.
This chapter discusses the following topics:
14.1 Memory Organization
Each Gem session has a temporary object memory that is private to the Gem process and its corresponding session. This local object memory is divided into the following regions:
- new — Young temporary objects; includes two subspaces named eden and survivor
- old — Older temporary objects
- pom — Unmodified faulted-in objects, divided into ten subspaces
- perm — Faulted-in or created Classes and Metaclasses
- code — Instances of GsNMethod being executed, or recently executed
- mE — oopMap entries that map objId to in-memory objects for committed objects
Temporary objects are created in the new area of local object memory. When the new area fills up, a scavenge occurs which throws away unreferenced objects in new. After an object has survived a number of scavenges, it is copied to the old area. After the old area has grown by some amount or is almost full, a mark/sweep takes place, finding all live objects and then compacting the new, old, perm, and code areas as needed to remove dead objects.
Committed objects referenced by the session are copied from the shared page cache into the pom, perm, or code areas at the point they are first referenced by interpreter execution or a GCI call. (This is called a "copy-on-read" design.)
If a committed object in the pom area has been modified, it is copied to the old area if a scavenge occurs before the change is committed. Objects that are sent to the GCI client, such as GBS, are also moved to the old area, whether or not they are modified.
When the pom area becomes full, the contents of its oldest subspace (that is, the oldest 10%) are discarded, and that subspace is reused to continue faulting-in committed objects. Before the oldest subspace is recycled, any objects in the subspace that have been modified, or that are currently referenced from the interpreter stack, are copied to the old area.
At transaction commit, any committed objects that have been modified, and any new objects transitively reachable from those modified objects, are copied to new data pages in the shared cache. A transaction conflict check is then performed. If the commit succeeds, the in-memory state of all new objects copied to the shared cache is changed to "committed". The newly committed objects are now eligible to be removed from temporary memory by a mark/sweep or scavenge if they are no longer directly referenced from temporary objects.
14.2 Configuring Temporary Memory Usage
You may encounter an OutOfMemory error if you create too large a graph of live temporary objects at any time, or if you try to modify too many committed objects in a single transaction. OutOfMemory is a fatal error that terminates the session.
Very large numbers of Classes can also fill up temporary object memory. Any class that is referenced by message send or iteration is loaded into the perm area, and its method dictionaries, classHistory, and so on are loaded into the old area.
Persistent objects that are in the export set for a GCI client, such as GemBuilder for Smalltalk, are also moved to the old area. This includes objects that are replicated but not modified.
If you find that your application is running out of temporary memory, you can use several GemStone environment variables to help you identify which parts of your application are triggering garbage collection. Once you’ve done that, you can set GemStone configuration options to provide the needed memory.
Configuration Options
The values for these options are set when the gem or topaz -l process is initialized. You cannot change these values without restarting the VM. For more about these options, see the descriptions that begin here.
GEM_TEMPOBJ_CACHE_SIZE
The maximum size (in KB) of temporary object memory. (This limit also applies to linked Topaz sessions and linked GemBuilder applications.) When you only change this setting, and the other GEM_TEMPOBJ... configuration options use default values, then all of the various spaces remain in proportion to each other.
The following should normally be left at the default, so the system can calculate the appropriate values based on the setting for GEM_TEMPOBJ_CACHE_SIZE.
GEM_TEMPOBJ_MESPACE_SIZE
The maximum size (in KB) of the mE (Map Entries) space within temporary object memory.
GEM_TEMPOBJ_OOPMAP_SIZE
The size of the hash table (that is, the number of 8-byte entries) in the objId-to-object map within temporary object memory.
GEM_TEMPOBJ_PERMGEN_SIZE
The maximum size (in KB) of the PERM generation area in temporary object memory. The PERM generation areas holds faulted in or creates instances of Classes and Metaclasses.
GEM_TEMPOBJ_POMGEN_SIZE
The maximum size (in KB) of the POM generation area in temporary object memory. The POM generation area holds unmodified copies of committed objects that have been faulted into a Gem, and is divided into ten subspaces.
Methods for Computing Temporary Object Space
To find out how much space is left in the old area of temporary memory, the following methods in class System (category Performance Monitoring) are provided:
System _tempObjSpaceUsed
Returns the approximate number of bytes of temporary object memory being used to store objects.
System _tempObjSpaceMax
Returns the approximate maximum number of bytes of temporary object memory that are usable for storing objects.
System _tempObjSpacePercentUsed
Returns the approximate percentage of temporary object memory that is in use to store temporary objects. This is equivalent to the expression:
(System _tempObjSpaceUsed * 100) // System _tempObjSpaceMax.
Note that it is possible for the result to be slightly greater than 100%. Such a result indicates that temporary memory is almost completely full.
Sample Configurations
This section presents several sample configurations:
These examples assume that you have already set the GS_DEBUG_VMGC... environment variables (Configuration Options) to produce the resulting printouts. The examples shown are for Solaris and may vary on other platforms.
Default Configuration
A value of 75000 for GEM_TEMPOBJ_CACHE_SIZE (that is, 75 MB) produces a limit of about 35 MB of temporary plus modified-committed objects (old), space for a working set of about 40 MB of unmodified committed objects (pom), and a maximum memory footprint on the order of 160 MB.
The following example shows the printout for this configuration:
(vmGc spaceSizes: eden init 2048K max 9344K , survivor init 448K max 1600K,
vmGc old max 37440K, code max 10048K, perm max 5056K, pom 10 * 4224K = 42240K,
vmGc remSet 1156K, meSpace max 51544K oopMapSize 262144 max footprint 160M)
Larger old, Smaller pom
The following settings configure the application for a 20 MB working set of unmodified committed objects (smaller than the default), and a maximum of 100 MB of temporary plus modified objects (larger than the default).
GEM_TEMPOBJ_CACHE_SIZE = 100 MB;
GEM_TEMPOBJ_POMGEN_SIZE = 20 MB;
The following example shows the printout for this configuration:
(vmGc spaceSizes: eden init 2048K max 19200K , survivor init 384K max 3200K,
vmGc old max 76800K, code max 20480K, perm max 10240K, pom 10 * 2048K = 20480K,
vmGc remSet 1732K, meSpace max 73764K oopMapSize 262144 max footprint 229M)
Smaller old, Larger pom
The following settings configure an application with a large working set of committed objects and smaller temporary object space.
GEM_TEMPOBJ_CACHE_SIZE = 50000;
GEM_TEMPOBJ_POMGEN_SIZE = 100000;
The following example shows the printout for this configuration:
(vmGc spaceSizes: eden init 2048K max 9344K , survivor init 448K max 1600K,
vmGc old max 37440K, code max 10048K, perm max 5056K, pom 10 * 10048K = 100480K,
vmGc remSet 1732K, meSpace max 79512K oopMapSize 524288 max footprint 247M)
Debugging out-of-memory errors
When any of the following environment variables are set to a positive non-zero value, they have the effect described here for each Gem or linkable Topaz (topaz -l) process that you subsequently start. For all of these environment variables, the printout goes to the "output push" file of a linkable Topaz (topaz -l) session, for use in testing your application. If that file is not defined, the printouts go to standard output of the session’s Gem or topaz -l process.
The Gem service script gemnetdebug is designed to be used instead of gemnetobject, for debugging memory issues. Memory environment variables are documented in this script ($GEMSTONE/sys/gemnetdebug). When using this script, most of these environment variables remain commented out; you must uncomment them in order for them to have effect.
The contents of gemnetdebug are subject to change. For the most current information about these and other variables, examine the contents of gemnetdebug.
GS_DEBUG_COMPILE_TRACE
Trace method compiles. The following are valid values:
0 - no tracing
1 - one line (class,selector) of each method compiled
2 - in addition to above, bytecode disassembly
3 - in addition to above, native code assembly listing
GS_DEBUG_VMGC_MKSW_MEMORY_USED_SOFT_BREAK
At the end of each mark/sweep, if the percent of memory used is greater than the threshold specified by this variable, a SoftBreak (error 6003) is generated, and the threshold is raised by 5 percent. We suggest a setting of 75%.
GS_DEBUG_VMGC_MKSW_PRINT_C_STACK
The mark/sweep count at which to begin printing the C stack at each mark/sweep. This variable is very expensive, consuming two seconds plus the cost of fork() for each printout.
GS_DEBUG_VMGC_MKSW_PRINT_STACK
The mark/sweep count at which to begin printing the Smalltalk stack at each mark/sweep.
GS_DEBUG_VMGC_PRINT_MKSW
The mark/sweep count at which to begin printing mark/sweeps.
GS_DEBUG_VMGC_PRINT_MKSW_MEMORY
The mark/sweep count at which to begin printing detailed memory usage (20 lines) for each mark/sweep.
GS_DEBUG_VMGC_PRINT_MKSW_MEMORY_USED
Specifies when Smalltalk stack printing starts as the application approaches OutOfMemory conditions. At the end of each mark/sweep, if the percent of memory used is greater than the threshold specified by this variable, the mark/sweep is printed, the Smalltalk stack is printed, and the threshold is raised by 5 percent. In a situation producing an OutOfMemory error, you should get several Smalltalk stacks printed in the Gem log file before the session dies.
GS_DEBUG_VMGC_PRINT_SCAV
The scavenge count at which to begin printing scavenges. Once this takes effect, all mark/sweeps will also be printed. Be aware that printing scavenges can produce large quantities of output.
GS_DEBUG_VMGC_SCAV_PRINT_C_STACK
The scavenge count at which to begin printing the C stack at each scavenge. This variable is very expensive, consuming 2 seconds plus the cost of fork() for each printout.
GS_DEBUG_VMGC_SCAV_PRINT_STACK
The scavenge count at which to begin printing the Smalltalk stack at each scavenge. Be aware that this print activity can produce large quantities of output.
GS_DEBUG_VMGC_VERBOSE_OUTOFMEM
Automatically call the primitive for System class>>_vmPrintInstanceCounts:0 when an OutOfMemory error occurs, and also print the Smalltalk stack. (For details about this method, see the comments in the image.) This applies to each Gem or linkable Topaz (topaz -l) process that you subsequently start.
GS_DEBUG_VMGC_VERIFY_LOGOUT
Verify object memory at logout, for use when memory corruption or GC bugs are suspected.
GS_DEBUG_VMGC_VERIFY_MKSW
The mark/sweep count at which to begin verifying object memory before and after each mark/sweep.
GS_DEBUG_VMGC_VERIFY_SCAV
The scavenge count at which to begin verifying object memory before and after each scavenge. Once this takes effect, GS_DEBUG_VMGC_VERIFY_MKSW will also be in effect. Be aware that this activity uses significant amounts of CPU time.
GS_DEBUG_VM_PRINT_TRANS
Print transaction boundaries (begin/commit/abort) in the log file.
Recording Out of Memory Information to CSV file
The Out of memory and almost out of memory information that is written to a Gem log can also be directed to write to a disk log file in (CSV) comma separated format, which can be loaded into spreadsheet applications for analysis.
To configure your system to write such a CSV file, set the configuration option GEM_TEMPOBJ_OOMSTATS_CSV to TRUE.
If a Gem or linked topaz process encounters an almost out of memory or out of memory condition, a file will be created or appended to with the name gemnetobjectpid.csv or topazpid.csv.
The config parameter GEM_TEMPOBJ_OOMSTATS_CSV and the environment variable GS_DEBUG_VMGC_VERBOSE_OUTOFMEM operate independently.
Signal on low memory condition
When a session runs low on temporary object memory, there are actions it can take to avoid running out of memory altogether. By enabling handling for the signal AlmostOutOfMemory, an application can take appropriate action before memory is entirely full. This signal is asynchronous, so may be received at any time memory use is greater than the threshold the end of an in-memory markSweep. However, if the session is executing a user action, or is in index maintenance, the error is deferred and generated when execution returns.
When performing index operations, such as creating indexes for large collections, the IndexManager can be configured to use this facility to automatically commit when memory is low. Committing objects allows them to be removed from memory, since they can be re-loaded as needed from the persistent object. See the Programming Guide for details on IndexManager autoCommit, and for more information on handling AlmostOutOfMemory.