1. Installing GemStone/S 64 Bit Version 3.5.1

1. Shared memory

The upper limit for shared memory single segment size and total usage should be set to values larger than your desired Shared Page Cache size, and not more than 75% of your real memory size.

The single segment maximum size, shmmax, is set in bytes, and the total shared memory limit, shmall, is configured in pages, with a base page size of 4KB. Note that the results of ipcs may be reported in kbytes.

For example, if you have 8192 MB of real memory:

8192 MB * .75 = 6144 MB

6144 MB * 220 = 6442450944 bytes

6442450944 / 4K = 1572864

To set shared memory sizes, you would append the following text to the /etc/sysctl.conf file. The settings are read from this file during the boot process.

# Shared Memory setting for GemStone

kernel.shmall = 1572864

kernel.shmmax = 6442450944

For more details, consult your Linux operating system documentation.

2. Semaphores

You may need to increase the settings for semaphores. These settings are configured by setting kernel.sem to a 4-element array, with the equivalent to the old semmsl, semmns, semopm, and semmni. For example, append the following to the /etc/sysctl.conf file.

kernel.sem=1000 512000 64 2048

The first element sets the maximum number of semaphores per id (per semaphore set). This parameter limits the number of GemStone sessions that can log in to a particular Stone and connect to its shared page cache.

On the Stone’s node, this parameter must provide two semaphores for each user who will log in to that Stone from any node plus an overhead of four. In distributed systems, nodes that have only user sessions must provide two semaphore for each user session on that node plus an overhead of one.

The number of semaphores actually requested for a particular shared page cache depends on the GemStone configuration file read by the process that starts the cache and is (SHR_PAGE_CACHE_NUM_PROCS * 2) + 1.

The second value sets the total number of semaphores in the system, which must be increased to along with the first.

3. File Descriptors

Each user session requires two file descriptors, and others are needed for extents, transaction logs, and other overhead. The default setting for fs.file-max setting is usually sufficient.

4. Locking the Shared Page Cache in memory

If you intend to lock the shared page cache into memory via the stone configuration option SHR_PAGE_CACHE_LOCKED, then the linux user starting the stone must either have the Linux capability CAP_IPC_LOCK, or have a RLIMIT_MEMLOCK resource limit set greater than the size of the SPC.

5. OOM Killer

If your system runs low on memory, the Linux OOM killer may select GemStone processes to terminate. To protect the shared page cache and other critical GemStone processes, each GemStone process’s oom_score_adj, which is used to select processes to terminate, is adjusted. If the userid that will be running the server processes has the CAP_SYS_RESOURCE privilege, critical GemStone processes have their oom_score_adj reduced, making them safer; if the user does not have CAP_SYS_RESOURCE, then non-critical processes such as Gems have their score increased, so they will be selected rather than more critical processes.

To set CAP_SYS_RESOURCE on kernels v2.6.32 and later, set the capability on the executables:

a. Install libcap2-bin

b. for i in pgsvrmain gem stoned shrpcmonitor; do sudo setcap cap_sys_resource=pe $GEMSTONE/sys/$i ; done

c. for i in startstone topaz; do sudo setcap cap_sys_resource=pe $GEMSTONE/bin/$i ; done

6. PAM

If you are using UNIX authentication for GemStone logins, or if you run NetLDI as root with setuid (i.e. not in guest mode), you must have PAM (Pluggable Authentication Module) configured on the server. You may include a specific GemStone authorization service name, or allow the default “other” authentication definitions to be used.

PAM authentication definitions are in files under the directory /etc/pam.d. Alternatively, they can be lines in the configuration file /etc/pam.conf, but this usage is deprecated on many distributions. On these distributions, the presence of the /etc/pam.d directory will cause /etc/pam.conf to be ignored.

The specific GemStone service file names are gemstone.gem for user authentication, and gemstone.netldi for a NetLDI running with authentication.

The libraries that are specified in the stack depend on how you are configuring PAM to perform the authentication. The examples below are for PAM configured to invoke LDAP for authentication.

For example, to allow GemStone UNIX authentication, which uses PAM, to authenticate via LDAP, create a file named /etc/pam.d/gemstone.gem with the following contents:

auth	required	pam_ldap.so

For NetLDI authentication, again using LDAP, create a file named /etc/pam.d/gemstone.netldi with the following contents:

auth	required	pam_ldap.so

Red Hat, by default, installs a file /etc/pam.d/other which disables “other” authentication. On Ubuntu, it is enabled by default. You can allow the “other” authentication stack to be used for GemStone authentication by ensuring that the file /etc/pam.d/other has the following contents:

auth	required	pam_ldap.so

Consult your System Administrators for more information on how authentication is handled on your system.

7. Transparent Huge Memory Pages

The default size for memory pages on Linux is 4KB. Linux supports 2MB and 1GB Huge Pages; this document, and GemStone code, refers to these page sizes and all larger-than-default memory pages using the generic term large pages. Using large pages will improve performance for large repositories with large shared page caches.

On Linux, you may explicitly allocate Huge Pages to an application, which allows you to ensure that GemStone will be using these pages; this is described in the next section, Huge Memory Pages.

The Linux kernel (in versions later than 2.6.38) also supports transparent huge pages, which are 2MB in size (/sys/kernel/mm/transparent_hugepage/hpage_pmd_size returns 2097152). Transparent huge pages are limited to anonymous memory regions. If they are enabled (they are by default), transparent huge pages are managed by the kernel without application level configuration.

Linux systems should ensure:

• /sys/kernel/mm/transparent_hugepage/enabled
  should be set to "always" or "madvise".
• /sys/kernel/mm/transparent_hugepage/defrag
  should be set to "always" or "madvise".
• /sys/kernel/mm/transparent_hugepage/shmem_enabled
  should be set to "madvise", "within_size", or "always".

To confirm these settings are correct in your Linux environment, cat the file; the enabled parameter is within the square brackets. E.g.,

unix> cat /sys/kernel/mm/transparent_hugepage/enabled

always [madvise] never

To confirm that GemStone is using transparent huge pages, after GemStone has been started up, execute:

cat /proc/shrpcmonPid/smaps

and look for the /SYSV line, and the subsequent value for ShmemPmdMapped. Note that this command is expensive, and should not be run unnecessarily.

8. Huge Memory Pages

Linux supports explicit configuration for 2MB and 1GB large pages (on Linux, Huge Pages), as well as transparent large pages (as described in the previous section).

A Linux installation may be configured to support either 2MB and 1GB large pages, or both. Using large pages may improve performance for large repositories with large shared page caches; by explicitly configuring large pages, they are allocated on boot and the GemStone executable can be explicitly configured to use them.

To use explicitly configured large pages, you must determine the huge page size and the number of huge pages needed, configure Linux to allocate the required number of pages, and configure GemStone to use large pages.

To configure the use of large pages:

a. Determine pages sizes available on your Linux system, and decide on the size you intend to use. Smaller repositories may use 2MB pages, but repositories with multi-GB shared page cache sizes may benefit more from 1GB pages.

To determine the default huge page size, execute:

unix> grep -e '^Huge' /proc/meminfo

which reports, for example,

   HugePages_Total:    5000

   HugePages_Free:     2756

   HugePages_Rsvd:        0

   HugePages_Surp:        0

   Hugepagesize:       2048 kB

To determine if you have another page size available, execute:

unix> ls /sys/kernel/mm/hugepages/

If you have 2MB pages, it will return:

   hugepages-2048kB

If you have both 2MB and 1GB pages, it will report:

   hugepages-1048576kB  hugepages-2048kB

b. Determine the number of huge pages that will be needed, based on your GemStone configuration. This is calculated by the utility largememorypages. This utility needs several details to compute the number of required pages: the shared page cache size, the maximum number of GemStone processes, and the number of shared counters. These can be provided by arguments or read from an existing configuration file.

largememorypages [-e path ] [-z path] [-F cacheFrames | -M cacheKB]

  [ -P maxProcesses] [-C maxSharedCounters] [-p largeMemoryPageSize]

You may specify the required values using a valid configuration file. This is parsed as a Stone configuration file and the necessary values extracted.

-e path
specifies an executable config file (same as startstone -e).

-z path
specifies a system config file (same as startstone -z).

If a config file is not specified, then -M or -F and -P and -C are required. These arguments can be used in addition to the configuration file, in which case they override values set or computed in the configuration file:

-F cacheFrames
shared cache size expressed in 16 KB frames

-M cacheKB
shared cache size with a optional units suffix.

-P maxProcesses
setting for SHR_PAGE_CACHE_NUM_PROCS

-C maxSharedCounters
setting for SHR_PAGE_CACHE_NUM_SHARED_COUNTERS.

-p largeMemoryPageSize
setting for SHR_PAGE_CACHE_LARGE_MEMORY_PAGE_SIZE_MB. Optional; the default is 2MB, may also specify 1 GB (or equivalent using appropriate units).

The following example is for an (approximately) 20GB shared page cache, 200 processes, and 1900 shared counters, with 2 MB memory pages:

unix> largememorypages -p 2MB -M 20GB -C 1900 -P 200

Cache config is 1310720 pages = 20480 MB,  total is 21384 MB,  overhead 4% of configured size

Large page size requested is: 2 MB.

Large page overhead: 1.12 MB

For 1310720 pages, 200 processes and 1900 shared counters, 10 pusherThreads

  required cache size is 22422749184 bytes.

Number of 2 MB large pages required: 10692

<with further update commands>

c. Configure Linux to use large pages, according to the instructions for your Linux distribution. The following commands may be used. You must execute these as root.

To enable large/huge pages until the next system reboot:

echo numpages > /sys/kernel/mm/hugepages/hugepages-hugePageSize/nr_hugepages

For example:

unix> echo 10692 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages

To permanently enable large pages:

echo "vm.nr_hugepages=numpages" >> /etc/sysctl.conf

Note nr_hugepages may fail if insufficient contiguous memory is available. You can confirm the number of large memory pages that are available for use using grep -e '^Huge' /proc/meminfo.

d. Ensure that the executables can use large pages.

You can configure the executables to use large pages, by using setcap:

unix> setcap cap_ipc_lock=pe $GEMSTONE/sys/startshrpcmon

unix> setcap cap_ipc_lock=pe $GEMSTONE/sys/shrpcmonitor

Alternatively, the SPC monitor process can be run with an effective user ID of root:

chown root $GEMSTONE/sys/shrpcmonitor $GEMSTONE/sys/startshrpcmon

chmod u+s $GEMSTONE/sys/shrpcmonitor $GEMSTONE/sys/startshrpcmon

Configure GemStone to request large pages by set the configuration option SHR_PAGE_CACHE_LARGE_MEMORY_PAGE_POLICY. This can be set to 1 or 2; with a setting of 1, the cache will be started anyway if the request for large pages is denied, while a setting of 2 indicates that startup should fail if large pages cannot be allocated.

If you are using 1GB pages, you will also need to update the setting for SHR_PAGE_CACHE_LARGE_MEMORY_PAGE_SIZE_MB. By default, this is set to 2; if you are using 1GB pages, set this to 1024.

9. System clock

The system clock must be set to the correct time. When GemStone opens the repository at startup, it compares the current system time with the recorded checkpoint times as part of a consistency check. A system time earlier than the time at which the last checkpoint was written may be taken as an indication of corrupted data and prevent GemStone from starting. The time comparisons use GMT.

10. TCP keepalive option

GemStone processes ordinarily use the TCP keepalive option to determine how long they will wait after communications activity ceases unexpectedly. This setting can be useful for reaping stale RPC Gems, but the operating system default may not be appropriate for this purpose. For further information, refer to your operating system documentation.

11. Unset LD_BIND_NOW

On some Linux distributions, setting the environment variable LD_BIND_NOW may result in process startup failures due to loading incorrect shared libraries.

Licensed Customer key file

Licenced customers can email keyfiles@gemtalksystems.com or contact GemTalk Technical Support to request a keyfile for version 3.5.1 for their platform or platforms. Keyfiles for previous versions of GemStone are not valid with v3.5.1.

Community key file

The GemStone distribution includes a community key file, community.starter.key., with product and system limits per the Community and Web Edition License. See https://gemtalksystems.com/licensing for details on the license terms.

If you do not install a custom keyfile, this starter keyfile will be used instead.

Installing a keyfile

To specify the location and name of the keyfile using the KEYFILE configuration parameter, edit the configuration file that will be used by the v3.5.1 stone to include the location and name of the keyfile.

You may also put the keyfile in the default location, $GEMSTONE/sys/gemstone.key. This requires modifying the write permissions of the $GEMSTONE/sys directory; ensure you change this back to not writable, after this update.