1. Installing GemStone/S 64 Bit Version 3.7

1. Locking the Shared Page Cache in memory

It is recommended that you have sufficient memory to avoid the need to swap to disk. If there is the risk of swapping, you can avoid problems with GemStone performance (at the expense of other processes running on this machine) by locking the shared page cache into memory. Locking the cache into memory is configured by the parameter SHR_PAGE_CACHE_LOCKED.

If you are using huge pages (Huge Memory Pages), the cache is inherently locked, so there is no further action needed.

Locking the cache into memory requires that the user starting GemStone has permission to lock the amount of memory required by the cache. There are several ways to do this.

• Edit /etc/security/limits.conf to give the administrative user permission to lock larger memory segments:

   gsAdminUser soft memlock sizeInKB

   gsAdminUser hard memlock sizeInKB

Note that after editing this file, you do not need to reboot, but you will need a new shell.

• While not recommended for security reasons, you may give the Shared Page Cache Monitor executable the Linux capability CAP_IPC_LOCK.

os$ setcap cap_ipc_lock=pe $GEMSTONE/sys/shrpcmonitor 

Note that setting this capability will prevent statmonitor and gdb from attaching to shrpcmonitor; statmonitor will not be able to record certain process statistics, and pstack cannot get C stack traces. See statmonitor additional permission.

2. OOM Killer

If your system runs low on memory, the Linux OOM killer may select GemStone processes to terminate. To protect the shared page cache and other critical GemStone processes, the process’s oom_score_adj, which is used to select processes to terminate, is reduced, making them less likely to be selected and killed.

Reducing the oom_score_adj requires configuring the process to have that capability. To set the capability on the required executables:

os$ setcap cap_sys_resource=pe $GEMSTONE/sys/stoned 

os$ setcap cap_sys_resource=pe $GEMSTONE/sys/pgsvrmain 

Only the Stone and pgsvrmain executables need the capability; other critical processes spawned by the Stone, such as the Shared Page Cache Monitor, inherit the Stone’s protection and adjust their oom_score_adj, and therefore also are protected. The capability is dropped as soon as the oom_score_adj has been updated.

Likewise, the pgsvrmain, which is responsible for spawning critical processes for distributed configurations with remote caches, protects processes such as the remote cache.

To examine the oom_score_adj for a process, check the contents of the file:

os$ cat /proc/pid/oom_score_adj 

3. Transparent Huge Memory Pages

The default size for memory pages on Linux is 4KB. Linux supports 2MB and 1GB Huge Pages; this document, and GemStone code, refers to these page sizes and all larger-than-default memory pages using the generic term large pages. Using large pages will improve performance for large repositories with large shared page caches.

The Linux kernel allows you to explicitly reserve 2MB or 1GB large pages for a specific application (such as GemStone), and also allows you to take advantage of 2MB transparent large pages, which are managed automatically without application level configuration.

Specifically allocated large memory pages are described in the next section, Huge Memory Pages.

Transparent huge pages are limited to anonymous memory regions. If they are enabled (they are enabled by default), transparent huge pages are managed by the kernel without application level configuration.

In general we recommend settings of "madvise" or "advise", which allow GemStone to use transparent huge pages. Settings of "always" will allow GemStone to use transparent huge pages, but may affect overall performance either positively or negatively.

Linux systems should ensure:

• /sys/kernel/mm/transparent_hugepage/enabled
  should be set to "madvise".
• /sys/kernel/mm/transparent_hugepage/defrag
  should be set to "madvise".
• /sys/kernel/mm/transparent_hugepage/shmem_enabled (on all except some older OS versions)
  should be set to "advise".

To confirm these settings are correct in your Linux environment, cat the file; the enabled parameter is within the square brackets. E.g.,

os$ cat /sys/kernel/mm/transparent_hugepage/enabled

always [madvise] never

To confirm that GemStone is using transparent huge pages, after GemStone has been started up, execute:

os$ cat /proc/pid/smaps

and look for the /SYSV line, and check that the subsequent value for ShmemPmdMapped is nonzero.

4. Huge Memory Pages

Linux supports explicit configuration for 2MB and 1GB large pages that are specifically reserved for an application, as well as transparent large pages (as described in the previous section).

A Linux installation may be configured to use either 2MB and 1GB large pages, or both. Using large pages may improve performance for large repositories with large shared page caches; by explicitly configuring large pages, they are allocated on boot and the GemStone executable can be explicitly configured to use them.

To use explicitly configured large pages, you must determine the page size and the number of pages needed, configure Linux to allocate the required number of pages, and configure GemStone to use these pages.

To configure the use of large pages:

a. Determine the Linux Page Size

From the page sizes that are available on your Linux system, you must decide on the size you intend to use. Smaller repositories may use 2MB pages, but repositories with multi-GB shared page cache sizes may benefit more from 1GB pages.

To determine the default huge page size, execute:

os$ grep -e '^Huge' /proc/meminfo

which reports, for example,

   HugePages_Total:    5000

   HugePages_Free:     2756

   HugePages_Rsvd:        0

   HugePages_Surp:        0

   Hugepagesize:       2048 kB

To determine if you have another page size available, execute:

os$ ls /sys/kernel/mm/hugepages/

If you have 2MB pages, it will return:

   hugepages-2048kB

If you have both 2MB and 1GB pages, it will report:

   hugepages-1048576kB  hugepages-2048kB

b. Determine the required number of Linux Pages

Calculate the number of huge pages that will be needed, based on your GemStone configuration. This is calculated by the utility largememorypages. This utility needs several details to compute the number of required pages: the shared page cache size, the maximum number of GemStone processes, and the number of shared counters. These can be provided by arguments or read from an existing configuration file. When using a configuration file, the file must have large memory pages (SHR_PAGE_CACHE_LARGE_MEMORY_PAGE_POLICY) enabled, and page size (SHR_PAGE_CACHE_LARGE_MEMORY_PAGE_SIZE_MB) set, if the calculated default page size is not correct.

The largememorypages computation by default is for the Stone’s cache. When configuring large pages for remote cache, the values are somewhat smaller. Use the -r option to compute large page requirements for a remote cache.

You may specify the required values using an existing configuration file, or by passing in the relevant values as argument. A configuration file is parsed as a Stone configuration file and the necessary values extracted; the configuration file must have large memory pages enabled.

If a configuration file is not specified, then -M or -F, and -p, -P and -C are required. These arguments can be used in addition to the configuration file, in which case they override values set or computed in the configuration file:

largememorypages [-e path ][-z path][-r][-F cacheFrames | -M cacheKB]

  [-P maxProcesses][-C maxSharedCounters][-p largeMemoryPageSize]

-e path
specifies an executable configuration file (same as startstone -e).

-z path
specifies a system configuration file (same as startstone -z).

-F cacheFrames
shared cache size expressed in 16 KB frames

-M cacheKB
shared cache size with a optional units suffix.

-P maxProcesses
setting for SHR_PAGE_CACHE_NUM_PROCS

-C maxSharedCounters
setting for SHR_PAGE_CACHE_NUM_SHARED_COUNTERS.

-p largeMemoryPageSize
setting for SHR_PAGE_CACHE_LARGE_MEMORY_PAGE_SIZE_MB. Options are 2MB and 1GB (or equivalent using appropriate units).

The following example is for an (approximately) 20GB shared page cache, 200 processes, and 1900 shared counters, with 1 GB memory pages:

os$ largememorypages -p 1GB -M 20GB -C 1900 -P 200

Cache kind is: Primary SPC (stone shared page cache)

Cache config is 1373248 pages = 21457 MB,  total is 22528

MB,  overhead 4% of configured size

  HashTableSize 2097152 TotalPces 3483008 freeListPces

1385856

Large page size requested is: 1024 MB.

Large page overhead: 0.09 MB

[Info]: Increasing number of pages from 1310720 to 1373248

to reduce large memory page overhead.

For 1373248 pages, 200 processes and 1900 shared counters, 0

pusherThreads

  minimum sizing for cache shmmax 23622320128, shmall

5767168.

Number of 1024 MB large pages required: 22

<with further update commands>

c. Allow large memory pages to be allocated and accessed by GemStone processes

To ensure processes have permission to use large memory pages, you need to do one of the following. You must execute these commands as root.

• Define a huge pages group.

Select or create a group that includes the Linux user that will be starting up the shared page cache, and all users who will be accessing the shared page cache

Then create a file /etc/sysctl.d/64-gemstone-local.conf, to set this group’s numeric id; add the line:

vm.hugetlb_shm_group = numericGidOfGroup

To determine the group id based on the group name, execute

os$ getent group nameOfGroup

This will take effect on reboot; you can apply immediately by executing:

os$ sysctl --system

To verify the current value, cat /proc/sys/vm/hugetlb_shm_group, which will show the group id. To enable large pages transiently without rebooting, you can set the gid of the group directly in /proc/sys/vm/hugetlb_shm_group. This will be reset on reboot if not configured as by /etc/sysctl.d/64-gemstone-local.conf.

• While not recommended for security reasons, you may instead give the Shared Page Cache Monitor the cap_ipc_lock capability:

os$ setcap cap_ipc_lock=pe $GEMSTONE/sys/shrpcmonitor

Setting this capability will prevent statmonitor and gdb from attaching to that process; statmonitor will not be able to record certain process statistics, and pstack cannot get C stack traces. See statmonitor additional permission.

d. Configure large pages on Linux

Linux must be configured according to the instructions for your Linux distribution. You must execute these commands as root.

(1) To enable hugepages, the file /etc/default/grub needs to contain an entry for GRUB_CMDLINE_LINUX_DEFAULT that defines the number of 2MB and 1 GB huge pages.

Edit the file /etc/default/grub, to append a line defining the number of each size of pages to the GRUB_CMDLINE_LINUX_DEFAULT entry, for example to configure with 1GB pages to support the 20GB cache in the above example:

"hugepagesz=1G hugepages=22 hugepagesz=2M hugepages=0 default_hugepagesz=1G"

(2) Run the command to update the kernel config.

On Ubuntu:

os$ /usr/sbin/update-grub

On Red Hat and CentOS:

os$ /usr/sbin/grub2-mkconfig -o /boot/grub2/grub.cfg

On Red Hat and CentOS UEFI systems:

os$ /usr/sbin/grub2-mkconfig -o /boot/efi/EFI/centos/grub.cfg

(3) Reboot the system:

os$ shutdown -r now

e. Configure GemStone to use large pages

To configure GemStone to request large pages, set the configuration option SHR_PAGE_CACHE_LARGE_MEMORY_PAGE_POLICY. This can be set to 1 or 2; with a setting of 1, the cache will be started anyway if the request for large pages is denied, while a setting of 2 indicates that startup should fail if large pages cannot be allocated.

Setting SHR_PAGE_CACHE_LARGE_MEMORY_PAGE_SIZE_MB to 2 (for 2 MB) or 1024 (for 1GB) is recommended; otherwise, the default is based on the current configuration of 1 GB, 2 MB, and/or the Linux default huge page size.

5. PAM

If you are using UNIX authentication for GemStone logins, or if you run NetLDI as root with setuid (i.e. not in guest mode), you must have PAM (Pluggable Authentication Module) configured on the server. You may include a specific GemStone authorization service name, or allow the default “other” authentication definitions to be used.

PAM authentication definitions are in files under the directory /etc/pam.d. Alternatively, they can be lines in the configuration file /etc/pam.conf, but this usage is deprecated on many distributions. On these distributions, the presence of the /etc/pam.d directory will cause /etc/pam.conf to be ignored.

The specific GemStone service file names are gemstone.gem for user authentication, and gemstone.netldi for a NetLDI running with authentication.

The libraries that are specified in the stack depend on how you are configuring PAM to perform the authentication. The examples below are for PAM configured to invoke LDAP for authentication.

For GemStone UNIX authentication, which uses PAM, to authenticate via LDAP, create a file named /etc/pam.d/gemstone.gem with the following contents:

auth	required	pam_ldap.so

For NetLDI authentication, again using LDAP, create a file named /etc/pam.d/gemstone.netldi with the following contents:

auth	required	pam_ldap.so

Red Hat, by default, installs a file /etc/pam.d/other which disables “other” authentication. On Ubuntu, it is enabled by default. You can allow the “other” authentication stack to be used for GemStone authentication by ensuring that the file /etc/pam.d/other has the following contents:

auth	required	pam_ldap.so

Consult your System Administrators for more information on how authentication is handled on your system.

6. System clock

The system clock must be set to the correct time. When GemStone opens the repository at startup, it compares the current system time with the recorded checkpoint times; a system time earlier than the last checkpoint time may be taken as an indication of corrupted data and prevent GemStone from starting. The time comparisons use GMT.

7. TCP keepalive option

GemStone processes ordinarily use the TCP keepalive option to determine how long they will wait after communications activity ceases unexpectedly. This setting can be useful for reaping stale RPC Gems, but the OS default may not be appropriate for your system. For further information, refer to your operating system documentation.

8. /etc/systemd/logind.conf must disable RemoveIPC

To avoid Stone shutdown when the session that started the Stone logs out, ensure that

/etc/systemd/logind.conf

includes the line

RemoveIPC=no

With the default value of yes, when the session that started the Stone logs out, the Stone's shared semaphore array is deleted, causing fatal errors.

9. statmonitor additional permission

If you are running with the NetLDI owned as root with the s bit set, then statmonitor running as an ordinary user, including as the administrative user, will not be able to collect certain memory statistics for Gems started by other users.

• You can enable statmonitor to still collect these statistics by giving it cap_sys_ptrace.

os$ setcap cap_sys_ptrace=pe $GEMSTONE/bin/statmonitor

• Alternatively, statmonitor can be run as root with s bit set:

os$ cd $GEMSTONE/sys

os$ chown root $GEMSTONE/bin/statmonitor

os$ chmod u+s $GEMSTONE/bin/statmonitor

If you have set the Linux capacity cap_ipc_lock for shrpcmonitor, for huge pages or locking (this is permitted but not recommended), statmonitor will not be able to collect memory statistics for the shared page cache monitor, and the above workarounds may ba applied. The cap_sys_resource capability, for OOM protection, is dropped after the process has adjusted its oom_score_adj, and is not an issue.

10. Unset LD_BIND_NOW

On some Linux distributions, setting the environment variable LD_BIND_NOW may result in process startup failures due to loading incorrect shared libraries.

Licensed Customer key file

Keyfiles for GemStone version 3.6.x and earlier are not valid with v3.7. Licenced customers can email keyfiles@gemtalksystems.com or contact GemTalk Technical Support to request a keyfile for version 3.7 for their platform or platforms.

Community key file

The GemStone distribution includes a community key file, community.starter.key., with product and system limits per the Community and Web Edition License. See https://gemtalksystems.com/licensing for details on the license terms.

If you do not install a custom keyfile, this starter keyfile will be used instead.

Installing a keyfile

To specify the location and name of the keyfile using the KEYFILE configuration parameter, edit the configuration file that will be used by the v3.7 stone to include the location and name of the keyfile.

You may also put the keyfile in the default location, $GEMSTONE/sys/gemstone.key. This requires modifying the write permissions of the $GEMSTONE/sys directory; ensure you change this back to not writable, after this update.