This chapter describes the procedure for installing GemStone/S 64 Bitâ„¢ version 3.7.2, and for additional OS configurations that can optimize your GemStone application.
If you are upgrading to this release from a previous version of GemStone/S 64 Bit, after installing v3.7.2, follow the instructions in the appropriate later chapter of this Installation Guide. These upgrade instructions will provide details on steps that need to be taken before and after the installation described here.
If you are installing using RPM (Red Hat Package Manager) on a Red Hat-compatible machine, the installation instructions are somewhat different; see the separate document Installation Guide for LinuxRPM for v3.7.2.
The following list summarizes the steps to install GemStone/S 64 Bit.
Before you install GemStone/S 64 Bit, ensure that the following system requirements are satisfied. Additional system resources may be necessary to support large applications.
GemStone/S 64 Bit v3.7 and later will not run on x86_64 CPUs older than listed. These versions are built using machine instructions that are not available on older CPUs. v3.7.2 cannot run on these CPUs, regardless of the Linux OS version.
Extent files cannot be accessed via CIFS, and NFS requires special configuration.
GemStone performs testing on a mixture of Red Hat, CentOS, and Rocky servers; these are all considered fully certified platforms. Any reference to Red Hat applies to any Red Hat-compatible distribution.
Note that you must also have a supported CPU; GemStone/S 64 Bit v3.7.2 will not run on CPUs older than listed under Platform.
A C debugger allows C-level stack traces when a GemStone error occurs, or when using the pstack command. While not required for GemStone execution, it is strongly recommended that a debugger be installed, so diagnostic error stacks will be available.
GemStone requires a C/C++ compiler if you are parsing C header files for creation of FFI interface classes, as described in the Programming Guide, or developing C or C++ code for user actions or a C or C++ application; as described in the GemBuilder for C manual.
If you have committed your FFI generated code, or compiled your user actions or application, the C/C++ compiler is not needed for application execution.
The compiler is required for application execution only when using GsTsExternalSession to access remote Stones running a different version of the GemStone server.
An X Windows server allows you to use GemStone’s graphical VSD application on Linux. Alternatively, GemStone statistical data may be viewed on Windows, by transferring the data files, or by mounting the file system on Windows. X Windows is not required for GemStone execution.
For further OS configuration and optimizations, see Additional Configuration of the Operating System |
The GemStone distribution should be installed as a regular user, not as the root user. Some later installation steps do require login as root. Starting GemStone and and GemStone administration should be done as a regular user, not as root.
For GemStone administration, select an existing user, or create a specific user to be the GemStone administrative user. This administrative user is referred to in these instructions as gsAdminUser.
1. Log in as gsAdminUser (the GemStone administrator) to the machine on which you are installing GemStone. This part of the installation should not be done as root, to ensure all the files are not owned by root.
2. Select the drive on which you will install the GemStone software, and the installation directory on this drive, InstallDir. Make this directory the current working directory. Avoid an NFS-mounted partition or one containing UNIX swap space.
3. GemStone/S 64 Bit is provided as a compressed file, GemStone64Bit3.7.2-x86_64.Linux.zip or GemStone64Bit3.7.2-arm64.Linux.zip. Move this distribution file to the directory location in which GemStone will be installed, InstallDir.
4. Unzip the distribution file using unzip.
$os unzip GemStone64Bit3.7.2-x86_64.Linux.zip
$os unzip GemStone64Bit3.7.2-arm64.Linux.zip
The InstallDir now contains the GemStone distribution in a directory named GemStone64Bit3.7.2-x86_64.Linux or GemStone64Bit3.7.2-arm64.Linux.
The GemStone distribution is now installed.
In addition to subdirectories, the installed GemStone distribution directory also contains the text files version.txt, which identifies this particular product and release of GemStone, and externals.sha.txt, which lists SHAs for external projects included in this version.
After installing the GemStone distribution, you need to set or reset environment variables, ensure you have the appropriate keyfile installed, and that your NetLDI user is configured correctly.
To use GemStone, you will need to define the GEMSTONE environment variable, and in most cases, set the OS search path.
1. The environment variable GEMSTONE should be set to the full pathname (starting with a slash) of your new GemStone installation directory. For example:
os$ export GEMSTONE=InstallDir/GemStone64Bit3.7.2-platform
2. Set your OS search path to include the $GEMSTONE/bin directory.
os$ export PATH=$GEMSTONE/bin:$PATH
3. If this machine has run previous versions of GemStone, or will run multiple versions, it is important to review all GemStone-related environment variables for ones that are incorrect for the new version, such as GEMSTONE_NRS_ALL, GEMSTONE_EXE_CONF, and GEMSTONE_SYS_CONF.
Check for existing GemStone environment variables:
os$ env | grep GEM
If any environment variables exist that are not appropriate for the new installation, you must specifically unset each one.
The GemStone/S distribution includes a community keyfile, which will be used if a keyfile is not specified.
You may use a licenced customer keyfile (that is, a GemTalk-provided keyfile for your license) of the correct platform from any v3.7.x version with v3.7.2; keyfiles from 3.6.x and earlier are not valid with v3.7.2.
A keyfile must be located where GemStone can find it on startup, in order:
You may continue to use your v3.7.x keyfile with version 3.7.2. If you are upgrading from v3.6.x or earlier and need a new keyfile, or you have questions about your keyfile or license limits, email keyfiles@gemtalksystems.com, or contact GemTalk Technical Support.
The GemStone distribution includes a community keyfile, community.starter.key, with product and system limits per the Community and Web Edition License. See https://gemtalksystems.com/licensing for details on the license terms.
The configuregs and installgs scripts offer to install a keyfile for you.
To specify the location and name of a customized keyfile using the KEYFILE configuration parameter, edit the configuration file that will be used by the v3.7.2 stone to include the location and name of the keyfile.
You may also put the keyfile in the default location, $GEMSTONE/sys/gemstone.key. This requires adding write permission to the $GEMSTONE/sys directory; ensure that you remove write permission after the update.
The NetLDI permits remote processes to interact with the repository. A NetLDI is required for some local and all remote sessions to log into GemStone. Logins can specify a NetLDI by name or by port number.
Defining a NetLDI service name is only necessary if you will use named NetLDIs. If you start a NetLDI using a port number, then no configuration of the system services database is needed. The login parameters can use the NetLDI’s port rather than a name.
If you are using named NetLDIs, and your configuration includes multiple machines, (including clients on Windows or other platforms), you must update the services database on each node with an entry mapping the NetLDI name to the same port number.
1. Determine how the services file is setup on your system. If your system is using /etc/services, the installgs script can setup a named NetLDI for you.Otherwise, consult your system administrators.
2. Determine whether a service for the NetLDI name, such as gs64ldi, is already defined. The GemStone distribution includes an executable that will allow you to check for a definition:
os$ $GEMSTONE/install/getservbyname gs64ldi
s_name=gs64ldi s_port = 50377 s_proto = tcp
3. If your desired NetLDI name is not defined, add an entry similar to the following to the system services database:
gs64ldi 50377/tcp #GemStone/S 64 Bit 3.7.2
Choose a port number that is not being used by another service. The port number should be in the range 49152 <= port <= 65535, to conform to IANA standards (http://www.iana.org/assignments/port-numbers).
If several machines will be running GemStone, the system services database must be updated for each machine, including Windows clients as well as UNIX nodes. The port number must be the same for every machine.
If you are installing GemStone on a machine that has not previously had a version of GemStone installed, running configuregs (or the equivalent manual process) is required to perform the setup of the /opt/gemstone/locks and log directories. These directories must exist and be accessible in order to run GemStone.
If the /opt/gemstone/locks and log directories already exist, running configuregs is optional.
The configuregs script must be executed with root permissions; either logged in as root, or using sudo. the $GEMSTONE environment variable must be defined.
os$ sudo -E $GEMSTONE/install/configuregs
This has the following options:
The files in the installation will be chowned to be owned by this user. Enter gsAdminUser.
The files in the installation will be chgrouped to this group. Depending on your NetLDI configuration, all users that will use GemStone should be in this group.
Enter the path to a keyfile, if you have a keyfile you wish to install. If you leave this empty (the default), by pressing enter, the Stone may be started using the community key file, or you may install your own key file later. If you supply the path to a keyfile, this is renamed and copied, to $GEMSTONE/sys/gemstone.key.
configuregs can add the NetLDI name and port number to /etc/services. Unless you are familiar with this step and know your port lookup goes through /etc/services, use the default n.
configuregs can add Out-of-Memory killer protection to key GemStone process. If you answer the default y, configuregs will add the cap_sys_resource capability to certain executables.
On some Linux distributions, the default for RemoveIPC is yes, which can result in the GemStone repository dying with an error when the user that started GemStone logs out. To be safe, we recommend explicitly setting RemoveIPC=no in /etc/systemd/logind.conf. Answering yes to this question will add the line to this file, unless it is already present.
configuregs writes a file answers.txt in the $GEMSTONE/data directory. This is a plain-text that can be edited if desired, and used to repeat the operations done by configuregs without re-answering each question, if you are installing on a different host or on a new GemStone release.
os$ sudo -E $GEMSTONE/install/doconfiguregs /var/tmp/gsanswers.txt
After configuregs completes, the GemStone distribution is ready to use.
The legacy installgs script verifies your environment, creates lock and log file directories, sets up the extent files, and can be used to configure certain types of security for multi-user systems.
For new users and for test and development systems without strict security requirements, it is recommended to use the configuregs script (here), and not running installgs.
Running installgs is needed if you want to configure the NetLDI to run in root mode with the s bit set.
The installgs script must be executed with root permissions; either logged in as root, or using sudo. the $GEMSTONE environment variable must be defined.
os$ cd $GEMSTONE/install
os$ sudo ./installgs
This has the following options:
The default location for server lock files and NetLDI log files is /opt/gemstone. If the directory does not exist, the installation script asks a number of questions allowing you to create /opt/gemstone and the subdirectories locks and log, and to set access (770) to these directories.
If you answer yes, the script will prompt you for the owner and group you want to use. Refer to Chapter 1 of the System Administration Guide for more information about setting owner and group permissions.
If you answer no, the permissions will remain the same as when the files were extracted from the distribution media.
The default gives only the owner read and write access (600) through ordinary UNIX commands. Other users can read and write the repository extent through a GemStone session. If you choose not to protect the repository, the executables run under ownership of the user who invokes them.
Default: Set the repository permission to 600, and leave the setuid bit applied.
The NetLDI permits remote processes to interact with the repository. There are two ways to set up a NetLDI so that it can provide services to all GemStone users: it can run as root, or it can run in guest mode with a captive account.
To run NetLDIs as root, accept the default “yes” response. Ownership of the NetLDI executable is changed to root, and the setuid bit is set. Any GemStone user will be able to start a NetLDI process that is accessible to all GemStone users because it will always run as root. For certain services, users will need to authenticate themselves by supplying a password.
To run NetLDIs in guest mode with a captive account, answer “no” to the prompt, because those modes are not permitted if the NetLDI runs as root. “Guest mode” means that GemStone users do not have to supply a UNIX password to use NetLDI services. The “captive account” is an account that owns all processes the NetLDI starts; typically, it is the GemStone administrative account that owns the files. You must start the NetLDI while logged in as that account.
If you are running a development system without strict security requirements, we recommend not using the default.
Default: Change ownership of the netldi executable to root, and set its setuid bit.
installgs offers to set up a new GemStone repository in $GEMSTONE/data. This is only appropriate if you wish to create a new repository. It performs the equivalent of the setup described in .
Default: Place a writable copy of extent0.dbf in $GEMSTONE/data.
installgs offers to start a NetLDI that is running in root with s-bit set; this should only be used if you have answered yes to “Allow NetLDI to Run as Root?”.
Default: Do not start a NetLDI at this time.
After running installgs, log out as user root. Further work is done as the GemStone administrative user.
GemStone/S 64 Bit v3.7.2 is now installed and configured. |
There are a number of Operating System-level configurations that can affect your GemStone environment. This includes a number of settings that are important for very large applications, but are not needed for simple test installations.
Modern Linux installations default to settings that support most GemStone configurations; there is no longer a need to set kernel.shmall, kernel.shmmax, kernel.sem, or fs.file-max. The default limits on modern Linux kernels are sufficiently large even for very large GemStone configurations.
Setup of OOM Killer protection is done by the configuregs script. You do not need to take further action if you answered [y] to that question when running configuregs.
If your system runs low on memory, the Linux OOM killer may select GemStone processes to terminate. To protect the shared page cache and other critical GemStone processes, the process’s oom_score_adj, which is used to select processes to terminate, is reduced, making them less likely to be selected and killed.
Reducing the oom_score_adj requires configuring the process to have that capability. To set the capability on the required executables:
os$ setcap cap_sys_resource=pe $GEMSTONE/sys/stoned
os$ setcap cap_sys_resource=pe $GEMSTONE/sys/pgsvrmain
Only the Stone and pgsvrmain executables need the capability; other critical processes spawned by the Stone, such as the Shared Page Cache Monitor, inherit the Stone’s protection and adjust their oom_score_adj, and therefore also are protected. The capability is dropped as soon as the oom_score_adj has been updated.
Likewise, the pgsvrmain, which is responsible for spawning critical processes for distributed configurations with remote caches, protects processes such as the remote cache.
To examine the oom_score_adj for a process, check the contents of the file:
os$ cat /proc/pid/oom_score_adj
The default size for memory pages on Linux is 4KB. Linux supports 2MB and 1GB Huge Pages; this document, and GemStone code, also refers to these page sizes and all larger-than-default memory pages using the generic term large pages. Using large pages will improve performance for large repositories with large shared page caches.
The Linux kernel allows you to explicitly reserve 2MB or 1GB large pages for a specific application (such as GemStone), and also allows you to take advantage of 2MB transparent large pages, which are managed automatically without application level configuration.
Reserved huge memory pages are described in the next section, Reserved Huge Memory Pages. These are recommended for the Shared page Cache. Transparent huge memory pages improve performance for Gems with large temporary object memory.
Transparent huge pages are limited to anonymous memory regions. If they are enabled (they are enabled by default), transparent huge pages are managed by the kernel without application level configuration.
In general we recommend settings of "madvise" or "advise", which allow GemStone to use transparent huge pages. Settings of "always" will allow GemStone to use transparent huge pages, but may affect overall performance either positively or negatively.
To confirm these settings are correct in your Linux environment, cat the file; the enabled parameter is within the square brackets. E.g.,
os$ cat /sys/kernel/mm/transparent_hugepage/enabled
always [madvise] never
Linux supports explicit configuration for 2MB and 1GB huge pages that are specifically reserved for an application, as well as transparent huge pages (as described in the previous section).
A Linux installation may be configured to use either 2MB and 1GB large pages, or both. Using large pages improves performance for large repositories with large shared page caches; by explicitly configuring large pages, they are allocated on boot and the GemStone executable can be explicitly configured to use them.
To use explicitly configured large pages, you must determine the page size and the number of pages needed, configure Linux to allocate the required number of pages, and configure GemStone to use these pages.
To configure the use of large pages:
a. Determine the Linux Page Size
From the page sizes that are available on your Linux system, you must decide on the size you intend to use.
To determine if your CPU supports 1 GB huge pages, execute:
os$ cat /proc/cpuinfo | grep -o pdpe1gb | head -1
b. Determine the required number of Linux Pages
Calculate the number of huge pages that will be needed, based on your GemStone configuration. This is calculated by the utility largememorypages. This utility needs several details to compute the number of required pages: the shared page cache size, the maximum number of GemStone processes, and the number of shared counters. These can be provided by arguments or read from an existing configuration file. When using a configuration file, the file must have large memory pages (SHR_PAGE_CACHE_LARGE_MEMORY_PAGE_POLICY) enabled, and page size (SHR_PAGE_CACHE_LARGE_MEMORY_PAGE_SIZE_MB) set, if the calculated default page size is not correct.
The largememorypages computation by default is for the Stone’s cache. When configuring large pages for remote cache, the values are somewhat smaller. Use the -r option to compute large page requirements for a remote cache.
You may specify the required values using an existing configuration file, or by passing in the relevant values as argument. A configuration file is parsed as a Stone configuration file and the necessary values extracted; the configuration file must have large memory pages enabled.
If a configuration file is not specified, then -M or -F, and -p, -P and -C are required. These arguments can be used in addition to the configuration file, in which case they override values set or computed in the configuration file:
largememorypages [-e path ][-z path][-r][-F cacheFrames | -M cacheKB]
[-P maxProcesses][-C maxSharedCounters][-p largeMemoryPageSize]
-e path
specifies an executable configuration file (same as startstone -e).
-z path
specifies a system configuration file (same as startstone -z).
-F cacheFrames
shared cache size expressed in 16 KB frames
-M cacheKB
shared cache size with a optional units suffix.
-P maxProcesses
setting for SHR_PAGE_CACHE_NUM_PROCS
-C maxSharedCounters
setting for SHR_PAGE_CACHE_NUM_SHARED_COUNTERS.
-p largeMemoryPageSize
setting for SHR_PAGE_CACHE_LARGE_MEMORY_PAGE_SIZE_MB. Options are 2MB and 1GB (or equivalent using appropriate units).
The following example is for an (approximately) 20GB shared page cache, 200 processes, and 1900 shared counters, with 1 GB memory pages:
os$ largememorypages -p 1GB -M 20GB -C 1900 -P 200
Cache kind is: Primary SPC (stone shared page cache)
Cache config is 1373248 pages = 21457 MB, total is 22528
MB, overhead 4% of configured size
HashTableSize 2097152 TotalPces 3483008 freeListPces
1385856
Large page size requested is: 1024 MB.
Large page overhead: 0.09 MB
[Info]: Increasing number of pages from 1310720 to 1373248
to reduce large memory page overhead.
For 1373248 pages, 200 processes and 1900 shared counters, 0
pusherThreads
minimum sizing for cache shmmax 23622320128, shmall
5767168.
Number of 1024 MB large pages required: 22
<with further update commands>
c. Allow large memory pages to be allocated and accessed by GemStone processes
To ensure processes have permission to use large memory pages, you need to do one of the following. You must execute these commands as root.
Select or create a group that includes the Linux user that will be starting up the shared page cache, and all users who will be accessing the shared page cache
Then create a file /etc/sysctl.d/64-gemstone-local.conf, to set this group’s numeric id; add the line:
vm.hugetlb_shm_group = numericGidOfGroup
To determine the group id based on the group name, execute
os$ getent group nameOfGroup
This will take effect on reboot; you can apply immediately by executing:
os$ sysctl --system
To verify the current value, cat /proc/sys/vm/hugetlb_shm_group, which will show the group id. To enable large pages transiently without rebooting, you can set the gid of the group directly in /proc/sys/vm/hugetlb_shm_group. This will be reset on reboot if not configured as by /etc/sysctl.d/64-gemstone-local.conf.
os$ setcap cap_ipc_lock=pe $GEMSTONE/sys/shrpcmonitor
Setting this capability will prevent statmonitor and gdb from attaching to that process; statmonitor will not be able to record certain process statistics, and pstack cannot get C stack traces. See statmonitor additional permission.
d. Configure large pages on Linux
Linux must be configured according to the instructions for your Linux distribution. You must execute these commands as root.
(1) To enable hugepages, the file /etc/default/grub needs to contain an entry for GRUB_CMDLINE_LINUX_DEFAULT that defines the number of 2MB and 1 GB huge pages.
Edit the file /etc/default/grub, to append a line defining the number of each size of pages to the GRUB_CMDLINE_LINUX_DEFAULT entry, for example to configure with 1GB pages to support the 20GB cache in the above example:
"hugepagesz=1G hugepages=22 hugepagesz=2M hugepages=0 default_hugepagesz=1G"
(2) Run the command to update the kernel config.
os$ /usr/sbin/update-grub
os$ /usr/sbin/grub2-mkconfig -o /boot/grub2/grub.cfg
On Red Hat and CentOS UEFI systems:
os$ /usr/sbin/grub2-mkconfig -o /boot/efi/EFI/centos/grub.cfg
os$ shutdown -r now
e. Configure GemStone to use large pages
To configure GemStone to request large pages, set the configuration option SHR_PAGE_CACHE_LARGE_MEMORY_PAGE_POLICY. This can be set to 1 or 2; with a setting of 1, the cache will be started anyway if the request for large pages is denied, while a setting of 2 indicates that startup should fail if large pages cannot be allocated.
Setting SHR_PAGE_CACHE_LARGE_MEMORY_PAGE_SIZE_MB to 2 (for 2 MB) or 1024 (for 1GB) is recommended; otherwise, the default is based on the current configuration of 1 GB, 2 MB, and/or the Linux default huge page size.
It is recommended that you have sufficient memory to avoid the need to swap portions of the shared page cache to disk. If there is the risk of swapping, you can avoid problems with GemStone performance (at the expense of other processes running on this machine) by locking the shared page cache into memory. Locking the cache into memory is configured by the parameter SHR_PAGE_CACHE_LOCKED.
If you are using huge pages (Reserved Huge Memory Pages), the cache is inherently locked, so there is no further action needed.
Locking the cache into memory requires that the user starting GemStone has permission to lock the amount of memory required by the cache. There are several ways to do this.
gsAdminUser soft memlock sizeInKB
gsAdminUser hard memlock sizeInKB
Note that after editing this file, you do not need to reboot, but you will need a new shell.
os$ setcap cap_ipc_lock=pe $GEMSTONE/sys/shrpcmonitor
Note that setting this capability will prevent statmonitor and gdb from attaching to shrpcmonitor; statmonitor will not be able to record certain process statistics, and pstack cannot get C stack traces. See statmonitor additional permission.
If you are using UNIX authentication for GemStone logins, or if you run NetLDI as root with setuid (i.e. not in guest mode), you must have PAM (Pluggable Authentication Module) configured on the server. You may include a specific GemStone authorization service name, or allow the default “other” authentication definitions to be used.
PAM authentication definitions are in files under the directory /etc/pam.d. Alternatively, they can be lines in the configuration file /etc/pam.conf, but this usage is deprecated on many distributions. On these distributions, the presence of the /etc/pam.d directory will cause /etc/pam.conf to be ignored.
The specific GemStone service file names are gemstone.gem for user authentication, and gemstone.netldi for a NetLDI running with authentication.
The libraries that are specified in the stack depend on how you are configuring PAM to perform the authentication. The examples below are for PAM configured to invoke LDAP for authentication.
For GemStone UNIX authentication, which uses PAM, to authenticate via LDAP, create a file named /etc/pam.d/gemstone.gem with the following contents:
auth required pam_ldap.so
For NetLDI authentication, again using LDAP, create a file named /etc/pam.d/gemstone.netldi with the following contents:
auth required pam_ldap.so
Red Hat, by default, installs a file /etc/pam.d/other which disables “other” authentication. On Ubuntu, it is enabled by default. You can allow the “other” authentication stack to be used for GemStone authentication by ensuring that the file /etc/pam.d/other has the following contents:
auth required pam_ldap.so
Consult your System Administrators for more information on how authentication is handled on your system.
The system clock must be set to the correct time. When GemStone opens the repository at startup, it compares the current system time with the recorded checkpoint times; a system time earlier than the last checkpoint time may be taken as an indication of corrupted data and prevent GemStone from starting. The time comparisons use GMT.
Disable of RemoveIIPC is done by the configuregs script. You do not need to take further action if you answered [y] to that question when running configuregs.
To avoid Stone shutdown when the session that started the Stone logs out, ensure that
/etc/systemd/logind.conf
RemoveIPC=no
On some platforms/versions, the default value is yes. This means that when the session that started the Stone logs out, the Stone's shared semaphore array is deleted, causing fatal errors. This excluded processes started by sessions running as root or as UNIX users with uids lower than 1000, which are defined as system users.
If you are running with the NetLDI owned as root with the s bit set, then statmonitor running as an ordinary user, including as the administrative user, will not be able to collect certain memory statistics for Gems started by other users.
os$ setcap cap_sys_ptrace=pe $GEMSTONE/bin/statmonitor
os$ cd $GEMSTONE/sys
os$ chown root $GEMSTONE/bin/statmonitor
os$ chmod u+s $GEMSTONE/bin/statmonitor
If you have set the Linux capacity cap_ipc_lock for shrpcmonitor, for huge pages or locking (this is permitted but not recommended), statmonitor will not be able to collect memory statistics for the shared page cache monitor, and the above workarounds may ba applied. The cap_sys_resource capability, for OOM protection, is dropped after the process has adjusted its oom_score_adj, and is not an issue.