1. Administration of the GemStone/S Environment

Figure 1.1   The GemStone Object Server

Keys parts that define the server configuration include:

Server Processes

• The Stone repository monitor process acts as a resource coordinator. It synchronizes critical repository activities and ensures repository consistency.
• The shared page cache (SPC) is a shared memory segment that holds the pages on which data and metainformation are stored. Pages are read into frames in the cache. Pages are read from the disk extent files into frames, and new pages are allocated and held in frames. A larger cache provides better performance since more frames can hold a larger percent of the repository data in memory. The cache may be sized to be large enough to hold all data in the extents.
• The shared page cache monitor process creates and maintains the shared page cache. The monitor balances page allocation among processes, ensuring that a few users or large objects do not monopolize the cache.
• The Admin Gem performs administrative garbage collection tasks.
• The Reclaim Gem performs reclaim, cleaning up old versions of objects and dead objects, so that the pages can be reused.
• The Symbol Gem is responsible for creating all new Symbols, based on session requests that are managed by the Stone.

Disk-based Persistence

• Objects are stored on disk in one or more extents, which can be files in the file system, data in raw partitions, or a mixture.
• Transaction logs permit recovery of committed data if a system crash occurs, and in full logging mode allows transaction logs to be used with GemStone backups for full recovery of committed transactions in case of disk failure.

Session Processes

• Gem sessions can be Gem processes in an RPC (Remote Procedure Call) login, or bound with the client application using shared libraries in a linked login.

The terms “Gem” and “session” are both used to refer to the logged-in Gem, although this may be either a Gem process that is separate from the application process, or a linked application process that also contains the Gem.

Communications process

• The NetLDI (Network Long Distance Information) listens on a configured port for connections, to establish logins and perform other tasks in distributed systems.

Stone

A configured GemStone server is started by the startstone command, which starts the Stone repository monitor. The Stone in turn starts the shared page cache monitor and other server processes. The extent files are attached, and a transaction log opened for writing.

The Stone is named; by default, gs64stone. Only one Stone with a given name can run on a particular node.

You may run more than one Stone on a node, as long as names are different and the extent files and transaction logs are in different locations or have different names.

Shared Page Cache

The GemStone shared page cache system has two parts: the shared page cache itself, a shared memory segment; and a monitor process, the shared page cache monitor (shrpcmonitor). Figure 1.2 shows the connections between these and other components on a single node.

The shared page cache resides in a segment of the operating system’s virtual memory that is available to any authorized process. When the Stone or a Gem session process needs to access an object in the repository, it first checks to see whether the page containing that object is already in the cache. If the page is already present, the process reads the object directly from shared memory. If the page is not present, the process reads the page from the disk into the cache, where all of its objects also become available to other processes.

The shared page cache monitor also has a name, which is derived from the name of the Stone repository monitor and the Host Identifier; for instance, gs64stone~d7e2174792b1f787.

Each Stone has a single shared page cache on its own node, and may have remote page caches on other nodes in distributed configurations (discussed in detail in Chapter 5). The Stone spawns the shared page cache monitor automatically during startup, and the shared page cache monitor creates the shared memory region and allocates the semaphores for the system. All sessions that log into this Stone connect to this shared page cache and monitor process.

Figure 1.2   Pages in the Shared Page Cache

---

NetLDI

A NetLDI also must be configured and explicitly started using the startnetldi command. While there are some cases, in which there are only linked logins, that do not require a NetLDI, most installations will need to start a NetLDI.

Each NetLDI has a name, but may also be referred to by its listening port number. The default name is gs64ldi; by using this name and setting this up in the services database on all your nodes, you can skip the step of specifying the NetLDI.

You may run more than one NetLDI on a node, related to the same GemStone system or an entirely different system, as long as the names and ports are different.

If you are using more than one NetLDI on a particular node, and one of them is running with the default name, use particular care to ensure that all logins contact the correct NetLDI.

Logging in Gem Sessions

Client applications include:

• the topaz command line tool, provided as part of the server
• client Smalltalk applications such, as VisualWorks and VA Smalltalk using GemBuilder for Smalltalk (GBS)
• client Java applications using GemBuilder for Java (GBJ)
• clients using open-source products such as GsDevKit and Rowan for GemStone
• web applications
• other custom GCI application

Client application are configured to load the GCI shared library, provided as a .dll, .dynlib, or .so file, containing the Gem API.

The login commands include, of course, GemStone authorization details. The specification of the Stone and node must be provided, and the location where the Gem process is to be run. The details need to include the name or port of the NetLDI that sets up the login.

The NetLDI uses that information to fork the Gem process, trigger the startup of any other required server processes, and setup the interprocess communications.

NRS (Network Resource String)

GemStone’s native NRS syntax is used to specify the name and location of each part of the GemStone system. These are widely used in utility commands and in login parameters, to specify the location and name of the Stone and NetLDI, and where the other GemStone processes should run.

The most commonly used syntax, as in the examples in this manual, has the form:

!@nodeNameOrId#netldi:netldiNameOrPort!stoneOrGemService

• The “@nodeNameOrId” can be omitted for the local node.
• the “#netldi:netldiNameOrPort” can be omitted if the NetLDI is running with the default name gs64ldi, and the name and port are configured in the services database.
• stoneOrGemService is the name of the running Stone, or gemnetobject or another Gem service. If you omit both node and netldi, you do not need the ! dividers; you can use only stoneOrGemService.

GemStone UserIds and login

Logins to GemStone are done as GemStone users, which are instances of UserProfile. UserProfiles have names and passwords that are unrelated to UNIX userIds, although GemStone UserProfiles can be created that correspond to the UNIX account names.

Login authentication can be configured to use UNIX or LDAP, in which case either the names must match, or the UserProfile configured with the mapping. UserProfiles can also be configured with single sign-on using Kerberos, by setting up the appropriate mapping within GemStone.

There are several built-in system accounts, including SystemUser, which is similar to a “root” user, used for upgrades; and DataCurator, the administrative user for tasks such as backups and user administration. Other system accounts are used for garbage collection and symbol creation.

When GemStone UserProfile authentication is controlled by GemStone, there are a number of ways to restrict password choice, require password changes, and disable accounts in cases such as too many failed logins. Internally, passwords are stored only in the encrypted form and there is no facility to decrypt them.

A number of important operations, such as code modification and garbage collection, may be restricted to certain users by using privileges. A UserProfile may be granted privileges that allow them to perform these operations.

Authorization to Access Data

Within GemStone, Objects are associated with Security Policies, that allow specific GemStone Users or Groups of users to write, read, or have no access to particular objects. This provides a highly granular way of protecting critical data.

File access and authorization

As a multiuser system, GemStone must allow applications running under different UNIX userIds to create processes that access the repository disk files, while ensuring that the disk files are protected against unauthorized access or modification, both intentional and accidental. This involves managing the permissions for important files and controlling the process owners and groups. Configuring security is described in Chapter 4.

Initial login requests to GemStone connect using SSL (Secure Socket Layer; more specifically, TLS protocol using OpenSSL). For processes on the same machine, communications after the login continue using primarily shared memory. Remote socket connections may be SSL or not, depending on your security requirements.

Object snapshot views

On login, each GemStone user acquires a “view” of the objects in the repository (equivalent to an DBMS term snapshot). This snapshot view is maintained as long as you are using it, even if the objects have been modified by other users. You may make modifications to the objects in your snapshot view, or create new objects, but these changes are transient unless and until you commit, in which case they become persistent and can be viewed by other users.

Making new objects persistent also requires that they be connected to an existing persistent object. Objects that are not reachable by other objects are subject to garbage collection.

If another user has made a conflicting change, your commit may fail, and you can get a report of the conflicts. You may need to discard the changes. This can be avoided by locking objects prior to modifying them; reduced-conflict classes allows certain kinds of conflict to be automatically resolved.

Session transactional state

Each Gem session is either “in transaction”, not in transaction, or in “transactionless” state (transactionless is a specialized state designed for idle sessions; most behavior described in this manual is concerned with sessions that are either in or out of transaction.

Sessions that are in transaction may commit changes. Sessions that are not in transaction can view data and modify objects, but these modifications are transient. To make persistent changes, you must begin a transaction, make the changes, and commit successfully.

Both sessions in and not in transaction can abort. An abort discards modifications. When a session commits or aborts, its snapshot view is updated to the most recent one, including any changes by other users.

If a session attempts to commit changes that conflict with changes made by another session, the commit will fail.

Commit records

GemStone is designed to accommodate large numbers of users. Each user may have his or her own snapshot view of the repository; and each of these snapshot views must be maintained as long as they are in use. GemStone maintains these views as Commit Records.

Over time, each of these sessions will commit changes (or abort), at which point the previous snapshot view is no longer required. However, since commit records are sets of changes from a previous snapshot view, GemStone must maintain the commit record of the session that has been logged in without commit or abort for the longest time (the oldest commit record) and every intermediate commit record up to the current one.

Commit records are stored in the repository, and a large commit record backlog can use a large amount of repository space. GemStone can signal sessions that are causing a backlog, but applications in multiuser system should be designed to abort or commit and respond to signals to avoid creating backlogs. A long-lasting commit record backlog can fill up all disk space such that the GemStone cannot avoid shutting down.

GemStone Installation

The GemStone installation process, as described in the Installation Guide for your platform, describes both how to configure your operating system, and the details of installing GemStone.

After installation, you should have a directory containing the executables, shared libraries, extents and other required or useful files. Not all are required, and they do not need to be located in this shared directory structure.

The GemStone installation directory is generally referenced by the environment variable $GEMSTONE. You will usually need to have this environment variable defined, as well as having the $GEMSTONE/bin directory on your machine search path.

Lock file directory

In addition to the normal installation directory, GemStone requires access to two directories under /opt/gemstone/:

• /opt/gemstone/locks is used for the hostId that uniquely identifies this node, and for lock files, which among other things provide the names, PIDs, ports, and other important data for GemStone processes, used in interprocess communication and reported by gslist.

Lock files (processName..LCK) are normally deleted when the GemStone process exits. To clear out lock files of processes that exited abnormally, use gslist -c.

If there is an unexpected shutdown, the lock files remain in the /opt/gemstone/locks/ directory. On restart, it is possible for a kernel process to reuse this PID. If the owner is root, GemStone cannot reliably determine the status of the process and thus cannot safely delete the lock file. These lock files must be manually deleted.

It is recommended that systems be setup so that on boot, the lock files in /opt/gemstone/locks/*.LCK are deleted automatically on system restart.

Do not delete /opt/gemstone/locks/gemstone.hostid.

• /opt/gemstone/log is the default location for NetLDI log files, if startnetldi does not explicitly specify a location using the -l option.

If /opt/gemstone/ does not exist, GemStone may use /usr/gemstone/ instead.

Alternatively, you can use the environment variable GEMSTONE_GLOBAL_DIR to specify a different location. Since the files in this location control visibility of GemStone processes to one another, all GemStone processes that interact must use the same directory.

Extents, Tranlogs, and disk space

GemStone data is preserved on disk in one or more extent files. These extents include the classes and objects that make up GemStone, as well as the classes and objects that make up your application. When GemStone is shutdown, these extents provide the complete set of objects that compose the application.

Each time a session commits changes to the repository, the changes are made on pages in the shared page cache, which are eventually written to disk by the AIO page server. A commit also writes a record in the current transaction log, which is immediately written to disk.

While GemStone is running, it periodically makes a checkpoint, at which point all dirty pages in the shared page cache are written to disk, and the root page, which holds critical global information on the repository, is updated to a new consistent state. GemStone also writes a checkpoint on an orderly shutdown, and before a backup.

Between checkpoints, updates to the repository are recorded in the transaction logs. If GemStone shuts down unexpectedly, it will startup at the point of the most recent checkpoint in the repository. Changes since that checkpoint are restored from the transaction logs.

Disk Usage for Extents and Transaction Logs

On a single disk, multiple processes writing to both extents and transaction logs will encounter contention, since only one physical write to disk media at a time is possible.

For this reason, if you are not using a SAN or a RAID device that avoids single disk contention, using multiple physical disk drives is recommended for better performance.

Using SSD devices will provide the best performance.

Process Log Files

Each GemStone process creates or appends to a log file. These log files include process details, startup configuration information, and messages about any errors that occur. The location and file name of most processes is configurable.

When the process exits, by default some processes leave their log files in place for possible later diagnostic use. Other process types delete their log files on a clean exit. A process that exits with a error never deletes its log file.

Note that linked sessions, which do not have an independent Gem process, do not create log files. Log file output is sent to stdout for the client application.

System Clock and GemStone times

The system clock should be set to the correct time. When GemStone opens the repository at startup, it compares the current system time with the recorded checkpoint times as part of a consistency check. A system time earlier than the time at which the last checkpoint was written may be taken as an indication of corrupted data and prevent GemStone from starting.

Configuration files

GemStone uses configuration files to hold the specifications for most important configuration details. This includes the names and locations of the extent files, the sizes of the various caches, and many parameters designed for tuning. While there are many parameters, only a few of them are required; in most cases the default is sufficient.

On a new Gemstone system, you must determine the basic configuration before starting. The details on how to establish the configuration are described in Chapter 2, “Configuring the GemStone Server” for the server and Chapter 3, “Configuring Gem Session Processes” for Gems.

GemStone distinguishes between the system configuration file, which applies to the Stone and other system processes, and application configuration files, which apply for Gem sessions (linked and RPC). If all GemStone processes are on a single node, the parameters can all be in a single configuration file. Multiple files can be setup, which apply to different tasks or reside on different nodes.

Environment variables

There are a number of environment variables that GemStone uses. The most important of these is $GEMSTONE, which is required for administration, and indicates the directory in which GemStone is installed.

Other environment variables provide file locations for configuration files themselves, and other information that is needed before a process is able to read the configuration file.

Finally, some logging and debugging settings are provided as environment variables to allow debugging without affecting other sessions in a multiuser system.

Utility command arguments

When GemStone is started up, the startup tasks are performed by utility commands, such as startstone to start the stone. These command-line tools accept optional arguments, including such things as process names, port numbers, and log file locations.

Run-time configurations

Some configuration details are fixed at startup, so making changes requires stopping and restarting the Stone (which stops all other processes except the NetLDI) or Gem (which means logging out). However, other configuration settings can be modified at runtime by executing Smalltalk code.

Example Configurations

As an example of how the scale of application characteristics affect key GemStone configuration settings, the following table provides some examples of “average” configurations.

While these may be useful in planning, the actual values will be based on your particular hardware and application requirements, and most systems will require additional tuning for optimal performance. GemStone can generate performance statistics, both during development and in production systems. These statistics can be displayed and analyzed using the VSD (Visual Statistics Display) utility to understand bottlenecks and determine both changes that are needed, and to study the effect of configuration and application changes.

GemStone Professional Services can provide expert assistance in establishing your configuration, tuning configurations for performance, and ensuring your configuration can accommodate predicted growth.

Recovery vs. Restore

Provided whatever caused the error has been corrected and there is no file damage, a Stone startup following an unexpected shutdown will automatically read the transaction log entries that follow the last checkpoint recorded in the repository, and replay these transactions, recovering the GemStone repository with all committed transactions at the point of shutdown. This is termed “recovery”.

If the extent disk files are damaged, you may need to restore from backup. This involves using a previously-made backup, and replaying all transaction logs that were generated since the backup was made. For this to succeed, you must have a complete set of backup files, and each of the transaction logs that were generated since the time that the backup was started.

Developing a Failover Strategy

The particular strategy to use to protect your data, depends on what the tolerance for data loss is, how soon your application must be available again following an unexpected outage, and the resources available to support a strategy.

Making regular backups and using full transaction logging mode provides the minimal amount of reliability.

Assuming that backups are kept on a separate disk, mirroring the transaction logs (using OS-level tools) avoids the risk of a disk failure causing loss of committed transactions.

A warm or hot standby system can allow the fastest failover, by keeping a secondary system running at all times. In this case, only a minimal amount of further restore is needed before the standby is usable.