5. Connecting Distributed Systems

Distributed configuration with Local Gem

In Figure 5.1, an application communicates with a Gem session process on the server node by way of RPC calls. This configuration lets the Gem execute Smalltalk code in the repository without first bringing complex objects across the network. The Gem directly accesses the shared page cache that was started by the Stone repository monitor. This configuration is considered a “Local” Gem, since the Gem is local to the Stone.

Figure 5.1   Gem Session on Server Node

---

Distributed configuration with Remote Gem

In Figure 5.2, the application and the Gem are linked in a single process that runs on the remote node (i.e, not the Stone’s node). This configuration avoids the overhead of RPC calls, but may increase network traffic if large objects must be brought across the network.

When a Gem is on a remote node, it needs a shared page cache on that node. This is started by the first Gem on that node. Page server processes are also needed to handle pages between the two nodes.

Figure 5.2   Gem Session on Remote Node

---

Network

The nodes in your system can be any combination of GemStone-supported platforms, as long as they have a network connection. Inter-node communication is handled via sockets.

Nodes in which the Stone, or a Gem process, will run must be platforms supported by GemStone/S 64 Bit.

Client-process only nodes such as the remote node in Figure 5.1 can include supported client platforms, such as Windows, as well as server platforms.

GemStone NetLDIs

The NetLDIs are the glue holding a distributed GemStone system together. Each NetLDI spawns other GemStone processes on request, and reports the location of GemStone services on its node to remote processes that must connect to those services.

In a distributed system, each node where a Stone repository monitor or Gem session process runs must have its own NetLDI process running. However, nodes that run only linked applications, or client applications without a Gem on that node, do not require a NetLDI.

The NetLDI listens on a well-known port for requests for RPC logins or other services.

NRS Syntax

GemStone uses Network Resource Strings (NRS) to specify the name and location of each part of the GemStone system. NRS is widely used in utility commands and in login parameters, to specify the location and name of the Stone and NetLDI, and where the other GemStone processes should run.

The most useful parts to know are:

!@node#netldi:netldi!stoneOrGemService

• The “@node” can be omitted for the local node, and you may use an IP address instead of the name of the node.
• the “#netldi:netldi” can be omitted if the netldi is running with the default name gs64ldi and that name is recorded in the services database. The netldi can be specified by name or by port number.
• stoneOrGemService is the name of the running Stone, or gemnetobject or another Gem service. This part is required; if you omit both node and netldi, you do not need the ! dividers; you can use simply stoneOrGemService.

Appendix C, “Network Resource Strings (NRS)” describes NRS in detail.

In particular, you may find it very useful to set the GEMSTONE_NRS_ALL environment variable if you are using a NetLDI name other than the default.

Stone

When logging into to a Stone on the local machine, the stone name is sufficient. If the Stone is on a separate node, you can specify the location of the stone, and the name of the NetLDI on the Stone’s node, using NRS.

For example, to connect to a stone named devstone on the node oboe, with the default-named NetLDI gs64ldi:

topaz> set gemstone !@oboe!devstone

Connecting to the RPC Gem

When a client requests an RPC login, a Gem session process needs to be spawned to fullfil the login request. To do this, the login parameters include the gem service (in topaz, the gemnetid), usually gemnetobject, gemnetdebug, or the name of a custom gem script.

For example,

topaz> set gemnetid gemnetobject

Using gemnetobject alone as here, specifies to run the gem on the same node as the client application.

By specifying an NRS, in the gem service login parameter that includes the node and the NetLDI on which you want the Gem to run, you can instruct GemStone on which processes need to be started and where. This can be the same node as the Stone. It can also be on another node entirely, provided GemStone is installed and a NetLDI is running on that node.

For example, if you intend the gem to run on a machine named lark, with the default-named NetLDI gs64ldi:

topaz> set gemnetid !@lark!gemnetobject

Note that if the gem service NRS includes a node name and the stone NRS does not include a node name, then the Stone is expected to be on the node in the Gem NRS, not on the client node.

Shared Page Cache

There is always a shared page cache on the Stone’s node. Any Gems that are on the Stone’s node (local Gems) will use that shared page cache.

When a GemStone login specifies that a Gem should run on a remote node, then a shared page cache and the associated processes, shared page cache monitor and free frame page server, need to be started on that node. Normally, the cache is started up by a NetLDI running on the remote node, although a linked session (but not an RPC session) can also initiate the startup.

When the remote Gem wants to access a page in the repository, it first checks the shared page cache on its own node. If the page is not found, the Gem communicates with the Remote Gem Page Server on the server node, checking in the shared cache on the server node and if necessary, reading the page from the disk.

Figure 5.3   Shared Page Cache with Remote Gem

---

GemStone Page Servers

Gems on remote nodes require page server processes to support operations between the remote Gem and cache and the Stone.

• When a Gem connects to a repository extent across the network, it requires a Remote Gem Page Server on the Stone’s node, associated with the remote node. This is spawned if it is not already running. This page server looks up pages in the Stone’s cache and performs extent reads, on behalf of the gem. There is one page server per remote node; each Gem on that node is services by a thread within that page server.
• On the remote node, a Remote Cache Page Server supports the Stone in managing the remote shared page cache, by starting the cache and removing pages that are being recycled.

Port use in GemStone

GemStone’s interprocess and internode communications are established via socket connections. A number of key GemStone processes listen on well known ports for a connection, which initiates the handshakes that establish interprocess socket connections.

Processes that listen on well known ports include the NetLDI, the Stone, and the Shared Page Cache Monitor. These named GemStone processes create a serviceName..LCK at startup, in the directory /opt/gemstone/locks or another location as described here. This file encodes the well-known port that the process is listening on. When the NetLDI (for example) contacts a particular Stone, it first looks up the well-known port for that Stone by locating the lock file for that Stone, then contacts the Stone on that port.

The NetLDI’s port is specified on startup in a number of ways, as described here.

The ports used by the Stone and Shared Page Cache Monitor can be specified using STN_WELL_KNOWN_PORT_NUMBER and SHR_WELL_KNOWN_PORT_NUMBER configuration options, respectively. These must be valid port numbers that are not already in use.

Disrupted Communications

Several incidents can disrupt communications between the GemStone server and remote processes in a distributed configuration. This can include network glitches, network overload, crashes or unexpected shut down of a remote process or the entire remote node, or of the network channel itself.

GemStone ordinarily depends on the network protocol keepalive option to detect that a remote process no longer exists because of an unexpected event. The keepalive interval is set globally by the operating system, typically at two hours. When that interval expires, the GemStone process tries to obtain a response from its partner. The parameters governing these attempts also are set by the operating system, with up to 10 attempts in 15 minutes being typical. If no response is received, the local GemStone process acts as if its partner was terminated abnormally. The keepalive interval may need to be adjusted.

Firewalls between GemStone processes require that the communications ports be configured to be open.

Local Gems only

If you are only running Gems on the same node as the Stone, there are only a few steps required on the remote node.

• Install the client shared libraries. This can be done by a complete server installation, or by copying the small number of required library files to the client, and configuring your client search paths appropriately.

Details on how to do this, and the various options, are in the GemStone/S 64 Bit Installation Guide for the client platform.

• If you are using named NetLDIs, you must configure your client system services database to map the same NetLDI name to the same port as defined on the server. If you are using a local services database, edit /etc/services or, on Windows, \WINDOWS\system32\drivers\etc\services to add the appropriate entry.

Remote Gems

Configuring GemStone so that Gem sessions or linked applications can run on the remote node is much the same as configuring the Gem and cache a session process on the server, which is described in Chapter 3, “Configuring Gem Session Processes”.

The remote node will need access to a number of executables and shared libraries within the server installation. You can either repeat the installation from the GemStone distribution media, or mount the directory on the server node that contains $GEMSTONE.

Some additional details points to keep in mind:

• A remote node (on which a Gem is running) must have its kernel configured for shared memory similarly to how it is configured on the primary server node.
• Only nodes running a Stone need a GemStone key file, not nodes running remote sessions.
• It’s best if each node has its own directory for /opt/gemstone/log and /opt/gemstone/locks (or /usr/gemstone/log and /usr/gemstone/locks, or other location under $GEMSTONE_GLOBAL_DIR). If these directories are on an NFS-mounted partition, make sure that two nodes are not using the same directories. Each Stone and NetLDI needs a unique lock file. Shared log files may make it impossible to diagnose problems.
• If you are using named NetLDIs, you must configure your client system services database to map the same NetLDI name to the same port as defined on the server. This is not necessary if you specify NetLDI by port number. See the Installation Guide for details.
• Unless you run the NetLDIs in guest mode with a captive account, users must have an account on, and authorized network access to, all nodes that are part of the GemStone network for the repository they will be using. This includes the nodes on which the Stone Repository monitor and the user’s Gem session process reside.

Unless you run the NetLDIs in guest mode with a captive account, the user who starts the Stone repository monitor ordinarily needs an account on every node where a Gem session process will run.

Configuration Examples

GemStone supports several configurations in which the application communicates with the Gem session process by using remote procedure calls (RPCs), as well as linked applications. This section presents examples that illustrate the processes and interprocess connections within the following distributed applications:

• RPC Application on a Remote Node with Local Gem
• Linked Application on a Remote Node
• RPC Application on a Remote Node with Remote Gem
• RPC Application, Gem, and Stone on Three Nodes
• Distributed System with a Mid-Level Cache

RPC Application on a Remote Node with Local Gem

If the Gem session process is going to run on the server node (as shown in Figure 5.4), an RPC application uses a NetLDI on that node to start a Gem session process. Unless the NetLDI is running in guest mode with a captive account, the application user must provide authentication to the NetLDI. The login parameters include the specification for the Gem network object (gemnetobject). For more information about network objects and how to invoke them, see Connecting to the RPC Gem.

Figure 5.4   RPC Application on a Remote Node with remote Gem

---

To login an RPC session with the Gem on the Stone’s host, set the login parameters as follows:

• Set the Stone’s name:

topaz> set gemstone stoneName

The hostname is optional in this case; by default, it will use the host specified by the NRS in the gem service parameter (gemnetid)

• Set GemStone user name and password:

topaz> set username DataCurator

topaz> set password swordfish

You may omit the password; you will be prompted for it on login.

• Set host authentication, if your NetLDI is not running in guest mode:

topaz> set hostusername unixUserAccountName

topaz> set hostpassword passwordForUnixUserAccount

• Set the gem service, specifying the stone’s host name, and the name of the NetLDI if necessary. For example, either one of the following:

topaz> set gemnetid !@serverhost!gemnetobject

topaz> set gemnetid !@serverhost#netldi:NetLdiName!gemnetobject

Linked Application on a Remote Node

Figure 5.5 shows how a linked application on a remote node communicates with a Stone and repository on the primary server node.

Figure 5.5   Connecting a Linked Application to a Remote Server

---

In this configuration, a remote, subordinate shared page cache is started on the remote node, along with associated server processes. The configuration of this remote cache is set by the first session to log in on this node. After the last session on this node logs out, the cache will shut down.

In addition, this configuration requires a Remote Gem page server on the server node, which reads pages from the repository on behalf of the gem.

Because the shared page cache is readable and writable only by its owner and members of the same group (protection 660), the user running the application may need to belong to that group. See Shared Page Cache.

Set the login parameters as follows:

• Set the Stone’s name, one of the following:

topaz> set gemstone !@serverhost!stoneName

topaz> set gemstone !@serverhost#netldi:NetLdiName!stoneName

• Set GemStone user name and password.

topaz> set username DataCurator

topaz> set password swordfish

You may omit the password; you will be prompted for it on login.

• Set host authentication, if your NetLDI is not running in guest mode:

topaz> set hostusername unixUserAccountName

topaz> set hostpassword passwordForUnixUserAccount

• Do not set the gem service; doing so specifies that the login will be RPC, rather than linked.

RPC Application on a Remote Node with Remote Gem

The configuration shown in Figure 5.6 shows the RPC application and its session process running on the same node. This configuration is similar to a linked application, but in this case, a NetLDI is required on the remote node in order to launch the RPC Gem session process.

Figure 5.6   Starting the Session Process on a Remote Node

---

• Set the Stone’s name, one of the following:

topaz> set gemstone !@serverhost!stoneName

topaz> set gemstone !@serverhost#netldi:NetLdiName!stoneName

You must specify the Stone’s host, since the otherwise it will assume the Stone is on the same node as the Gem, given in the gemnetid’s NRS.

• Set GemStone user name and password.

topaz> set username DataCurator

topaz> set password swordfish

You may omit the password; you will be prompted for it on login.

• Set host authentication, if your NetLDI is not running in guest mode.

topaz> set hostusername unixUserAccountName

topaz> set hostpassword passwordForUnixUserAccount

• Set the gem service, specifying the stone’s host name, and the name of the NetLDI if necessary. For example, one of the following:

topaz> set gemnetid !@remotenode!gemnetobject

topaz> set gemnetid !@remotenode#netldi:NetLdiName!gemnetobject

RPC Application, Gem, and Stone on Three Nodes

The RPC application, session process, and Stone can run on three separate nodes. The application runs on its node and connects to a Gem session process on the Gem’s node, which is sometimes referred to as the Gem Server. That session process communicates with the repository on the primary server node by way of a page server.

Again we see that a NetLDI must be running on each node where part of GemStone executes (but is not required on the application node, which runs only the RPC application).

The following diagram shows two separate RPC applications connecting to Gems running on the same node, the Gem Server node.

Figure 5.7   RPC Applications using a Gem Server Node

---

The directions for this configuration are the same as for the previous configuration, RPC Application on a Remote Node with Remote Gem; however, the application itself is executing on a third node.

• Set the Stone’s name, one of the following:

topaz> set gemstone !@serverhost!stoneName

topaz> set gemstone !@serverhost#netldi:NetLdiName!stoneName

You must specify the Stone’s host, since the otherwise it will assume the Stone is on the same node as the Gem, given in the gemnetid’s NRS.

• Set GemStone user name and password.

topaz> set username DataCurator

topaz> set password swordfish

You may omit the password; you will be prompted for it on login.

• Set host authentication, if your NetLDI is not running in guest mode.

topaz> set hostusername unixUserAccountName

topaz> set hostpassword passwordForUnixUserAccount

• Set the gem service, specifying the stone’s host name, and the name of the NetLDI if necessary. For example, one of the following:

topaz> set gemnetid !@remotenode!gemnetobject

topaz> set gemnetid !@remotenode#netldi:NetLdiName!gemnetobject

Distributed System with a Mid-Level Cache

In a distributed system over a Wide Area Network (WAN), with many remote nodes that are topographically distant from the Stone but close to each other, a mid-level cache can improve performance for the remote sessions. In this configuration, the RPC application and the session process may be on the same or different nodes, with the mid-level cache and Stone running on separate nodes. In Figure 3.8, the Gem session process (on the leaf node) connects to a mid-level cache (on the mid-level cache node). The session process communicates with the repository (on the primary server node) by way of a page server.

When the Gem needs a page but can't find it in its local cache, it first looks in the mid-level cache. If the Gem can't find the page in the mid-level cache, it then forwards the request to the page server on the Stone’s host.

The Stone’s page server aggregates the responses from the page servers on each of the Gem’s shared caches, and returns a combined response to the Stone. This reduces the number of round trips from the Stone to distant nodes.

If a mid-level cache is in use, then for each Gem process using the mid-level cache, all the shared caches to which the Gems are attached are subordinate to that mid-level cache.

Figure 5.8   An Application with a Mid-Level Cache

---

Setting up a configuration with a mid-level cache requires that the session execute code following login to start a mid-level cache on a specified host, or to connect to an existing mid-level cache. Unlike the other remote configurations discussed in this chapter, the configuration is not established entirely by configuration settings and login arguments.

The Gem must be on a node that is remote from the Stone, and the request to connect to a mid-level cache must specify a node that is neither the Stone’s nor the Gem’s node.

If a Gem is running on the same machine as a mid-level cache, that Gem will use the mid-level cache as its local cache.

topaz> set gemstone !@Server!gs64stone

topaz> set username DataCurator

topaz> set gemnetid !@gemNode!gemnetobject

topaz> login

GemStone Password?

<details omitted>

successful login

topaz 1>

topaz 1> printit

  System midLevelCacheConnect: 'midLevelCacheNode'

    cacheSizeKB: 16000 maxSessions: 12

%

true

How the Login Process starts Session Processes

When a remote client applications requests a login, it communicates with the appropriate processes on the server. This process is ordinarily transparent to the user.

The following examples show the sequence of steps that occur during login in order to start the required processes and connect them appropriately.

RPC Application with Gem on Server Node

The simplest configuration for a distributed configuration is a client application running on a remote node, that connects to a RPC Gem session on the server node. The following examples describe the sequence of steps that connect this configuration during login.

Figure 5.9   Connecting an RPC Application with Gem on Server Node

---

1. The client application contacts the NetLDI on the server node and requests a session.

2. The NetLDI invokes gemnetobject or a similar script to start an RPC Gem.

3. The Gem performs the login to the Stone.

4. The application client connects to the Gem.

5. The Gem attaches extent files.

Linked Application on Remote Node

A remote linked login is more involved, since the remote shared page cache and associated processes are also started up.

In this example, there is a NetLDI running on the remote node. This is not required for a linked login; in the case of a linked login with no NetLDI running, the tasks performed by the NetLDI are handled by the linked gem. .

Figure 5.10   Connecting a Linked Application to a Remote Server

---

1. The client application contacts the NetLDI on the server node, requesting the Stone's listening port.

2. The client application contacts the Stone on its listening port, requesting login.

3. Stone requests the Remote Cache page server on remote node, if there is not already an SPC on the remote node. If there is a cache already there, continue with step 6.

4. The NetLDI on the remote node launches the Remote Cache page server.

5. The Remote Cache page server launches the Shared Page Cache Monitor, which initializes the shared page cache.

6. Stone connects to the Remote Cache page server.

7. The client application requests a Remote Gem page server, if there is not already a Remote Gem page server for this node. If a Remote Gem page server already present, continue with step 9.

8. The NetLDI on the server node launches the Remote Gem page server.

9. The client application connects to the Remote Gem page server.

10. A newly launched Remote Gem page server attaches the extent files.

If You Still Have Trouble

If you still can’t log in to GemStone from an application on a remote node, try logging in on the server node as the same UNIX user account. We suggest that you first try a linked application, such as topaz -l, and when that works, move on to an RPC application (such as topaz or the equivalent topaz -r), still on the server.

os$ ipcs -m

IPC status from <running system> as of Mon Aug 07 16:22:27 PDT 2023

T     ID     KEY        MODE       OWNER    GROUP

Shared Memory:

m    768 0x4c177155 --rw-rw----  gsadmin     pubs