3. Connecting Distributed Systems
This chapter tells how to set up GemStone/S 64 Bit in a distributed environment:
3.1 Overview
A properly configured network system is nearly transparent to GemStone users, but it requires additional steps by the system administrator. Users must be given access to all the workstations that will run their GemStone processes. Pointers to network services must be set up, and file and process specifications must include the node name in addition to the file name and path. Because processes are running on different nodes, the log files are spread throughout the network, and troubleshooting may become more complicated.
The nodes in your system can be any combination of GemStone-supported platforms, as long as they are connected by means of TCP/IP.
Although the Network File System (NFS) can be used to share executables, libraries, and configuration files, they are not recommended, and by default disallowed, for sharing repository extents and tranlogs.
Figure 3.1 and Figure 3.2 show two typical distributed configurations in which an application on a remote node is logged in to a repository and Stone repository monitor running on a server node.
Distributed configuration with Local Gem
In Figure 3.1, an application communicates with a Gem session process on the server node by way of RPC calls. This configuration lets the Gem execute Smalltalk code in the repository without first bringing complex objects across the network. The Gem can access the shared page cache that was started by the Stone repository monitor.
Figure 3.1 Gem Session on Server Node
This configuration is considered a “Local” Gem, since the Gem is local to the Stone.
With this configuration, GemStone components other than the shared libraries do not need to be installed on the remote node. The GemStone server shared library files allow the application to make RPC calls to the Gem. This configuration allows applications on Windows (as well as other platforms) to connect to a GemStone server.
For instructions on setting up this configuration, see RPC Application on a Remote Node with Local Gem.
Distributed configuration with Remote Gem
In Figure 3.2, the application and the Gem are linked in a single process that runs on the remote node. This configuration avoids the overhead of RPC calls, but in some applications it may increase network traffic if large objects must be brought across the network.
When a user on a remote node logs in to the repository, if there is not already a shared page cache running on that remote node, a remote cache is started. The Stone and the Gem session process each use a GemStone page server to access data pages residing on the other node. Multiple Gem sessions, both linked and RPC, may share the shared page cache and other server processes that run on the remote node.
This configuration requires a full installation of the GemStone server on the remote node, although the Stone and associated processes will not be run there, nor will extents or tranlogs be located on the remote note.
For instructions on setting up this configuration, see Linked Application on a Remote Node.
GemStone NetLDIs
The GemStone network server process is called NetLDI (Network Long Distance Information). The NetLDIs are the glue holding a distributed GemStone system together. Each NetLDI reports the location of GemStone services on its node to remote processes that must connect to those services. It also spawns other GemStone processes on request.
In a distributed system, each node where a Stone repository monitor or Gem session process runs must have its own NetLDI; you do not need a NetLDI on nodes that are running only linked applications, or RPC applications with the gems on a different node.
You start a NetLDI directly by invoking the startnetldi command (described here). The NetLDI, in turn, starts Gem session processes and page servers on demand. (See the following section for more about page servers.) These child processes belong by default to the user account of the process requesting the service—sometimes that account is a user logging in to GemStone, other times it is the account that started the repository monitor.
Because most operating systems only let the root account start processes that will be owned by other accounts, your NetLDI must be setup correctly if it is to serve more than one user. There are two ways to accomplish this:
- Set the owner and S bit for $GEMSTONE/sys/netldid so that the NetLDI runs as root.
- Run netldi in captive account mode, in which all processes started by the NetLDI belong to a single user account.
For more details on configuring NetLDI security, see NetLDI configuration and Network Security.
NetLDI Ports and Names
The NetLDI listens for all requests on a single, fixed TCP/IP port, either as provided by the startnetldi -P option, or by querying the OS using getservbyname(). During the installation process, the NetLDI name is added to network services database (for a simple installation, the /etc/services file), and is assigned a port number.
The default name of the NetLDI process is gs64ldi. You may add your own NetLDI name instead, or in addition to this, and include multiple NetLDI names with different ports.
To access NetLDIs by name in a distributed system, the same name and port must be defined on every node. This includes all server nodes that will run Stones, Gems, or Caches, and also client nodes, Windows as well as UNIX, even if they do not run a NetLDI. This allows client nodes to reference a NetLDI by name during login.
To avoid the need to update the network services database, you can configure the well-known port by starting NetLDI using the startnetldi -P option, or by specifying a valid, unused port instead of a name in startnetldi. You must then reference the NetLDI using the port number, not the name; lookup by name requires an entry in /etc/services.
If you do not use the default NetLDI name gs64ldi, then you need to specify the NetLDI by name or port in NRS, using a line of the form #netldi:netLdiName. For example,
!@hostname.com#netldi:MyNetLDI!stoneName
To avoid entering the netldi with each NRS, you can add this information to the GEMSTONE_NRS_ALL environment variable for each user. For example:
$ GEMSTONE_NRS_ALL=#netldi:MyNetLDI
$ export GEMSTONE_NRS_ALL
For more information about GEMSTONE_NRS_ALL, see How To Use Network Resource Strings.
Stone and Shared Page Cache Monitor
Every named GemStone process — Stone, NetLDI, and Shared Page Cache Monitor — creates a serviceName..LCK at startup, in the directory /opt/gemstone/locks or /usr/gemstone/locks, or in a directory specified by GEMSTONE_GLOBAL_DIR (as discussed here). The .LCK file holds the well-known port that the process is listening on. When the NetLDI (for example) contacts a particular Stone, it first looks up the well-known port for that Stone by locating the lock file for that Stone, then contacts the Stone on that port.
The ports used by the Stone and Shared Page Cache Monitor can be specified using STN_WELL_KNOWN_PORT_NUMBER and SHR_WELL_KNOWN_PORT_NUMBER configuration options, respectively. These must be valid port numbers that are not already in use.
GemStone Page Servers
GemStone uses specialized page servers for various purposes. As described in Chapter 1, GemStone repository I/O is carried out by the AIO page server, and the Free Frame page server adds free frames to the free frame list. For more about page servers, see Tuning Page Server Behavior.
Sessions on remote hosts also require page servers:
- When a Gem connects to a repository extent across the network, a Remote Gem page server is spawned on the Stone’s node, if a page server associated with the remote node is not already running. This page server looks up pages in the Stone’s cache and performs extent reads, on behalf of the gem
- On the remote node, a Remote Cache page server supports the Stone in managing the remote shared page cache.
GemStone Network Objects
GemStone uses the concept of network objects to encompass the services that a NetLDI can provide to a client. In addition to the page server, other network objects include the following services requested by the Stone at startup: the shared page cache monitor, SymbolGem, and garbage collection (GcGem) sessions.
The network object most visible to users is the Gem session process requested by an RPC application. This object can be gemnetobject or the name of a custom Gem. The request can be sent to the NetLDI on the same node to start a local session process, or (by using a network resource string) to a NetLDI on another node to start a process there.
The NetLDI first tries to map the requested object to the path of an executable by looking for an entry in $GEMSTONE/sys/services.dat. That file contains an entry for the standard Gem session process:
gemnetobject $GEMSTONE/sys/gemnetobject
For example, when you enter “gemnetobject” as a session login parameter (such as for gemnetid in Topaz), the NetLDI uses the services.dat file to map the request to the script $GEMSTONE/sys/gemnetobject. Similarly, an object name can be entered while setting up a GemBuilder session (as Name of Gem Service) or other application. Application programmers provide the name as a parameter to GciSetNet().
If your application uses a custom Gem executable, you can edit services.dat to include the appropriate mapping. For the procedure, see How To Install a Custom Gem.
If the NetLDI does not find the requested object in services.dat, it searches for an executable with that name in the user’s $HOME directory. If you have a private Gem executable, place the executable in $HOME and then enter its name in place of gemnetobject during a GemStone login. Because of the search order, the private name must not be the same as that of an object in services.dat. The name must be the name of a file in $HOME, not a pathname.
Shared Page Cache in Distributed Systems
When a GemStone login includes a Gem running on a remote node, either a linked login or an RPC login with the Gem on the remote node, then a shared page cache and shared page cache monitor are started on that node, along with a free frame page server to service the remote shared page cache. Normally, the cache is started up by a NetLDI running on the remote node, although a linked session can also initiate the startup.
When the remote Gem wants to access a page in the repository, it first checks the shared page cache on the remote node. If the page is not found, the Gem uses a pgsvrmain on the server node, checking in the shared cache on that node and then, if necessary, reading the page from the disk. See Figure 3.3.
Disrupted Communications
Several incidents can disrupt communications between the GemStone server and remote nodes in a distributed configuration. This can include node crashes, firewalls, and loss of the communications channel itself.
GemStone ordinarily depends on the network protocol keepalive option to detect that a remote process no longer exists because of an unexpected event. The keepalive interval is set globally by the operating system, typically at two hours. When that interval expires, the GemStone process tries to obtain a response from its partner. The parameters governing these attempts also are set by the operating system, with up to 10 attempts in 15 minutes being typical. If no response is received, the local GemStone process acts as if its partner was terminated abnormally. In some cases, you may wish to adjust the keepalive interval to allow for a longer timeout.
3.2 NetLDI configuration and Network Security
This section describes the levels to which the system administrator can set the GemStone authentication requirement.
Running as Root with Authentication
When the NetLDI is started in default mode, it performs authentication for starting client processes. Since running the NetLDI in this mode from an ordinary user account requires that only that user may start processes, this mode is normally used with the NetLDI running as root, with the setuid bit set on the executables.
When running in this mode, the file permissions and ownership for the NetLDI executable should look similar to this:
-r-sr-xr-x 1 root gsadmin 674488 Mar 7 11:29 netldid
The UNIX host login name and password that will be used for authentication are set using the application’s user interface, such as the HostUserName and HostPassword parameters in Topaz.
NOTE
Authentication is always done using the “real” user id, not the effective user id as set by the S bit on GemStone executables.
In this mode, PAM (Pluggable Authentication Modules) is used to authenticate, and the system should either configure a service with the name gemstone.netldi, or ensure that the default PAM authentication will allow logins. The actual means of authentication, such as LDAP, depends on how your system administrators have configured your UNIX installation.
Setting host Username and Password
Your application’s login interface will generally let you specify a UNIX login name and password for the node on which you will be running an RPC Gem session process. For example, Topaz lets you set these as variables:
topaz> set hostusername yourLogin
topaz> set hostpassword yourPassword
GemBuilder for Smalltalk (GBS) provides similar fields in its login dialog, and other GemStone products and tools also provide ways to enter this information. .
You may specify the login name and password in NRS syntax. When you use topaz, GBS, or other GemStone tools, the fields for host user name and host password are used to build the NRS containing the authentication information. This is then passed to the NetLDI. For example, if you set the Topaz login parameters HostUserName and HostPassword, the application puts them in an NRS like the following:
'!@Server#auth:HostUserName@HostPassword!gemnetobject'
Although it is less convenient for ordinary use, this can be done manually, by entering the authorization modifier directly using the Topaz GemNetId parameter. For example:
topaz> set gemnetid !@Server#auth:name@password!gemnetobject
Authentication Levels
When running the NetLDI as root with setuid, there are two levels of authentication:
Default
In the default level of authentication, authentication is required each time a NetLDI attempts to start certain processes for a client, even if that process is to run on the node where the user is logged in. These situations always require authentication:
- Starting an RPC Gem session process, even on the same node.
- Creating or restoring a GemStone backup using a device on a node remote from the Gem performing the operation.
- Using copydbf between non-NFS mounted nodes.
Once a Stone or Gem is running, the NetLDI treats it as a trusted client and starts the page servers needed by a remote login without authentication. Simple network information requests, such as a request to look up a port number, also do not require authentication.
Secure Mode
startnetldi -s starts the NetLDI in secure mode. All accesses are authenticated, including simple requests to look up a server name. This mode affects the waitstone command and such user actions as connecting a session process to a remote Stone (a NetLDI is asked to look up the Stone’s address. In secure mode, authentication is needed before a Gem or Stone can start a page server to access an extent or shared page cache on another node.
Running in Guest Mode with Captive Account
As an alternative, you may start the NetLDI with a captive account and in guest mode, which together provide the NetLDI security.
startnetldi -aname starts the NetLDI in captive account mode. All child processes created by the NetLDI will belong to the single, designated account name. This mode can be used when the NetLDI runs either as root or as name, but is normally combined with captive account.
The effect of captive account mode is much like setting the S bits on executables, but it only affects ownership of processes started by the NetLDI, not linked applications invoked directly by the user.
The captive account can be an ordinary user account or one created for that purpose, such as a GemStone administrative account. Log files by default will be in the captive account’s home directory.
startnetldi -g starts the NetLDI in guest mode. No accesses are authenticated. Guest mode is not permitted if the NetLDI will run as root, so while it can be used without a captive account, it is normally combined with captive account mode.
Running the NetLDI in guest mode with captive account serves multiple users with the convenience of guest mode and with improved security, because the child processes no longer belong to accounts of individual users who request services.
The principal advantage of this combination is that the NetLDI can spawn processes on behalf of multiple users without being run as root. To make this capability possible, the captive account must own the netldi process. Change the file permissions and ownership for the NetLDI executable to remove the setuid bit. The resulting permissions look like this:
-r-xr-xr-x 1 gsadmin gsadmin 674488 Mar 7 11:29 netldid
A disadvantage of the captive account for some applications is that the Gem session process will perform all I/O as that account, not as the account running the application — all file-ins, file-outs, and System class >> performOnServer:.
The captive account mode differs from the setuid method in that captive account mode affects all services started by the NetLDI, including any ad hoc processes, which are processes started from the user’s home directory. (The NetLDI looks in the user’s home directory if it cannot find a service listed in $GEMSTONE/sys/services.dat.) If you prefer, you can prohibit such ad hoc services by specifying the -n option when starting the NetLDI.
This is the procedure to configure guest and captive account:
Step 1. Create an OS account to own the GemStone distribution tree and serve as the captive account. We refer to this account as gsadmin.
Step 2. Make gsadmin the owner of the distribution tree, and set the setuid bit for any linked GemStone executables that run on the server node. Make the repository extents accessible only by gsadmin (mode 600). For instructions, see To Set File Permissions for the Server.
Step 3. Make sure that gsadmin has execute permission for $GEMSTONE/sys/netldid. The setgid bit should NOT be set on the netldid executable ; it should look similar to this:
-r-xr-xr-x 1 gsadmin gsadmin 674488 Mar 7 11:29 netldid
Step 4. Log in as the captive account (such as gsadmin). Then start the NetLDI in guest mode and captive account mode. For instance:
% startnetldi -g -a gsadmin
You may specify other startnetldi options; for more details about this command and its options, see startnetldi.
Table 3.1 shows how guest mode and captive account mode combinations affect NetLDI operation.
3.3 How To Use Network Resource Strings
The network resource strings (NRS) allows you to specify the location of each part of the GemStone system. Use an NRS on a network system where you would use a process or file name on a single-node system. For example, suppose you want to know whether a Stone named development33 is running. If the Stone is on the local node, use this command:
$ waitstone development33 -1
If the Stone is on a remote node, use a command like this instead:
$ waitstone !@oboe!development33 -1
where oboe is the Stone’s node.
Since a NetLDI is responsible for providing most services on a particular node, you will need a NetLDI to be running on the remote host (such as oboe) in order for queries and commands such as this to be performed. If the NetLDI on that host is not running with the default name (gs64ldi), then you can include this within the NRS. For example, if the NetLDI on oboe is running with the name netldi33, then you may use this:
$ waitstone !@oboe#netldi:netldi33!development33 -1
NRS syntax allows you to further specify authorization information, log directories and configuration parameters. among other things. For more information, see Appendix C, “Network Resource String Syntax”.
- You can use an Internet address in “dot” form, such as 120.0.0.4, to identify the remote node.
- You may specify the NetLDI by listening port, rather than by name.
- If using the C shell, note that each “!” must be preceded by a backslash (\).
Appendix B, “GemStone Utility Commands”, indicates which options of each UNIX-level GemStone command can be specified as an NRS.
To Set a Default NRS
To make it easier to enter NRS settings, you may set these via an environment variable, which you can set in the environment on each node.
The environment variable GEMSTONE_NRS_ALL determines which modifiers GemStone will use by default in each NRS it processes on your behalf. For instance, you can cause all Gem session process logs to be created with a specific name in a specific directory.
- If you set GEMSTONE_NRS_ALL before starting a NetLDI, which is a system-wide service, that setting is passed to all its children and becomes the default for all users of that service.
- If you set GEMSTONE_NRS_ALL before starting a Stone, an application, or a utility (such as copydbf), that setting applies only to your own processes and does not affect other users.
Because these settings are defaults, they take effect only if an explicit setting is not provided for the same modifier in a specific NRS setting or command-line argument.
For example, to use the #dir modifier to set the current (working) directory for NetLDI child processes, use code such as the following:
$ GEMSTONE_NRS_ALL=#dir:/user2/apps/logs
$ export GEMSTONE_NRS_ALL
This causes, for example, the gemnetobject log files to be written to the directory /user2/apps/logs; if this directory is not writable, an error is generated. If this is not set, the default is the user’s home directory
3.4 Configuring GemStone on Remote Nodes
If you are only running local Gems; in other words, only the client application will run on the remote node, most of the configuration for the Gem is on the server node. However, if you will run Gems or Linked sessions on nodes remote from the Stone, then more configuration is required on the remote node.
Local Gems only
If you are only running Gems on the same node as the Stone, there are only a few steps required on the remote node.
- Install the client shared libraries. This can be done by a complete server installation, or by copying the small number of required library files to the client, and configuring your client search paths appropriately.
Details on how to do this, and the various options, are in the GemStone/S 64 Bit Installation Guide for the client platform.
- If you are using named NetLDIs, you must configure your client system services database to map the same NetLDI name to the same port as defined on the server. If you are using a local services database, edit /etc/services or, on Windows, \WINDOWS\system32\drivers\etc\services to add the appropriate entry.
Remote Gems
Configuring GemStone so that Gem sessions or linked applications can run on the remote node is much the same as configuring the Gem and cache a session process on the server, which is described in Chapter 2, “Configuring Gem Session Processes”.
The remote node will need access to a number of executables and shared libraries within the server installation. You can either repeat the installation from the GemStone distribution media, or mount the directory on the server node that contains $GEMSTONE.
If you repeat the installation on the remote node, we recommend that you also run $GEMSTONE/install/installgs. In particular, you should make the same selections regarding the ownership and group for the GemStone files as you did on the primary server node. You can save disk space later by deleting initial repositories ($GEMSTONE/data/extent0.dbf and $GEMSTONE/bin/*.dbf) and the complete upgrade ($GEMSTONE/upgrade) and seaside ($GEMSTONE/seaside) directories.
Some additional details points to keep in mind:
- A remote node (on which a Gem is running) must have its kernel configured for shared memory similarly to how it is configured on the primary server node.
- Only nodes running a Stone need a GemStone key file, not nodes running remote sessions.
- It’s best if each node has its own directory for /opt/gemstone/log and /opt/gemstone/locks (or /usr/gemstone/log and /usr/gemstone/locks, or other location under $GEMSTONE_GLOBAL_DIR). If these directories are on an NFS-mounted partition, make sure that two nodes are not using the same directories. Each Stone and NetLDI needs a unique lock file. Shared log files may make it impossible to diagnose problems.
- If you are using named NetLDIs, you must configure your client system services database to map the same NetLDI name to the same port as defined on the server. If you are using a local services database, edit /etc/services or, on Windows, \WINDOWS\system32\drivers\etc\services to add the appropriate entry.
- Unless you run the NetLDIs in guest mode with a captive account, users must have an account on, and authorized network access to, all nodes that are part of the GemStone network for the repository they will be using. This includes the nodes on which the Stone Repository monitor and the user’s Gem session process reside.
Unless you run the NetLDIs in guest mode with a captive account, the user who starts the Stone repository monitor ordinarily needs an account on every node where a Gem session process will run.
Configuration Examples
GemStone supports several configurations in which the application communicates with the Gem session process by using remote procedure calls (RPCs), as well as linked applications. This section presents examples that illustrate the processes and interprocess connections within the following distributed applications:
- RPC Application on a Remote Node with Local Gem
- Linked Application on a Remote Node
- RPC Application on a Remote Node with Remote Gem
- RPC Application, Gem, and Stone on Three Nodes
- Distributed System with a Mid-Level Cache
Determine if you will run RPC or Linked application
GemStone shared libraries are provided in two variants: linked and RPC. In order to login a linked session, you must be running an application bound to the linked libraries, such as topaz -l. To login an RPC session, you may run an application bound with the RPC libraries or with the linked applications.
A linked application may have only one linked login, but may login RPC sessions in addition to (or instead of) the linked login. RPC sessions applications are not able to login linked sessions.
Set the environment
The instructions assume that you are already set up to run GemStone applications, as described in Chapter 2. In particular, you must have defined the GEMSTONE environment variable and invoked $GEMSTONE/bin/gemsetup.sh or its equivalent, so the system search path include $GEMSTONE/bin.
Configure NetLDI and authentication mechanism
For all these examples, you must have a NetLDI running on the Stone’s host. In many cases you will also need a NetLDI on the remote node or nodes. These should be configured according to your security requirements, as described under NetLDI configuration and Network Security.
RPC Application on a Remote Node with Local Gem
If the Gem session process is going to run on the server node (as shown in Figure 3.4), an RPC application uses a NetLDI on that node to start a Gem session process. Unless the NetLDI is running in guest mode with a captive account, the application user must provide authentication to the NetLDI. The login parameters include the specification for the Gem network object (gemnetobject). For more information about network objects and how to invoke them, see GemStone Network Objects.
Figure 3.4 RPC Application on a Remote Node with remote Gem
To login an RPC session with the Gem on the Stone’s host, set the login parameters as follows:
topaz> set gemstone stoneName
The hostname is optional in this case; by default, it will use the host specified by the NRS in the gem service parameter (gemnetid)
topaz> set username DataCurator
topaz> set password swordfish
You may omit the password; you will be prompted for it on login.
topaz> set hostusername unixUserAccountName
topaz> set hostpassword passwordForUnixUserAccount
- Set the gem service, specifying the stone’s host name, and the name of the NetLDI if necessary. For example, either one of the following:
topaz> set gemnetid !@serverhost!gemnetobject
topaz> set gemnetid !@serverhost!netldi:NetLdiName!gemnetobject
Linked Application on a Remote Node
Figure 3.5 shows how a linked application on a remote node communicates with a Stone and repository on the primary server node.
Figure 3.5 Connecting a Linked Application to a Remote Server
In this configuration, a remote, subordinate shared page cache is started on the remote node, along with associated server processes. The configuration of this remote cache is set by the first session to log in on this node. After the last session on this node logs out, the cache will shut down.
In addition, this configuration requires a Remote Gem page server on the server node, which reads pages from the repository on behalf of the gem.
Because the shared page cache is readable and writable only by its owner and members of the same group (protection 660), the user running the application may need to belong to that group. See To Set Ownership and Permissions for Session Processes.
Set the login parameters as follows:
topaz> set gemstone !@serverhost!stoneName
topaz> set gemstone !@serverhost!netldi:NetLdiName!stoneName
topaz> set username DataCurator
topaz> set password swordfish
You may omit the password; you will be prompted for it on login.
topaz> set hostusername unixUserAccountName
topaz> set hostpassword passwordForUnixUserAccount
RPC Application on a Remote Node with Remote Gem
The configuration shown in Figure 3.6 shows the RPC application and its session process running on the same node. This configuration is similar to a linked application, but in this case, a NetLDI is required on the remote node in order to launch the RPC Gem session process.
Figure 3.6 Starting the Session Process on a Remote Node
topaz> set gemstone !@serverhost!stoneName
topaz> set gemstone !@serverhost!netldi:NetLdiName!stoneName
You must specify the Stone’s host, since the otherwise it will assume the Stone is on the same node as the Gem, given in the gemnetid’s NRS.
topaz> set username DataCurator
topaz> set password swordfish
You may omit the password; you will be prompted for it on login.
topaz> set hostusername unixUserAccountName
topaz> set hostpassword passwordForUnixUserAccount
- Set the gem service, specifying the stone’s host name, and the name of the NetLDI if necessary. For example, one of the following:
topaz> set gemnetid !@remotenode!gemnetobject
topaz> set gemnetid !@remotenode!netldi:NetLdiName!gemnetobject
RPC Application, Gem, and Stone on Three Nodes
The RPC application, session process, and Stone can run on three separate nodes. The application runs on its node and connects to a Gem session process on the Gem’s node, which is sometimes referred to as the Gem Server. That session process communicates with the repository on the primary server node by way of a page server.
Again we see that a NetLDI must be running on each node where part of GemStone executes (but is not required on the application node, which runs only the RPC application).
The following diagram shows two separate RPC applications connecting to Gems running on the same node, the Gem Server node.
Figure 3.7 RPC Applications using a Gem Server Node
The directions for this configuration are the same as for the previous configuration, RPC Application on a Remote Node with Remote Gem; however, the application itself is executing on a third node.
topaz> set gemstone !@serverhost!stoneName
topaz> set gemstone !@serverhost!netldi:NetLdiName!stoneName
You must specify the Stone’s host, since the otherwise it will assume the Stone is on the same node as the Gem, given in the gemnetid’s NRS.
topaz> set username DataCurator
topaz> set password swordfish
You may omit the password; you will be prompted for it on login.
topaz> set hostusername unixUserAccountName
topaz> set hostpassword passwordForUnixUserAccount
- Set the gem service, specifying the stone’s host name, and the name of the NetLDI if necessary. For example, one of the following:
topaz> set gemnetid !@remotenode!gemnetobject
topaz> set gemnetid !@remotenode!netldi:NetLdiName!gemnetobject
Distributed System with a Mid-Level Cache
In a distributed system over a Wide Area Network (WAN), with many remote nodes that are topographically distant from the Stone but close to each other, a mid-level cache can improve performance for the remote sessions. In this configuration, the RPC application and the session process may be on the same or different nodes, with the mid-level cache and Stone running on separate nodes. In Figure 3.8, the Gem session process (on the leaf node) connects to a mid-level cache (on the mid-level cache node). The session process communicates with the repository (on the primary server node) by way of a page server.
When the Gem needs a page but can't find it in its local cache, it first looks in the mid-level cache. If the Gem can't find the page in the mid-level cache, it then forwards the request to the page server on the Stone’s host.
The Stone’s page server aggregates the responses from the page servers on each of the Gem’s shared caches, and returns a combined response to the Stone. This reduces the number of round trips from the Stone to distant nodes.
If a mid-level cache is in use, then for each Gem process using the mid-level cache, all the shared caches to which the Gems are attached are subordinate to that mid-level cache.
Figure 3.8 An Application with a Mid-Level Cache
Setting up a configuration with a mid-level cache requires that the session execute code following login to start a mid-level cache on a specified host, or to connect to an existing mid-level cache. Unlike the other remote configurations discussed in this chapter, the configuration is not established entirely by configuration settings and login arguments.
The Gem must be on a node that is remote from the Stone, and the request to connect to a mid-level cache must specify a node that is neither the Stone’s nor the Gem’s node.
If a Gem is running on the same machine as a mid-level cache, that Gem will use the mid-level cache as its local cache.
Connection Methods
System Class methods in the Shared Cache Management category allow you to connect to a mid-level cache.
midLevelCacheConnect: hostName
Attempts to connect to a mid-level cache on the specified host, if the cache already exists. The session's Gem process must be on a machine different from the machine running the Stone process.
midLevelCacheConnect: hostName cacheSizeKB: aSize
maxSessions: nSess
If a mid-level cache does not already exist on the specified host, and aSize > 0,
attempts to start the cache and connect to it. If a cache is already running on the host, this method attempts to connect to the cache and ignores the other arguments.
The size of the mid-level cache is controlled by the method argument aSize, rather than by configuration parameters (as with other shared caches).
topaz> set gemstone !@Server!gs64stone
topaz> set username DataCurator
topaz> set gemnetid !@gemNode!gemnetobject
topaz> login
GemStone Password?
<details omitted>
successful login
topaz 1>
topaz 1> printit
System midLevelCacheConnect: 'midLevelCacheNode'
cacheSizeKB: 2048 maxSessions: 10
%
Reporting Methods
System Class methods in the Shared Cache Management category return lists of the shared caches on your system.
Returns a String that lists all shared caches that the Stone process is managing, not including the cache on the Stone machine.
Similar to remoteCachesReport, but only includes the mid-level caches.
3.5 Troubleshooting Remote Logins
Logging into GemStone from a remote node requires proper system configuration of the remote node and frequently requires permission for network access from the primary server to the remote node as well as from the remote node to the primary server.
- The UNIX kernel on the remote node should meet shared memory and semaphore requirements similar to those for the server, although smaller sizes may be sufficient.
- Make sure that NetLDIs are running on all nodes that require them (see the figure for your configuration). Also make sure that the NetLDIs have the same port number in /etc/services. All nodes must be listed in /etc/hosts.
- If an RPC application is being started (that is, one with a separate Gem session process), make sure the user who starts the application has entered their host username and host password, or that NetLDI is running in guest mode with a captive account. The owner of the Gem process needs an account on the node where the Gem will run and needs write access to the Gem log, typically in $HOME. Ownership and permissions for $GEMSTONE/sys/netldid must be appropriate for the authentication system in use (details can be found under Running as Root with Authentication or Running in Guest Mode with Captive Account), and the directories in /opt/gemstone must be writable.
- Make sure that the user who started the Stone has an account on the remote node. This user also must have write permission for $HOME so that log files for the remote node can be created, unless steps are taken to create the log files in another directory.
- Check any GEMSTONE environment variables for definitions that point to a previous version: env | grep GEM.
How the Login Process starts Session Processes
When a remote client applications requests a login, it communicates with the appropriate processes on the server. This process is ordinarily transparent to the user.
The following examples show the sequence of steps that occur during login in order to start the required processes and connect them appropriately.
RPC Application with Gem on Server Node
The simplest configuration for a distributed configuration is a client application running on a remote node, that connects to a RPC Gem session on the server node. The following examples describe the sequence of steps that connect this configuration during login.
Figure 3.9 Connecting an RPC Application with Gem on Server Node
1. The client application contacts the NetLDI on the server node and requests a session.
2. The NetLDI invokes gemnetobject or a similar script to start an RPC Gem.
3. The Gem performs the login to the Stone.
Linked Application on Remote Node
A remote linked login is more involved, since the remote shared page cache and associated processes are also started up.
In this example, there is a NetLDI running on the remote node. This is not required for a linked login; in the case of a linked login with no NetLDI running, the tasks performed by the NetLDI are handled by the linked gem. .
Figure 3.10 Connecting a Linked Application to a Remote Server
1. The client application contacts the NetLDI on the server node, requesting the Stone's listening port.
2. The client application contacts the Stone on its listening port, requesting login.
3. Stone requests the Remote Cache page server on remote node, if there is not already an SPC on the remote node. If there is a cache already there, continue with step 6.
4. The NetLDI on the remote node launches the Remote Cache page server.
5. The Remote Cache page server launches the Shared Page Cache Monitor, which initializes the shared page cache. The Remote Cache pages server also launches the free frame page server.
6. Stone connects to the Remote Cache page server.
7. The client application requests a Remote Gem page server, if there is not already a Remote Gem page server for this node. If a Remote Gem page server already present, continue with step 9.
8. The NetLDI on the server node launches the Remote Gem page server.
9. The client application connects to the Remote Gem page server.
10. A newly launched Remote Gem page server attaches the extent files.
If You Still Have Trouble
If you still can’t log in to GemStone from an application on a remote node, try logging in on the server node as the same UNIX user account. We suggest that you first try a linked application, such as topaz -l, and when that works, move on to an RPC application (such as topaz or the equivalent topaz -r), still on the server.
Try Linked Topaz on the Server
A linked application on the server offers the least complicated kind of login because the server’s shared page cache is already running and no network facilities are used. Any problems are likely to involve access permission for the shared page cache or the repository extents, which can also block attempts to log in from a client node.
- Make sure that the owner of the topaz process ($GEMSTONE/bin/topaz) can access the shared page cache. Use the UNIX command ipcs -m to display permissions, owner, and group for shared memory; for example:
Server% ipcs -m
IPC status from <running system> as of Thu Mar 12 16:22:27 PDT 2015
T ID KEY MODE OWNER GROUP
Shared Memory:
m 768 0x4c177155 --rw-rw---- gsadmin pubs
Compare the owner and group returned by ipcs with the owner of the Topaz process. You can use the ps command to determine the owner; for example,
ps -ef | grep topaz. (The switches may be different on your system.)
A typical problem arises when root owns the Stone process and the shared page cache because their group ordinarily will be a special one to which Topaz users do not belong. Related problems may occur with a linked GemBuilder session. The third-party Smalltalk may be installed without the S bits and therefore may rely on group access to the shared page cache and repository. For background information, see To Set Ownership and Permissions for Session Processes.
To correct a shared page cache access failure, either change the owner and group of the setuid files or have the Stone started by a user whose primary group is one to which other GemStone users belong. Unlike file permissions, the shared page cache permissions cannot be set directly.
Try Topaz RPC on the Server
The next step should be to try running Topaz on the server with a separate Gem session process. This configuration relies on the NetLDI to start a Gem session process, and that process, not the application itself, must be able to access the shared page cache and repository extent.
- Make sure that a NetLDI is running on the server by invoking gslist. The default name is gs64ldi. If you need to start a NetLDI, the command is startnetldi.
GemStone uses the NetLDI to start a Gem session process that does repository I/O in this configuration. For the NetLDI to start processes for anyone other than its owner, it must be owned by root or it must be started in guest mode and captive account mode by someone logged in as the captive account.
The NetLDI writes a log file with the default name
/opt/gemstone/log/gs64ldi.log; this may be overridden by the startnetldi -l option. Using the gslist -x command will provide the location of all log files. The log file contents may help you diagnose problems. (See the following discussion, “Check NetLDI Log Files.“)
- Make sure that the owner of the resulting Gem session process ($GEMSTONE/sys/gem) can access the shared page cache and extent0.dbf through group membership or S bits. The troubleshooting is the same as that given here for the topaz executable.
The user who starts topaz (or the NetLDI captive account when it is in use) must have write permission for $HOME so that the session process can create a log file there. (For a workaround for situations where write permission is not allowed, see To Set a Default NRS.)
Check NetLDI Log Files
Troubleshooting on a distributed GemStone system can be complicated. What looks like a hung process may actually be caused by incorrect NRS syntax or by another node on the network going down. The information for analyzing problems may be found in log files on all the nodes used by GemStone.
Where the log file messages include NRS strings, be sure to check their syntax. The problem may be as simple as an incorrect NRS or one that was not expanded by the shell as you intended.
If you can’t identify the problem from the standard log messages, try running the NetLDI in debug mode, which puts additional information in the log. The command line is startnetldi [netLdiName] -d. For further details, see startnetldi.