3. Connecting Distributed Systems
This chapter tells how to set up GemStone/S 64 Bit in a distributed environment:
3.1 Overview
A properly configured network system is nearly transparent to GemStone users, but it requires additional steps by the system administrator. Users must be given access to all the workstations that will run their GemStone processes. Pointers to network services must be set up, and file and process specifications must include the node name in addition to the file name and path. Because processes are running on different nodes, the log files are spread throughout the network, and troubleshooting may become more complicated.
The nodes in your system can be any combination of GemStone-supported platforms, as long as they are connected by means of TCP/IP.
Although the Network File System (NFS) can be used to share executables, libraries, and configuration files, they are not recommended, and by default disallowed, for sharing repository extents and tranlogs.
Figure 3.1 and Figure 3.2 show two typical distributed configurations in which an application on a remote node is logged in to a repository and Stone repository monitor running on a server node.
In Figure 3.1, an application communicates with a Gem session process on the server node by way of RPC calls. This configuration lets the Gem execute Smalltalk code in the repository without first bringing complex objects across the network. The Gem can access the shared page cache that was started by the Stone repository monitor. For instructions on setting up this configuration, see To Run the Gem Session Process on the Stone’s Node.
Figure 3.1 Gem Session Process on Server Node
In Figure 3.2, the application and the Gem are linked in a single process that runs on the remote node. This configuration avoids the overhead of RPC calls, but in some applications it may increase network traffic substantially if large objects must be brought across the network. The Stone repository monitor starts a shared page cache on the remote node when the first user from that node logs in to the repository. The Stone and the Gem session process each use a GemStone page server to access data pages residing on the other node. For instructions on setting up this configuration, see To Run a Linked Application on a Remote Node.
GemStone NetLDIs
The GemStone network server process is called NetLDI (Network Long Distance Information). The NetLDIs are the glue holding a distributed GemStone system together. Each NetLDI reports the location of GemStone services on its node to remote processes that must connect to those services. It also spawns other GemStone processes on request.
In a distributed system, each node where a Stone repository monitor or Gem session process runs must have its own NetLDI. You do not need a NetLDI on nodes that are running only linked applications, or RPC applications with the gems on a different node.
You start a NetLDI directly by invoking the startnetldi command (described here). The NetLDI, in turn, starts Gem session processes and page servers on demand. (See the following section for more about page servers.) These child processes belong by default to the user account of the process requesting the service—sometimes that account is a user logging in to GemStone, other times it is the account that started the repository monitor.
Because most operating systems only let the root account start processes that will be owned by other accounts, your NetLDI must be setup correctly if it is to serve more than one user. There are two ways to accomplish this:
- Set the owner and S bit for $GEMSTONE/sys/netldid so that the NetLDI runs as root.
- Run netldi in captive account mode, in which all processes started by the NetLDI belong to a single user account.
For more details on configuring NetLDI security, see How To Arrange Network Security.
NetLDI Ports and Names
The NetLDI listens for all requests on a single, fixed TCP/IP port, either as provided by the startnetldi -P option, or by querying the OS using getservbyname(). During the installation process, the NetLDI name is added to network services database (for a simple installation, the /etc/services file), and is assigned a port number.
The default name of the NetLDI process is gs64ldi. You may add your own NetLDI name instead, or in addition to this, and include multiple NetLDI names with different ports.
To access NetLDIs by name in a distributed system, the same name and port must be defined on every node. This includes all server nodes that will run Stones, Gems, or Caches, and also client nodes, Windows as well as UNIX, even if they do not run a NetLDI. This allows client nodes to reference a NetLDI by name during login.
To avoid the need to update the network services database, you can configure the well-known port by starting NetLDI using the startnetldi -P option, or by specifying a valid, unused port instead of a name in startnetldi. You must then reference the NetLDI using the port number, not the name; lookup by name requires an entry in /etc/services,
If you do not use the default NetLDI name gs64ldi, then you need to specify the NetLDI by name or port in NRS, using a line of the form #netldi:netLdiName. For example,
!@hostname.com#netldi:MyNetLDI!stoneName
To avoid entering the netldi with each NRS, you can add this information to the GEMSTONE_NRS_ALL environment variable for each user. For example:
$ GEMSTONE_NRS_ALL=#netldi:MyNetLDI$ export GEMSTONE_NRS_ALL
For more information about GEMSTONE_NRS_ALL, see How To Use Network Resource Strings.
Stone and Shared Page Cache Monitor
Every named GemStone process — Stone, NetLDI, and Shared Page Cache Monitor — creates a serviceName..LCK at startup, in the directory /opt/gemstone/locks or /usr/gemstone/locks, or in a directory specified by GEMSTONE_GLOBAL_DIR (as discussed here). The .LCK file holds the well-known port that the process is listening on. When the NetLDI (for example) contacts a particular Stone, it first looks up the well-known port for that Stone by locating the lock file for that Stone, then contacts the Stone on that port.
The ports used by the Stone and Shared Page Cache Monitor can be specified using STN_WELL_KNOWN_PORT_NUMBER and SHR_WELL_KNOWN_PORT_NUMBER configuration options, respectively. These must be valid port numbers that are not already in use.
GemStone Page Servers
GemStone repository I/O is carried out by page server processes, running the executable file pgsvrmain. The Stone repository monitor creates one or more page servers at startup, to perform asynchronous I/O to the repository. There will also be Free Frame page servers that are dedicated to added free frames to the free frame list. For more about page servers, see Tuning Page Server Behavior.
Sessions on remote hosts also require page servers. For each process that connects to a repository extent across the network, the NetLDI service spawns a pgsvrmain on the stone’s node.
GemStone Network Objects
GemStone uses the concept of network objects to encompass the services that a NetLDI can provide to a client. In addition to the page server, other network objects include the following services requested by the Stone at startup: the shared page cache monitor, SymbolGem, and garbage collection (GcGem) sessions.
The network object most visible to users is the Gem session process requested by an RPC application. This object can be gemnetobject or the name of a custom Gem. The request can be sent to the NetLDI on the same node to start a local session process, or (by using a network resource string) to a NetLDI on another node to start a process there.
The NetLDI first tries to map the requested object to the path of an executable by looking for an entry in $GEMSTONE/sys/services.dat. That file contains an entry for the standard Gem session process:
gemnetobject $GEMSTONE/sys/gemnetobject
For example, when you enter “gemnetobject” as a session login parameter (such as for gemnetid in Topaz), the NetLDI uses the services.dat file to map the request to the script $GEMSTONE/sys/gemnetobject. Similarly, an object name can be entered while setting up a GemBuilder session (as Name of Gem Service) or other application. Application programmers provide the name as a parameter to GciSetNet().
If your application uses a custom Gem executable, you can edit services.dat to include the appropriate mapping. For the procedure, see How To Install a Custom Gem.
If the NetLDI does not find the requested object in services.dat, it searches for an executable with that name in the user’s $HOME directory. If you have a private Gem executable, place the executable in $HOME and then enter its name in place of gemnetobject during a GemStone login. Because of the search order, the private name must not be the same as that of an object in services.dat. The name must be the name of a file in $HOME, not a pathname.
Shared Page Cache in Distributed Systems
When the remote session logs in to the repository, the Stone repository monitor uses a NetLDI and page server (pgsvrmain) on the remote node to start a monitor process, and that monitor requests its local NetLDI to create a local shared page cache. When the remote Gem wants to access a page in the repository, it first checks the shared page cache on the remote node. If the page is not found, the Gem uses a pgsvrmain on the server node, checking in the shared cache on that node and then, if necessary, reading the page from the disk. See Figure 3.3.
Disrupted Communications
Several incidents can disrupt communications between the GemStone server and remote nodes in a distributed configuration. This can include node crashes, firewalls, and loss of the communications channel itself.
GemStone ordinarily depends on the network protocol keepalive option to detect that a remote process no longer exists because of an unexpected event. The keepalive interval is set globally by the operating system, typically at two hours. When that interval expires, the GemStone process tries to obtain a response from its partner. The parameters governing these attempts also are set by the operating system, with up to 10 attempts in 15 minutes being typical. If no response is received, the local GemStone process acts as if its partner was terminated abnormally. In some cases, you may wish to adjust the keepalive interval to allow for a longer timeout.
3.2 How To Arrange Network Security
This section describes the levels to which the system administrator can set the GemStone authentication requirement.
Running as Root with Authentication
When the NetLDI is started in default mode, it performs authentication for starting client processes. Since running the NetLDI in this mode from an ordinary user account requires that only that user may start processes, this mode is normally used with the NetLDI running as root, with the setuid bit set on the executables.
When running in this mode, the file permissions and ownership for the NetLDI executable should look similar to this:
-r-sr-xr-x 1 root gsadmin 674488 Mar 7 11:29 netldid
The UNIX host login name and password that will be used for authentication are set using the application’s user interface, such as the HostUserName and HostPassword parameters in Topaz.
NOTE
Authentication is always done using the “real” user id, not the effective user id as set by the S bit on GemStone executables.
In this mode, PAM (Pluggable Authentication Modules) is used to authenticate, and the system should either configure a service with the name gemstone.netldi, or ensure that the default PAM authentication will allow logins. The actual means of authentication, such as LDAP, depend on how your system administrators have configured your UNIX installation.
Setting host Username and Password
Your application’s login interface will generally let you specify a UNIX login name and password for the node on which you will be running an RPC Gem session process. For example, Topaz lets you set these as variables:
topaz> set hostusername yourLogin
topaz> set hostpassword yourPassword
GemBuilder for Smalltalk (GBS) provides similar fields in its login dialog, and other GemStone products and tools also provide ways to enter this information. .
You may specify the login name and password in NRS syntax. When you use topaz, GBS, or other GemStone tools, the fields for host user name and host password are used to build the NRS containing the authentication information. This is then passed to the NetLDI. For example, if you set the Topaz login parameters HostUserName and HostPassword, the application puts them in an NRS like the following:
'!@Server#auth:HostUserName@HostPassword!gemnetobject'
Although it is less convenient for ordinary use, this can be done manually, by entering the authorization modifier directly using the Topaz GemNetId parameter. For example:
topaz> set gemnetid !@Server#auth:name@password!gemnetobject
Authentication Levels
When running the NetLDI as root with setuid, there are two levels of authentication:
Default
In the default level of authentication, authentication is required each time a NetLDI attempts to start certain processes for a client, even if that process is to run on the node where the user is logged in. These situations always require authentication:
- Starting an RPC Gem session process, even on the same node.
- Creating or restoring a GemStone backup using a device on a node remote from the Gem performing the operation.
- Using copydbf between nodes.
Once a Stone or Gem is running, the NetLDI treats it as a trusted client and starts the page servers needed by a remote login without authentication. Simple network information requests, such as a request to look up a port number, also do not require authentication.
Secure Mode
startnetldi -s starts the NetLDI in secure mode. All accesses are authenticated, including simple requests to look up a server name. This mode affects the waitstone command and such user actions as connecting a session process to a remote Stone (a NetLDI is asked to look up the Stone’s address. In secure mode, authentication is needed before a Gem or Stone can start a page server to access an extent or shared page cache on another node.
Running in Guest Mode with Captive Account
As an alternative, you may start the NetLDI with a captive account and in guest mode, which together provide the NetLDI security.
startnetldi -aname starts the NetLDI in captive account mode. All child processes created by the NetLDI will belong to the single, designated account name. This mode can be used when the NetLDI runs either as root or as name, but is normally combined with captive account.
The effect of captive account mode is much like setting the S bits on executables, but it only affects ownership of processes started by the NetLDI, not linked applications invoked directly by the user.
The captive account can be an ordinary user account or one created for that purpose, such as a GemStone administrative account. Log files by default will be in the captive account’s home directory.
startnetldi -g starts the NetLDI in guest mode. No accesses are authenticated. Guest mode is not permitted if the NetLDI will run as root, so while it can be used without a captive account, it is normally combined with captive account mode.
Running the NetLDI in guest mode with captive account serves multiple users with the convenience of guest mode and with improved security, because the child processes no longer belong to accounts of individual users who request services.
The principal advantage of this combination is that the NetLDI can spawn processes on behalf of multiple users without being run as root. To make this capability possible, the captive account must own the netldi process. Change the file permissions and ownership for the NetLDI executable to remove the setuid bit. The resulting permissions look like this:
-r-xr-xr-x 1 gsadmin gsadmin 674488 Mar 7 11:29 netldid
A disadvantage of the captive account for some applications is that the Gem session process will perform all I/O as that account, not as the account running the application — all file-ins, file-outs, and System class >> performOnServer:.
The captive account mode differs from the setuid method in that captive account mode affects all services started by the NetLDI, including any ad hoc processes, which are processes started from the user’s home directory. (The NetLDI looks in the user’s home directory if it cannot find a service listed in $GEMSTONE/sys/services.dat.) If you prefer, you can prohibit such ad hoc services by specifying the -n option when starting the NetLDI.
This is the procedure to configure guest and captive account:
Step 1. Create an OS account to own the GemStone distribution tree and serve as the captive account. We refer to this account as gsadmin.
Step 2. Make gsadmin the owner of the distribution tree, and set the setuid bit for any linked GemStone executables that run on the server node. Make the repository extents accessible only by gsadmin (mode 600). For instructions, see To Set File Permissions for the Server.
Step 3. Make sure that gsadmin has execute permission for $GEMSTONE/sys/netldid. The setgid bit should NOT be set on the netldid executable ; it should look similar to this:
-r-xr-xr-x 1 gsadmin gsadmin 674488 Mar 7 11:29 netldid
Step 4. Log in as the captive account (such as gsadmin). Then start the NetLDI in guest mode and captive account mode. For instance:
% startnetldi -g -a gsadmin
You may specify other startnetldi options; for more details about this command and its options, see startnetldi.
Table 3.1 shows how guest mode and captive account mode combinations affect NetLDI operation.
3.3 How To Use Network Resource Strings
Once you have chosen the remote and server nodes, network resource strings (NRS) allow you to specify the location of each part of the GemStone system. Use an NRS on a network system where you would use a process or file name on a single-node system. For example, suppose you want to know whether a Stone is running. If the Stone is on the local node, use this command:
$ waitstone gemStoneName -1
If the Stone is on a remote node, use a command like this instead:
$ waitstone !@oboe!gemStoneName -1
where oboe is the Stone’s node. You can also use an Internet address in “dot” form, such as 120.0.0.4, to identify the remote node. Note that each “!” must be preceded by a backslash (\) when your command will be processed by the C shell.
Appendix B, “GemStone Utility Commands”, indicates which options of each UNIX-level GemStone command can be specified as an NRS. Besides location, an NRS can describe the network resource type so that GemStone can more accurately interpret the command line. Sometimes an NRS can also include your authorization to use that resource. For more information, see Appendix C, “Network Resource String Syntax”.
To Set a Default NRS
You can set a default NRS header (the part between “! ... !”) by setting the environment variable GEMSTONE_NRS_ALL. This variable determines which modifiers GemStone will use by default in each NRS it processes on your behalf. For instance, you can cause all Gem session process logs to be created with a specific name in a specific directory.
- If you set GEMSTONE_NRS_ALL before starting a NetLDI, which is a system-wide service, that setting is passed to all its children and becomes the default for all users of that service.
- If you set GEMSTONE_NRS_ALL before starting a Stone, an application, or a utility (such as copydbf), that setting applies only to your own processes and does not affect other users.
Because these settings are defaults, they take effect only if an explicit setting is not provided for the same modifier in a specific request.
Use the #dir modifier to set the current (working) directory for NetLDI child processes, such as gemnetobject. Without this setting, the default is the user’s home directory. If the directory specified does not exist or is not writable at run time, an error is generated. For example:
$ GEMSTONE_NRS_ALL=#dir:/user2/apps/logs
$ export GEMSTONE_NRS_ALL
For further information about the modifiers and templates available, see Appendix C, “Network Resource String Syntax”.
3.4 How To Set Up a Remote Session
Configuring a Gem session process on a remote node is much the same as configuring a session process on the server, which is described in Chapter 2, “Configuring Gem Session Processes”. Keep the following points in mind:
- A remote node (on which a Gem is running) must have its kernel configured for shared memory similarly to how it is configured on the primary server node.
- Only nodes running a Stone need a GemStone key file, not nodes running remote sessions.
- If your site doesn’t run NIS, add each node in the GemStone network to /etc/hosts.
- If your site doesn’t run NIS, add the NetLDI entry to /etc/services on each node. Be sure to specify the same name and network port number each time.
- It’s best if each node has its own directory for /opt/gemstone/log and /opt/gemstone/locks (or /usr/gemstone/log and /usr/gemstone/locks, or other location under $GEMSTONE_GLOBAL_DIR). If these directories are on an NFS-mounted partition, make sure that two nodes are not using the same directories. Each Stone and NetLDI needs a unique lock file. Shared log files may make it impossible to diagnose problems.
- Unless you run the NetLDIs in guest mode with a captive account, users must have an account on, and authorized network access to, all nodes that are part of the GemStone network for the repository they will be using. This includes the nodes on which the Stone Repository monitor and the user’s Gem session process reside.
- Unless you run the NetLDIs in guest mode with a captive account, the user who starts the Stone repository monitor ordinarily needs an account on every node where a Gem session process will run.
You can either repeat the installation from the GemStone distribution media, or mount the directory on the server node that contains $GEMSTONE; these options are described in the following sections.
To Duplicate the GemStone Installation
If you repeat the installation on the remote node, we recommend that you also run $GEMSTONE/install/installgs. In particular, you should make the same selections regarding the ownership and group for the GemStone files as you did on the primary server node. You can save disk space later by deleting initial repositories ($GEMSTONE/data/extent0.dbf and $GEMSTONE/bin/*.dbf) and the complete upgrade ($GEMSTONE/upgrade) and seaside ($GEMSTONE/seaside) directories.
To Share a GemStone Directory
The following example prepares to run an application and Gem session process on a remote node using a shared software directory on the server. The GEMSTONE environment variable points to the shared installation directory, which is on the node Server and is NFS-mounted as /Server/users/gs64stone.
Step 1. Set the GEMSTONE environment variable to point to the NFS-mounted installation directory, and then invoke gemsetup:
(Bourne or Korn shell)
$ GEMSTONE=/Server/users/gs64stone
$ export GEMSTONE
$ . $GEMSTONE/bin/gemsetup.sh
(C shell)
Remote% setenv GEMSTONE /Server/users/gs64stone
Remote% source $GEMSTONE/bin/gemsetup.csh
Step 2. If they do not already exist, create the GemStone log and locks directories on the local node. (NetLDIs use this log directory.) You may need to have a system administrator do this for you as root.
# cd /opt
# mkdir gemstone gemstone/log gemstone/locks
# chmod 777 gemstone gemstone/log gemstone/locks
Configuration Examples
GemStone supports several configurations in which the application communicates with the Gem session process by using remote procedure calls (RPCs). Although the calls to network routines inevitably are time-consuming, they are essential when the application runs on a different node from the Gem, and they are desirable during code development because they isolate the application and Gem address spaces.
Use of RPC configurations for production repositories should be based on careful analysis of system loads and network traffic to select the most efficient configuration for a particular application. The RPC configuration may be desirable when the application accesses large or complex objects that would saturate the network if they were brought across it on a frequent basis.
This section presents examples that illustrate the following distributed applications:
- A linked application connected to a Stone on another node (To Run a Linked Application on a Remote Node).
- An RPC application with both the session process and the Stone on the server node (To Run the Gem Session Process on the Stone’s Node).
- An RPC application with the session process on the application’s node (To Run the Gem and Stone on Different Nodes).
- An RPC application in which all three are on different nodes (To Run the Application, Gem, and Stone on Three Nodes).
Two other examples show how to set up an extent on a node that is remote from the Stone, and how to use copydbf between nodes.
To Run a Linked Application on a Remote Node
Figure 3.4 shows how a linked application on a remote node communicates with a Stone and repository on the primary server node. This configuration typically is the best choice when you must offload some processes from a server node, especially when the application accesses relatively small objects or small groups of large objects.
Figure 3.4 Connecting a Linked Application to a Remote Server
Two NetLDIs and two page servers ordinarily are required. NetLDIs start the page servers on request of the Stone and the application. Numbers show the order in which these processes are started:
- One page server (1) lets the Stone start a shared page cache and monitor (2) on the remote node. The page server and monitor processes will be owned by the user who started the Stone (or by the captive account), so the owner must have an account on the remote node. The cache itself will have the same owner and group as the Stone. The linked application must have permission to access the cache, either through group membership or through an S bit on the application executable.
- The other page server (3) lets the Gem session process (the linked application) access the repository on the primary server. There will be one such page server process on the primary server node for each session logged in from a remote node; its owner (which may be a captive account) must have an account on the primary server. The page server process must have read-write permission for the repository, either through group membership or through an S bit on the pgsvrmain executable.
Because the shared page cache is readable and writable only by its owner and members of the same group (protection 660), the user running the application may need to belong to that group. See To Set Ownership and Permissions for Session Processes.
The following steps set up a linked application on the remote node. They use software in an NFS-mounted installation on the primary server node. (That directory is already mounted on the remote node as /Server/gs64stone.)
Step 1. Set the GEMSTONE environment variable to point to the installation directory, and then invoke gemsetup:
(Bourne or Korn shell)
$ GEMSTONE=/Server/gs64stone
$ export GEMSTONE
$ . $GEMSTONE/bin/gemsetup.sh
(C shell)
Remote% setenv GEMSTONE /Server/gs64stone
Remote% source $GEMSTONE/bin/gemsetup.csh
Step 2. Verify that a Stone and NetLDI are running on the primary server node. One way to do this verification is to use the gslist utility. For example:
Remote% gslist -m serverName
(The -m option tells gslist to list only processes that are running on the specified node. The options for this command are described in more detail under gslist.
Step 3. Start a NetLDI on the remote node.
- To start the NetLDI for password authentication, make sure $GEMSTONE/sys/netldid is owned by root and has the S bit set. Issue this command (on some operating systems, you may have to issue it as root):
Remote% startnetldi
- To start the NetLDI in guest mode (authentication is not required), make sure $GEMSTONE/sys/netldid does NOT have the S bit set. Log in as the captive account name, then issue this command:
Remote% startnetldi -g -aname
Step 4. Start the linked application (for instance, Topaz) on the remote node, then set the GemStone login parameter to include the name of the primary server node in network resource syntax. For instance, to log in to Topaz as DataCurator:
Remote% topaz -l
topaz> set gemstone !@Server!gs64stone
topaz> set username DataCurator
topaz> login
GemStone Password?
successful login
topaz 1>
To Run the Gem Session Process on the Stone’s Node
If the Gem session process is going to run on the server node (as shown in Figure 3.5), an RPC application uses a NetLDI on that node to start a Gem session process. Unless the NetLDI is running in guest mode with a captive account, the application user must provide authentication to the NetLDI. You should also specify the Gem network object (gemnetobject) that matches your UNIX shell on the server. For more information about network objects and how to invoke them, see GemStone Network Objects.
The following procedure assumes that you are already set up to run GemStone applications, as described in Chapter 2. In particular, you must have defined the GEMSTONE environment variable and invoked $GEMSTONE/bin/gemsetup.sh or its equivalent.
Figure 3.5 Starting a Session Process on the Server Node
Step 1. Make sure that the NetLDI and Stone are running on the server. One way to do this is to use the gslist command. For example:
% gslist -m serverName
Step 2. Log in to the application node and start the RPC version of your application (for instance, Topaz), then set UserName. For example:
% topaz
(header details omitted)
topaz> set username DataCurator
Step 3. Unless the NetLDI is running in guest mode, set the application login parameters, such as HostUserName and HostPassword. For example:
topaz> set hostusername yourLogin
topaz> set hostpassword yourPassword
Step 4. Set GemNetId to gemnetobject. Because the session process is to run on the server, be sure to include the node name in the GemNetId NRS. (It’s not necessary to set the GemStone login parameter when the Stone repository monitor runs on the same node as the Gem.) For example:
topaz> set gemnetid !@Server!gemnetobject
Step 5. Log in to the repository:
topaz> login
GemStone Password?
(startup details omitted)
successful login
topaz 1>
At this point, you are logged in to a Gem session process on the server node. That session process acts as a server to Topaz RPC and as a client to the Stone.
To Run the Gem and Stone on Different Nodes
The configuration shown in Figure 3.6 is unusual in that the RPC application and its session process are running on the same node. (While this configuration might be desirable during application development, a linked application, if it is available, probably would give better performance.)
The NetLDIs and page servers function similarly to those described for the linked application (see To Run a Linked Application on a Remote Node). In Figure 3.6, however, the NetLDI also starts the RPC Gem session process at the request of the application.
Figure 3.6 Starting the Session Process on a Remote Node
Step 1. Unless the NetLDIs are running in guest mode, decide how you will provide access so that application can start a Gem session process on the remote node.
Set the application login parameters, such as HostUserName and HostPassword, after you start the application. For example:
topaz> set hostusername yourLogin
topaz> set hostpassword yourPassword
Step 2. Log in to the remote node and start a NetLDI.
- To start the NetLDI for password authentication, make sure that $GEMSTONE/sys/netldid is owned by root and has the S bit set. Issue this command (on some operating systems, you may have to issue it as root):
Remote% startnetldi
- To start the NetLDI in guest mode (authentication is not required), make sure $GEMSTONE/sys/netldid does NOT have the S bit set. Log in as the captive account name, then issue this command:
Remote% startnetldi -g -aname
Step 3. Start the RPC version of your application (for instance, Topaz):
Remote% topaz
Step 4. Set GemNetId to gemnetobject. This network object identifies scripts that start a session process. For example:
topaz> set gemnetid gemnetobject
Step 5. Set the GemStone name, using NRS syntax to specify its location on the primary server node. Then set the UserName and log in. For example:
topaz> set gemstone !@Server!gs64stone
topaz> set username DataCurator
topaz> login
GemStone Password?
(startup details omitted)
successful login
topaz 1>
To Run the Application, Gem, and Stone on Three Nodes
The RPC application, session process, and Stone can run on three separate nodes, as shown in Figure 3.7. The application runs on its node and connects to a Gem session process on the Gem’s node. That session process communicates with the repository on the primary server node by way of a page server.
Again we see that a NetLDI must be running on each node where part of GemStone executes (but not necessarily on the application node, which runs only the RPC application).
Figure 3.7 Connecting an RPC Application, Three Nodes
The network access problem is similar to that in other RPC configurations: unless the NetLDI on the Gem node is running in guest mode, you must provide authentication to start the Gem session process.
Step 1. Unless the NetLDI on the Gem node is running in guest mode, set the application login parameters, such as HostUserName and HostPassword, after you start the application. For example:
topaz> set hostusername yourLogin
topaz> set hostpassword yourPassword
Step 2. Log in to the Gem’s node and start the NetLDI.
- To start the NetLDI for password authentication, make sure that $GEMSTONE/sys/netldid is owned by root and has the S bit set. Issue this command (on some operating systems, you may have to issue it as root):
Remote% startnetldi
- To start the NetLDI in guest mode (authentication is not required), make sure $GEMSTONE/sys/netldid does NOT have the S bit set. Log in as the captive account name, then issue this command:
Remote% startnetldi -g -aname
Step 3. Log in to the application node. Start the RPC version of your application (for instance, Topaz):
Application% topaz
Step 4. Set GemNetId to gemnetobject, and include the location, gemNode, in the NRS. For example:
topaz> set gemnetid !@gemNode!gemnetobject
Step 5. Use NRS syntax to specify the location and name of the repository. Then set the username and log in. In Topaz, for example, set GemStone and UserName:
topaz> set gemstone !@Server!gs64stone
topaz> set username DataCurator
topaz> login
GemStone Password?
(details omitted)
successful login
topaz 1>
At this point, your Topaz application on the application node has logged you in to a Gem session process on the Gem’s node, and the session process has logged in to the repository on the server.
To Run a Distributed System with a Mid-Level Cache
In a distributed system over a Wide Area Network (WAN), with many remote nodes that are topographically distant from the Stone but close to each other, a mid-level cache can improve performance for the remote sessions. In this configuration, the RPC application and the session process may be on the same or different nodes, with the mid-level cache and Stone running on separate nodes. In Figure 3.8, the Gem session process (on the leaf node) connects to a mid-level cache (on the mid-level cache node). The session process communicates with the repository (on the primary server node) by way of a page server.
When the Gem needs a page but can't find it in its local cache, it first looks in the mid-level cache. If the Gem can't find the page in the mid-level cache, it then forwards the request to the page server on the Stone’s host.
The page manager's pgsvr aggregates the responses from the pgsvrs on each of the gemSCs, and returns a combined response to the page manager. This reduces the number of round trips from the page manager to distant nodes.
If a mid-level cache is in use, then for each Gem process using the mid-level cache, all the shared caches to which the Gems are attached are subordinate to that mid-level cache.
Setting up a configuration with a mid-level cache requires that the session execute code following login to start a mid-level cache on a specified host, or to connect to an existing mid-level cache. Unlike the other remote configurations discussed in this chapter, the configuration is not established entirely by configuration settings and login arguments.
The Gem must be on a node that is remote from the Stone, and the request to connect to a mid-level cache must specify a node that is neither the Stone’s nor the Gem’s node.
If a Gem is running on the same machine as a mid-level cache, that Gem will use the mid-level cache as its local cache.
Connection Methods
System Class methods in the Shared Cache Management category allow you to connect to a mid-level cache.
midLevelCacheConnect: hostName
Attempts to connect to a mid-level cache on the specified host, if the cache already exists. The session's Gem process must be on a machine different from the machine running the Stone process.
midLevelCacheConnect: hostName cacheSizeKB: aSize
maxSessions: nSess
If a mid-level cache does not already exist on the specified host, and aSize > 0,
attempts to start the cache and connect to it. If a cache is already running on the host, this method attempts to connect to the cache and ignores the other arguments.
The size of the mid-level cache is controlled by the method argument aSize, rather than by configuration parameters (as with other shared caches).
topaz> set gemstone !@Server!gs64stone
topaz> set username DataCurator
topaz> set gemnetid !@gemNode!gemnetobject
topaz> login
GemStone Password?
<details omitted>
successful login
topaz 1>
topaz 1> printit
System midLevelCacheConnect: 'midLevelCacheNode' cacheSizeKB: 2048 maxSessions: 10
%
Reporting Methods
System Class methods in the Shared Cache Management category return lists of the shared caches on your system.
Returns a String that lists all shared caches that the Stone process is managing, not including the cache on the Stone machine.
Similar to remoteCachesReport, but only includes the mid-level caches.
3.5 Troubleshooting Remote Logins
Logging into GemStone from a remote node requires proper system configuration of the remote node and frequently requires permission for network access from the primary server to the remote node as well as from the remote node to the primary server.
- The UNIX kernel on the remote node should meet shared memory and semaphore requirements similar to those for the server, although smaller sizes may be sufficient.
- Make sure that NetLDIs are running on all nodes that require them (see the figure for your configuration). Also make sure that the NetLDIs have the same port number in /etc/services. All nodes must be listed in /etc/hosts.
- If an RPC application is being started (that is, one with a separate Gem session process), make sure the user who starts the application has entered their host username and host password, or that NetLDI is running in guest mode with a captive account. The owner of the Gem process needs an account on the node where the Gem will run and needs write access to the Gem log, typically in $HOME. Ownership and permissions for $GEMSTONE/sys/netldid must be appropriate for the authentication system in use (details can be found under Running as Root with Authentication or Running in Guest Mode with Captive Account), and the directories in /opt/gemstone must be writable.
- Make sure that the user who started the Stone has an account on the remote node. This user also must have write permission for $HOME so that log files for the remote node can be created, unless steps are taken to create the log files in another directory.
- Check any GEMSTONE environment variables for definitions that point to a previous version: env | grep GEM.
If You Still Have Trouble
If you still can’t log in to GemStone from an application on a remote node, try logging in on the server node as the same UNIX user account. We suggest that you first try a linked application, such as topaz -l, and when that works, move on to an RPC application (such as topaz or the equivalent topaz -r), still on the server.
Try Linked Topaz on the Server
A linked application on the server offers the least complicated kind of login because the server’s shared page cache is already running and no network facilities are used. Any problems are likely to involve access permission for the shared page cache or the repository extents, which can also block attempts to log in from a client node.
- Make sure that the owner of the topaz process ($GEMSTONE/bin/topaz) can access the shared page cache. Use the UNIX command ipcs -m to display permissions, owner, and group for shared memory; for example:
Server% ipcs -m
IPC status from <running system> as of Thu Mar 13 16:22:27 PDT 2014
T ID KEY MODE OWNER GROUP
Shared Memory:
m 768 0x4c177155 --rw-rw---- gsadmin pubs
Compare the owner and group returned by ipcs with the owner of the Topaz process. You can use the ps command to determine the owner; for example,
ps -ef | grep topaz. (The switches may be different on your system.)
A typical problem arises when root owns the Stone process and the shared page cache because their group ordinarily will be a special one to which Topaz users do not belong. Related problems may occur with a linked GemBuilder session. The third-party Smalltalk may be installed without the S bits and therefore may rely on group access to the shared page cache and repository. For background information, see To Set Ownership and Permissions for Session Processes.
To correct a shared page cache access failure, either change the owner and group of the setuid files or have the Stone started by a user whose primary group is one to which other GemStone users belong. Unlike file permissions, the shared page cache permissions cannot be set directly.
Try Topaz RPC on the Server
The next step should be to try running Topaz on the server with a separate Gem session process. This configuration relies on the NetLDI to start a Gem session process, and that process, not the application itself, must be able to access the shared page cache and repository extent.
- Make sure that a NetLDI is running on the server by invoking gslist. The default name is gs64ldi. If you need to start a NetLDI, the command is startnetldi.
GemStone uses the NetLDI to start a Gem session process that does repository I/O in this configuration. For the NetLDI to start processes for anyone other than its owner, it must be owned by root or it must be started in guest mode and captive account mode by someone logged in as the captive account.
The NetLDI writes a log file with the default name
/opt/gemstone/log/gs64ldi.log; this may be overridden by the startnetldi -l option. Using the gslist -x command will provide the location of all log files. The log file contents may help you diagnose problems. (See the following discussion, “Check NetLDI Log Files.“)
- Make sure that the owner of the resulting Gem session process ($GEMSTONE/sys/gem) can access the shared page cache and extent0.dbf through group membership or S bits. The troubleshooting is the same as that given here for the topaz executable.
The user who starts topaz (or the NetLDI captive account when it is in use) must have write permission for $HOME so that the session process can create a log file there. (For a workaround for situations where write permission is not allowed, see To Set a Default NRS.)
Check NetLDI Log Files
Troubleshooting on a distributed GemStone system can be complicated. What looks like a hung process may actually be caused by incorrect NRS syntax or by another node on the network going down. The information for analyzing problems may be found in log files on all the nodes used by GemStone.
Where the log file messages include NRS strings, be sure to check their syntax. The problem may be as simple as an incorrect NRS or one that was not expanded by the shell as you intended.
If you can’t identify the problem from the standard log messages, try running the NetLDI in debug mode, which puts additional information in the log. The command line is startnetldi [netLdiName] -d. For further details, see startnetldi.