5. Connecting Distributed Systems

Previous chapter

Next chapter

This chapter tells how to set up GemStone/S 64 Bit in a distributed environment:

Overview
An introduction to the GemStone processes and network objects that facilitate distributed GemStone systems.

Configuring GemStone on Remote Nodes
Examples for setting up typical distributed client-server configurations.

Troubleshooting Remote Logins
What to do when there are problems with remote logins.

Note that GemStone also supports distributed configurations in which all connections are initiated from the Stone and secured with X.509 certificates. The architectural details, processes, login parameters, and sequence of operations is entirely different with X509-secured GemStone, and are described in the GemStone/S 64 Bit X509-Secured GemStone System Administration Guide

5.1  Overview

GemStone is designed to be highly efficient and easy to use in a networked environment with many nodes. However, this requires additional steps by the system administrator so that the server processes can be found and accessed, and so server processes can connect to one another. Because processes are running on different nodes, the log files are spread throughout the network, and troubleshooting may become more complicated.

Figure 5.1 and Figure 5.2 show two typical distributed configurations in which an application on a remote node is logged in to a repository and Stone repository monitor running on a server node.

Distributed configuration with Local Gem

In Figure 5.1, an application communicates with a Gem session process on the server node by way of RPC calls. This configuration lets the Gem execute Smalltalk code in the repository without first bringing complex objects across the network. The Gem directly accesses the shared page cache that was started by the Stone repository monitor. This configuration is considered a “Local” Gem, since the Gem is local to the Stone.

Figure 5.1   Gem Session on Server Node

Distributed configuration with Remote Gem

In Figure 5.2, the application and the Gem are linked in a single process that runs on the remote node (i.e, not the Stone’s node). This configuration avoids the overhead of RPC calls, but may increase network traffic if large objects must be brought across the network.

When a Gem is on a remote node, it needs a shared page cache on that node. This is started by the first Gem on that node. Page server processes are also needed to handle pages between the two nodes.

Figure 5.2   Gem Session on Remote Node

Network

The nodes in your system can be any combination of GemStone-supported platforms, as long as they have a network connection. Inter-node communication is handled via sockets.

Nodes in which the Stone, or a Gem process, will run must be platforms supported by GemStone/S 64 Bit.

Client-process only nodes such as the remote node in Figure 5.1 can include supported client platforms, such as Windows, as well as server platforms.

GemStone NetLDIs

The NetLDIs are the glue holding a distributed GemStone system together. Each NetLDI spawns other GemStone processes on request, and reports the location of GemStone services on its node to remote processes that must connect to those services.

In a distributed system, each node where a Stone repository monitor or Gem session process runs must have its own NetLDI process running. However, nodes that run only linked applications, or client applications without a Gem on that node, do not require a NetLDI.

The NetLDI listens on a well-known port for requests for RPC logins or other services.

NRS Syntax

GemStone uses Network Resource Strings (NRS) to specify the name and location of each part of the GemStone system. NRS is widely used in utility commands and in login parameters, to specify the location and name of the Stone and NetLDI, and where the other GemStone processes should run.

The most useful parts to know are:

!@node#netldi:netldi!stoneOrGemService
  • The “@node” can be omitted for the local node, and you may use an IP address instead of the name of the node.
  • the “#netldi:netldi” can be omitted if the netldi is running with the default name gs64ldi and that name is recorded in the services database. The netldi can be specified by name or by port number.
  • stoneOrGemService is the name of the running Stone, or gemnetobject or another Gem service. This part is required; if you omit both node and netldi, you do not need the ! dividers; you can use simply stoneOrGemService.

Appendix C, “Network Resource Strings (NRS)” describes NRS in detail.

In particular, you may find it very useful to set the GEMSTONE_NRS_ALL environment variable if you are using a NetLDI name other than the default.

Stone

When logging into to a Stone on the local machine, the stone name is sufficient. If the Stone is on a separate node, you can specify the location of the stone, and the name of the NetLDI on the Stone’s node, using NRS.

For example, to connect to a stone named devstone on the node oboe, with the default-named NetLDI gs64ldi:

topaz> set gemstone !@oboe!devstone

Connecting to the RPC Gem

When a client requests an RPC login, a Gem session process needs to be spawned to fullfil the login request. To do this, the login parameters include the gem service (in topaz, the gemnetid), usually gemnetobject, gemnetdebug, or the name of a custom gem script.

For example,

topaz> set gemnetid gemnetobject

Using gemnetobject alone as here, specifies to run the gem on the same node as the client application.

By specifying an NRS, in the gem service login parameter that includes the node and the NetLDI on which you want the Gem to run, you can instruct GemStone on which processes need to be started and where. This can be the same node as the Stone. It can also be on another node entirely, provided GemStone is installed and a NetLDI is running on that node.

For example, if you intend the gem to run on a machine named lark, with the default-named NetLDI gs64ldi:

topaz> set gemnetid !@lark!gemnetobject

Note that if the gem service NRS includes a node name and the stone NRS does not include a node name, then the Stone is expected to be on the node in the Gem NRS, not on the client node.

Shared Page Cache

There is always a shared page cache on the Stone’s node. Any Gems that are on the Stone’s node (local Gems) will use that shared page cache.

When a GemStone login specifies that a Gem should run on a remote node, then a shared page cache and the associated processes, shared page cache monitor and free frame page server, need to be started on that node. Normally, the cache is started up by a NetLDI running on the remote node, although a linked session (but not an RPC session) can also initiate the startup.

When the remote Gem wants to access a page in the repository, it first checks the shared page cache on its own node. If the page is not found, the Gem communicates with the Remote Gem Page Server on the server node, checking in the shared cache on the server node and if necessary, reading the page from the disk.

Figure 5.3   Shared Page Cache with Remote Gem

GemStone Page Servers

Gems on remote nodes require page server processes to support operations between the remote Gem and cache and the Stone.

  • When a Gem connects to a repository extent across the network, it requires a Remote Gem Page Server on the Stone’s node, associated with the remote node. This is spawned if it is not already running. This page server looks up pages in the Stone’s cache and performs extent reads, on behalf of the gem. There is one page server per remote node; each Gem on that node is services by a thread within that page server.
  • On the remote node, a Remote Cache Page Server supports the Stone in managing the remote shared page cache, by starting the cache and removing pages that are being recycled.

Port use in GemStone

GemStone’s interprocess and internode communications are established via socket connections. A number of key GemStone processes listen on well known ports for a connection, which initiates the handshakes that establish interprocess socket connections.

Processes that listen on well known ports include the NetLDI, the Stone, and the Shared Page Cache Monitor. These named GemStone processes create a serviceName..LCK at startup, in the directory /opt/gemstone/locks or another location as described here. This file encodes the well-known port that the process is listening on. When the NetLDI (for example) contacts a particular Stone, it first looks up the well-known port for that Stone by locating the lock file for that Stone, then contacts the Stone on that port.

The NetLDI’s port is specified on startup in a number of ways, as described here.

The ports used by the Stone and Shared Page Cache Monitor can be specified using STN_WELL_KNOWN_PORT_NUMBER and SHR_WELL_KNOWN_PORT_NUMBER configuration options, respectively. These must be valid port numbers that are not already in use.

Disrupted Communications

Several incidents can disrupt communications between the GemStone server and remote processes in a distributed configuration. This can include network glitches, network overload, crashes or unexpected shut down of a remote process or the entire remote node, or of the network channel itself.

GemStone ordinarily depends on the network protocol keepalive option to detect that a remote process no longer exists because of an unexpected event. The keepalive interval is set globally by the operating system, typically at two hours. When that interval expires, the GemStone process tries to obtain a response from its partner. The parameters governing these attempts also are set by the operating system, with up to 10 attempts in 15 minutes being typical. If no response is received, the local GemStone process acts as if its partner was terminated abnormally. The keepalive interval may need to be adjusted.

Firewalls between GemStone processes require that the communications ports be configured to be open.

5.2  Configuring GemStone on Remote Nodes

If you are only running local Gems; in other words, only the client application will run on the remote node, most of the configuration for the Gem is on the server node. However, if you will run Gems or Linked sessions on nodes remote from the Stone, then more configuration is required on the remote node.

Local Gems only

If you are only running Gems on the same node as the Stone, there are only a few steps required on the remote node.

  • Install the client shared libraries. This can be done by a complete server installation, or by copying the small number of required library files to the client, and configuring your client search paths appropriately.

Details on how to do this, and the various options, are in the GemStone/S 64 Bit Installation Guide for the client platform.

  • If you are using named NetLDIs, you must configure your client system services database to map the same NetLDI name to the same port as defined on the server. If you are using a local services database, edit /etc/services or, on Windows, \WINDOWS\system32\drivers\etc\services to add the appropriate entry.

Remote Gems

Configuring GemStone so that Gem sessions or linked applications can run on the remote node is much the same as configuring the Gem and cache a session process on the server, which is described in Chapter 3, “Configuring Gem Session Processes”.

The remote node will need access to a number of executables and shared libraries within the server installation. You can either repeat the installation from the GemStone distribution media, or mount the directory on the server node that contains $GEMSTONE.

Some additional details points to keep in mind:

  • A remote node (on which a Gem is running) must have its kernel configured for shared memory similarly to how it is configured on the primary server node.
  • Only nodes running a Stone need a GemStone key file, not nodes running remote sessions.
  • It’s best if each node has its own directory for /opt/gemstone/log and /opt/gemstone/locks (or /usr/gemstone/log and /usr/gemstone/locks, or other location under $GEMSTONE_GLOBAL_DIR). If these directories are on an NFS-mounted partition, make sure that two nodes are not using the same directories. Each Stone and NetLDI needs a unique lock file. Shared log files may make it impossible to diagnose problems.
  • If you are using named NetLDIs, you must configure your client system services database to map the same NetLDI name to the same port as defined on the server. This is not necessary if you specify NetLDI by port number. See the Installation Guide for details.
  • Unless you run the NetLDIs in guest mode with a captive account, users must have an account on, and authorized network access to, all nodes that are part of the GemStone network for the repository they will be using. This includes the nodes on which the Stone Repository monitor and the user’s Gem session process reside.

Unless you run the NetLDIs in guest mode with a captive account, the user who starts the Stone repository monitor ordinarily needs an account on every node where a Gem session process will run.

Configuration Examples

GemStone supports several configurations in which the application communicates with the Gem session process by using remote procedure calls (RPCs), as well as linked applications. This section presents examples that illustrate the processes and interprocess connections within the following distributed applications:

Determine if you will run RPC or Linked application

GemStone shared libraries are provided in two variants: linked and RPC. In order to login a linked session, you must be running an application bound to the linked libraries, such as topaz -l. To login an RPC session, you may run an application bound with the RPC libraries or with the linked applications.

A linked application may have only one linked login, but may login RPC sessions in addition to (or instead of) the linked login. RPC sessions applications are not able to login linked sessions.

Set the environment

The instructions assume that you are already set up to run GemStone applications, as described in Chapter 3. In particular, you must have defined the GEMSTONE environment variable and invoked $GEMSTONE/bin/gemsetup.sh or its equivalent, so the system search path include $GEMSTONE/bin.

Configure NetLDI and authentication mechanism

For all these examples, you must have a NetLDI running on the Stone’s host. In many cases you will also need a NetLDI on the remote node or nodes. These should be configured according to your security requirements, as described under NetLDI configuration.

RPC Application on a Remote Node with Local Gem

If the Gem session process is going to run on the server node (as shown in Figure 5.4), an RPC application uses a NetLDI on that node to start a Gem session process. Unless the NetLDI is running in guest mode with a captive account, the application user must provide authentication to the NetLDI. The login parameters include the specification for the Gem network object (gemnetobject). For more information about network objects and how to invoke them, see Connecting to the RPC Gem.

Figure 5.4   RPC Application on a Remote Node with remote Gem

To login an RPC session with the Gem on the Stone’s host, set the login parameters as follows:

  • Set the Stone’s name:
topaz> set gemstone stoneName

The hostname is optional in this case; by default, it will use the host specified by the NRS in the gem service parameter (gemnetid)

  • Set GemStone user name and password:
topaz> set username DataCurator
topaz> set password swordfish

You may omit the password; you will be prompted for it on login.

  • Set host authentication, if your NetLDI is not running in guest mode:
topaz> set hostusername unixUserAccountName
topaz> set hostpassword passwordForUnixUserAccount
  • Set the gem service, specifying the stone’s host name, and the name of the NetLDI if necessary. For example, either one of the following:
topaz> set gemnetid !@serverhost!gemnetobject
topaz> set gemnetid !@serverhost#netldi:NetLdiName!gemnetobject

Linked Application on a Remote Node

Figure 5.5 shows how a linked application on a remote node communicates with a Stone and repository on the primary server node.

Figure 5.5   Connecting a Linked Application to a Remote Server

In this configuration, a remote, subordinate shared page cache is started on the remote node, along with associated server processes. The configuration of this remote cache is set by the first session to log in on this node. After the last session on this node logs out, the cache will shut down.

In addition, this configuration requires a Remote Gem page server on the server node, which reads pages from the repository on behalf of the gem.

Because the shared page cache is readable and writable only by its owner and members of the same group (protection 660), the user running the application may need to belong to that group. See Shared Page Cache.

Set the login parameters as follows:

  • Set the Stone’s name, one of the following:
topaz> set gemstone !@serverhost!stoneName
topaz> set gemstone !@serverhost#netldi:NetLdiName!stoneName
  • Set GemStone user name and password.
topaz> set username DataCurator
topaz> set password swordfish

You may omit the password; you will be prompted for it on login.

  • Set host authentication, if your NetLDI is not running in guest mode:
topaz> set hostusername unixUserAccountName
topaz> set hostpassword passwordForUnixUserAccount
  • Do not set the gem service; doing so specifies that the login will be RPC, rather than linked.

RPC Application on a Remote Node with Remote Gem

The configuration shown in Figure 5.6 shows the RPC application and its session process running on the same node. This configuration is similar to a linked application, but in this case, a NetLDI is required on the remote node in order to launch the RPC Gem session process.

Figure 5.6   Starting the Session Process on a Remote Node

 
  • Set the Stone’s name, one of the following:
topaz> set gemstone !@serverhost!stoneName
topaz> set gemstone !@serverhost#netldi:NetLdiName!stoneName

You must specify the Stone’s host, since the otherwise it will assume the Stone is on the same node as the Gem, given in the gemnetid’s NRS.

  • Set GemStone user name and password.
topaz> set username DataCurator
topaz> set password swordfish

You may omit the password; you will be prompted for it on login.

  • Set host authentication, if your NetLDI is not running in guest mode.
topaz> set hostusername unixUserAccountName
topaz> set hostpassword passwordForUnixUserAccount
  • Set the gem service, specifying the stone’s host name, and the name of the NetLDI if necessary. For example, one of the following:
topaz> set gemnetid !@remotenode!gemnetobject
topaz> set gemnetid !@remotenode#netldi:NetLdiName!gemnetobject

RPC Application, Gem, and Stone on Three Nodes

The RPC application, session process, and Stone can run on three separate nodes. The application runs on its node and connects to a Gem session process on the Gem’s node, which is sometimes referred to as the Gem Server. That session process communicates with the repository on the primary server node by way of a page server.

Again we see that a NetLDI must be running on each node where part of GemStone executes (but is not required on the application node, which runs only the RPC application).

The following diagram shows two separate RPC applications connecting to Gems running on the same node, the Gem Server node.

Figure 5.7   RPC Applications using a Gem Server Node

The directions for this configuration are the same as for the previous configuration, RPC Application on a Remote Node with Remote Gem; however, the application itself is executing on a third node.

  • Set the Stone’s name, one of the following:
topaz> set gemstone !@serverhost!stoneName
topaz> set gemstone !@serverhost#netldi:NetLdiName!stoneName

You must specify the Stone’s host, since the otherwise it will assume the Stone is on the same node as the Gem, given in the gemnetid’s NRS.

  • Set GemStone user name and password.
topaz> set username DataCurator
topaz> set password swordfish

You may omit the password; you will be prompted for it on login.

  • Set host authentication, if your NetLDI is not running in guest mode.
topaz> set hostusername unixUserAccountName
topaz> set hostpassword passwordForUnixUserAccount
  • Set the gem service, specifying the stone’s host name, and the name of the NetLDI if necessary. For example, one of the following:
topaz> set gemnetid !@remotenode!gemnetobject
topaz> set gemnetid !@remotenode#netldi:NetLdiName!gemnetobject

Distributed System with a Mid-Level Cache

In a distributed system over a Wide Area Network (WAN), with many remote nodes that are topographically distant from the Stone but close to each other, a mid-level cache can improve performance for the remote sessions. In this configuration, the RPC application and the session process may be on the same or different nodes, with the mid-level cache and Stone running on separate nodes. In Figure 3.8, the Gem session process (on the leaf node) connects to a mid-level cache (on the mid-level cache node). The session process communicates with the repository (on the primary server node) by way of a page server.

When the Gem needs a page but can't find it in its local cache, it first looks in the mid-level cache. If the Gem can't find the page in the mid-level cache, it then forwards the request to the page server on the Stone’s host.

The Stone’s page server aggregates the responses from the page servers on each of the Gem’s shared caches, and returns a combined response to the Stone. This reduces the number of round trips from the Stone to distant nodes.

If a mid-level cache is in use, then for each Gem process using the mid-level cache, all the shared caches to which the Gems are attached are subordinate to that mid-level cache.

Figure 5.8   An Application with a Mid-Level Cache

Setting up a configuration with a mid-level cache requires that the session execute code following login to start a mid-level cache on a specified host, or to connect to an existing mid-level cache. Unlike the other remote configurations discussed in this chapter, the configuration is not established entirely by configuration settings and login arguments.

The Gem must be on a node that is remote from the Stone, and the request to connect to a mid-level cache must specify a node that is neither the Stone’s nor the Gem’s node.

If a Gem is running on the same machine as a mid-level cache, that Gem will use the mid-level cache as its local cache.

Connection Methods

System Class methods in the Shared Cache Management category allow you to connect to a mid-level cache.

midLevelCacheConnect: hostName

Attempts to connect to a mid-level cache on the specified host, if the cache already exists. The session's Gem process must be on a machine different from the machine running the Stone process.

midLevelCacheConnect: hostName cacheSizeKB: aSize
maxSessions: nSess

If a mid-level cache does not already exist on the specified host, and aSize > 0,
attempts to start the cache and connect to it. If a cache is already running on the host, this method attempts to connect to the cache and ignores the other arguments.

The size of the mid-level cache is controlled by the method argument aSize, rather than by configuration parameters (as with other shared caches).

For example,

topaz> set gemstone !@Server!gs64stone
topaz> set username DataCurator
topaz> set gemnetid !@gemNode!gemnetobject
topaz> login
GemStone Password?
<details omitted>
successful login
topaz 1>
topaz 1> printit
  System midLevelCacheConnect: 'midLevelCacheNode'
    cacheSizeKB: 16000 maxSessions: 12
%
true
Reporting Methods

To determine what node a session is using as a mid level cache, use System class >> midLevelCacheAddress.

System Class methods in the Shared Cache Management category return lists of the shared caches on your system.

System class >> remoteCachesReport
Returns a String that lists all shared caches that the Stone process is managing, not including the cache on the Stone machine.

System class >> midLevelCachesReport
Similar to remoteCachesReport, but only includes the mid-level caches.

5.3  Troubleshooting Remote Logins

Logging into GemStone from a remote node requires proper system configuration of the remote node and frequently requires permission for network access from the primary server to the remote node as well as from the remote node to the primary server.

It can be useful to enable debug mode for each of the NetLDIs, which will report additional details about connection attempts. Restart the NetLDI with the -d option, or use the netldidebug.

How the Login Process starts Session Processes

When a remote client applications requests a login, it communicates with the appropriate processes on the server. This process is ordinarily transparent to the user.

The following examples show the sequence of steps that occur during login in order to start the required processes and connect them appropriately.

RPC Application with Gem on Server Node

The simplest configuration for a distributed configuration is a client application running on a remote node, that connects to a RPC Gem session on the server node. The following examples describe the sequence of steps that connect this configuration during login.

Figure 5.9   Connecting an RPC Application with Gem on Server Node

1. The client application contacts the NetLDI on the server node and requests a session.

2. The NetLDI invokes gemnetobject or a similar script to start an RPC Gem.

3. The Gem performs the login to the Stone.

4. The application client connects to the Gem.

5. The Gem attaches extent files.

Linked Application on Remote Node

A remote linked login is more involved, since the remote shared page cache and associated processes are also started up.

In this example, there is a NetLDI running on the remote node. This is not required for a linked login; in the case of a linked login with no NetLDI running, the tasks performed by the NetLDI are handled by the linked gem. .

Figure 5.10   Connecting a Linked Application to a Remote Server

1. The client application contacts the NetLDI on the server node, requesting the Stone's listening port.

2. The client application contacts the Stone on its listening port, requesting login.

3. Stone requests the Remote Cache page server on remote node, if there is not already an SPC on the remote node. If there is a cache already there, continue with step 6.

4. The NetLDI on the remote node launches the Remote Cache page server.

5. The Remote Cache page server launches the Shared Page Cache Monitor, which initializes the shared page cache.

6. Stone connects to the Remote Cache page server.

7. The client application requests a Remote Gem page server, if there is not already a Remote Gem page server for this node. If a Remote Gem page server already present, continue with step 9.

8. The NetLDI on the server node launches the Remote Gem page server.

9. The client application connects to the Remote Gem page server.

10. A newly launched Remote Gem page server attaches the extent files.

If You Still Have Trouble

If you still can’t log in to GemStone from an application on a remote node, try logging in on the server node as the same UNIX user account. We suggest that you first try a linked application, such as topaz -l, and when that works, move on to an RPC application (such as topaz or the equivalent topaz -r), still on the server.

Try Linked Topaz on the Server

A linked application on the server offers the least complicated kind of login because the server’s shared page cache is already running and no network facilities are used. Any problems are likely to involve access permission for the shared page cache or the repository extents, which can also block attempts to log in from a client node.

  • Make sure that the owner of the topaz process ($GEMSTONE/bin/topaz) can access the shared page cache. Use the UNIX command ipcs -m to display permissions, owner, and group for shared memory; for example:
os$ ipcs -m
IPC status from <running system> as of Mon Aug 07 16:22:27 PDT 2023
T     ID     KEY        MODE       OWNER    GROUP
Shared Memory:
m    768 0x4c177155 --rw-rw----  gsadmin     pubs

Compare the owner and group returned by ipcs with the owner of the Topaz process. You can use the ps command to determine the owner; for example,
ps -ef | grep topaz. (The switches may be different on your system.)

A typical problem arises when root owns the Stone process and the shared page cache because their group ordinarily will be a special one to which Topaz users do not belong. Related problems may occur with a linked GemBuilder session. The third-party Smalltalk may be installed without the S bits and therefore may rely on group access to the shared page cache and repository. For background information, see File Permissions.

To correct a shared page cache access failure, either change the owner and group of the setuid files or have the Stone started by a user whose primary group is one to which other GemStone users belong. Unlike file permissions, the shared page cache permissions cannot be set directly.

  • Make sure the owner of the Topaz process has read-write access to the extent files.
Try Topaz RPC on the Server

The next step should be to try running Topaz on the server with a separate Gem session process. This configuration relies on the NetLDI to start a Gem session process, and that process, not the application itself, must be able to access the shared page cache and repository extent.

  • Make sure that a NetLDI is running on the server by invoking gslist. The default name is gs64ldi. If you need to start a NetLDI, the command is startnetldi.

GemStone uses the NetLDI to start a Gem session process that does repository I/O in this configuration. For the NetLDI to start processes for anyone other than its owner, it must be owned by root or it must be started in guest mode and captive account mode by someone logged in as the captive account.

The NetLDI writes a log file with the default name
/opt/gemstone/log/gs64ldi.log; this may be overridden by the startnetldi -l option. Using the gslist -x command will provide the location of all log files. The log file contents may help you diagnose problems. (See the following discussion, “Check NetLDI Log Files.“)

  • Make sure that the owner of the resulting Gem session process ($GEMSTONE/sys/gem) can access the shared page cache and extent0.dbf through group membership or S bits. The troubleshooting is the same as that given here for the topaz executable.
  • The user who starts topaz (or the NetLDI captive account when it is in use) must have write permission for $HOME so that the session process can create a log file there. (For a workaround for situations where write permission is not allowed, see Controlling log file directory locations.)

Check NetLDI Log Files

Troubleshooting on a distributed GemStone system can be complicated. What looks like a hung process may actually be caused by incorrect NRS syntax or by another node on the network going down. The information for analyzing problems may be found in log files on all the nodes used by GemStone.

Where the log file messages include NRS strings, be sure to check their syntax. The problem may be as simple as an incorrect NRS or one that was not expanded by the shell as you intended.

If you can’t identify the problem from the standard log messages, try running the NetLDI in debug mode, which puts additional information in the log. The command line is startnetldi [netLdiName] -d. For further details, see startnetldi.

 

Previous chapter

Next chapter