4. Running GemStone

To Start GemStone

Follow these steps to start GemStone following installation or an orderly shutdown. (To recover from an abnormal shutdown, refer to Recovering from an Unexpected Shutdown.)

Step 1. Set the GEMSTONE environment variable to the full pathname (starting with a slash) of the directory where GemStone is installed. Ordinarily this directory has a name like GemStone64Bit3.3.0-x86_64.Linux (depending on your platform). For example:

$ GEMSTONE=/users/GemStone64Bit3.3.0-x86_64.Linux

$ export GEMSTONE

If you have been using another version of GemStone, be sure you update or unset previous settings of these environment variables:

• GEMSTONE
• GEMSTONE_SYS_CONF
• GEMSTONE_EXE_CONF
• GEMSTONE_NRS_ALL

Step 2. Set your UNIX path. One way to do this is to use one of the gemsetup scripts. There is one version for users of the Bourne and Korn shells and another for users of the C shell. These scripts also set your man page path to include the GemStone man pages. Note that these scripts append to the end of your path or man path; you will need to manually remove references to older versions of GemStone.

(Bourne or Korn shell)
$ . $GEMSTONE/bin/gemsetup.sh

or (C shell)
% source $GEMSTONE/bin/gemsetup.csh

Step 3. Start GemStone by using the startstone command:

% startstone [gemStoneName]

where gemStoneName is optional and is the name you want the repository monitor to have. The default name is gs64stone. See startstone for additional information.

To Troubleshoot Stone Startup Failures

If the Stone repository monitor fails to start in response to a startstone command, it’s likely that the cause is one of the following. Inspect the Stone log for clues (the default location is $GEMSTONE/data/gs64stone.log).

• The GemStone key file is missing or invalid.
• The shared page cache cannot be attached.
• A problem with the extents: a missing extent or one that is in use by another process.
• A problem with transaction logs: a log needed for recovery is missing, or the log directory or device does not exist.
• The repository has become corrupted.

GemStone is unable to open the file

!TCP@pelican#dbf!/users/GemStone/data/extent0.dbf.

reason = File = /users/GemStone/data/extent0.dbf

DBF Op: Open; DBF Record: -1;

Error: open() failure; System Codes: errno=2, ENOENT, The file or

directory specified cannot be found

An error occurred opening the repository for exclusive access.

Stone startup has failed.

GemStone is unable to open the file

!TCP@pelican#dbf!/users/GemStone/data/extent0.dbf.

reason = File = /users/GemStone/data/extent0.dbf

DBF Op: Open; DBF Record: -1; Error: exclusive open:  File is open

by another process.; System Codes:

errno=11, EAGAIN, No more processes (due to process table full, 

user quotas, or insufficient memory)

An error occurred opening the repository for exclusive access.

Stone startup has failed.

An error occurred in recovery for extentId 2:

fileName= !TCP@pelican#dbf!/users/GemStone/data/extent1.dbf

File already exists; you must delete it before recovery can succeed.

Extent 0 was not cleanly shutdown; recovery is needed.

<Repository startup statistics>

Repository startup is from checkpoint = (fileId 6, blockId 3)

ERROR: cannot find log file(s) to recover repository.

To proceed without tranlogs and lose transactions committed

since the last checkpoint use "-N" switch on your startstone

command.

An error occurred when attempting to start repository recovery.

Waiting for aiowrites to complete

Stone startup has failed.

To Troubleshoot NetLDI Startup Failures

If the NetLDI service fails to start in response to a startnetldi command, it’s likely that the cause is one of the following:

• The NetLDI is to run as root but the guest mode option is specified. This combination is not allowed.
• The account starting the NetLDI does not have permission to create or append to its log file.
• The account starting the NetLDI does not have read and execute permission for $GEMSTONE/sys/netldid.

Check the NetLDI log for clues. By default, the NetLDI log (netLdiName.log) is located in /opt/gemstone/log. On some systems, this file may be located in /usr/gemstone/log, and may be overridden using the -l option to the startnetldi command, or by setting $GEMSTONE_GLOBAL_DIR.

To Define a GemStone Session Environment

In order to start a GemStone session, the following must be defined through your operating system environment:

• Where GemStone is installed

All GemStone users must have a GEMSTONE environment variable that points to the GemStone installation directory, such as
/installDir/GemStone64Bit3.3.0-x86_64.Linux (depending on your platform). The directory $GEMSTONE/bin should be in your search path for commands. For an example, see the next topic, “To Start a Linked Session”.

• Which configuration parameters to use

Because each GemStone session can have its own configuration file, some users may need a second environmental variable, such as GEMSTONE_EXE_CONF. If no other file is found, the session uses system defaults. For further information, see How GemStone Uses Configuration Files.

To Start a Linked Session

The following steps show how to start a linked application (here, the linked version of Topaz). The steps for setting the GEMSTONE environment variable and the operating system path for a session are the same as those given here for starting a repository monitor. They are repeated here for convenience.

The procedure assumes that the Stone repository monitor has already been started and has the default name gs64stone.

Step 1. Set the GEMSTONE environment variable to the full pathname (starting with a slash) of the directory where GemStone is installed. Ordinarily this directory has a name like GemStone64Bit3.3.0-x86_64.Linux (depending on your platform). For example:

$ GEMSTONE=/installDir/GemStone64Bit3.3.0-x86_64.Linux

$ export GEMSTONE

If you have been using another version on GemStone, be sure you update or delete previous settings of these environment variables:

• GEMSTONE
• GEMSTONE_SYS_CONF
• GEMSTONE_EXE_CONF
• GEMSTONE_NRS_ALL

Step 2. Set your UNIX path. One way to do this is to use one of the gemsetup scripts. There is one version for users of the Bourne and Korn shells and another for users of the C shell. These scripts also set your man page path to include the GemStone man pages.

(Bourne or Korn shell)
$ . $GEMSTONE/bin/gemsetup.sh

or (C shell)
% source $GEMSTONE/bin/gemsetup.csh

Step 3. Start linked Topaz:

% topaz -l

Step 4. Set the UserName login parameter:

topaz> set username DataCurator

Step 5. Log in to the Gem session. It will query you for the password.

topaz> login

GemStone Password?

[Info]: LNK client/gem GCI levels = 860/860

[Info]: libssl-3.3.0-64.so: loaded

[Info]: User ID: DataCurator

[Info]: Repository: gs64stone

[Info]: Session ID: 6

[Info]: GCI Client Host: <Linked>

[Info]: Page server PID: -1

[Info]: Login Time: 03/12/2015 11:36:47.508 PDT

Gave this VM preference for OOM killer, Wrote to /proc/6923/oom_adj value 4

[03/12/2015 11:36:47.510 PDT]

  gci login: currSession 1 linked session

successful login

topaz 1> 

At this point, you are logged in to a Gem session process, which is linked with the application. The session process acts as a server to Topaz and as a client to the Stone. Information about Topaz is in the manual GemStone Topaz Programming Environment.

When you are ready to end the GemStone session, you can log out of GemStone and exit Topaz in one step by invoking the Topaz exit command:

topaz 1> exit

To Start an RPC Session

The following steps show how to start an RPC application (here, the RPC version of Topaz) on the server node. The procedure assumes that the Stone is running under the default name gs64stone and that you are already set up to run a GemStone session as described in Step 1 and Step 2 of the previous example (“To Start a Linked Session”).

Sessions that login RPC use SRP (Secure Remote Password) and SSL to authenticate passwords for login. If the Gem is running on the server node, the connection reverts to normal socket communication after login completes.

The following steps demonstrate an RPC login from topaz:

Step 1. Use gslist to find out if a NetLDI is already running. The default name for the NetLDI is gs64ldi.

% gslist

Status Version  Owner      Started    Type  Name

------ -------- --------- ------------ ------ ----

exists 3.3.0    gsadmin   Mar 11 12:02 cache gs64stone~1c9fa07f0412665

exists 3.3.0    gsadmin   Mar 11 12:02 Stone  gs64stone

exists 3.3.0    gsadmin   Mar 11 10:13 Netldi gs64ldi

If necessary, start a NetLDI following the instructions under Starting a NetLDI.

Step 2. Start the RPC application (such as Topaz), then set the UserName.

topaz> set username DataCurator

Step 3. Unless the NetLDI is running in guest mode with a captive account, set the application login parameters, such as HostUserName and HostPassword, after you start the application. For example:

topaz> set hostusername yourUnixId

topaz> set hostpassword yourPassword

Step 4. Set GemNetId (the name of the Gem service to be started) to gemnetobject. This script starts the separate Gem session process for you. For example:

topaz> set gemnetid gemnetobject

Step 5. Log in to the GemStone session.

topaz> login

GemStone Password?

[Info]: libssl-3.3.0-64.so: loaded

[03/12/2015 11:36:47.777 PDT]

  gci login: currSession 1 rpc gem processId 6943

successful login

topaz 1> 

At this point, you are logged in through a separate Gem session process that acts as a server to Topaz RPC and as a client to the Stone repository monitor.

When you are ready to end the GemStone session, you can log out of GemStone and exit Topaz by in one step by invoking the Topaz exit command:

topaz 1> exit

To Troubleshoot Session Login Failures

Several factors may prevent successful login to the repository:

• Your GemStone key file may establish a maximum number of user sessions that can simultaneously be logged in to GemStone. (Note that a single user may have multiple GemStone sessions running simultaneously.) The limit itself is encoded in the keyfile used to start the stone (by default, $GEMSTONE/sys/gemstone.key), and reported in the stone log on startup. Look for a line like this:

SESSION MAX: The licensed concurrent session max is 10.

• The STN_MAX_SESSIONS configuration option can restrict the number of logins to fewer than a particular key file allows. An entry in the Stone log file shows the maximum at the time the Stone started. Look for a line like this:

SESSION CONFIGURATION: The maximum number of concurrent sessions is 40

• The SHR_PAGE_CACHE_NUM_PROCS configuration option restricts the number of sessions that can attach to a particular shared page cache. This is computed based on the setting for STN_MAX_SESSIONS.

Multi-threaded operations use additional slots for their working threads while they are executing. If you are close to your session limit, these operations may prevent other sessions from logging in.

• The UNIX kernel must provide two semaphores for each session that wants to attach to the shared page cache. See Reviewing Kernel Tunable Parameters.
• The UNIX kernel file descriptor limit can restrict the number of sessions, and GemStone executables attempt to raise that limit. For information, see the discussions under Estimating Server File Descriptor Needs and Estimating Session File Descriptor Needs. On some operating systems, you can examine the kernel limit by invoking ulimit.
• The owner of the Gem or a linked application process must have write access to the extent file and to the shared page cache. Use the UNIX command ipcs -m to display permissions, owner, and group for shared memory. For example:

server% ipcs -m

IPC status from <running system> as of Mon March 9 16:21:08 PDT 2015

T      ID      KEY         MODE         OWNER      GROUP

Shared Memory:

m      25089   0x4c000ed5  --rw-rw----  gsadmin    users

Typical problems occur with linked applications, which may be installed without the S bit and therefore rely on group access to the shared page cache and the repository.

• If the session is using a separate (RPC) gem process, see Troubleshooting Remote Logins.

Stopping Logged-in Sessions

Privileges required: SessionAccess and SystemControl

There are a number of methods on System class that can be used to stop a specific session, or all sessions:

stopSession: aSessionId
Stop the specified session; any transactions that the session was in are aborted, and the session is terminated. This method does not stop the GcGems or SymbolGem.

terminateSession: aSessionId timeout: timeoutSeconds
Stop the specified session; any transactions that the session was in are aborted, and the session is terminated. Waiting up to timeoutSeconds for the session to complete terminating before returning. This method can be used to stop the GcGems. but not the SymbolGem.

stopUserSessions
Stops all sessions other than system Gems; does not stop the GcGems nor SymbolGem. Any transactions that any of the sessions were in are aborted.

NOTE
Be aware that it may take as long as a minute for a session to terminate after you send stopSession:.If the Gem is responsive, it usually terminates within milliseconds. However, if a Gem is not active (for example, sleeping or waiting on I/O), the Stone waits one minute for it to respond before forcibly logging it out. You can bypass this timeout by sending terminateSession:timeout:

Normal Shutdown Message

If you see a shutdown message in the system log file, GemStone has stopped in response to a stopstone command or a Smalltalk System shutdown method:

--- 04/20/15 17:16:35 PDT ---

LoginsSuspended is set to 1 by DataCurator from session 5

SHUTDOWN command was received from user DataCurator session 5 gem processId 29188.

Waiting for aiowrites to complete

Waiting for NetRead thread to stop

Now stopping GemStone.

After a normal shutdown, restart GemStone in the usual manner. For instructions, see Starting the GemStone Server of this chapter.

Disk Failure or File System Corruption

GemStone prints several different disk read error messages to the GemStone log file. For example:

Repository Read failure,

fileName = !#dbf!/users/gemstone/data/extent0.dbf

PageId = 94

File = /users/gs64stone/data/extent0.dbf

too few bytes returned from read()

DBF Operation Read; DBF record 94, UNIX codes: errno=34,...

	"A read error occurred when accessing the repository."

If you see a message similar to the above, or if your system administrator identifies a disk failure or a corrupted file system, try to copy your extents to another node or back them up immediately. The copies may be bad, but it is worth doing, just in case. If you’re lucky, you may be able to copy them back after the underlying problem is solved and start again with the current committed state of your repository.

Otherwise, you may need to restore the repository. For details, see the restore procedures in Chapter 9.

Shared Page Cache Error

If you find a message similar to the following in the GemStone log, the shared page cache (SPC) monitor process (shrpcmonitor) died. The SPC monitor log, $GEMSTONE/data/gemStoneName_pcmonnnnn.log,may indicate the reason.

--- 04/06/15 15:07:19 PDT ---

    The stone’s connection to the local shared cache monitor was lost.

    Error Text: ’Network partner has disconnected.’

The unexpected shutdown of a Gem process may, in rare cases, result in a “stuck spin lock” error that brings down the shared page cache monitor and the Stone. GemStone uses spin locks to coordinate access to critical structures within the cache. In most cases, the monitor can recover if a Gem dies while holding a spin lock, but not all spin locks can be recovered safely. Stuck spin locks may result from a Gem crash, but a typical cause is the use of kill -9 to kill an unwanted Gem process. If you must halt a Gem process, be sure to use only kill or kill -TERM so that the Gem can perform an orderly shutdown.

Use startstone to restart GemStone. For instructions, see Starting the GemStone Server.

Fatal Error Detected by a Gem

If a Gem session process detects a fatal error that would cause it to halt and dump a core image, the Stone repository monitor may do the same when it is notified of the event. This response on the part of the Stone is configurable through the STN_HALT_ON_FATAL_ERR configuration option. When that option is set to True and a Gem encounters a fatal error, the Stone prints a message like this in its log file:

Fatal Internal Error condition in Gem

   when halt on fatal error was specified in the config file

By default, STN_HALT_ON_FATAL_ERR is set to False. That setting causes the Stone to attempt to keep running if a Gem encounters a fatal error; it is the recommended setting for GemStone in a production system. You can set STN_HALT_ON_FATAL_ERR to True during development and testing to provide additional checks for potential risks.

Some Other Shutdown Message

In the event of other shutdown messages in the GemStone log:

1. Consider whether the shutdown might have been caused by a disk failure or a corrupt file system, especially if you see an unexpected message such as Object not found. If you suspect one of these conditions, start with a page audit of the repository file (see Repository Page and Object Audit).

If the page audit fails, refer to Disk Failure or File System Corruption, and consult your operating system administrator.

If the audit succeeds, continue to the next step.

2. If you don’t suspect disk failure or a corrupt file system, try using startstone to restart GemStone. For instructions, see Starting the GemStone Server.

3. If the restart fails, you may have to restore the repository. For details, see the restore procedures in Chapter 9.

No Shutdown Message

If the GemStone log doesn’t contain a shutdown message, there has probably been a power failure or an operating system crash. In that event, the Stone repository monitor automatically recovers committed transactions the next time it starts. Use startstone to restart GemStone, as described under Starting the GemStone Server. See startstone for more information on this command.

Loading the object table at startup

When starting the repository, the object table is not loaded into memory, and initial accesses can take an excessively long time. If you encounter degraded application performance for a period after restarting the Stone, you may want to start up using cache warming. There is an initial heavy I/O load at Stone startup with cache warming, but subsequent application performance should be consistent with your normal application performance. You may choose to preload just the object table pages, which are most important for performance. Alternatively, you can also preload data pages, which will improve performance if you have a large cache with a relatively fixed working set of data pages.

There are two ways to perform cache warming on startup.

• The configuration parameters STN_CACHE_WARMER and STN_CACHE_WARMER_SESSIONS enable and configure cache warming, respectively.
• Alternatively, you may run the startcachewarmer utility. See startcachewarmer for details.

Making efficient use of remote caches

When running a system on which many users log in simultaneously, consider using remote caches so that you don’t need to run all Gem processes on the same machine. There are a couple of ways to optimize this. The following configuration options are of particular interest:

• To allow Gems to make more efficient use of the large cache, set the value of the GEM_PGSVR_FREE_FRAME_CACHE_SIZE configuration option to increase the size of the Gem free frame cache. For example:

GEM_PGSVR_FREE_FRAME_CACHE_SIZE = 25;

• To improve performance on remote caches, set the value of the GEM_PGSVR_UPDATE_CACHE_ON_READ configuration option to True so that remote Gem sessions will update their local caches. For example:

GEM_PGSVR_UPDATE_CACHE_ON_READ = TRUE;

Disk Space and Commit Record Backlogs

Sessions only update their view of the repository when they commit or abort. The repository must keep a copy of each session’s view so long as the session is using it, even if other sessions frequently commit changes and create new views (commit records). Storing the original view and all the intermediate views uses up space in the repository, and can result in the repository running out of space. To avoid this problem, all sessions in a busy system should commit or abort regularly.

For a session that is not in a transaction, if the number of commit records exceeds the value of STN_CR_BACKLOG_THRESHOLD, the Stone repository monitor signals the session to abort by signaling TransactionBacklog (also called “sigAbort”). If the session does not abort, the Stone repository monitor terminates it.

Sessions that are in transaction are not subject to losing their view forcibly. Sessions in transaction enable receipt of the signal TransactionBacklog, and handle it appropriately, but it is optional. It is important that sessions do not stay in transaction for long periods in busy systems; this can result in the Stone running out of space and shutting down. However, sessions that run in automatic transaction mode are always in transaction; as soon as they commit or abort, they begin a new transaction. (For a discussion of automatic and manual transaction modes, see the “Transactions and Concurrency Control” chapter of the Programming Guide.)

To avoid running out of disk space, we recommend that you use manual transaction mode whenever possible. To enter manual transaction mode:

topaz> printit

System transactionMode: #manualBegin

%

At the point that this session needs to commit a change, begin a transaction manually, then make the changes:

topaz> printit

System beginTransaction .

AllUsers addNewUserWithId: #Jane password: 'gemstone' .

System commitTransaction

%

After you commit (or abort) the transaction, your session will return to waiting outside of a transaction.

Handling signals indicating a commit record backlog

Even in manual transaction mode, it is possible to cause a commit record backlog, depending on how your system is configured. Sessions should ensure that they commit or abort regularly, or set up sigAbort handlers to abort when requested by the Stone. A sigAbort handler may be as simple as this:

Exception 

   installStaticException: 

   [ :exception :GSdictionary :errID :array |

      System abortTransaction.

      System enableSignaledAbortError).

Exception 

   installStaticException: 

   [ :exception :GSdictionary :errID :array |

      System continueTransaction.

      System enableSignaledFinishTransactionError).