GemStone/S 64 Bit System Administration Guide

1.1 Basic GemStone/S 64 Bit Architecture

1.2 Starting GemStone and Logging In

• Stone

• Shared Page Cache

  • Shared Cache Size

  • OS memory and Swapping

• NetLDI

• Logging in Gem Sessions

  • Linked vs. RPC

  • Gemservers for Large Configurations

• NRS (Network Resource String)

1.3 Authentication and Authorization

• GemStone UserIds and login

• Authorization to Access Data

• File access and authorization

1.4 Transactions and commit records

• Object views

• Session transactional state

• Commit records

1.5 Files and Directories

• GemStone Installation

  • GemStone shared libraries

• Lock file directory

  • Host Identifier

• Extents, Tranlogs, and disk space

• Disk Usage for Extents and Transaction Logs

  • Raw Partitions

• Process Log Files

• System Clock and GemStone times

  • TimeZones

1.6 Options for Configuring

• Configuration files

• Environment variables

• Utility command arguments

• Run-time configurations

• Example Configurations

1.7 Avoiding risk of Data loss

• Recovery vs. Restore

• Developing a Failover Strategy

  • Verifying strategy

1.8 Running a Second Repository

2.1 Server Components

• The Server Configuration File

• Before you begin

• Number of User Sessions

• Shared Page Cache

  • Verifying OS support for sufficient Shared Memory

  • Other shared page caches

2.2 Configuring Extents and Transaction Logs

• Recommendations About Disk Usage

  • Raw Partitions

  • NFS

• Configuring the Repository Extents

  • Estimating Extent Size

  • Choosing the Extent Location

  • Extent disk configuration

  • Setting a Maximum Size for an Extent

  • Pregrowing Extents to a Fixed Size

  • Pregrowing Extents to the Maximum Size

  • Allocating Data to Multiple Extents

• Configuring the Transaction Logs

  • Logging Mode

  • Estimating Disk Space for Transaction Logs

  • Choosing the Log Location and Size Limit

• How To Set Up a Raw Partition

• Sample Setup for Extent on Raw Partition

• Changing Between Files and Raw Partitions

  • Moving an Extent to a Raw Partition

  • Moving an Extent to the File System

  • Moving Transaction Logging to a Raw Partition

  • Moving Transaction Logging to the File System

• Server Response to Gem Fatal Errors

2.3 How To Access the Server Configuration at Run Time

• To Access Current Settings at Run Time

• To Change Settings at Run Time

2.4 Tuning Server Performance

• Tuning the Shared Page Cache

  • Adjusting the Cache Size

  • Clustering Objects That Are Accessed Together

• Controlling Checkpoint Frequency

• Tuning Page Server Behavior

  • To Add AIO Page Server threads

  • Free Frame Page Server

  • Process Free Frame Caches

• Running Cache Warming

  • Configure what is loaded into the cache

  • Configure when warming is run

3.1 Overview

• Linked and RPC Applications

  • Note on terminology

• The Session Configuration File

3.2 Configuring Gem Session Processes

• Local vs. Remote

  • Local

  • Remote

• Gem Memory Requirements

• Additional Configuration for Remote Gems

3.3 Set the Gem Configuration Options

• Configure Temporary Object Space

• Configure SSL for remote Gem sessions

• Native Code

3.4 How To Access the Configuration at Run Time

• To Access Current Settings at Run Time

• To Change Settings at Run Time

4.1 Overview

• Administrative user account

• Login Parameters and Gem process

4.2 The NetLDI

• NetLDI Ports and Names

4.3 NetLDI configuration

• Configuration Decisions

  • NetLDI behavior: who will own spawned processes?

  • Authentication: which requests need to be authenticated?

• Setting up the NetLDI Configuration

  • 1. Setting up NetLDI as root with Authentication

  • Starting the NetLDI

  • Login parameters

  • Using Kerberos authentication

  • 2. Setting up NetLDI in Guest Mode with Captive Account

  • Starting the netldi

  • Login

4.4 File Permissions

• Using the Setuid Bit

• Alternative: Use Group Write Permission

• Shared Page Cache

• File Permissions for Other Files and Directories

4.5 Linked Gem Sessions

5.1 Overview

• Distributed configuration with Local Gem

• Distributed configuration with Remote Gem

• Network

• GemStone NetLDIs

• NRS Syntax

• Stone

• Connecting to the RPC Gem

• Shared Page Cache

• GemStone Page Servers

• Port use in GemStone

• Disrupted Communications

5.2 Configuring GemStone on Remote Nodes

• Local Gems only

• Remote Gems

• Configuration Examples

  • RPC Application on a Remote Node with Local Gem

  • Linked Application on a Remote Node

• RPC Application on a Remote Node with Remote Gem

• RPC Application, Gem, and Stone on Three Nodes

• Distributed System with a Mid-Level Cache

5.3 Troubleshooting Remote Logins

• How the Login Process starts Session Processes

  • RPC Application with Gem on Server Node

  • Linked Application on Remote Node

• If You Still Have Trouble

  • Check NetLDI Log Files

6.1 Starting the GemStone Server

• To Start GemStone

• To Troubleshoot Stone Startup Failures

  • Missing or Invalid Key File

  • Shared Page Cache Cannot Be Attached

  • Extent Missing or Access Denied

  • Extent Open by Another Process

  • Extent Already Exists

  • Other Extent Failures

  • Transaction Log Missing

  • Other Startup Failures

• Listing Running Servers

6.2 Starting a NetLDI

• To Troubleshoot NetLDI Startup Failures

6.3 Starting a GemStone Session

• To Define a GemStone Session Environment

• To Start a Linked Session

• To Start an RPC Session

• To Troubleshoot Session Login Failures

• Identifying and Stopping Logged-in Sessions

6.4 Shutting Down Sessions, the Object Server, and NetLDI

• Stopping Logged-in Sessions

• Stopping the Stone

• Stopping the NetLDI

• Using OS kill

6.5 Logins without a stone running

• Solo login from topaz

• Object creation and memory use

• Solo sessions other than from topaz

6.6 Recovering from an Unexpected Shutdown

• Clean Shutdown Message

• Disk Failure or File System Corruption

• Shared Page Cache Error

• Fatal Error Detected by a Gem

• Some Other Shutdown Message

• No Shutdown Message

7.1 GemStone Process Logs

• Finding log files

• Stone Log

• Shared Page Cache Monitor Log

• Admin Gem Log

• Reclaim Gem Log

• Free Frame Page Server Log

• AIO Page Server Log

• Page Manager Log

• Symbol Gem Log

• NetLDI Log

• Gem Logs and logs related to Gem Sessions

  • Linked Gems

  • RPC Gems on Stone’s host

  • Remote RPC or Linked Gems

• Further control over log file location and name

• Logsender and logreceiver Logs

• Other Log Files

• Summary of GemStone Process Log Behaviors

• Managing log files

  • Retaining or deleting all log files

  • Customizing individual process deletion behavior

• Localizing timestamps in log files

• Programmatically adding messages to logs

7.2 Repository Page and Object Audit

• Page Audit

• Object Audit and Repair

  • Performing the Object Audit

  • Error Recovery

7.3 Profiling Repository Contents

7.4 Monitoring Performance

• Statmonitor and VSD

  • Reliable constant monitoring

• Programmatic Access to Cache Statistics

• Host Statistics

  • Host Statistics for processes

  • Host Statistics for OS

8.1 GemStone Users

• UserProfiles

• AllUsers

• Special System Users

• UserProfile Data

  • User ID

  • Authentication Scheme

  • Password

  • Default Object Security Policy

  • Privileges

  • Symbol Lists

• DeletedUserProfile and AllDeletedUsers

8.2 UserProfileGroups

• AllGroups

• Groups for object authorization

• Create a group

• Delete a group

8.3 Creating and Removing Users

• Creating Users

• Removing Users

• Users and Group membership

  • Adding a user to a group

  • Removing a user from a group

  • Querying for Group members

  • Querying for a User’s Groups

8.4 Administering Users

• List Existing Users

• Modifying the UserId

• Modifying Password

• Modifying defaultObjectSecurityPolicy

• Modifying Privileges

• Modifying SymbolLists

• Disable and Enable User Logins

• Disable and Enable Commits by User

8.5 Configuring GemStone Authentication

• Configuring GemStone Login Security

• Limiting Choice of Passwords

• Disallowing Particular Passwords

• Disallowing Reuse of Passwords

• Password Aging – Require Periodic Password Changes

• Account Aging – Disable Inactive Accounts

• Enabling Account Aging and lastLoginTime

• Limit Logins Until Password Is Changed

• Limit Concurrent Sessions by a Particular UserId

• Limit Login Failures

8.6 Configuring UNIX Authentication

8.7 Configuring LDAP Authentication

• LDAP authentication without anonymous binds

• Validating passwords

8.8 Configure SingleSignOn Authentication

• Kerberos concepts

• KerberosPrincipal and AllKerberosPrincipals

• Setting up Kerberos Authentication in GemStone

• Using Groups to authenticate with Kerberos

• KerberosPrincipal available to all users

8.9 Tracking User Logins

• Login logging

• Login Hook

  • Debugging

9.1 The Repository and Extents

• Repository Growth

• How To Check Free Space

9.2 Adding and Removing Extents

• To Add an Extent While the Stone is Running

  • Possible Effects on Other Sessions

  • Repository>>createExtent:

  • Repository>>createExtent:withMaxSize:

• To Remove an Extent

9.3 Reallocating Existing Objects Among Extents

• To Reallocate Objects Among a Different Number of Extents

• To Reallocate Objects Among the Same Number of Extents

9.4 Shrinking the Repository

9.5 Checking Page Fragmentation

9.6 Disk Space and Commit Record Backlogs

• Handling signals indicating a commit record backlog

9.7 Recovering from Disk-Full Conditions

• Repository Full

10.1 Overview

• Logging Modes

• Restoring Transactions to a Restored Backup

  • How the Logs Are Used

10.2 How To Manage Full Logging

• To Archive Logs

• To Add a Log at Run Time

• To Force a New Transaction Log

• To Initiate a Checkpoint

• To Change to Partial Logging

10.3 How To Manage Partial Logging

• To Change to Full Logging

10.4 How To Recover from Tranlog-Full Conditions

• Transaction Log Space Full

11.1 Overview

• Warm and Hot Standbys

• Version Compatibility

11.2 Types of Backups

11.3 How To Make an Extent Snapshot Backup

• Extent Snapshot Backup when the Repository is shutdown

• Extent Snapshot Backup when the Repository is running

11.4 How To Make a Smalltalk Full Backup

• Compressed Backups

• Performance Optimization

  • Impact on other sessions

  • Increasing session threads

  • Backups and Garbage Collection

• Monitoring and Verification

  • statmonitor and cache statistics

  • Verifying a Backup is Readable

  • Checking Backup Start and Completion

  • Details about the backup

11.5 How to Restore from Backup

• Restoring from an Extent Snapshot Backup

• Restoring from a Full Backup

  • Steps in restoring a full backup

11.6 How to Make and Restore Secure Backup

• Creating a secure backup

  • Setting the certificate directories

  • Create an unencrypted secure backup

  • Create an encrypted backup

• Restoring a secure backup

  • Restore an unencrypted backup

  • Restore an encrypted backup

• Verifying the digital signature

11.7 How to Restore Transaction Logs

• Finalize by commitRestore

  • commitRestore without restoreFromCurrentLogs

  • After restoring a backup with corrupted objects

11.8 Special Cases and Errors in Restore

• Missing or Corrupted Objects in Full Backup

• Restoring Logs up to a Specific Log

• Restoring Logs to a Point in Time

• Precautions When Restoring a Subset of Transaction Logs

  • Fork-in-Time Scenario

• Errors While Restoring Transaction Logs

  • Missing Transaction Log File

  • Truncated or Corrupt Transaction Log File

• Recovering from File System Problems

12.1 Overview

12.2 Warm Standby

• Setup and run the warm standby

• Activate the warm standby in case of failure in the primary

12.3 Hot Standby

• Hot standby processes

  • logsender

  • logreceiver

• Continuous Restore Mode

• Transaction Record Transmittal

• Multiple standby repositories

• To setup and run the hot standby

  • Activate the hot standby in case of failure in the primary

• Planned failovers

• Connecting using SSL Mode

• Added transaction logs to the master

12.4 Tuning a Warm or Hot Standby

• Tuning Reclaim

13.1 Memory Organization

13.2 Configuring Temporary Memory Usage

• Configuration Options

• Methods for Computing Temporary Object Space

• Debugging out-of-memory errors

• Recording Out of Memory Information to CSV file

• Signal on low memory condition

14.1 Basic Concepts

• Shadow or Dead?

• What Happens to Garbage?

• Admin and Reclaim Gems

• Admin and Reclaim Gem configuration parameters

• GemStone’s Garbage Collection Mechanisms

  • Marking

  • Reclaiming

• GcLock

• Symbol Garbage Collection

14.2 MarkForCollection

• Impact on Other Sessions

• Scheduling markForCollection

14.3 Epoch Garbage Collection

• Running Epoch Garbage Collection

• Tuning Epoch

• Cache Statistics

14.4 Reclaim

• Tuning Reclaim

  • Reclaim Configuration Parameters

  • Reclaim Commit Frequency

  • Controlling the impact of reclaim

  • Speeding up reclaim

  • Avoiding disk space issues

• Cache Statistics

14.5 Running Admin and Reclaim Gems

• Configuring Admin Gem

• Configuring Reclaim Gem

• Starting GcGems

• Stopping GcGems

• Adjusting the number of Reclaim sessions

14.6 Further Tuning Garbage Collection

• Multi-Threaded Scan

  • Tuning Multi-Threaded Scan

  • Memory Impact

• Identifying Sessions Holding Up Voting

• Tuning Write Set Union Sweep

• Identifying Sessions Holding Up Page Reclaim

• Finding References to Objects that prevent garbage collection

  • Using GsBitmaps

  • Finding all references

  • Full reference path

• Finding large objects that are using excessive space

  • Identify Larger Objects in the Repository

  • Finding named objects that are large

A.1 How GemStone Uses Configuration Files

• System Configuration File

• Executable Configuration File

• Creating or Using a System Configuration File

• Creating an Executable Configuration File

• Nesting Configuration Files

• Naming Executable Configuration Files

• Determining which Configuration Files are in use by a Gem or Stone

• Alternate ways to specify configuration parameter values

• Naming Conventions for Configuration Options

A.2 Configuration File Syntax

• Errors in Configuration Files

  • Syntax Errors

  • Option Value Errors

A.3 Configuration Options

• CONFIG_WARNINGS_FATAL

• DBF_ALLOCATION_MODE

• DBF_EXTENT_NAMES

• DBF_EXTENT_SIZES

• DBF_PRE_GROW

• DBF_SCRATCH_DIR

• DUMP_OPTIONS

• GEM_ABORT_MAX_CRS

• GEM_CACHE_WARMER_ARGS

• GEM_CACHE_WARMER_MID_CACHE_ARGS

• GEM_COMPRESS_TRANLOG_RECORDS

• GEM_FREE_FRAME_CACHE_SIZE

• GEM_FREE_FRAME_LIMIT

• GEM_FREE_PAGEIDS_CACHE

• GEM_HALT_ON_ERROR

• GEM_KEEP_MIN_SOFTREFS

• GEM_KERBEROS_KEYTAB_FILE

• GEM_KEYRING_DIRS

• GEM_MAX_SMALLTALK_STACK_DEPTH

• GEM_NATIVE_CODE_ENABLED

• GEM_PGSVR_COMPRESS_PAGE_TRANSFERS

• GEM_PGSVR_FREE_FRAME_CACHE_SIZE

• GEM_PGSVR_FREE_FRAME_LIMIT

• GEM_PGSVR_UPDATE_CACHE_ON_READ

• GEM_PGSVR_USE_SSL

• GEM_READ_AUTH_ERR_STUBS

• GemRemoteCommit

• GEM_REPOSITORY_IN_MEMORY

• GEM_RPCGCI_TIMEOUT

• GEM_RPC_KEEPALIVE_INTERVAL

• GEM_RPC_USE_SSL

• GEM_SOLO_EXTENT

• GEM_STATMONITOR_ARGS

• GEM_STATMONITOR_MID_CACHE_ARGS

• GEM_SOFTREF_CLEANUP_PERCENT_MEM

• GEM_TEMPOBJ_AGGRESSIVE_STUBBING

• GEM_TEMPOBJ_CACHE_SIZE

• GEM_TEMPOBJ_CONSECUTIVE_MARKSWEEP_LIMIT

• GEM_TEMPOBJ_MESPACE_SIZE

• GEM_TEMPOBJ_OOMSTATS_CSV

• GEM_TEMPOBJ_OOPMAP_SIZE

• GEM_TEMPOBJ_PERMGEN_SIZE

• GEM_TEMPOBJ_POMGEN_PRUNE_ON_VOTE

• GEM_TEMPOBJ_POMGEN_SCAVENGE_INTERVAL

• GEM_TEMPOBJ_POMGEN_SIZE

• GEM_TEMPOBJ_SCOPES_SIZE

• GEM_TEMPOBJ_START_ADDR

• INCLUDE

• KEYFILE

• LOG_WARNINGS

• NETLDI_HostAgentUser_cert

• NETLDI_HostAgentUser_key

• NETLDI_PORT_RANGE

• NETLDI_START_MIDCACHE

• NETLDI_WARMER_ARGS

• SHR_NUM_FREE_FRAME_SERVERS

• SHR_PAGE_CACHE_LARGE_MEMORY_PAGE_POLICY

• SHR_PAGE_CACHE_LARGE_MEMORY_PAGE_SIZE_MB

• SHR_PAGE_CACHE_LOCKED

• SHR_PAGE_CACHE_NUM_PROCS

• SHR_PAGE_CACHE_NUM_SHARED_COUNTERS

• SHR_PAGE_CACHE_PERMISSIONS

• SHR_PAGE_CACHE_SIZE_KB

• SHR_PUSH_TO_MIDCACHES_THREADS

• SHR_SPIN_LOCK_COUNT

• SHR_TARGET_FREE_FRAME_COUNT

• SHR_WELL_KNOWN_PORT_NUMBER

• STN_ADMIN_GC_SESSION_ENABLED

• STN_ALLOCATE_HIGH_OOPS

• STN_ALLOW_NFS_EXTENTS

• STN_ALLOW_NO_SESSION_INIT

• STN_ANONYMOUS_SSL

• STN_CACHE_WARMER

• STN_CACHE_WARMER_ARGS

• STN_CACHE_WARMER_SESSIONS

• STN_CACHE_WARMER_WAIT_MODE

• STN_CHECKPOINT_INTERVAL

• STN_COMMIT_QUEUE_THRESHOLD

• STN_COMMIT_RECORD_BM_CACHING

• STN_COMMIT_RECORD_QUEUE_SIZE

• STN_COMMIT_TOKEN_TIMEOUT

• STN_COMMITS_ASYNC

• STN_CR_BACKLOG_THRESHOLD

• STN_DISABLE_LOGIN_FAILURE_LIMIT

• STN_DISABLE_LOGIN_FAILURE_TIME_LIMIT

• STN_DISKFULL_TERMINATION_INTERVAL

• STN_EPOCH_GC_ENABLED

• STN_EXTENT_IO_FLAGS

• STN_FREE_FRAME_CACHE_SIZE

• STN_FREE_SPACE_THRESHOLD

• STN_GEM_ABORT_TIMEOUT

• STN_GEM_INITIAL_TRANSACTION_MODE

• STN_GEM_LOSTOT_TIMEOUT

• STN_GEM_PGSVR_CONNECT_TIMEOUT

• STN_GEM_PRIVATE_PGSVR_ENABLED

• STN_GEM_TIMEOUT

• STN_GROUP_COMMITS

• STN_HALT_ON_FATAL_ERR

• STN_LISTENING_ADDRESSES

• STN_LOGIN_LOG_DIR

• STN_LOGIN_LOG_ENABLED

• STN_LOGIN_LOG_HALT_ON_ERROR

• STN_LOGIN_LOG_MAX_SIZE

• STN_LOG_IO_FLAGS

• STN_LOG_LOGIN_FAILURE_LIMIT

• STN_LOG_LOGIN_FAILURE_TIME_LIMIT

• STN_LOOP_NO_WORK_THRESHOLD

• STN_MAX_AIO_RATE

• STN_MAX_AIO_REQUESTS

• STN_MAX_GC_RECLAIM_SESSIONS

• STN_MAX_LOGIN_LOCK_SPIN_COUNT

• STN_MAX_REMOTE_CACHES

• STN_MAX_SESSIONS

• STN_MAX_VOTING_SESSIONS

• STN_NUM_AIO_WRITE_THREADS

• STN_NUM_GC_RECLAIM_SESSIONS

• STN_NUM_LOCAL_AIO_SERVERS

• STN_OBJ_LOCK_TIMEOUT

• STN_PAGE_MGR_COMPRESSION_ENABLED

• STN_PAGE_MGR_MAX_WAIT_TIME

• STN_PAGE_MGR_PRINT_TIMEOUT_THRESHOLD

• STN_PAGE_MGR_REMOVE_MAX_PAGES

• STN_PAGE_MGR_REMOVE_MIN_PAGES

• STN_PGSVR_PORT_RANGE

• STN_REMOTE_CACHE_PGSVR_TIMEOUT

• STN_REMOTE_CACHE_STARTUP_TIMEOUT

• STN_REMOTE_CACHE_TIMEOUT

• STN_SHR_TARGET_PERCENT_DIRTY

• STN_SIGNAL_ABORT_AGGRESSIVE

• STN_SIGNAL_ABORT_CR_BACKLOG

• STN_SMC_SPIN_LOCK_COUNT

• STN_STATMONITOR_ARGS

• STN_STONE_CACHE_STARTUP_TIMEOUT

• STN_SYMBOL_GC_ENABLED

• STN_SYMBOL_GEM_TEMPOBJ_CACHE_SIZE

• STN_TRAN_FULL_LOGGING

• STN_TRAN_LOG_DEBUG_LEVEL

• STN_TRAN_LOG_DIRECTORIES

• STN_TRAN_LOG_LIMIT

• STN_TRAN_LOG_PREFIX

• STN_TRAN_LOG_SIZES

• STN_TRANQ_TO_RUNQ_THRESHOLD

• STN_WELL_KNOWN_PORT_NUMBER

A.4 Runtime-only Configuration Options

• DelayAutoServiceSigAbort

• GemAutoServiceSigAbort

• GemCommitConflictDetails

• GemCommitStubsForNpObjects

• GemConvertArrayBuilder

• GemConfigFileNames

• GemDropCommittedExportedObjs

• GemExceptionSignalCapturesStack

• LogOriginTime

• SessionInBackup

• StnConfigFileNames

• StnCurrentTranLogDirId

• StnCurrentTranLogNames

• StnLogFileName

• StnLogGemErrors

• StnLoginsSuspended

• StnMaxReposSize

• StnMaxSessions

• StnStandbyRole

• StnSunsetDate

• StnTranLogOriginTime

copydbf

• Compression of copydbf output

• Using copydbf to access information on a file

• copydbf with remote nodes

• copydbf with raw partitions

gemnetobject

• Examples

• Error checking

gslist

largememorypages

pageaudit

printlogs

• Examples

pstack

removedbf

searchlogs

• Examples

startcachewarmer

startlogreceiver

startlogsender

startnetldi

• netldid

startstone

• stoned

statmonitor

stoplogreceiver

stoplogsender

stopnetldi

stopstone

topaz

vsd

waitstone

A.1 Using NRS

• GsNetworkResourceString

• GEMSTONE_NRS_ALL

• Further arguments to gemnetobject

• Controlling log file directory locations

• Controlling log file names

A.2 NRS Syntax

Non-Numeric Constants

Numeric Constants

Repository and GsObjectSecurityPolicies

Global Variables and Collections

Current TimeZone

• Zoneinfo

• Utilities

Public Environment Variables

• System Variables Used by GemStone

Reserved Environment Variables