2. Scripting with Topaz and SuperDoit
Many tasks in any software environment can be made easier or automated using scripts. GemStone is no exception, and operations such as markForCollection and backup are often scripted to start automatically at certain times.
In GemStone, scripts rely on topaz, GemStone’s command line interface. Topaz itself does not provide flow of control, nor conditional execution. For fully featured scripting, you can combine topaz with shell scripting such as bash, or use the SuperDoit environment, to allow script arguments, conditional execution, create sequences of operations, and similar scripting requirements.
Standard and Solo logins
A standard GemStone login requires a running Stone, and creates a session in a GemStone multi-user transactional environment. The login can perform any operation that its GemStone and UNIX UserId allows, including markForCollection and other operations that are often scripted.
There are tasks, however, that do not actually require a Stone to be running, and can be done with less risk and lower impact by using solo logins. Solo logins, described under Solo Logins, allow a single-user, read-only login that can load and execute GemStone Smalltalk code.
- You can execute Smalltalk code, without impacting other users.
- Using an empty extent and a non-privileged user, anyone can execute GemStone Smalltalk code, such as for text processing, with minimal security concerns.
- A solo session can filein topaz files containing GemStone classes and methods and execute code invoking them, provided that code execution does not require a Stone, nor perform a commit or abort. Code that attempts to commit or abort will signal an error.
- You cannot perform operations that require a Stone and are inherently multi-user, such as markForCollection or backup.
- You can use GemStone’s external session protocols to login remotely to a running Stone to perform operations such as markForCollection or backup. This requires the login credentials for the running Stone.
While solo logins cannot replace standard logins for the most common scripting needs, they provide additional options for executing GemStone code from the command line.
2.1 Topaz scripting
Creating a script
To execute markForCollection or other simple commands, only a simple script is needed.
Create the topaz script
To start with, create a file containing the login instructions and the GemStone code you wish to execute.
For example, this might be the contents of a text file script that runs markForCollection, with the name runmfc.topaz:
set user DataCurator password swordfish gemstone gs64stone
login
! run garbage collection mark/sweep
exec SystemRepository markForCollection %
exit
Verify using the topaz input command
The topaz input command can execute any file containing topaz commands, such as the runmfc.topaz just created. Since the login information is embedded in the contents of this file, you do not need to do a separate login.
To verify your script, start topaz and input the script file. Note that if successful, this will in fact run markForCollection on your Stone.
topaz 1> input runmfc.topaz
Pass the script on the topaz command line.
The topaz executable is invoked on the command line, and for scripting you can pass in a the topaz script to be executed on the command line, using the -I or -S topaz argument.
- The -I command, (initialization) suppresses automatic use of a .topazini, and suppresses echo of output to the console.
- The -S command (scripting) does read a .topazini; this can be suppressed by also specifying -i. The -S also does not suppress echoing output; this can be suppressed manually by also specifying -q. In other words, topaz -I scriptName is equivalent to topaz -i -q -S scriptName.
- You may pass in two scripts, using both -S and -I scripts; the -I script is always executed first.
unix> topaz -I runmfc.topaz > MFC.out
Further embed in a UNIX shell script
To allow a simple shell command to execute the code, rather than requiring the topaz arguments and redirect, you can embed the topaz script within a UNIX shell script.
For example, create a file runMFC with the following contents:
#! /bin/bash
#set -x
$GEMSTONE/bin/topaz -l -I runmfc.topaz <<EOF >>MFC.out
EOF
Alternatively, you can include the topaz script code in the shell script. For example:
#! /bin/bash
#set -x
$GEMSTONE/bin/topaz -il <<EOF >>MFC.out
set user DataCurator password swordfish gemstone gs64stone
login
! run garbage collection mark/sweep
exec SystemRepository markForCollection %
exit
EOF
Either script can be executed (once given the appropriate permissions) by executing:
unix > ./runMFC
Security Concerns
The above scripts embed the GemStone password in the script, which may not be appropriate, depending on the security requirements for your environment.
You can segregate the login details into a separate file, which can be automatically located following the rules for .topazini, or by passing in using the -I topaz argument.
Using a separate initialization file allows you to move the user name and password to a separate file, and thus not visible in the script code itself.
Given the following two scripts:
set gemstone gs64stone
login
! run garbage collection mark/sweep
exec SystemRepository markForCollection %
exit
set user DataCurator password swordfish
Then you can execute, for example:
unix> topaz -I myini -S runmfc.topaz > MFC.out
Or skip creating the file runmfc.topaz, and create a UNIX script with the following contents:
#! /bin/bash
#set -x
$GEMSTONE/bin/topaz -l -I myini <<EOF >>MFC.out
set gemstone gs64stone
login
! run garbage collection mark/sweep
exec SystemRepository markForCollection %
exit
EOF
2.2 Scripting with topaz she-bang
When writing solo scripts (that do not require a Stone), you can avoid writing shell script code altogether. Topaz she-bang scripting has some restrictions:
- The environment variable $GEMSTONE must be set
- she-bang scripts execute in a solo session, that is, do not log into a running stone.
- she-bang scripts run in a linked topaz session
- no topaz command line options can be passed in
- No .topazini files are read
The first line of the she-bang can be defined two ways:
#!/usr/bin/env topaz
$GEMSTONE/bin must be on the machine executable search path.
#!fullPathToExecutable/topaz
GEMSTONE/bin does not need to be on the machine executable search path, but fullPathToExecutable must be the full path, not including environment variables.
For example, if you define a executable text file, myscript, with the following contents:
#!/usr/bin/env topaz
set user DataCurator password swordfish
login
run
| files sz |
files := GsFile
contentsOfDirectory: '$GEMSTONE/data/tranlog*.dbf'
onClient: false.
sz := 0.
files do: [:ea |
sz := sz + (GsFile sizeOfOnServer: ea)
].
'tranlogs consume ', (sz / 1024) asInteger asString, ' KB'.
%
This file can be executed, with or without a running stone, to report the sum total size of tranlog files in the given directory.
Note that this will (in a default configuration) execute using the empty distribution extent, and checks the file sizes on disk rather than using any internal representation in the repository that is actually generating the transaction logs.
2.3 Script Arguments
The Topaz command line supports additional arguments, which can appear after a double dash at the end of the command line. Anything after this double dash are ignore by topaz.
These command line arguments are available from Smalltalk code using System commandLineArguments.
unix> topaz -l -- these are some arguments
...
topaz 1> exec System commandLineArguments printString%
anArray( 'topaz', '-l', '--', 'these', 'are', 'some', 'arguments')
Passing arguments using bash scripts
By using bash to invoke topaz, you can create executables with command line arguments, that use topaz scripting to perform operations.
The following example uses three files to demonstrate writing a command-line script with one argument, that executes GemStone Smalltalk code.
The top-level bash script invokes topaz with the login information in a topaz initialization file and the actual script file, passing in a single argument. For simplicity, this example does no error checking. An actual script, of course, should check that the arguments are present and valid.
Example 2.1 Example files demonstrating bash script with argument
Contents of executable gettranlogspace:
#!/bin/bash
export GEMSTONE=/lark1/users/gsadmin/3.7
$GEMSTONE/bin/topaz -lq -I $GEMSTONE/scripts/myini -S
$GEMSTONE/scripts/fileSpaceUsed.tpz -- $1
This invokes linked topaz (-l), passing in the initialization file (-I) and the script file (-S). The use of the -q flag suppresses topaz output that will clutter the output. Arguments following -- are passed to topaz, in this case, the bash shell’s (0-based) second element, which is the argument (a file system path).
By setting $GEMSTONE explicitly and specifying the path to topaz, there is no need to perform GemStone environment setup before executing this script; it can be run in any environment.
Contents of file myini:
set user DataCurator pass swordfish
set gemstone gs64stone
set solologin on
This sets solo login, so no stone login is required. This script logs in as DataCurator, but another userId with limited privileges would be sufficient, since the script will only perform limited operations.
Contents of file fileSpaceUsed.tpz:
login
run
| dir files sz |
dir := System commandLineArguments last.
files := GsFile
contentsOfDirectory: dir
onClient: false.
sz := 0.
files do: [:ea | sz := sz + (GsFile sizeOfOnServer: ea)].
GsFile gciLogClient: dir, ': files consume total ',
(sz / 1024) asInteger asString, ' KB'.
%
logout
exit
The use of System commandLineArguments gets the full set of tokens that invoked topaz, so the last element is the bash argument. Again, for simplicity no error checking is done in this example.
2.4 SuperDoit
The SuperDoit scripting environment allows you to write command line scripts entirely in Smalltalk. SuperDoit uses the topaz features described earlier in this chapter, and provides a framework for handling arguments and usage sections, as well as other features that support writing complex scripts.
- $GEMSTONE must be defined
- $GEMSTONE/bin must be on your search path
- For stone scripts, $GEMSTONE/version.txt must be present.
Script Types
SuperDoit scripts come in two variants:
-
stone
executes the script as a normal login to a running Stone. Login details are provided in a topaz initialization file (.topazini, or another name), which may be provided on the command line using the -I argument, or in one of the standard .topazini locations: the current working directory, or the home directory of the user executing the script.
All the details required for a login to a Stone in your specific environment must be available, in the topaz initialization file, $GEMSTONE_NRS_ALL, and/or other environment variables.
By convention, these scripts end in .stone. You may use the template $GEMSTONE/examples/superDoit/template.stone to start developing your script.
-
solo
executes the script as a solo session. Commits, and operations that require a stone, are not allowed, and solo scripts cannot login RPC. Solo scripts may use the rowan extent, $GEMSTONE/bin/extent0.rowan.dbf (as was the default in previous releases) for their solo extent, or the base GemStone image, $GEMSTONE/bin/extent0.dbf. The only time the distinction would matters if you are using or avoiding Rowan-specific behavior.
Solo scripts do not automatically look for a standard .topazini file. Solo scripts use a solo-specific topaz initialization file, located within the SuperDoit project, which logs in as SystemUser. You may use the -I argument to specify a different topaz initialization file.
By convention, solo scripts end in .solo. For clarity, to distinguish scripts that specifically use the base Gemstone image or the Rowan extent, the script may end in .gs_solo or .rw_solo. You may use the template $GEMSTONE/examples/superDoit/template.rw_solo or $GEMSTONE/examples/superDoit/template.gs_solo to start developing your script.
Script structure
A SuperDoit script consists of a number of sections, each one started by a keyword, and ending with %. All sections are optional.
The Smalltalk code that the script executed is in a section with the keyword doit. Note that while this looks similar to topaz syntax, it is a keyword defining the code execution section; topaz syntax is not understood in solo and stone scripts.
A minimal SuperDoit script might be, for example,
#!/usr/bin/env superdoit_solo
doit
Time now asString
%
In addition to a doit section, there are number of sections that you may include depending on your script’s requirements. Most commonly, you will want to include sections for script command line options, in the options section, and usage information in a usage section.
An example of a script with the options, usage, and doit sections:
#!/usr/bin/env superdoit_stone
# Print the contents of DbfHistory with a byte limit
options
{SuperDoitOptionalOptionWithRequiredArg
long: 'bytes' short: 'b'}
%
usage
$basename [-h] [-D] [-b <number>] [-- [-I <topazini>]]
Return DbfHistory for the stone specified by <topazini>, or
if not specified, the default .topazini for the environment.
The optional -b argument specifies to return only the first
<number> bytes of the DbfHistory.
%
doit
self bytes isNil
ifTrue: [DbfHistory]
ifFalse: [DbfHistory copyFrom: 1 to:
(self bytes asInteger min: DbfHistory size)]
%
This script show how access to the argument is handled: self bytes returns the contents of the command line argument specified using -b or --bytes.
Also note that the usage here is simplified; the arguments --help, --Debug, and --bytes may also be used on the command line. See the section under Arguments.
Finally, the argument to topazini that is specifically mentioned in the usage is only one of a number of arguments to topaz that can be passed to the SuperDoit script and forwarded to topaz. These arguments, if used, are separated from the script arguments by --. See the section Arguments to Topaz.
Commonly Used Sections
doit
holds the body of the script
options
specifies command line arguments; see the possible specifications under Arguments.
usage
provide usage text. This can be in any format; all text between the usage and closing % is printed in response to the -h or --help argument to the script.
input
specifies code (topaz or gs) that is filed in prior to executing the doit.
customoptions
Allows you to override the default arguments for help and debug.
Complex Scripting
The following sections allow functionality to be broken down into supporting methods and associated state.
method
Define a method on the script itself
instvars
Define instance variables for the script, which can be accessed by the methods defined in method, method: and classmethod: sections.
Rowan Support
The following are advanced sections to support Rowan repositories:
projectshome
Declare the value of the ROWAN_PROJECTS_HOME environment variable.
specs
Specify an array of Rowan load specification STON objects used to load external projects into the image.
specurls
Specify a list of spec urls that reference the location of a Rowan load specification STON object.
Arguments
Command line arguments may be defined using name-value pairs, names without values, or as positional arguments.
The options section defines the named and name-value command line arguments; positional arguments are accessed in the doit section, sending self positionalArgument to return the array of zero or more positional arguments that were included on the command line.
Named and name-value pairs can be specified either using getopts syntax -Character value (referred to as "short" in SuperDoit), or --String=value ("long").
The long version is always supported; you may also specify a short version when defining the script arguments.
Arguments with values may be required or optional, and optional arguments may have a default value specified. See the specifications on Defining Script Arguments.
Standard arguments
By default, all scripts have the help and debug options:
-h or --help
Returns usage information specified in the usage section
-D or --Debug
If an exception occurs, the script stops in topaz rather than exiting. This allows you to debug using topaz commands such as where.
--gemDebug
If terminal is connected to stdout, bring up debugger. If not, dump stack to stdout and wait for topaz to attach using topaz DEBUGGEM command.
To exclude help and/or debug options, you can define a section in your script for customoptions.
Note that since the help section is processed and returned in Smalltalk code, executing a script to return the help text requires a successful login to the Stone or solo extent.
Defining Script Arguments
The script options are specified as an array of objects (in GemStone Array Builder syntax, curly-brace delimited and period separated), specified using the following expressions.
SuperDoitOptionalOptionWithNoArg long:
SuperDoitOptionalOptionWithNoArg long:short:
SuperDoitOptionalOptionWithRequiredArg long:
SuperDoitOptionalOptionWithRequiredArg long:default:
SuperDoitOptionalOptionWithRequiredArg long:short:
SuperDoitOptionalOptionWithRequiredArg long:short:default:
SuperDoitRequiredOptionWithRequiredArg long:
SuperDoitRequiredOptionWithRequiredArg long:short:
For example, to specify an argument that may be supplied with -b value, with --bytes value, or omitted:
options
{SuperDoitOptionalOptionWithRequiredArg
long: 'bytes' short: 'b'}
%
Arguments to Topaz
In addition to your script-specific arguments, you may also include arguments that are directives to topaz. These are included following a -- on the command line.
For example, to pass in both a script argument and a topazini:
getDbfHistory.stone -b 100 -- -I .topazini
This can be used to pass in any other topaz argument, for example to override the default use the rowan extent to execute a SuperDoit script:
simple.solo -- -C GEM_SOLO_EXTENT=$GEMSTONE/bin/extent0.dbf
Most topaz directives can be used, and will override those specified by the SuperDoit infrastructure or the configured environment. Any use of the topaz -S option is ignored, and solo scripts may only log in linked.
Usage
The usage section includes a section of text that will be displayed when the script is invoked with -h or --help.
usage
$basename [-h] [-D] [-b <number>] [-- [-I <topazini>]]
Return DbfHistory for the stone specified by <topazini>, or
if not specified, the default .topazini for the environment.
The optional -b argument specifies to return only the first
<number> bytes of the DbfHistory.
%
The usage section is optional. If it is not included in the script, when -h or -help is invoked, the default help with -h/--help/-D/--debug options is displayed.
Coding in the doit section
The doit section consists of GemStone Smalltalk code. Within the doit, self corresponds to the instance of SuperDoitExecution. This provides access to the script arguments and environment, and set of convenience helper methods.
Available methods include (but are not limited to):
executionStoneName
Returns the name of the running stone; for solo, this will be gs64stone.
globalNamed: globalName
Return a global variable with the given name.
globalNamed: globalName ifAbsent: notfoundblock
Return a global variable with the given name, executing the notfoundblock if it does not exist.
positionalArgs
Returns an array containing all positional arguments from the command line.
stderr
The stderr that the script writes to.
stdout
The stdout that the script writes to.
noResult
At the end of the script, this allows you to return with no messages on the command line.
exit: errmsg withStatus: anInteger
Exit the script, writing errmsg to stderr and with the OS process exit status set to anInteger.
exitWithStatus: anInteger
Exits the script, with the OS process exit status set to anInteger.
isSolo
Return true if in a solo session.
log: anObject
Write the argument in STON format to stdout.
logErrorMessage: aString
Write the given string on stderr.
Output
At the end of a superDoit script execution, it displays the final object (or the exception, if an exception occurred), as a printable String, or the exception if an error occurred.
To avoid this output, the final statements of the script can send self noResult.