VSD provides a graphical way to view the data that is recorded by statmonitor. When you open VSD on this data, you will see a very large number of statistics that were recorded for potentially a very large number of GemStone processes. Not all of these are going to be useful; your task is to locate the interesting and useful information.
You can start VSD from your windowing system or from the command line.
You may start VSD and automatically open a specific data file by:
prompt> vsd statmonDataFileName
When VSD is started, it displays the Main window. This window has a single instance and remains open while you are running VSD. When you are within VSD, the F12 key will bring the Main window to the top.
The Main window displays the list of processes and other information about the statmonitor file; here, you will locate and select processes and statistics for which you want to view charts.
This window also provides menus that allow you to set configuration options that control how your charts are displayed. The details for the various menu options are described in Section 2.8 .
Most of these display options are written to a .vsdrc file when VSD exists, which is read on startup. The next time you open VSD you will get the same options. Details of .vsdrc are here.
If you opened VSD with a particular data file or files as an argument, the main window will display the processes listed from that file. Otherwise, you need to load a data file, or start monitoring a running server.
To load a statmonitor file into VSD, use the menu item Main > Load Data File(s)...
This brings up a file dialog that allows you to browse for an existing statmonitor output file.
If you know the name of the file you would like to load, you can also type the full path in the File entry box.
You may load a statmonitor data file that is still being updated by the live system. As new data is added to this file, you have the option to have vsd update its view.
If you select File > Auto Update, VSD automatically updates your display, and any associated charts, any time the data file changes. Otherwise, if auto update is disabled, you can choose File > Update periodically to update the displays when you choose.
You can load multiple statmonitor data files, either keeping the statistics separate by using the Main > Load Data File(s)... menu option, or appending them as if they were a single data set by using the Main > Append Data File(s)... menu option.
For more details on working with multiple files, see Working with Multiple Files.
The statmonitor utility that is part of the GemStone server allows you to specify a sampling time in milliseconds rather than seconds, by using the -I option. Since versions of VSD earlier then v4.0 did not handle samples of less than one second, statmonitor historically did not write exact timestamps for subsecond sample rates; it treated the timestamp a counter that was incremented with each sample. The result was a series of timestamps that looked like 1-second intervals but represented statistic values at the requested sample increments.
GemStone/S 64 Bit version 3.3 and later, and GBS, write subsecond timestamps, adding milliseconds to the timestamp if needed. Earlier versions of GemStone, however, write relative timestamps for subsecond samples.
For statistics from older GemStone servers, there is no way to distinguish a true 1-second sample rate from a subsecond sample rate with nominal 1-second timestamps. When analyzing data with subsecond samples from earlier versions of the GemStone server, you must be aware of the sample rate when you load the file, and ensure you use relative timestamps and a time format of seconds.
You can use VSD to start statmonitor and automatically read directly from the data file this statmonitor is writing to. This options is not available on Windows.
In order to monitor, you must have $GEMSTONE set to a GemStone server installation, and the machine search path set to include $GEMSTONE/bin, to allow VSD to provide a list of running GemStone caches to monitor. However, you can monitor any GemStone version or product, regardless of the version of GemStone specified in the environment of VSD. VSD uses the current GemStone environment only to list available caches. It reads information on the processes, and starts the correct version of statmonitor for the selected stone.
Step 1. Bring up the Monitor window using the menu item Main > Monitor...
The Monitor window will display a list of all shared caches on this machine.
Step 2. Select the cache you wish to monitor, and the monitoring options.
These options are a subset of those offered by the server statmonitor utility. Options are:
Step 3. Click the Start button to start monitoring.
This will start statmonitor, and create a statmonitor data file in the specified location, or in the current working directory. If statmonitor startup fails, the Monitor status window will appear which shows the statmonitor command line with the error message.
The Monitor window will remain open. You can use the Dismiss button to close the window; this does not affect monitoring.
VSD loads the data file as soon as it has some samples in it (unless you have explicitly turned off Auto Update).
The statmonitor data file that is created by monitoring will grow as long as you are monitoring. If you have a large system and collect samples frequently, this file may become very large.
Step 4. Click the Stop button to stop monitoring.
If you have dismissed the Monitor window, you can re-open it using the menu item Main > Monitor. Select the cache you have been monitoring to enable the Stop button.
Note that the statistics data that is created will remain, and can be viewed later. It must be explicitly deleted if you no longer need it.
After you’ve loaded the data file or started monitoring, the VSD window looks something like this:
The upper list pane contains a list of all the entities—processes, thread-based sessions within processes, and other entities such as the disks—for which data is recorded in the statmonitor data. Depending on the options used to collect the data, this may include all GemStone processes or a small subset, and may include a number of non-GemStone entities including the operating system, disks, etc. While these are not all strictly “processes”, in the following discussion this term is used to include all entities that appear in the process list.
For each process, session, or other entity, the display includes:
If you are monitoring, processes that are currently running and for which samples are being taken are shown in green, and these number will change as new data samples are taken.
GemStone processes fall into a number of types, which govern which statistics apply. The common process types are:
Stn — Stone (and Page Manager in cases where this is a thread in the Stone )
Shrpc — Shared page cache monitor
Pgsvr — AIO or free frame page server, or a page server for a remote session
Gem — Gem; includes system Gems, RPC Gems, and linked sessions such as linked Topaz
Additional types will appear if you are also monitoring OS information, including platform-specific statistics.
This is assigned by statmonitor or GBS’s Stat Monitor, and displayed in VSD. It may be assigned for Gem processes within the GemStone server; see Process names in GemStone.
To reduce the number of processes that are visible in the list, the processes of a given type can be hidden, using the menu item Hide Instance By Type....
By selecting a column header in the main windows, you can sort all the processes by that column’s attribute. The current sort column is indicated by an up or down arrow next to the column label.
Within primary sort order, the results are further sorted by a secondary sort attribute. This is indicated by the column label in italic font. To specify the secondary sort attribute, right click on the column label.
Note that the secondary sort key direction (ascending or descending) is always the same as the direction of the primary sort.
For example, if you want to find the Stone, you can sort by Type, descending, which sorts the type Stn first. Or, if you know the session ID of the Gem, you can sort by SessionId and scroll until you find that Gem.
When you select a process, the list of statistics relevant for that process is displayed in the statistics pane. Multiple selection is allowed.
The pop-up menu option Select allows you to select all processes matching a particular criteria. The options are:
By type —select all processes of the same type as the currently selected process. Does nothing if no process is selected.
By statistic—select all processes for which the currently selected statistics applies. Does nothing if no statistic is selected.
The lower pane displays a list of statistics.
Each process type has a specific set of statistics that are relevant for that process. Many of these are unique for that process type; for example, CommitCount applies to Gem processes, while TotalCommits is meaningful for the Stone. Others, such as UserTime, apply to all processes.
When no process is selected, all statistics in the system are shown. When a particular process is selected, only statistics appropriate for that process type are listed. When multiple processes are selected, only statistics that are common to all the selected process types will be listed.
You may scroll down the list to locate the statistic you want. Typing in the first character will select the first statistic with a name beginning with that letter.
Notice that some statistics are listed in bold font, while others are in regular font. Statistics with the font in bold have non-zero values recorded.
You may hide statistics for which the value of that statistic is 0 for the entire sample set. These are the statistics that are listed in non-bold font.
To hide statistics where there are no non-zero values, check No Flatlines in the Main menu or in the right mouse button popup menu. Note that "No Flatlines" means "No Flatlines at zero", and does not apply to statistics whose values present a flat line at a non-zero value.
Each statistic has a characteristic called a level reflecting the amount of background knowledge about GemStone processes needed to use it with understanding. You can set up VSD to list only those statistics that are at, or below, a certain level of complexity.
To establish the levels of statistics that you want to display in VSD, choose Statistic Level from the VSD window’s Main menu:
The level of statistics you select will be recorded in your .vsdrc file, and will be used automatically the next time you start VSD. See VSD Configuration Files for details on VSD configuration files.
While many statistics have intuitive names, you will often need to look up details. Appendix B, “Statistics Reference”, has a complete list of statistics.
You should not expect to understand every statistic recorded for your system. Some of these statistics require an intimate knowledge of GemStone server internals, some are experimental, and some may only apply under specific server conditions. GemStone documentation, Techtips, and analysis from GemStone Technical Support will include specific statistics that are particularly helpful.
The information in the current version of VSD, including more detail, is displayed in the Statistics Information Window. To open the Statistic Information, use the menu item Main > Show Statistics Info.
This singleton window can be left open; it will display the details for any selected statistics as it is selected. You can also lookup statistics using the drop down.
After you have selected one or more processes and one or more statistics, you can display these results in a new Chart Window by clicking on the New Chart button. You can create a named chart by entering a name in the Chart name entry field.
If you have a chart open, you can add the selected statistics to an existing chart. Select the chart by name in the drop-down with the Chart name entry field, then click on the Add Line button. The Chart name is set to the most recently selected chart, so if you do not explicitly select a chart, the statistics will be displayed in the most recently selected chart.
The following shows the main window with one process selected, the Stone, and two statistics selected, TotalAborts and TotalCommits.
To view data for particular statistics, you will open a Chart Window. These VSD statistics charts allow you to view and analyze the data for the statistics you have selected. You may have many Chart Windows open.
There are a variety of ways to control the display and to perform operations on selected lines. The details for the various menu options are described under Section 2.8 .
When you first open a chart, all the values for all statistics are displayed with its default settings. Each line has a unique color and/or pattern.
At the bottom is the Legend - the list of process statistics that are viewed in this chart. Each is described by Process name and statistics name.
The axis of the graph are labeled; the horizontal axis is always time.
The vertical axis is labeled with the number of the units for all the statistics; this may or may not be meaningful, if the statistics used different units of measurement. For example, comparing the counts of the number of commits and the time waiting for I/O, by units, does not directly make sense. However, it can be very useful to see how each varies relative to the other over time.
You can customize and manipulate a VSD chart in many ways to make your analysis easier. A few of the more useful are described here.
You can select an individual line from the Legend at the bottom, or by clicking on that line in the graph. Selecting a line colors it red and displays information about that line.
When a line is selected, the following information is available:
When you have a selected line, you can copy the value of a point on the line using the Chart windows pop-up menu item Copy Y value. This menu item is only displayed when you right click while the cursor is over a point on the active line.
The copied value can be pasted into another application using control-V. Middle-button paste depends on the target application copy buffer and may or may not work.
The filter allows you to view either absolute values, rates of change per second or sample. Filter options are:
No Filter — Displays the values for the statistic exactly as they are expressed in the statmonitor text file.
PerSample — Displays the difference between two consecutive samples of the statistic.
PerSecond — Displays the difference between two consecutive samples of the statistic, divided by the number of elapsed seconds between the two samples.
Aggregate — Displays a running total for the per-sample deltas of the statistic.
When you have multiple lines with widely different ranges of values, you can increase the visibility of a line relative to the other line by scaling it.
Select the line, and in the upper right, use the arrow indicators to scale up or down the line.
Using the Line > Compute Scale or the Chart > Compute Scale All menu options will calculate and apply a scale based on the chart data values.
By default, S is selected for the transformation setting, between the filter list and the scaling value entry field. When S is selected, the numeric scrolling field is used to scale the selected line.
You may also select other options, rather than S. This allows you to transform the line by adding, subtracting, multiplying, or by dividing the value in the numeric scrolling field.
The menu item Line > Log Info will open (or append to an already open log window), summary information about a single line. For example:
A very useful tool is collecting statistics on a section of line. For example, you may want to know the commit rate over a single period of intense activity, rather than for the entire period of sampling, or the rate of growth of a CommitRecordCount.
The menu item Line > Log Delta will allow you to select two point on the selected line. It then opens (or appends to an already open log window), summary information about that section of line.
The useful information will be different for counter statistics vs. values. For CommitCount, which is a counter type and therefore already a delta, the average will be of interest; for values is as CommitRecordCount that provide unit values, the value delta will be useful.For more on statistics types, see the descriptions here.
There are many ways to control what you are viewing to make analysis of your data easier. Here are a few that are particularly useful.
Normally, when opening a chart or adding lines to an existing chart, one line is added for each selected process for each selected statistic. If you have selected a large number of processes — for example, all Gems — this may be cumbersome to see in the resulting chart.
In cases like this, you can combine values for a single statistic for all the selected processes. To combine values, in the Main VSD Window, check the value Combine in the Main menu or in the right mouse button popup menu.
By default, all lines in a graph are plotted according to Y-axis units scaled on the left-hand side of the chart. You can instead plot some of the lines according to units scaled on the right-hand side of the chart, which will allow them to use a different scale. This is useful when the lines use different units of measurement from those plotted on the left.
To change the side on which a line is selected, with the line selected in a Chart Window, uncheck the value Graph on left axis on the Line menu.
When monitoring a running GemStone server, over a long period of time there may be many samples recorded, so that the most recent samples are in a very narrow section of the graph along the right side.
Since in many cases the most recent statistics are of particular interest, you can configure the display to only show the samples for a fixed period of time before the latest loaded sample; as new samples are loaded, the older ones are no longer displayed.
The File menu item Set Rolling Time Window... , or Cntrl-shift-R, brings up the following dialog:
Rolling left trim, like auto update, applies to the current file, which is shown at the top of the Main window, in the field labeled File:. The name of the current file is displayed at the bottom of the Rolling Left Trim Window.
When statmonitor records statistics for Gem processes, it records them with the name “Gem”, or for linked topaz logins, “Topaz”. This can be modified to an application-meaningful name in the GemStone server, so this name is recorded in the statmonitor data. This can be particularly useful for regular maintenance processes, such as your backup or markForCollection, that are run from scripts.
The options to set the process name vary by the GemStone server version.
topaz> set cachename MyProcessName
can be executed to set the cache name. This is available for linked sessions in v2.4 and later, and additionally for RPC sessions in v3.4 and later.
System cacheName: 'myName'
Depending on how soon you can execute this after login and the statmonitor sample rate, you may have one or more samples showing the original default name.
Timestamps are recorded in statmonitor data files using GMT time. When these files are read into GemStone, VSD will translate these times into the current machine’s local time.
If you are viewing the data in VSD in the same time zone as the system that generated the statmonitor data, this means that VSD will display dates in the correct time, including adjusting for daylight savings time.
However, if you are viewing statistics in a different time zone, the timestamps will appear to be incorrect.
To correct this, use the main menu option Change Time Offset..., and select the number of hours plus or minus to adjust VSD’s display.
For example, if you are in U.S. Pacific Time and viewing statistics that were generated by a system running in U.S. Eastern Time, which is three hours ahead, add three hours:
VSD templates let you quickly create a chart with a commonly used set of statistics. Templates are helpful if you find yourself performing the same task frequently in VSD — for example, monitoring the same five or six statistics. By creating a template for the statistics that you want to monitor most frequently, you can automate the task of building charts.
VSD is shipped with a set of predefined templates. You can open charts on these templates using Template > New Template Chart, and choosing any of the predefined templates. If you create your own custom templates, they will also appear in this list.
You can create a custom template based on a chart you have created in VSD, or by editing the formatted text file that contains the templates. See VSD Configuration Files for more on the text file and how it is formatted, and editing this file outside of the GUI.
To create a custom template from VSD:
Step 1. set up a chart with the processes, statistics, scaling, and so on, that you want in your template.
Step 2. Chose the Chart menu item Save Template. This will prompt you to enter a name for the custom template. Hit the enter key when you are done.
You can now use your new template, from the Template > New Template Chart menu item.
if you want to remove your template, edit the .vsdtemplates file, and use the menu item Chart > Reload Template File to reload the templates file.
When you create a template that includes processes that are Gems or PageServers, the name of the process, as well as the type, is included in the template. This helps avoid an excessive number of lines appearing when the chart is opened.
The name that is included does not include any serial numbers that are displayed with the name in VSD. So for example, if the chart on which you are creating a template includes a Gem named Topaz19, the template will use Topaz* as the name. When you later use the template to create a chart, lines will be added for all Gems with the name Topaz.
Since you may have many Topaz processes, and cannot determine which one is performing a particular task such as backup or MFC, including Gem and Page server statistics in templates requires more consideration. Providing the Gem with a name, as described under Process names in GemStone, is one solution.
VSD provides a number of ways to work with multiple statmonitor data files. While analysis is often done using a single statmonitor data file or monitoring session, there are a number of cases in which you need to work with multiple files. For example:
Production systems should normally run statmonitor at all times, and archive the results for some period of time, to ensure that this information is available for diagnosis if a problem occurs. This usually involves automatically restarting statmonitor periodically, for example, starting a new data file every day at midnight.
When studying the progress of a operation such as a markForCollection, the statistics of interest may span more than one data file.
If you have statistics for remote caches, you may wish to display these along with the statistics for the Stone’s cache to track the interactions.
For example, if you are studying the effect of a configuration or code change on performance, you may have data sets for two runs of a particular operation, and want to compare them to understand the differences.
You can load multiple statmonitor files using the menu item Main > Load Data File(s)...., or by specifying multiple files on the command line.
If your multiple data files are logically a single data set (for example, you start statmonitor on the same repository, shut statmonitor down, and start statmonitor again), then you can append the second and later data files to the first file, allowing you to see all the processes and samples at once.
To open multiple files so they are appended, use the menu item Main > Append Data File(s), or the -a command line argument. The File list displays the name of the first file, with [Appended] following.
You must ensure that the files to be appended are specified in the correct order. The append operation does not detect and prevent loading data sets that are in incorrect order or overlap, and loading invalid sequences results in unpredictable behavior and possible errors.
VSD always has a current file, which is displayed in the File: list. Operations on the File menu apply to the current file. If you have multiple files loaded, you can use the File: drop down to change the current file.
Each loaded file has an sequence number, in the order in which they are loaded; this number is displayed in the File: field and drop-down list and in the File column in the process list.
In Single File mode, the current file is the only file that is displayed. In Multiple File mode, all files are displayed. This is toggled using the Main menu item Single File Mode.
In multiple file mode, you can see which file is associated with a given process in the File column. The following example shows three sequential sample sets on the same stone and topaz process.
When you ave multiple files loaded separately, the legend in the Chart Windows indicates which file the statistics values originated with by appending a [N] to the statistic name for each file, except File 1. The N corresponds to the File list index sequence number of that file, no number indicates it is from file 1.
If you are testing the impact of changes in code or configuration values, you may have multiple sets of statmonitor data that represent similar operations, that are offset in time. You can view these data sets overlaid by using relative time stamps, so each data set is charted with the same starting point.
The following chart includes the CommitCount for the Topaz process in each of the three files. Since Absolute Time stamps is unchecked, they all are displayed as if the starting time was the same.
Note the index following the process name in the legend that indicates which file each line is based on.
When GemStone software generates statmonitor data, each statistic is recorded with a meaningful name, such as "CommitCount". GemStone also provides application-defined statistics, which have non-meaningful names: for Gems, SessionStat00... SessionStat47, and for the Stone, GlobalStat00..GlobalStat47. These can be programmatically updated by customer applications to track application-specific information, and may be used differently by different application processes.
Some GemStone processes, such as markForCollection, reclaim, and backup, use Session statistics to track internal progress. While this is designed for GemStone Engineering analysis, this may be helpful in understanding your system.
Since analysis is easier with meaningful names, there are several kinds ways to manage aliases in VSD:
Statistics can be globally renamed in VSD, by including a directive:
sl_stat -alias existingStatName newName
in .vsdconfig or .vsdrc. This causes all statistics named existingStatName to be replaced by newName whereever it is used. newName should not be an existing statistic name.
VSD custom aliases allow you to associate a name, description, and other details that VSD uses for display, with an existing statistic name. This association is set by the name of the process (and, optionally, by GemStone version).
This is designed in particular to define meanings for SessionStat00.. SessionStat47. However, aliases can ba applied to any Gem statistic.
The aliases are defined via commands that are included in a VSD support file, either .vsdconfig or .vsdrc. To update an alias, you must shut down VSD, edit the file, and restart.
The setCtrAlias command has the following arguments:
setCtrAlias CacheNamePattern StatName Alias Type Filter Units Description [versionPatterns [writeToStartupFile]]
CacheNamePattern - cache name of processes to match, case sensitive
StatName - name of an existing cache statistic
Type - one of counter, counter64, uvalue, uvalue64, svalue, svalue64.
Filter - one of none, persecond or persample.
Units - Unit of measure for the statistic.
Description - A brief description of the statistic.
versionPatterns (optional) - * for all GemStone versions, or a list of GemStone version numbers that the alias applies to. This is primarily intended for the system aliases described starting here.
writeToStartupFile (optional) - Boolean (1 or 0) indicating if the alias will be written to .vsdrc. 1 means write the alias definition to .vsdrc, 0 means do not write to .vsdrc.
For example, if you have a process that acts as a watchdog to kill misbehaving sessions, your code can programmatically update session stat 0 with the number of sessions killed. To view the statistics in VSD, you can add the alias ’NumSessionsKilled’ to SessionStat00.
Defining the following alias in .vsdrc:
setCtrAlias WatchDog SessionStat00 NumSessionsKilled counter64
none {The number of sessions killed} * 1
In VSD, when you load the statmonitor file or monitor, it will display the statistic name for the WatchDog process as:
SessionStat00 (NumSessionsKilled)
The Statistics Information window will provide the description.
Process name matching can includes a *, which will match any portion of a string or nothing. However, use the * with caution, since process names with or without a * are treated as unique patterns, and only the one that appears first will be used. Once a process name is matched to WatchDog, WatchDog*, or *, other potential matches to the process name are ignored, and the statistics will not be applied.
You may set aliases in either .vsdconfig or .vsdrc. If you may run older versions of VSD on this node that will access the same .vsdconfig and.vsdrc, you should ensure that the definitions are conditionalized:
if {[llength [namespace which setCtrAlias]]} {
definitions of aliases
}
If the alias is defined in .vsdconfig with writeToStartupFile as 0 (false) or if Save Settings On Exit is false, then the alias definition will not be written to .vsdrc, but will be read from .vsdconfig.
If you wish to add aliases to .vsdrc, either writeToStartupFile must be true, or Save Settings On Exit must be false, otherwise the definition will be lost next time VSD is shut down.
The code that implements important GemStone operations such as reclaim and markForCollection collect a number of session statistics, to provide additional information for GemTalk Engineering analysis. These are undocumented, and many of them are obscure in the absence of source code. However, there are some that may be useful or suggestive when analyzing problems or performance issues.
The VSD distribution includes vsd.internal.stat.aliases.tcl, which contains aliases for the session statistics internally produced by GemStone operations. These are defined per GemStone version, and the statistics definition for each process in specific versions are provided, in addition to the alias names.
Note that since internal session statistics are not a supported or documented feature, there may be inaccuracies or version-specific differences that VSD does not detect.
Aliases work by matching the pattern of the Gem name in the statmonitor data file. For example, the ReclaimGem has a name like GcReclaim6. Internal session statistics aliases are automatically applied to named sessions.
Processes such as markForCollection and restoreFromBackup are executed by ordinary Gems, which by default have the cache name GemN or Topaz. If you know the session that ran the given operation, you can apply the alias by name.
To do this, in the Main window, select the session, and use the right menu button pop up menu. Apply Alias Template For Type > provides a list of templates that you can select from to apply to the selected Gem process.
You may also set the cache name to exactly match the template definition. For example, for an MFC session, execute:
System cacheName: ’MarkForCollection’.
And the MarkForCollection template will be automatically applied when the statmonitor file is loaded.
The Main Window process pane pop up menu item Remove Aliases is enabled if the process has an alias, either a custom alias from .vsdconfig or .vsdrc, or an applied system alias. Using the Remove Aliases removes either type of alias.
Removing custom aliases does not affect writing these alias to .vsdrc on shutdown; removes in this way are transient.
A system alias can be reapplied using the Apply Alias Template For Type menu. You cannot reapply custom alias; you will need to restart VSD to retrieve these aliases.
When running multiple instances of VSD, it can be helpful to be able to distinguish the VSD charts belonging to different instances. To support this, you may set the background color of the VSD instance. Once selected, all VSD windows for that VSD instance will display this color in all windows and charts.
Setting a master background color changes the window background, which ordinarily is a light gray, and also changes the background color for process lists, fields, and graphs which ordinarily have a white background.
The color may be set in a number of ways:
TK color names and RGB values are provided here:
http://www.tcl.tk/man/tcl8.6/TkCmd/colors.htm
TK provides over 750 color names describing over 500 unique colors.
After setting a background color, for example:
Note that the background color is not saved to the .vsdrc file, since it is intended as a way to distinguish different instances of VSD on the same machine, rather than as a configured color for VSD in general.
This section describes the menu items available on the Main and Chart Windows in VSD.
Select from the templates and open a new chart using that template |
|
If selected, for templates that have patterns using *, open templates using only the selected processes. |