5. X509 Mid Level Cache
This chapter describes how to setup and login using an X509-secured mid level cache (MLC).
5.1 Overview
In a distributed system over a Wide Area Network (WAN), with many remote nodes that are topographically distant from the Stone but close to each other, a mid-level cache can improve performance for the remote sessions.
This configuration involves not at least three nodes, which of which will contain a shared page cache:
- the Stone's node, which contains the Stone’s SPC
- the mid level cache node, containing the mid-level cache. Gems may run on the mid-level cache node in which case the mid-level cache acts as the Gem's cache.
- the leaf node where the Gem is running; the Gem's client application and be on the leaf node or on yet another node. This node contains the remote cache; it is remote from the Stone, but local to the Gem, so the term "leaf" is used to avoid confusion.
When the Gem needs a page but can't find it in its leaf cache, it first looks in the mid-level cache. If the page is not in found in the mid-level cache, it then forwards the request to the page server on the Stone’s host.
X509-secured Mid-level caches
You may configure your system to use an X509-secured mid-level cache on a node that will service X509-secured Gems on other leaf nodes.
As with ordinary remote caches, x509-secured mid-level caches are not compatible with non-secured mid-level caches, remote caches, NetLDIs, or Gems.
X509-secured mid level caches, like secured remote caches, require the NetLDI be started up and the full authentication initiated from the Stone’s node, before a Gem can connect and start using the mid-level cache.
The following are additional differences from X509-secured remote caches:
- An X509-secured NetLDI must be started on the mid-level cache node, before the connection from the Gem is initiated. This must be started with a configuration file including NETLDI_START_MIDCACHE, which specifically allows this Netldi to start a mid-level cache.
- During startup of the MLC cache, the MLC node’s NetLDI starts a HostAgent process on the mid-level cache node; this is distinct from the HostAgent on the Stone’s node. To allow login as HostAgentUser, the HostAgentUser’s certificates and private key must be configured. This extra HostAgent allows this cache to operate as a mid-level cache.
- The extra MLC HostAgent is counted as a session and will prevent the mid-level cache from automatically timing out and shutting down the way ordinary remote caches do.
5.2 Configuring and Starting the X509 Mid Level Cache
The following instructions provide the steps to startup, then connect to, a mid-level cache node.
The examples use the name brisbane for the mid-level cache node name, and as with the examples in Chapter 3, the Stone named devstone running on a node named alcatraz, and a remote x509 Gem is running on fiji. The NetLDI is accessed via the port number 54321.
Starting the mid-level cache NetLDI
1. Create Certificates and configure on mid-cache host
Create certificates
A HostAgent is started on the mid-level cache node, which logs into the Stone as HostAgentUser. This requires cert files for the HostAgentUser on the mid-level cache node.
The mid-level cache node also requires Host certificates.
unix> $GEMSTONE/bin/x509/newuser -s devstone HostAgentUser
unix> $GEMSTONE/bin/x509/newhost -s devstone brisbane
Copy to a conveniently accessible location
For convenience, you may create a new flat directory in which to place the required certificate files, and access it via an environment variable. These examples use $MyRemoteCertDir.
export MyRemoteCertDir=dirName
cp $GEMSTONE_CERT_DIR/stones/devstone/stoneCA/stoneCA-devstone.cert.pem $MyRemoteCertDir
cp $GEMSTONE_CERT_DIR/stones/devstone/hosts/brisbane/brisbane.chain.pem $MyRemoteCertDir
cp $GEMSTONE_CERT_DIR/stones/devstone/hosts/brisbane/brisbane.privkey.pem $MyRemoteCertDir
cp $GEMSTONE_CERT_DIR/stones/devstone/users/HostAgentUser/HostAgentUser.chain.pem $MyRemoteCertDir
cp $GEMSTONE_CERT_DIR/stones/devstone/users/HostAgentUser/HostAgentUser.privkey.pem $MyRemoteCertDir
Set up log directories
To make it easier to locate the log files and diagnose connection issues, it is recommended to create a log file directory, and direct all related logs to that directory.
These examples use the environment variables $remoteLogDir to refer to these directories
brisbane> mkdir $GEMSTONE/logs
brisbane> export remoteLogDir=$GEMSTONE/logs
2. Start the mid-level cache’s NetLDI
As with leaf X509 NetLDIs, the mid-level cache NetLDI has the responsibility of starting up the remote cache, which it does on instructions by the starthostagent utility on the Stone’s node.
Define a configuration file
One of the arguments to a X509-secured remote cache is a configuration file, which includes the parameters used to configure the remote cache, and for mid-level caches, there are additional configuration parameters required.
NETLDI_START_MIDCACHE = true ;
NETLDI_HostAgentUser_cert = pathForHostAgentUserCert ;
NETLDI_HostAgentUser_key = pathForHostAgentUserPrivateKey ;
You may also optionally include NETLDI_PORT_RANGE and other remote cache configuration options in this configuration file. Supported options are listed here.
Start the Mid-Cache NetLDI
The Stone must be running before you start the x509-secured mid-cache NetLDI.
For the mid-cache certificate-only NetLDI, you must include the arguments -S -U -R -J which allow you to pass in the required certificates, and the required -D argument to set the directory for log files.
In addition, you must also pass in the remote configuration file using the -E argument, which must include the configuration parameter settings NETLDI_START_MIDCACHE (set to true), NETLDI_HostAgentUser_cert and NETLDI_HostAgentUser_key.
For details on the startnetldi arguments, see Utility details for X509.
Example
To start a mid-level cache on node brisbane, the configuration file midcache.conf could contain:
NETLDI_START_MIDCACHE = true;
NETLDI_HostAgentUser_cert = $MyRemoteCertDir/HostAgentUser.chain.pem;
NETLDI_HostAgentUser_key = $MyRemoteCertDir/HostAgentUser.privkey.pem;
and the following would start the NetLDI on the mid-level cache node:
brisbane> startnetldi -E midcache.conf -S -D $remoteLogDir
-U $MyRemoteCertDir/brisbane.chain.pem
-R $MyRemoteCertDir/brisbane.privkey.pem
-J $MyRemoteCertDir/stoneCA-devstone.cert.pem 54321
3. Start the two HostAgents from the Stone’s node
When starthostagent is executed on the Stone’s node, and the -m argument node is running a mid-cache enabled X509-secured NetLDI, then connections are initiated that setup the argument node to use for a mid-level cache.
As with regular x509-secured remote nodes, a HostAgent is started on the Stone’s node to service that remote node, and the remote NetLDI is instructed to start a remote shared cache on its node.
In addition, the mid-cache NetLDI starts a HostAgent on the mid-level cache node. This remote HostAgent allows this node to act as a mid-level cache.
Internal Details
The MLC NetLDI starts the MLC HostAgent by initiating a topaz -r session, and executing the .ini file $GEMSTONE/sys/midhostagenttopaz.ini.
The Gem session logs in securely as HostAgentUser, using the CA cert passed in to startnetldi with the -J argument and the HostAgentUser’s certificate and private key as configured in the -E configuration file.
Once a secure login is established, the topaz session executes code to become the HostAgent.
Flow of Operations
The following diagrams shows the order of operations in the startup of a mid-level cache.
In this diagram, the steps are as follows:
1. The user starts the remote NetLDI using appropriate flags including the certificate flags, and the -E configuration file with NETLDI_START_MIDCACHE = true. These arguments configure the NetLDI to start in certificate-only mode, listen on a public port, and await connection.
2. The user invokes starthostagent with the appropriate arguments, specifying the node and mid-level NetLDI to connect to, and the required certificate files. The starthostagent process contacts the already-running cert-only NetLDI on the Stone's node.
3. The Stone’s NetLDI starts (forks) the HostAgent.
4. As is done for any X509 remote cache, the HostAgent logs in.
5. The HostAgent contacts the mid-level NetLDI on its public port, and performs mutual authentication using host credentials, creating secure connection (A).
6. The HostAgent connects and authenticates with the mid-level NetLDI again, and requests to fork a remote cache pageserver. The secure connection creates (B), which will be inherited by the remote cache page server.
7. As for any remote X509 cache, the mid-level NetLDI forks the remote cache pageserver based the -E configuration values, and starts the associated processes.
8. The remote NetLDI spawns a topaz process that will be used to start the Mid Level Cache HostAgent, passing along the location of the cert files for the login in step (9), which are in the NetLDI’s -E configuration file and its -J CA cert.
9. topaz requests the X509 secured login as HostAgentUser, using the supplied certs. This authentication creates secure connection (C), which will remain active, since topaz is the GCI client for the HostAgent.
10. The remote NetLDI selects and listens on a port N, which will be used by the mid-level HostAgent to accept a connection from the Stone’s HostAgent, in step (12). The NetLDI spawns the MLC HostAgent Gem on the mid-level cache node.
11. The remote NetLDI informs the HostAgent on the Stone’s node of listening port N for the HostAgent Gem, via secure connection (A).
12. The Stone’s HostAgent contacts the MLC HostAgent on listening port N and authenticates, establishing secure connection (E). Note that (D) is skipped, to make the following mid-level cache connection diagram clearer.
13. The MLC HostAgent, via secure connection (E), requests the Stone’s HostAgent to update the Stone, indicating it is running as a mid-level cache node.
14. The Stone’s HostAgent passes the information to the Stone.
NetLDI, cache, and HostAgent startup are complete, and a Gem can now connect to the mid-level cache on this node.
5.3 Connecting to a mid-level cache
Connecting a Gem to a mid-level cache is operationally much like connecting an ordinary session to an ordinary mid-level cache.
The Gem must perform an x509-secured login. After successful login, the Gem session connects to a mid-level cache using:
System class >> midLevelCacheConnect:
You may use the same mid level cache connect methods that were used in previous versions to make connections between an ordinary gem and an ordinary mid-level cache. These methods include keywords with arguments for configure the mid-cache; these arguments are ignored when connecting an X509-secured Gem to an x509-secured mid level cache.
System class >> midLevelCachesReport
To see that the cache type x509mid is present.
Flow of Operations
The following diagrams shows the order of operations when a Gem connects to a mid-level cache.
Figure 5.2 Connecting Leaf Gem to Mid-level cache
When an X509 mid-level cache connection is requested, the mid-level cache and associated processes must already have been started, so it is a matter of starting a thread on the mid-level cache to act as a pageserver, and connecting it securely to the Gem and the Gem’s HostAgent pgsvr thread.
This diagram is simplified to avoid details that are shown in previous diagrams. The (D) and (E) secure connections are existing secured connections setup during the initial login of the remote X509 Gem (Figure 3.2) and during the startup of the Mid Level cache (Figure 5.1).
1. The application calls System midLevelCacheConnect:, specifying the mid-level cache node.
2. Via secure connection (D), the Gem requests the mid-level cache connection. The Stone’s HostAgent knows that there is a mid-level cache on that node, and returns the listening port for that MLC HostAgent to the Gem.
3. The Gem contacts the MLC HostAgent on the listening port returned by the Stone in Step (2). The Gem presents the certificates and private key used for the X509 login that started that Gem to the MLC HostAgent, to establish secure connection (F).
4. The MLC HostAgent starts a pgsvr thread for the Gem.
5. The MLC HostAgent passes the listening port for the mid-cache pgsvr thread to the Gem.
6. The Gem passes the listening port for the mid-cache pgsvr thread to the HostAgent on the Stone.
7. The HostAgent on the Stone’s node contacts the mid-cache pgsvr thread in the MLC HostAgent, and they authenticate, establishing secure connection (G). This connection is between the MLC’s HostAgent pgsvr thread and the pgsvr thread for that session in the HostAgent on the Stone’s node.
The mid level cache is now connected and used by the Gem for page accesses.
You can verify that a session is using a mid-level cache by sending
System >> midLevelCacheAddress
Which returns the IP address of a mid-level cache that this session is connected to, or nil if this session is not using a mid-level cache.
Example
In this example, the topaz application is running on remote leaf node fiji, logging into the Stone on alcatraz. After login, it connects to the mid-level cache on brisbane.
fiji> topaz
<startup>
topaz > set cert $MyRemoteCertDir/DataCurator.chain.pem
topaz > set key $MyRemoteCertDir/DataCurator.privkey.pem
topaz > set cacert $MyRemoteCertDir/stoneCA-devstone.cert.pem
topaz > set netldi localhost:54321
topaz > login
<login details>
topaz 1 > run
System midLevelCacheConnect: 'brisbane'
%
true
topaz 1> run
System midLevelCacheAddress
%
10.94.162.81
Reconnecting
When there are multiple mid-level caches connected to the Stone, you can disconnect from one mid-level cache and reconnect to a different one in a single operation, using the method:
System class >> midLevelCacheReconnect: hostName
This is similar to midLevelCacheConnect:, but if the current session is connected to a mid-level cache, it will be disconnected before it tries to connect to the new mid-level cache.