Multi-Process RPC Listener Host System for RC5

[jpbro]

What is this?

Short answer: A multi-process host system for vbRichClient5 RPC Listeners.

Longer answer: Typically you have a single RC5 CRpcListener object instantiated in your server EXE, configured to run with mulitple listeners in a thread pool (e.g. an 8 listener handler thread pool in a single EXE). The new approach used here spawns multiple processes, each with a single listener handler thread pool.

Why?

There are some advantages to using multiple processes vs. multiple threads, such as:

• Each process has access to a full 2GB of memory, which can be advantageous for high memory workloads (1 connection gets 2GB of memory if needed).
  
  
• If a process crashes for any reason, at most 1 user connection will be affected. Crashed processes will be automatically respawned in short order, minimizing downtime.
  
  
• Related to #2 - if you have a long-running RPC call that times-out, RC5 will terminate the listener handler thread, which can lead to problems. In a process model, we can unload/reload in this case to maintain overall system stability. TODO: I'm looking for a good way to detect when this happens so I can restart the appropriate listener host process (would appreciate some feedback from Olaf on this - maybe an event in CRpcListener could be raised?)
  
  
• We can periodically unload/reload listener processes while leaving existing connections in other listener process unaffected. This can ensure that any bugs like memory leaks, unreleased locks, etc.. have minimal impact. Of course, it is critical to fix these bugs ASAP but sometimes availability is also critical while you hunt down a hard to squash bug. This is also useful for testing system resiliency by removing listeners periodically we can ensure our upstream client apps/reverse proxies are able to cleanly and quietly handle dead listeners by trying another port.

What's the catch?

There are some disadvantages to using multiple process vs. multiple threads, such as:

• Processes use more memory than threads. Each Rc5RpcHost.exe takes about 20MB before any memory used by your app classes.
  
  
• It can be difficult to manage startup, monitoring, and shutdown of multiple processes. I think I've handled this on your behalf though using Windows Job Objects and optional monitoring an outside App.ExeName & PID shutdown. All spawned Rc5RpcHost.exe processes should shutdown when required when the parent app closes, or when a /shutdown command is issued against a single Rc5RpcHost.exe process. The system also watches all spawned listener processes for crashes/unscheduled shutdown and will respawn as them as required.
  
  I've been running a slightly different version of Rc5RpcHost.exe in production for a few months and have 0% unscheduled downtime, so things are looking quite resilient. That said, there may be bugs that will need to be worked out and I am committed fixing them as they are reported to me.
  
  
• Your RPC listeners will use 1 TCP/IP port per process (e.g. 32 listener processes will use 32 ports), whereas with the traditional thread pool approach you use a single port for multiple threads. This means you will have to either change you client app to randomly pick a port to connect to for each RPC call (e.g. pick a random number from 22222 to 22222+<# of listeners>-1) OR (my preferred method) configure a server like Nginx to stream connections it receives on a single port to one of your available backend RPC listeners. With the stream/reverse proxy approach you need only a single WAN facing open port and Nginx can bounce around backend ports to a free listener for your connecting client application. Needless to say, there's a bit more configuration work fo you to do here vs. the single process/multi-thread approach.

Should I use it?

I recommend you weigh the above pros and cons and decide if the multi-process approach is suitable for you.

Getting Started

1. Extract the source files from the archive located here: Rc5RpcHost.zip
   
   
2. If you don't already have vbRichClient5 installed on your development system, download the files from vbrichclient.com and put them in the extracted Rc5RpcHost\bin\libs\ folder and register vbRichClient5.dll. If you already have vbRichClient5 installed on your development machine, copy vbRichClient5.dll, vb_cairo_sqlite.dll, and DirectCOM.dll to the extracted Rc5RpcHost\bin\libs\ folder but do NOT register them there.
   
   
3. If you don't already have VbPcre2.dll & pcre2_16.dll installed on your development computer, download them from here to the extracted Rc5RpcHost\bin\libs\ folder and then register VbPcre2.dll. If you already have VbPcre2.dll and pcre2_16.dll installed on your development computer, copy those files from the installed location to the extracted Rc5RpcHost\bin\libs\ folder but do NOT register them.
   
   
4. Open extracted Rc5RpcHost\Rc5RpcHost.vbp file and compile Rc5RpcHost.exe to the extracted Rc5RpcHost\bin\ folder.
   
   
5. Copy your existing RPC DLLs to the extracted Rc5RpcHost\bin\libs\RPCDlls\ folder. If you don't already have any DLLs that you are using for RC5 RPC calls, then let me know and I can provide instructions on how to create a test DLL.
   
   
6. When the above is complete your folder tree should look like this:

Rc5RpcHost\
Rc5RpcHost\<A bunch of VB6 source files>
Rc5RpcHost\bin\
Rc5RpcHost\bin\Rc5RpcHost.exe
Rc5RpcHost\bin\libs\
Rc5RpcHost\bin\libs\vbRichClient5.dll
Rc5RpcHost\bin\libs\vb_cairo_sqlite.dll
Rc5RpcHost\bin\libs\DirectCOM.dll
Rc5RpcHost\bin\libs\VbPcre2.dll
Rc5RpcHost\bin\libs\pcre2_16.dll
Rc5RpcHost\bin\libs\RPCDlls\
Rc5RpcHost\bin\libs\RPCDlls\YourRpcDll.dll

Using Rc5RpcHost.exe

Once the above steps have been performed, you can start the Rc5RpcHost.exe with the following required parameters:

/start <NumberOfListenerProcessesToSpawn>
- For example /start 5 will start one Rc5RpcHost.exe process in spawner/monitor mode and 5 Rc5RpcHost.exe process in listener mode.

/host <IpAddress>
- For example /host 127.0.0.1 will configure all listener processes to bind to address 127.0.0.1.

/baseport <Port>
- For example /baseport 22222 will configure all listener processes to bind to port 22222+offset. So for /start 5 /host 127.0.0.1 /baseport 22222, your will have 5 listener processes bound to 127.0.0.1:22222, 127.0.0.1:22223, 127.0.0.1:22224, 127.0.0.1:22225, 127.0.0.1:22226 (one process per IP : port).

<optional> /parentproc <AppExeName:PID>
- For example, /parentproc MyService.exe:12345 will tell the Rc5RpcHost.exe processes to monitor an application named MyService.exe with a PID of 12345 for shutdown. If MyService.exe PID 12345 ever closes, then all Rc5RpcHost.exe processes that are monitoring will also close. This is useful if you have a parent service that you want to tie your Rc5RpcHost.exe processes to.

You can shutdown all of your Rc5RpcHost.exe processes at any time by using the /shutdown command line parameter.

Once you've started your Rc5RpcHost.exe processes, they will behave just like your old single process RPC listener. This means your client applications can connect to and make RPC calls just like before using the vbRichClient5.CRpcConnection class & .RPC method.


Hope some of you find this project useful, and Happy New Year to everyone at VBForums!