Robustness - representing recovery from a crash

[Techno]

What would be the best way in terms of trying to make an application 100% uptime (mission critical)?

now im not referring to try catch blocks.
Lets say we have an application which is processing some data in a mutlithreaded environment.

The application either hangs, crashes, exits (user closes it/ends the task) or the server has a power failure.

We want to be able to still process that message that it was processing in some way (and any others which have not been processed).

in such a requirement, what should be done? What techniques or methods should be used to ensure that the messages will be processed?

[abhijit]

Have you looked into Advanced Queuing

I believe the SQL Server equivalent is Service Broker

[Techno]

thats is correct however im not looking in terms of the DB context at all

scenario:

systems are in clustered mode.
if one of the nodes goes down, the other kicks in and takes over. great.

but what if the WCF service crashes, for whatever reason - how do you recover from that? I dont think bindings at this point matter at all but it would be using the msmqintegration binding (not netmsmq)

[szlamany]

The WCF SERVICE completion of a request has to be a response to the client with a status.

It's the client responsibility to ensure the request gets a proper success status.

The WCF SERVICE needs to send response ASAP - but only after the logging of the request gets done - not necessarily completion of the processing of the request - but the logging of it in some fashion.

The WCF SERVICE takes responsibility for pushing the data to the destination - and tells the client success.

Is this what you are asking?

You say this

The application either hangs, crashes, exits (user closes it/ends the task) or the server has a power failure

Well that seems to be several applications in one sentence - right? Each piece has it's own recovery possibilities.

[Techno]

correct. thanks for pointing that out. sorry got tons of things in my mind and trying to seperate them but proves difficult!

the service does NOT necessarily need to send a response back to the client because it could be a OneWay system.... client sends and thats it (fire and forget).

the context im talking about is MSMQ and WCF (using native binding/mode - msmqintegration binding).

So lets say the WCF service crashing during processing of a message which was delivered (in a transactional queue) to the WCF service. how can the WCF service recover from this and continue?
lets say if there was some unhandled exception during the service.... is it possible for WCF service to recover? and if so - how?

I played with a sample on MSDN which was downloaded and shows an MSMQ to WCF example. (directory: WF_WCF_Samples\WCF\Basic\Binding\MSMQIntegration\MsmqToWcf\CS)

now, in this case, client sends message to MSMQ, server receives and processes it. great.
But on the server side, if I did a throw new Exception("Test"); then client doesnt get notified about it - fine, I expect that and that is the behavior.

but at this point, any NEW messages coming through the queue would NOT be processed by the service. I had to purge the queue and resend requests, something that shouldnt be done nor would I want to.

any thoughts on this?

[szlamany]

Wow - never did WCF with message queues - just googled a bit to see where you are at - sorry nothing I've ever done.

I'm developing some service apps right now - but doing proof-of-concept at this point so I'm doing them in a winform for presentation to the team I'm working with.

[Techno]

haha. no worries my friend
ditto im also doing a POC

but im not sure if it matters or not here. For example if you had a WCF client talking to a WCF server AND the method calls are OneWay - the client wont get the exceptions (as you would expect).
WCF server is faulted. so how can it recover from the fault?

I guess from the client side, they would need to abort the connection and recreate the channel factory and start communicating again.... right?

[abhijit]

In the scenario, you just described, if the client doesn't get the exception, it doesn't get a success code either.

So the client SHOULD check not just for a presence of an exception, but also for an absence of a success code.

[Techno]

how do you mean? What would be the point on checking for an absense of a success code?
when a method in a contract on a WCF service is marked as OneWay then the client does a fire and forget. so why would and how would the client check for any form of any status code? it cant.

[szlamany]

I believe the OP is asking how the service can re-start and see transactions that it should have been still processing...

So lets say the WCF service crashing during processing of a message which was delivered (in a transactional queue) to the WCF service. how can the WCF service recover from this and continue?
lets say if there was some unhandled exception during the service.... is it possible for WCF service to recover? and if so - how?

I do not know the restrictions of WCF services - if this was me I would "log" the message as being "in progress" when the service starts processing it. Then "log" it as "done" when complete.

Whenever the service has to re-start I would look to see if any transactions are "in progress" and handle them first.

I don't know if this is in the realm of what you can do with a WCF service - seems you could...

[Techno]

thanks!

well yes of course, normal logging will be put in place. I guess because this system must be 100% uptime (mission critical), its difficult to try and find something or some concept and play with it or even to begin to write your own from scratch especially when it is a POC where too much time shouldnt be spent "demo'ing" it but making sure you have concrete facts and a simple example. But then I guess the POC varies depending upon the nature of the beast.

I also think from my perspective, nothing is clear yet in terms of how the WCF service will be hosted. Either as a Windows Service or Console app or IIS/WAS.

IIS/WAS seems a great option.

but I think in either case, whatever hosting you go for, the service must be recoverable and continue to work without manual intervention.

[abhijit]

Ok. I can relate to this scenario. We have a product that has a batch upload feature built into it.
This batch upload feature takes an XML (file or stream) and then writes records to the database.

The way this works is that the feature takes a stream and writes to a file and places it in a folder. Then it queues one file after the other. At the end of each successful write, it moves the file from that folder into another folder. So if you have a crash or restart or anything like that, it will pick up from where it left.

You could configure the upload feature and set to to Auto or Manual. Auto will automatically pick up from where it left. In case of Manual, the user has to actually hit the submit button to resubmit the files in that folder.

[szlamany]

...well yes of course, normal logging will be put in place.

I'm not talking "normal logging" - I'm talking more like how MS SQL Server has a TRANSACTION LOG that allows it to "re-start" and rollback half done transactions.

Can you "serialize" the MSMQ message and store it in a file? Doing this in the WCF service when the message is first gotten?

[Techno]

of course. that will work in most scenarios and makes sense. however when it comes to MSMQ and WCF...slightly different . I do apologise I wasnt being so clear on the original post but wanted to make sure I didnt get into specifics... but now its noted I do need to get into specifics about the technologies concerned.

Let me give you an example of what I discovered. I do not know if it implies on any other WCF services but would be good to know.

WCF Client sends msg to MSMQ
WCF server gets this message from MSMQ.
WCF server throws an exception
WCF server stops processing messages even if there is a new client connection.

the only way to get the server to work again is to purge the MSMQ queue - something you dont want to do. so this is what sparked the question on why the WCF server service stops processing messages

[Techno]

I'm not talking "normal logging" - I'm talking more like how MS SQL Server has a TRANSACTION LOG that allows it to "re-start" and rollback half done transactions.

Can you "serialize" the MSMQ message and store it in a file? Doing this in the WCF service when the message is first gotten?

gotcha my friend.

MSMQ can be transactional yes. And that is what will be used. from what ive read, if there is an error in processing then the message will be put back in the queue ready to be processed again until the service detects its a poison messages.

but I think the concern here is to get the WCF service to still function incoming messages AFTER it had faulted.