Modular Conundrum

[Shaggy Hiker]

I'm working on a project that is based on modules and plug-ins. Any of these modules might alter some part of a database in various ways. One module might write or edit one table, while another module might write or edit other tables.

The modules all communicate via events raised on a common object. Therefore, when a person using module A enters a certain change that would edit data managed by module A, the module raises a Proposed Edit event such that all other modules that might be affected by the edit will have a chance to accept or reject the edit. If no module rejects the edit, then module A makes its changes and raises an Edit event, at which time all the other modules make any changes they need to make.

That's the basic pattern, but there's a problem. If I don't share a transaction during the edit, then module A can successfully edit their tables, but module B might fail to edit their table. In some cases, the failure by B should rollback the changes made by A. Therefore, it would make sense for module A to share an open transaction for all the other modules to use while making their changes, and this gets either committed or rolled back, depending on whether everyone was successful, or not.

The problem with that design is that a person could write a new module C that had a bug which caused it to fail to update correctly. That would cause it to signal that the transaction failed, such that all the other modules, which had been working up to that point, would no longer work because this new module kept canceling the transactions. In other words, the modularity of the program has now been lost. Every module depends entirely on every other module that even cares about its data.

In most cases, what would be lost if certain modules failed would be minor, whereas what would be lost if other modules failed would be catastrophic, as there would be no way to re-create the lost data with any kind of ease.

As long as this explanation is, it may not be sufficient. At least it might start a discussion. Do the actions of all the modules act in the same transaction, or should they be left to succeed or fail on their own?

[NickThissen]

It seems logical to me that the modules are really separate entities and have their own database communication. Your example makes this clear: if someone writes a module C that has a bug, then everyone using module C has a completely useless application that no longer does anything, all because module C doesn't work. That would be like Windows would stop working when you install a buggy application.

If something goes wrong in module C, then that should not affect all other modules. Unless of course module C depends on some other module (not sure if that is a possiblity). I could imagine one module writing some data that corresponds to data written by another module. In that case, if one fails, so should the other. But this implies a direct link between the two modules. If this is a possibility you should keep this in mind in your design; give a module to possibility to block other modules' work, but leave this off by default.

Perhaps you are worried the database work might be slower if you setup a new connection/transaction for every module, but surely the number of modules will be relatively small (at least not hundreds, right?) so I wouldn't worry about that too much.

[Shaggy Hiker]

Unless of course module C depends on some other module (not sure if that is a possiblity).

It is this very point that I have been pondering for the last couple hours. The problem with a modular program is that I have to consider what will happen for modules that haven't even been contemplated yet. Just answering that question for the modules that HAVE been created is a problem that will take a few hours of thought.

I agree that it is simply wrong to have all the actions by all the modules performed inside the same transaction, so I will stop even considering that option. I was headed in that direction already, but my mind is feeling sluggish, at the moment.

I then began contemplating what the consequences are for certain actions working followed by others failing. That's going to be a joy, as well. Some sequences of failures would have no significant impact. Some spurious records would exist, but nobody would ever see them because they could be rejected easily enough. However, there are some sequences that would result in numbers not adding up right. The only thing I notice about that, so far, is that this will always be a result of either a bug (in which case I fix it), or something going wrong with a SQL action (a real exception, which should be exceptional). Therefore, I'm thinking of logging any SQL failure somehow. I guess I can't put it in the database (actually, I think I will try), so a log file would make sense. A file containing the SQL command that failed to run (there's effectively nothing but raw SQL statements throughout the whole program for reasons that I won't go into), the module and function that it failed on, the exception that was raised, and the time at which this occurred. I should then be able to add an administration module that would have a chance at being able to fix corrupted data by running those commands at a later time.

This may be overly cautious, on my part. After all, I'm not really talking about network connections, so if the connection failed for one transaction, it will fail for the rest, and every other cause would just be symptoms of a bug. Still, it seems like the way to go.

[NickThissen]

Why not let your modules raise an event when such an exception occurs? Then your main application handles that event and, in turn, notifies all other modules that an exception occured and on which module. Each module is then responsible for reacting to exceptions in other modules. For most modules, probably, they don't need to react at all, but for the few that do need to take action, there's the opportunity to do just that.

Still, logging exceptions is always a good idea, so certainly don't skip that part. I wouldn't put the logs in the database though, if your exception is caused by not being able to connect to the database there would be no way to log that. A simple text file (or xml file more complex log files) is probably better.

[Shaggy Hiker]

Right, logging to the DB does have a few obvious flaws. Still, I suspect that the majority of the writes would be a result of bugs. I'm already logging information about that, and raising some events associated with errors and exceptions (not JUST exceptions). The purpose for those events was solely for logging. After reading what you just wrote, I think I need to elevate that mechanism in importance and functionality. I have never run into a situation where one module can respond to the failure of another module, but that's not to say that it can't happen, so the notification is good.

I might as well note that the main module doesn't handle or notify. All modules share access to a single object, created by the main module on startup, and passed to all the other modules as they are created, which raises all events. Any module can raise any event and respond to any event because they all share that common object.

[Evil_Giraffe]

First up: Do you actually need this system to be modular in the sense you're describing? Plugin systems are so fraught with issues that in my experience it is better to avoid them unless you have a very compelling reason, and the only compelling reason I have ever come across is when you have a commercial piece of software. My understanding was that you wrote in-house LOB software?

If you decide that a plug-in system really is what you need, then you need to think of it differently. Your "ProposedEdit" event sounds like multiple plugins can "own" a piece of data. This is a sure-fire recipe for a complete headache. (Plus, you can imagine a malicious plugin that simply rejects all proposed edits)

My general advice here would be to completely rethink what you're doing, but without knowing exactly what problem your app is trying to solve, and how the modules will be delineated in terms of their responsibilities, it's a bit hard to give anything more specific.

[Shaggy Hiker]

Don't read this unless you feel like it. I'm not sure that it is coherent, as I was thinking about some design alterations as I was writing it, and don't feel like editing it to give it a start and an end.

First up: Do you actually need this system to be modular in the sense you're describing? Plugin systems are so fraught with issues that in my experience it is better to avoid them unless you have a very compelling reason, and the only compelling reason I have ever come across is when you have a commercial piece of software. My understanding was that you wrote in-house LOB software?

That's roughly true. The situation, in this case, is that the only thing certain is that whatever is written will be incomplete. Entire branches of the program may or may not even be thought of, yet, so the point behind the modular approach was to attempt to write something that will work now, while leaving it open for further expansion. The one thing most notable about the problem domain is its unusual fluidity. Identifying the core elements that have persisted for decades is harder than identifying the parts that have changed over the years. Still, those core elements do exist, and the fluid parts can be managed.

It would certainly be possible to write this as a single piece of software. I noticed a while back that the modular system is not quite as useful for extension as I had originally hoped, though that is simply because I will probably be the only one extending it. Still, the design has a certain elegance to it. The various pieces have very sharp boundaries with carefully thought out interactions. The goal is that, when a new piece of data that needs to be included is identified, the module for creating that can snap into the existing program without roiling other parts.

If you decide that a plug-in system really is what you need, then you need to think of it differently. Your "ProposedEdit" event sounds like multiple plugins can "own" a piece of data

.

After thinking about it further, I believe this is not the case, though I thought it was (to some extent) when I wrote this. The data that was causing me doubt was a case where the same items are being counted in two entirely different ways. The count must come out the same, but the two modules have independent counts. This is FAR from ideal in any database, but it is the situation I am faced with.

To put it in more concrete terms, the program deals with fish in a hatchery. These fish are naturally ordered into groups based on a couple properties (such as species, brood year, stock, and at least one other category). This grouping arises from the origin of the fish, and after that point, most of those properties can only be distinguished by DNA testing, at best (stock, while critical to management, may not be distinguishable even through DNA testing).

At some later date, the fish are marked. There are a variety of marks used, and these marks do not line up cleanly along the boundaries of the groups (sometimes they do, othertimes they don't), but marks are readily distinguishable, and are one of the primary reporting metrics.

So the one accounting method is based on numbers by those initial properties, and the other accounting method is based on numbers by marks. The sum of each must be the same, but the borders between the groups can be fluid (and all of the numbers are fuzzy, though they are treated as if they are not). Naturally, there are other complications, but overall, it's an interesting problem with a fairly elegant solution involving matrixes.

If some fish die, it is easy to account for that loss by reducing the numbers in one location. However, if the fish are marked, the loss must also be accounted for by the fish marking system. Only these two pieces are really interdependent.

After thinking about that, I guess I need to consider whether those two pieces should be consolidated further. All the rest of the data will be subsidiary to those elements. As long as they work, the rest can fail without consequence.

[Evil_Giraffe]

Ah, it's the fish hatchery still? The more I hear about your domain, the more interesting it sounds.

Okay, so when you say "modular", you don't necessarily mean plug-ins? It sounds like you're wanting ongoing development on a piece of software that is deployed internally to your business? That is fairly standard for LOB development. Plugins are a solution to an issue you don't have - you have control over the deployment of your application.

However, with iterative development you do have other issues to account for. You don't know upfront exactly what the future requirements will be. This, again, is de facto for LOB systems. The answer to this is intense effort on keeping technical debt under control, focusing on the core fundamentals of OO (SOLID principles etc). One part of managing the complexity is Separation of Concerns - this can be described as "modular" I guess, so if that's how you meant it, you're definitely on the right track, but it's not what I think most developers would mean by the term.

Now, as to your issue with the counting. It strikes me from what you explain that "the two counts must be equal" is not an inherent property of the system. Rather, it is a business result that this software is supposed to help manage. Therefore, if the counts don't match up, that is not a system error. It is a business error that users need to be alerted to, and have a way to either correct the counts if they were wrong, or take some action with the actual fish stocks and update the system with the results, or perform "write-offs" (I suspect they don't use this particular term with fish?).

Therefore, one "module" really does not affect the other. Where the link is is that the UI for the user should present the two counts next to each other and highlight when the numbers do not match, perhaps?

You might want to have a quick flick through the series of articles by Ayende about the Macto prison management system he is "designing" as an example. The sections on counting the prisoners seems quite relevant to your problem.
http://ayende.com/blog/tags/macto

[Milk]

Funnily enough I was having a chat about counting fish with one of my more fishy clients just the other day. They had one particular hatchery where the counting error was all over the place, it turned out one member of staff had really bad eyesight.

[Evil_Giraffe]

Is... is that a... euphemism? I... actually, no. I don't want to know.

[Milk]

No euphemism, genuine conversation about estimating the number of juvenile wrasse at a particular Marine Environmental Research Lab in bonny Scotland.

[Shaggy Hiker]

You have to be careful when working with wrasse. Say the wrong thing in the wrong company, and you could get slapped with a suit....or just slapped.

Okay, so when you say "modular", you don't necessarily mean plug-ins? It sounds like you're wanting ongoing development on a piece of software that is deployed internally to your business? That is fairly standard for LOB development. Plugins are a solution to an issue you don't have - you have control over the deployment of your application.

Actually, I do mean modular. I have a whole lot less control over this than I would like. The expectation is entirely that other agencies will be writing plug-ins....though the way I feel about it at the moment, the true expectation should be that the whole project will collapse in a flurry of inter-agency conflict, but it is fun while it lasts.

By the way, as a result of Evil Gs first post, I have realized that I made a fundamental mistake in the project. I REALLY wanted to isolate parts of the project. The core would be the concept of rearing units and groups of fish in the rearing units. That's complicated enough, but it is the essential fundamental of a fish hatchery: Fish, and where they are located. I wanted everything else to be as nearly detachable as was possible. Each piece has a single class that implements one or two interfaces that place it into one or two categories of items that the main program can deal with via the interface. All the rearing units are also isolated into dlls though they don't implement the interface. Instead, they are all classes derived from a common base class. Each derived class handles a single shape of a rearing unit, and knows how to draw itself, how to deal with having a divider screen added to it (adding one screen to a rectanglular raceway divides it into two pieces, but adding one divider screen to a torroidal or circular raceway doesn't divide it at all, only adding a second one divides it, and so forth). The main program holds a list of the base class, and runtime polymorphism does the rest.

In keeping with this isolation of pieces, I decided that marking fish should be a distinct act isolated into a module. That was a mistake. What I described earlier was true, all fish are categorized by stock, species, brood year, something else, and marks on the fish. The fact that fish don't receive marks until they have grown to a certain size, is irrelevant. A fish prior to being marked still has a distinct mark: Nothing. Once I realized that, I found that there was a much better way to write the thing such that the question of transactions goes away.

All criticisms of any design can make it stronger. In this case, EG's skepticism about the design based on what description I gave, prompted me to look at a core design decision differently, which broke me out of a rigid mind-set about the isolation of different parts.