Hi -
In my application, I need to be able to guarantee that processing for a re-used conversation is completed prior to starting processing the next (re-used) conversation. My application is based on the concepts from the sample posted on Remus's blog: http://blogs.msdn.com/remusrusanu/archive/2007/05/02/recycling-conversations.aspx#comments). Essentially (in this sample), we create a new conversation for each SPID and re-use the conversation, so that messages are sent through the queue (and processed in order) for each SPID. SPID was used in the sample code as an example of some application-specific "thing" that you care about message ordering for. To prevent a conversation from living forever (using up log/resources), they are ended after 1 hour using DialogTimer and a customer message type.
My conundrum is this:
Assume conversation 1 (on SPID 1) is flooded with a large number of messages just before the conversation timer expires. The DialogTimer then expires before the target queue is drained. The sample code (mentioned above) then creates a new conversation for the same SPID (with a DialogTimer of 1 hour). Until the queue for conversation1 is drained, we have 2 conversations being processed for the same SPID. This same problem would occur in any application where you re-use conversations for a period of time (using DialogTimer), and then start a new conversation when the DialogTimer expires.
So although I like having the idea fof being able to re-use conversations, I would need to guarantee that conversation 1 is finished processing before conversation 2 starts processing (for the same SPID, to be consistent with the sample above). If I could get these 2 conversations into the same conversation group on the target queue, the CG locking would solve the problem. But because conversation groups only apply to the initiator queue (when you begin dialog with related conversation), I have no "out of the box" way to control how the conversation groups are associated on the target queue. Remus posted an idea here (bottom of thread): http://forums.microsoft.com/MSDN/ShowPost.aspx?PostID=182646&SiteID=1, which was to just send a special message at the beginning of each new conversation (containing the conversation group to use), and then doing a move conversation to conversationgroupid on the target queue. I've tried this solution, and the problem is 1.) if the set convo message fails for some reason, the conversation group is not set and 2.) if the target queue seems to reject most of my move conversation commands with the error "The destination conversation group '<conversation guid>' is invalid." - which I am guessing is due to the fact that this convo group id is being used on the initiator as well.
Any ideas?
Thanks!
Terryc
Well...I have a solution...after re-reading Remus's idea here: http://forums.microsoft.com/MSDN/ShowPost.aspx?PostID=182646&SiteID=1, I realized that the target queue should not share the same conversation group as the initiator's conversation group. So I've created a table to track conversationgroupid as it pertains to the key I am using (SPID in the example above), and a proc to find the conversationgroupid if one exists (...and also insert the existing one into the tracking table if that SPID is not yet being tracked, and update the tracking table if the group no longer exists in sys.conversation_groups). I pass the key (SPID) along as the first message on the conversation, which triggers the target activation proc to perform this lookup to find/track the conversation group for that SPID on the target side.
So it seems to be working.
|||But the problem still remains. No matter what you do and how you arrange the conversation groups or the conversation keys, you will be faced with the same issue: at one moment you will be activated because the message N (sent on dialog D2) is in the queue, but the message N-1 (sent on dialog D1) has not yet arrived. You had already read the message N from queue, so what do you do with it?
To solve this problem you have to be really fancy on the target side and have a staging table where you can keep out of order messages until the in-order message arrives. You also need a tracking sequence number that spans dialogs to be able to detect the in-order or out-of-order conditions in the first place, and whenever an in-order message arrives you have to lookup the staging table to process also every formerly out-of-order message that now is in-order.
My sugestion si to revisit the sender side and consider what needs to be in order and what not, and keep the messages that are required to be in order in the same conversation. Usually there is a criteria (eq. same transaction, or same business object) that applies to the order requirement and can be mapped to a conversation.
|||I think I have that covered; I am using your idea for re-using conversations on the sender side, so the messages that need to be sent in order are sent on the same conversations.
On the target side, I am setting the conversations to a specific conversation group (as described above, the first message on each conversation is triggers a "move conversation" on the target side). I only ever have one conversation per "thing I care about message ordering for" until the Dialog expires, at which time a new conversation is started for that "thing"; so conversation D1 will have been processing for an hour before conversation D2 arrives to be processed. As I mentioned above, this is where I need some mechanism on the target to ensure that D1 completes processing before D2 begins.
When conversation D2 arrives, it sets itself to the same conversation group on the target as D1, and so D2 will wait until D1 completes before starting, because the group lock applies to all conversations in the conversation group, and there can only be one receive process per conversation group (please correct me if I'm wrong). So, I don't need to save messages off to a staging table.
Does that make sense? Please let me know if I've missed something... Thanks
|||In your example (link above) for re-using conversations (putting each SPID on its own conversation), how would you solve this problem? SPID 1 messages are sent on D1 until D1 expires and is replaced by D2, at which point you need to ensure message order across 2 conversations (until D1 finishes processing). In your SPID/re-use example, would you build the fancy tracking into your code on the target side, or could you just use conversation groups on the target side for D1 and D2?
Is there a clean way to re-use conversations with DialogTimer/[EndOfStream] (referening your example again)? People who use this SPID/conversation re-use concept need to be aware of the risk that messages could be processed out of order on when conversations overlap for the same SPID.
I appreciate your help in solving this issue! Thanks
|||It all depends on what is being sent.
The conversation re-use paradigm only applies to certain types of message patterns, when traffic is sent only one direction and there are no distinct business object updates being sent. Typical examples of such trafic are audit and change tracking. If the messages are audit records, then what is important is that no record is lost, but order is largely irelevant. In that case I would periodically reuse dialogs w/o concern to order.
If the messages are changes tracking (triggers on tables sending a message with each UPDATE/INSERT/DELETE) then the best (and maybe only) way to guarantee order is to send all updates on one long lived dialog. This means that all updates are then serialized on the sender side, thus preventing scalability. But there are cases when such update tracking makes perfect sense, eg. many remote sites doing updates at a slow rate that are all cumulated to one big central site. Other case the view of the 'replica' site where the changes are being tracked/applied does not have to be in perfect ACID synchronicity with the sender (eg. DW real-time ETL applications) and in that case the updates can be applied out of order, providing that the last update is always preserved (a row version or timestamp is needed for that) and such application can send using one conversation per SPID and reuse conversations.
When the requirements are more stringent on the 'replica' the updates have to be guaranteed that they are preserving order and the original transaction boundaries have to be kept as well (to prevent improper dirty reads), then the only possible way is to read and send the changes after they are applied, and that always boils down to a log reader agent that can scan the log and send all changes over one long lived conversation without introducing complete serialization on the sender transactions.
The conclusion is that order can only be guaranteed over one and only one long lived conversation. But more often than not the place to look is at the requirement to preserve order and relax it in a meaningful way (ie. w/o breaking correctness), and this always depends on the business meaning of the traffic.
HTH,
~ Remus
P.S. I mean 'serialization' in the sense of serial execution, not XML/binary wire formatting.
