MSMQ, NServiceBus

In the previous blog post we went over some of the MSMQ bascis. In this blog post we’ll touch on how to monitor, troubleshoot and backup MSMQ.

I’ve been using NServiceBus with the MSMQ transport for a while now and have faced some issues. Unfortunately, information about MSMQ is pretty scarce and sometimes outdated. This blog post contains links to resources that I’ve found useful for understanding some of the best practices around MSMQ administration.

Monitoring

Events

MSMQ logs informational, warning and error events under the Application Log. When accessing an object fails or succeeds, it also adds an audit entry in the Security Log. All message queuing events contain the “MSMQ” text in the Source column.

If you have enabled End-to-End tracing, you can find the trace events in Application and Services Logs/Microsoft/Windows/MSMQ/End2End.

Dead-Letter Queues

When a message expires, the queue manager puts it in one of the dead letter queues. This ensures that messages are not lost. A message expires when one of its timers (Time-To-Reach-Queue or Time-To-Be-Received) expire. There are two dead-letter queues, one for non-transactional messages and one for transactional messages. In order to keep track of expired messages, you should monitor the dead-letter queues.

External Transactions

MSMQ can take part in external transactions. You can monitor these transactions by using the Distributed Transaction Coordinator Transaction Statistics and Transaction List views. Since internal transactions don’t go through MSDTC, these can’t be monitored using this approach.

Performance Counters

MSMQ provides performance counters that are helpful for monitoring its performance.

The MSMQ Service performance object contains global information about the Message Queuing Service:

  • Total bytes in all queues – This is very important, since you don’t want to run out of disk space
  • Total messages in all queues
  • MSMQ Incoming Messages
  • MSMQ Outgoing Messages
  • Incoming Messages/sec
  • Outgoing Messages/sec
  • Sessions – The total count of open network sessions

The MSMQ Queue performance object contains counters for each individual queue. If you want to monitor the Dead-Letter queues, you should select Computer Queues:

  • Bytes in Queue
  • Messages in Queue
  • Bytes in Journal Queue
  • Messages in Journal Queue

Since MSMQ is disk intensive, it is recommended to spread out MSMQ’s storage files over multiple disks.

Troubleshooting

The good news is that, usually, MSMQ just works. Most of the problems that you encounter are caused by either DTC or not having enough disk space.

For investigating DTC issues, you should use DTCPing. Here is a blog post documenting how to troubleshoot DTC issues.

Disk space depends on the throughput of messages that flow through your system, their size, and the amount of time it takes to recover from a failure (and start processing messages again). You don’t want to start losing messages because you can’t store them on disk. You should monitor disk space using performance counters. If there is not enough disk space, you can either increase the MSMQ computer quota and maybe change the MSMQ storage location.

If you want to find out more about MSMQ troubleshooting, check out these links:

Backup and Restore

MSMQ comes with a command line utility – mqbkup – for backup and restore. Using this tool you can backup storage files, log files, transaction files, and Registry settings. I’ve also heard that QueueExplorer is a good tool for managing MSMQ, although I haven’t tried it yet.

Conclusion

In my limited experience with MSMQ, I have ran into a couple of issues. Most of them were caused by the fact that I didn’t monitor it well enough or I didn’t understand how it works under the hood. In this blog post I’ve summarized some of the best practices around MSMQ monitoring and troubleshooting. I’ll probably update this post whenever I learn something new about MSMQ.

NServiceBus

Recently, I stumbled upon an interesting problem related to NServiceBus Saga concurrency and coarse-grained locks when using NHibernate Persistence.

With NServiceBus, concurrent access to an existing saga instance works out of the box. This is because it relies on the underlying database system. With NHibernate, it relies on optimistic concurrency, by adding a Version column to your Saga Data. But there is a catch: NHibernate doesn’t support Coarse-Grained Locks. Let’s first explore this general problem and then we’ll get back to the NServiceBus Saga.

Coarse-Grained Locks

With a coarse-grained lock you basically lock a group of entities together. If you’re familiar with DDD, then you know that an aggregate should be a consistency boundary. When I save an aggregate, I want to save the entire aggregate, to ensure it’s consistent. Why would you need this? Here’s an example:

Let’s take the already familiar example of an Order with OrderLines, Order being the aggregate root. We’re using optimistic concurrency control for each of our aggregate parts. In our domain there is an invariant that dictates that the total number of items  in an order can’t be more than 10. Two commands that would increase the quantity could be processed in parallel and break the invariant. This is because in each transaction, the invariant would hold and each transaction would commit, since they update different lines. Now you’re in an inconsistent state.

If we could lock the entire aggregate (order and order lines), then the second transaction would fail and rollback, maintaining a consistent state. One way to achieve this when using optimistic concurrency is to update the root’s version whenever an aggregate part is updated.

Continue Reading

NServiceBus

Framework designers usually make sure there is a way to decouple infrastructure code from business code. Usually this comes in the form of hooks that you can use you to inject custom code. The benefit is twofold: you don’t have duplicated code and you don’t break the Single Responsibility Principle. If you’re using ASP.NET Core/MVC/WebApi, then you have ActionFilters. NServiceBus has its own mechanisms that you can use. In this blog post we’ll discuss two ways of hooking into the message processing pipeline that are useful for managing units of work in NServiceBus : IManageUnitsOfWork and the NServiceBus pipeline. There are many use cases where these prove useful. Here are some examples:

  • Manage the unit of work (i.e. the session in NHibernate and RavenDB)
  • Logging & Monitoring
  • Error Handling
  • Authorization

You definitely don’t want to add these concerns to your handlers. Here are two approaches that can help you to keep the handlers clean of infrastructural concerns.

Continue Reading

Books

Enterprise Integration Patterns by Gregor Hohpe and Bobby Woolf is a seminal book on Messaging. Every developer working on messaging solutions should read it. Even though it’s more than 10 years old, the content is still relevant. This is because the book focuses on patterns and principles and it’s technology agnostic. The authors do a good job on describing how you can combine the patterns together and present the trade-offs of different approaches.

Enterprise Integration Patterns

Continue Reading