Home > exchange 2007, network management, server 2003, troubleshooting, Uncategorized, windows, windows server > Exchange 2007: Messages stuck in Submission Queue

Exchange 2007: Messages stuck in Submission Queue

We recently received reports of message delivery delays in our Exchange organization. We run Exchange 2007, so I checked out the Hub Transport Servers and discovered that messages were piling up in the Submission queues on both of the main hub transports. Restarting the Microsoft Exchange Transport service didn’t get things going again, so I turned to the Application Log to try to figure out what was going on.

I noticed some errors coming from the source MSExchange Extensibility, with an event ID of 1050. They read as follows:

The execution time of agent ‘ScanMail Routing Agent’ exceeded 300000 (milliseconds) while handling event ‘OnSubmittedMessage’. This is an unusual amount of time for an agent to process a single event. However, Transport will continue processing this message.

From the event, you can tell we’re using TrendMicro ScanMail as part of our edge protection. It looked like the ScanMail scanning agent was stuck on a message and couldn’t proceed. So, off to the services console I go. ScanMail has three services installed on the server, including its Master Service. The other services seemed happy, but the Master service timed out when I attempted to stop it. I checked the service properties and found its related executable file and killed it through Task Manager. The service was then recorded in the Services Console as being stopped, and after starting the service, mail started flowing again on that Hub Transport server. I repeated those steps on the second Hub Transport server, and messages came flooding out from there as well.

I checked that ScanMail was up to date on both servers, and they appeared to be. I checked through Trend Micro’s site nonetheless, and found that there appeared to be a related update. The links:

List of updates for ScanMail 8: http://www.trendmicro.com/download/product.asp?productid=8
Direct link to the relevant update: ScanMail 8.0 for Microsoft Exchange Patch 3
The README file: http://www.trendmicro.com/ftp/documentation/readme/smex_80_win_en_patch3_Readme.TXT

The relevant portion of the changelog:

21. While ScanMail for Microsoft Exchange 8.0 is running, after a few hours, email messages will start to queue. If user re-starts the ScanMail for Microsoft Exchange services, the application will hang at “Stopping”.

I was technically on vacation while this all went down, so I had to hand of the update to someone else to apply, but no one was available for that weekend. The patch was not manually applied on the servers that weekend, but the issue did not return. We’ve now been running smoothly for about two weeks.

Hopefully you don’t run across the issue, and hopefully this post will be of some help if you do!

JaS

About these ads
  1. September 5, 2008 at 7:07 pm

    As a matter of fact I did come across this very same issue, only with a different antivirus software: GFI MailSecurity. I followed your indications and it worked wonderfully. It’s 10 PM here now and I am too tired to look for an update on the AV but I will do it first thing tomorrow.

    Thanks a lot for your post.

  2. justanothersysadmin
    September 5, 2008 at 9:37 pm

    Hi Rafael,

    I’m glad that I could be of help! This one hit us smack in the middle of the day in a hosted Exchange environment; talk about having to think on your feet! I remember having to fight our way through that, so I must congratulate you on working your way through what I’m sure felt like a pretty long day for you!

    If this is something that is occurring across multiple message hygiene providers, maybe there’s some sort of a ‘gotcha’ in the API? Anyway, that’s irrelevant at this point, I guess.

    Good luck with the patching!

    JaS

  3. LeeM
    September 11, 2008 at 8:25 pm

    Thanks for this post, same thing happened to us, mail not affected though. Scanmail master service wont start, Ill battle this over the weekend.
    LeeM

  4. Daniel
    January 30, 2010 at 9:58 pm

    Hi guys, I have the same problem.
    I’m running Forefront Protection 2010 for Exchange on the exchange 2007 edge role.
    Could anybody fix this problem?

    Daniel

  5. John
    September 25, 2012 at 3:01 am

    This same thing happened to me on Friday but in my exchange 2010 environment. Except I use McAfee so the routing agent was named slightly different. But basically all my submission queues went up to 2000 which triggered the resource threshold and started shutting down exchange components. Very annoying.

  1. November 18, 2013 at 2:46 pm

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

%d bloggers like this: