sendmail queues to /var/spool/clientmqueue and never sends

Question

This is a continuation of my previous question here. The troubleshooting has gotten me a little further, and I think I'm facing a new set of problems now. Originally, I found out that sendmail wasn't listening on port 25 anymore, but I fixed this by increasing the values of the QueueLA and RefuseLA. I can telnet to port 25, and send email manually using SMTP commands. However, when I try sendmail:

[netcool1 (root)/]$ sendmail -v [email protected] < ~/testmessage
[email protected]... queued
[netcool1 (root)/]$ mailq
/var/spool/mqueue is empty
                Total requests: 0
[netcool1 (root)/]$ cd /var/spool/mqueue/
[netcool1 (root)/var/spool/mqueue]$ ls -la
total 6
drwxr-x---   2 root     bin         2048 Jan 11 11:54 .
drwxr-xr-x  13 root     bin          512 Jul 10  2009 ..
[netcool1 (root)/var/spool/mqueue]$ cd ../clientmqueue/
[netcool1 (root)/var/spool/clientmqueue]$ ls -la
total 76
drwxrwx---   2 smmsp    smmsp      22016 Jan 11 12:19 .
drwxr-xr-x  13 root     bin          512 Jul 10  2009 ..
-rw-rw----   1 smmsp    smmsp        100 Jan 11 07:03 dfq0BC3b8H007834
-rw-rw----   1 smmsp    smmsp         99 Jan 11 07:33 dfq0BCXba1008078
-rw-rw----   1 smmsp    smmsp         15 Jan 11 12:02 dfq0BH29Zu010211
-rw-rw----   1 smmsp    smmsp         15 Jan 11 12:13 dfq0BHDeEA010348
-rw-rw----   1 smmsp    smmsp        117 Jan 11 12:16 dfq0BHGuda010384
-rw-rw----   1 smmsp    smmsp        117 Jan 11 12:18 dfq0BHI9Q4010389
-rw-rw----   1 smmsp    smmsp         15 Jan 11 12:19 dfq0BHJjEv010410
-rw-rw----   1 smmsp    smmsp        871 Jan 11 07:03 qfq0BC3b8H007834
-rw-rw----   1 smmsp    smmsp        870 Jan 11 07:33 qfq0BCXba1008078
-rw-rw----   1 smmsp    smmsp        542 Jan 11 12:02 qfq0BH29Zu010211
-rw-rw----   1 smmsp    smmsp        562 Jan 11 12:13 qfq0BHDeEA010348
-rw-rw----   1 smmsp    smmsp        583 Jan 11 12:16 qfq0BHGuda010384
-rw-rw----   1 smmsp    smmsp        583 Jan 11 12:18 qfq0BHI9Q4010389
-rw-rw----   1 smmsp    smmsp        562 Jan 11 12:19 qfq0BHJjEv010410
-rw-------   1 smmsp    smmsp         34 Jan 11 12:12 sm-client.pid
[netcool1 (root)/var/spool/clientmqueue]$ sendmail -q -v
[netcool1 (root)/var/spool/clientmqueue]$ ls -la
total 76
drwxrwx---   2 smmsp    smmsp      22016 Jan 11 12:19 .
drwxr-xr-x  13 root     bin          512 Jul 10  2009 ..
-rw-rw----   1 smmsp    smmsp        100 Jan 11 07:03 dfq0BC3b8H007834
-rw-rw----   1 smmsp    smmsp         99 Jan 11 07:33 dfq0BCXba1008078
-rw-rw----   1 smmsp    smmsp         15 Jan 11 12:02 dfq0BH29Zu010211
-rw-rw----   1 smmsp    smmsp         15 Jan 11 12:13 dfq0BHDeEA010348
-rw-rw----   1 smmsp    smmsp        117 Jan 11 12:16 dfq0BHGuda010384
-rw-rw----   1 smmsp    smmsp        117 Jan 11 12:18 dfq0BHI9Q4010389
-rw-rw----   1 smmsp    smmsp         15 Jan 11 12:19 dfq0BHJjEv010410
-rw-rw----   1 smmsp    smmsp        871 Jan 11 07:03 qfq0BC3b8H007834
-rw-rw----   1 smmsp    smmsp        870 Jan 11 07:33 qfq0BCXba1008078
-rw-rw----   1 smmsp    smmsp        542 Jan 11 12:02 qfq0BH29Zu010211
-rw-rw----   1 smmsp    smmsp        562 Jan 11 12:13 qfq0BHDeEA010348
-rw-rw----   1 smmsp    smmsp        583 Jan 11 12:16 qfq0BHGuda010384
-rw-rw----   1 smmsp    smmsp        583 Jan 11 12:18 qfq0BHI9Q4010389
-rw-rw----   1 smmsp    smmsp        562 Jan 11 12:19 qfq0BHJjEv010410
-rw-------   1 smmsp    smmsp         34 Jan 11 12:12 sm-client.pid
[netcool1 (root)/var/spool/clientmqueue]$ cat sm-client.pid 
10344
/usr/lib/sendmail -Ac -q15m
[netcool1 (root)/var/spool/clientmqueue]$ 

I've been working on this for three days now, any help would be appreciated.

Edit: Contents of submit.mc (the only version I could find under /etc/mail was under /etc/mail/cf/cf):

$ more submit.mc 
divert(-1)
#
# Copyright (c) 2001, 2002 Sendmail, Inc. and its suppliers.
#       All rights reserved.
#
# By using this file, you agree to the terms and conditions set
# forth in the LICENSE file which can be found at the top level of
# the sendmail distribution.
#
#ident  "@(#)submit.mc  1.3     02/12/29 SMI"
#

#
#  This is the prototype file for a set-group-ID sm-msp sendmail that
#  acts as a initial mail submission program.
#

divert(0)dnl
VERSIONID(`$Id: submit.mc,v 8.6.2.4 2002/12/29 03:54:34 ca Exp $')
define(`confCF_VERSION', `Submit')dnl
define(`__OSTYPE__',`')dnl dirty hack to keep proto.m4 from complaining
define(`_USE_DECNET_SYNTAX_', `1')dnl support DECnet
define(`confTIME_ZONE', `USE_TZ')dnl
define(`confDONT_INIT_GROUPS', `True')dnl
dnl
dnl If you use IPv6 only, change [127.0.0.1] to [IPv6:::1]
FEATURE(`msp', `[127.0.0.1]')dnl

I'm wondering if this still has something to do with the RefuseLA and QueueLA values. I've set them impossibly high (QueueLA = 50, RefuseLA = 60) because this server really only sends emails from scripts and has a high LA due to many multi-threaded listeners. The increased RefuseLA got sendmail to listen on port 25 again (see previous question), but QueueLA doesn't seem to have any effect on the queuing. However, suddenly last night around 1900, the server sent through all the test emails that I had sent during the day.

Solution

Since the default value of QueueLA is 8 * numproc and for RefuseLA 12 * numproc a value of 50 may not be that ridiculous. Try however swapping the values of QueueLA and RefuseLA so that QueueLA > RefuseLA. This trick comes from Vixie and Avolio (Sendmail Theory and Practice):

"Setting them as shown here gives Sendmail a range of load average in which it is capable of delivering messages from its queue but incapable of receiving new messages. This is intentional. If you set Ox to be less than OX, Sendemail has instead a range of load average in which it can receive new mail (thus adding to the queue) but cannot deliver any queued mail."

Let us know what happens

Comments

[Tim]

Do logs show anything of interest?

[coding_hero]

No, there's not anything in /var/log/syslog, and /var/adm/messages only had sendmail errors in it up until I freed up disk space in the last question. Nothing from sendmail today.

[adamo]

Keep in mind that you have two queues. One (the one you expect) for sendmail.cf which is for the process that listens on port 25 and one for submit.cf which you view by executing mailq -C /etc/mail/submit.cf . So can you edit the post and show us what the contents of submit.mc (not .cf) are?

[coding_hero]

Changed QueueLA to 60 and RefuseLA to 50, tried it and same result. Queued message. Then upped QueueLA to 70 and RefuseLA to 60, same result. Is it odd that I can't find logs for sendmail anywhere? There is no syslog, and the last entries from sendmail in /var/adm/messages were from a few days ago when I had a full filesystem. For that matter, the '-Ac -q' flags aren't flushing out /var/spool/clientmqueue when run manually.

[adamo]

Run sendmail -Ac -v -q and see what it tells you. The same without -Ac for the normal sendmail process. Also do you have a feeling of the average load on the system?

[coding_hero]

$ sendmail -Ac -v -q Skipping queue run -- load average too high The load averages tend to run around 35, since there's quite a few multithreaded processes on this server.

[coding_hero]

Wait, I think I just fixed it. I was changing the QueueLA and RefuseLA values in sendmail.cf, but not submit.cf (yes, I know I'm not supposed to edit them directly). Once I changed them in submit.cf, it started working again. This was a good troublshooting command I found: sendmail -v -q -d3.30. Thanks for your help, @adamo!