CommuniMail - Cmaild Appears To Hang When Processing Over 500 Messages

Message
Author
RickSanson
Posts: 36
Joined: Mar 14, 2007 6:27 pm

Cmaild Appears To Hang When Processing Over 500 Messages

#1 Postby RickSanson » Jan 11, 2008 12:06 am

Hi!

I am trying to work through a problem where C-Mail falls appears to fall down over 500 messages.

Using top this is what I see...

PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME CPU CMD
19644 root 23 0 1404M 1.0G 824 R 33.4 89.1 15:07 1 perl

This is a dual processor box with 2gb ram. 500 messages are processed in about 10 minutes and the "perl" ram consumption varies but over 500 messages perl's ram consumption climbs and then remains fixed at the above value (1404M).

Please help! I need to move as many as 5000!

admin
Site Admin
Posts: 1839
Joined: Mar 22, 2006 12:32 pm

Re: Cmaild Appears To Hang When Processing Over 500 Messages

#2 Postby admin » Jan 11, 2008 12:14 pm

RickSanson wrote: PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME CPU CMD
19644 root 23 0 1404M 1.0G 824 R 33.4 89.1 15:07 1 perl
Is that possible to provide cmaild.log? Does it finish sending 500 emails?

RickSanson
Posts: 36
Joined: Mar 14, 2007 6:27 pm

#3 Postby RickSanson » Jan 11, 2008 1:00 pm

I'm doing a few more runs in testing but here is what I've found:

cmaild (listed as the command "perl") apparently puts all message elements into memory. The more messages to send the more memory it requires until it exceeds system memory which it then begins to consume swap space. Once all swap space is consumed the perl process is simply dropped without notice.

I notice than at 500 messages perl requires about 1.25MB ram per 100 messages until above around 800. 1500 messages consumed "4GB" ram (physical is 2GB). So above 800 messages the process consumes about 2.5MB per 100 messages.

This logs at this time only report back a successful run but when the perl process is dropped no log is written...

Thank you for your help!

RickSanson
Posts: 36
Joined: Mar 14, 2007 6:27 pm

#4 Postby RickSanson » Jan 11, 2008 7:46 pm

Hi!

I built another machine that has two non-parity strip set for both the root and the swap partitions. I did thsi to see if I could get anything out of the swap...

I did 500 messages (html only) using 369MB ram with 0 swap
I did 1000 messages (html only) using 1410MB ram with 256KB swap
I then tried 2000 messages which wound up to 3067MB ram and 1.7GB swap --- Perl died right there, I got an Out of memory message (from top) then saw the perl process come back began to consume more memory and swap but with little processor utilization...


Here is the tail of the cmail_daemon.log

send to:[email protected]
send to:[email protected]
send to:[email protected]
.. Successfully send: 1008 emails, 0 with errors. Spend 1546 seconds.

2008-01-11 14:30:25 STOP deamon


2008-01-11 14:31:40 Deamon START and found 1 active Send Requests


Please help! I'm in serious trouble with this...

RickSanson
Posts: 36
Joined: Mar 14, 2007 6:27 pm

#5 Postby RickSanson » Jan 13, 2008 3:27 pm

Is anyone sending over 1000 messages?

Anyone? Any ideas?

RickSanson
Posts: 36
Joined: Mar 14, 2007 6:27 pm

#6 Postby RickSanson » Dec 14, 2008 2:40 am

Just as a follow up...

Using flat files (csv) is NOT efficient. The problem here is that there is no garbage collection and the scripts (perl) will consume memory until the machine pukes, kills perl without warning...

However with MySQL I've pushed 20,000 messages per hour consuming not more than 1% memory...