Discussion:
Question about Qmail relating to MX retry processing
(too old to reply)
Marc Perkel
2006-11-24 21:52:07 UTC
Permalink
I'm not a Qmail user. I have a spam filtering operating where I do front
end filtering for about 3000 domains. Many of the servers that send mail
to my filtering network are running Qmail and there seems to be a
problem and I'm wondering if someone can address it. I'm running Exim
myself.

One of my tricks to filter spam is a gray listing like trick that
detects suspicious hosts and returns a temp error on the lowest MX
number. Spammers often don't retry but real email servers would, in
theory, retry the next level up in the MX chain and the secondary server
will accept the email.

Servers that I do this with include servers with no or bad reverse
lookup, Host names with pattens that look like residential machines, and
servers listed in black lists that are not reliable enough to block, but
usually are spammers.

The idea being the but profiling these servers and returning a temp
error (421) on the lower MX that the good servers who would be a false
positive would retry to a different server that would accept it.

But - it seems like servers who are running Qmail only send to the
lowest MX and don't retry the higher MX. Is this so? Or does it apply
only to old versions?

When Exim gets a temp error on the lowest MX it immediately retury all
the IP addresses of the higher MX servers. If they all fail then the
server wait for a period of time and tries them all in order again. But
people are telling me that Qmail is broken on this issue.

So - is this so? Can someone let me know how Qmail works on MX retries?

Thanks in Advance
Uncle George
2006-11-25 00:10:13 UTC
Permalink
your analysis sorta reminds me of a child trying to get permission from
an adult. It mom says no, then there is always dad. When mom said no
way in hell, amongst (savy) parents that means absolutely no. In a
perfect world, there's no need to ask pop.
But if mom didnt reply ( ie out shopping ) , then the answer from pop
would be binding.

I suppose it all revolves around if u mean that the IP connection
failed, or the SMTP protocol failed would cause another MX record to be
used. My *opinion* is that if the IP connection failed, then another MX
can be tried until a connection is made. If the SMTP protocol failed,
then that answer is suppose to be good for all MX records. No need to
seek out another parent: sort-of-speek.

It would seem that if one server greylisted, then they all should
greylist. Just my *opinion* mind you. As eventually you will find that
the errant spammers will find the hole in your filtering scheme, and not
bother with the lowest, or highest, and just try them all until they get
what they want.

I'm curious as to why they ("people are telling me ") think that their
scheme is appropriate? All I see is that spammers will adapt to try all
MX records, wasting your bandwidth, and server time. And in your case,
having spam processed ( as I suppose, greylist is just the first barrier ).
Post by Marc Perkel
I'm not a Qmail user. I have a spam filtering operating where I do
front end filtering for about 3000 domains. Many of the servers that
send mail to my filtering network are running Qmail and there seems to
be a problem and I'm wondering if someone can address it. I'm running
Exim myself.
One of my tricks to filter spam is a gray listing like trick that
detects suspicious hosts and returns a temp error on the lowest MX
number. Spammers often don't retry but real email servers would, in
theory, retry the next level up in the MX chain and the secondary
server will accept the email.
Servers that I do this with include servers with no or bad reverse
lookup, Host names with pattens that look like residential machines,
and servers listed in black lists that are not reliable enough to
block, but usually are spammers.
The idea being the but profiling these servers and returning a temp
error (421) on the lower MX that the good servers who would be a false
positive would retry to a different server that would accept it.
But - it seems like servers who are running Qmail only send to the
lowest MX and don't retry the higher MX. Is this so? Or does it apply
only to old versions?
When Exim gets a temp error on the lowest MX it immediately retury all
the IP addresses of the higher MX servers. If they all fail then the
server wait for a period of time and tries them all in order again.
But people are telling me that Qmail is broken on this issue.
So - is this so? Can someone let me know how Qmail works on MX retries?
Thanks in Advance
Marc Perkel
2006-11-25 00:28:51 UTC
Permalink
The think about spammers is that they try to deliver to as many people
as they can so coming back to try to get spam to my domains is a lot of
work then they can find other servers that are easier.

I also use it for load balancing. I have my biggest server cluster on my
lowest MX. But in some rare cases if the load levels start creeping up
what I do is start returning 4xx errors for certain countries or
blacklisted on lists that I can't count on being 100% reliable, or if
the load levels get really high it tells the sender to spill over to the
backup servers that are less loaded. I have 3 servers on the second tier
to process mail that is deferred by the primary server or if the primary
server dies or is otherwise offline. So just because one server might
return a 4xx error doesn't mean any other server will. All that means is
this server isn't ready.

And - there is the plain language of the MX specs that clearly say that
the reason there are higher MX records is for this reason. I guess I'm
confused as to who Qmail doesn't follow the spec.

So here's the problem. A server running qmail from what appears to be a
dynamic IP or a server running Qmail that has a bad reverse lookup tries
to email one of the domains that I process and gets a 4xx error on the
main server. All other MTAs try the secondary MX and it acceots the
message. But Qmail keeps trying the lowest MX and it eventually gives up.

Or - the main server overloads for some reason. (sometimes 4 gigs still
isn't enough ram!) so it throws up a 4xx to everything. All other MTAs
try the backup MX and the message goes through. But Qmail doesn't and
thus the message is delayed till qmail retries.

The bottom line is that there is a spec and it seems that Qmail doesn't
follow the spec. So what's up with that?
Post by Uncle George
your analysis sorta reminds me of a child trying to get permission from
an adult. It mom says no, then there is always dad. When mom said no
way in hell, amongst (savy) parents that means absolutely no. In a
perfect world, there's no need to ask pop.
But if mom didnt reply ( ie out shopping ) , then the answer from pop
would be binding.
I suppose it all revolves around if u mean that the IP connection
failed, or the SMTP protocol failed would cause another MX record to be
used. My *opinion* is that if the IP connection failed, then another MX
can be tried until a connection is made. If the SMTP protocol failed,
then that answer is suppose to be good for all MX records. No need to
seek out another parent: sort-of-speek.
It would seem that if one server greylisted, then they all should
greylist. Just my *opinion* mind you. As eventually you will find that
the errant spammers will find the hole in your filtering scheme, and not
bother with the lowest, or highest, and just try them all until they get
what they want.
I'm curious as to why they ("people are telling me ") think that their
scheme is appropriate? All I see is that spammers will adapt to try all
MX records, wasting your bandwidth, and server time. And in your case,
having spam processed ( as I suppose, greylist is just the first barrier ).
Post by Marc Perkel
I'm not a Qmail user. I have a spam filtering operating where I do
front end filtering for about 3000 domains. Many of the servers that
send mail to my filtering network are running Qmail and there seems
to be a problem and I'm wondering if someone can address it. I'm
running Exim myself.
One of my tricks to filter spam is a gray listing like trick that
detects suspicious hosts and returns a temp error on the lowest MX
number. Spammers often don't retry but real email servers would, in
theory, retry the next level up in the MX chain and the secondary
server will accept the email.
Servers that I do this with include servers with no or bad reverse
lookup, Host names with pattens that look like residential machines,
and servers listed in black lists that are not reliable enough to
block, but usually are spammers.
The idea being the but profiling these servers and returning a temp
error (421) on the lower MX that the good servers who would be a
false positive would retry to a different server that would accept it.
But - it seems like servers who are running Qmail only send to the
lowest MX and don't retry the higher MX. Is this so? Or does it apply
only to old versions?
When Exim gets a temp error on the lowest MX it immediately retury
all the IP addresses of the higher MX servers. If they all fail then
the server wait for a period of time and tries them all in order
again. But people are telling me that Qmail is broken on this issue.
So - is this so? Can someone let me know how Qmail works on MX retries?
Thanks in Advance
Uncle George
2006-11-25 00:42:24 UTC
Permalink
Well, not exactly from an RFC, but from
http://en.wikipedia.org/wiki/MX_record
The sending agent then attempts to establish an SMTP
<http://en.wikipedia.org/wiki/Simple_Mail_Transfer_Protocol>
connection to one of these servers, starting with the one with the
smallest preference number, delivering the message to the first server
with which a connection can be made.
I guess the operative word would be *connection*. The SMTP dialog
appears to have established a connetion.

But if you read on to the next paragraph of the WIKI, one senses that
they dont mean connection, but rather the acceptance of the e-mail.
Which appears to be your position.
The think about spammers is that they try to deliver to as many people
as they can so coming back to try to get spam to my domains is a lot
of work then they can find other servers that are easier.
I also use it for load balancing. I have my biggest server cluster on
my lowest MX. But in some rare cases if the load levels start creeping
up what I do is start returning 4xx errors for certain countries or
blacklisted on lists that I can't count on being 100% reliable, or if
the load levels get really high it tells the sender to spill over to
the backup servers that are less loaded. I have 3 servers on the
second tier to process mail that is deferred by the primary server or
if the primary server dies or is otherwise offline. So just because
one server might return a 4xx error doesn't mean any other server
will. All that means is this server isn't ready.
And - there is the plain language of the MX specs that clearly say
that the reason there are higher MX records is for this reason. I
guess I'm confused as to who Qmail doesn't follow the spec.
So here's the problem. A server running qmail from what appears to be
a dynamic IP or a server running Qmail that has a bad reverse lookup
tries to email one of the domains that I process and gets a 4xx error
on the main server. All other MTAs try the secondary MX and it acceots
the message. But Qmail keeps trying the lowest MX and it eventually
gives up.
Or - the main server overloads for some reason. (sometimes 4 gigs
still isn't enough ram!) so it throws up a 4xx to everything. All
other MTAs try the backup MX and the message goes through. But Qmail
doesn't and thus the message is delayed till qmail retries.
The bottom line is that there is a spec and it seems that Qmail
doesn't follow the spec. So what's up with that?
Post by Uncle George
your analysis sorta reminds me of a child trying to get permission from
an adult. It mom says no, then there is always dad. When mom said no
way in hell, amongst (savy) parents that means absolutely no. In a
perfect world, there's no need to ask pop.
But if mom didnt reply ( ie out shopping ) , then the answer from pop
would be binding.
I suppose it all revolves around if u mean that the IP connection
failed, or the SMTP protocol failed would cause another MX record to be
used. My *opinion* is that if the IP connection failed, then another MX
can be tried until a connection is made. If the SMTP protocol failed,
then that answer is suppose to be good for all MX records. No need to
seek out another parent: sort-of-speek.
It would seem that if one server greylisted, then they all should
greylist. Just my *opinion* mind you. As eventually you will find that
the errant spammers will find the hole in your filtering scheme, and not
bother with the lowest, or highest, and just try them all until they get
what they want.
I'm curious as to why they ("people are telling me ") think that their
scheme is appropriate? All I see is that spammers will adapt to try all
MX records, wasting your bandwidth, and server time. And in your case,
having spam processed ( as I suppose, greylist is just the first barrier ).
Post by Marc Perkel
I'm not a Qmail user. I have a spam filtering operating where I do
front end filtering for about 3000 domains. Many of the servers that
send mail to my filtering network are running Qmail and there seems
to be a problem and I'm wondering if someone can address it. I'm
running Exim myself.
One of my tricks to filter spam is a gray listing like trick that
detects suspicious hosts and returns a temp error on the lowest MX
number. Spammers often don't retry but real email servers would, in
theory, retry the next level up in the MX chain and the secondary
server will accept the email.
Servers that I do this with include servers with no or bad reverse
lookup, Host names with pattens that look like residential machines,
and servers listed in black lists that are not reliable enough to
block, but usually are spammers.
The idea being the but profiling these servers and returning a temp
error (421) on the lower MX that the good servers who would be a
false positive would retry to a different server that would accept it.
But - it seems like servers who are running Qmail only send to the
lowest MX and don't retry the higher MX. Is this so? Or does it
apply only to old versions?
When Exim gets a temp error on the lowest MX it immediately retury
all the IP addresses of the higher MX servers. If they all fail then
the server wait for a period of time and tries them all in order
again. But people are telling me that Qmail is broken on this issue.
So - is this so? Can someone let me know how Qmail works on MX retries?
Thanks in Advance
Marc Perkel
2006-11-25 01:06:46 UTC
Permalink
Post by Uncle George
Well, not exactly from an RFC, but from
http://en.wikipedia.org/wiki/MX_record
The sending agent then attempts to establish an SMTP
<http://en.wikipedia.org/wiki/Simple_Mail_Transfer_Protocol>
connection to one of these servers, starting with the one with the
smallest preference number, delivering the message to the first
server with which a connection can be made.
I guess the operative word would be *connection*. The SMTP dialog
appears to have established a connetion.
But if you read on to the next paragraph of the WIKI, one senses that
they dont mean connection, but rather the acceptance of the e-mail.
Which appears to be your position.
Suppose your mail server relies on an NFS mount and suppose the NFS
server is down. The server might connect but in the process of receiving
the email the server fails because it can't write to something it needs
to write to. So it returns a 4xx error. The is a backup server on a
higher MX that is there to accept email in case something like this
happens. But Qmail won't send the email to the backup MX?

There's a spec - everyone else follows the spec - except qmail.
Marc Perkel
2006-11-25 01:19:22 UTC
Permalink
Post by Marc Perkel
Post by Uncle George
Well, not exactly from an RFC, but from
http://en.wikipedia.org/wiki/MX_record
The sending agent then attempts to establish an SMTP
<http://en.wikipedia.org/wiki/Simple_Mail_Transfer_Protocol>
connection to one of these servers, starting with the one with the
smallest preference number, delivering the message to the first
server with which a connection can be made.
I guess the operative word would be *connection*. The SMTP dialog
appears to have established a connetion.
But if you read on to the next paragraph of the WIKI, one senses that
they dont mean connection, but rather the acceptance of the e-mail.
Which appears to be your position.
Suppose your mail server relies on an NFS mount and suppose the NFS
server is down. The server might connect but in the process of receiving
the email the server fails because it can't write to something it needs
to write to. So it returns a 4xx error. The is a backup server on a
higher MX that is there to accept email in case something like this
happens. But Qmail won't send the email to the backup MX?
There's a spec - everyone else follows the spec - except qmail.
You're beating a dead horse. The best you're going to get here is an
argument about what the RFC requires. You're not going to get every
qmail site to patch their system. So the bottom line is, you've got to
say goodbye to this particular technique you're using.
- Ron
I'm not sure that's the case. It just means the those who use qmail AND
(have bad reverse lookup OR are on a dynamic IP address) will not be
able to send email to my domains UNLESS I whitelist their host.
Uncle George
2006-11-25 01:58:55 UTC
Permalink
Post by Marc Perkel
I'm not sure that's the case. It just means the those who use qmail
AND (have bad reverse lookup OR are on a dynamic IP address) will not
be able to send email to my domains UNLESS I whitelist their host.
I guess i'm at a loss here. Exactly how would trying each MX record in
turn prevent your filtering techniques from processing reverse lookups,
or dynamic ip's.

but the guy is correct. qmail is owned by one person, and is not likely
to change. The followers of qmail, I think, would have adopted such a
MX strategy, if it was *exactly* spec'd that way into one of their
mandatory patches.
Marc Perkel
2006-11-28 05:15:48 UTC
Permalink
Post by Uncle George
Post by Marc Perkel
I'm not sure that's the case. It just means the those who use qmail
AND (have bad reverse lookup OR are on a dynamic IP address) will not
be able to send email to my domains UNLESS I whitelist their host.
I guess i'm at a loss here. Exactly how would trying each MX record in
turn prevent your filtering techniques from processing reverse
lookups, or dynamic ip's.
but the guy is correct. qmail is owned by one person, and is not
likely to change. The followers of qmail, I think, would have
adopted such a MX strategy, if it was *exactly* spec'd that way into
one of their mandatory patches.
I process about 4 million messages a day. As you know graylisting is a
popular spam fighting technique and gets rid of spam using 4xx errors
and counts on the fact that spam zombies don't retry. But greylisting
has its drawbacks like delaying good email. So I don't use it.

What I do instead is sort of selective graylisting targeting dynamic IPs
and bad RDNS. But instead of delaying them for a long time with
graylisting I only use the tem error on the lowest IP and accept it on
the next highest IP. And this works for all MTAs except qmail that will
keep trying the lowest MX till it times out.

But - it only happens to qmail users with dynamic IPs or bad RDNS. And I
can add IPs to a braiddead hosts list to bypass it if someone complains.
Uncle George
2006-11-25 01:35:17 UTC
Permalink
Post by Marc Perkel
But Qmail won't send the email to the backup MX?
Not sure if i would use NFS as a qmail queue storage. But that another
IMHO.
Qmail will do the second MX once u close up the service listening on
port 25 of the broken machine. For many default qmails, u have about 6
days to disable the port to the broken MTA before the e-mail is deemed
undeliverable.
Post by Marc Perkel
There's a spec - everyone else follows the spec - except qmail.
I'm sure that there is a spec. I'm not sure why the followers of the RFC
spec have not yet chimed in. Do u know where the spec spells this out?
Charles Cazabon
2006-11-25 04:10:44 UTC
Permalink
Post by Marc Perkel
But Qmail won't send the email to the backup MX?
Not if it is able to make a tcp connection to a higher-priority/lower-distance
one.
Post by Marc Perkel
There's a spec - everyone else follows the spec - except qmail.
Your assertion is incorrect. Re-read RFC 974:

If the list of MX RRs is not empty, the mailer should try to deliver the
message to the MXs in order (lowest preference value tried first). The
mailer is required to attempt delivery to the lowest valued MX.
Implementors are encouraged to write mailers so that they try the MXs in
order until one of the MXs accepts the message, or all the MXs have been
tried.

Only trying the lowest-distance MX is permitted. If you think otherwise,
you're discarding mail, and you've got no noe to blame but yourself.

If you want to make your hare-brained scheme work, don't accept the connection
in the first place, rather than accepting it and not accepting the message.

Charles
--
--------------------------------------------------------------------------
Charles Cazabon <***@discworld.dyndns.org>
Read http://pyropus.ca/personal/writings/12-steps-to-qmail-list-bliss.html
My services include qmail consulting. See http://pyropus.ca/ for details.
--------------------------------------------------------------------------
Marc Perkel
2006-11-25 05:57:43 UTC
Permalink
Post by Charles Cazabon
Post by Marc Perkel
But Qmail won't send the email to the backup MX?
Not if it is able to make a tcp connection to a higher-priority/lower-distance
one.
Post by Marc Perkel
There's a spec - everyone else follows the spec - except qmail.
If the list of MX RRs is not empty, the mailer should try to deliver the
message to the MXs in order (lowest preference value tried first). The
mailer is required to attempt delivery to the lowest valued MX.
Implementors are encouraged to write mailers so that they try the MXs in
order until one of the MXs accepts the message, or all the MXs have been
tried.
Only trying the lowest-distance MX is permitted. If you think otherwise,
you're discarding mail, and you've got no noe to blame but yourself.
If you want to make your hare-brained scheme work, don't accept the connection
in the first place, rather than accepting it and not accepting the message.
Charles
What that is saying is if your app is so braindead that it can only tru
one IP then ise the lowest but if you want to do it right you try them
all. Every other MTA other than qmail does it right.

So - why not have a configuration option allowing you to CHOOSE what way
you want to do it. That way people who want their qmail to act like all
the other MTAs can SELECT that is they want it?
Pawel Panek
2006-11-25 07:32:19 UTC
Permalink
Post by Charles Cazabon
Post by Marc Perkel
But Qmail won't send the email to the backup MX?
Not if it is able to make a tcp connection to a
higher-priority/lower-distance one.
Post by Marc Perkel
There's a spec - everyone else follows the spec - except qmail.
If the list of MX RRs is not empty, the mailer should try to
deliver the message to the MXs in order (lowest preference value
tried first). The mailer is required to attempt delivery to the
lowest valued MX. Implementors are encouraged to write mailers so
that they try the MXs in order until one of the MXs accepts the
message, or all the MXs have been tried.
yep, it is written right here: "accepts the messeage". I think it is far
different from statment: "accepts the connection".
Post by Charles Cazabon
Only trying the lowest-distance MX is permitted. If you think
otherwise, you're discarding mail, and you've got no noe to blame but
yourself.
If you want to make your hare-brained scheme work, don't accept the
connection in the first place, rather than accepting it and not
accepting the message.
is this statement also included in RFCs? Don't you think you have gone too
far? Acceptance of connection == acceptance of message? You are suggesting
that Marc's solution should work this way: reject connection if it can't
accept the message. So why not to use this suggestion to all conditions
which cause rejecting of message (bad rcpt, over quota, virus, spam scan at
smtp, etc)? I know why - it is impossilble to check these conditions without
the message. Chicken and egg problem. So acceptance of connection doesn't
mean that the message will be accepted. So if message is rejected (even
temporaily) why not to try next MX (as RFC 974 said)? Is Qmail following the
specs or it is not?
Post by Charles Cazabon
Charles
Greets,
Pawel Panek
Harald Hanche-Olsen
2006-11-25 09:56:13 UTC
Permalink
+ "Pawel Panek" <***@inet.pl>:

| So if message is rejected (even temporaily) why not to try next MX
| (as RFC 974 said)? Is Qmail following the specs or it is not?

Actually, RFC 974 is obsoleted by RFC 2821, so people should quote it
instead. (Though RFC 974 was the relevant one when qmail was
written.)

As to why qmail does what it does, there was an extremely brief
discussion on the qmail mailing list on 1996-06-25, and an equally
brief discussion on comp.protocols.tcp-ip in the days before.

http://groups.google.com/group/comp.mail.misc/browse_frm/thread/8987d5e1c3c4698f/9eaefb25953624c6?lnk=st&q=&rnum=1#9eaefb25953624c6

Nobody seemed to care much about the question back then. If I were to
guess, I'd say that djb chose what he did from the reasoning that
backup MXs are there to deal with a host being down, and of course a
host being down will not accept a connection. (Hosts were probably
down a lot more in those days.)

Be that as it may, changing qmail's behaviour on this issue would
probably be nontrivial, due to the way responsibility is divided up
between qmail-send and qmail-remote.

- Harald
Uncle George
2006-11-25 12:48:29 UTC
Permalink
Post by Harald Hanche-Olsen
| So if message is rejected (even temporaily) why not to try next MX
| (as RFC 974 said)? Is Qmail following the specs or it is not?
Actually, RFC 974 is obsoleted by RFC 2821, so people should quote it
instead. (Though RFC 974 was the relevant one when qmail was
written.)
as per section 5 of RFC 2821 circa 2001: Address Resolution and Mail
Handling

To provide reliable
mail transmission, the SMTP client MUST be able to try (and retry)
each of the relevant addresses in this list in order, until a
delivery attempt succeeds. However, there MAY also be a configurable
limit on the number of alternate addresses that can be tried. In any
case, the SMTP client SHOULD try at least two addresses.

Only a lawyer would like this. you *MUST* be able to try, but you
*SHOULD* try at least 2. It would have been real easy
to have said MUST both times.

It would appear that qmail is not compliant with todays RFC. Since
qmail was engineered before 2001, it was in compliance with the older
specs, except that there appears to have been some underlying notion
that all MX's should be tried.
So its sort of like saying, in the 60's, the optional car seat ( lap )
belts were available. But seat belts were mandatory in the 90's. Does
that make all the cars without seat belts undrivable? ( i dont know
exactly when seat belts were introduced, nor made mandatory )
Marc Perkel
2006-11-25 17:23:49 UTC
Permalink
Post by Uncle George
Post by Harald Hanche-Olsen
| So if message is rejected (even temporaily) why not to try next MX
| (as RFC 974 said)? Is Qmail following the specs or it is not?
Actually, RFC 974 is obsoleted by RFC 2821, so people should quote it
instead. (Though RFC 974 was the relevant one when qmail was
written.)
as per section 5 of RFC 2821 circa 2001: Address Resolution and Mail
Handling
To provide reliable
mail transmission, the SMTP client MUST be able to try (and retry)
each of the relevant addresses in this list in order, until a
delivery attempt succeeds. However, there MAY also be a configurable
limit on the number of alternate addresses that can be tried. In any
case, the SMTP client SHOULD try at least two addresses.
Only a lawyer would like this. you *MUST* be able to try, but you
*SHOULD* try at least 2. It would have been real easy
to have said MUST both times.
It would appear that qmail is not compliant with todays RFC. Since
qmail was engineered before 2001, it was in compliance with the older
specs, except that there appears to have been some underlying notion
that all MX's should be tried.
So its sort of like saying, in the 60's, the optional car seat ( lap )
belts were available. But seat belts were mandatory in the 90's. Does
that make all the cars without seat belts undrivable? ( i dont know
exactly when seat belts were introduced, nor made mandatory )
Qmail has a longer evolution that most all other MTAs. In the early days
there were almost no secondary MX servers. Mail went from pine to smtp
where it was delivered to the end user whi received it in Pine. The
world has changed and it would be nice if Qmail had an option to be
compatible with these changes.
Markus Stumpf
2006-11-26 02:36:39 UTC
Permalink
Post by Marc Perkel
In the early days
there were almost no secondary MX servers.
Bullshit. In the early days there were a lot of secondary and even third and
fourth level MX hosts. Links were flaky and overloaded, systems were offline
(over night) and it was often a big gain to get the message e.g. over the
Atlantic bottleneck (in the night), so they could be delivered (over cheaper
links) within the country as the systems came back in the morning.
Post by Marc Perkel
Mail went from pine to smtp
In the early days there was no "pine". Mail went from "mail" or "elm"
to sendmail, sendmail sent on to e.g. campus wide gateways and those
made the WAN connections. Also pine does not do MX delivery IMHO, it
uses configured MSAs.

I agree that MTAs should try other MX hosts with the same distance as
the MX with the shortest distance on temp failure, but if one or more of
them are available and signal an error (be it permanent or temporary) the
sending MTA has to honor that and not send to MX hosts with larger distance.
I've seen MTAs (not spammers) not even honoring permanent error codes and
trying to deliver to other MTAs or backups instead. These programmers and
admins should be shot on sight.

Also your "idea" of the modified greylisting is pretty useless from my
experience, as the spammers either immediately try all available MX hosts
on errors anyway or they even start with the larger distance MX hosts first
(reverse MX algorithm) as the backup MX hosts have weaker policies and
usually don't know about users. That way the spammers get their mail
delivered and the burden is on the receivers and the faked senders get
all the bounces from the backups that cannot send to the primaries.

\Maex
Matthew R. Dempsky
2006-11-25 16:57:51 UTC
Permalink
Post by Harald Hanche-Olsen
Be that as it may, changing qmail's behaviour on this issue would
probably be nontrivial, due to the way responsibility is divided up
between qmail-send and qmail-remote.
Matthias Andree wrote a patch that changes this behavior[1].

[1] http://www-dt.e-technik.uni-dortmund.de/~ma/qmail/patch-qmail-1.03-rfc2821.diff
Marc Perkel
2006-11-25 18:15:23 UTC
Permalink
Email routing in front emd spam filtering.

+-------------------------+
| |
^ V
Internet >-----> Backup MX Primary MX >-----> Customer's Email Server
V ^
| |
+--------------------------+


The idea here being that an overloaded Primary MX would return 4xx errors causing mail
to route through the Backup MX servers. Thus a 4xx for the Primary doesn't meant
that the backup will return 4xx.
Harald Hanche-Olsen
2006-11-25 19:56:43 UTC
Permalink
+ "Matthew R. Dempsky" <***@alkemio.org>:

| On Sat, Nov 25, 2006 at 10:56:13AM +0100, Harald Hanche-Olsen wrote:
| > Be that as it may, changing qmail's behaviour on this issue would
| > probably be nontrivial, due to the way responsibility is divided up
| > between qmail-send and qmail-remote.
|
| Matthias Andree wrote a patch that changes this behavior[1].
|
| [1] http://www-dt.e-technik.uni-dortmund.de/~ma/qmail/patch-qmail-1.03-rfc2821.diff

I see. But that only changes one thing: If the server greets the
client with a temporary error, then qmail-remote will go to the next
item on the list of MXs.

Unfortunately, RFC 2821 only allows two codes on the initial greeting:
220 and 554. I suspect the rationale behind this is that if the
server knows it has a temporary problem so it cannot receive mail,
then it should not accept connections in the first place.

Also, without further work, if the patched qmail-remote gets a
temporary error on greeting (or fails to connect) for each MX, it will
log the misinformative message "Sorry, I wasn't able to establish an
SMTP connection. (#4.4.1)".

The first place in the SMTP dialog where a temporary error is allowed
is in response to the MAIL command. I suppose one could further
develop the patch to go to the next MX if that happens, but again it
really should keep a record of what really happened and log something
more appropriate than the above #4.4.1 message.

- Harald
Marc Perkel
2006-11-25 17:19:30 UTC
Permalink
Post by Harald Hanche-Olsen
| So if message is rejected (even temporaily) why not to try next MX
| (as RFC 974 said)? Is Qmail following the specs or it is not?
Actually, RFC 974 is obsoleted by RFC 2821, so people should quote it
instead. (Though RFC 974 was the relevant one when qmail was
written.)
As to why qmail does what it does, there was an extremely brief
discussion on the qmail mailing list on 1996-06-25, and an equally
brief discussion on comp.protocols.tcp-ip in the days before.
http://groups.google.com/group/comp.mail.misc/browse_frm/thread/8987d5e1c3c4698f/9eaefb25953624c6?lnk=st&q=&rnum=1#9eaefb25953624c6
Nobody seemed to care much about the question back then. If I were to
guess, I'd say that djb chose what he did from the reasoning that
backup MXs are there to deal with a host being down, and of course a
host being down will not accept a connection. (Hosts were probably
down a lot more in those days.)
Be that as it may, changing qmail's behaviour on this issue would
probably be nontrivial, due to the way responsibility is divided up
between qmail-send and qmail-remote.
- Harald
Perhaps the question needs to be revisited because the world have
evolved since them. 1996 was before spam was invented. :)
Adam McKenna
2006-11-27 16:30:20 UTC
Permalink
Post by Marc Perkel
Perhaps the question needs to be revisited because the world have
evolved since them. 1996 was before spam was invented. :)
I suggest you donate to djb's writing fund along with your request.

http://cr.yp.to/donations.html#writing

--Adam
Harald Hanche-Olsen
2006-11-25 09:18:09 UTC
Permalink
+ Uncle George <***@gatworks.com>:

| As eventually you will find that the errant spammers will find the
| hole in your filtering scheme, and not bother with the lowest [...]

AFAIK, this is already the case. Much spam is sent directly to an MX
other than the lowest numbered one, without bothering with the lowest
one at all. Imagine that - spammers intentionally violating Internet
protocols! 8-)

- Harald
Marc Perkel
2006-11-25 15:09:14 UTC
Permalink
Post by Harald Hanche-Olsen
| As eventually you will find that the errant spammers will find the
| hole in your filtering scheme, and not bother with the lowest [...]
AFAIK, this is already the case. Much spam is sent directly to an MX
other than the lowest numbered one, without bothering with the lowest
one at all. Imagine that - spammers intentionally violating Internet
protocols! 8-)
- Harald
Yes, another trick I do is my highest MX record returns a 4xx on
everything that connect to it. Spammers try to hit the highest one first
thinking there's less spam filtering on it. As soon as the get the 4xx
error they go away and never come back. I get rid of about 350,000 spams
a day using that trick.

Which of course makes another point about the idea that if one server
says 4xx that the other servers won't. In my case the highest MX and the
lowest MX are on the same computer.
Markus Stumpf
2006-11-26 12:34:44 UTC
Permalink
Post by Marc Perkel
Yes, another trick I do is my highest MX record returns a 4xx on
everything that connect to it. Spammers try to hit the highest one first
thinking there's less spam filtering on it. As soon as the get the 4xx
error they go away and never come back. I get rid of about 350,000 spams
a day using that trick.
I really doubt that.
I agree that it helps to some extent, but I have been running such a
"trick" for about 2 years now. It rejected with 4xx at the end of the
DATA command at 2 longer distance MX hosts. This way one has the whole
message and I used that to feed it to a DCC system and build a SHA1 sum
of the message. The shortest distance MX is using greylisting.

I see tons of spammers connecting the whole MX set upwards and
downwards. So you probably force the spammers to 350000 useless
connects a day to your servers but I really doubt that you also get rid
of those 350000 spam mails a day with that "trick".

\Maex
Marc Perkel
2006-11-25 16:20:57 UTC
Permalink
Let me explain some of the things I do. My business is
junkemailfilter.com and it is likely from what I can tell the most
accurate spam filter and most efficient on the planet. (Efficient in the
amount of computing power it takes to process mail). What is my secret?
It's not a secret, but I use every trick that exists and several I
invented myself.

My service is that I do front end spam filtering. Email from the world
comes in, I process it, and send the good email on to their existing
server. Users set 3 MX records as follows:

mx.junkemailfilter.com - 10
mx.junkemailfilter.net - 20
mx.junkemailfilter.org - 30

The lowest MX record points to one main server and I have a second
server that runs Spam Assassin. The second MX points to 2-3 IP addresses
(I'm adding on) and in a remote location so that if there is a failure
on the main server the other servers will continure to process and
forward email until the main server comes back up. The highest MX just
returns 4xx on anything that connects to it. It's just a spam bait MX
for spammers that try to go in the back door.

There are a number of things I do that are different than most spam
filters. I try to catch spam based on the spammers behavour first and
then look at the message content last. I also actively look for non spam
as well as spam bypassing spam filtering based on a MySQL reputation
database. Using these tricks I only have to feed SpamAssassin about 2%
of the messages I process, which make my system very efficient.

I'm using Exim to do the tricky stuff. It is very versatile and can do
just about anything I want. So if you're wondering how I do that with
Qmail - I don't. I'm just here investigating why Qmail does unusual MX
processing. See my other thread for details.

Spam is different than regular email in several ways. Spammers are out
to get your money. So spammers are always trying to get you to DO
something. And that is something that you can often trap for.

Spammers are also working from lists that are mined from web sites and
such and the spammer has no relationship to the recipient, and there are
often ways you can tell that gives the spammer away.

Spammers are usually on the move. Since most places don't allow spamming
they have to set up, spam, and then move on. Or they are using windows
zombies on home computers.

Spammers goal is to spam as many people as possible. So they go for
those easiest to spam. If they encounter resistence they move on. They
don't usually keep retrying until the message is delivered. Real email
will keep trying becuse getting the specific message delivered is important.

Spammers also try to do tricky stuff to get past spam filters that only
spammers do. Once you identify the trick then it becomes a tell that
gives away all spammers using that trick with 100% accuracy.

So - getting to my point. My 3 MX levels have completely different
behavour relating to the use of 4xx errors. As I said my highest MX
always returns 4xx on everything just to send away spammers who are
going in the back door. And like I said this gets rid of about 350,000
spams a day with 0 processing power. In fact - if any of you want to cut
about 15% of your spam all you have to do is create a new highest MX and
point it to a dead IP address and you'll get a little less spam.

On the lowest MX, the main server, there are classifications of what I
call suspicious hosts. This is a category where 99% of the email comes
from spammers, but you can't just block it because the 1% good email
would bounce. These include hosts from dynamic IP ranges, blacklists
like spamcop that is just not quite good enough to just bounce on, and
hosts with bad reverse DNS. So the idea here is if I force them to retry
the second MX by returning 4xx errors then the hit and run spammers go
away and the ones who retry will usually be the 1% of good email that I
need to pass. This leads to a significant reduction in spam. Over 90%
actually.

Many virus infected spam zombies are not that sophisticated and they are
only smart enough to send to the lowest MX. Real MTAs are expected to be
smarter programs and will retry them all. That is one of the tricks I
use to separate real MTAs from virus infected zombies.

Every trick I use reduces some spam and I have thousands of tricks. The
front end levels take out the bulk of it and as the mesage progresses
through the system more and more spam is sheared off. I also have a fast
track whitelisting system to shear off the good email up front and pass
it on quickly also reducing system load and false positives.

Additionally the second tier servers are also backup should bad things
happen to that main servers the backups kick in and process email
without the end users experiencing any down time.

I also use it for load balancing. If for some reason the load levels go
above 80 on the main server I start returning 4xx errors to incoming
connections as a way of limiting the problem, letting the server to
process the jobs causing the load and taking new mail after the load
drops down again. The expectation is that on peak load that the second
tier servers will get the traffic instead allowing the email to pass
through them and on to the destination. And it works very well for those
sending mail with all other MTAs than qmail. The qmail senders have to
wait for the main server to accept the mail due to the way qmail works.

So - my point is, 4xx errors only mean that the server you are connected
to is not ready for some reason and it doesn't mean that any other
server on my network will also return a 4xx. The concept that if one
server returns 4xx that it speaks for all servers is a concept that I
have never heard until yesterday in this forum and it only exists in the
qmail world.

So - I have to say that I'm confused and surprized by this and still
trying to comprehend why qmail behaves differently than every other MTA
in this regard.
Charles Cazabon
2006-11-27 14:31:27 UTC
Permalink
Marc Perkel <***@perkel.com> wrote:
[much snippage]
Post by Marc Perkel
So - my point is, 4xx errors only mean that the server you are connected
to is not ready for some reason and it doesn't mean that any other
server on my network will also return a 4xx. The concept that if one
server returns 4xx that it speaks for all servers is a concept that I
have never heard until yesterday in this forum and it only exists in the
qmail world.
No.

Every server which you advertise as an MX for your domain is, by definition,
authoritative for your domain. Client MTAs have *no* obligation to
immeditately try one of your other servers after being told "Fuck off, I'm
busy" by the one they happen to connect to.
Post by Marc Perkel
So - I have to say that I'm confused and surprized by this and still
trying to comprehend why qmail behaves differently than every other MTA
in this regard.
A misinterpretation of the relevant standards does not make qmail's behaviour
incorrect in any fashion. And note that saying "But RFC2821 is out now,
that's the relevant standard" demonstrates an amazing lack of clue on the part
of the utterer.

Charles
--
--------------------------------------------------------------------------
Charles Cazabon <***@discworld.dyndns.org>
Read http://pyropus.ca/personal/writings/12-steps-to-qmail-list-bliss.html
My services include qmail consulting. See http://pyropus.ca/ for details.
--------------------------------------------------------------------------
Marc Perkel
2006-11-27 15:26:50 UTC
Permalink
Post by Charles Cazabon
[much snippage]
Post by Marc Perkel
So - my point is, 4xx errors only mean that the server you are connected
to is not ready for some reason and it doesn't mean that any other
server on my network will also return a 4xx. The concept that if one
server returns 4xx that it speaks for all servers is a concept that I
have never heard until yesterday in this forum and it only exists in the
qmail world.
No.
Every server which you advertise as an MX for your domain is, by definition,
authoritative for your domain. Client MTAs have *no* obligation to
immeditately try one of your other servers after being told "Fuck off, I'm
busy" by the one they happen to connect to.
No - on my system a 4xx error only means that that particular server
isn't ready. It means try the other servers or come back later. It
certianly dos not mean that all the servers aren't ready. That's what I
use EXIM and not Qmail. All other MTAs except Qmail behave this way.
it's what the spec says. Qmail does it wrong.
Jason Frisvold
2006-11-27 19:30:58 UTC
Permalink
No - on my system a 4xx error only means that that particular server isn't
ready. It means try the other servers or come back later. It certianly dos
not mean that all the servers aren't ready. That's what I use EXIM and not
Qmail. All other MTAs except Qmail behave this way. it's what the spec says.
Qmail does it wrong.
Hrm.. I find this interesting and slightly disturbing. Can you
please provide the spec and highlight the passage which identifies
this behavior as correct?

I'm not sure I like the idea of a mail server immediately trying
another MX if the initial one it contacted was busy. This just
creates additional load on the other MXs and can cause additional
meltdown. I believe the "correct" way for this to work is to queue
the message when a 4XX is received and retry later. When later
arrives, the MX lookup is repeated and at that point it's possible
that an alternative server is chosen.
--
Jason 'XenoPhage' Frisvold
***@gmail.com
Uncle George
2006-11-27 19:57:58 UTC
Permalink
Post by Jason Frisvold
No - on my system a 4xx error only means that that particular server isn't
ready. It means try the other servers or come back later. It
certianly dos
not mean that all the servers aren't ready. That's what I use EXIM and not
Qmail. All other MTAs except Qmail behave this way. it's what the spec says.
Qmail does it wrong.
Hrm.. I find this interesting and slightly disturbing. Can you
please provide the spec and highlight the passage which identifies
this behavior as correct?
I'm not sure I like the idea of a mail server immediately trying
another MX if the initial one it contacted was busy. This just
creates additional load on the other MXs and can cause additional
meltdown. I believe the "correct" way for this to work is to queue
the message when a 4XX is received and retry later. When later
arrives, the MX lookup is repeated and at that point it's possible
that an alternative server is chosen.
as per section 5 of RFC 2821 circa 2001: Address Resolution and Mail
Handling
To provide reliable
mail transmission, the SMTP client MUST be able to try (and retry)
each of the relevant addresses in this list in order, until a
delivery attempt succeeds. However, there MAY also be a configurable
limit on the number of alternate addresses that can be tried. In any
case, the SMTP client SHOULD try at least two addresses.
Charles Cazabon
2006-11-27 20:07:52 UTC
Permalink
Post by Jason Frisvold
No - on my system a 4xx error only means that that particular server isn't
ready. It means try the other servers or come back later. It certianly dos
not mean that all the servers aren't ready. That's what I use EXIM and not
Qmail. All other MTAs except Qmail behave this way. it's what the spec
says. Qmail does it wrong.
Hrm.. I find this interesting and slightly disturbing.
[...]
Post by Jason Frisvold
I'm not sure I like the idea of a mail server immediately trying another MX
if the initial one it contacted was busy. This just creates additional load
on the other MXs and can cause additional meltdown. I believe the "correct"
way for this to work is to queue the message when a 4XX is received and
retry later.
That is correct. Any MTA which immediately tries another delivery upon
receiving a temporary deferral is broken.

Charles
--
--------------------------------------------------------------------------
Charles Cazabon <***@discworld.dyndns.org>
Read http://pyropus.ca/personal/writings/12-steps-to-qmail-list-bliss.html
My services include qmail consulting. See http://pyropus.ca/ for details.
--------------------------------------------------------------------------
Marc Perkel
2006-11-27 20:28:12 UTC
Permalink
Post by Charles Cazabon
That is correct. Any MTA which immediately tries another delivery upon
receiving a temporary deferral is broken.
So - Qmail does it right and every other MTA does it wrong?
Uncle George
2006-11-27 20:59:12 UTC
Permalink
Post by Marc Perkel
Post by Charles Cazabon
That is correct. Any MTA which immediately tries another delivery upon
receiving a temporary deferral is broken.
So - Qmail does it right and every other MTA does it wrong?
Qmail does it correctly, as it was designed before 2001.
Qmail does not do it correctly because the latest RFC suggests a
different SMTP strategy.

One might think that after 5 years, the debate over this issue would
have been resolved. And there would be some progress towards
implementation. Maybe like what the spec's says, except also allow a
configurable zero or more MX retries. For those who want to be in
current RFC graces, then set it to min 2. For those who want no retries,
then set it to max 1.

IMHO, i'm not please with the retry of all MX's as an smtp strategy.
Even less so as a strategy for spam.
Marc Perkel
2006-11-29 11:36:21 UTC
Permalink
Qmail does it correctly, as it was designed before 2001. Qmail does
not do it correctly because the latest RFC suggests a different SMTP
strategy.
One might think that after 5 years, the debate over this issue would
have been resolved. And there would be some progress towards
implementation. Maybe like what the spec's says, except also allow a
configurable zero or more MX retries. For those who want to be in
current RFC graces, then set it to min 2. For those who want no
retries, then set it to max 1.
Yes - it would be nice if Qmail (without having to apply a third part
path) would have the *option* of processing MX records the way Exim,
Sendmail, Postfix, and Exchange and every other MTA does. Is it too much
to ask for the option?
Randy Adamczyk
2006-11-29 15:02:34 UTC
Permalink
Post by Marc Perkel
Yes - it would be nice if Qmail (without having to apply a third
part path) would have the *option* of processing MX records the way
Exim, Sendmail, Postfix, and Exchange and every other MTA does. Is
it too much to ask for the option?
even if qmail had this option, how would you convince every qmail
admin in the world to use it the way you want them to? you have
changed the way your mailserver receives email. now you're expecting
others to adapt their setup to yours.

_you_ are the one who has to either accept the consequences of your
(mis)configuration, or change your setup if you can't live with the
consequences.

so long,
randy
Fabio Busatto
2006-11-29 16:06:54 UTC
Permalink
Post by Marc Perkel
Yes - it would be nice if Qmail (without having to apply a third part
path) would have the *option* of processing MX records the way Exim,
Sendmail, Postfix, and Exchange and every other MTA does. Is it too much
to ask for the option?
Yes. It's too much.
qmail development is performed only by djb, and none of us can do "non-third
part patches", because we are not djb and none of our patches can be included
in "official" qmail release without djb approval.
Are you sad because of qmail behavior and you want your ideas to be included
in qmail code? Mail djb, not this list (and good luck, eheheh :).
Do you want patches for qmail? They are there, and now everyone in this list
is informed about the problem and the possible solution.
Are you flaming about qmail and djb decisions? Sorry, this is not the right
place, and now I think you've exposed your opinion clearly.
Just give up, you can obtain nothing from here more than the "third-part patch",
so, what's the problem now?
If you mail djb and get a positive answer, let us know where we can download
qmail-1.04.tar.gz.

Have a good life.
Fabio
Russ Nelson
2006-11-29 21:00:35 UTC
Permalink
Post by Marc Perkel
Yes - it would be nice if Qmail (without having to apply a third part
path) would have the *option* of processing MX records the way Exim,
Sendmail, Postfix, and Exchange and every other MTA does. Is it too much
to ask for the option?
Yes. It's useless for delivering email to properly configured hosts.

But let's say it existed. Your technique for blocking spam is a
FUSSP. You're assuming that spammers will not change their methods
once they are seen to fail. Their rule? "Three MX records means: try
the middle one first." All your efforts will be for naught.
--
--my blog is at http://blog.russnelson.com | You can do any damn thing
Crynwr sells support for free software | PGPok | you want, as long as you
521 Pleasant Valley Rd. | +1 315-323-1241 | don't expect somebody else
Potsdam, NY 13676-3213 | Sheepdog | to pick up the pieces.
Matt Thoene
2006-11-27 21:09:20 UTC
Permalink
Hello Marc,
Post by Marc Perkel
Post by Charles Cazabon
That is correct. Any MTA which immediately tries another delivery upon
receiving a temporary deferral is broken.
So - Qmail does it right and every other MTA does it wrong?
It's been a while since I've worked with Postfix but I am pretty sure
they too do not immediately try backups when they get a temporary
deferral. The mail is placed in the deferred queue and tries again
later.
--
Matt
Marc Perkel
2006-11-27 22:34:39 UTC
Permalink
Post by Matt Thoene
Hello Marc,
Post by Marc Perkel
Post by Charles Cazabon
That is correct. Any MTA which immediately tries another delivery upon
receiving a temporary deferral is broken.
So - Qmail does it right and every other MTA does it wrong?
It's been a while since I've worked with Postfix but I am pretty sure
they too do not immediately try backups when they get a temporary
deferral. The mail is placed in the deferred queue and tries again
later.
Nope - Postfix will immediately go up the MX list trying all servers.
I'm sure of that because I have a test account on a Postfix server and I
often test it. Only Qmail - and no other MTA will refuse to retry the
higher MX on a 4xx error. I originally came here to report it as a bug
and figured that it was only in very old qmail versions. I'm quite
surprized to find it was deliberate and there's a cult like attachment
to the problem.
Charles Cazabon
2006-11-28 01:49:55 UTC
Permalink
Only Qmail - and no other MTA will refuse to retry the higher MX on a 4xx
error. I originally came here to report it as a bug and figured that it was
only in very old qmail versions. I'm quite surprized to find it was
deliberate and there's a cult like attachment to the problem.
Instead of actually trying to understand the way that the protocol works (and
that means taking into account the way it works in practice, not just in
theory), you tried to guess the right behaviour from a sample population of
one (!). That, not suprisingly, gave you the wrong answer.

It's a common error, as actually trying to understand the protocol properly is
a lot of hard work. But don't blame qmail because you got it wrong -- and
there's no doubt that you did. You've illustrated that amply in this thread.

Charles
--
--------------------------------------------------------------------------
Charles Cazabon <***@discworld.dyndns.org>
Read http://pyropus.ca/personal/writings/12-steps-to-qmail-list-bliss.html
My services include qmail consulting. See http://pyropus.ca/ for details.
--------------------------------------------------------------------------
Charles Cazabon
2006-11-27 20:56:15 UTC
Permalink
Post by Marc Perkel
Post by Charles Cazabon
That is correct. Any MTA which immediately tries another delivery upon
receiving a temporary deferral is broken.
So - Qmail does it right and every other MTA does it wrong?
No. Most MTAs also back off before trying again if they receive a temporary
deferral.

Note the difference between "failure to connect", "failure to get a banner",
and "temporary deferral".

Charles
--
--------------------------------------------------------------------------
Charles Cazabon <***@discworld.dyndns.org>
Read http://pyropus.ca/personal/writings/12-steps-to-qmail-list-bliss.html
My services include qmail consulting. See http://pyropus.ca/ for details.
--------------------------------------------------------------------------
Harald Hanche-Olsen
2006-11-27 19:53:58 UTC
Permalink
+ Marc Perkel <***@perkel.com>:

| No - on my system a 4xx error only means that that particular server
| isn't ready. It means try the other servers or come back later.

In that case, it would probably be better to just stop listening on
the smtp port while the load is too high. But you want to tailor your
response to what host is connecting: Too bad then that unix doesn't
let you move such decisions into userland. If you're listening on a
port, you accept all incoming connections no matter what, and then the
best you can do is either just drop the connection, or try to tell the
client to go away, and this is where you clash with established
protocol.

Are there Unix variants that will let you decide in userland what to
do about an incoming SYN? The Hurd perhaps? AFAIK, one of the design
goals of the Hurd is precisely to move stuff from the kernel into
userland.

- Harald
Donald Nash
2006-11-27 23:52:13 UTC
Permalink
--On November 27, 2006 8:53:58 PM +0100 Harald Hanche-Olsen
Post by Harald Hanche-Olsen
Are there Unix variants that will let you decide in userland what to
do about an incoming SYN?
None that I know of. This feature was included in DEC Ultrix, but only for
DEC's adaptation of DECnet to the socket API. As I recall, you could use
setsockopt() to mark an AF_DECNET socket in such a way that listen() would
return without actually accepting the incoming connection. You could then
use getpeername() to determine the identity of the remote system, and then
use setsockopt() either to accept or reject the connection. This must have
been at least 15 years ago, and it has been a constant source of low-level
amazement to me ever since that it was never adapted to TCP/IP, not even by
DEC for Ultrix.
--
Donald L. Nash, <***@its.utexas.edu>
Information Technology Services, The University of Texas at Austin
Paul Jarc
2006-11-28 00:09:43 UTC
Permalink
Post by Harald Hanche-Olsen
Are there Unix variants that will let you decide in userland what to
do about an incoming SYN?
It could be done by manipulating firewall rules, although that's
hardly convenient.


paul
Charles Cazabon
2006-11-28 01:37:30 UTC
Permalink
Post by Paul Jarc
Post by Harald Hanche-Olsen
Are there Unix variants that will let you decide in userland what to
do about an incoming SYN?
It could be done by manipulating firewall rules, although that's
hardly convenient.
Don't the BSD and Linux networking stacks let you do this with the netfilter
stuff? I'm sure they let you pass packets to a userspace application for
accept/reject/drop or more complex processing.

Charles
--
--------------------------------------------------------------------------
Charles Cazabon <***@discworld.dyndns.org>
Read http://pyropus.ca/personal/writings/12-steps-to-qmail-list-bliss.html
My services include qmail consulting. See http://pyropus.ca/ for details.
--------------------------------------------------------------------------
Michael Sierchio
2006-11-29 22:23:30 UTC
Permalink
Post by Charles Cazabon
Don't the BSD and Linux networking stacks let you do this with the netfilter
stuff? I'm sure they let you pass packets to a userspace application for
accept/reject/drop or more complex processing.
Yes, the IPFIREWALL/ipfw on FreeBSD has the concept of a DIVERT
socket, in which a packet is passed to a userland daemon for
processing and reinjection into the ruleset. */natd*/ is
implemented this way.
Adam McKenna
2006-11-28 18:15:26 UTC
Permalink
Post by Harald Hanche-Olsen
Are there Unix variants that will let you decide in userland what to
do about an incoming SYN? The Hurd perhaps? AFAIK, one of the design
goals of the Hurd is precisely to move stuff from the kernel into
userland.
I would imagine the easiest way to accomplish this is to figure out how many
connections your server can handle, and set the number of available SMTP TCP
connections to that value.

--Adam
Tyler
2006-11-27 21:09:44 UTC
Permalink
Post by Marc Perkel
No - on my system a 4xx error only means that that particular server
isn't ready. It means try the other servers or come back later. It
certianly dos not mean that all the servers aren't ready. That's what I
use EXIM and not Qmail. All other MTAs except Qmail behave this way.
it's what the spec says. Qmail does it wrong.
I've been waiting to weigh in on this topic. There are a few issues
here that are completely overblown or misinterpreted. (Sorry about the
long post).

From the other thread, the relevant RFC passage was quoted from RFC2821:

To provide reliable mail transmission, the SMTP client MUST be able
to try (and retry) each of the relevant addresses in this list in
order, until a delivery attempt succeeds. However, there MAY also
be a configurable limit on the number of alternate addresses that
can be tried. In any case, the SMTP client SHOULD try at least two
addresses.

If you look carefully, at the top of the RFC, "SHOULD" is defined:

3. SHOULD This word, or the adjective "RECOMMENDED", mean that
there may exist valid reasons in particular circumstances to
ignore a particular item, but the full implications must be
understood and carefully weighed before choosing a different
course.

So, since QMail is capable of trying and retrying different MXes, it is
RFC compliant. The act of choosing not to try another MX when it can
contact a lower priority MX is not of itself a violation of the RFC,
since that sentence is only a recommendation.

As Harald Hanche-Olsen pointed out, RFC 2821 does not allow 4xx replies
upon connect, so, if anything, it is your configuration which is not RFC
compliant. According to the RFC, the only codes you can give at connect
is 220 or 554:

The SMTP protocol allows a server to formally reject a transaction
while still allowing the initial connection as follows: a 554
response MAY be given in the initial connection opening message
instead of the 220.

Given the actual communication, if you are responding to the initial
connection with a 4xx series error, QMail is actually functioning
properly (this was just touched upon by Charles Cazabon):

4yz Transient Negative Completion reply
The command was not accepted, and the requested action did not
occur. However, the error condition is temporary and the action
may be requested again.
[-snip-]
A rule of thumb to determine whether a reply fits into the 4yz or
the 5yz category (see below) is that replies are 4yz if they can
be successful if repeated without any change in command form or in
properties of the sender or receiver.

A 4xx series response tells the sending MTA that those commands probably
would have worked, but something is temporarily wrong. It tells the
sending MTA to "try again later". It mentions nothing of trying another
MX upon this failure. Additionally, the sending MTA must back off:

The sender MUST delay retrying a particular destination after one
attempt has failed. In general, the retry interval SHOULD be at
least 30 minutes; however, more sophisticated and variable strategies
will be beneficial when the SMTP client can determine the reason for
non-delivery.

This says nothing of trying another MX, and furthermore, depending on
how you define 'destination', could mean that immediately trying another
MX for the same destination domain violates the RFC.

Seeing as your your lowest and highest MXes intend never to actually
accept the mail, they should be returning a 5xx series errors,
specifically 554 "No SMTP service here", however, that a permfail that
would cause the sending MTA to bounce the messages.

Tyler
Uncle George
2006-11-27 21:38:49 UTC
Permalink
Post by Tyler
To provide reliable mail transmission, the SMTP client MUST be able
to try (and retry) each of the relevant addresses in this list in
order, until a delivery attempt succeeds.
I think you are missing the *MUST* in the first sentence. There is no
try of any other MX if a connection is made, but delivery fails for any
reason.
The SHOULD addresses a configurable limit of MX tries.
Larry Weldon
2006-11-27 22:24:38 UTC
Permalink
Post by Uncle George
Post by Tyler
To provide reliable mail transmission, the SMTP client MUST be able
to try (and retry) each of the relevant addresses in this list in
order, until a delivery attempt succeeds.
I think you are missing the *MUST* in the first sentence. There is no
try of any other MX if a connection is made, but delivery fails for any
reason.
You correctly point out that it says "MUST be _able_ to" - that is, it
does NOT say "MUST try and retry".

Larry
Tyler
2006-11-27 22:50:54 UTC
Permalink
Post by Uncle George
Post by Tyler
To provide reliable mail transmission, the SMTP client MUST be able
to try (and retry) each of the relevant addresses in this list in
order, until a delivery attempt succeeds.
I think you are missing the *MUST* in the first sentence. There is no
try of any other MX if a connection is made, but delivery fails for any
reason.
The SHOULD addresses a configurable limit of MX tries.
IMHO, QMail does try and retry each of the relevant addresses in the MX
list, in order, until a delivery attempt succeeds. The issue is that
different people interpret this in different ways:

Some people think this should mean:
-> TRY1: "MX5 is dead, MX10 is greylisting, MX15 is greylisting, MX20 is
greylisting"
-> TRY2: "MX5 is dead, MX10 accepts message"

While QMail thinks this should mean:
-> TRY1: "MX5 is dead, MX10 is greylisting"
-> TRY2: "MX10 accepts message"

Is there a reason to go onto MX15 or MX20? MX10 says that it'll accept
the mail, but that it's temporarily unable to do so. Trying the other
MXes just generates additional unnecessary traffic. The only problem
occurs when the receiving MTA lies about it's ability to accept the
message, or a temporary failure that takes more than your maximum queue
time to resolve.

"MAY" addresses the configurable MX retry limit; "SHOULD" recommends
that QMail try more than one MX, even though it's not required to:

In any case, the SMTP client SHOULD try at least two addresses.

So, as I said before, as long as QMail is _capable_ of trying more than
one MX, it doesn't have to, and is still RFC compliant. (Not that I put
a whole lot of stock in everything being RFC compliant.)

Tyler
Uncle George
2006-11-27 23:57:37 UTC
Permalink
Post by Tyler
-> TRY1: "MX5 is dead, MX10 is greylisting, MX15 is greylisting, MX20
is greylisting"
-> TRY2: "MX5 is dead, MX10 accepts message"
-> TRY1: "MX5 is dead, MX10 is greylisting"
u really mean:
TRY1: "MX5 is dead, MX10 is greylisting, MX15 dont bother, MX20 dont
bother"
Post by Tyler
-> TRY2: "MX10 accepts message"
Is there a reason to go onto MX15 or MX20? MX10 says that it'll
accept the mail, but that it's temporarily unable to do so. Trying
the other MXes just generates additional unnecessary traffic.
Yes, maybe MX15 will accept the message! Maybe MX20 will accept the
message. You wont know till you try.

As the original poster stated, a long time ago, his service greylist's
everyone on the lowest MX. But apparently accepts, without greylisting,
on the next MX. This on the presumption that spammers will only try the
lowest MX, and ignore the higher MX's. Real MTA's, as per 2001 spec's,
would have tried at least the 2 very lowest MX's, if not the
preconfigured limit, or all of the MX's .

Its not for me to say what the reasons ( or reasonableness ) are for why
this SMTP strategy is acceptable for some. Or even why such matters have
to be codified in an RFC .
Marc Perkel
2006-11-28 05:22:38 UTC
Permalink
Post by Uncle George
Yes, maybe MX15 will accept the message! Maybe MX20 will accept the
message. You wont know till you try.
As the original poster stated, a long time ago, his service
greylist's everyone on the lowest MX. But apparently accepts, without
greylisting, on the next MX. This on the presumption that spammers
will only try the lowest MX, and ignore the higher MX's. Real MTA's,
as per 2001 spec's, would have tried at least the 2 very lowest MX's,
if not the preconfigured limit, or all of the MX's .
Its not for me to say what the reasons ( or reasonableness ) are for
why this SMTP strategy is acceptable for some. Or even why such
matters have to be codified in an RFC .
Except for the qmail problem, if someone wanted to get dir of say 90% of
all their spam here's an easy way to do it.

You have 3 MX records. The highest and lowest both return 4xx on
everything. The middle MX accepts all. Then you block with a few of the
better black lists and you have a spam filter that is more accurate than
using Spam Assassin by itself.

I use both. When I get down to the last 10% I can cull out half the good
email with my whitelist tricks and that leaves only 5% for SA.
Charles Cazabon
2006-11-28 13:39:05 UTC
Permalink
Except for the qmail problem, if someone wanted to get [rid] of say 90% of
all their spam here's an easy way to do it.
You'll excuse us for not taking that number seriously, as there's still some
traces of fecal matter on it.

Charles
--
--------------------------------------------------------------------------
Charles Cazabon <***@discworld.dyndns.org>
Read http://pyropus.ca/personal/writings/12-steps-to-qmail-list-bliss.html
My services include qmail consulting. See http://pyropus.ca/ for details.
--------------------------------------------------------------------------
Ted Fines
2006-11-28 14:59:17 UTC
Permalink
Post by Charles Cazabon
Except for the qmail problem, if someone wanted to get [rid] of say 90%
of
all their spam here's an easy way to do it.
You'll excuse us for not taking that number seriously, as there's still some
traces of fecal matter on it.
Charles
--
--------------------------------------------------------------------------
Read http://pyropus.ca/personal/writings/12-steps-to-qmail-list-bliss.html
My services include qmail consulting. See http://pyropus.ca/ for details.
--------------------------------------------------------------------------
Where's Dave Sill when you need him? Well, I guess this will have to do.
Loading Image...
DAve
2006-11-28 15:15:26 UTC
Permalink
Post by Marc Perkel
Except for the qmail problem, if someone wanted to get [rid] of
say 90% of
all their spam here's an easy way to do it.
You'll excuse us for not taking that number seriously, as there's still some
traces of fecal matter on it.
Charles
Where's Dave Sill when you need him? Well, I guess this will have to do.
http://www.apecity.com/lwtqml.gif
Ha ha ha ha ha ha ha ha ahhhhh ha.

You forgot the percentage where someone gets called an a%^hole or
f*&cking stupid.

I just LOVE those, they reaffirm my chosen career path ;^)

DAve
--
Three years now I've asked Google why they don't have a
logo change for Memorial Day. Why do they choose to do logos
for other non-international holidays, but nothing for
Veterans?

Maybe they forgot who made that choice possible.
Marc Perkel
2006-11-27 22:42:28 UTC
Permalink
From the other thread, the relevant RFC passage was quoted from RFC2821:

To provide reliable mail transmission, the SMTP client MUST be able
to try (and retry) each of the relevant addresses in this list in
order, until a delivery attempt succeeds.


Seems clear to me. Unless you want to argue what MUST means.

However, there MAY also
be a configurable limit on the number of alternate addresses that
can be tried. In any case, the SMTP client SHOULD try at least two
addresses.

What this says is that if you decide to have some limits that the limits
should be set at 2 or more. Since Qmail doesn't have a configurable
limit then the MUST part applies.

Clearly it is saying that in any case that at least 2 mx records must
best tested. For some reason you are ignoring the plain language of the
RFC. It could have been written more precisely but you really have to
stretch to not get what it is clearly saying.

Qmail is out of compliance with RFC282.
Richard Archer
2006-11-27 22:52:55 UTC
Permalink
Post by Marc Perkel
Qmail is out of compliance with RFC282.
Hey, if you want to drop all mail from qmail servers into
the bit bucket, that's fine with me.

However if you want to run a reliable and useful service
then you need to accept the fact that qmail does what it
does and isn't going to change no matter how much you
whine about it. Just get on with the job of making your
system work within these known parameters.

...R.
Tyler
2006-11-27 23:06:16 UTC
Permalink
Post by Tyler
To provide reliable mail transmission, the SMTP client MUST be able
to try (and retry) each of the relevant addresses in this list in
order, until a delivery attempt succeeds.
Seems clear to me. Unless you want to argue what MUST means.
However, there MAY also
be a configurable limit on the number of alternate addresses that
can be tried. In any case, the SMTP client SHOULD try at least two
addresses.
What this says is that if you decide to have some limits that the limits
should be set at 2 or more. Since Qmail doesn't have a configurable
limit then the MUST part applies.
Clearly it is saying that in any case that at least 2 mx records must
best tested. For some reason you are ignoring the plain language of the
RFC. It could have been written more precisely but you really have to
stretch to not get what it is clearly saying.
Qmail is out of compliance with RFC282.
The SMTP client MUST be able to try (and retry) each of the relevant
addresses in this list in order, until a delivery attempt succeeds.
QMail does have the ability to try and retry each MX, for example, when
none of the MXes are reachable.
Post by Tyler
There MAY also be a configurable limit on the number of alternate
addresses that can be tried.
As far as I am aware, this is not configurable in QMail, but is not
required.
Post by Tyler
In any case, the SMTP client SHOULD try at least two addresses.
This is a recommendation, it does not have to try at least two
addresses. So, as long as QMail is ABLE to try and retry according to
the first sentence, it is in compliance.


Whether you like it or not, QMail is (at least arguably), RFC compliant,
whereas your spam solution, as I mention in the other thread, is not RFC
compliant.

Either way, with QMail as one of the top 4 most popular MTAs (arguably
the 2nd most popular), and used by giants like Yahoo, whether it's RFC
compliant or not, it's widely used and accepted, and changing every
install of QMail so it works with your system isn't a feasible option.

Tyler
Markus Stumpf
2006-11-27 23:22:53 UTC
Permalink
Post by Marc Perkel
What this says is that if you decide to have some limits that the limits
should be set at 2 or more. Since Qmail doesn't have a configurable
limit then the MUST part applies.
Read on in the same section:

Although the capability to try multiple alternative addresses is
required, specific installations may want to limit or disable the use
of alternative addresses. The question of whether a sender should
attempt retries using the different addresses of a multihomed host
has been controversial. The main argument for using the multiple
addresses is that it maximizes the probability of timely delivery,
and indeed sometimes the probability of any delivery; the counter-
argument is that it may result in unnecessary resource use. Note
that resource use is also strongly determined by the sending strategy
discussed in section 4.5.4.1.

Now look at "4.5.4.1 Sending Strategy"

The sender MUST delay retrying a particular destination after one
attempt has failed. In general, the retry interval SHOULD be at
least 30 minutes; however, more sophisticated and variable strategies
will be beneficial when the SMTP client can determine the reason for
non-delivery.

All that you cite is that a sending MTA must have the capability to use
different/alternate MX hosts. It says nothing about using them, when
using them and in which interval and for what reasons.

For standards it doesn't help to grep for a keyword and cite a phrase.
For standards you have to read the whole paper and understand it.
Post by Marc Perkel
Qmail is out of compliance with RFC282.
Says who? You? *lol*

\Maex
Marc Perkel
2006-11-28 16:17:48 UTC
Permalink
Post by Markus Stumpf
Post by Marc Perkel
What this says is that if you decide to have some limits that the limits
should be set at 2 or more. Since Qmail doesn't have a configurable
limit then the MUST part applies.
This document technically contradicts itself. Not written by a lawyer.
As you can see in one place it says MUST and then gives an exception. Or
it says REQUIRED but then another exception. So lets see what it really
means
Post by Markus Stumpf
Although the capability to try multiple alternative addresses is
required,
Note REQUIRED.
Post by Markus Stumpf
specific installations may want to limit or disable the use
of alternative addresses.
I believe what it means here is that the capability is required but you
might want to put limits on it because of resources.
Post by Markus Stumpf
The question of whether a sender should
attempt retries using the different addresses of a multihomed host
has been controversial. The main argument for using the multiple
addresses is that it maximizes the probability of timely delivery,
and indeed sometimes the probability of any delivery; the counter-
argument is that it may result in unnecessary resource use.
This is what lawyers cal dictum. It's a discussion. The reason for
trying all MX is timely delivery. The exception is resources. If for
example you are running your server on a dialup connection at 56k you
can make an exception because retries use up your limited bandwidth.
Post by Markus Stumpf
Note
that resource use is also strongly determined by the sending strategy
discussed in section 4.5.4.1.
Now look at "4.5.4.1 Sending Strategy"
The sender MUST delay retrying a particular destination after one
attempt has failed.
By destination this means host. That means you don't keep hammering the
same host over and over. But it does not mean that you don't try other
servers. If it did then there would be no point to multiple MX records.
Post by Markus Stumpf
In general, the retry interval SHOULD be at
least 30 minutes; however, more sophisticated and variable strategies
will be beneficial when the SMTP client can determine the reason for
non-delivery.
Exim has a nice timing feature where after a number of hours the retry
interval gets lomger.
gauze
2006-11-28 16:26:50 UTC
Permalink
Post by Marc Perkel
Exim has a nice timing feature where after a number of hours the retry
interval gets lomger.
http://www.micronicos.com/whatsnew/qmail-retry-timing.html


brian
--
Never be afraid to tell the world who you are.
-- Anonymous
11:20:01 up 4 days, 11:53, 5 users, load average: 0.10, 0.27, 0.33
Tullio Andreatta ML
2006-11-28 17:16:45 UTC
Permalink
Post by Marc Perkel
Post by Markus Stumpf
Post by Marc Perkel
What this says is that if you decide to have some limits that the limits
should be set at 2 or more. Since Qmail doesn't have a configurable
limit then the MUST part applies.
...
Post by Marc Perkel
Post by Markus Stumpf
Although the capability to try multiple alternative addresses is
required,
Note REQUIRED.
So qmail MUST try multiple alternative addresses. And in effect IT DOES
THAT (if connection cannot be established).
Post by Marc Perkel
Post by Markus Stumpf
specific installations may want to limit or disable the use
of alternative addresses.
I believe what it means here is that the capability is required but you
might want to put limits on it because of resources.
I believe what it means here is that the capability can be subjected to
policy restriction. For qmail, policy rules are:
- if a connection cannot be established, try next MX.
- if a connection succeeds, but server is unable to accept transaction,
don't try next MX.
Compliant.
Post by Marc Perkel
Post by Markus Stumpf
The question of whether a sender should
attempt retries using the different addresses of a multihomed host
has been controversial. The main argument for using the multiple
addresses is that it maximizes the probability of timely delivery,
and indeed sometimes the probability of any delivery; the counter-
argument is that it may result in unnecessary resource use.
This is what lawyers cal dictum. It's a discussion. The reason for
trying all MX is timely delivery. The exception is resources. If for
example you are running your server on a dialup connection at 56k you
can make an exception because retries use up your limited bandwidth.
"The main argument ... is", so it's not the only argument. Another
argument may exists: so "if the main server is up, try only that server
because sending message to the final recipient is faster" is another
good argument. Counter-argument is "but if sysadmin screwed it's
configuration, misused protocol, activated brain-dead anti-spam
measures, it will not receive the message" ...
Post by Marc Perkel
Post by Markus Stumpf
Note
that resource use is also strongly determined by the sending strategy
discussed in section 4.5.4.1.
Now look at "4.5.4.1 Sending Strategy"
The sender MUST delay retrying a particular destination after one
attempt has failed.
By destination this means host.
Why? "Particular destination" may be also a mailbox, a domain, an IP
address. Qmail "particular destination" is "domain".
Post by Marc Perkel
That means you don't keep hammering the
same host over and over.
And if you send the message to 2nd host, 2nd host will then "hammer"
first host for you? In Italy we call that "mafia" ... :-)
Post by Marc Perkel
But it does not mean that you don't try other
servers. If it did then there would be no point to multiple MX records.
If SMTP client can't connect to first MX, may be 2nd MX can. So qmail
send message to 2nd MX if connection cannot be established.
--
Tullio Andreatta

Disclaimer: "Please treat this email message in a reasonable way, or we
might get angry" ( http://www.goldmark.org/jeff/stupid-disclaimers )
Uncle George
2006-11-28 19:40:15 UTC
Permalink
Post by Tullio Andreatta ML
So qmail MUST try multiple alternative addresses. And in effect IT DOES
THAT (if connection cannot be established).
"So qmail MUST try multiple alternative addresses" - until the message
is successfully accepted. Qmail does not do this for all cases on which
the message is not accepted.
Post by Tullio Andreatta ML
Post by Marc Perkel
Post by Markus Stumpf
specific installations may want to limit or disable the use
of alternative addresses.
I believe what it means here is that the capability is required but you
might want to put limits on it because of resources.
I believe what it means here is that the capability can be subjected to
- if a connection cannot be established, try next MX.
- if a connection succeeds, but server is unable to accept transaction,
don't try next MX.
Unfortunately, its not for qmail to decide policy. Its up to the admins,
air-head management, and yokels to deceide for them selfs what the
policy restrictions should be. Once the "try all MX's" requirement is
implimeted, then administratively the try all mx's can be reduced to
just 2. I suspect that the limit can be further reduced to 1.
Post by Tullio Andreatta ML
Compliant.
Post by Marc Perkel
Post by Markus Stumpf
The question of whether a sender should
attempt retries using the different addresses of a multihomed host
has been controversial. The main argument for using the multiple
addresses is that it maximizes the probability of timely delivery,
and indeed sometimes the probability of any delivery; the counter-
argument is that it may result in unnecessary resource use.
This is what lawyers cal dictum. It's a discussion. The reason for
trying all MX is timely delivery. The exception is resources. If for
example you are running your server on a dialup connection at 56k you
can make an exception because retries use up your limited bandwidth.
"The main argument ... is", so it's not the only argument. Another
argument may exists: so "if the main server is up, try only that server
because sending message to the final recipient is faster" is another
good argument. Counter-argument is "but if sysadmin screwed it's
configuration, misused protocol, activated brain-dead anti-spam
measures, it will not receive the message" ...
I suppose the point here is that the sysadmin would/should be fired?
Clients would move onto other organizations that sysadmin better? etc.
Darwin had something to say on this subject.
Both MUA and MTA have administative choices in how the facility should
run. If the MTA does not want all the servers hit, then limit the number
of published MX's. If the MUA does not want to try all the MX's, then
set the limit to (min) 2.

BTW: if "you are running your server on a dialup", then you probably
have only one MX, and this discourse becomes academic.
Lars Hansson
2006-11-28 02:42:32 UTC
Permalink
Post by Tyler
To provide reliable mail transmission, the SMTP client MUST be able
to try (and retry) each of the relevant addresses in this list in
order, until a delivery attempt succeeds.
Seems clear to me. Unless you want to argue what MUST means.
It only says it has to be able to do it, it doesnt say it *has* to do it.
Post by Tyler
Clearly it is saying that in any case that at least 2 mx records must
best tested. For some reason you are ignoring the plain language of the
RFC. It could have been written more precisely but you really have to
stretch to not get what it is clearly saying.
Like how it is precise about not issuing 4xx replies on connection?
Post by Tyler
Qmail is out of compliance with RFC282.
What's out of compliance is your 4xx reply on connection and there's nothing
unclear about that.

---
Lars Hansson
Russ Nelson
2006-11-29 20:52:18 UTC
Permalink
Post by Marc Perkel
Qmail does it wrong.
You're right. You should go away, use Exim, and stop posting to this
mailing list.
--
--my blog is at http://blog.russnelson.com | You can do any damn thing
Crynwr sells support for free software | PGPok | you want, as long as you
521 Pleasant Valley Rd. | +1 315-323-1241 | don't expect somebody else
Potsdam, NY 13676-3213 | Sheepdog | to pick up the pieces.
Harald Hanche-Olsen
2006-11-27 19:31:14 UTC
Permalink
+ Charles Cazabon <***@discworld.dyndns.org>:

| A misinterpretation of the relevant standards does not make qmail's
| behaviour incorrect in any fashion. And note that saying "But
| RFC2821 is out now, that's the relevant standard" demonstrates an
| amazing lack of clue on the part of the utterer.

I am the utterer in this case. Would you care to elaborate? Before
responding, please note that I have not at any point supported Marc's
position on this point. (Nor have I opposed it, other than by
pointing out that a 4xx response in the greeting is a violation of the
standard -- both RFC 2821 and RFC 821.) In fact, I fail to see any
significant difference between RFC 2821 and earlier RFCs on the
particular point under discussion. Neither of them is very clear on
precisely what circumstances demand moving to the next MX, which is
precisely the question that djb raised in that usenet posting ages
ago, and which drew next to no response at all.

- Harald
Charles Cazabon
2006-11-27 20:13:00 UTC
Permalink
Post by Harald Hanche-Olsen
| A misinterpretation of the relevant standards does not make qmail's
| behaviour incorrect in any fashion. And note that saying "But
| RFC2821 is out now, that's the relevant standard" demonstrates an
| amazing lack of clue on the part of the utterer.
I am the utterer in this case. Would you care to elaborate?
I wasn't referring to your reference to the spec in this thread. I was
referring to those who think an MTA is broken because it doesn't implement a
newly-minted RFC like 2821 the day after it comes out.

The SMTP spec moves *slowly*, precisely because SMTP has been around for a
long time, and it takes years (if not decades, now) to make sure that "old"
clients have died off. Backwards-incompatible changes can only be introduced
at a glacial pace, if at all. Some would say 2821 was actually *too*
progressive due to the problems with the DRUMS process.

Charles
--
--------------------------------------------------------------------------
Charles Cazabon <***@discworld.dyndns.org>
Read http://pyropus.ca/personal/writings/12-steps-to-qmail-list-bliss.html
My services include qmail consulting. See http://pyropus.ca/ for details.
--------------------------------------------------------------------------
Markus Stumpf
2006-11-27 23:30:05 UTC
Permalink
Post by Charles Cazabon
I wasn't referring to your reference to the spec in this thread. I was
referring to those who think an MTA is broken because it doesn't implement a
newly-minted RFC like 2821 the day after it comes out.
*smile*
RFC2821 obsoletes RFC821, but RFC821 is still the STD document (STD 10)
for SMTP.

\Maex
Arkadiusz Dykiel
2006-11-30 22:01:15 UTC
Permalink
Hello Marc,
Post by Marc Perkel
My business is
junkemailfilter.com and it is likely from what I can tell the most
accurate spam filter
I know better but I won't mention them here :)
Post by Marc Perkel
It's not a secret, but I use every trick that exists and several I
invented myself.
I think that's why you and some of your customers had been kicked of
LKML.

[advanced, tricky 4xx and other tricks]

All I wanted to say is that your approach to spam is wrong.
Graylisting is good for now but will be obsolete when spammers will
learn the trick and make zombies into MTA-zombies (with own queue and
retry procedures). The best for future is a combination of RBL, p0f +
neural learning and some statistical (not behavioral like
Spamassassin) analysis.
Post by Marc Perkel
The concept that if one
server returns 4xx that it speaks for all servers is a concept that I
have never heard until yesterday in this forum and it only exists in the
qmail world.
So - I have to say that I'm confused and surprized by this and still
trying to comprehend why qmail behaves differently than every other MTA
in this regard.
As you may know by now qmail is not actively maintained by its author
since 1998.
--
Best regards,
Arkadiusz Dykiel
Pawel Panek
2006-12-01 14:48:58 UTC
Permalink
----- Original Message -----
Post by Arkadiusz Dykiel
(...)
All I wanted to say is that your approach to spam is wrong.
Graylisting is good for now but will be obsolete when spammers will
learn the trick and make zombies into MTA-zombies (with own queue and
retry procedures). The best for future is a combination of RBL, p0f +
neural learning and some statistical (not behavioral like
Spamassassin) analysis.
Do you use some implementation of neural nets to filter spam emails? If yes
could you point some more info about it?
Post by Arkadiusz Dykiel
(...)
--
Best regards,
Arkadiusz Dykiel
Regs,
Pawel Panek
Arkadiusz Dykiel
2006-12-01 16:13:27 UTC
Permalink
Hello Pawel,
Post by Pawel Panek
Do you use some implementation of neural nets to filter spam emails? If yes
could you point some more info about it?
http://aisk.tuxland.pl/
http://aisk.tuxland.pl/faq.html

I don't use it myself (breaks TLS) but find it extremely interesting.

The new test version of simscan uses p0f http://www.inter7.com/?page=simscan
--
Best regards,
Arkadiusz Dykiel
Ajai Khattri
2006-12-01 16:57:17 UTC
Permalink
Post by Arkadiusz Dykiel
http://aisk.tuxland.pl/
Kind of ironic - a see a lot of spam from TPNET these days :-)
--
Aj. (***@bitblit.net)
Marc Perkel
2006-11-25 00:43:42 UTC
Permalink
Post by Marc Perkel
One of my tricks to filter spam is a gray listing like trick that
detects suspicious hosts and returns a temp error on the lowest MX
number. Spammers often don't retry but real email servers would, in
theory, retry the next level up in the MX chain and the secondary server
will accept the email.
Servers that I do this with include servers with no or bad reverse
lookup, Host names with pattens that look like residential machines, and
servers listed in black lists that are not reliable enough to block, but
usually are spammers.
The idea being the but profiling these servers and returning a temp
error (421) on the lower MX that the good servers who would be a false
positive would retry to a different server that would accept it.
Having seen it before, I refer to this trick as the "I don't ever want
my e-mail trick". As you've discovered, that's the end result you get
from applying it. So long as you only want to get mail from servers
that mimic Sendmail's MX processing, it works. Otherwise, if the lowest
distance MX responds and tempfails, in essence saying "come back later",
then that's what qmail does. It's a feature, rather than a bug. If
that MX were not to respond at all, then qmail would contact the next
lowest distance MX.
- Ron
(who loved his MarxMenu to pieces, back in the day)
Thanks for being a MarxMenu fan.

I don't see it as mimicing Sendmail. I see it as following the spec. And
from what I can tell every other MTA follows the spec. If the lowest MX
says 4xx then you try the next one. That is the spec. The idea of having
the spec is failover. If I have 3 MX records and the lowest says "not
ready" then that's why I have the others.
Marc Perkel
2006-11-25 00:53:09 UTC
Permalink
Suppose the lowest MX had more than one IP address associated with that
name. Would Qmail try all the IPs of the lowest MX if it got a 4xx error
on one of them?
Continue reading on narkive:
Loading...