Basic rate limit filter

[Georgi Petrov]

Hello all,

Recently milter-greylist was added to Virtualmin GPL as a mail rate
limiting option. The default setting is pretty basic and not satisfactory.
Therefore I will try to propose more advanced settings/setup.

Milter-greylist is of course way more capable than just limiting the rate
of the emails sent, but as this is one really important feature lets start
with it.

Rate limiting is important because:


- brute foreced/hijacked mail accounts can be used for sending huge
amount of spam. I've seen something like 100K emails sent in a night. Of
course the IP gets listed as spammer immediately. Rate limiting can slow
this down. The admin will have time to react (possibly the admin will be
warned).
- in a shared environment unfair users can send spam using their own
accounts
- websites get hacked and scripts that send spam can be installed there
- there are certain DoS attacks that can block the email service. For
example huge amount of incoming emails to existing users, containing virus.
The virus protection (if any) will block them, but the whole service can
slow down or completely become blocked.


What we want to achieve with our basic rate limiting:


- mail sent from localhost (via PHP, sendmail or webmail) should be
limited using the user as key. We should block the user and not the IP as
if we block the IP (127.0.0.1), nobody will be able to send anymore. We
should probably use the "from" field, as there is no sasl authentication
for localhost, but scamming the "from" field is not possible (I should
double check this).
- mails sent via email clients (Thunderbird etc.) should be limited by
the IP or the user. If we use the user it is probably better to use the
sasl authentication user, not the "from" field, as it can be scammed. If we
use the IP this can be combined with the next rule, but it will also block
multiple users using the same IP, one of which is sending too many emails.
- mail coming from other mail servers for local users should be limited
by the IP. This is for DoS prevention only, the limit here should be high.
- all users/IPs should be greylisted if they go over the ratelimit as we
want them to be able to send emails after some time and not blacklist them
forever.


Here is the original Virtualmin ratelimit rule, which is obviously not good
at all (the actual number of emails is controlled by the user via the
webinterface):

ratelimit "virtualmin_limit" rcpt 30 / 30m
racl blacklist from /.*/ ratelimit "virtualmin_limit" msg "Message quota
exceeded"
racl whitelist default

If somebody sends too many emails from localhost, it gets blacklisted and
nobody can send anymore.

Here is my modified version:

ratelimit "virtualmin_limit" rcpt 30 / 30m key "%f"
racl greylist from /.*/ ratelimit "virtualmin_limit" delay 31m autowhite 0m
msg "Message quota exceeded"
racl whitelist default

It works as expected limiting per user. If somebody sends too many emails
from PHP all the emails wait in the queue until the user drops below the
limit. The drawback is that it does not treat local and non local email
differently. So it needs improvement.

Here are my specific questions:

- How I can achieve two different rate limit rules for localhost and non
localhost (I have tried with some rules but they didn't work)?
- Is there a better way to limit the emails from PHP? They cannot be
rejected, so they go in the mail queue and wait there. The queue tries to
resend them every few minutes and they are rejected again until the rate
drops under the limit.
- Can we achieve some sort admin notification if the ratelimit is blocking
somebody? Now I can have warning notifications if the mail queue gets very
big.

Thanks a lot for your ideas, comments and time!

Georgi

[manu-S783fYmB3Ccdnm+]

Post by Georgi Petrov
- How I can achieve two different rate limit rules for localhost and non
localhost (I have tried with some rules but they didn't work)?

I guess you used an ACL whith clause addr 127.0.0.1, but perhaps the
message was sent from ::1 (IPv6) ?

Post by Georgi Petrov
- Is there a better way to limit the emails from PHP? They cannot be
rejected, so they go in the mail queue and wait there. The queue tries to
resend them every few minutes and they are rejected again until the rate
drops under the limit.

You need your PHP application to talk SMTP instead of using the system
mail/sendmail/whatever command.

Post by Georgi Petrov
- Can we achieve some sort admin notification if the ratelimit is blocking
somebody? Now I can have warning notifications if the mail queue gets very
big.

You could add a urlcheck clause at the end of your racl blacklist ACL:
if ratelimit does not match it is not evaluated, and if ratelimit
matches, the evaluation continue to the urlcheck clause.

Your urlcheck clause will always evaluate to true but it will send the
signal to a webservice where you can do whatever you want.

An alternative is to log the even, and have another process parsing the
logs.

[Georgi Petrov]

Hello Emmanuel,

Thanks a lot for your comments!

Here is what I did try (the limits are very low to let me test without
sending too many emails):

ratelimit "virtualmin_limit_local" rcpt 3 / 10m key "%f"
racl greylist addr 127.0.0.1 ratelimit "virtualmin_limit_local" delay 11m
autowhite 0m msg "Message quota exceeded"

ratelimit "virtualmin_limit" rcpt 3 / 10m key "%i"
racl greylist from /.*/ ratelimit "virtualmin_limit" delay 11m autowhite 0m
msg "Message quota exceeded"

racl whitelist default

The first rate limit rule works fine (the emails indeed come from
127.0.0.1). The problem is that the local emails also hit the second
ruleset and eventually 127.0.0.1 gets greylisted if the combined amount of
local emails is over the limit. The second ruleset should work for
everything except 127.0.0.1 but I didn't find a way to invert the logic
there. (Also if some message hits one rule it still propagates down so it
can also hit another, the logic is not like in iptables for example). Maybe
I can use the regular expression syntax to invert the logic, but in this
case I need to use the actual full localhost name which is not as
convenient as the IP. Can you advise me on that?


Yes, if the PHP application is able to talk SMTP then it can be much
easier, but only few can so we need to keep the mail(); function working. I
should investigate if it is possible to fake the "FROM" if you indeed use a
PHP appllication that comes from 127.0.0.1 and talks SMTP. In this case we
should use the SASL authentication user and create one more rate limit rule.

When adding more keys, for example

ratelimit "virtualmin_limit_local" rcpt 3 / 10m key "%f%i"

is the logic AND or is it OR? From what I see it is just string comparison
so the logic is AND. Is there a way to achieve logic OR? For example either
the first value is the same or the second? The key value, can it take
regular expression?

The admin notification is better if we log the rate_limit event then other
scripts will parse the log under some cron schedule and send proper
warnings. I think this one is already available in Virtualmin/Webmin

Thanks again for your time!

Georgi

Post by manu-S783fYmB3Ccdnm+

Post by Georgi Petrov
- How I can achieve two different rate limit rules for localhost and non
localhost (I have tried with some rules but they didn't work)?

I guess you used an ACL whith clause addr 127.0.0.1, but perhaps the
message was sent from ::1 (IPv6) ?

Post by Georgi Petrov
- Is there a better way to limit the emails from PHP? They cannot be
rejected, so they go in the mail queue and wait there. The queue tries to
resend them every few minutes and they are rejected again until the rate
drops under the limit.

You need your PHP application to talk SMTP instead of using the system
mail/sendmail/whatever command.

Post by Georgi Petrov
- Can we achieve some sort admin notification if the ratelimit is

blocking

Post by Georgi Petrov
somebody? Now I can have warning notifications if the mail queue gets

very

Post by Georgi Petrov
big.

if ratelimit does not match it is not evaluated, and if ratelimit
matches, the evaluation continue to the urlcheck clause.
Your urlcheck clause will always evaluate to true but it will send the
signal to a webservice where you can do whatever you want.
An alternative is to log the even, and have another process parsing the
logs.
--
Emmanuel Dreyfus
http://hcpnet.free.fr/pubz

[Emmanuel Dreyfus]

Post by Georgi Petrov
The first rate limit rule works fine (the emails indeed come from
127.0.0.1). The problem is that the local emails also hit the second
ruleset and eventually 127.0.0.1 gets greylisted if the combined amount of
local emails is over the limit. The second ruleset should work for
everything except 127.0.0.1 but I didn't find a way to invert the logic
there.

have a not addr 127.0.0.1 clause in the second ACL, or add a
"racl whitelist addr 127.0.0.1" between the two ACLs.

Post by Georgi Petrov
ratelimit "virtualmin_limit_local" rcpt 3 / 10m key "%f%i"
is the logic AND or is it OR? From what I see it is just string comparison
so the logic is AND. Is there a way to achieve logic OR? For example either
the first value is the same or the second? The key value, can it take
regular expression?

Define two ratelimit and use two ACL, that makesa OR.

[Georgi Petrov]

Hello Emmanuel,

Thanks a lot for your advice! It indeed worked. The rules are now:

ratelimit "virtualmin_limit_local" rcpt 3 / 10m key "%f"
racl greylist addr 127.0.0.1 ratelimit "virtualmin_limit_local" delay 11m
autowhite 0m msg "Message quota exceeded"

ratelimit "virtualmin_limit" rcpt 3 / 10m key "%i"
racl greylist not addr 127.0.0.1 ratelimit "virtualmin_limit" delay 11m
autowhite 0m msg "Message quota exceeded"

racl whitelist default

I think it should be fine like this.

I will try to read a little more and invent some way of automated outgoing
spam checking and blacklisting.

Thanks again!

Georgi

Post by Emmanuel Dreyfus

Post by Georgi Petrov
The first rate limit rule works fine (the emails indeed come from
127.0.0.1). The problem is that the local emails also hit the second
ruleset and eventually 127.0.0.1 gets greylisted if the combined amount

of

Post by Georgi Petrov
local emails is over the limit. The second ruleset should work for
everything except 127.0.0.1 but I didn't find a way to invert the logic
there.

have a not addr 127.0.0.1 clause in the second ACL, or add a
"racl whitelist addr 127.0.0.1" between the two ACLs.

Post by Georgi Petrov
ratelimit "virtualmin_limit_local" rcpt 3 / 10m key "%f%i"
is the logic AND or is it OR? From what I see it is just string

comparison

Post by Georgi Petrov
so the logic is AND. Is there a way to achieve logic OR? For example

either

Post by Georgi Petrov
the first value is the same or the second? The key value, can it take
regular expression?

Define two ratelimit and use two ACL, that makesa OR.
--
Emmanuel Dreyfus