Multiwan connections addendum

After some testing and some thought, there are several things I felt I should also document regarding the multiwan connection solution I posted earlier.

Turn rp_filter OFF

It appears that rp_filter causes problems with NAT and connection marking. From what I could tell, packets tend to 'lose' their mark and thus get routed out the wrong interface without this turned off. However, it should be noted that if you turn this off, you need to take care of the anti-spoofing functionality it provides in your firewall script.

To turn it off, I edited this file: /etc/init.d/networking; and set it to echo "0" to /proc/sys/net/ipv4/conf/*/rp_filter instead of "1". Don't forget to change the text in the function spoofprotect() to say "NOT setting up IP spoofing protection".

Traffic destined for the router itself (TCP, ICMP and otherwise)

The posted solution also will not route packets to the router itself correctly. It may not be entirely noticable as some traffic may loop around (come in one interface, exit another). To solve this, add two additional rules to the routing policy database:

ip rule add from {ip_wan1_interface_on_router} table wan_one
ip rule add from {ip_wan2_interface_on_router} table wan_two

Optimizations to the MARK rules

Here are some minor optimizations to the MARK rules which make them more specific to what we are marking. They are not strictly necessary but may provide some more insight into someone just looking at your mangle table and trying to figure out what is going on.

instead of:
iptables -t mangle -A PREROUTING -i eth1 -j MARK --set-mark 1

use:
iptables -t mangle -A PREROUTING -i eth1 -m state --state NEW -m mark --mark 0 -j MARK --set-mark 1

instead of:
iptables -t mangle -A PREROUTING -i eth2 -j MARK --set-mark 2

use:
iptables -t mangle -A PREROUTING -i eth2 -m state --state NEW -m mark --mark 0 -j MARK --set-mark 2

Empirical tests indicate that packets that are not explicitly marked otherwise have a mark of 0.