Sunday, May 14, 2006

That's Neat!

In addition to working with network and systems, I am also an amateur photographer. For the longest time, I was an unconvinced film addict and stuck to my film based equipment as the picture quality is undeniably better than what most digital cameras can produce particularly at higher ISOs. However, with a recent discovery, this perception is starting to change.

I read in a digital photography book recently about a noise-reduction filter/program called NeatImage. When I tried out the program, and applied its noise reduction algorithms to my digital photographs, I was amazed. Pictures I used to consider second-rate because of the noise could now be filtered to look so much better. See the examples they have on their Web site.

Give it a try. The demo version allows you to process an unlimited number of images, one image at a time, provided you are ok with the output being lossy jpg. That isn't so bad if you do the noise reduction part as your last step in your workflow. For a nominal fee, you can purchase Home and Professional versions of the program which don't have these limitations.

Monday, May 08, 2006

Multiwan connections addendum

After some testing and some thought, there are several things I felt I should also document regarding the multiwan connection solution I posted earlier.

Turn rp_filter OFF

It appears that rp_filter causes problems with NAT and connection marking. From what I could tell, packets tend to 'lose' their mark and thus get routed out the wrong interface without this turned off. However, it should be noted that if you turn this off, you need to take care of the anti-spoofing functionality it provides in your firewall script.

To turn it off, I edited this file: /etc/init.d/networking; and set it to echo "0" to /proc/sys/net/ipv4/conf/*/rp_filter instead of "1". Don't forget to change the text in the function spoofprotect() to say "NOT setting up IP spoofing protection".

Traffic destined for the router itself (TCP, ICMP and otherwise)

The posted solution also will not route packets to the router itself correctly. It may not be entirely noticable as some traffic may loop around (come in one interface, exit another). To solve this, add two additional rules to the routing policy database:

ip rule add from {ip_wan1_interface_on_router} table wan_one
ip rule add from {ip_wan2_interface_on_router} table wan_two

Optimizations to the MARK rules

Here are some minor optimizations to the MARK rules which make them more specific to what we are marking. They are not strictly necessary but may provide some more insight into someone just looking at your mangle table and trying to figure out what is going on.

instead of:
iptables -t mangle -A PREROUTING -i eth1 -j MARK --set-mark 1

use:
iptables -t mangle -A PREROUTING -i eth1 -m state --state NEW -m mark --mark 0 -j MARK --set-mark 1

instead of:
iptables -t mangle -A PREROUTING -i eth2 -j MARK --set-mark 2

use:
iptables -t mangle -A PREROUTING -i eth2 -m state --state NEW -m mark --mark 0 -j MARK --set-mark 2

Empirical tests indicate that packets that are not explicitly marked otherwise have a mark of 0.


Friday, May 05, 2006

Conning the Mark: Multiwan connections using IPTables, MARK, CONNMARK and iproute2

Over the past few months, I have been configuring a replacement multi-wan NAT router/firewall for work. My collegues and I decided to use Voyage Linux (a derivative of Debian Linux for embedded devices) on a Soekris net4801 box. See also the pictures on my coworker's (cyboc) blog.

Unlike other organizations who use their multi wan connections to do automatic load balancing, and traffic shaping, we simply use our extra WAN connection for redundancy. Both connections DNAT to an internal server with two distinct external IP addresses. The idea is that users can access the server using either of the IP addresses, though they might normally prefer one over the other. Users would be able to switch to the other connection should the one they were using provide a less than optimal result. No automatic load balancing is required nor desired. In simple terms, our network looks something like this:


As part of this configuration, we wanted to have network traffic that came in on one interface properly exit again through that same interface. I was able to configure most of the firewall and NAT parts of the router with relative ease using iptables but was stumped when it came to the routing table and how to route packets in and out of their own respective interfaces.

The Problem Defined

Traditional routing tables generally only allow for one default gateway at a time. Multiple default gateways have to be specified in priority sequence. Thus, there is no guarantee that an incoming packet on one line will receive a reply routed back through that same interface. At best, the return packet will go out the default gateway or some other static gateway defined according to the routing table. Furthermore, traditional routing tables only allow destination based routing. That is, we can create specific routing entries to dictate a route given a destination address but not based on the source address.

Enter IPROUTE2

After some research, I discovered that IPROUTE2 solves a lot of my problems. IPRoute2, amongst many other things, allows for source based routing, and also allows for routing based on packet markers. More on packet markers later.

Be warned though, IPROUTE2 is a rather complex beast! The user manual is far from friendly, and it took me a few tries to get it to do what I wanted to do.

Attempt 1: Source based routing

My first attempt at my problem involved source-based routing: Since most of the time, users will be using WAN connection #1 for this server, route all traffic originating from the IP address of my server out WAN connection 1. This works, however, it requires a manual change to the routing table when WAN connection 1 goes down. An administrator would have to switch the source based routing rule to now say route all traffic originating from server IP address out WAN connection #2.

Wouldn't it be simpler if the router could somehow just remember what connection the packet came in and route subsequent replies through that same interface?

Attempt 2: Packet marking based routing

My second attempt at my problem centered around being able to track connections and routing accordingly. To do this, I discovered iptables' packet marking and connection marking.

In short, iptables has two types of targets that one can use to mark packets: CONNMARK and MARK. CONNMARK marks a connection. Once marked, packets in the same "conversation" are also marked with the same CONNMARK indicator.

Another marker is the packet marker denoted by iptables' MARK target. (Couldn't they have come up with better names?!) The MARK target only marks individual packets. They are not resilient like the connmark indicators - i.e. they only retain their value for the duration of that one packet's lifespan.

Now when I first went diving into this, I erroneously thought that one could simply set the CONNMARK when a packet came in one WAN line, and have the routing tables detect that connmark and route accordingly. As I soon discovered though, iproute2 only recognizes packet MARKs not CONNMARKs. Thus, to do what I wanted, the CONNMARK value had to be copied to the MARK value each time a packet was about to be routed.

Solution Part 1: Configuring the mangle table in iptables

Given the above restrictions with CONNMARK and MARK, I devised in plain English the steps I want my router to take when marking packets and when routing.

  • If this is the first packet in a connection (i.e. it doesn't have a CONNMARK nor a MARK) then, set the MARK of the packet to 1 or 2 depending on which line it came in. Save this MARK to the CONNMARK value and accept the packet for routing.
  • If, however, a CONNMARK does exist, then restore that CONNMARK to the MARK value. Check to see what the MARK value is. If it is 1 or 2, then ACCEPT the packet for routing.
Once the packet is accepted for routing, route basis these rules:

  • If the packet has a MARK value of 1 then use the routing table for WAN connection #1.
  • Else if the packet has a MARK value of 2, then use the routing table for WAN connection #2.

Now that you understand the English algorithm, I will translate it into pseudocode in the same order in which it must appear in iptables' mangle table:

  • Restore the packet's CONNMARK to the MARK. (If one doesn't exist, then no mark is set.)
  • If packet MARK is 1, then it means that there is already a connection mark and the original packet came in on WAN #1, so ACCEPT.
  • Else, we need to mark the packet. If the packet is incoming on eth1 then set MARK to 1
  • If packet MARK is 2, then it means there is already a connection mark and the original packet came in on WAN #2, so ACCEPT.
  • Else, we need to mark the packet. If the packet is incoming on eth2 then set MARK to 2
  • Save MARK to CONNMARK. This rule will be hit only if the previous rules (2, and 4) did not match. A new mark would have been written according to rules (3 and 5) and it is saved here to the connection mark indicator.

Finally, the actual iptables commands:
iptables -A PREROUTING -t mangle -j CONNMARK --restore-mark
iptables -A PREROUTING -t mangle --match mark --mark 1 -j ACCEPT
iptables -A PREROUTING -t mangle -i eth1 -j MARK --set-mark 1
iptables -A PREROUTING -t mangle --match mark --mark 2 -j ACCEPT
iptables -A PREROUTING -t mangle -i eth2 -j MARK --set-mark 2
iptables -A PREROUTING -t mangle -j CONNMARK --save-mark

Solution Part 2: Configuring iproute2 to route according to the packet markers

Now that the connection and packets are marked as they come in, we need to instruct the routing table to route according to the markers on each packet. This is done using the Routing Policy database available in iproute2. In essence, this database defines a bunch of rules which when matched, ask the router to consider specific routing tables rather than the default routing table. In this way, we can define specific rules that say when the packet has a marker value of say "1", use wan_one routing table. Similarly if the packet has marker value of "2", use the wan_two routing table.

Several things need to be done in order to put all this together:

1. Modify the file /etc/iproute2/rt_tables.
2. Add two custom tables at the bottom of the file. Number the table numbers similar to your packet marker numbers for simplicity.
myrouter:/etc/iproute2# more rt_tables
#
# reserved values
#
255 local
254 main
253 default
0 unspec
#
# local
#
1 wan_one
2 wan_two

3. Define each routing table (wan_one and wan_two) by specifying rules specific to that connection. Note, however, that you must also specify rules that dictate how other packets will behave as well (notably packets destined for the local LAN). This is because once in the special routing table, the routing process does not consult your default routing table anymore. This is what I have in my two routing tables:
myrouter:/etc/iproute2# ip route show table wan_one
172.16.1.0/24 dev eth0 scope link
default via 149.99.251.145 dev eth1

myrouter:/etc/iproute2# ip route show table wan_two
172.16.1.0/24 dev eth0 scope link
default via 66.119.160.1 dev eth2

These are the commands I entered to get the routing tables above:
ip route add 172.16.1.0/24 dev eth0 table wan_one
ip route add default via 149.99.251.145 dev eth1 table wan_one

ip route add 172.16.1.0/24 dev eth0 table wan_two
ip route add default via 66.119.160.1 dev eth2 table wan_two

4. Next, you must define the iproute2 rules that will tell iproute2 to use the special routing tables. Do this by issuing the following commands:
ip rule add fwmark 1 table wan_one prio 1024
ip rule add fwmark 2 table wan_two prio 1025

Note: the prio (priority) numbers are simply there to ensure that they get placed in the right order and relatively near the top of the rules. You may need to adjust this number if you have other rules in your policy database.

You can verify that the rules were entered correctly by issuing an ip rule show command.
myrouter:/usr/local/sbin# ip rule show
0: from all lookup local
1024: from all fwmark 0x1 lookup wan_one
1025: from all fwmark 0x2 lookup wan_two
32766: from all lookup main
32767: from all lookup default

5. Add a default gateway to the default routing table to define the default path unmarked packets must take.

Conclusion

You're done! Packets now coming in wan connection one should be marked with 1, which then get routed according to table wan_one. Similarly for wan_two.

A few interesting notes in addition:
  • I have not described here any of the firewalling or nat processes. Obviously you need to have these setup and tested correctly before doing the CONNMARKing and MARKing.
  • Packets originating from inside the LAN will not receive a connection mark at first, and thus will fall through to the default routing table. They will route out the default gateway specified there. However, the first ack packet and every subsequent related packet should receive a connection mark, and follow one of the special routing tables.
  • Because of this peculiar behaviour for packets originating from inside the LAN, and because of the nature of network address translation, it is necessary to explicitly state the ISP's gateway in each of the default rules in the special tables. In other words, it is not enough to simply put "ip route add default dev eth2 table wan_two". Instead, this should be issued: "ip route add default via 66.119.160.1 dev eth2 table wan_two".
  • Debugging the above solution can be a bit of a pain. I found that the iptables (mangling) part of the whole exercise can be done relatively easily through logging and the "iptables -L --line-numbers -n -v -t mangle" command, but there is no equivalent functionality in iproute2. This, probably more than anything caused more grief when things weren't working than anything else.
  • I have posted an addendum to this article which includes a few important details left out in this article.

Thursday, May 04, 2006

What's in a name? Welcome to my technical blog.

Many years ago (1999), long before the age of blogs, I was asked to write a column for an online e-zine called Hello World. Although the magazine never survived more than a few pilot issues, I was pleasantly surprised, while doing a google search, to find my articles still hanging around. I've always been meaning to resurrect my column, and I think that now, during the age of blogs, is possibly the right time.

Though a bit dated, I have provided links to the old articles for continuity sake.
By the way, in case you are wondering where the name "Developing Rapids" comes from, it is a play on the late 90s cliche "Rapid Development". It is also the title of my first article.