Advanced static routing
Using iproute2 to set up source based routing is actually pretty simple... see: http://www.lartc.org/howto/lartc.rpdb.html#LARTC.RPDB.SIMPLE
I used this tonight, so that traffic on eth3 can go out eth8, while normal traffic goes out eth0.
Source policy routing with OpenVPN
Source policy routing enabled the shrewd network administrator to do some really power things: extruded networking, route some packets (eg TCP) one direction and other packets (UDP) in another, etc.
Here's how I've configured my home firewall to encrypt all connections to a server (a.b.c.d) in a co-location facility. Essentially, anything packets from the LAN (192.168.5.0/24) are put into a new routing table (as defined in /etc/iproute2/rt_tables). This new table contains the special routes:
ip rule add from 192.168.5.0/24 to a.b.c.d/32 table nyet ip route add table nyet default via 10.10.10.5 ip route add 10.10.10.5 dev tun2 table nyet ip route add a.b.c.d/32 via 10.10.10.5 table nyet
In this particular case, any workstation connecting to a.b.c.d (resolved via standard public DNS) will be sent through an OpenVPN tunnel. Yet, the encrypted VPN traffic will not be sent through the VPN tunnel, since OpenVPN itself will still use the main route table.
I think this is a pretty nifty way of forcing security. So even if an email client isn't configured to use SSL for IMAP/HTTP/SMTP, all traffic will be encrypted anyhow.
I've done a whole bunch of tests with this, and latency is not noticably affected. Another neat aspect is that LZO compression can be enabled for encreased throughput...
An even tricker variation of this is to route Internet bound traffic over the VPN tunnel.
GRE Tunnels
Here's a small crib note on how to set up GRE tunnels on Linux. It's quite simple, actually - and compatible with Cisco GRE tunnels:
ip tunnel add gre_black mode gre remote a.b.c.d ip link set gre_black up ip addr add 10.5.5.1/30 dev gre_black
Once the GRE device has an IP address, you can then statically route (or use OSPF/RIP/BGP) any network over it. Kind of like a VPN tunnel, except less overhead and CPU cycles required.
See also Networking, Cisco