Quite recently, I was in charge of running the temporary network at the Norwegian Wood festival at Frognerparken, Oslo, and it did require some constructive configuration to pull it off.
To start off, the Internet connection was bad. This was mainly due to the location which was too far from the main connection point, and as such, the speed of the Internet lines were very poor. We ended up using two ADSL lines of 12Mbps/0.6Mbps (D/U) and one SHDSL line of 4Mbps/4Mbps. This configuration was both bad and good at the same time because we had systems which required stabile lines with little to no other traffic sharing the same line. We could then dedicate one of the lines to these systems while we shared the rest of the lines for general Internet usage.
I’ll start off explaining the Internet connection setup and the hardware we had available.
- A Dell OptiPlex GX620
- A number of Cisco SB200 8port managed switches
- A number of Mikrotik RB750GL routers
- A number of Mikrotik SXT wireless routers
- A number of wireless APs
- Two Thompson ADSL modems
- One Cisco SHDSL modem
The network was based on a core gigabit trunk backbone running on the Cisco and the Mikrotik switches and routers. Both the two ADSL modems and the Cisco SHDSL modem got its own VLAN which were trunked into the server enabling the server to access the Internet through all three available paths.
After a discussion on how to build the client networks (the wireless and wired networks going to different parts of the organization), we ended up on splitting each of the different parts into their own subnets. This was mainly because we wanted to have the option of controlling the path each of these subnets chose when accessing the Internet as well as controlling whether the subnet should be forwared directly onto the Internet or if it was routed through the transparent squid proxy.
Each of the different subnets were assigned their own VLAN and trunked through the entire core network which enabled us to access each of these subnets anywhere on site.
We did set up a wireless link using the Mikrotik SXT devices, which I must confess, is a couple totally awsome devices! These devices were used as point-to-point links which trunked all the VLANs effectively replacing a wired backbone trunk. We have successfully tested these devices over a distance of 5.1km getting throughput exceeding 85Mpbs of simplex traffic, and a sustained throughput of 65Mbps full duplex communication. In our setup, the line-of-sight distance were approx. 150m which gave us a simplex throughput of 100Mbps line speed (only limited by the SXTs 100Mbps ethernet port).
Routing and server setup
Now, time for the interresting parts – network routing and configuration!
I’ve already mentioned the server (the Dell OptiPlex GX620) getting all the Internet lines tagged from the ADSL and SHDSL modems. These were configured to use the following subnets and VLANs.
- ADSL #1 : 10.10.0.1/24 VLAN 10
- ADSL #2 : 10.20.0.1/24 VLAN 20
- SHDSL : 192.168.1.1/24 VLAN 30
The server had its interfaces set up as follows:
# eth0.10: Transit: ADSL1
iface eth0.10 inet static
# eth0.20: Transit: ADSL2
iface eth0.20 inet static
# eth0.30: Transit: SHDSL
iface eth0.30 inet static
The reason why the SHDSL subnet was 192.168.1.0/24 is because the Cisco router was configured this way, and I did not see any reason why I had to change it, and because I did not have the login credentials for the device itself. It did not matter as I changed the subnets of both of the ADSL modems so the subnets would not overlap. The original plan was to change the SHDSL subnet to 10.30.0.0/24, but it was not that important as to spend much time and energy setting it up.
After configuring the Internet trunks, we configured the server with three additional routing tables, 10 adsl1, 20 adsl2 and 30 shdsl. This were done using the iproute utility, changing the rt_tables file to look like this:
# file: /etc/iproute/rt_tables
# reserved values
By having different routing tables for each of the Internet links, we were able to control where to route external traffic based which subnet the client was assigned to.
Next up were the NAT rules on the server. As the server had three possible ways of accessing the Internet, we had to create rules which NATed the traffic out on the different links. This were accomplished using the masquerade feature of iptables and the rules looked like this:
iptables -t nat -A POSTROUTING -o eth0.10 -j MASQUERADE
iptables -t nat -A POSTROUTING -o eth0.20 -j MASQUERADE
iptables -t nat -A POSTROUTING -o eth0.30 -j MASQUERADE
By now, we need to set up the different subnets and VLANs for the client networks. This is easy using /etc/network/interfaces and adding the following:
# eth0.100: Cust: nw12-area0
iface eth0.100 inet static
# eth0.101: Cust: nw12-area1
iface eth0.101 inet static
# eth0.102: Cust: nw12-area2
iface eth0.102 inet static
# eth0.103: Cust: nw12-area3
iface eth0.103 inet static
# eth0.104: Cust: nw12-area4
iface eth0.104 inet static
# eth0.105: Cust: nw12-area5
iface eth0.105 inet static
# eth0.106: Cust: nw12-area6
iface eth0.106 inet static
# eth0.107: Cust: nw12-area7
iface eth0.107 inet static
The #eth0.10x: Cust: <string> is for letting observium know the port description such as to use the buildt-in features. The following page explains how to enable your Linux server to define interface comments – Observium interface ifAlias
As you can see we had a number of client networks, some for internal use, and some for external clients requiring Internet connectivity through our uplinks.
For DNS we installed Bind9 and set it up as a local caching DNS server and set up ISCs DHCP server for each of the configured subnets.
Next thing to do is to configure the next-hop rules for ip route. Since we have three outgoing links to choose from, we use the ip routes ability to define multiple default gateways and use the kernels ability to weight these next hop definitions.
ip route add default scope global \
nexthop via 10.10.0.1 dev eth0.10 weight 2 \
nexthop via 10.20.0.1 dev eth0.20 weight 2 \
nexthop via 192.168.1.1 dev eth0.30 weight 1
The default gateway as set above uses all the available outgoing lines. The weighting states that two out of five outgoing connections should be routed through ADSL#1, two through ADSL#2 and the last through the SHDSL link. One may adjust these values, but although the connections will be weight-multiplexed over all the outgoing links, you have no control over the throughput each of the links will be exposed to (more on this later).
By now, after enabling ip forwarding in the kernel, your clients should be NATed out over all the available links and you should be able to utilize all the available bandwitdh at your disposal.
The next post will contain information on how to set up squid3 as a transparent proxy, how to decide which outgoing links the transparent proxy will use, how to force one client subnet out over a specific link and how to do traffic shaping over a specific VLAN using tc qdisc.