Ubuntu 12.04 Xen DomU – failed to run bootloader: -3

Got this error after upgrading the kernel on a Ubuntu 12.04 virtual machine the other day:

# xl create machine.cfg 
Parsing config file machine.cfg
failed to run bootloader: -3

The reason for this is that ubuntu always keeps the old kernel when upgrading to a new. This creates two entries in grub.cfg which makes pygrub go: “WHAT? I’m not able to handle this!”.

Luckily it is an easy fix.

There are a few steps to this, all which are easy to get through.

  • First, edit your DomU’s config file in /etc/xen/machine.cfg
    • Comment the bootloader = “pygrub” line
    • Add the lines kernel, ramdisk and extra. Example is below (adjust the kernel version accordingly):

name = "machine"

memory = 4096

disk = ['phy:/dev/xen/machine,xvda,w']

vif = ['']

#bootloader = "pygrub"

vcpus = 4

kernel = "/boot/vmlinuz-3.2.0-30-generic"
ramdisk = "/boot/initrd.img-3.2.0-30-generic"
extra = "root=/dev/xvda1"

  • This enables you to boot the machine using Dom0’s kernel.
  • Once booted, you need to remove the old kernel. I think apt-get autoremove does the trick, if not, you need to remove the previous kernel using dpkg.
  • After you have removed the old kernel, run update-grub and make sure it only finds one kernel
  • Power off the machine and revert the config file to the pygrub configuration (remove kernel, ramdisk and extra)
  • Boot your machine and grab a coffee!

EFI dualboot Ubuntu 12.04 and Windows 8 in Raid0 on Sony Vaio S

Recently I got my hands on a Sony Vaio S with the following specs:

  • Model: SVS13A1Y9ES
  • Specs: link

The machine has two 128GB SSD disks which can be configured in RAID0 using a built-in software raid-controller.

I wanted to run a dualboot setup with Windows 8 for gaming and Ubuntu Linux for everything else.

I tried and successfully installed and booted Ubuntu using BIOS Legacy mode, but failed repeatedly using EFI. After a couple of days of fooling around, I found a EFI setup which worked – WHY? Because I want to.

I’m writing this mainly because I am guaranteed to forget the procedure until the next time I try to do this. If it helps others stuck with the same problem it is just a bonus!

The main reason for not wanting to install Ubuntu and Windows to separate drives is because of this:

tormsl@eir:~$ dd if=/dev/zero of=testfile bs=1M count=20000
20000+0 records in
20000+0 records out
20971520000 bytes (21 GB) copied, 34.0132 s, 617 MB/s

I’ve just wrote this down as a list of steps, please comment on this post if there’s something wrong or you are having issues following the steps.

  • Enable RAID setup in BIOS and enter the configuration mode by pressing CTRL-I
  • Create RAID0 using both SSDs
  • Boot ubuntu live x64 and create new GPT partition table on /dev/mapper/isw_xxxx_Volume0 using gparted or similar (not fdisk)
  • Install Windows by selecting the entire empty raid set
  • Boot ubuntu live x64 and shrink the main ntfs volume used to install windows (/dev/mapper/isw_xxxx_Volume0p4)
  • Make sure Windows boots correctly
  • Install Ubuntu 12.04 x64 using the alternate install media
  • Create two new partitions (but do not change anything)
  • /dev/mapper/isw_xxxx_Volume0p5	linux-swap
    /dev/mapper/isw_xxxx_Volume0p6	ext4
  • Install Ubuntu like normal and when asked, install Grub to “/dev/mapper/isw_xxxx_Volume0”
  • After the install, your machine will not be able to boot as the installation of grub has destroyed the Windows EFI boot files
  • You now need to boot the Windows 8 install DVD which starts the install process. Cancel this process by clicking the X in the first installation window. As the Windows installer closes, it should leave you at the previously installed Windows login screen. Log into windows and start the “Automatic Repair” utility as described here.
  • Make sure Windows boots correctly without the CD-ROM after the automatic repair process
  • Boot ubuntu live x64. We are now going to make the EFI boot process load Grub instead of the Windows bootloader.
  • After the boot, install grub-efi and grub-efi-amd64 using apt-get
  • Mount the partition of where you installed Ubuntu in the previous step to /mnt and mount the EFI partition to /mnt/boot/efi
  • mount /dev/mapper/isw_xxxx_Volume0p6 /mnt
    mount /dev/mapper/isw_xxxx_Volume0p2 /mnt/boot/efi
  • Install grub
  • grub-install --root=/mnt /dev/mapper/isw_xxxx_Volume0
  • You should now be able to find the grub .efi binary in /mnt/boot/efi/EFI/ubuntu/grubx64.efi
  • We now need to rename the windows .efi bootfiles and replace these with the grub one.
  • sudo mv /mnt/boot/efi/EFI/Boot/bootx64.efi /mnt/boot/efi/EFI/Boot/bootx64.efi.old
    sudo mv /mnt/boot/efi/EFI/Microsoft/Boot/bootmgfw.efi /mnt/boot/efi/EFI/Microsoft/Boot/bootmgfw.efi.old
    sudo cp /mnt/boot/efi/EFI/ubuntu/grubx64.efi /mnt/boot/efi/EFI/Boot/bootx64.efi
    sudo cp /mnt/boot/efi/EFI/ubuntu/grubx64.efi /mnt/boot/efi/EFI/Microsoft/Boot/bootmgfw.efi
  • Unmount and reboot the machine without the ubuntu-live CDROM/USB-stick. Grub should now be the primary bootloader and you should be able to boot into ubuntu.
  • In ubuntu, we need to activate the Windows 8 menuentry in Grub.
  • Edit /etc/default/grub and comment the two lines containing the word HIDDEN
  • Now, we need to find the fs_uuid of the EFI partition in order to chainload the Windows bootloader. Run the command below which will give you the fs_uuid
  • grub-probe --target=fs_uuid /boot/efi/EFI/Microsoft/Boot/bootmgfw.efi.old
  • Open the file /etc/grub.d/40_custom and paste the following
  • menuentry "Windows 8" {
    	search --fs-uuid --no-floppy --set=root <fs_uuid from previous step>
    	chainloader (${root})/EFI/Microsoft/Boot/bootmgfw.efi.old
  • Run update-grub and reboot the machine.
  • You should now be able to boot into Windows using Grub.

load balancing multiple ISP connections using iproute and iptables

Quite recently, I was in charge of running the temporary network at the Norwegian Wood festival at Frognerparken, Oslo, and it did require some constructive configuration to pull it off.

To start off, the Internet connection was bad. This was mainly due to the location which was too far from the main connection point, and as such, the speed of the Internet lines were very poor. We ended up using two ADSL lines of 12Mbps/0.6Mbps (D/U) and one SHDSL line of 4Mbps/4Mbps. This configuration was both bad and good at the same time because we had systems which required stabile lines with little to no other traffic sharing the same line. We could then dedicate one of the lines to these systems while we shared the rest of the lines for general Internet usage.

I’ll start off explaining the Internet connection setup and the hardware we had available.


  • A Dell OptiPlex GX620
  • A number of Cisco SB200 8port managed switches
  • A number of Mikrotik RB750GL routers
  • A number of Mikrotik SXT wireless routers
  • A number of wireless APs
  • Two Thompson ADSL modems
  • One Cisco SHDSL modem

The network

The network was based on a core gigabit trunk backbone running on the Cisco and the Mikrotik switches and routers. Both the two ADSL modems and the Cisco SHDSL modem got its own VLAN which were trunked into the server enabling the server to access the Internet through all three available paths.

After a discussion on how to build the client networks (the wireless and wired networks going to different parts of the organization), we ended up on splitting each of the different parts into their own subnets. This was mainly because we wanted to have the option of controlling the path each of these subnets chose when accessing the Internet as well as controlling whether the subnet should be forwared directly onto the Internet or if it was routed through the transparent squid proxy.

Each of the different subnets were assigned their own VLAN and trunked through the entire core network which enabled us to access each of these subnets anywhere on site.

We did set up a wireless link using the Mikrotik SXT devices, which I must confess, is a couple totally awsome devices! These devices were used as point-to-point links which trunked all the VLANs effectively replacing a wired backbone trunk. We have successfully tested these devices over a distance of 5.1km getting throughput exceeding 85Mpbs of simplex traffic, and a sustained throughput of 65Mbps full duplex communication. In our setup, the line-of-sight distance were approx. 150m which gave us a simplex throughput of 100Mbps line speed (only limited by the SXTs 100Mbps ethernet port).

Routing and server setup

Now, time for the interresting parts – network routing and configuration!

I’ve already mentioned the server (the Dell OptiPlex GX620) getting all the Internet lines tagged from the ADSL and SHDSL modems. These were configured to use the following subnets and VLANs.

  • ADSL #1 : VLAN 10
  • ADSL #2 : VLAN 20
  • SHDSL : VLAN 30

The server had its interfaces set up as follows:

# eth0.10: Transit: ADSL1
auto eth0.10
iface eth0.10 inet static
# eth0.20: Transit: ADSL2
auto eth0.20
iface eth0.20 inet static
# eth0.30: Transit: SHDSL
auto eth0.30
iface eth0.30 inet static

The reason why the SHDSL subnet was is because the Cisco router was configured this way, and I did not see any reason why I had to change it, and because I did not have the login credentials for the device itself. It did not matter as I changed the subnets of both of the ADSL modems so the subnets would not overlap. The original plan was to change the SHDSL subnet to, but it was not that important as to spend much time and energy setting it up.

After configuring the Internet trunks, we configured the server with three additional routing tables, 10 adsl1, 20 adsl2 and 30 shdsl. This were done using the iproute utility, changing the rt_tables file to look like this:

# file: /etc/iproute/rt_tables
# reserved values
255 local
254 main
253 default
0 unspec
# local
#1 inr.ruhep
10 adsl1
20 adsl2
30 shdal

By having different routing tables for each of the Internet links, we were able to control where to route external traffic based which subnet the client was assigned to.

Next up were the NAT rules on the server. As the server had three possible ways of accessing the Internet, we had to create rules which NATed the traffic out on the different links. This were accomplished using the masquerade feature of iptables and the rules looked like this:

iptables -t nat -A POSTROUTING -o eth0.10 -j MASQUERADE
iptables -t nat -A POSTROUTING -o eth0.20 -j MASQUERADE
iptables -t nat -A POSTROUTING -o eth0.30 -j MASQUERADE

By now, we need to set up the different subnets and VLANs for the client networks. This is easy using /etc/network/interfaces and adding the following:

# eth0.100: Cust: nw12-area0
auto eth0.100
iface eth0.100 inet static
# eth0.101: Cust: nw12-area1
auto eth0.101
iface eth0.101 inet static
# eth0.102: Cust: nw12-area2
auto eth0.102
iface eth0.102 inet static
# eth0.103: Cust: nw12-area3
auto eth0.103
iface eth0.103 inet static
# eth0.104: Cust: nw12-area4
auto eth0.104
iface eth0.104 inet static
# eth0.105: Cust: nw12-area5
auto eth0.105
iface eth0.105 inet static
# eth0.106: Cust: nw12-area6
auto eth0.106
iface eth0.106 inet static
# eth0.107: Cust: nw12-area7
auto eth0.107
iface eth0.107 inet static

The #eth0.10x: Cust: <string> is for letting observium know the port description such as to use the buildt-in features. The following page explains how to enable your Linux server to define interface comments – Observium interface ifAlias

As you can see we had a number of client networks, some for internal use, and some for external clients requiring Internet connectivity through our uplinks.

For DNS we installed Bind9 and set it up as a local caching DNS server and set up ISCs DHCP server for each of the configured subnets.

Next thing to do is to configure the next-hop rules for ip route. Since we have three outgoing links to choose from, we use the ip routes ability to define multiple default gateways and use the kernels ability to weight these next hop definitions.

ip route add default scope global \
nexthop via dev eth0.10 weight 2 \
nexthop via dev eth0.20 weight 2 \
nexthop via dev eth0.30 weight 1

The default gateway as set above uses all the available outgoing lines. The weighting states that two out of five outgoing connections should be routed through ADSL#1, two through ADSL#2 and the last through the SHDSL link. One may adjust these values, but although the connections will be weight-multiplexed over all the outgoing links, you have no control over the throughput each of the links will be exposed to (more on this later).

By now, after enabling ip forwarding in the kernel, your clients should be NATed out over all the available links and you should be able to utilize all the available bandwitdh at your disposal.

The next post will contain information on how to set up squid3 as a transparent proxy, how to decide which outgoing links the transparent proxy will use, how to force one client subnet out over a specific link and how to do traffic shaping over a specific VLAN using tc qdisc.

defgw, iptables, iproute2 and NAT

Ok, not sure how to present this project, so I’ll dive right in.

Say you have a Linux box running openvpn which is responsible for routing several subnets over the VPN link. The VPN is setup such that the default gateway on the box are set to the other VPN endpoint. By doing this, all traffic are directed over the VPN and everybody’s happy.

Say you now want to direct traffic from one of the subnets to another gateway, and at the same time you’d like to NAT it.

Now, if you have one single gateway configured, this isn’t really an issue. A simple solutions would then be to NAT the traffic based on the source address of the packets with a simple iptables rule like this:
# iptables -t nat -A POSTROUTING -o eth0 -s -j MASQUERADE
Here we tell iptables to rewrite all packets going out on interface eth0 which has a source address of The MASQUERADE target will pick the external address on eth0 and rewrite the source address in the IP header with this address.

The problem with this solution is that the kernel will lookup the destination for the packet in the main routing table, which in our case is configured with a default route over the VPN link. In other words, we need some additional rules, both in iproute and iptables to make this work.

I have created a network diagram to visualize the setup in question.

Network Diagram

What we see from the above figure is the two VPN endpoints, one being our router, the other is the VPN provider. We have several subnets connected to eth1 and our connection to the ISP is through eth0. The VPN interface is tap0. The ISPs gateway is Our private subnet,, is going to be NAT’ed through the ISPs gateway, and not over the VPN like all the others.

To summarize, we are going to do this:

  1. Create an alternate routing table which has the ISP’s gateway as the default
  2. Mark packets with a source address of
  3. Write a rule which matches the mark and tells the kernel to look in the alternate routing table when deciding where to send the packet
  4. NAT marked packets

Lets dive into it!

First we create an alternate routing table. This is easily done by editing iproute2s rt_tables file.

# vim /etc/iproute2/rt_tables
Make this file look like this:

# reserved values
255 local
254 main
253 default
0 unspec
# local
#1 inr.ruhep
4 alternate

Save and close the file.

Now, it’s time to populate the table with the rules we need. In our case, we need atleast three rules, one being the default gateway, the other being the subnet on eth1 and the last being the subnet for eth0 (the ISP linknet).

# ip route add dev eth0 src table 4
# ip route add dev eth1 src table 4
# ip route add default via table 4

All right, now that this is done we can check out the results:
# ip route show table 4
default via dev eth0 dev eth0 scope link src dev eth1 scope link src

Lets add the iptables configuration. iptables needs two rules, one for the mangle chain which will mark packets arriving from before any routing decisions are made, and one for the nat chain which will SNAT the packets when leaving. We will mark the incomming packets with 0x4 which conveniently matches our routing table id.
# iptables -t mangle -A PREROUTING -s -j MARK --set-mark 0x4
# iptables -t nat -A POSTROUTING -m mark --mark 0x4 -j SNAT --to-source

We’re almost done. What is left is to chose the correct routing table based on the mark (0x4), which we set on the incomming packets from We do this by adding some routing rules in iproute2.
# ip rule add from table 4
# ip rule add fwmark 0x4 table 4

I’ve created a small diagram showing our packet path.

iptables flow diagram

The solid figures are the ones we utilize, and the dashed figures are the ones we don’t care about. This diagram does not show all available paths through netfilter/iptables, but it shows the ones we need to care about.
In the mangle/PREROUTING table, we mark the incomming packet. This mark is then matched in the first routing decision which will select our alternate routing table when making the next-hop decision. The mark is then used for the last time in the nat/POSTROUTING table.

And that should be it.

You should now be able to access the internet from NAT’ed through the ISPs gateway and not over the VPN.

Some comments…

  • I had to add the “ip rule add from table 4” to get this to work, don’t really know why…
  • Remember to enable ip forwarding
  • I read that if you have rp_filter enabled, this could mess up NAT’ing when marking packets

Please do comment or send me an mail if you are trying somthing similar and like to share your setup or are having difficulties using iproute2 or iptables.

VLAN Access Ports on MikroTik Routers

Coming from a Cisco environment, configuring access ports for VLANs are relatively simple. When I recently invested in a MikroTik RB1200 router, I was surprised that I could not find any way of simply setting one of the ethernet ports as an access-port for a configured VLAN. Whenever you configure a VLAN and attach it to an ethernet port, it leaves the device tagged. There was no configuration option which allowed me to configure a physical port to strip the VLAN tag before a packet leaves the device. But, after a few hours of intense searching and trial and error, I found a way!

It seems that you need to configure a bridge which you attach both the physical interface and the vlan interface as bridge-ports. I will show an example where I configure one port as a VLAN trunk (ether9), adding VLAN 4 and 10, and two access-ports for VLAN 4 (ether1,ether2) and two for VLAN 10 (ether3,ether4).

First, we configure the VLANs. In RouterOS, every VLAN has to belong to an interface, so we attach the VLANs to ether9.

/interface vlan add name=vlan4 interface=ether9 vlan-id=4
/interface vlan add name=vlan10 interface=ether9 vlan-id=10

Now that we have the VLANs, we can configure the bridges. We need two bridges, one for VLAN 4 and one for VLAN10

/interface bridge add name=br-vlan4
/interface bridge add name=br-vlan10

For each of these bridges, we add the physical interfaces as well as the VLAN we want to configure access-ports for.

/interface bridge port add bridge=br-vlan4 interface=ether1
/interface bridge port add bridge=br-vlan4 interface=ether2
/interface bridge port add bridge=br-vlan4 interface=vlan4
/interface bridge port add bridge=br-vlan10 interface=ether3
/interface bridge port add bridge=br-vlan10 interface=ether4
/interface bridge port add bridge=br-vlan10 interface=vlan10

As you see from above, we add ether1, ether2 and vlan4 to the br-vlan4 bridge, the same goes for vlan10. This will for all intents and purposes behave the exact way as Cisco’s way of saying:

# switchport mode access
# switchport access vlan 4

I must say, that this way of configuring access-ports seems a little strange at first, but when you think about it – RouterOS is basically a Linux Operating system below the configuration interface.

I hope this post helps newcomers like me enjoy their brand new RouterOS experience!

Securing your IPv6 enabled Linux desktop

I recently enabled my desktop computer with IPv6, and with it comes some security implications.

I use openvpn to tunnel my IPv6 subnet (/64) from my IPv6 enabled server hosted by Blix Solutions AS to provide IPv6 to my client machines. Now, I have my local IPv4 subnet NAT’ed with some good iptables rules so I don’t usually run iptables for IPv4 within my local subnet. With IPv6 though, I don’t want to do firewalling on the router, and thus will have to secure every machine locally. My router runs neighbor and router advertisement to the clients through the openvpn tunnel.

I run Ubuntu 10.10 on my desktop, and with it comes ip6tables. It behaves pretty much like the normal iptables, but has some extra features which comes with the IPv6 territory.

The only services I want to export globally is the ssh service and the icmp ech request, the rest will be dropped. I have created a set of rules which will enable the desktop computer to receive what it needs from the router (advertisements and solicitations), open the SSH service and drop the rest.

I created a file containing these rules in /etc/ip6tables.rules

-A INPUT -p ipv6-icmp -m icmp6 –icmpv6-type 128 -j ACCEPT
-A INPUT -p ipv6-icmp -m icmp6 –icmpv6-type 135 -j ACCEPT
-A INPUT -p ipv6-icmp -m icmp6 –icmpv6-type 136 -j ACCEPT
-A INPUT -p ipv6-icmp -m icmp6 –icmpv6-type 133 -j ACCEPT
-A INPUT -p ipv6-icmp -m icmp6 –icmpv6-type 134 -j ACCEPT
-A INPUT -p tcp -m tcp –dport 22 -j ACCEPT

To enable the script, you’ll need to run this command (as root, or sudo depending on whether or not you have made the file globally readable).
ip6tables-restore < /etc/ip6tables.rules Now, to run the script every time the network is enabled, you can use this trick: Create a file in /etc/network/if-pre-up.d/ named ip6tables-load and make it executable (chmod a+rx). In this file, paste the following: #!/bin/bash if [ -f /etc/ip6tables.rules ] ; then ip6tables-restore < /etc/ip6tables.rules fi Now, your desktop is secured even when IPv6 is enabled!

Fix Dell Vostro v130 Gigabit Ethernet in Ubuntu 10.10

If you, like me, have experienced some issues with the internal network interface shipped with the Dell Vostro v130 notebook, you should take a look at this post.

I noticed that my v130 seems to hang as network traffic on the internal gigabit ethernet interface increased. It seems that others were having the same issue.

I first tried updating the BIOS from version 3 to 4 (v4), and now I was able to do a PXE install from a local mirror. But when running a large file copy operation through the interface, the kernel locked up again.

I went on the web and found a new driver from the network card manufacturer, and tried installing this. This did not immediately fix the problem due to an error in the kernel shipped with Ubuntu 10.10, the kernel selects the wrong module while booting the OS. The kernel loads the r8169 module while the interface module from RealTek are named r8168.

What seems to have fixed the problem was to blacklist the r8169 module and tell the operating system to load the r8168 module instead.

If you need a detailed walk through, read on, if you have another solution, please post a comment for me to read.

— walk through —

Fetch the driver from RealTek on this (choose Linux driver in the UNIT section) page, and save it to a local folder.

Unpack the file

$ tar vjxf r8168-8.aaa.bb.tar.bz2
$ cd r8168-8.aaa.bb
$ sudo ./autorun.sh

The driver will now compile and install itself onto your system.

Next, we will blacklist the r8169 module so it doesn’t load during system boot. Edit /etc/modprobe.d/blacklist.conf and add the following line to the end of the file:

blacklist r8169

Now, to load the correct module during boot, edit the /etc/modules file and add the following to the end of the file:


Now, reboot your computer and when up, do a lsmod and check that the r8168 module is loaded.

If it is, you should be able to ifup eth0 and use your network interface without hangs or crashes – ENJOY!

Dedicated PulseAudio network sound server without X

This article will explain how you configure a Ubuntu server to act as a network sound server playing sounds for all configured machines on your network. It even has a small program which binds to your keyboard and allows you to control your sound servers output volume from any client.

Firstly, you’ll need pulseaudio. Get it from the repositories by apt-getting it

sudo apt-get install pulseaudio

As we don’t run X or any desktop environment on our sound-server, you’ll need to run pulseaudio in system mode. Although this is not recommended by the pulseaudio developers, it works quite well for this purpose. But don’t blame me if your server gets hacked – see this link.

Firstly, open /etc/default/pulseaudio and change the following lines to look like this:


Then open /etc/pulse/system.pa and edit this section:

### Automatically load driver modules depending on the hardware available
.ifexists module-hal-detect.so
load-module module-hal-detect
### Alternatively use the static hardware detection module (for systems that
### lack udev support)
load-module module-detect

to look like this:

### Automatically load driver modules depending on the hardware available
.ifexists module-udev-detect.so
load-module module-udev-detect
### Alternatively use the static hardware detection module (for systems that
### lack udev support)
load-module module-detect

Now, there’s just one ting remaining, the network service. PulseAudio comes with a module which listens to network requests, and it just needs to be activated. Open /etc/pulse/system.pa and add the following line:

load-module module-native-protocol-tcp auth-ip-acl=;

Where is your local subnet.

Restart the computer and you should be able to play music on your server from any machine on your network.

Client setup:

There are two ways of configuring a client,

1. Install padevchooser ( sudo apt-get install padevchooser) and start it. Click the icon and select Default Server -> Other… Input your servers ip-address or hostname and port 4713 (server:4713). Start your music and your sound should play on the server.

2. Configure the pulse client to globally select the correct server. TODO: write this 😀

Linking /dev/sdX to physical device

I have for quite some time been running a software-RAID5 array using 5 harddrives with mdadm in Linux. I have never had a disk failure, but I am not looking forward to the day it happens, why?

I have 5 nearly identical harddrives and I have no idea which one of them is connected to the equivalent /dev/sdX. I know the array will refuse to activate if it is missing more then one drive, and such I could just unplug the drives one by one and see if I hit the correct one, but this is no optimal solution.

While looking at the hardware stats with lshw, I notised that the harddrives listed there have its serial number printed in cleartext, which solves my problem. If a disk fails, I only need to do a lshw to find its serial number and swap the correct disk without any doubt that I have unplugged the correct one.

Solution is:

sudo lshw -C disk