New, Improved LXC on Gentoo

So I’ve cloned the lxc repository to try to get the latest version running (because I can be stupid that way).

One of the first problems I ran into was getting the configure script to recognize that I truly do have python (and, since it’s gentoo, all the associated development pieces) installed.

I don’t know much about autogen, et al. But fireeye’s chapter on PKG_CHECK_MODULES led me to the “solution”.

That module (in this case) winds up using pkg-config to compare versions. In this situation, it’s looking for version details in /usr/lib/pkgconfig/python3.pc to see whether I have 3.2 or later installed. That file doesn’t exist on my system.

What does exist is /usr/lib/pkgconfig/python-3.3.pc which should happily inform the config system that I’m good to go.

I’m almost inclined to label this a bug in gentoo’s eselect mechanism. But it could be architectural differences of opinion. Whatever. `ln -s /usr/lib/pkgconfig python3.pc` got me moving forward.

Gentoo Hosts, LXC, and Ubuntu 14 Guests


I’m trying to get the beta version of Ubuntu 14 running as some sort of virtualized guest inside a gentoo host (don’t judge…I have valid reasons). This post will probably become obsolete very quickly.

Start With the Obvious

Canonical has tentatively released a preliminary Vagrant box of their cloud version of Trusty. So I tried it first. I can’t seem to get networking going over NAT at all. Virtual Box errors out when I try to set up host-only networking. For anyone who cares, the error from the GUI is:

Failed to create the host-only network interface.

VBoxNetAdpCtl: Error while adding new interface: VBoxNetAdpCtl: ioctl failed for 
/dev/vboxnetctl: Invalid argument


Result Code: NS_ERROR_FAILURE (0x80004005)
Component: HostNetworkInterface
Interface: IHostNetworkInterface {87a4153d-6889-4dd6-9654-2e9ff0ae8dec}

I get the same basic error code from vagrant.

Trying to use bridged networking winds up basically taking down my host. I suspect it has something to do with not releasing loopbacks…basic terminal commands quit returning, hard disk writes block, etc. Years ago, I had a job at a company that used dial-up for internet access. I tried running linux, very briefly. I had the same sorts of errors where doing something basic would freeze, wait for the corporate modem to do its thing, and then play nicely until that connection timed out and it would happen all over again.

It may be something totally different, but it feels like something similar. (The actual problem would be pretty difficult to trouble-shoot: I’m running a heavily customized kernel, and docker is setting up its own bridged connection, even though the actual docker daemon crashes because of some problem with that connection…this is why you should always do development work inside a VM from the get-go).

New Old-School

My next thought was to give LXC a shot. I really want ubuntu inside a real VM so I’m using its default kernel, but at this point I just desperately wanted to get something going. So I set about looking into LXC from gentoo.

The current LXC version in portage is 0.8, which seems to have the same limitations as the CLI available under ubuntu precise. I can see that a machine’s running, but there wasn’t an obvious way to connect automatically: the ultimate problem I’m trying to solve here is to have scripts log in and do useful work (and the VM servers we have available are far too slow).

So I spent a lot of time customizing my kernel for cgroups, trying to clear out all the errors that show up with lxc-checkconfig. I finally managed to clear up all of them except the complaint that “Cgroup memory controller: missing”. Googling for that turned out to be a complete waste of time (and the main motivation for writing this post).

I eventually broke down, cloned lxc from github, and started building it from scratch. Its README warns to carefully check the summary output from configure. It suggests trying to run –enable-feature on any features it doesn’t find that you want so you can see what the problem was.The missing pieces that stood out (for now…Apparmor is on my list) in there was:

 - init script type(s):
Security features:
 - cgmanager: no

At the time, I didn’t realize that the blank “init script type(s):” line was interesting, but now it seems symptomatic.

So I tried rerunning `configure –enable-cgmanager` and received the error “No package ‘libcgmanager’ found”. That seemed as quick and easy as installing a package under gentoo ever is. I started trying to emerge it. And was promptly shocked that it wasn’t obvious.


So I started googling around and ran into the hotbed of nasty political worms that’s swirling around the politics involved in cgroups.

I don’t know enough about the actual issue to have much of an opinion either way. I don’t even know enough to try to summarize the actual issues involved. Except that they seem to culminate in what I ran into:

Moving forward, if you want to run cgroups (and, in particular, the cgroupmanager), you’re going to be forced to switch to systemd. Which I really don’t want to do, if only because I’m basically being told I don’t have a choice in the matter. Whether I want to or not is really beside the point. For now, I don’t have time to make that kind of major switch.

Ubuntu/Debian from Scratch

This led me down another rabbit trail, which basically amounted to installing ubuntu by hand (I had a running system with a spare disk partition, but nothing I could use to boot from an ISO). Ultimately, it’s not that different from installing gentoo: use `debootstrap` to get the equivalent of a Stage 3 environment, mount the special partitions like /proc, chroot into it, configure it, add a user, then set up grub to also boot from that new partition.

I ran into some issues doing this:

  1. Ubuntu really doesn’t like the passwords that gentoo creates in /etc/shadow. It seems to have something to do with the hashing algorithm. Or maybe the salt. Actually, this bit me while I was still trying to get things going through an lxc container going…I don’t really care about cgroup protection. I was able to overcome this by doing a standard old-fashioned chroot to have ubuntu add its own users.
  2. I started using debootstrap through the lxc-ubuntu template. The documentation of both leaves out a really important detail: /etc/apt/sources.list is really vital. If that file’s missing (why would I have something like?), it defaults to searching the kernel mirrors for a debian release. This error showed up in a complaint about missing GPG key rings and an inability to find something along the lines of (TODO: I should really dig up that actual error to make life easier on google). I started hacking into the lxc-template, but the real solution was just to run the command as `sudo MIRROR= lxc-create -t ubuntu -n container-name — –release trusty –arch amd64`. (I probably didn’t need the arch argument, but it’s what worked, and it was well after midnight at this point).
  3. I had to dig into the way grub2 generates its menus to basically hard-code a kernel/root combination using the 40custom script (or something along those lines…I don’t have the box handy to check). It shouldn’t be a big deal, but I’m not happy with the solution I came up with–especially since I seem to build a new kernel once a week or so.
  4. I couldn’t get networking to happen automatically inside ubuntu. I wound up mounting my gentoo partition from ubuntu, changing into my kernel source tree, running `make modules_install`, then slapping together a script to run something like `ifconfig up eth0` and `dhcpclient eth0` (again…I don’t have that box handy to check). It certainly isn’t a good solution, but it was good enough for now. And I don’t plan to spend any more time in ubuntu land than I must.

EDIT 2014MAR08, 11:46:

LXC seems to have gotten smarter about the “cgroup memory controller” problem. I’m not sure how, but the latest version from github seems happy with my setup. (At least, lxc-checkconfig returns all green statuses).