Ubuntu – LXC: Create privileged container that can execute mknod

I'm going to dispel a few myths here.

This is just a bad idea. I'm sorry. – Jacob Mar 5 at 20:30

I don't see how this is a bad idea. It's really just a chroot inside a chroot. On one hand, it could possibly decrease performance in some negligible manner (nothing compared to running a VM inside a VM). On the other hand, it's likely to be more secure (e.g. more isolated from the root host system and it's constituents).

Do you actually have a real reason to do this? Please remember that questions here should be about actual problems that you face. – Zoredache Mar 5 at 21:52

I agree 100% with the poster's following comment. Furthermore, I think it's safe to assume that everybody who posts a question on here likely thinks that they have a real reason to do [ it ]..

I think, that lxc should be able to simplify VM migration(and backup+recovery too). But I'm not sure about cases, when there is no access to host OS(cheap vps for example). – Mikhail Mar 6 at 11:17

I actually came across this question back in June when I was first diving into LXC for PaaS/IaaS projects, and I was particularly interested in the ability to allow users to emulate cloud environments for development purposes.

LXCeption. We're too deep. – Tom O'Connor Mar 6 at 22:46

I laughed a little bit when I read this one, but that's not, at all, the case :)

Anyway, I eventually set up a VirtualBox environment with a stock install of Ubuntu 12.04 LTS Server Edition after reading all this, thinking that this was 100% possible. After installing LXC, I created a new container, and installed LXC inside the container with apt-get. Most of the installation progressed well, but resulted in error eventually due to a problem with the cgroup-lite package, whose upstart job failed to start after the package had been installed.

After a bit of searching, I came across this fine article at stgraber.org (the goodies are hiding under the "Container Nesting" section):

sudo apt-get install lxc
sudo lxc-create -t ubuntu -n my-host-container -t ubuntu
sudo wget https://www.stgraber.org/download/lxc-with-nesting -O /etc/apparmor.d/lxc/lxc-with-nesting
sudo /etc/init.d/apparmor reload
sudo sed -i "s/#lxc.aa_profile = unconfined/lxc.aa_profile = lxc-container-with-nesting/" /var/lib/lxc/my-host-container/config
sudo lxc-start -n my-host-container
(in my-host-container) sudo apt-get install lxc
(in my-host-container) sudo stop lxc
(in my-host-container) sudo sed -i "s/10.0.3/10.0.4/g" /etc/default/lxc
(in my-host-container) sudo start lxc
(in my-host-container) sudo lxc-create -n my-sub-container -t ubuntu
(in my-host-container) sudo lxc-start -n my-sub-container

Installing that AppArmor policy and restarting the daemon did the trick (don't forget to change the network ranges, though!). In fact, I thought that particular snippet was so important that I mirrored it @ http://pastebin.com/JDFp6cTB just in case the article ever goes offline.

After that, sudo /etc/init.d/cgroup-lite start succeeded and it was smooth sailing.

So, yes, it is possible to start an LXC container inside of another LXC container :)

EDIT #2:

Wow, can't believe I missed this:

When first installing lxc (version: 0.8.0~rc1-8+deb7u1) ("apt-get install lxc") it will ask you in what directory you want the containers to go.

EDIT #2 END

EDIT:

Apparently if you install "lxc" from source with the command:

make install --localstatedir=/home

it should allow automatic creation of a VE in the /home directory. This will impact all lxc-* tools.

EDIT END

You need to edit "lxc-create" (for me it's in "/usr/bin"). In that file where it defines "localstatedir" and "lxc_path" you need to change that.

Based on what you want, you would need:

localstatedir=/home
lxc_path=${localstatedir}/al/lxc

Or possibly:

localstatedir=/home/al
lxc_path=${localstatedir}/lxc

My setup:

localstatedir=/home
lxc_path=${localstatedir}/lxc

I have just figured this out; it creates the container without any errors, but I have not tested it beyond that. As for the line "LXC_DIRECTORY=" in the config file "/etc/default/lxc", it needs to match the line "lxc_path=" in the file "lxc-create" or it will create a rootfs in two different directories.

So your config file "/etc/default/lxc" would need to read:

LXC_DIRECTORY="/home/al/lxc"

and mine reads:

LXC_DIRECTORY="/home/lxc"

This is of course a workaround, and I do not know the impact this has on other lxc-* tools. They may need to be edited as needed also; perhaps someone else can chime in on how to install lxc with this built into the program.