Running multiple containers within virtual hosts in Mininet

Mininet is a popular network emulator that allows us to create virtual networks using virtual hosts, switches, and controllers. Recently, I faced the need to run multiple Docker containers within virtual hosts in Mininet. This article dwells on why it is challenging and how I eventually accomplished it.

Context

Before we begin, let me explain my motivation. I am a researcher working on Service Function Chaining (SFC) and I recently set about building a testbed that I can use to evaluate algorithms used for SFC placement. For the uninitiated, SFC placement is about building a chain of Virtual Network Functions (VNFs) such as firewalls, load balancers, Intrusion Detection Systems (IDS), etc., and finding the optimal physical servers and links to embed these functions on. One server may accommodate several VNFs and we virtually link these VNFs together over the physical links to form SFCs.

I decided to use Mininet to emulate the physical substrate network. Mininet provides both virtual hosts and switches, so building a simple substrate network was straightforward. However, deploying multiple VNFs on a virtual host proved to be a challenge. This is because virtual hosts in Mininet offer only network isolation.

The challenge

This means Mininet doesn’t offer filesystem or process isolation. Processes running on a virtual host in Mininet actually run on the Mininet host and use the host’s filesystem. This becomes a problem when you want to deploy multiple VNFs on one virtual host, especially when you want two instances of the same VNF. To illustrate this, let’s use the following example.

Two web servers and a client are connected using a switch. The web servers run a Flask application that uses a file to keep track of the number of API requests. Every time a client sends a request, the app increments this number in the file and returns it as a part of the response.

The two servers should run an instance of this application each. Ideally, we will expect each server to maintain and return its own counter values. Let’s see if that’s the case here by sending a request from the client to the two servers in Mininet.

When we run the code, we can observe that the servers return 0 and 1. This means both servers share the same file. Let’s try to send one more request to each server to confirm.

As we can see, both servers are using the same file to persist the counter. This is because, in Mininet, virtual hosts share the same filesystem. Besides, we can also see that the two Python processes are running on the Mininet host by running the `ps -al` command.

Using Docker containers as virtual hosts

We can try to implement network, filesystem, and process isolation on virtual hosts. However, since containerization already facilitates this, why don’t we simply commission its service? Here is where Containernet comes in handy. Containernet is a fork of Mininet that replaces the virtual hosts in Mininet with Docker containers offering us complete isolation. You can find a guide on how to get started here.

So, let’s see if Containernet can solve our issue. I am going to now use the Python Docker library to build our web server into a Docker image and replace the two virtual hosts in the previous example with two Docker containers.

This time, as expected, the servers have returned two independent counter values as they maintain their own files thanks to the filesystem isolation provided by Docker.

Docker in Docker containers

However, this is not enough. Even though each virtual host is now self-contained, running multiple VNFs within a container is still going to pose issues. For starters, running two instances of the same VNF within a container will be problematic as there exists no process or filesystem isolation within a container. Ideally, we should deploy VNFs as containers to overcome such issues. However, since we use a container itself as a virtual host, this means that we will have to deploy containers within containers.

This may seem wild, but it is very much a possibility. In fact, Docker has an official image that allows you to spin up Docker containers within a Docker container. You can access these images by using the `dind` tag. However, there is a catch.

Enabling the privileged mode

To deploy containers within containers, we have to run the host container in the privileged mode. Doing so grants containers access to the kernel space of the host Operating System. We can do this easily by passing the `–privileged` flag to the Docker `run` command. However, Containernet doesn’t allow you to start a container in the privileged mode and this is disabled by default.

Consequently, the only way to resolve this is by modifying the code. There is already a Pull Request to the repository with the necessary changes to add this feature but it has not been merged yet. So, either you will have to modify the code yourself and run it or you can use my fork of the repository that contains the changes.

Running multiple containers within a virtual host

Let’s now see if we can run two instances of the web server within one Docker container successfully. To that end, I will be using a server and a client connected by a switch. The server is a Docker container that runs the Docker-in-Docker image. Two instances of the web server will be deployed as Docker containers within this Docker container.

Here, one container is mapped to port 80 on the host and the other to 8080.

As we can see, we are able to run these two servers without any issues as Docker containers within the host Docker container. Accordingly, we can use Containernet and the Docker-in-Docker image to deploy multiple containers within a virtual host in Mininet. By using this strategy, I can now emulate an environment where multiple VNFs are deployed on a physical server.

Summary

In this article, we first saw the motivation behind needing to run multiple containers within a virtual host and the challenge posed by the absence of filesystem and process isolation in Mininet hosts. Subsequently, we discussed how we can leverage Containernet to use Docker containers as virtual hosts in Mininet, providing us with complete isolation. Finally, we dealt with how we can use the Docker-in-Docker image to deploy multiple VNFs within the host container. You can find all the code I used in this article here.