Run kubelet in Hyperd pod

[enzian]

Hi Guys

I'd like to set up small kubernetes clusters for a workshop and since we're running hyperd I thought I'd try an run the kubelet in a hyperd pod and let it start all it's dependencies in there!

So my question is - Is it possible to get the kubelet to run inside a hyperd VM? The main challenge I see is getting Docker to run inside the VM so the Kubelet can start controllers, apisurfaces etc... Does anybody know if a) that's possible and b) has any advice on how it could be done?

[bergwolf]

@enzian You can run docker in hyperd though you need to mount all cgroups mountpoints on your own. But after that, docker runs w/o issues.

IMO, the main challenge of running kubernetes inside hyperd is the missing systemd. hyperstart runs as the init program in the guest and thus systemd cannot run (since it requires to be the init process as well). If you can setup kubernetes without systemd, you can run it inside hyperd.

[enzian]

@bergwolf any pointers as to where I can find the mountpoints needed?

[bergwolf]

For example this is the cgroups mountpoints in my local machine:

tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755)
cgroup on /sys/fs/cgroup/unified type cgroup2 (rw,nosuid,nodev,noexec,relatime)
cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,name=systemd)
cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices)
cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset)
cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids)
cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event)
cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb)
cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct)
cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio)
cgroup on /sys/fs/cgroup/rdma type cgroup (rw,nosuid,nodev,noexec,relatime,rdma)
cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer)
cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio)
cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory)

You can find out a list of supported cgroups in /proc/cgroups and then mount them properly. I don't remember exactly what is required to run docker but all needed cgroups are already included in the hyperstart kernel image.

$cat /proc/cgroups
#subsys_name    hierarchy       num_cgroups     enabled
cpuset  3       2       1
cpu     7       67      1
cpuacct 7       67      1
blkio   8       67      1
memory  12      101     1
devices 2       67      1
freezer 10      2       1
net_cls 11      2       1
perf_event      5       2       1
net_prio        11      2       1
hugetlb 6       2       1
pids    4       71      1
rdma    9       1       1

[enzian]

hmm, I'll try that one and report back here :-)

[enzian]

ok, here's what I've tried so far:

I used the docker:stable-dind image and wantet to run this like this:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  labels:
    run: my-shell
  name: my-shell
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      run: my-shell
  strategy:
  template:
    metadata:
      labels:
        run: my-shell
    spec:
      containers:
      - args:
        - sh
        image: docker:stable-dind
        imagePullPolicy: Always
        name: my-shell
        resources:
          limits:
            cpu: 500m
            memory: 1048Mi
        volumeMounts:
        - mountPath: /sys/fs/cgroup
          name: cgroup-storage
        - mountPath: /var/run
          name: docker-sock
      volumes:
      - emptyDir: {}
        name: cgroup-storage
      - emptyDir: {}
        name: docker-sock

Then I attach to the pod in kubernetes and try running dockerd which then fails with:

WARN[2018-06-19T14:56:45.325877326Z] could not change group /var/run/docker.sock to docker: group docker not found
can't create unix socket /var/run/docker.sock: listen unix /var/run/docker.sock: bind: no such device or address

I'm not sure what's going wrong and google did not help much so far. Why would dockerd not create a socket in this directory?