Implement the approach in the form of a daemon

[flongo82]

We need to implement a daemon acting as a hypervisor. Right now, best thing to do is to implement it in Node.js. Performance could be an issue but we will deal with that later.

The daemon should have a REST interface for administration APIs. At least, it should provide two calls: create() and destroy(). For the time being, only a human-friendly name for the virtual rasp to be created (or destroyed) could be enough as parameter. Later on, we can add a JSON file with pin mapping information (see #3).

Moreover, the daemon should be notified when a pin is exported or imported from within a virtual rasp and mount the corresponding folders accordingly (see #5).

[flongo82]

This is the current implementation:

[flongo82]

As a first step, it is necessary to focus on porting the logic that is implemented in the two shell scripts directly into a Node.js API server. Basically, after such a work, the "bash shell" actor should be gone.

This means that we need to:

• Understand how to interact with LXD from a Node.js script: is there any library that we can use? If there are no libraries, we can probably call a shell command directly from the Node.js code.
• Understand how to manipulate the filesystem from a Node.js script: is there any library to do that? Also in this case, if no libraries are available we can call shell commands directly from the Node.js code.
• Understand how to call the node-folder-mirroring.js from the Node.js API server code. This can surely be done given that it is already a Node.js code. We just need to modify node-folder-mirroring.js to be a module that can be imported from the Node.js API server code and invoked.

In order to implement the Node.js API it is okay to use express.

[maratstepanov]

understand how to interact with LXD from a Node.js script: is there any library that we can use? If there are no libraries, we can probably call a shell command directly from the Node.js code.

There are few packages like node-lxd and lxd, but they do not allow to fully manage lxd, specifically to change configuration of container. Moreover, they are just wrappers, which translate calls into shell commands. Therefore, I suggest using the following construction:

var exec = require('child_process').exec;
exec(`lxc config device remove ${name} devices disk`, (error) => {
      if (!error) {
        console.log(`disk was removed from ${name}`);
      } else {
        console.log(`An error has occured during disk removal from ${name}: ${error}`);
      }
    });

I have checked - it works.

Understand how to manipulate the filesystem from a Node.js script: is there any library to do that? Also in this case, if no libraries are available we can call shell commands directly from the Node.js code.

There is fs library, which allows to work with filesystem - read file contents, remove it etc. glob library allows to search requred files by mask.
For example, code which removes log files:

fs= require('fs');
glob = require('glob');
glob(`/home/ubuntu/test_gpio_mirroring/log_*_${ame}`,function(err,files){
     if (err) throw err;
     files.forEach(function(item,index,array){
          fs.unlink(item, function(err){
               if (err) throw err;
               console.log(item + " deleted");
          });
     });
  });

In order to mount/umount, mountutils library could be used.

var mountpoints = [`/gpio_mnt/${req.body.name}/sys/devices/platform/soc/3f200000.gpio`,`/gpio_mnt/${req.body.name}/sys/class/gpio`];
  mountpoints.forEach (function(mountpoint,i,mountpoints) {
      mountutil.umount(mountpoint, false, { "removeDir": true }, function(result) {
        if (result.error) {
          console.log('Error during ' + mountpoint + 'unmounting:'  + result.error);
        } else {
          console.log(`successfully unmounted ${mountpoint}`);
        }
      });
  });

Understand how to call the node-folder-mirroring.js from the Node.js API server code. This can surely be done given that it is already a Node.js code. We just need to modify node-folder-mirroring.js to be a module that can be imported from the Node.js API server code and invoked. 

The main question is: should node-folder-mirroring.js run in separate from API node.js process or not. If we run it separately, mirroring will not depend on API server. If we run it in the same process, then with API server restart we lose all running node-folder-mirroring.js scripts and should restart them if want mirroring to work correctly.

[maratstepanov]

I've came to understanding: everything should be in one daemon, because it's stated in the topping:

We need to implement a daemon acting as a hypervisor.
The daemon should have a REST interface for administration APIs

[flongo82]

Your question is correct and there are pros and cons in both approaches.

I would say, first of all, let's have a single process running both the REST API and the mirroring functions for all the virtual rasps. Then we could split it leveraging on some more advanced form of process spawning. For example, see: https://dzone.com/articles/understanding-execfile-spawn-exec-and-fork-in-node.

Of course, if the REST API daemon crashes, then we will need to restart it and restart all the mirroring for all the virtual rasp. So, we will need a place to store the names of all the virtual rasps and then reboot them dealing with the fact that the corresponding containers will already be there.

By the way, this is also true when having multiple separated processes, e.g., what happens if the physical rasp reboots? In this case, even if the daemon and the mirroring processing are separated, we will always need a way to restart the latter ones.

However, as a first step, I will not think about these problems and go for a single process daemon.

What do you think?

[maratstepanov]

Let's start with the simplest approach - everything in the one process.
We can keep names of mountings which should be active in some place (db/file) and perform all mounting jobs during start of the process in the future.

[flongo82]

I would say let's keep it simple and use a file to keep track of the virtual rasps. It should be a JSON file because while right now we have only names, later on we will also have information about pin mapping.

[flongo82]

Also, please fork this repository and perform pull requests when you have major commits so that we keep everything in one place.

[maratstepanov]

Could you explain what this line is doing?

MYGID=$(($MYUID + 'lxc exec "$name" -- sed -nr "s/^gpio:x:([0-9]+):.*/\1/p" /etc/group\'))

 I don't think that fully undestand why $MYUID concatenates with id of gpio user.  

[flongo82]

It is not a concatenation. In bash the $((...)) construct allows mathematical operations. It is an actual sum. The group ID outside the container is given by the group ID inside the container plus a certain value that is given by the UID of the root user of the container on the host machine. Basically all the IDs of users and groups inside the container need to be summed with that value to have the correspondent on the host machine.

[maratstepanov]

I've prepared the server script, to which you can push JSON with container name, and it would do everything declared in "current implementation of launch virtual rasp).
There are some additional packages to install - I have to write a user manual to install them (generally npm install xxx yyy zzz).
Script currently stucks on fuse.mount: this is blocking procedure, which does not allow script to go on. It looks like fork() method suggested by you would be nice soluction to overcome this behaviour.
I will fork the repository to show it.

[flongo82]

So right now it is possible to launch a single virtual rasp, right? I guess fork() can be a good solution. It also allows to have interprocess communication between parent and child processes:

https://nodejs.org/api/child_process.html#child_process_child_send_message_sendhandle_options_callback

This could be useful when we are going to implement more advanced features because I would not implement all the logic in the mirroring script.

As soon as you perform a pull request, I will review the code and accept it. Please, put the installation manual in a file different from the current README and do not delete the bash scripts.

[maratstepanov]

So right now it is possible to launch a single virtual rasp, right?
Correct.

Sorry, I have messed up a bit. fuse.mount does not block server, and it keeps accepting messages. I have forgotten to add response, therefore it did not answer to client, and I thought that it was blocked by fuse. Currently it works.
Performed a pull request