Use pod local tmp based builddir instead of shared storage git-repos

[antbob]

The rationale behind this change is to avoid performance, resource consumption and concurrency issues associated with the builddir located on a shared storage like netapp-nfs or aws-efs. The problem with shared storage in this case is builddir often contains an extremely large amount of small files those file systems are ill designed to deal with.

In practical terms after switching to aws-efs to be able to scale the deployment (we do not have netapp-nfs available in our cluster) we are constantly running into run_package_prep/git_prepare_package_sources timing out with:

Operation timed out after 300 s / 600 s

and generally taking a long time (minutes instead of seconds) when working with builddir on shared storage.

AFAIK there is no need to place the builddir on a shared storage, instead the agent local pod storage can be used for this purpose which is much faster and takes less resources. The largest source bases sit at about 4Gb magnitude eg Firefox/Thunderbird and thus the local pod storage has enough space to host the builddir without the need to provision any additional storage resources. As long as cleanup of no longer needed builddirs works fine there should not be any space problems.

The choice to host it under tmp is because there is no need to provision ephemeral storage via deployment files as tmp acts the same way as ephemeral storage in this case anyway.

[nforro]

IIUC this only works because no privileged tools are being run on unpacked sources. That makes sense, but I'm afraid that won't be true forever, see e.g. #814.

[majamassarini]

IIUC this only works because no privileged tools are being run on unpacked sources. That makes sense, but I'm afraid that won't be true forever, see e.g. #814.

I believe #814 is compatible with this change. For now I think we are good to go with it.