RFC: Filesystem permissions and environments with Docker

In its current implementation, Drupalizer has a hard dependency on Docker to manage the configuration of development environment, which currently includes: PHP, Apache, MySQL, NPM and Python.

By using Linux namespaces, Docker creates an independent view of system resources for every container. A command run in a container, does not inherit environment or attributes of the shell where it was invoked.

This way, if a script writes a file to disk, it will be created with different UIDs when the script is executed inside and outside of Docker.

We will try to execute fab init in a clear project that uses Drupalizer and examine the properties of all files that were created. The init task will be executed for the first time, so the src/drupal/ directory does not contain anything yet.

$ fab init
[...]

$ find src/drupal/ -type f -exec stat -c %U {} \; | sort -u
root
ubuntu
www-data

The execution of init task produced files owned by three different system users, among which is the root user who ends up owning at least the following file src/drupal/sites/all/drush/aliases.drushrc.php, and the source of all modules and themes that were downloaded as dependencies during site installation.

It blocks automation (at least) in the following ways:

1. It is impossible to run two successful sequential builds in Jenkins without manual intervention.

During a Jenkins build, files in the build workspace will be removed in two cases: (a) automatically by Jenkins at the beginning of the build (which is what Jenkins 2 does); (b) by Drush make command which will try to move Drupal source tree into src/drupal/ after downloading it. In both cases, continuous deployment with Jenkins will be blocked until someone with root access manually deletes the whole build workspace.

2. It is impossible to run fab init && fab release && fab deploy as a single command in neither local environment, nor Jenkins. As its first step, Drupalizer deploy command will try to empty the contents of the src/ directory and will instantly fail because it cannot remove files not owned by the current user.

3. If we choose, for some project, to commit compiled CSS files alongside with the SASS sources, it will break git pull command in the following scenario:

a) execute CSS preprocessor build task from inside of Docker environment; b) grunt/gulp deletes and recreates all CSS files but now they are owned by the root user; c) execute git pull to fetch new commits that contain modifications of those CSS files; d) Git will not be able to apply changes to files not owned by the current user and fill exit with an error.

4. Translation sources, cache files and images written to disk by Apache and Drush from inside of Docker cannot be removed or updated in local environment without sudo which requires manual intervention. This may happen if Drush is used both inside and outside of Docker for debugging purposes, for example.

Mitigation

In case of files owned by www-data, it is possible to mitigate the problem of ownership by adding the main user to the www-data group. This way, files and directories created by Apache user inside of Docker can be deleted in local environment. But this approach has the following drawbacks:

a) the main user (ubuntu, jenkins, etc.) has to be manually added to www-data group on every host;

b) all files and directories created by Apache inside of Docker must be group-writable which is not the default creation mode. This, again, requires manual configuration of umask on each and every created container (this breaks continuous deployment), or this has to be specified in Dockerfile;

c) this approach is not portable because the name and UID of Apache user is not guaranteed to be the same on all Linux machines. For example, Ubuntu uses www-data user with UID 33, while CentOS uses apache user with UID 48. So if Drupalizer is used on CentOS host, file operations will still require sudo command.

d) this approach cannot be scaled to work with files owned by root user. Only the root user can remove those files and directories.

Proposed solution

In order to turn Drupalizer into a reliable continuous deployment tool that will not unpredictably block builds, it must be to ensured that all executed tasks and commands always run with same UID and GID of the main user who invokes the fab command.

This is a more predictable behavior that will contribute to transparency of environments and decrease dependency on Docker. For example, if file ownership is properly maintained, it does not matter if drush dl features is executed in local environment or inside of Docker - the result will be the same.

To achieve this behavior, it is necessary to implement the following steps:

1. Detect the UID and GID when any Fabric task is executed or Docker image is built.

2. Configure Apache to run with these UID and GID.

3. Connect to the Docker container using the correct user in order to have these UID and GID after login; OR use sudo to run every command as the correct user.

4. Ensure that none of the commands executed by Fabric tasks require root privilege on the container in order to simplify execution of commands. All configuration of the Docker environment must be done either when an image is built or when a container is first created.

Drawbacks

While this solution makes Drupalizer usage more predictable, it introduces certain new expectations on how the Docker image will behave and interact with Drupalizer. As a minimum, Docker image should be able to receive UID and GID that we want to use to execute commands.

The most simple approach would be to always use a pre-configured Dockerfile included into Drupalizer distribution. But this will make Drupalizer less flexible because projects that rely on it will loose ability to use custom Docker images.

An alternative approach will be to ship Drupalizer with a set of integration scripts, and simply require those scripts to be included into Dockerfile.