Google Summer of Code 2017 Final Report

The idea:

The aim of my project was to support installation of containerised applications in amahi. Before I started my GSOC project, amahi supported applications which used a specific stack only. The goal of this project was to add support for different stacks by using docker.

Progress of Work

Initially we started with small goals and kept testing stability of our changes and kept moving on.

  • First we added support for running php5 applications in Amahi using docker. The problem was that F-25 and F-26 had pre-installed php7 and so the apps which required php5 were failing. We could have used other solutions to fix this problem but we didn’t want a stick and glue solution and hence we used docker containers.
  • In beginning we made a simple php5 Hello World app work inside Amahi and then we added database support to that hello-world app. Pull Request: Docker containers: the beginning added the feature of supporting php5 apps.
  • Along with this we also built docker images of a few legacy PHP5 applications which we had to support, namely
    • coppermine
    • osticket
    • fengoffice.

After this the project took momentum. Once we were confident enough that this system would work properly we started shifting our focus on including next-gen apps with varied stacks into Amahi. As we were trying to add apps we came across different problems related to container configuration and management and decided to use docker-compose.

Once we shifted to docker-compose things became quite smooth.

  • We had to remove a large chunk of code written and rewrite things in a much simpler way.
  • So what we did essentially was that I modified things in code such that a docker-compose.yml file is made available for each app on amahi.org (as install script) and to run an app that docker-compose.yml file is downloaded and modified as per the requirement and the app can be run using docker-compose easily. We don’t have to worry about managing the container after that. So the idea was to write a docker-compose file for each container application, a parser on the HDA will read the docker-compose.yml file and modify it as per the system and then run it using a single command.
  • This enabled us to run complex applications which required multiple containers to run in isolation easily. Docker-compose takes care of running and linking different containers running in isolation. This gave us a lot of flexibility and made the work of adding apps much easy and stable.
  • To implement this I had to essentially rewrite the whole thing. I removed a lot of code written earlier and added new and less code and things worked like a charm. I felt that I should have taken this approach in the beginning itself and that would have saved at least 20-30 days. But that’s ok I guess, I wouldn’t have taken this approach if I did not experiment for those 20 days.

From then on adding apps became a breeze and I added some really interesting apps to Amahi like

From now on-wards if we have official image for some application available then they can be easily integrated into Amahi by simple writing a docker-compose file and adding it to amahi.org. If you are interested to know the process for that, you can refer to this : Amahi Documentation

Important Diffs and PRs

I categories all the commits done into 3 major diffs and 2 Pull requests

Diffs

Tested node js and ruby on rails stack, Implemented a test suite for testing apps

Shifted to docker compose for managing containers and achieved a much stable system

Code cleanup and comments

Pull requests

  1. Pull Request: Docker containers : the beginning
  2. Pull Request: A stable version. Gitlab and Redmine working

Code Contributed by me.

In case if you want to have a look in detail the work done you can see my fork of the project along with other repositories that I maintained for this:

My Amahi Platform fork 

Amahi Containers Base images and install scripts 

Wiki Documentation of the work done

Reports

I did write a few reports to discuss various aspects of the problem statement.

How to integrate docker images to Amahi?  

Shifting to docker-compose

Challenges

Some major challenges during the project:

  • Understanding Amahi Ecosystem : Amahi as a software is very complex. It took a while to get the development environment up and ready. Thanks to VirtualBox to make things easy.
  • How to integrate docker into the existing Amahi system? Defining the whole architecture of what fits where? We had in length discussions of how things could get impacted and we always had performance in mind.
  • Creating lightweight docker images of legacy applications : To support old php5 apps we had to build lightweight images. We ultimately used the Linux Alpine Project  to keep things light.
  • Container management : We wanted Amahi and Docker to have minimum interaction. Initially the code was getting complicated because we were using docker api to manage containers and their configuration. We solved this problem by shifting to docker-compose.

Future Work

Cleanup – too many images.

Presently when a containerised app is uninstalled the container is stopped and removed but the image used to run the container stays (Note the difference between container and image). I am not removing the image since images are generally large and what if users decides to reinstall the app? This has to be thought of more. Adding the feature of deleting image after installation is essentially 1 line of code. Going forward we might have to come up with a mechanism for clearing images which haven’t been used for a long time.

Reducing Download Size

We can’t build and maintain containers for all apps. Vendors provide official containers but some of them are huge in size. To fix this problem we can reduce the download to that of a normal installation by using an apt-cache, gem server or npm server on hda itself.

The idea is following: Instead of we building images on our server we can push the Dockerfile to the client and the client can build the docker image. While building the image they will download the required packages, gem files, node dependencies or whatever. If we have a mechanism to cache this download so that all the subsequent builds can use this data then we can save a lot of Internet usage. One possible way of doing this was running apt-cache, gem server and npm server on the client itself.

Updating apps

With containers updates can be really easy. We can support single click update of apps. One possible way of implementing this feature would be to create an update button which on click would stop the running container, download the latest image of the app (which is the updated app) and restart the container. Though implementing this from docker perspective is not a big deal at all but it can get tricky as well if there are some major upgrades in code then the vendor must take care of data migration from older version to newer versions in their image itself. But this problem of migration exists without containers as well.

Support for advance configuration.

With containers we can limit the cpu/memory/disk usage of each app. We just have to modify the docker-compose.yml file for the app for these changes without doing any modification to the source code. If required, this can be used.

Collecting Logs

It’s a new feature, we will need to collect a lot of metrics from the users to understand how this feature is working and how it can be improved. Some of those features include

  • The CPU info of systems which are running Amahi.
  • RAM and Storage information.
  • Logs of containerised apps to debug errors

Comments

comments