Commit 8564977d authored by Christian Kuhn's avatar Christian Kuhn
Browse files

Merge remote-tracking branch 'infra-bamboo-remote-agent/master' into...

Merge remote-tracking branch 'infra-bamboo-remote-agent/master' into feature/merge-infra-bamboo-remote-agent
parents 8757aa4a 13fa578f
# EditorConfig is awesome: http://EditorConfig.org
# top-most EditorConfig file
root = true
# Unix-style newlines with a newline ending every file
[*]
end_of_line = lf
insert_final_newline = true
indent_size = 4
indent_style = tab
[*.yml]
indent_size = 2
# This is a basic workflow to help you get started with Actions
name: CI
on:
push:
branches: [ master ]
jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
- uses: dorny/paths-filter@v2
id: filter
with:
filters: |
changes:
- 'baseimage/**'
- name: Login to GitHub Container Registry
if: steps.filter.outputs.changes == 'true'
uses: docker/login-action@v1
with:
registry: ghcr.io
username: TYPO3IncTeam
password: ${{ secrets.CR_PAT }}
- name: Login to Docker Hub
if: steps.filter.outputs.changes == 'true'
uses: docker/login-action@v1
with:
username: ${{ secrets.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_TOKEN }}
- name: Build Base Image
if: steps.filter.outputs.changes == 'true'
run: make build_baseimage
- name: Release Base Image
if: steps.filter.outputs.changes == 'true'
run: make release_baseimage
build-images:
runs-on: ubuntu-latest
needs: [ build ]
strategy:
fail-fast: false
matrix:
image: [ php53, php54, php55, php56, php70, php71, php72, php73, php74, php80, bamboo, js ]
steps:
- uses: actions/checkout@v2
- uses: dorny/paths-filter@v2
id: filter
with:
base: ${{ github.ref }}
filters: |
matrix:
- '${{ matrix.image }}/**'
- name: Login to GitHub Container Registry
if: steps.filter.outputs.matrix == 'true'
uses: docker/login-action@v1
with:
registry: ghcr.io
username: TYPO3IncTeam
password: ${{ secrets.CR_PAT }}
- name: Login to Docker Hub
if: steps.filter.outputs.matrix == 'true'
uses: docker/login-action@v1
with:
username: ${{ secrets.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_TOKEN }}
- name: Build ${{ matrix.image }} Image
if: steps.filter.outputs.matrix == 'true'
run: make build_${{ matrix.image }}
- name: Release ${{ matrix.image }} Image
if: steps.filter.outputs.matrix == 'true'
run: make release_${{ matrix.image }}
.DS_Store
.vagrant
*.swp
build_baseimage
build_php53
build_php54
build_php55
build_php56
build_php70
build_php71
build_php72
build_php73
build_php74
build_bamboo
build_js
.idea
Copyright (c) 2013-2015 Phusion Holding B.V.
Copyright (c) 2016 Morton Jonuschat
Copyright (c) 2018 Christian Kuhn
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in
all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
THE SOFTWARE.
This diff is collapsed.
# A set of Docker containers for TYPO3 testing infrastructure
## Introduction
This [repository](https://bitbucket.typo3.com/projects/T3COM/repos/bamboo-remote-agent/browse) contains
Docker container build scripts used within the [TYPO3 GmbH](https://typo3.com) infrastructure
to execute the TYPO3 CMS core tests and other build and packaging jobs.
The containers may be used by anyone to execute tests locally in order to have the exact
same environment as the "pre-merge tests" run for the TYPO3 CMS core.
The latest compiled versions of those containers can be pulled from [Docker Hub](https://hub.docker.com/r/typo3gmbh/) or from [Github Container Registry](https://github.com/orgs/TYPO3GmbH/packages).
## Architecture
Docker containers can be stacked: An image can use another image below to build its
own stuff on-top of that. This feature is used here.
### baseimage: A minimal Ubuntu base image modified for Docker-friendliness
This is the lowest layer of images put on top of each other.
typo3gmbh/baseimage is a Docker image based off of Phusion's baseimage-docker, but has been
modified to run on Ubuntu 18.04 and removes features deemed unnecessary for a modern baseimage.
baseimage is a special [Docker](https://www.docker.com) image that is configured for
correct use within Docker containers. It is Ubuntu, plus:
* Modifications for Docker-friendliness.
* Administration tools that are especially useful in the context of Docker.
baseimage is a fork from [passenger-docker](https://github.com/phusion/passenger-docker).
### phpXY
Images on top of baseimage. The images contain php in one version per container, nodejs and
some other packages like graphicsmagick.
* Single images per PHP version. There is an image coming with PHP 7.0 and an image for PHP 7.1 and so on.
Users can use these images to execute unit, functional, acceptance and JS tests in an environment that is
identical to the core testing infrastructure. Note that some core tests need additional containers that
run a database or selenium with chrome.
### bamboo-remote-agent
typo3gmbh/bamboo-remote-agent adds the bamboo test runner on top of the baseimage images for integration in
TYPO3 GmbH testing infrastructure. Users usually don't have to deal with these images and use the phpXY ones instead.
## Compiling and Uploading
A Makefile takes care of container building. One obvious reason to re-compile is when
container definitions have changed or added, another one is to have builds with younger
packages (eg. younger php versions). To create a new set of containers, these steps should be done:
* Prepare new semver versions in Makefile (and commit/push change): Each container has an own
version, raise at least the patch level.
* If files in one of the version directories get changed, the corresponding version is built automatically by Github
- version information is taken from the Makefile.
### New Base Image Release
If you want to use a new base image, adjust the base image version in all Dockerfiles. The change detection will then
take care of rebuilding the corresponding images.
## Manually Compiling and uploading
To create a new set of containers manually, these steps should be done:
* Pick a machine that has good network connectivity and a young docker engine installed.
* Drop all containers (docker rmi) that are involved in the build: the ubuntu 18.04 one, baseimage,
phpXY bulids and agent build. This forces docker to fetch / compile fresh versions of
everything.
* Prepare new semver versions in Makefile (and commit/push change): Each container has an own
version, raise at least the patch level.
* 'make build' will build all containers a-new. The php containers can be built in parallel,
this will drastically increase build server load, but reduce time. A 'make -j8 build' would build
8 php containers in parallel, after baseimage has been built.
* 'make release' will add tags and push to docker hub. It will ask for according credentials.
If it doesn't, but reject the push' run 'docker login docker.io' and log in with the credentials from LastPass.
## Packages Included
### PHP Images
The images are based on Ubuntu 18.04 and usually contain the following extensions:
* bcmath
* bz2
* cli
* common
* curl
* dev
* gd
* gmp
* imap
* intl
* json
* ldap
* mbstring
* mysql
* opcache
* pgsql
* pspell
* readline
* soap
* sqlite3
* sqlsrv
* xml
* xmlrpc
* xsl
* zip
* apcu
* pear
* redis
* memcached
* xdebug
Additionally, the following packages / tools are installed:
* re2c
* graphicsmagick
* imagemagick
* zip
* unzip
* sqlite3
* nodejs / yarn / npm
* curl
* less
* vim
* psmisc
* net-tools
* iputils-ping
* ncdu
* dirmngr
* gpg-agent
* ack-grep
* bzip2
* pbzip2
* patch
* openssh-client
* git
* language-pack-de
* parallel
* netcat
### JS Image
The JavaScript image is based on node 12 and contains yarn in addition.
FROM typo3gmbh/baseimage:3.0
MAINTAINER TYPO3 GmbH <info@typo3.com>
ADD . /pd_build
RUN /pd_build/system_services.sh && \
/pd_build/bamboo-agent.sh && \
/pd_build/utilities.sh && \
/pd_build/finalize.sh
CMD ["/sbin/my_init"]
\ No newline at end of file
#!/bin/bash
set -e
source /pd_build/buildconfig
set -x
# We don't know which user will execute bamboo, we have to use 777 here
mkdir -p /srv/bamboo/bin
chmod 0777 /srv/bamboo
chmod 0777 /srv/bamboo/bin
# Download bamboo remote agent
curl -SL --progress-bar https://bamboo.typo3.com/agentServer/agentInstaller/ -o /srv/bamboo-installer.jar
# Configure properties
cp -a /pd_build/config/bamboo-capabilities.properties /srv/bamboo/bin
# Enable agent
cp -a /pd_build/runit/bamboo-agent /etc/service/bamboo-agent
chmod 777 /etc/service/bamboo-agent
\ No newline at end of file
#!/usr/bin/python3 -u
import os, os.path, sys, stat, signal, errno, argparse, time, json, re
KILL_PROCESS_TIMEOUT = 5
KILL_ALL_PROCESSES_TIMEOUT = 5
LOG_LEVEL_ERROR = 1
LOG_LEVEL_WARN = 1
LOG_LEVEL_INFO = 2
LOG_LEVEL_DEBUG = 3
SHENV_NAME_WHITELIST_REGEX = re.compile('[^\w\-_\.]')
log_level = None
terminated_child_processes = {}
class AlarmException(Exception):
pass
def error(message):
if log_level >= LOG_LEVEL_ERROR:
sys.stderr.write("*** %s\n" % message)
def warn(message):
if log_level >= LOG_LEVEL_WARN:
sys.stderr.write("*** %s\n" % message)
def info(message):
if log_level >= LOG_LEVEL_INFO:
sys.stderr.write("*** %s\n" % message)
def debug(message):
if log_level >= LOG_LEVEL_DEBUG:
sys.stderr.write("*** %s\n" % message)
def ignore_signals_and_raise_keyboard_interrupt(signame):
signal.signal(signal.SIGTERM, signal.SIG_IGN)
signal.signal(signal.SIGINT, signal.SIG_IGN)
raise KeyboardInterrupt(signame)
def raise_alarm_exception():
raise AlarmException('Alarm')
def listdir(path):
try:
result = os.stat(path)
except OSError:
return []
if stat.S_ISDIR(result.st_mode):
return sorted(os.listdir(path))
else:
return []
def is_exe(path):
try:
return os.path.isfile(path) and os.access(path, os.X_OK)
except OSError:
return False
def import_envvars(clear_existing_environment = True, override_existing_environment = True):
if not os.path.exists("/etc/container_environment"):
return
new_env = {}
for envfile in listdir("/etc/container_environment"):
name = os.path.basename(envfile)
with open("/etc/container_environment/" + envfile, "r") as f:
# Text files often end with a trailing newline, which we
# don't want to include in the env variable value. See
# https://github.com/phusion/baseimage-docker/pull/49
value = re.sub('\n\Z', '', f.read())
new_env[name] = value
if clear_existing_environment:
os.environ.clear()
for name, value in new_env.items():
if override_existing_environment or not name in os.environ:
os.environ[name] = value
def export_envvars(to_dir = True):
if not os.path.exists("/etc/container_environment"):
return
shell_dump = ""
for name, value in os.environ.items():
if name in ['HOME', 'USER', 'GROUP', 'UID', 'GID', 'SHELL']:
continue
if to_dir:
with open("/etc/container_environment/" + name, "w") as f:
f.write(value)
shell_dump += "export " + sanitize_shenvname(name) + "=" + shquote(value) + "\n"
with open("/etc/container_environment.sh", "w") as f:
f.write(shell_dump)
with open("/etc/container_environment.json", "w") as f:
f.write(json.dumps(dict(os.environ)))
_find_unsafe = re.compile(r'[^\w@%+=:,./-]').search
def shquote(s):
"""Return a shell-escaped version of the string *s*."""
if not s:
return "''"
if _find_unsafe(s) is None:
return s
# use single quotes, and put single quotes into double quotes
# the string $'b is then quoted as '$'"'"'b'
return "'" + s.replace("'", "'\"'\"'") + "'"
def sanitize_shenvname(s):
return re.sub(SHENV_NAME_WHITELIST_REGEX, "_", s)
# Waits for the child process with the given PID, while at the same time
# reaping any other child processes that have exited (e.g. adopted child
# processes that have terminated).
def waitpid_reap_other_children(pid):
global terminated_child_processes
status = terminated_child_processes.get(pid)
if status:
# A previous call to waitpid_reap_other_children(),
# with an argument not equal to the current argument,
# already waited for this process. Return the status
# that was obtained back then.
del terminated_child_processes[pid]
return status
done = False
status = None
while not done:
try:
# https://github.com/phusion/baseimage-docker/issues/151#issuecomment-92660569
this_pid, status = os.waitpid(pid, os.WNOHANG)
if this_pid == 0:
this_pid, status = os.waitpid(-1, 0)
if this_pid == pid:
done = True
else:
# Save status for later.
terminated_child_processes[this_pid] = status
except OSError as e:
if e.errno == errno.ECHILD or e.errno == errno.ESRCH:
return None
else:
raise
return status
def stop_child_process(name, pid, signo = signal.SIGTERM, time_limit = KILL_PROCESS_TIMEOUT):
info("Shutting down %s (PID %d)..." % (name, pid))
try:
os.kill(pid, signo)
except OSError:
pass
signal.alarm(time_limit)
try:
try:
waitpid_reap_other_children(pid)
except OSError:
pass
except AlarmException:
warn("%s (PID %d) did not shut down in time. Forcing it to exit." % (name, pid))
try:
os.kill(pid, signal.SIGKILL)
except OSError:
pass
try:
waitpid_reap_other_children(pid)
except OSError:
pass
finally:
signal.alarm(0)
def run_command_killable(*argv):
filename = argv[0]
status = None
pid = os.spawnvp(os.P_NOWAIT, filename, argv)
try:
status = waitpid_reap_other_children(pid)
except BaseException as s:
warn("An error occurred. Aborting.")
stop_child_process(filename, pid)
raise
if status != 0:
if status is None:
error("%s exited with unknown status\n" % filename)
else:
error("%s failed with status %d\n" % (filename, os.WEXITSTATUS(status)))
sys.exit(1)
def run_command_killable_and_import_envvars(*argv):
run_command_killable(*argv)
import_envvars()
export_envvars(False)
def kill_all_processes(time_limit):
info("Killing all processes...")
try:
os.kill(-1, signal.SIGTERM)
except OSError:
pass
signal.alarm(time_limit)
try:
# Wait until no more child processes exist.
done = False
while not done:
try:
os.waitpid(-1, 0)
except OSError as e:
if e.errno == errno.ECHILD:
done = True
else:
raise
except AlarmException:
warn("Not all processes have exited in time. Forcing them to exit.")
try:
os.kill(-1, signal.SIGKILL)
except OSError:
pass
finally:
signal.alarm(0)
def run_startup_files():
# Run /etc/my_init.d/*
for name in listdir("/etc/my_init.d"):
filename = "/etc/my_init.d/" + name
if is_exe(filename):
info("Running %s..." % filename)
run_command_killable_and_import_envvars(filename)
# Run /etc/rc.local.
if is_exe("/etc/rc.local"):
info("Running /etc/rc.local...")
run_command_killable_and_import_envvars("/etc/rc.local")
def start_runit():
info("Booting runit daemon...")
pid = os.spawnl(os.P_NOWAIT, "/usr/bin/runsvdir", "/usr/bin/runsvdir",
"-P", "/etc/service")
info("Runit started as PID %d" % pid)
return pid
def wait_for_runit_or_interrupt(pid):
try:
status = waitpid_reap_other_children(pid)
return (True, status)
except KeyboardInterrupt:
return (False, None)
def shutdown_runit_services():
debug("Begin shutting down runit services...")
os.system("/usr/bin/sv down /etc/service/*")
def wait_for_runit_services():
debug("Waiting for runit services to exit...")
done = False
while not done:
done = os.system("/usr/bin/sv status /etc/service/* | grep -q '^run:'") != 0
if not done:
time.sleep(0.1)
def main(args):
import_envvars(False, False)
export_envvars()
if not args.skip_startup_files:
run_startup_files()
runit_exited = False
exit_code = None
if not args.skip_runit:
runit_pid = start_runit()
try:
exit_status = None
if len(args.main_command) == 0:
runit_exited, exit_code = wait_for_runit_or_interrupt(runit_pid)
if runit_exited:
if exit_code is None: