docker: os: Allow caching vuls db between builds
Vuls database files are quite large and take a lot of time to download,
occasionally leading to out-of-memory issues on certain Jenkins slave
build nodes.
To overcome this limitation, allow the build process to cache its data
between subsequent runs:
- if the build host machine (i.e. build server) already has a file at
/opt/akraino/validation/tests/os/vuls/db.tar.gz,
that file will be included in the Docker build context and extracted
during build (Dockerfile ADD implicitly does that for .tar.gz files),
then brought up to date and re-archived;
- to enable vuls DB caching, one can create an (initially) empty tar.gz
file at the expected location on the host (build) machine using:
$ make -C docker/os .init_db_cache
This is a one time (manual) job, `make build` or equivalent calls will
ensure the db.tar.gz file is kept up to date afterwards;
Implementation quirks:
- getting the updated db.tar.gz from the os docker image requires us to
spawn the image as a running container (since Docker does not allow
extracting files from the image directly), so we handle this step as
a post-docker-build step via another double-colon `.build` make
target;
- since we want to perform both a pre-docker-build (copying build
server persistent db.tar.gz to the current build dir, i.e. inside the
Docker build context) and a post-docker-build step (copying back the
updated db.tar.gz from the os docker image to the build server
persistent location), we leverage double-colon Makefile targets for
`.build`:
* pre-docker-build .build (from docker/os/Makefile before the include
../build.mk statement) is executed first;
* .build from ../build.mk (common for all Docker builds in validation)
is then normally executed, resulting in a tagged docker image;
* post-docker-build .build (from docker/os/Makefile after the include
statement) is executed last;
While at it, fix some related issues:
- fix build issue caused by a new (changed upstream) dependency on
python 'cryptography' package >= 2.5 (which would require SSL and FFI
libs inside the destionation container where pip tries to compile the
new requirement) by adding the new requirement to
pip-requirements.txt, so we also create a wheel for it in the build
container;
- reduce verbosity of `go-cve-dictionary fetchnvd` to
eliminate some non-printable characters from the job output;
- reduce verbosity of wget dot progress;
- add timestamp inside os build container to force Docker cache
invalidation of database fetch/update step (to avoid missing upstream
changes masked by Docker cache);
- split build container vuls database fetch into 2 separate steps
covering tool download/installation, respectively fetch/update, so we
leverage the Docker cache for the first step while always invalidating
the cache for the second;
- `gost fetch` should use a number of threads comparable with the number
of available processors;
JIRA: VAL-101
Change-Id: I41d7bec0e72c92da1596abd67e2c1306ef9ffffa
Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>