bluval: Add Jenkins log-parser rules

Instead of creating the log-parser rules on the fly from the CI job,
add the rules to the current repository, allowing the Validation
committers to tweak them independently (not through ci-management
changes).

While at it, extend and improve the previous rules by:
- raising a warning instead of an error for robot FAIL messages (which
  will later be changed via ci-management to _not_ fail the CI job run);
- separating the test suites into log parsing sections;
- raising an info for PASS robot messages (so they are listed in the
  log-parser links section for better readability);
- fail the CI job only if critical tests failed;

For logparser section grouping to work as expected, fix one output
buffering issue that used to print the issued commands after their
output (and also fix another output buffering related issue to keep
things uniform).

JIRA: VAL-104

Change-Id: I1506d84a44926e80913562f67f24885a98f94df7
Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>

diff --git a/bluval/blucon.py b/bluval/blucon.py
index 0d5d7ca..2fce9d0 100644 (file)
--- a/bluval/blucon.py
+++ b/bluval/blucon.py
@@ -66,7 +66,7 @@ def invoke_docker(bluprint, layer):
 
     args = [cmd]
     try:
-        print('\nInvoking {}'.format(args))
+        print('\nInvoking {}'.format(args), flush=True)
         subprocess.call(args, shell=True)
     except OSError:
         #print('Error while executing {}'.format(args))

diff --git a/bluval/bluval.py b/bluval/bluval.py
index 63b301f..73cc68a 100644 (file)
--- a/bluval/bluval.py
+++ b/bluval/bluval.py
@@ -68,7 +68,7 @@ def run_testcase(testcase):
 
     print('Executing testcase {}'.format(name))
     print('show_stopper {}'.format(show_stopper))
-    print('Invoking {}'.format(args))
+    print('Invoking {}'.format(args), flush=True)
     try:
         status = subprocess.call(args, shell=False)
         if status != 0 and show_stopper.lower() == "true":

diff --git a/bluval/rules.txt b/bluval/rules.txt
new file mode 100644 (file)
index 0000000..f2273ff
--- /dev/null
+++ b/bluval/rules.txt
@@ -0,0 +1,10 @@
+# Separate results based on suite (group the messages by suite)
+start /(?:Executing testcase ).*$/
+
+# Only fail if some critical tests failed
+debug /critical test.*passed, 0 failed/
+error /critical test.*passed/
+
+# Suite/subsuite test results will be marked as info/warn
+info /PASS/
+warn /FAIL/