If a test executed by test_runner raises signals.TestAbortAll before any failures or errors are received, the test will exit with status code 0 instead of what I expected to be 1. This is due to not executing ok = False
|
ok = True |
|
for config in test_configs: |
|
runner = TestRunner( |
|
log_dir=config.log_path, testbed_name=config.testbed_name |
|
) |
|
with runner.mobly_logger(console_level=console_level): |
|
runner.add_test_class(config, test_class, tests) |
|
try: |
|
runner.run() |
|
ok = runner.results.is_all_pass and ok |
|
except signals.TestAbortAll: |
|
pass |
|
except Exception: |
|
logging.exception('Exception when executing %s.', config.testbed_name) |
|
ok = False |
|
if not ok: |
|
sys.exit(1) |
To make this more confusing, if a test failure or error is experienced before TestAbortAll is raised, the test will exit with status code 1.
I believe both of these behaviors oppose most users' expectations of the implication of aborting a test. I would expect the test to always return status code 1 whenever an abort occurs. WDYT about adding ok = False to the except signals.TestAbortAll statement?
At the very least, some doc comments above each signal type would be valuable for setting expectations.
If a test executed by
test_runnerraisessignals.TestAbortAllbefore any failures or errors are received, the test will exit with status code0instead of what I expected to be1. This is due to not executingok = Falsemobly/mobly/test_runner.py
Lines 71 to 87 in 991ecb2
To make this more confusing, if a test failure or error is experienced before
TestAbortAllis raised, the test will exit with status code1.I believe both of these behaviors oppose most users' expectations of the implication of aborting a test. I would expect the test to always return status code
1whenever an abort occurs. WDYT about addingok = Falseto theexcept signals.TestAbortAllstatement?At the very least, some doc comments above each signal type would be valuable for setting expectations.