incompatibile with ember-exam?

[SergeAstapov]

Hi there!

First of all - thank you for great addon!

I'm not sure it's necessarily a defect, maybe just "how things work". If my assumptions are correct - maybe worth to add this as a note somewhere in docs.

We use ember-exam to parallelize tests sugar following command:

ember exam --split=4 --parallel=1

As we started to write more fastBoot tests, they started to flow into different partitions.
At the same time what we observed is that FastBoot tests sporadically fail.

My assumption is:
when tests run, ember-cli-fastboot-testing creates instance of testemMiddleware (here).

As we parallelize tests and run them in 4 browser instances at the same time, FastBoot tests may be split between different browser instances.
After every test run afterEach hook calls /__cleanup-mocks url (from here, defined here).

Due to multiple threads of test runners but single instance of testemMiddleware - mocks may be cleaned up while some other test is running, which may cause tests to fail sporadically (execution order and execution timing in not guarantied when we have 4 browser instances running in parallel).

Please share your thoughts on this.

As a mitigation, we changed test command to following:

ember exam --split=4 --parallel=1 --filter='!Fastboot |' && ember test --filter='Fastboot |'

[ryanto]

Yup, absolutely a bug! Thanks for the great write up, that made it easy to understand.

Also thanks for posting your fix, hopefully that will help anyone else facing this for the time being.

My guess is that we should have beforeEach setup a guid that creating and cleaning up mocks can use. This way cleanup only clears out mocks that were created during its test run.

[SergeAstapov]

@ryanto Thank you for quick response!

looks like nock supports "scopes" - https://github.com/nock/nock#scope-filtering, may be an option here

[suchitadoshi1987]

@ryanto did we come up with a solution for this issue? looks like we are seeing a similar issue which potentially might be caused due to this.

[ryanto]

Hi @suchitadoshi1987

Right now the best thing to do is the test filtering based on Serge's post so that you only run a single fastboot testing process.

ember exam --split=4 --parallel=1 --filter='!Fastboot |' && ember test --filter='Fastboot |'