Disaggregated stats wrong total

[noliveleger]

Fixes #190.

[coveralls]

Coverage increased (+0.02%) to 83.865% when pulling 43e3aa0 on 190-get-disaggregated-stats-bad-total into 2d68ce0 on master.

[noliveleger]

Per discussion with @jnm, we may need to show stats for None values of the grouping field (when field is not restricted).
Perhaps we should open another issue for this. It won't be added to this PR because of this:
https://github.com/kobotoolbox/formpack/blob/master/src/formpack/schema/fields.py#L308-L310.

[jnm]

Could you include a unit test with this PR?

[jnm]

I'm having trouble figuring out where this happens. I found these:

formpack/src/formpack/reporting/autoreport.py

Lines 161 to 162 in 09c600f

	if value is not None:
	counters['__submissions__'] += 1

formpack/src/formpack/reporting/autoreport.py

Lines 84 to 85 in 09c600f

	counter.update(values)
	counter['__submissions__'] += 1

...but they both increment counter['__submissions__'] only when the value is not None.

[noliveleger]

@jnm,

Let's say, we have a form with two questions (select one), not mandatory:

Favorite coffee:

• Nespresso
• Keurig
• Tim Horton

Type of coffee:

• Regular
• Espresso
• Latte

Three users submit data (first case):
First one:
- Favorite coffee: Nespresso
- Type of coffee: Regular

Second one:
- Favorite coffee: Tim Horton
- Type of coffee: Latte

Third one:
- Favorite coffee: Keurig
- Type of coffee: No choices checked

Disaggregating the stats by grouping by Favorite coffee should return 3 Counters

• None => {'keuring': 1}
• Latte => {'tim_horton': 1}
• Regular => {'nespresso': 1}

Now, a fourth user submits this (second case):
- Favorite coffee: No choices checked
- Type of coffee: latte

It should still return the 3 sames counters, except that Latte Counter should have changed for
{'None': 1, 'tim_horton': 1}

Having that said, the if condition handles the first case, the else handles the second one.

[jnm]

@noliveleger, sorry for abandoning this for so long. I'm trying once again to understand it, but I've found that the unit test passes without any of the other changes:

(e) john@world:~/Documents/kobo/dockdev/formpack$ git log origin/master...
commit 95fe7eeda02130df5e476a9421f78ddbb4eca161 (HEAD -> cherry-pick-191-test, origin/cherry-pick-191-test, master)
Author: Olivier Léger <olivierleger@gmail.com>
Date:   Mon Feb 11 16:02:57 2019 -0500

    Added a unittest for disaggregated stats with empty answers
(e) john@world:~/Documents/kobo/dockdev/formpack$ pytest -v | grep test_disaggregate_without
tests/test_autoreport.py::TestAutoReport::test_disaggregate_without_answers PASSED [ 22%]

Here is the branch with the unit test commit cherry-picked on top of master: https://github.com/kobotoolbox/formpack/tree/cherry-pick-191-test.

Did we implement a fix for #190 somewhere along the line after you opened this PR, or does the test need to be modified?

[noliveleger]

@jnm, no. Unfortunately, my test is totally wrong. The issue is that total_count exceeds the number of submissions when data is submitted with null values. It does not test this at all. The fixture seems wrong too. Let me correct that.

[joshuaberetta]

@noliveleger what's the status of this PR?

[noliveleger]

@noliveleger what's the status of this PR?

I've applied requested changes ;-)
It's pretty old, we should discussed (as the back-end team) if it's still an improvement and merge if it is.