Repository navigation
bug: @egen and @collapse can not summarize over missing values in columns? #190
Description
When I use columns that have missing values to calculate something in a groupby it seems to throw error. E.g:
On the above df if I run
@with df @egen birt_year = minimum(birth_year), by(person_id)I get
ERROR: MethodError: reducing over an empty collection is not allowed; consider supplying `init` to the reducerBut if I run
@with df begin
@mvencode birth_year, mv(99999)
@egen byear = minimum(birth_year), by(person_id)
endIt throws no error.
Activity
korenmiklos commentedon Aug 13, 2024
The issue is that all values in a group are missing. So when we do
skipmissing, the result will be an empty vector. We may need to special case this. I think we need to returnmissingin all these cases.korenmiklos commentedon Aug 5, 2026
Related: I hit the same root cause from the other side in #233 —
meanandsumreturnNaNand0.0on an all-missing group rather than erroring, so the failure is silent there.Both look like the same underlying thing: the output column is typed from the non-missing element type (
Union{Missing,Float64}in,Float64out), somissingcannot be represented. Widening the result when the input is missing-typed would letminimum/maximumreturnmissinginstead of needing aninit, and fix the silent cases at the same time.Return missing from aggregators when a group has no usable values