BUG: Failure to parse BirdNet live raven format

[blaiseli]

Describe the bug

I came across crowsetta while searching for a way to convert the ".selections.txt" file exported by the BirdNet Live smartphone app (which, according to the documentation, is supposed to be in raven pro format) into something that I could load in Audacity. However Transcriber.from_file fails as follows:

SchemaError: column 'annotation' not in dataframe. Columns in dataframe: ['Selection', 'View', 'Channel', 'Begin File', 'begin_time_s', 'end_time_s', 'low_freq_hz', 'high_freq_hz', 'Common Name', 'Scientific Name', 'Confidence', 'Survey Time (UTC)', 'Confirmed', 'Confirmed At (UTC)']

To Reproduce
Steps to reproduce the behavior:

I have only tested in one case. I used the .selections.txt found in the zip archive exported from a BirdNet Live session and tried to load it as follows:

scribe = crowsetta.Transcriber(format="raven")
annots = scribe.from_file(<path to my .selections file>)

Desktop (please complete the following information):

• Operating System: Linux
• Version 5.1.2

Additional context

This is my first attempt at using crowsetta, and I only partially looked at the tutorial. I may not be using it as intended.

[NickleDave]

Hi @blaiseli thank you for raising this issue and for the clear explanation of the problem you're running into.

I think it should work if you specify an annot_col (the name of the column with the annotations).

Can you try like so?

import crowsetta

scribe = crowsetta.Transcriber(format="raven")
raven = scribe.from_file("/path/to/your/selections.txt", annot_col="Common Name")

Please let me know if that works.

If you still get an exception raised, please reply with the entire snippet of what you ran as well as the entire traceback, and please if possible attach the selections.txt file (you will probably have to compress it in a zip or something like that).

Note that my snippet will rename the "Common Name" column to annotation, and throw away the other columns. Looking at this again, I am not super satisfied with this design; I think it would be better to only validate that the columns required for the bounding boxes are present, and then just allow all other columns to remain with their given names. I'm open to making that change if you would find it useful.

[blaiseli]

Hello, and thanks for your answer.

What you suggested enabled me to go one step further.

Now I encounter another error:

SchemaError: expected series 'low_freq_hz' to have type float64, got int64

Here is my code:

selections = "BirdNET_Live_2026-08-24_09-39-11_#125.selections.txt"
import crowsetta
scribe = crowsetta.Transcriber(format="raven")
annots = scribe.from_file(selections, annot_col="Common Name")

And here is my selections.txt file:

BirdNET_Live_2026-08-24_09-39-11_#125.selections.txt

The full traceback:

---------------------------------------------------------------------------
SchemaError                               Traceback (most recent call last)
Cell In[4], line 1
----> 1 annots = scribe.from_file(selections, annot_col="Common Name")

File ~/Documents/prive/Divers/Nature/Oiseaux/bird_venv_2/lib/python3.11/site-packages/crowsetta/transcriber.py:203, in Transcriber.from_file(self, annot_path, *args, **kwargs)
    143 def from_file(
    144     self, annot_path, *args, **kwargs
    145 ) -> "Union[crowsetta.interface.SeqLike,crowsetta.interface.BBoxLike]":  # noqa: F821
    146     """Load annotations from a file.
    147 
    148     Parameters
   (...)    201     >>> generic_seq.to_csv('./data/bfsongrepo/notmats.csv')
    202     """
--> 203     return self._format_class.from_file(annot_path, *args, **kwargs)

File ~/Documents/prive/Divers/Nature/Oiseaux/bird_venv_2/lib/python3.11/site-packages/crowsetta/formats/bbox/raven.py:127, in Raven.from_file(cls, annot_path, annot_col, audio_path)
    125 columns_map.update({annot_col: "annotation"})
    126 df.rename(columns=columns_map, inplace=True)
--> 127 df = RavenSchema.validate(df)
    129 return cls(
    130     df=df,
    131     annot_path=annot_path,
    132     annot_col=annot_col,
    133     audio_path=audio_path,
    134 )

File ~/Documents/prive/Divers/Nature/Oiseaux/bird_venv_2/lib/python3.11/site-packages/pandera/api/pandas/model.py:241, in DataFrameModel.validate(cls, check_obj, head, tail, sample, random_state, lazy, inplace)
    226 @classmethod
    227 @docstring_substitution(validate_doc=BaseSchema.validate.__doc__)
    228 def validate(
   (...)    236     inplace: bool = False,
    237 ) -> DataFrame[Self]:
    238     """%(validate_doc)s"""
    239     return cast(
    240         DataFrame[Self],
--> 241         cls.to_schema().validate(
    242             check_obj, head, tail, sample, random_state, lazy, inplace
    243         ),
    244     )

File ~/Documents/prive/Divers/Nature/Oiseaux/bird_venv_2/lib/python3.11/site-packages/pandera/api/pandas/container.py:178, in DataFrameSchema.validate(self, check_obj, head, tail, sample, random_state, lazy, inplace)
    166     check_obj = check_obj.map_partitions(  # type: ignore [operator]
    167         self._validate,
    168         head=head,
   (...)    174         meta=check_obj,
    175     )
    176     return check_obj.pandera.add_schema(self)
--> 178 return self._validate(
    179     check_obj=check_obj,
    180     head=head,
    181     tail=tail,
    182     sample=sample,
    183     random_state=random_state,
    184     lazy=lazy,
    185     inplace=inplace,
    186 )

File ~/Documents/prive/Divers/Nature/Oiseaux/bird_venv_2/lib/python3.11/site-packages/pandera/api/pandas/container.py:198, in DataFrameSchema._validate(self, check_obj, head, tail, sample, random_state, lazy, inplace)
    188 def _validate(
    189     self,
    190     check_obj: pd.DataFrame,
   (...)    196     inplace: bool = False,
    197 ) -> pd.DataFrame:
--> 198     return self.get_backend(check_obj).validate(
    199         check_obj,
    200         schema=self,
    201         head=head,
    202         tail=tail,
    203         sample=sample,
    204         random_state=random_state,
    205         lazy=lazy,
    206         inplace=inplace,
    207     )

File ~/Documents/prive/Divers/Nature/Oiseaux/bird_venv_2/lib/python3.11/site-packages/pandera/backends/pandas/container.py:116, in DataFrameSchemaBackend.validate(self, check_obj, schema, head, tail, sample, random_state, lazy, inplace)
    111 components = self.collect_schema_components(
    112     check_obj, schema, column_info
    113 )
    115 # run the checks
--> 116 error_handler = self.run_checks_and_handle_errors(
    117     error_handler,
    118     schema,
    119     check_obj,
    120     column_info,
    121     sample,
    122     components,
    123     lazy,
    124     head,
    125     tail,
    126     random_state,
    127 )
    129 if error_handler.collected_errors:
    130     if getattr(schema, "drop_invalid_rows", False):
    131         # if the failure cases are a string, it means the error is
    132         # a schema-level error.

File ~/Documents/prive/Divers/Nature/Oiseaux/bird_venv_2/lib/python3.11/site-packages/pandera/backends/pandas/container.py:203, in DataFrameSchemaBackend.run_checks_and_handle_errors(self, error_handler, schema, check_obj, column_info, sample, components, lazy, head, tail, random_state)
    192         else:
    193             error = SchemaError(
    194                 schema,
    195                 data=check_obj,
   (...)    201                 reason_code=result.reason_code,
    202             )
--> 203         error_handler.collect_error(
    204             get_error_category(result.reason_code),
    205             result.reason_code,
    206             error,
    207             result.original_exc,
    208         )
    210 return error_handler

File ~/Documents/prive/Divers/Nature/Oiseaux/bird_venv_2/lib/python3.11/site-packages/pandera/api/base/error_handler.py:117, in ErrorHandler.collect_error(self, error_type, reason_code, schema_error, original_exc)
    110 """Collect schema error, raising exception if lazy is False.
    111 
    112 :param error_type: type of error
    113 :param reason_code: string representing reason for error
    114 :param schema_error: ``SchemaError`` object.
    115 """
    116 if not self._lazy:
--> 117     raise schema_error from original_exc
    119 # delete data of validated object from SchemaError object to prevent
    120 # storing copies of the validated DataFrame/Series for every
    121 # SchemaError collected.
    122 if hasattr(schema_error, "data"):

File ~/Documents/prive/Divers/Nature/Oiseaux/bird_venv_2/lib/python3.11/site-packages/pandera/backends/pandas/container.py:238, in DataFrameSchemaBackend.run_schema_component_checks(self, check_obj, schema, schema_components, lazy)
    234     # disable coercion at the schema component level since the
    235     # dataframe-level schema already coerced it.
    236     schema_component.coerce = False  # type: ignore
--> 238     result = schema_component.validate(
    239         check_obj, lazy=lazy, inplace=True
    240     )
    242     check_passed.append(is_table(result))
    243 except SchemaError as err:

File ~/Documents/prive/Divers/Nature/Oiseaux/bird_venv_2/lib/python3.11/site-packages/pandera/api/dataframe/components.py:148, in ComponentSchema.validate(self, check_obj, head, tail, sample, random_state, lazy, inplace)
    120 def validate(
    121     self,
    122     check_obj,
   (...)    128     inplace: bool = False,
    129 ):
    130     """Validate a series or specific column in dataframe.
    131 
    132     :check_obj: data object to validate.
   (...)    146 
    147     """
--> 148     return self.get_backend(check_obj).validate(
    149         check_obj,
    150         schema=self,
    151         head=head,
    152         tail=tail,
    153         sample=sample,
    154         random_state=random_state,
    155         lazy=lazy,
    156         inplace=inplace,
    157     )

File ~/Documents/prive/Divers/Nature/Oiseaux/bird_venv_2/lib/python3.11/site-packages/pandera/backends/pandas/components.py:154, in ColumnBackend.validate(self, check_obj, schema, head, tail, sample, random_state, lazy, inplace)
    148 if getattr(schema, "drop_invalid_rows", False):
    149     # replace the check_obj with the validated
    150     check_obj = validate_column(
    151         check_obj, column_name, return_check_obj=True
    152     )
--> 154 validated_column = validate_column(
    155     check_obj,
    156     column_name,
    157     return_check_obj=True,
    158 )
    159 if schema.parsers and validated_column is not None:
    160     if is_table(validated_column):

File ~/Documents/prive/Divers/Nature/Oiseaux/bird_venv_2/lib/python3.11/site-packages/pandera/backends/pandas/components.py:107, in ColumnBackend.validate.<locals>.validate_column(check_obj, column_name, return_check_obj)
    105 except SchemaError as err:
    106     err.column_name = column_name
--> 107     error_handler.collect_error(
    108         get_error_category(err.reason_code), err.reason_code, err
    109     )

File ~/Documents/prive/Divers/Nature/Oiseaux/bird_venv_2/lib/python3.11/site-packages/pandera/api/base/error_handler.py:117, in ErrorHandler.collect_error(self, error_type, reason_code, schema_error, original_exc)
    110 """Collect schema error, raising exception if lazy is False.
    111 
    112 :param error_type: type of error
    113 :param reason_code: string representing reason for error
    114 :param schema_error: ``SchemaError`` object.
    115 """
    116 if not self._lazy:
--> 117     raise schema_error from original_exc
    119 # delete data of validated object from SchemaError object to prevent
    120 # storing copies of the validated DataFrame/Series for every
    121 # SchemaError collected.
    122 if hasattr(schema_error, "data"):

File ~/Documents/prive/Divers/Nature/Oiseaux/bird_venv_2/lib/python3.11/site-packages/pandera/backends/pandas/components.py:81, in ColumnBackend.validate.<locals>.validate_column(check_obj, column_name, return_check_obj)
     77 try:
     78     # make sure the schema component mutations are reverted after
     79     # validation
     80     _orig_name = schema.name
---> 81     validated_check_obj = super(ColumnBackend, self).validate(
     82         check_obj,
     83         schema.set_name(column_name),
     84         head=head,
     85         tail=tail,
     86         sample=sample,
     87         random_state=random_state,
     88         lazy=lazy,
     89         inplace=inplace,
     90     )
     91     # revert the schema component mutations
     92     schema.name = _orig_name

File ~/Documents/prive/Divers/Nature/Oiseaux/bird_venv_2/lib/python3.11/site-packages/pandera/backends/pandas/array.py:136, in ArraySchemaBackend.validate(self, check_obj, schema, head, tail, sample, random_state, lazy, inplace)
    129     error_handler.collect_error(
    130         get_error_category(exc.reason_code),
    131         exc.reason_code,
    132         exc,
    133     )
    135 # run the core checks
--> 136 error_handler = self.run_checks_and_handle_errors(
    137     error_handler,
    138     schema,
    139     check_obj,
    140     head=head,
    141     tail=tail,
    142     sample=sample,
    143     random_state=random_state,
    144 )
    146 if lazy and error_handler.collected_errors:
    147     if getattr(schema, "drop_invalid_rows", False):

File ~/Documents/prive/Divers/Nature/Oiseaux/bird_venv_2/lib/python3.11/site-packages/pandera/backends/pandas/array.py:200, in ArraySchemaBackend.run_checks_and_handle_errors(self, error_handler, schema, check_obj, **subsample_kwargs)
    189         else:
    190             error = SchemaError(
    191                 schema=schema,
    192                 data=check_obj,
   (...)    198                 reason_code=result.reason_code,
    199             )
--> 200             error_handler.collect_error(
    201                 get_error_category(result.reason_code),
    202                 result.reason_code,
    203                 error,
    204                 original_exc=result.original_exc,
    205             )
    207 return error_handler

File ~/Documents/prive/Divers/Nature/Oiseaux/bird_venv_2/lib/python3.11/site-packages/pandera/api/base/error_handler.py:117, in ErrorHandler.collect_error(self, error_type, reason_code, schema_error, original_exc)
    110 """Collect schema error, raising exception if lazy is False.
    111 
    112 :param error_type: type of error
    113 :param reason_code: string representing reason for error
    114 :param schema_error: ``SchemaError`` object.
    115 """
    116 if not self._lazy:
--> 117     raise schema_error from original_exc
    119 # delete data of validated object from SchemaError object to prevent
    120 # storing copies of the validated DataFrame/Series for every
    121 # SchemaError collected.
    122 if hasattr(schema_error, "data"):

SchemaError: expected series 'low_freq_hz' to have type float64, got int64

[NickleDave]

Thank you for your detailed reply, and for sharing the file

I think what's happening here (in case it's not obvious) is that pandas parses low_freq_hz as int, and then when we validate the DataFrame with pandera, we get an error, because we require that column to be a float (and I'm guessing we'll have the same issue with high_freq_hz)

I'm an imposter DSP expert, but I'm pretty sure that FFT frequencies should be float, see: https://dsp.stackexchange.com/questions/67973/why-can-we-have-non-integer-frequency-bins-in-fft

So if the BirdNet app wants to be as pedantic as I'm being right now, they should save the csv with decimal points to make it very clear those values are floats. But that's an annoying implementation detail you shouldn't have to think about as a user.

So I think the fix will be to change: https://github.com/vocalpy/crowsetta/blob/main/src/crowsetta/formats/bbox/raven.py#L28C1-L29C57

    low_freq_hz: Series[float] = pandera.pandas.Field()
    high_freq_hz: Series[float] = pandera.pandas.Field()

to:

    low_freq_hz: Series[float] = pandera.pandas.Field(coerce=True)
    high_freq_hz: Series[float] = pandera.pandas.Field(coerce=True)

I'm a bit swamped right now but will try to test this out locally by the end of the week, and if it resolves this issue, I will release a patch version with this change