Fix reading records when additional fields exist and should be skipped - #179
Merged
AdrianStrugala merged 2 commits intoJul 14, 2026
Conversation
Contributor
Author
|
@AdrianStrugala Appreciate if you can take a look at this, as we are currently experiencing issues with this in our systems. Thanks! |
Contributor
Author
|
Hi @AdrianStrugala sorry to ping you but we'd really like to get this resolved in case we have other mismatched schemas like this in our systems. Any chance you can expedite this one? |
Owner
|
Hello @robertcoltheart, I have just merged your PR and released a new version |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes: #151 (possibly)
Not sure if this fixes the above issue, but we encountered exactly this error recently.
Setup
We have a write schema which has an additional field in an array schema. This field has been added in the middle of the other fields, which is a valid Avro schema change. The array has multiple items in it. Our read schema does not have this field, and we expect the field to be skipped correctly.
Behavior observed
The first array item to be read is processed correctly, and the read steps are cached. However the 2nd read, which uses those cached steps, aborts on the first occurrence of a "skipped" field, and leaves the rest of the byte stream for the array item untouched. A subsequent read of any field results in incorrectly parsed values and exceptions. In our case we saw stack overflow, but also saw other "bad format" etc exceptions as well.
Fix
The fix is to continue reading the fields when a "skipped" field is encountered. This is already the behavior for 'first reads' of an object, this just mirrors the behavior for the cached steps as well. I've also added a very specific unit test which tests this behavior.