Repository navigation
More Tweaks - #195
Draft
mukunku wants to merge 34 commits into
Draft
More Tweaks#195mukunku wants to merge 34 commits into
mukunku wants to merge 34 commits into
Conversation
… plus better list of struct handling (LIST_OF_NESTED_STRUCTS_TEST now works for ParquetNET!)
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
Nested-data loss, export correctness issues, and undisposed preview images remain unresolved.
4 open issues
What changed in this PR
Expands ParquetNET’s support for nested data and previously skipped files, alongside viewer usability and error-handling updates.
Changes:
- Reworks nested list, struct, and map decoding and preserves unsigned array types.
- Adds regression tests and removes remaining ParquetNET file-test skips.
- Updates date-format validation, preview reuse, loading controls, and exception handling.
| File | Description |
|---|---|
| src/ParquetViewer/MainForm.Helpers.cs | Updates loading controls and export error handling. |
| src/ParquetViewer/MainForm.cs | Handles memory and access failures. |
| src/ParquetViewer/Helpers/UtilityMethods.cs | Validates and cleans date/time formats. |
| src/ParquetViewer/Helpers/SemanticVersion.cs | Refines version equality and comparison contracts. |
| src/ParquetViewer/Helpers/ExtensionMethods.cs | Returns wrapped Hugging Face image data. |
| src/ParquetViewer/CustomDateFormatInputForm.cs | Uses stricter format validation. |
| src/ParquetViewer/Controls/ParquetGridView.cs | Reuses and restores preview windows. |
| src/ParquetViewer/AppSettings.cs | Validates stored custom formats. |
| src/ParquetViewer.Tests/MetadataViewerTests.cs | Updates struct expectations and assertions. |
| src/ParquetViewer.Tests/ListValueBuilderTests.cs | Tests nested null and empty lists. |
| src/ParquetViewer.Tests/HelperTests.cs | Checks formatting across cultures. |
| src/ParquetViewer.Tests/EngineTests.cs | Adds regressions and removes engine skips. |
| src/ParquetViewer.Engine.ParquetNET/Types/StructValue.cs | Removes the engine-specific struct wrapper. |
| src/ParquetViewer.Engine.ParquetNET/ParquetViewer.Engine.ParquetNET.csproj | Retains the Types folder entry. |
| src/ParquetViewer.Engine.ParquetNET/ParquetSchemaElement.cs | Caches levels and updates struct classification. |
| src/ParquetViewer.Engine.ParquetNET/ParquetEngine.Processor.cs | Reworks nested collection reconstruction. |
| src/ParquetViewer.Engine.ParquetNET/ParquetColumnData.cs | Preserves signed and unsigned array types. |
| src/ParquetViewer.Engine.ParquetNET/MapValueBuilder.cs | Adds recursive map reconstruction. |
| src/ParquetViewer.Engine.ParquetNET/LogicalTypeMetadataResolver.cs | Extracts logical-type metadata conversion. |
| src/ParquetViewer.Engine.ParquetNET/ListValueBuilder.cs | Revises nested-list level handling. |
| src/ParquetViewer.Engine.ParquetNET/Helpers.cs | Removes obsolete level helpers. |
| src/ParquetViewer.Engine.ParquetNET/FieldLevels.cs | Centralizes repetition and definition levels. |
Review effort: Balanced
| var valueColumn = await ReadColumnAsync(groupReader, valueField, cancellationToken); | ||
| return new MapValueBuilder(parentRepetitionBoundary, keyField.Levels, keyColumn.RepetitionLevels, | ||
| keyColumn.DefinitionLevels, keyValues, keyField.ClrType, valueField.ClrType, | ||
| valueValues: valueColumn.Data.ToArray()); |
| rows.Add(currentRow); | ||
|
|
||
| var dataEnumerable = Engine.Helpers.PairEnumerables(keyDataEnumerable, valueDataEnumerable, DBNull.Value); | ||
| return rows.Skip((int)skipRecords).Take((int)readRecords).ToList(); |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR contains some changes to the ParquetNET engine to support more files.
This PR also has some other tweaks. I'll probably release this as v4.3.1 soon