Skip to content

Validate multiLine reads against runaway quote consumption #18

Description

@Iwan-Dyke

Problem

With multiLine=true (merged in #17), a malformed file with unmatched quotes causes Spark to silently swallow multiple lines into a single field. The result is fewer rows with bloated field values — no error raised, _corrupt_record not triggered.

This is the inverse of the #16 bug: previously corruption showed as extra rows, now it shows as missing rows with oversized fields.

Suggested approach

  • After read, compare actual row count against a raw newline count heuristic
  • Or flag any field value exceeding a configurable max length
  • Or both, behind a validate_multiline=True option

Acceptance criteria

  • A file with unmatched quotes produces a warning or error, not silent corruption
  • Validation does not break well-formed multiLine files
  • Behaviour is documented

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions