Skip to content

Distinct option doesn't work with a small size of file #2

Description

@MamoruAsagami

When and an input file is small enough to fit in a single chunk, distinct option doesn't remove duplicated items.

The reason seems because ExternalMergeSort#mergeSortedChunksNoPartialMerge(List sortedChunks) doesn't honor config.distinct as you see below.

    if (sortedChunks.size() == 1) {
        File sortedChunk = sortedChunks.get(0);
        return new ChunkFile<T>(sortedChunk, serializer, comparator, config.cleanup);
    } else {
        List<ChunkFile<T>> cfs = new ArrayList<ChunkFile<T>>(sortedChunks.size());
        for  (File file : sortedChunks) {
            cfs.add(new ChunkFile<T>(file, serializer, comparator, config.cleanup));
        }
        return new MergeSortedIterator<T,ChunkFile<T>>(cfs, comparator, config.distinct);
    }

It works fine if you eliminate the above optimizing if-true section.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions