Skip to content

get_files(directory) is broken on both the S3 and local drivers #218

Description

@tmgbedu

Summary

get_files(directory) does not work on either driver, in two different ways. Listing a subdirectory is the common case (get_files("backups"), get_files("audio")), and neither driver handles it.

1. S3: passing a directory always returns an empty list

def get_files(self, directory=None):
    bucket = self.get_resource().Bucket(self.get_bucket())

    if directory:
        objects = bucket.objects.all().filter(Prefix=directory)
    else:
        objects = bucket.objects.all()

    files = []
    for my_bucket_object in objects.all():
        if "/" not in my_bucket_object.key:      # ← here
            files.append(File(my_bucket_object, my_bucket_object.key))

    return files

The filter is presumably meant to skip "subdirectories", but it tests the whole key, not the part below the prefix. Any key matched by a non-empty Prefix necessarily contains the prefix followed by /, so it is always discarded:

keys in bucket:              ['backups/a.dump', 'backups/b.dump', 'root.txt', 'audio/x/y.mp3']
matched by Prefix='backups': ['backups/a.dump', 'backups/b.dump']
kept by `"/" not in key`:    ['root.txt']          ← and root.txt is not even under the prefix

So get_files("backups") returns [] whenever the prefix is non-empty. Only a root-level listing returns anything.

Suggested fix

Strip the prefix before testing for a nested separator:

prefix = f"{directory.rstrip('/')}/" if directory else ""
for obj in objects.all():
    relative = obj.key[len(prefix):]
    if relative and "/" not in relative:
        files.append(File(obj, obj.key))

2. Local: file contents are read from the wrong path

def get_files(self, directory=""):
    file_path = self.get_path(directory)
    files = []
    for f in os.listdir(file_path):
        if not isfile(join(file_path, f)):
            continue

        files.append(File(self.get(f), f))     # ← here

    return files

os.listdir yields bare names, so for get_files("audio") the loop calls self.get("tts-1.mp3"). get() resolves through get_path(), which joins against the disk root, not root/audio. The file is looked up at root/tts-1.mp3, misses, and get() swallows the FileNotFoundError and returns None.

The result is a list of File objects with None content — no error, just silently empty files. It only appears to work when directory is empty.

Suggested fix

files.append(File(self.get(join(directory, f)), f))

Why it matters

The two drivers also disagree about what get_files means, which makes it unusable in disk-agnostic code: on local it lists files in a directory, on S3 it only ever lists the bucket root. Anything that needs "the most recent file under this prefix" has to drop to boto3 directly or maintain its own manifest.

A fake driver equivalent would be worth adding alongside, since FakeDriver currently has no get_files at all.

Environment

  • fastapi-startkit 0.56.0
  • Python 3.13.7

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions