The complete reference for every script-facing builtin, client method, and
configuration accessor exposed by the s3 module. For an overview,
installation, and a quickstart, see the README.
The module exposes four top-level builtins via load("s3", …) —
create_client, validate_bucket_name, validate_object_key,
get_supported_services — plus a set of configuration accessors
(get_<key> / set_<key>) generated from the module's options. A
create_client(...) call returns a Client object that carries the bucket and
object methods (put_object, get_object, list_objects, presign_url, …).
Credentials are never passed from a script.
create_clientdoes not acceptaccess_key/secret_key/session_token; they are host-injected only (see Configuration). Passing them as keyword arguments raises anunexpected keyword argumenterror.
- Module functions
- Client object
- Client information
- URL generation
- Bucket operations
- Object operations
- Object-writing options
- Provider detection
- Configuration
These are the names registered by LoadModule, loadable via
load("s3", …).
create_client(service_type?, region?, endpoint?, force_path_style?, use_ssl?, timeout?, max_retries?, part_size?, concurrency?, enable_logging?, user_agent?)
Creates an S3 client. Every parameter is optional; an argument omitted falls
back to the corresponding module config option (see
Configuration). If service_type is "auto" or empty, the
provider is auto-detected (see Provider detection).
Credentials are not accepted here. access_key, secret_key, and
session_token are host-injected only; passing them raises an unexpected keyword argument error.
Parameters:
service_type(string): S3 service type ("aws","minio","cloudflare", …);"auto"(the default) enables provider detection.region(string): S3 region.endpoint(string): Custom S3 endpoint URL.force_path_style(bool): Force path-style addressing.use_ssl(bool): Use SSL/TLS for connections.timeout(int): Per-request timeout in seconds; bounds each HTTP request to the service.max_retries(int): Maximum attempts per request, including the first try (so3means up to two retries), default3. Passing0here means "use the configured default" (like any unset numeric argument), so it resolves to3— it does not mean zero retries. The effective value is always applied and takes precedence over an ambientAWS_MAX_ATTEMPTS. To instead defer to the SDK/env default, set the module option itself to0(S3_MAX_RETRIES=0orset_max_retries(0)), which is left unset on the client.part_size(int): Multipart upload part size in bytes.concurrency(int): Number of concurrent operations.enable_logging(bool): Enable SDK request/response logging.user_agent(string): Custom user-agent token appended to the SDK user agent.
Returns: A Client object (see Client object).
Errors: Fails on an invalid configuration, or if the underlying AWS SDK client cannot be constructed.
Example:
load("s3", "create_client")
# Region is enough for AWS auto-detection; credentials come from the host.
client = create_client(region="us-west-2")
# MinIO via an explicit endpoint
minio = create_client(service_type="minio", endpoint="localhost:9000", use_ssl=False)
# Cloudflare R2 (auto-detected from the endpoint)
r2 = create_client(endpoint="https://<account>.r2.cloudflarestorage.com")Reports whether name is a valid S3 bucket name. A valid name is 3–63
characters, starts and ends with a lowercase letter or number, contains only
lowercase letters, numbers, dots, and hyphens, has no consecutive dots, is not
formatted as an IP address, and does not start with xn-- or end with
-s3alias.
Parameters:
name(string): Bucket name to validate.
Returns: bool — True if the name is valid, else False.
Example:
load("s3", "validate_bucket_name")
print(validate_bucket_name("my-bucket")) # True
print(validate_bucket_name("My_Bucket")) # False (uppercase + underscore)Reports whether key is a valid S3 object key. A valid key is non-empty, at
most 1024 bytes long, and contains no ASCII control characters in the range
0x00–0x0F.
Parameters:
key(string): Object key to validate.
Returns: bool — True if the key is valid, else False.
Example:
load("s3", "validate_object_key")
print(validate_object_key("docs/report.pdf")) # True
print(validate_object_key("")) # False (empty)Lists the supported service-type strings the module knows.
Parameters: None
Returns: list[str] of service-type strings (e.g. "aws", "minio",
"cloudflare").
Example:
load("s3", "get_supported_services")
for service in get_supported_services():
print(service)The value returned by create_client is a Client. It is immutable, truthy,
and unhashable (cannot be used as a dict key). It exposes the methods below via
attribute access (e.g. client.put_object(...)).
Returns the client's effective configuration. Secret values are never
exposed — only *_set booleans report whether each credential is present.
Parameters: None
Returns: A struct with the fields: service_type, region, endpoint,
force_path_style, use_ssl, timeout, max_retries, part_size,
concurrency, enable_logging, user_agent, plus access_key_set,
secret_key_set, and session_token_set booleans.
Example:
info = client.get_client_info()
print(info.service_type, info.region, info.endpoint)
print("credentials present:", info.access_key_set and info.secret_key_set)Builds a public HTTP(S) URL for an object using the client's own region /
endpoint / use_ssl / service_type configuration. (No request is made; the
URL is constructed from config.)
Parameters:
bucket(string): Bucket name.key(string): Object key.
Returns: str — the public URL.
Example:
public = client.get_public_url("my-bucket", "hello.txt")Generates a pre-signed URL valid for expires_in seconds.
Parameters:
bucket(string): Bucket name.key(string): Object key.expires_in(int): Validity in seconds (default:3600).method(string):"GET","PUT", or"HEAD"(case-insensitive; default:"GET").
Returns: str — the pre-signed URL.
Errors: Fails if method is anything other than GET, PUT, or HEAD,
or if presigning fails.
Example:
signed = client.presign_url("my-bucket", "hello.txt", expires_in=3600)
upload = client.presign_url("my-bucket", "upload.bin", method="PUT")Creates a bucket. When region is omitted, the client's configured region is
used.
Parameters:
bucket(string): Bucket name.region(string, optional): Region in which to create the bucket.
Returns: None
Example:
client.create_bucket("my-bucket")
client.create_bucket("eu-bucket", region="eu-west-1")Deletes a bucket. With force=True, all of the bucket's objects are deleted
first. If any object cannot be deleted (object lock, governance, permissions),
the call fails with an error naming the first failure rather than reporting a
false success — the bucket is left intact so the condition is not silently
swallowed.
Parameters:
bucket(string): Bucket name.force(bool): IfTrue, delete the bucket's objects before deleting the bucket (default:False).
Returns: None
Example:
client.delete_bucket("temp-bucket", force=True)Lists the buckets in the account.
Parameters: None
Returns: list[dict] — one dict per bucket (the bucket-info fields below).
Example:
for bucket in client.list_buckets():
print(bucket["name"], bucket["creation_date"])Reports whether a bucket exists.
Parameters:
bucket(string): Bucket name.
Returns: bool
Example:
if client.bucket_exists("my-bucket"):
print("exists")Returns comprehensive bucket information. Several fields are gathered with best-effort calls (versioning, encryption, CORS, policy, tags); a provider that lacks a feature simply omits the corresponding data rather than failing.
Parameters:
bucket(string): Bucket name.
Returns: dict with the fields name, creation_date, region,
location, versioning_status, public_access_blocked, has_policy,
has_cors, encryption_enabled, encryption_type, object_count,
total_size, storage_class, tags, owner, bucket_type.
Example:
info = client.get_bucket_info("my-bucket")
print(info["versioning_status"], info["encryption_enabled"])Uploads an object from an in-memory string.
Parameters:
bucket(string): Bucket name.key(string): Object key.content(string): Object body.**options: Object-writing options (see Object-writing options).
Returns: None
Example:
client.put_object("my-bucket", "hello.txt", "Hello, World!")
client.put_object("my-bucket", "report.json", body, content_type="application/json")Uploads an object by reading a local file. (One of the module's two local-filesystem touch points.)
Parameters:
bucket(string): Bucket name.key(string): Object key.file_path(string): Path to the local file to upload.**options: Object-writing options (see Object-writing options).
Returns: None
Example:
client.put_object_file("my-bucket", "images/logo.png", "/tmp/logo.png",
content_type="image/png")Downloads an object and returns its body as a string.
Parameters:
bucket(string): Bucket name.key(string): Object key.
Returns: str — the object body.
Example:
content = client.get_object("my-bucket", "hello.txt") # => "Hello, World!"Downloads an object and writes it to a local file. (The module's second local-filesystem touch point.)
Parameters:
bucket(string): Bucket name.key(string): Object key.file_path(string): Destination path for the downloaded object.
Returns: None
Example:
client.get_object_file("my-bucket", "images/logo.png", "/tmp/logo.png")Deletes an object.
Parameters:
bucket(string): Bucket name.key(string): Object key.
Returns: None
Example:
client.delete_object("my-bucket", "obsolete.txt")Lists objects in a bucket. The object list is returned directly (not wrapped in a result envelope).
Parameters:
bucket(string): Bucket name.prefix(string): Only return keys beginning with this prefix (default: all).delimiter(string): Group keys by this delimiter (default: none).max_keys(int): Maximum number of items to return in total (default:1000). S3 returns at most 1000 items per request, so a largermax_keysis satisfied by transparently paginating and concatenating pages until the total is reached — you are not silently capped at one page. Memory use is bounded bymax_keys, so raise it deliberately for very large buckets. When adelimiteris set, grouped common prefixes count toward this total alongside objects (S3 counts them together).
Returns: list[dict] — one dict per object (the object-info fields below).
Note: the flat list carries objects only. When
delimiteris set, the groupedcommon_prefixes(the pseudo-folders) are not returned by this method; list without a delimiter to enumerate the keys themselves.
Example:
for obj in client.list_objects("my-bucket", prefix="docs/"):
print(obj["key"], obj["size"])Reports whether an object exists.
Parameters:
bucket(string): Bucket name.key(string): Object key.
Returns: bool
Example:
if client.object_exists("my-bucket", "hello.txt"):
print("present")Returns object metadata.
Parameters:
bucket(string): Bucket name.key(string): Object key.
Returns: dict with the fields key, size, last_modified, etag,
content_type, content_encoding, content_disposition, content_language,
cache_control, expires, storage_class, checksum_algorithm, version_id,
is_latest, owner, metadata, tags. For list_objects entries
checksum_algorithm carries the object's checksum algorithm (e.g. "SHA256")
when present; version_id there is empty because a plain object listing carries
no version — it is populated only where the API returns one (e.g.
get_object_info).
Example:
info = client.get_object_info("my-bucket", "hello.txt")
print(info["size"], info["content_type"], info["etag"])Sets object metadata and properties in place (implemented as a self-copy with the metadata directive set to replace).
Parameters:
bucket(string): Bucket name.key(string): Object key.**options: Object-writing options (see Object-writing options).
Returns: None
Example:
client.set_object_info(
"my-bucket", "document.pdf",
content_type="application/pdf",
cache_control="max-age=3600",
metadata={"author": "Ada", "version": "1.0"},
tags={"project": "alpha"},
)Copies an object from one location to another. When any object-writing option is supplied, the metadata directive is set to replace.
Parameters:
src_bucket(string): Source bucket name.src_key(string): Source object key.dst_bucket(string): Destination bucket name.dst_key(string): Destination object key.**options: Object-writing options (see Object-writing options).
Returns: None
Example:
client.copy_object(
"src-bucket", "src/file.txt",
"dst-bucket", "dst/file.txt",
metadata={"copied": "true"},
)The object-writing methods — put_object, put_object_file, set_object_info,
and copy_object — accept these optional keyword arguments:
| Option | Type | Description |
|---|---|---|
content_type |
string | MIME type of the object |
metadata |
dict | User-defined metadata (string keys and values) |
tags |
dict | Object tags (string keys and values) |
cache_control |
string | Cache-Control header |
content_disposition |
string | Content-Disposition header |
content_encoding |
string | Content-Encoding header |
content_language |
string | Content-Language header |
expires |
string | Expiry timestamp as an RFC 3339 string |
Provider feature support varies (for example, Cloudflare R2 does not support S3
object tagging). The module degrades gracefully and surfaces clear errors for
operations a provider does not support; branch on
client.get_client_info().service_type when needed.
Omit service_type (or set it to "auto") to let the client detect the
provider from the endpoint, region, or host-injected access key. Detection runs
a priority-ordered rule engine:
| Priority | Signal | Examples |
|---|---|---|
| Highest | Endpoint pattern | amazonaws.com→AWS, r2.cloudflarestorage.com→R2, digitaloceanspaces.com→DigitalOcean, wasabisys.com→Wasabi, backblazeb2.com→Backblaze, aliyuncs.com→Alibaba |
| High | Special region | region="auto"→Cloudflare R2 |
| Medium | Access-key pattern | AKIA…/ASIA…→AWS, 32-char hex→R2 (host key, not script) |
| Lower | Region format | us-west-2→AWS, nyc3/fra1→DigitalOcean |
| Lowest | Endpoint shape | localhost:9000→MinIO, min.io domain→MinIO |
get_supported_services() returns the service-type strings the module knows:
| Service | Service type | Default region |
|---|---|---|
| Amazon S3 | "aws" |
us-east-1 |
| MinIO | "minio" |
us-east-1 |
| DigitalOcean Spaces | "digitalocean" |
nyc3 |
| Linode Object Storage | "linode" |
us-east-1 |
| Wasabi | "wasabi" |
us-east-1 |
| Backblaze B2 | "backblaze" |
us-west-000 |
| Cloudflare R2 | "cloudflare" |
auto |
| Scaleway | "scaleway" |
fr-par |
| Alibaba Cloud OSS | "alibaba" |
cn-hangzhou |
| Google Cloud Storage | "google" |
us-central1 |
| Oracle Cloud | "oracle" |
us-ashburn-1 |
| IBM Cloud | "ibm" |
us-south |
if client.get_client_info().service_type == "aws":
client.put_object(bucket, key, content, tags={"env": "prod"})
else:
client.put_object(bucket, key, content) # e.g. R2 has no taggingEach module configuration option is exposed to scripts as generated accessor
builtins (loaded from the s3 module alongside the functions above):
get_<key>()— returns the current value of the option.set_<key>(value)— sets the option (returnsNone).
An option's value resolves in priority order: an explicit set_<key> value, the
environment variable (S3_<KEY>), then the default. The non-secret options
are also accepted as keyword arguments to create_client(...) (a
script-supplied value overrides the module default).
Secret options are set-only. The three credential options — access_key,
secret_key, session_token — are secret: they expose only a set_<key>
accessor and no getter (so a secret value can never be read back from a
script). They are also not accepted as create_client(...) keyword
arguments (passing them raises an unexpected keyword argument error). Provide
credentials via the S3_ACCESS_KEY / S3_SECRET_KEY / S3_SESSION_TOKEN
environment variables, the host config, or the AWS default credential chain.
client.get_client_info() reports only access_key_set / secret_key_set /
session_token_set booleans, never the values.
Host-only file-access policy. file_root and allow_unsafe_file_paths govern
which local paths put_object_file / get_object_file may touch. They are
host-only: only their getters (get_file_root, get_allow_unsafe_file_paths)
are generated — set_file_root and set_allow_unsafe_file_paths are
intentionally NOT exposed, so a script cannot widen its own filesystem reach.
By default those two methods confine every local path under file_root (empty =
the process working directory), rejecting any path that escapes it via .. or a
symlink; an "absolute" path is re-anchored under the root rather than reaching the
real host path. allow_unsafe_file_paths=true is the host's explicit opt-out that
disables the confinement.
max_object_size is a third host-only lever: it bounds how many bytes
get_object reads into memory, so a hostile or oversized object can't exhaust
host memory. Only get_max_object_size is generated (no set_max_object_size);
0 disables the limit and the default is 256 MiB; a negative value is a
misconfiguration and falls back to the default (fail-safe, never fail-open).
| Option | Getter | Setter | Type | Env var | Default | Description |
|---|---|---|---|---|---|---|
service_type |
get_service_type |
set_service_type |
string | S3_SERVICE_TYPE |
"auto" |
S3 service type (aws, minio, cloudflare, …); "auto" enables detection |
access_key |
(secret — none) | set_access_key |
string | S3_ACCESS_KEY |
"" |
Access key ID (secret; host-injected, set-only, not a create_client kwarg) |
secret_key |
(secret — none) | set_secret_key |
string | S3_SECRET_KEY |
"" |
Secret access key (secret; host-injected, set-only, not a create_client kwarg) |
session_token |
(secret — none) | set_session_token |
string | S3_SESSION_TOKEN |
"" |
Session token for temporary credentials (secret; host-injected, set-only, not a create_client kwarg) |
region |
get_region |
set_region |
string | S3_REGION |
"us-east-1" |
S3 region |
endpoint |
get_endpoint |
set_endpoint |
string | S3_ENDPOINT |
"" |
Custom S3 endpoint URL |
force_path_style |
get_force_path_style |
set_force_path_style |
bool | S3_FORCE_PATH_STYLE |
false |
Force path-style addressing |
use_ssl |
get_use_ssl |
set_use_ssl |
bool | S3_USE_SSL |
true |
Use SSL/TLS for connections |
timeout |
get_timeout |
set_timeout |
int | S3_TIMEOUT |
30 |
Per-request timeout in seconds |
max_retries |
get_max_retries |
set_max_retries |
int | S3_MAX_RETRIES |
3 |
Maximum attempts per request, incl. the first try |
part_size |
get_part_size |
set_part_size |
int | S3_PART_SIZE |
5242880 |
Multipart upload part size in bytes (5 MiB) |
concurrency |
get_concurrency |
set_concurrency |
int | S3_CONCURRENCY |
3 |
Number of concurrent operations |
enable_logging |
get_enable_logging |
set_enable_logging |
bool | S3_ENABLE_LOGGING |
false |
Enable debug logging |
user_agent |
get_user_agent |
set_user_agent |
string | S3_USER_AGENT |
"Starlark-S3/1.0" |
Custom user agent string |
file_root |
get_file_root |
set_file_root — host-only, not generated |
string | S3_FILE_ROOT |
"" |
Root that put_object_file/get_object_file paths are confined under ("" = working directory) |
allow_unsafe_file_paths |
get_allow_unsafe_file_paths |
set_allow_unsafe_file_paths — host-only, not generated |
bool | S3_ALLOW_UNSAFE_FILE_PATHS |
false |
Disable the file_root confinement (host opt-out) |
max_object_size |
get_max_object_size |
set_max_object_size — host-only, not generated |
int | S3_MAX_OBJECT_SIZE |
268435456 |
Max bytes get_object reads into memory (256 MiB; 0 = unlimited) |
The env var for any option is S3_ + the option name uppercased.
Example:
load(
"s3",
"create_client",
# non-secret getters
"get_service_type", "get_region", "get_endpoint", "get_force_path_style",
"get_use_ssl", "get_timeout", "get_max_retries", "get_part_size",
"get_concurrency", "get_enable_logging", "get_user_agent",
# non-secret setters
"set_service_type", "set_region", "set_endpoint", "set_force_path_style",
"set_use_ssl", "set_timeout", "set_max_retries", "set_part_size",
"set_concurrency", "set_enable_logging", "set_user_agent",
# secret setters (no getters)
"set_access_key", "set_secret_key", "set_session_token",
)
set_region("eu-west-1")
print(get_region()) # "eu-west-1"
client = create_client() # uses region=eu-west-1