[Feature] Modernization: one requests transport with base_url, OAuth, adaptive rate limits and typed errors

CHANGELOG.md

@@ -7,6 +7,32 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+- BREAKING: Drop support for Python 3.8 and 3.9, which are end of life. The minimum supported version is now Python 3.10.
+- Add support for Python 3.14 and 3.15 ([#95](https://github.com/anvilco/python-anvil/issues/95)).
+- BREAKING: The `anvil` command line tool's dependencies, `click` and `tabulate`, are now the `cli` extra: `pip install 'python-anvil[cli]'` ([#96](https://github.com/anvilco/python-anvil/issues/96)). Importing `python_anvil` no longer imports the CLI.
+- BREAKING: One `requests`-based transport replaces `gql` for GraphQL and the `ratelimit` decorators for REST. Both dependencies are dropped.
+  - `Anvil(api_key, base_url=...)` sends every request to one host: REST to `{base_url}/api/...`, GraphQL to `{base_url}/graphql`. `base_url` defaults to production. The `endpoint_url` argument is removed, and `environment` does nothing: the rate limits now come from the API.
+  - OAuth access tokens are supported: `Anvil(access_token=...)`. JWTs are sent as-is, other tokens base64 encoded, like node-anvil.
+  - Rate limits are learned from the first response that carries the `X-RateLimit-Limit` and `X-RateLimit-Interval-Ms` headers. Requests made before the first response arrives wait for it. The client counts requests over a sliding window, like the API, and only waits when the limit has been reached. A request counts from when it is sent until an interval after its response arrives, so the API, which counts from when the request reached it, never sees more in its window than the client allowed. Every client in the process with the same `base_url` and credentials shares one limiter, and so the limits the API taught it; the starting `request_limit` only applies to the first of them. A `429` is retried after its `Retry-After` plus 50ms, or after a second when there is no usable `Retry-After`, for as long as it takes. The `retry` option is removed.
+  - Every request, REST included, is retried on `500`, `502`, `503` and `504`, and when it couldn't connect, up to `max_retries` times (5) with exponential backoff from 100ms, the statuses and count `gql`'s transport retried for GraphQL requests before. Mutations are retried too, as before. A request that was sent but got no complete response, like a read timeout, is not retried, since the API may have processed it; `gql` retried those too. `max_retries=0` turns it off.
+  - API errors raise `AnvilRequestError`, with the normalized `errors` (GraphQL errors, REST errors and validation errors with their `fields`), the `status_code`, the `headers` and any partial `data`. An error is any status outside 2xx, or a 2xx GraphQL response with `errors`; a non-JSON error, like a `502` from a proxy, keeps the start of its body. The REST methods used to raise a bare `Exception`; GraphQL methods raised `gql`'s errors. `AnvilRequestException` and `AnvilException` remain as aliases, and still accept a plain message, but their base class is now `Exception` rather than `BaseException`, so `except Exception` catches them.
+  - Methods return the data: the PDF or zip `bytes` for `fill_pdf`, `generate_pdf` and `download_documents`, and the response's `data` dict for the GraphQL methods. A JSON response to a binary method, like `fill_pdf` with `outputFormat: "url"`, is returned parsed. The `include_headers`, `debug` and `retry` keyword arguments are removed and now raise `TypeError`, as do the `gql` options `query` and `mutate` used to pass through; the headers are on `AnvilRequestError`.
+  - `data_type="stream"` on the binary methods returns the `requests.Response`, to stream a large PDF or zip.
+  - Requests time out after 5 minutes by default (`timeout`). They never did before.
+  - A `requests.Session` can be passed in (`session`), e.g. to use a proxy, and `user_agent`, `request_limit` and `request_limit_ms` match node-anvil's options.
+- BREAKING: `request_rest` now sends a request, `request_rest("POST", "/api/v1/...", json=...)`, like node-anvil's `requestREST`. A full URL is accepted if it is on the client's `base_url`, and refused otherwise. The `python_anvil.api_resources.requests` module with `RestRequest`, `PlainRequest`, `GraphqlRequest` and `FullyQualifiedRequest`, and `request_fully_qualified`, are removed.
+- BREAKING: Removed with `gql`: `python_anvil.http.GQLClient`, `get_local_schema` and `get_gql_ds`, `Anvil.gql_client`, and `HTTPClient`'s old signature and `(content, status_code, headers)` return. `python_anvil.constants` loses `GRAPHQL_ENDPOINT`, `REST_ENDPOINT`, `ANVIL_HOST`, `VALID_HOSTS`, `RETRIES_LIMIT`, `REQUESTS_LIMIT` and `RATELIMIT_ENV`; `DEFAULT_BASE_URL` is the one host now. The mutation builders' `mutation` template attribute is gone (pass `mutation=` to the method instead); `mutation_res_query` still selects a builder's default response fields.
+- BREAKING: `generate_etch_signing_url` returns the URL. `get_etch_packet` returns the packet, and `remove_weld_data` whether the data was removed.
+- BREAKING: `query` and `mutate` take the operation as a string; `gql` documents and its DSL are gone. `request_graphql` is the same thing with node-anvil's name.
+- BREAKING: `get_casts` and `get_welds` query the API's paginated `casts` and `welds` fields, which the old queries no longer matched, and return up to 500 per organization. `get_current_user` no longer selects each organization's casts.
+- Add `get_etch_packet` and `remove_weld_data`, and `response_query` and `mutation` options on `create_etch_packet`, `forge_submit` and `get_etch_packet`, matching node-anvil.
+- Add `Anvil.prepare_graphql_file` (also `python_anvil.prepare_graphql_file`) for multipart uploads of a path, bytes or a file object, anywhere in a mutation's variables. Files given as open file objects used to be base64 encoded into the JSON; they still are unless prepared this way.
+- The `createEtchPacket` mutation gains `organizationEid` and `allowUpdates`, and the mutations and default response fields now match node-anvil's exactly. `forgeSubmit`'s default fields drop `resolvedPayload` and `weldData.status`, and gain `continueURL` and `payloadValue`; pass `response_query` to select others.
+- `forge_submit` with a `webhook_url` works. `ForgeSubmitPayload.webhook_url` was a pydantic `HttpUrl`, which the JSON encoder rejected, so it failed before sending; it is a `str` now.
+- `fill_pdf` sends `version_number=0` (it used to drop it), and `generate_etch_signing_url` is also available as `generate_etch_sign_url`.
+- The client's queries and mutations are checked against the API schema in CI, and `make update-schema` refreshes the schema from the published `@anvilco/types` package.
+- Add `test-client.py`, which runs every method against a real environment: `ANVIL_API_KEY=... make test-client`.
+
 ## [v5.0.3] - 2025-02-24
 
 - Package import now uses `importlib.metadata` to get the version and throws `PackageNotFoundError` if the package is not

Makefile

@@ -29,6 +29,10 @@ ci: lint test ## Everything CI runs
 test-client: install ## Run every method against a real environment (staging by default)
 	poetry run python test-client.py
 
+.PHONY: update-schema
+update-schema: install ## Refresh schema/anvil_schema.graphql from the published @anvilco/types
+	poetry run python dev/fetch_schema.py
+
 .PHONY: build
 build: install ## Build the sdist and wheel into dist/
 	rm -rf dist

dev/fetch_schema.py

@@ -0,0 +1,63 @@
+#!/usr/bin/env python3
+"""Refreshes schema/anvil_schema.graphql from the published @anvilco/types package.
+
+@anvilco/types (https://www.npmjs.com/package/@anvilco/types) ships the Anvil
+GraphQL schema, regenerated from production after every deploy. node-anvil
+validates its operations against that copy, and so does this client: using
+the same source keeps the two clients in step, and needs no API key.
+
+Usage:
+    poetry run python dev/fetch_schema.py [version]
+
+Defaults to the latest version. On any failure the existing file is untouched.
+"""
+
+from __future__ import annotations
+
+import io
+import json
+import sys
+import tarfile
+import urllib.request
+from pathlib import Path
+
+
+PACKAGE = "@anvilco/types"
+REGISTRY = "https://registry.npmjs.org"
+SCHEMA_IN_PACKAGE = "package/schema/anvil-schema.graphql"
+OUTPUT_PATH = Path(__file__).resolve().parent.parent / "schema" / "anvil_schema.graphql"
+
+
+def fetch_schema(version: str = "latest") -> tuple[str, str]:
+    """The schema SDL in the given version of @anvilco/types, and that version."""
+    metadata_url = f"{REGISTRY}/{PACKAGE.replace('/', '%2F')}"
+    with urllib.request.urlopen(metadata_url) as response:
+        metadata = json.load(response)
+
+    if version == "latest":
+        version = metadata["dist-tags"]["latest"]
+    tarball_url = metadata["versions"][version]["dist"]["tarball"]
+
+    with urllib.request.urlopen(tarball_url) as response:
+        tarball = response.read()
+
+    with tarfile.open(fileobj=io.BytesIO(tarball), mode="r:gz") as archive:
+        member = archive.extractfile(SCHEMA_IN_PACKAGE)
+        if member is None:
+            raise FileNotFoundError(
+                f"{SCHEMA_IN_PACKAGE} is not in {PACKAGE}@{version}"
+            )
+        return member.read().decode("utf-8"), version
+
+
+def main() -> None:
+    version = sys.argv[1] if len(sys.argv) > 1 else "latest"
+    schema, version = fetch_schema(version)
+    OUTPUT_PATH.write_text(schema)
+    print(
+        f"Wrote the schema from {PACKAGE}@{version} to {OUTPUT_PATH.relative_to(Path.cwd())}"
+    )
+
+
+if __name__ == "__main__":
+    main()

examples/create_etch_markdown.py

@@ -131,7 +131,7 @@ def main():
     # Create your packet
     # If overriding/adding new fields, use the modified payload from
     # `packet.create_payload()`
-    res = anvil.create_etch_packet(payload=packet, include_headers=True)
+    res = anvil.create_etch_packet(payload=packet)
     print(res)
 
 

examples/create_etch_markup.py

@@ -136,7 +136,7 @@ def main():
     # Create your packet
     # If overriding/adding new fields, use the modified payload from
     # `packet.create_payload()`
-    res = anvil.create_etch_packet(payload=packet, include_headers=True)
+    res = anvil.create_etch_packet(payload=packet)
     print(res)
 
 

examples/create_etch_upload_file_multipart.py

@@ -55,21 +55,15 @@ def main():
     file_dir = os.path.dirname(os.path.realpath(__file__))
     file_path = os.path.join(file_dir, filename)
 
-    # You can check manually if your file exists, however, the validator being
-    # used in the `GraphqlUpload` below will also check if the file exists.
+    # `prepare_graphql_file` marks the file for a multipart upload: it's streamed
+    # with the request rather than base64 encoded into the JSON. It takes a
+    # path, bytes, or an open binary file object. A path is opened when the
+    # request is sent.
     #
-    # if not os.path.exists(file_path):
-    #     raise FileNotFoundError('File does not exist. Please check `file_path` '
-    #                             'and ensure it points to an existing file.')
-
-    # file data must be read in as _bytes_, not text.
-    file = open(file_path, "rb")  # noqa: SIM115
-
-    # You can also provide a custom `content_type` if you needed.
-    # The Anvil library will guess the file's content_type by its file
-    # extension automatically, but this can be used to force a different
-    # content_type.
-    # f1.content_type = "application/pdf"
+    # The filename and mimetype are taken from the path. Pass `filename=` and
+    # `mimetype=` to override them, or when uploading bytes or a file object
+    # that has no name, like a BytesIO.
+    file = Anvil.prepare_graphql_file(file_path)
 
     # Upload the file and define signer field locations.
     file1 = DocumentUpload(
@@ -140,11 +134,8 @@ def main():
     # Create your packet
     # If overriding/adding new fields, use the modified payload from
     # `packet.create_payload()`
-    try:
-        res = anvil.create_etch_packet(payload=packet)
-        print(res)
-    finally:
-        file.close()
+    res = anvil.create_etch_packet(payload=packet)
+    print(res)
 
 
 if __name__ == "__main__":

examples/make_graphql_request.py

@@ -1,9 +1,12 @@
-import os
+# Send your own GraphQL queries with the Anvil API client.
+# Reference: https://www.useanvil.com/docs/api/graphql/reference/
+#
+# Run this from the project root:
+# ANVIL_API_KEY=YOUR_KEY python examples/make_graphql_request.py
 
-from gql.dsl import DSLQuery, dsl_gql
+import os
 
 from python_anvil.api import Anvil
-from python_anvil.http import get_gql_ds
 
 
 API_KEY = os.environ.get("ANVIL_API_KEY")
@@ -12,82 +15,44 @@
 
 
 def call_current_user_query(anvil: Anvil) -> dict:
-    """Get the user data attached to the current API key.
-
-    :param anvil:
-    :type anvil: Anvil
-    :return:
-    """
-    # See the reference docs for examples of all queries and mutations:
-    # https://www.useanvil.com/docs/api/graphql/reference/
-
-    # You can also use `gql`'s query builder. Below is the equivalent of the
-    # string above, but can potentially be a better interface if you're
-    # building a query in multiple steps. See the official `gql` docs for more
-    # details: https://gql.readthedocs.io/en/stable/advanced/dsl_module.html
-
-    # Use `ds` to create your queries
-    ds = get_gql_ds(anvil.gql_client)
-    ds_user_query_builder = ds.Query.currentUser.select(
-        ds.User.eid,
-        ds.User.name,
-        ds.User.organizations.select(
-            ds.Organization.eid,
-            ds.Organization.slug,
-            ds.Organization.name,
-            ds.Organization.casts.select(
-                ds.Cast.eid,
-                ds.Cast.name,
-            ),
-            ds.Organization.welds.select(
-                ds.Weld.eid,
-                ds.Weld.name,
-            ),
-        ),
+    """Get the user data attached to the current API key."""
+    # `request_graphql` (or its alias `query`) takes the query as a string and
+    # returns the response's `data`. It raises an `AnvilRequestError` if the
+    # API returns any errors.
+    res = anvil.request_graphql(
+        """
+        query CurrentUser {
+            currentUser {
+                eid
+                name
+                organizations {
+                    eid
+                    slug
+                    name
+                    casts { eid name }
+                    welds { eid name }
+                }
+            }
+        }
+        """
     )
-
-    ds_query = dsl_gql(DSLQuery(ds_user_query_builder))
-
-    res = anvil.query(query=ds_query, variables=None)
     return res["currentUser"]
 
 
-def call_weld_query(anvil: Anvil, weld_eid: str):
-    """Call the weld query.
-
-    The weld() query is an example of a query that takes variables.
-    :param anvil:
-    :type anvil: Anvil
-    :param weld_eid:
-    :type weld_eid: str
-    :return:
-    """
-
-    # You can also use `gql`'s query builder. Below is the equivalent of the
-    # string above, but can potentially be a better interface if you're
-    # building a query in multiple steps. See the official `gql` docs for more
-    # details: https://gql.readthedocs.io/en/stable/advanced/dsl_module.html
-
-    # Use `ds` to create your queries
-    ds = get_gql_ds(anvil.gql_client)
-    ds_weld_query_builder = ds.Query.weld.args(eid=weld_eid).select(
-        ds.Weld.eid,
-        ds.Weld.name,
-        ds.Weld.forges.select(
-            ds.Forge.eid,
-            ds.Forge.slug,
-            ds.Forge.name,
-        ),
+def call_weld_query(anvil: Anvil, weld_eid: str) -> dict:
+    """Call the weld query, an example of a query that takes variables."""
+    res = anvil.request_graphql(
+        """
+        query WeldQuery($eid: String!) {
+            weld(eid: $eid) {
+                eid
+                name
+                forges { eid slug name }
+            }
+        }
+        """,
+        variables={"eid": weld_eid},
     )
-
-    ds_query = dsl_gql(DSLQuery(ds_weld_query_builder))
-
-    # You can call the query with the string literal and variables like usual
-    # res = anvil.query(query=weld_query, variables=variables)
-
-    # Or, use only the `dsl_gql` query. `variables` not needed as it was
-    # already used in `.args()`.
-    res = anvil.query(query=ds_query)
     return res["weld"]