Add a Python generator example for data.ingest

_includes/code/client-libraries/python_v4.py

@@ -360,6 +360,33 @@
 # END BatchRateLimit
 
 
+# START BatchIngestGenerator
+import weaviate
+
+client = weaviate.connect_to_local()
+
+# A generator produces objects one at a time instead of building a list
+def article_titles():
+    for title in ["Multitenancy", "Database schema"]:
+        yield {"title": title}
+
+try:
+    articles = client.collections.use("WikiArticle")
+    # `ingest` accepts any iterable, including a generator
+    result = articles.data.ingest(article_titles())
+
+    if result.errors:
+        print(f"Number of failed imports: {len(result.errors)}")
+
+finally:
+    client.close()
+# END BatchIngestGenerator
+
+# Tests
+assert len(result.errors) == 0
+assert len(result.uuids) == 2
+
+
 import weaviate
 
 client = weaviate.connect_to_local()

_includes/code/howto/manage-data.import.py

@@ -588,7 +588,7 @@ def add_object(obj) -> None:
     {"title": f"Object {i+1}"} for i in range(5)
 ]
 
-collection = client.collections.get("MyCollection")
+collection = client.collections.use("MyCollection")
 
 # highlight-start
 # Use `stream` for server-side batching. The client will send data
@@ -616,6 +616,59 @@ def add_object(obj) -> None:
 client.collections.delete(collection.name)
 
 
+# ==================================================
+# ===== Server-side ingest from a generator =====
+# ==================================================
+
+# Re-create the collection
+client.collections.delete("MyCollection")
+client.collections.create(
+    "MyCollection",
+    vector_config=Configure.Vectors.self_provided()
+)
+
+# Create the source file used by the example below
+with open("my-data.jsonl", "w") as f:
+    for i in range(5):
+        f.write(json.dumps({"title": f"Object {i+1}"}) + "\n")
+    f.write("\n")  # A blank line, to show that the generator skips it
+
+# START ServerSideIngestGeneratorExample
+import json
+
+# Each line of the source file holds one JSON object
+def read_objects(path):
+    with open(path) as f:
+        for line in f:
+            line = line.strip()
+            if not line:  # Skip blank lines
+                continue
+            record = json.loads(line)
+            yield {"title": record["title"]}
+
+collection = client.collections.use("MyCollection")
+
+# highlight-start
+# `ingest` pulls objects from the generator as it goes
+result = collection.data.ingest(read_objects("my-data.jsonl"))
+# highlight-end
+
+if result.errors:
+    print(f"Number of failed imports: {len(result.errors)}")
+# END ServerSideIngestGeneratorExample
+
+# Tests
+assert len(result.errors) == 0
+assert len(result.uuids) == 5
+
+agg_result = collection.aggregate.over_all(total_count=True)
+assert agg_result.total_count == 5
+
+# Clean up
+client.collections.delete(collection.name)
+os.remove("my-data.jsonl")
+
+
 # ==================================================
 # ===== Server-side one-shot ingest =====
 # ==================================================

docs/weaviate/client-libraries/python/notes-best-practices.mdx

@@ -202,8 +202,6 @@ There are four methods to configure the batching behavior. They are `stream`, `d
 | `fixed_size` | The batch size and number of concurrent requests are fixed to sizes specified by the user.                                                                                                                                     | When you want to specify fixed parameters.                                |
 | `rate_limit` | The number of objects sent to Weaviate is rate limited (specified as n_objects per minute).                                                                                                                                    | When you want to avoid hitting third-party vectorization API rate limits. |
 
-If your objects are already in an in-memory list, `collection.data.ingest(objs)` is a one-shot convenience that uses server-side batching under the hood (no batching context required). It accepts plain property dicts or `DataObject` instances and returns the same `BatchObjectReturn` object as `insert_many`, making it a drop-in replacement for `insert_many` for large lists.
-
 #### Usage
 
 We recommend using a context manager as shown below.
@@ -247,6 +245,19 @@ These methods return a new context manager for each batch. Attributes that are r
 
 If the background thread that is responsible for sending the batches raises an exception during batch processing, the error is raised to the main thread.
 
+### One-shot ingest
+
+`collection.data.ingest(objs)` is a one-shot convenience that uses server-side batching under the hood (no batching context required). It accepts any iterable of plain property dicts or `DataObject` instances, and returns the same `BatchObjectReturn` object as `insert_many`. Pass a list of objects that you already hold in memory to use `ingest` as a drop-in replacement for `insert_many` on large lists. Pass a generator, or any other lazy iterable, to import from a source that does not fit in memory: the client sends each object to the server as the generator produces it.
+
+<FilteredTextBlock
+  text={PythonCode}
+  startMarker="# START BatchIngestGenerator"
+  endMarker="# END BatchIngestGenerator"
+  language="py"
+/>
+
+For a generator that reads a source file line by line, see [Batch import](../../manage-objects/import.mdx#server-side-batching).
+
 ### Error handling
 
 During a batch import, any failed objects or references will be stored for retrieval. Additionally, a running count of failed objects and references is maintained.
@@ -264,6 +275,8 @@ Note that these lists are reset when a batching process is initialized. So make
   language="py"
 />
 
+`collection.data.ingest()` does not use a batching context, so it reports failures through its return value instead. The `BatchObjectReturn` object that it returns exposes `result.has_errors` as a summary flag, and `result.errors` as a dictionary keyed by the position of each failed object in the input. Check `result.errors` directly, because it holds an entry for every failed object. The [one-shot ingest](#one-shot-ingest) example above shows this pattern.
+
 ### Batch vectorization
 
 import BatchVectorizationOverview from "/_includes/code/client-libraries/batch-import.mdx";

docs/weaviate/manage-objects/import.mdx

@@ -43,6 +43,15 @@ Open the `batch.stream()` context manager and add objects one at a time; the cli
       endMarker="# END ServerSideBatchImportExample"
       language="py"
     />
+
+You can also stream from a data source with `data.ingest()`. It accepts any iterable, so you can pass a generator that reads a source file record by record. Objects go to the server as the generator produces them, so the source never has to fit in memory. To import objects that you already hold in a list, see [Ingest an in-memory list](#ingest-an-in-memory-list).
+
+    <FilteredTextBlock
+      text={PyCode}
+      startMarker="# START ServerSideIngestGeneratorExample"
+      endMarker="# END ServerSideIngestGeneratorExample"
+      language="py"
+    />
   </TabItem>
   <TabItem value="ts" label="JavaScript/TypeScript">
 
@@ -397,8 +406,8 @@ You can batch create links from an object to another object through cross-refere
 
 If your dataset does not fit in memory, do not load it all at once. Instead, read the source file lazily and add objects to the import as you go:
 
-- With the server-side streaming context (`batch.stream()` in Python), add each object as you read it from the file. The client sends data at the pace the server requests, so memory usage stays flat.
-- In TypeScript, `data.ingest()` accepts any `Iterable`, so you can pass a lazy source (such as a generator that reads the file record by record) instead of a fully loaded array.
+- With the [server-side streaming context](#server-side-batching), add each object as you read it from the file. The client sends data at the pace the server requests, so memory usage stays flat.
+- In Python and TypeScript, the [one-shot import method](#server-side-batching) accepts any iterable, so you can pass a lazy source, such as a generator that reads the file record by record, instead of a fully loaded list.
 - With manual batching, apply the same pattern: add objects to the batch as you read them.
 
 For JSON files, use a streaming parser that yields one object at a time (such as `ijson` in Python). For CSV files, read the file in chunks (such as `pandas` with the `chunksize` parameter) rather than loading it whole.