Add a Python generator example for data.ingest - #484
Merged
Conversation
The batch import page taught the iterable/generator form of the one-shot ingest call to TypeScript readers only, even though the Python client accepts any iterable too. Add the Python counterpart so both clients document the same capability. * New CI-executed snippet reads a JSONL file line by line and yields property dicts straight into the ingest call, so the file never has to fit in memory. Blank lines are skipped explicitly instead of relying on a null check that raises on them. * The snippet sits in the server-side batching tab, next to the streaming context, and links to the in-memory list section for the eager form. * "Stream data from large files" now names Python alongside TypeScript for the lazy-iterable option. Verified against Weaviate 1.38.0 with the CI-pinned client: 5 uuids returned, empty errors dict, all 5 titles persisted.
The page's only mention of the one-shot ingest call opened with "If your objects are already in an in-memory list", which is wrong by omission: the parameter is an iterable, so a generator works just as well. The page had no mention of generators anywhere. Rewrite that paragraph to cover both the eager and the lazy case, add a short generator snippet, and link to the batch import page for the version that reads a file line by line. Ingest is deliberately not added to the batch sizing table: it is a second entry point to server-side batching, not a fifth sizing mode. Snippet verified against Weaviate 1.38.0 with the CI-pinned client: 2 uuids returned, empty errors dict, both titles persisted.
- import.mdx: name `data.ingest()` inside the Python tab, and drop the Python-only method name from the shared "Stream data from large files" prose, linking to the server-side batching section instead. - notes-best-practices.mdx: promote the one-shot ingest material out of "Batch sizing" into its own "One-shot ingest" section, so it no longer splits "#### Usage" from the batching-method table it refers to. - notes-best-practices.mdx: document the `BatchObjectReturn` that `data.ingest()` returns under "Error handling", which previously covered only the batching-context API. - Snippet cleanups: use `collections.use()`, and trim two code comments that overclaimed or duplicated the surrounding prose.
![orca-security-eu[bot]](https://avatars.githubusercontent.com/in/276244?s=60&v=4){kind=small}
There was a problem hiding this comment.
Orca Security Scan Summary
| Status | Check | Issues by priority | |
|---|---|---|---|
 Passed |
Infrastructure as Code |  0  0  0  0 |
View in Orca |
| Passed |
SAST | 0 0 0 0 |
View in Orca |
| Passed |
Secrets | 0 0 0 0 |
View in Orca |
| Passed |
Vulnerabilities | 0 0 0 0 |
View in Orca |
Contributor
There was a problem hiding this comment.
Pull request overview
This PR updates the Weaviate documentation and CI-executed Python snippets to demonstrate that collection.data.ingest() accepts any iterable (including generators), enabling streaming imports without holding the full dataset in memory.
Changes:
- Adds a Python generator-based
data.ingest()example to the batch import docs alongside the existing streaming-context example. - Refactors the Python best practices page to introduce a dedicated “One-shot ingest” section and clarifies ingest-specific error handling via
BatchObjectReturn. - Adds/updates CI-executed Python snippets (
manage-data.import.py,python_v4.py) to cover generator-based ingest and validate behavior against Weaviate 1.38.0.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated no comments.
| File | Description |
|---|---|
| docs/weaviate/manage-objects/import.mdx | Documents generator-based data.ingest() usage in the server-side batching section and updates large-file streaming guidance. |
| docs/weaviate/client-libraries/python/notes-best-practices.mdx | Adds a “One-shot ingest” section and updates error-handling guidance for ingest’s BatchObjectReturn. |
| _includes/code/howto/manage-data.import.py | Adds a generator-based ingest snippet with CI assertions and aligns collection access with collections.use(). |
| _includes/code/client-libraries/python_v4.py | Adds a generator-based ingest snippet (with assertions) for the Python v4 client examples. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a Python generator example for
data.ingest. #478 shipped the TypeScript one only, so Python readers were never toldingestaccepts any iterable and can stream without holding the dataset in memory.import.mdx— generator example in the Server-side batching tab, alongside the in-memory list formnotes-best-practices.mdx— theingestparagraph no longer assumes an in-memory list; moved into its own section, and### Error handlingnow coversBatchObjectReturnmanage-data.import.py,python_v4.py— new CI-executed snippets, run against Weaviate 1.38.0The examples check
result.errors, notresult.has_errors. That is deliberate:has_errorsstaysFalseon theingestpath (weaviate/weaviate-python-client#2107). Please don't simplify it.Merge this before the quickstart
ingestPR, which links to#error-handlingon the best practices page.