Repository navigation
Conversation
A negative limit in a request reached the database as a negative LIMIT and failed with an internal server error. It is now an InvalidArgumentsError, reported to the user as a 400. A limit of 0 is still allowed (and returns nothing). Assisted-by: claude-code/2.1.310:claude-opus-5-5
POST /datasets/<type>/<id>/properties/bulkFetch returns selected values from the properties dictionary of complete records, extracted by the database, so callers no longer need to download whole records to build a table of properties. At most 20 properties can be requested at once. Rows are ordered by (entry name, specification name) and paginated with a keyset cursor on the record item table's primary key, so every page is an index walk of bounded cost and large datasets can be fetched in batches without long-running requests. The total row count is optional. Pages are not a consistent snapshot: changes to the dataset between pages can cause rows to be missed or repeated (documented). In qcportal, iterate_properties walks the pages, with a progress bar sized by the total (which is only requested when the bar is shown), and get_properties_df builds its DataFrame from it. fetch_records=True instead takes the values from whole records (using and filling the cache), which views, offline datasets and requests for more than 20 properties always do, so both functions work with view files. Either way gives the same rows and the same DataFrame, including when no records match or a property is named "entry" or "specification". Entry and specification names not in the dataset are an error either way (rather than giving no rows), and duplicate property names are rejected by get_properties_df. Complete records with no properties no longer cause an error, and compile_values returns an empty DataFrame when no records match. Assisted-by: claude-code/2.1.310:claude-opus-5-5
qcportal imported plain tqdm, and most of its progress bars use disable=None, which only shows a bar when writing to a terminal. A Jupyter kernel's output is not a terminal, so these bars never appeared in notebooks. tqdm.auto uses the notebook widget there (which shows with disable=None), and plain tqdm everywhere else, so terminals and scripts are unchanged. InternalJob.watch now closes its progress bar even when waiting fails (such as on a timeout), so a notebook widget does not linger. Assisted-by: claude-code/2.1.310:claude-opus-5-5
Outlined inputs have a fixed height, so chips that wrap onto a second line spilled out of multi-selects and Autocompletes with chips, into the focus outline and under the label. Like multiline inputs, these now grow with their content (keeping the usual height as a minimum), and a multi-select's own inset is dropped so its chips sit inside the input. This applies to the Groups Autocompletes in the add user dialog and on the user profile page, and to the multi-selects in the Properties tab added next. Assisted-by: claude-code/2.1.310:claude-opus-5-5
A new tab on the dataset page shows a table of record properties, like qcportal's get_properties_df: pick properties (offered from the dataset's computed_properties) and, optionally, specifications, and each entry gets a row with a (specification, property) column for each value, with all-empty columns dropped. Short, flat arrays (such as a dipole) are shown inline; longer or nested arrays and objects are shown in the JSON viewer, collapsed (its copy button shows in the cell's padding, so hovering doesn't resize the column). Columns of numbers are right-aligned. Rows are in order of entry name, the order the values arrive in, so they stay in place while loading. Choices in the Properties and Specifications dropdowns are applied when the dropdown closes, so picking several starts a single fetch (deleting a chip applies right away). Values come from the new properties/bulkFetch endpoint, one page per request, with a progress bar based on the total row count from the first page. Each page is added to the values already loaded, and while loading the table is rebuilt at most every 2 seconds, so large datasets don't get slower to load as they go. Loading pauses automatically after 200,000 records, pointing to qcportal or a dataset view for more (with the option to continue). The stop button pauses loading; Resume continues from where it stopped. Both survive switching tabs. Results are kept until the dataset is refreshed (only for the current selection, since they can be large), and the computed properties are also only refetched on refresh, since that can take several seconds. The loaded values can be downloaded as a CSV file (one row per entry, like the table) or as JSON (one object per record, with values as stored). After stopping, the partial results can be downloaded, with "_partial" in the file name. Text that a spreadsheet would run as a formula is prefixed with an apostrophe in the CSV. Selections and table paging are part of the saved dataset view state (checked when restored, dropping names that are no longer available). Long property and specification names are truncated in removable chips (with the full name on hover). Assisted-by: opencode:vt-arc/GLM-5.3-thinking-high Assisted-by: claude-code/2.1.310:claude-opus-5-5
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Functions like
dataset.get_properties_dfwill fetch full record information and then get the properties that way. This is very slow for large datasets. So this PR adds the query to collect that information server-side and return it to the client, with appropriate pagination functionality.This is added as an option to the dataset functions, as well as a new tab on the web app (see below). While much faster, it can still be pretty slow for datasets over a few hundred thousand records.
As a side benefit, also fixes tqdm progress bars in jupyter notebooks.
Status