Fitting models clean by maclariz · Pull Request #273 · electronmicroscopy/quantem

maclariz · 2026-07-31T05:01:53Z

What problem this PR addreseses

This creates a new set of functions in diffraction to provide Digital Dark Field imaging, both with manual selection and cluster analysis. It is all in one py file split into three parts:

import of data into Vector class
regular DDF imaging that reproduces some functions in py4DSTEM
the starts of cluster analysis (mainly L1 clustering)

All can be built out once the module has been accepted

The initial import is based on my own centre finding, which I can provide, but you may have code elsewhere in the repo that produces the desired kx and ky centre positions, in which case just adjust my function to use that result.

What should the reviewer(s) do

Digital Dark Field functions introduced

Restructured and some cluster support

missed import

Double import

bugfix (relics of old code)

Added the function to read pointsarrays from py4DSTEM and fixed a minor bug in cluster real space plot

maclariz · 2026-07-31T05:04:18Z

And don't change "vector.py" in response to this. I initially changed it and then changed it back, so it looks changed but isn't. All the new stuff in in digital_dark_field_cluster.py and init.py was changed in diffraction to import the new functions into the repo

Now writing cluster labels into the vector at L1

Totally changes how we make images from the vector object. This really will speed up when we make complex selections in objects since we now just select with one mask, not a stack.

Complete set of functions using new img reduction method. Maybe I should also make a new method to reduce a selection to a diffraction pattern.

maclariz · 2026-08-04T02:16:11Z

So, functions rebuilt to work with a new image reduce method in Vector and all built out to L2 clustering with two methods. The image reduce could be generalised, and an option for reducing to a diffraction pattern could also be made.

arthurmccray

Hi Ian, overall looks good! I'm leaving a few comments in the files where errors would occur. A few more general things:

We are also in the process of moving Vector over to being base torch, (see #272) That will cause a problem next time you try and merge from dev, but I think an LLM should be able to adapt what you've added to torch without many issues. I don't think this needs to be solved now.
a lot of the things i point out in comments would be caught with a simple pre-commit. I would suggest checking out the CONTRIBUTING.md or just install and run ruff.
as a smaller thing, please refer to PEP8 for a style guide and conventions for naming (e.g. in the DBSCAN_L2 function)
please write a few tests for DDF (or have an LLM do so for you)

arthurmccray · 2026-08-25T21:52:26Z

 from quantem.diffraction.strain_autocorrelation import StrainMapAutocorrelation as StrainMapAutocorrelation
-from quantem.diffraction.model_fitting import ModelDiffraction as ModelDiffraction
+from quantem.diffraction.model_fitting import ModelDiffraction as ModelDiffraction
+from quantem.diffraction.digital_dark_field_cluster import *


Please do not import * , if you want certain features elevated in the namespace please import specific items (as is done in this file)

fair dos. Will name everything explicitly.

arthurmccray · 2026-08-25T21:52:49Z

+
+from tqdm import tqdm
+
+from sklearn.cluster import DBSCAN


sklearn isn't a dependency of quantem, so please guard the import with an error message saying something like "if you want to use this feature, please install scikit-learn to your environment"

I really suggest that you consider having scikit-learn as a dependency so we can leverage the tools they have built rather than having to reinvent the wheel ourselves.
Of course, that means we might have to take torch tensors and transform to numpy to access this, but my initial attempt at getting DBSCAN written for torch didn't work well. Maybe this can be done eventually, but until then, scikit-learn is the best show on the road for clustering algorithms.

arthurmccray · 2026-08-25T21:53:03Z

+
+from tqdm import tqdm
+
+from sklearn.cluster import DBSCAN


sklearn isn't a dependency of quantem, so please guard the import with an error message saying something like "if you want to use this feature, please install scikit-learn to your environment"

arthurmccray · 2026-08-25T21:54:04Z

+        assert isinstance(fig, Figure)
+        assert isinstance(ax, Axes)


these lines throw an import error

arthurmccray · 2026-08-25T21:55:19Z

+    if ordering == "sequential":
+        cluster_list = unique_labels[1:]
+    elif ordering == "size":
+        cluster_list = L1_unique_labels[1:][np.argsort(L1_all_cluster_sizes[1:])[::-1]]


gives a NameError

arthurmccray · 2026-08-25T21:56:42Z

+
+    """
+    assert isinstance(maskstack, np.ndarray), "the maskstack must be a numpy array"
+    assert maskstack.shape[-1] == pointsvector.flatten.shape[1], "the mask size does not match the Vector size"


should be .flatten(), and i'm not actually sure it's comparing row to row

arthurmccray · 2026-08-25T21:57:41Z

+
+    L1_unique_labels = np.unique(L1labels)[1:]
+
+    COMs = np.zeros_like(np.vstack((L1_unique_labels,L1_unique_labels)).T)


are COMs supposed to be integer valued? this will truncate sub-pixel offsets

easy to fix

arthurmccray · 2026-08-25T21:58:47Z

+    assert len(scaling)==len(fields), "the scalings and fields must have the same number of entries"
+
+    # We need to return a new Vector as it is changing length once we select only part of the data
+    pointsvector2 = pointsvector.copy()


this line isn't used

arthurmccray · 2026-08-25T21:59:33Z

+        plot_L1_clusters_kspace( 
+            pointsvector2, 
+            fields, 
+            kr_max_plot=int(pointsvector2.select_fields('kx').flatten().max()*1.05)


is this meant to be kr not kx?

Yes, this is intentional. It rather assumes the detector is square, which it probably is. In the rare cases where it is not (e.g. Merlin T4 in frame-based mode), kx_max > ky_max, so this sets the maximum radius used in deciding maximum axis limits to the largest k value possible +5%

arthurmccray · 2026-08-25T22:01:12Z

+    for rx in tqdm(range(Rshape[0])):
+        for ry in range(Rshape[1]):


I think there should be a way to vectorize this

maclariz added 8 commits July 30, 2026 10:17

Digital Dark Field

a8fb4c2

Digital Dark Field functions introduced

updating DDF functions

4aaa2a2

Restructured and some cluster support

Update __init__.py

8d3df94

Update digital_dark_field_cluster.py

24f1404

missed import

Update digital_dark_field_cluster.py

7fd35ba

missed import

Update digital_dark_field_cluster.py

e3f6bcd

Double import

Update digital_dark_field_cluster.py

fb97b10

bugfix (relics of old code)

Update digital_dark_field_cluster.py

a84cd57

Added the function to read pointsarrays from py4DSTEM and fixed a minor bug in cluster real space plot

maclariz and others added 6 commits August 2, 2026 10:01

Update digital_dark_field_cluster.py

7124dcc

Now writing cluster labels into the vector at L1

Vector additional methods

0aa5de6

Vector tests

2ce5b36

Adding filter_rows method

a9b94b2

imgreduce

9c15964

Totally changes how we make images from the vector object. This really will speed up when we make complex selections in objects since we now just select with one mask, not a stack.

Update digital_dark_field_cluster.py

0596768

Complete set of functions using new img reduction method. Maybe I should also make a new method to reduce a selection to a diffraction pattern.

arthurmccray self-assigned this Aug 20, 2026

arthurmccray requested changes Aug 25, 2026

View reviewed changes


		L1_unique_labels = np.unique(L1labels)[1:]

		COMs = np.zeros_like(np.vstack((L1_unique_labels,L1_unique_labels)).T)

		for rx in tqdm(range(Rshape[0])):
		for ry in range(Rshape[1]):

Uh oh!

Conversation

maclariz commented Jul 31, 2026

What problem this PR addreseses

What should the reviewer(s) do

Uh oh!

maclariz commented Jul 31, 2026

Uh oh!

maclariz commented Aug 4, 2026

Uh oh!

arthurmccray left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

arthurmccray left a comment •

edited

Loading