Skip to content

Eac/shear table ids #22

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
eacharles merged 9 commits into from
Oct 1, 2026
Merged

Eac/shear table ids #22

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
27b7a16
Add object_id and cluster_id to shear output tables
eacharles Oct 1, 2026
3f919a6
Add optional shear file join to buildJoinedObjectTable/buildJoinedClu…
eacharles Oct 1, 2026
1c14e8d
Add computeColumnStats for per-row aggregation across catalog columns
eacharles Oct 1, 2026
c92ff39
Add reduceJoinedTable for flexible reduction of joined DataFrames
eacharles Oct 1, 2026
0a49ae8
reduceJoinedTable: also drop bare-prefix columns from stats_cols
eacharles Oct 1, 2026
6739daf
Drop _n column from computeColumnStats output
eacharles Oct 1, 2026
1faa298
computeColumnStats: eliminate nanstd/nanmean RuntimeWarnings
eacharles Oct 1, 2026
a66ba97
Add n_sib_objects and n_sib_srcs to ObjectStatsTable
eacharles Oct 1, 2026
bc6de85
Notebook updates to use reduced join table
eacharles Oct 1, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
52 changes: 44 additions & 8 deletions examples/rubin_dp2/CheckDescShearMatch.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,27 +14,55 @@
"id": "1e088a65-bfa6-476d-8c77-9068925d8ba5",
"metadata": {},
"outputs": [],
"source": "# Standard imports\nimport matplotlib.pyplot as plt\nimport numpy as np\nimport tables_io\nimport hpmcm"
"source": [
"# Standard imports\n",
"import matplotlib.pyplot as plt\n",
"import numpy as np\n",
"import tables_io\n",
"import hpmcm"
]
},
{
"cell_type": "markdown",
"id": "8b53c63c",
"source": "## Configuration\n\nSet the tract number and define input file paths.",
"metadata": {}
"metadata": {},
"source": [
"## Configuration\n",
"\n",
"Set the tract number and define input file paths."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "156c3f8c-11d6-465d-8373-4e0bb33f5e74",
"metadata": {},
"outputs": [],
"source": "tract = 8006\n\n# Load shear catalogs and match output tables\nshear_desc_in = tables_io.read(f'{tract}/mdet-run-{tract}.parq')\nshear_desc_ns = tables_io.read(f'{tract}/mdet-run-reduced_uncleaned_{tract}_ns.parq')\nshear_desc_1m = tables_io.read(f'{tract}/mdet-run-reduced_uncleaned_{tract}_1m.parq')\nshear_desc_1p = tables_io.read(f'{tract}/mdet-run-reduced_uncleaned_{tract}_1p.parq')\nshear_desc_obj_stats = tables_io.read(f'{tract}/desc_md_match_object_stats.pq')\nshear_desc_obj_assoc = tables_io.read(f'{tract}/desc_md_match_object_assoc.pq')\ninput_files = [\n f'{tract}/mdet-run-reduced_uncleaned_{tract}_ns.parq',\n f'{tract}/mdet-run-reduced_uncleaned_{tract}_1m.parq',\n f'{tract}/mdet-run-reduced_uncleaned_{tract}_1p.parq',\n] \ncatalog_ids = [0, 1, 2]"
"source": [
"tract = 8006\n",
"\n",
"# Load shear catalogs and match output tables\n",
"shear_desc_in = tables_io.read(f'{tract}/mdet-run-{tract}.parq')\n",
"shear_desc_ns = tables_io.read(f'{tract}/mdet-run-reduced_uncleaned_{tract}_ns.parq')\n",
"shear_desc_1m = tables_io.read(f'{tract}/mdet-run-reduced_uncleaned_{tract}_1m.parq')\n",
"shear_desc_1p = tables_io.read(f'{tract}/mdet-run-reduced_uncleaned_{tract}_1p.parq')\n",
"shear_desc_obj_stats = tables_io.read(f'{tract}/desc_md_match_object_stats.pq')\n",
"shear_desc_obj_assoc = tables_io.read(f'{tract}/desc_md_match_object_assoc.pq')\n",
"input_files = [\n",
" f'{tract}/mdet-run-reduced_uncleaned_{tract}_ns.parq',\n",
" f'{tract}/mdet-run-reduced_uncleaned_{tract}_1m.parq',\n",
" f'{tract}/mdet-run-reduced_uncleaned_{tract}_1p.parq',\n",
"] \n",
"catalog_ids = [0, 1, 2]"
]
},
{
"cell_type": "markdown",
"id": "a6268c28",
"source": "## Validation plots — positional checks",
"metadata": {}
"metadata": {},
"source": [
"## Validation plots — positional checks"
]
},
{
"cell_type": "code",
Expand Down Expand Up @@ -62,7 +90,15 @@
"id": "c8db1973-1088-4a83-8fd0-3a85d99d981f",
"metadata": {},
"outputs": [],
"source": "# Join source catalog data onto the object stats table\njoined_shear_obj = hpmcm.output_tables.buildJoinedObjectTable(\n shear_desc_obj_stats,\n shear_desc_obj_assoc,\n input_files,\n catalog_ids,\n)"
"source": [
"# Join source catalog data onto the object stats table\n",
"joined_shear_obj = hpmcm.output_tables.buildJoinedObjectTable(\n",
" shear_desc_obj_stats,\n",
" shear_desc_obj_assoc,\n",
" input_files,\n",
" catalog_ids,\n",
")"
]
},
{
"cell_type": "code",
Expand Down Expand Up @@ -126,4 +162,4 @@
},
"nbformat": 4,
"nbformat_minor": 5
}
}
60 changes: 51 additions & 9 deletions examples/rubin_dp2/CheckJoins.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,48 +3,84 @@
{
"cell_type": "markdown",
"id": "e2cba6bd",
"source": "# Check joined tables\n\nVerifies that buildJoinedObjectTable correctly merges object statistics with per-catalog source columns.",
"metadata": {}
"metadata": {},
"source": [
"# Check joined tables\n",
"\n",
"Verifies that buildJoinedObjectTable correctly merges object statistics with per-catalog source columns."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "9cb3a6e4-5813-4b95-8b20-0449bc89bfb2",
"metadata": {},
"outputs": [],
"source": "# Standard imports\nimport tables_io\nimport hpmcm\nimport matplotlib.pyplot as plt\nimport numpy as np"
"source": [
"# Standard imports\n",
"import tables_io\n",
"import hpmcm\n",
"import matplotlib.pyplot as plt\n",
"import numpy as np"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "d56eba26-ca76-4cd1-b545-b0c3ef8cc75f",
"metadata": {},
"outputs": [],
"source": "tract = 8006\n\n# Input catalogs and their IDs for the meta-match\nall_files = [\n f'{tract}/object_reduced_{tract}.parq',\n f'{tract}/anacal_reduced_{tract}.parq',\n f'{tract}/desc_md_match_object_stats.pq',\n f'{tract}/rubin_md_match_object_stats.pq',\n]\nall_ids = list(range(len(all_files)))"
"source": [
"tract = 8006\n",
"\n",
"# Input catalogs and their IDs for the meta-match\n",
"all_files = [\n",
" f'{tract}/object_reduced_{tract}.parq',\n",
" f'{tract}/anacal_reduced_{tract}.parq',\n",
" f'{tract}/desc_md_match_object_joined.pq',\n",
" f'{tract}/rubin_md_match_object_joined.pq',\n",
"]\n",
"all_ids = list(range(len(all_files)))"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "9b34104d-db8c-4efe-a2d3-7b4c5bfc0bff",
"metadata": {},
"outputs": [],
"source": "# Load association and stats tables\nobject_assoc = hpmcm.output_tables.ObjectAssocTable.read(f'{tract}/all_match_object_assoc.pq', [])\nobject_stats = hpmcm.output_tables.ObjectStatsTable.read(f'{tract}/all_match_object_stats.pq', [])"
"source": [
"# Load association and stats tables\n",
"object_assoc = hpmcm.output_tables.ObjectAssocTable.read(f'{tract}/all_match_object_assoc.pq', [])\n",
"object_stats = hpmcm.output_tables.ObjectStatsTable.read(f'{tract}/all_match_object_stats.pq', [])"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "ed073cb6-005c-4e0e-95aa-05ef3f67161c",
"metadata": {},
"outputs": [],
"source": "# Build wide joined table — one row per object, source cols suffixed by catalog_id\njoined = hpmcm.output_tables.buildJoinedObjectTable(\n object_assoc,\n all_files,\n all_ids,\n)"
"source": [
"# Build wide joined table — one row per object, source cols suffixed by catalog_id\n",
"joined = hpmcm.output_tables.buildJoinedObjectTable(\n",
" object_stats,\n",
" object_assoc,\n",
" all_files,\n",
" all_ids,\n",
")"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "5688ec6c-8981-432f-99da-b0e5929d5cfd",
"metadata": {},
"outputs": [],
"source": "# Count objects detected in catalog 0 (bit 0 set)\n(np.bitwise_and(joined['catalog_mask'],1) == 1).sum()"
"source": [
"# Count objects detected in catalog 0 (bit 0 set)\n",
"(np.bitwise_and(joined['catalog_mask'],1) == 1).sum()"
]
},
{
"cell_type": "code",
Expand Down Expand Up @@ -132,7 +168,13 @@
"id": "3341a247-cf9b-4a26-8d71-2ebcdc3ed687",
"metadata": {},
"outputs": [],
"source": "# Histogram of fluxes split by which catalogs detected each object\nfor i in range(1,16):\n _ = plt.hist(joined['g_gaapPsfFlux_0'][joined['catalog_mask']==i], bins=np.logspace(0, 6, 61))\n_ = plt.xscale('log')\n_ = plt.yscale('log')"
"source": [
"# Histogram of fluxes split by which catalogs detected each object\n",
"for i in range(1,16):\n",
" _ = plt.hist(joined['g_gaapPsfFlux_0'][joined['catalog_mask']==i], bins=np.logspace(0, 6, 61))\n",
"_ = plt.xscale('log')\n",
"_ = plt.yscale('log')"
]
},
{
"cell_type": "code",
Expand Down Expand Up @@ -164,4 +206,4 @@
},
"nbformat": 4,
"nbformat_minor": 5
}
}
65 changes: 57 additions & 8 deletions examples/rubin_dp2/CheckObjectMatch.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,16 +3,26 @@
{
"cell_type": "markdown",
"id": "4b91779a",
"source": "# Check object matching results\n\nExamines per-catalog coverage, SNR distributions, and spatial layouts of the multi-catalog object match.",
"metadata": {}
"metadata": {},
"source": [
"# Check object matching results\n",
"\n",
"Examines per-catalog coverage, SNR distributions, and spatial layouts of the multi-catalog object match."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "b6d5608f-ba28-4a27-9c17-20da93f778cc",
"metadata": {},
"outputs": [],
"source": "# Standard imports\nimport matplotlib.pyplot as plt\nimport numpy as np\nimport tables_io\nimport hpmcm"
"source": [
"# Standard imports\n",
"import matplotlib.pyplot as plt\n",
"import numpy as np\n",
"import tables_io\n",
"import hpmcm"
]
},
{
"cell_type": "code",
Expand All @@ -22,7 +32,25 @@
"scrolled": true
},
"outputs": [],
"source": "tract = 8006\n\n# Input catalog files and IDs for the object meta-match\nall_obj_stats = tables_io.read(f'{tract}/all_match_object_stats.pq')\nall_obj_assoc = tables_io.read(f'{tract}/all_match_object_assoc.pq')\nobject_in = tables_io.read(f'{tract}/object_reduced_{tract}.parq')\nanacal_in = tables_io.read(f'{tract}/anacal_reduced_{tract}.parq')\ndesc_md_in = tables_io.read(f'{tract}/desc_md_match_object_stats.pq')\nrubin_md_in = tables_io.read(f'{tract}/rubin_md_match_object_stats.pq')\n\nall_files = [\n f'{tract}/object_reduced_{tract}.parq',\n f'{tract}/anacal_reduced_{tract}.parq',\n f'{tract}/desc_md_match_object_stats.pq',\n f'{tract}/rubin_md_match_object_stats.pq',\n]\nall_ids = list(range(len(all_files)))"
"source": [
"tract = 8006\n",
"\n",
"# Input catalog files and IDs for the object meta-match\n",
"all_obj_stats = tables_io.read(f'{tract}/all_match_object_stats.pq')\n",
"all_obj_assoc = tables_io.read(f'{tract}/all_match_object_assoc.pq')\n",
"object_in = tables_io.read(f'{tract}/object_reduced_{tract}.parq')\n",
"anacal_in = tables_io.read(f'{tract}/anacal_reduced_{tract}.parq')\n",
"desc_md_in = tables_io.read(f'{tract}/desc_md_match_object_stats.pq')\n",
"rubin_md_in = tables_io.read(f'{tract}/rubin_md_match_object_stats.pq')\n",
"\n",
"all_files = [\n",
" f'{tract}/object_reduced_{tract}.parq',\n",
" f'{tract}/anacal_reduced_{tract}.parq',\n",
" f'{tract}/desc_md_match_object_stats.pq',\n",
" f'{tract}/rubin_md_match_object_stats.pq',\n",
"]\n",
"all_ids = list(range(len(all_files)))"
]
},
{
"cell_type": "code",
Expand All @@ -40,23 +68,44 @@
"id": "f37611d3-aa7c-42c6-b925-63e7854f1b80",
"metadata": {},
"outputs": [],
"source": "# Build wide joined table\njoined_obj = hpmcm.output_tables.buildJoinedObjectTable(\n all_obj_stats,\n all_obj_assoc,\n all_files,\n all_ids,\n)"
"source": [
"# Build wide joined table\n",
"joined_obj = hpmcm.output_tables.buildJoinedObjectTable(\n",
" all_obj_stats,\n",
" all_obj_assoc,\n",
" all_files,\n",
" all_ids,\n",
")"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "3d658e93-ee0a-45e0-bd7e-e73925c22cf6",
"metadata": {},
"outputs": [],
"source": "# SNR distributions split by which catalogs detected each object\nbins = np.logspace(0, 5, 51)\nfor i in range(4):\n mask_bit = 1 << i\n mask = np.bitwise_and(joined_obj.catalog_mask, mask_bit) == mask_bit\n _ = plt.hist(joined_obj[mask].snr, bins=bins, alpha=0.1)\n_ = plt.xscale('log')\n_ = plt.yscale('log')"
"source": [
"# SNR distributions split by which catalogs detected each object\n",
"bins = np.logspace(0, 5, 51)\n",
"for i in range(4):\n",
" mask_bit = 1 << i\n",
" mask = np.bitwise_and(joined_obj.catalog_mask, mask_bit) == mask_bit\n",
" _ = plt.hist(joined_obj[mask].snr, bins=bins, alpha=0.1)\n",
"_ = plt.xscale('log')\n",
"_ = plt.yscale('log')"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "88a6fdeb-fb5f-426d-9835-7b5b75fe63fd",
"metadata": {},
"outputs": [],
"source": "# 2D histogram: which catalogs detected each object vs SNR\n_ = plt.hist2d(joined_obj.catalog_mask, joined_obj.snr, bins=(np.linspace(0.5, 15.5, 16), np.logspace(0, 5, 51)), norm='log')\n_ = plt.yscale('log')"
"source": [
"# 2D histogram: which catalogs detected each object vs SNR\n",
"_ = plt.hist2d(joined_obj.catalog_mask, joined_obj.snr, bins=(np.linspace(0.5, 15.5, 16), np.logspace(0, 5, 51)), norm='log')\n",
"_ = plt.yscale('log')"
]
},
{
"cell_type": "code",
Expand Down Expand Up @@ -99,4 +148,4 @@
},
"nbformat": 4,
"nbformat_minor": 5
}
}
Loading
Loading