From 4105b729f7f82844fd82f07baf0fb822b492e832 Mon Sep 17 00:00:00 2001 From: Jerry Combs Date: Thu, 29 Oct 2026 08:43:45 -0500 Subject: [PATCH 1/2] Share the tree between reads; give the server more of the container's memory A large project ran the server out of memory (#177): the schema and provisioning requests each read and parsed the whole tree again, and a page sends several at once. They now read TreeService's copy under its lock, checking the files first so a change written a moment ago is seen. The launcher lets Java use 75% of the container's memory instead of 25%; IDM_JAVA_OPTS (in idmweb.env) still overrides it. The first log line says how much heap the server has, and the quickstart and hosted install say what to do on an OutOfMemoryError. Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 13 + docs/hosted-install.md | 3 + docs/quickstart.md | 12 + hosted/docker/bin/idmweb | 4 +- .../dirxml/web/ProvisioningService.java | 344 +++++++++--------- .../java/com/pointblue/dirxml/web/Serve.java | 2 +- .../com/pointblue/dirxml/web/TreeService.java | 18 + .../pointblue/dirxml/web/VaultService.java | 204 ++++++----- .../com/pointblue/dirxml/web/Workspace.java | 2 + .../pointblue/dirxml/web/SharedModelTest.java | 67 ++++ 10 files changed, 407 insertions(+), 262 deletions(-) create mode 100644 src/test/java/com/pointblue/dirxml/web/SharedModelTest.java diff --git a/CHANGELOG.md b/CHANGELOG.md index 4b411bcf..c171af6a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,19 @@ and this project uses [Semantic Versioning](https://semver.org/spec/v2.0.0.html) ## [Unreleased] +### Fixed + +- A large project no longer runs the server out of memory on the schema and provisioning pages + (#177). Each of their requests read and parsed the whole tree again, and a page sends several at + once. They now read the tree the outline keeps, read again only when its files change. + +### Changed + +- The server may use three quarters of the container's memory (`-XX:MaxRAMPercentage=75`); Java's + default is a quarter. `IDM_JAVA_OPTS` in `idmweb.env` still overrides it (`-Xmx4g`). The first + log line says how much it has (`heap up to … MB`), and the quickstart and hosted install say + what to do when the log shows `OutOfMemoryError`. + ## [0.82.0] - 2026-10-28 Needs DirXMLDev 0.58.0. diff --git a/docs/hosted-install.md b/docs/hosted-install.md index 5c568ef7..2f65df3b 100644 --- a/docs/hosted-install.md +++ b/docs/hosted-install.md @@ -134,4 +134,7 @@ An agent (CI, a coding agent, a script) does not install anything: customer keeps logs; set `audit.keep=0` if nothing may be deleted. - Sessions are kept in `/sessions.json` (owner-only): a restart signs nobody out, and each person gets their last project and branch back on their next session. +- Memory: the server takes up to three quarters of the container's memory (its first log line + says `heap up to … MB`). Size the container for the largest project; `IDM_JAVA_OPTS=-Xmx…` in + `idmweb.env` sets the heap instead. - `bin/smoke-container.sh` runs the image end to end on a workstation with Docker. diff --git a/docs/quickstart.md b/docs/quickstart.md index 147aeecf..f3dd139b 100644 --- a/docs/quickstart.md +++ b/docs/quickstart.md @@ -193,6 +193,18 @@ command there (`docker exec idmweb idm vault.status --env lab`) binds as you do - **One per machine:** the container is named `idmweb`. - **The server log:** `docker logs idmweb` (add `--tail 200` for the end). When the page reports "an internal error", the stack trace is there; attach it to an issue. +- **Memory:** the server takes up to three quarters of the container's memory; its first log line + says how much (`heap up to … MB`). Docker's own limit can be as little as 2 GB, which a large + project can fill: the log then shows `OutOfMemoryError`. Give Docker more memory + (Docker Desktop: **Settings → Resources**; WSL: `memory=` in `.wslconfig`), or set the heap + yourself with a line in `idmweb.env`, which `setup` keeps: + + ``` + IDM_JAVA_OPTS=-Xmx4g + ``` + + Then `docker compose -f "$HOME/idmweb/compose.yml" up -d` again. The same setting reaches + `docker exec idmweb idm …`. - **Moving to a server for a team:** [hosted-install.md](hosted-install.md) is the same image behind a proxy with TLS and your identity provider. Push the folder's `repo.git` to GitHub or GitLab and point the server's project at it. diff --git a/hosted/docker/bin/idmweb b/hosted/docker/bin/idmweb index 31e1dfb7..2d9997ab 100755 --- a/hosted/docker/bin/idmweb +++ b/hosted/docker/bin/idmweb @@ -1,8 +1,10 @@ #!/bin/sh # The web server inside the container, on the same class path as idm. set -e +# Java takes a quarter of the container's memory unless told: let the server have three +# quarters (#177). IDM_JAVA_OPTS comes after, so an -Xmx there wins. IDM_HOME=${IDM_HOME:-/opt/idmweb} IDM_LIB=${IDM_LIB:-/opt/idm/lib} CP="$IDM_HOME/lib/dirxml-dev-web.jar:$IDM_HOME/lib/dirxml-dev.jar:$IDM_HOME/lib/dirxml-simulator.jar:$IDM_HOME/lib/dirxml-trace-core.jar:$IDM_HOME/lib/postgresql.jar:$IDM_HOME/lib/console-auth.jar:$IDM_HOME/lib/jldap.jar" for j in "$IDM_LIB"/*.jar; do [ -f "$j" ] && CP="$CP:$j"; done -exec java ${IDM_JAVA_OPTS:-} -Didmweb.home="$IDM_HOME" -Didm.home="$IDM_HOME" -cp "$CP" com.pointblue.dirxml.web.Serve "$@" +exec java -XX:MaxRAMPercentage=75 ${IDM_JAVA_OPTS:-} -Didmweb.home="$IDM_HOME" -Didm.home="$IDM_HOME" -cp "$CP" com.pointblue.dirxml.web.Serve "$@" diff --git a/src/main/java/com/pointblue/dirxml/web/ProvisioningService.java b/src/main/java/com/pointblue/dirxml/web/ProvisioningService.java index 111e5ed0..fd8a8485 100644 --- a/src/main/java/com/pointblue/dirxml/web/ProvisioningService.java +++ b/src/main/java/com/pointblue/dirxml/web/ProvisioningService.java @@ -42,6 +42,17 @@ public ProvisioningService(Path tree) { this.tree = tree.toAbsolutePath().normalize(); } + private TreeService models; + + /** Reads the tree through {@code t}'s copy instead of from disk each time (#177). */ + public void models(TreeService t) { + this.models = t; + } + + private T model(TreeService.ModelRead read) throws IOException { + return models != null ? models.withModel(read) : read.read(AsCodeReader.read(tree)); + } + private Driver driver(DriverSet ds, String name) { Driver d = ds.driver(name); if (d == null) { @@ -55,110 +66,111 @@ private Driver driver(DriverSet ds, String name) { /** A PRD: its definition, its request form's fields and the workflow graph. */ public Map prd(String driverName, String name) throws IOException { - DriverSet ds = AsCodeReader.read(tree); - Driver d = driver(ds, driverName); - Prd prd = d.provisioning.prd(name); - if (prd == null) { - throw new TreeService.NotFound("driver '" + driverName + "' has no PRD '" + name + "'"); - } - Map m = new LinkedHashMap<>(); - m.put("driver", d.name); - m.put("name", prd.name); - m.put("dir", "drivers/" + AsCodeWriter.fileSafe(d.name) + "/provisioning/prds/" + AsCodeWriter.fileSafe(prd.name)); - m.put("status", prd.definition == null ? null : blankToNull(prd.definition.getAttribute("status"))); - m.put("jsonForms", prd.isJsonForms()); - m.put("properties", prd.properties); - List bindings = new ArrayList<>(); - for (Prd.FormBinding b : prd.bindings()) { - Map one = new LinkedHashMap<>(); - one.put("activity", b.activityId); - one.put("form", b.formId); - List fields = new ArrayList<>(); - for (Prd.Field f : b.fields) { - Map fm = new LinkedHashMap<>(); - fm.put("name", f.name); - fm.put("dataType", f.dataType); - fm.put("control", f.controlType); - fields.add(fm); + return model(ds -> { + Driver d = driver(ds, driverName); + Prd prd = d.provisioning.prd(name); + if (prd == null) { + throw new TreeService.NotFound("driver '" + driverName + "' has no PRD '" + name + "'"); } - one.put("fields", fields); - bindings.add(one); - } - m.put("bindings", bindings); - List formNames = new ArrayList<>(); - for (Form f : d.provisioning.forms) { - formNames.add(f.name); - } - m.put("forms", formNames); - if (prd.process == null) { - m.put("flow", null); - return m; - } - Flow flow = Flow.of(prd); - Map fl = new LinkedHashMap<>(); - Map process = new LinkedHashMap<>(); - process.put("version", flow.version); - process.put("process-type", flow.processType); - process.put("flow-strategy", flow.flowStrategy); - process.put("setnotify", flow.setnotify); - process.put("restrict-view", flow.restrictView); - process.put("generate-comments", flow.generateComments); - process.put("default-completed-approval-status", flow.defaultCompletedApprovalStatus); - fl.put("process", process); - List acts = new ArrayList<>(); - for (Flow.Activity a : flow.activities) { - Map am = new LinkedHashMap<>(); - am.put("id", a.id); - am.put("kind", a.kind == Flow.Kind.UNKNOWN ? a.element.getTagName() : a.kind.element.replace("-activity", "")); - am.put("name", a.displayName(null)); - Map attrs = new LinkedHashMap<>(); - NamedNodeMap nm = a.element.getAttributes(); - for (int i = 0; i < nm.getLength(); i++) { - attrs.put(nm.item(i).getNodeName(), nm.item(i).getNodeValue()); - } - am.put("attrs", attrs); - Map parts = new LinkedHashMap<>(); - for (Node c = a.element.getFirstChild(); c != null; c = c.getNextSibling()) { - if (c instanceof Element && !"display-name".equals(c.getNodeName()) && !hasElementChild((Element) c)) { - parts.merge(c.getNodeName(), c.getTextContent().trim(), (x, y) -> x + "\n" + y); + Map m = new LinkedHashMap<>(); + m.put("driver", d.name); + m.put("name", prd.name); + m.put("dir", "drivers/" + AsCodeWriter.fileSafe(d.name) + "/provisioning/prds/" + AsCodeWriter.fileSafe(prd.name)); + m.put("status", prd.definition == null ? null : blankToNull(prd.definition.getAttribute("status"))); + m.put("jsonForms", prd.isJsonForms()); + m.put("properties", prd.properties); + List bindings = new ArrayList<>(); + for (Prd.FormBinding b : prd.bindings()) { + Map one = new LinkedHashMap<>(); + one.put("activity", b.activityId); + one.put("form", b.formId); + List fields = new ArrayList<>(); + for (Prd.Field f : b.fields) { + Map fm = new LinkedHashMap<>(); + fm.put("name", f.name); + fm.put("dataType", f.dataType); + fm.put("control", f.controlType); + fields.add(fm); } + one.put("fields", fields); + bindings.add(one); + } + m.put("bindings", bindings); + List formNames = new ArrayList<>(); + for (Form f : d.provisioning.forms) { + formNames.add(f.name); } - am.put("parts", parts); - am.put("xml", xml(a.element)); - am.put("linkTypes", a.kind.restrictsOutgoingLinkTypes() ? new ArrayList<>(new TreeSet<>(a.kind.allowedOutgoingLinkTypes())) : Flow.LINK_TYPES); - List items = new ArrayList<>(); - for (Flow.DataItem di : flow.dataItemsByActivity.getOrDefault(a.id, List.of())) { - Map dm = new LinkedHashMap<>(); - dm.put("name", di.name); - dm.put("dataType", di.dataType); - dm.put("source", di.source); - dm.put("target", di.target); - dm.put("targetType", di.targetType); - items.add(dm); + m.put("forms", formNames); + if (prd.process == null) { + m.put("flow", null); + return m; } - am.put("dataItems", items); - String form = null; - for (Flow.FormBinding fb : flow.formBindings) { - if (a.id.equals(fb.activityId)) { - form = fb.formId; + Flow flow = Flow.of(prd); + Map fl = new LinkedHashMap<>(); + Map process = new LinkedHashMap<>(); + process.put("version", flow.version); + process.put("process-type", flow.processType); + process.put("flow-strategy", flow.flowStrategy); + process.put("setnotify", flow.setnotify); + process.put("restrict-view", flow.restrictView); + process.put("generate-comments", flow.generateComments); + process.put("default-completed-approval-status", flow.defaultCompletedApprovalStatus); + fl.put("process", process); + List acts = new ArrayList<>(); + for (Flow.Activity a : flow.activities) { + Map am = new LinkedHashMap<>(); + am.put("id", a.id); + am.put("kind", a.kind == Flow.Kind.UNKNOWN ? a.element.getTagName() : a.kind.element.replace("-activity", "")); + am.put("name", a.displayName(null)); + Map attrs = new LinkedHashMap<>(); + NamedNodeMap nm = a.element.getAttributes(); + for (int i = 0; i < nm.getLength(); i++) { + attrs.put(nm.item(i).getNodeName(), nm.item(i).getNodeValue()); + } + am.put("attrs", attrs); + Map parts = new LinkedHashMap<>(); + for (Node c = a.element.getFirstChild(); c != null; c = c.getNextSibling()) { + if (c instanceof Element && !"display-name".equals(c.getNodeName()) && !hasElementChild((Element) c)) { + parts.merge(c.getNodeName(), c.getTextContent().trim(), (x, y) -> x + "\n" + y); + } } + am.put("parts", parts); + am.put("xml", xml(a.element)); + am.put("linkTypes", a.kind.restrictsOutgoingLinkTypes() ? new ArrayList<>(new TreeSet<>(a.kind.allowedOutgoingLinkTypes())) : Flow.LINK_TYPES); + List items = new ArrayList<>(); + for (Flow.DataItem di : flow.dataItemsByActivity.getOrDefault(a.id, List.of())) { + Map dm = new LinkedHashMap<>(); + dm.put("name", di.name); + dm.put("dataType", di.dataType); + dm.put("source", di.source); + dm.put("target", di.target); + dm.put("targetType", di.targetType); + items.add(dm); + } + am.put("dataItems", items); + String form = null; + for (Flow.FormBinding fb : flow.formBindings) { + if (a.id.equals(fb.activityId)) { + form = fb.formId; + } + } + am.put("form", form); + acts.add(am); } - am.put("form", form); - acts.add(am); - } - fl.put("activities", acts); - List links = new ArrayList<>(); - for (Flow.Link l : flow.links) { - Map lm = new LinkedHashMap<>(); - lm.put("from", l.source); - lm.put("to", l.target); - lm.put("type", l.type); - links.add(lm); - } - fl.put("links", links); - fl.put("linkTypes", Flow.LINK_TYPES); - m.put("flow", fl); - return m; + fl.put("activities", acts); + List links = new ArrayList<>(); + for (Flow.Link l : flow.links) { + Map lm = new LinkedHashMap<>(); + lm.put("from", l.source); + lm.put("to", l.target); + lm.put("type", l.type); + links.add(lm); + } + fl.put("links", links); + fl.put("linkTypes", Flow.LINK_TYPES); + m.put("flow", fl); + return m; + }); } private static boolean hasElementChild(Element e) { @@ -189,86 +201,88 @@ private static String blankToNull(String s) { /** A JSON form: its document, and each PRD activity that shows it. */ public Map form(String driverName, String name) throws IOException { - DriverSet ds = AsCodeReader.read(tree); - Driver d = driver(ds, driverName); - Form form = d.provisioning.formByName(name); - if (form == null) { - throw new TreeService.NotFound("driver '" + driverName + "' has no form '" + name + "'"); - } - Map m = new LinkedHashMap<>(); - m.put("driver", d.name); - m.put("name", form.name); - m.put("kind", form.kind.dir); - m.put("file", "drivers/" + AsCodeWriter.fileSafe(d.name) + "/provisioning/forms/" + form.kind.dir + "/" + AsCodeWriter.fileSafe(form.name) + ".form.json"); - m.put("json", form.json); - List usedBy = new ArrayList<>(); - for (Prd p : d.provisioning.prds) { - for (Prd.FormBinding b : p.bindings()) { - if (form.name.equals(b.formId)) { - Map u = new LinkedHashMap<>(); - u.put("prd", p.name); - u.put("activity", b.activityId); - List mapped = new ArrayList<>(); - for (Prd.Field f : b.fields) { - mapped.add(f.name); + return model(ds -> { + Driver d = driver(ds, driverName); + Form form = d.provisioning.formByName(name); + if (form == null) { + throw new TreeService.NotFound("driver '" + driverName + "' has no form '" + name + "'"); + } + Map m = new LinkedHashMap<>(); + m.put("driver", d.name); + m.put("name", form.name); + m.put("kind", form.kind.dir); + m.put("file", "drivers/" + AsCodeWriter.fileSafe(d.name) + "/provisioning/forms/" + form.kind.dir + "/" + AsCodeWriter.fileSafe(form.name) + ".form.json"); + m.put("json", form.json); + List usedBy = new ArrayList<>(); + for (Prd p : d.provisioning.prds) { + for (Prd.FormBinding b : p.bindings()) { + if (form.name.equals(b.formId)) { + Map u = new LinkedHashMap<>(); + u.put("prd", p.name); + u.put("activity", b.activityId); + List mapped = new ArrayList<>(); + for (Prd.Field f : b.fields) { + mapped.add(f.name); + } + u.put("fields", mapped); + usedBy.add(u); } - u.put("fields", mapped); - usedBy.add(u); } } - } - m.put("usedBy", usedBy); - return m; + m.put("usedBy", usedBy); + return m; + }); } /** The role and resource catalog of every driver that has one. */ public Map catalog() throws IOException { - DriverSet ds = AsCodeReader.read(tree); - List roles = new ArrayList<>(); - List resources = new ArrayList<>(); - List entities = new ArrayList<>(); - List drivers = new ArrayList<>(); - for (Driver d : ds.drivers) { - if (d.provisioning == null) { - continue; - } - drivers.add(d.name); - for (AppObject o : d.provisioning.objects) { - AppObject.Kind k = o.kind(); - if (k == AppObject.Kind.ENTITY) { - entities.add(entity(d.name, o)); + return model(ds -> { + List roles = new ArrayList<>(); + List resources = new ArrayList<>(); + List entities = new ArrayList<>(); + List drivers = new ArrayList<>(); + for (Driver d : ds.drivers) { + if (d.provisioning == null) { continue; } - if (k != AppObject.Kind.ROLE && k != AppObject.Kind.RESOURCE) { - continue; + drivers.add(d.name); + for (AppObject o : d.provisioning.objects) { + AppObject.Kind k = o.kind(); + if (k == AppObject.Kind.ENTITY) { + entities.add(entity(d.name, o)); + continue; + } + if (k != AppObject.Kind.ROLE && k != AppObject.Kind.RESOURCE) { + continue; + } + Map m = new LinkedHashMap<>(); + m.put("driver", d.name); + m.put("name", o.name()); + m.put("path", o.path()); + m.put("displayName", o.displayName()); + m.put("container", o.parentPath()); + m.put("attrs", o.attrs); + (k == AppObject.Kind.ROLE ? roles : resources).add(m); } - Map m = new LinkedHashMap<>(); - m.put("driver", d.name); - m.put("name", o.name()); - m.put("path", o.path()); - m.put("displayName", o.displayName()); - m.put("container", o.parentPath()); - m.put("attrs", o.attrs); - (k == AppObject.Kind.ROLE ? roles : resources).add(m); } - } - Map out = new LinkedHashMap<>(); - out.put("drivers", drivers); - out.put("roles", roles); - out.put("resources", resources); - out.put("entities", entities); - Map categories = new LinkedHashMap<>(); - for (Driver d : ds.drivers) { - if (d.provisioning != null) { - Map lists = new LinkedHashMap<>(); - for (AppCategories.For f : AppCategories.For.values()) { - lists.put(f.key, categories(d, f)); + Map out = new LinkedHashMap<>(); + out.put("drivers", drivers); + out.put("roles", roles); + out.put("resources", resources); + out.put("entities", entities); + Map categories = new LinkedHashMap<>(); + for (Driver d : ds.drivers) { + if (d.provisioning != null) { + Map lists = new LinkedHashMap<>(); + for (AppCategories.For f : AppCategories.For.values()) { + lists.put(f.key, categories(d, f)); + } + categories.put(d.name, lists); } - categories.put(d.name, lists); } - } - out.put("categories", categories); - return out; + out.put("categories", categories); + return out; + }); } /** A driver's role or resource category list (core #70): key, label, labels, how many use it; null when the tree has no list. */ diff --git a/src/main/java/com/pointblue/dirxml/web/Serve.java b/src/main/java/com/pointblue/dirxml/web/Serve.java index 310ef078..0d06e188 100644 --- a/src/main/java/com/pointblue/dirxml/web/Serve.java +++ b/src/main/java/com/pointblue/dirxml/web/Serve.java @@ -128,7 +128,7 @@ static void hosted(Path config) throws Exception { server.createContext("/auth/", new Auth(identity, sessions, security, audit, h.publicUrl)); server.createContext("/", new StaticFiles(web)); start(server); - System.out.println("DirXMLDevWeb " + VERSION + " hosted: " + h.publicUrl + " (bound " + h.bind + ":" + h.port + "), identity " + identity.name() + ", " + agents.size() + " agent(s), projects " + (h.projects.isEmpty() ? "none yet (setup on first visit)" : String.join(", ", h.projects.keySet())) + ", data " + h.dataDir); + System.out.println("DirXMLDevWeb " + VERSION + " hosted: " + h.publicUrl + " (bound " + h.bind + ":" + h.port + "), identity " + identity.name() + ", " + agents.size() + " agent(s), projects " + (h.projects.isEmpty() ? "none yet (setup on first visit)" : String.join(", ", h.projects.keySet())) + ", data " + h.dataDir + ", heap up to " + (Runtime.getRuntime().maxMemory() >> 20) + " MB"); Thread.currentThread().join(); } diff --git a/src/main/java/com/pointblue/dirxml/web/TreeService.java b/src/main/java/com/pointblue/dirxml/web/TreeService.java index 7fdf2a18..dddece17 100644 --- a/src/main/java/com/pointblue/dirxml/web/TreeService.java +++ b/src/main/java/com/pointblue/dirxml/web/TreeService.java @@ -88,6 +88,24 @@ private synchronized DriverSet model() throws IOException { return cached; } + /** Reads the model; it must not change it, nor keep a DOM node past its return. */ + public interface ModelRead { + T read(DriverSet ds) throws IOException; + } + + /** + * Runs {@code read} on the tree as last read, under this service's lock, for another service + * that only reads it. Each request used to read the whole tree again, and the schema page asks + * several at once, so a large tree filled the heap (#177). The lock also keeps two threads off + * one DOM, which isn't safe to read concurrently. The files are checked first (sizes and + * times only), so a change written a moment ago is read as before, without waiting for the + * watcher. + */ + public synchronized T withModel(ModelRead read) throws IOException { + watcher.refreshNow(); + return read.read(model()); + } + // ---- reads ---- public synchronized Map outline() throws IOException { diff --git a/src/main/java/com/pointblue/dirxml/web/VaultService.java b/src/main/java/com/pointblue/dirxml/web/VaultService.java index a22b2e1e..974c00fa 100644 --- a/src/main/java/com/pointblue/dirxml/web/VaultService.java +++ b/src/main/java/com/pointblue/dirxml/web/VaultService.java @@ -44,6 +44,17 @@ public VaultService(Path tree, EnvFiles files) { this.files = files; } + private TreeService models; + + /** Reads the tree through {@code t}'s copy instead of from disk each time (#177). */ + public void models(TreeService t) { + this.models = t; + } + + private T model(TreeService.ModelRead read) throws IOException { + return models != null ? models.withModel(read) : read.read(AsCodeReader.read(tree)); + } + /** The environments this tree is deployed to (hosted: the project's list); empty = every one the file names. */ private volatile List environmentFilter = List.of(); @@ -438,76 +449,78 @@ static String gate(String envName, String confirm) { /** The tree's vault schema: counts and source, every class (name, LDAP, kind, custom) and every attribute (name, LDAP, syntax, single, custom). */ public Map schema() throws IOException { - DriverSet ds = AsCodeReader.read(tree); - Map m = new LinkedHashMap<>(); - m.put("present", ds.schema != null); - if (ds.schema == null) { + return model(ds -> { + Map m = new LinkedHashMap<>(); + m.put("present", ds.schema != null); + if (ds.schema == null) { + return m; + } + m.putAll(ds.schema.summary()); + List classes = new ArrayList<>(); + for (com.pointblue.dirxml.dev.model.VaultSchema.ClassDef c : ds.schema.classes) { + Map cm = new LinkedHashMap<>(); + cm.put("name", c.name); + cm.put("ldap", c.ldap); + cm.put("kind", c.kind); + cm.put("container", c.container); + cm.put("custom", c.custom); + cm.put("superclasses", c.superclasses); + classes.add(cm); + } + m.put("classes", classes); + List attrs = new ArrayList<>(); + for (com.pointblue.dirxml.dev.model.VaultSchema.AttrDef a : ds.schema.attributes) { + Map am = new LinkedHashMap<>(); + am.put("name", a.name); + am.put("ldap", a.ldap); + am.put("syntax", a.syntax); + am.put("single", a.single); + am.put("custom", a.custom); + attrs.add(am); + } + m.put("attributes", attrs); return m; - } - m.putAll(ds.schema.summary()); - List classes = new ArrayList<>(); - for (com.pointblue.dirxml.dev.model.VaultSchema.ClassDef c : ds.schema.classes) { - Map cm = new LinkedHashMap<>(); - cm.put("name", c.name); - cm.put("ldap", c.ldap); - cm.put("kind", c.kind); - cm.put("container", c.container); - cm.put("custom", c.custom); - cm.put("superclasses", c.superclasses); - classes.add(cm); - } - m.put("classes", classes); - List attrs = new ArrayList<>(); - for (com.pointblue.dirxml.dev.model.VaultSchema.AttrDef a : ds.schema.attributes) { - Map am = new LinkedHashMap<>(); - am.put("name", a.name); - am.put("ldap", a.ldap); - am.put("syntax", a.syntax); - am.put("single", a.single); - am.put("custom", a.custom); - attrs.add(am); - } - m.put("attributes", attrs); - return m; + }); } /** One class with everything about it and the attributes it holds (its superclasses' included), for a picker. */ public Map schemaClass(String name) throws IOException { - DriverSet ds = AsCodeReader.read(tree); - if (ds.schema == null) { - throw new TreeService.NotFound("the tree has no schema"); - } - com.pointblue.dirxml.dev.model.VaultSchema.ClassDef c = ds.schema.classDef(name); - if (c == null) { - throw new TreeService.NotFound("no class '" + name + "' in the schema"); - } - Map m = new LinkedHashMap<>(); - m.put("name", c.name); - m.put("ldap", c.ldap); - m.put("oid", c.oid); - m.put("kind", c.kind); - m.put("container", c.container); - m.put("custom", c.custom); - m.put("superclasses", c.superclasses); - m.put("mandatory", c.mandatory); - m.put("optional", c.optional); - m.put("containment", c.containment); - m.put("naming", c.naming); - Set mandatory = new java.util.HashSet<>(); - mandatoryOf(ds.schema, c, mandatory, new java.util.HashSet<>()); - List all = new ArrayList<>(); - for (String a : ds.schema.attributesOf(c.name)) { - com.pointblue.dirxml.dev.model.VaultSchema.AttrDef d = ds.schema.attribute(a); - Map am = new LinkedHashMap<>(); - am.put("name", a); - am.put("ldap", d == null ? a : d.ldap); - am.put("syntax", d == null ? null : d.syntax); - am.put("single", d != null && d.single); - am.put("mandatory", mandatory.contains(a.toLowerCase(java.util.Locale.ROOT))); - all.add(am); - } - m.put("attributes", all); - return m; + return model(ds -> { + if (ds.schema == null) { + throw new TreeService.NotFound("the tree has no schema"); + } + com.pointblue.dirxml.dev.model.VaultSchema.ClassDef c = ds.schema.classDef(name); + if (c == null) { + throw new TreeService.NotFound("no class '" + name + "' in the schema"); + } + Map m = new LinkedHashMap<>(); + m.put("name", c.name); + m.put("ldap", c.ldap); + m.put("oid", c.oid); + m.put("kind", c.kind); + m.put("container", c.container); + m.put("custom", c.custom); + m.put("superclasses", c.superclasses); + m.put("mandatory", c.mandatory); + m.put("optional", c.optional); + m.put("containment", c.containment); + m.put("naming", c.naming); + Set mandatory = new java.util.HashSet<>(); + mandatoryOf(ds.schema, c, mandatory, new java.util.HashSet<>()); + List all = new ArrayList<>(); + for (String a : ds.schema.attributesOf(c.name)) { + com.pointblue.dirxml.dev.model.VaultSchema.AttrDef d = ds.schema.attribute(a); + Map am = new LinkedHashMap<>(); + am.put("name", a); + am.put("ldap", d == null ? a : d.ldap); + am.put("syntax", d == null ? null : d.syntax); + am.put("single", d != null && d.single); + am.put("mandatory", mandatory.contains(a.toLowerCase(java.util.Locale.ROOT))); + all.add(am); + } + m.put("attributes", all); + return m; + }); } /** The attributes a class must have, its superclasses' included (a mandatory attribute is mandatory wherever it is inherited). */ @@ -525,38 +538,39 @@ private static void mandatoryOf(com.pointblue.dirxml.dev.model.VaultSchema s, co /** A driver's application schema as the shim reported it: classes with their attributes; present=false when never read. */ public Map appSchema(String driver) throws IOException { - DriverSet ds = AsCodeReader.read(tree); - Driver d = ds.driver(driver); - if (d == null) { - throw new TreeService.NotFound("no driver '" + driver + "'"); - } - org.w3c.dom.Element app = d.config.get(Driver.APP_SCHEMA); - Map m = new LinkedHashMap<>(); - m.put("driver", driver); - m.put("present", app != null); - List classes = new ArrayList<>(); - if (app != null) { - m.put("hierarchical", app.getAttribute("hierarchical")); - for (org.w3c.dom.Element c : SchemaOps.children(app, "class-def")) { - Map cm = new LinkedHashMap<>(); - cm.put("name", c.getAttribute("class-name")); - cm.put("container", "true".equals(c.getAttribute("container"))); - List attrs = new ArrayList<>(); - for (org.w3c.dom.Element a : SchemaOps.children(c, "attr-def")) { - Map am = new LinkedHashMap<>(); - am.put("name", a.getAttribute("attr-name")); - am.put("type", a.getAttribute("type")); - am.put("multi", "true".equals(a.getAttribute("multi-valued"))); - am.put("required", "true".equals(a.getAttribute("required"))); - am.put("naming", "true".equals(a.getAttribute("naming"))); - attrs.add(am); + return model(ds -> { + Driver d = ds.driver(driver); + if (d == null) { + throw new TreeService.NotFound("no driver '" + driver + "'"); + } + org.w3c.dom.Element app = d.config.get(Driver.APP_SCHEMA); + Map m = new LinkedHashMap<>(); + m.put("driver", driver); + m.put("present", app != null); + List classes = new ArrayList<>(); + if (app != null) { + m.put("hierarchical", app.getAttribute("hierarchical")); + for (org.w3c.dom.Element c : SchemaOps.children(app, "class-def")) { + Map cm = new LinkedHashMap<>(); + cm.put("name", c.getAttribute("class-name")); + cm.put("container", "true".equals(c.getAttribute("container"))); + List attrs = new ArrayList<>(); + for (org.w3c.dom.Element a : SchemaOps.children(c, "attr-def")) { + Map am = new LinkedHashMap<>(); + am.put("name", a.getAttribute("attr-name")); + am.put("type", a.getAttribute("type")); + am.put("multi", "true".equals(a.getAttribute("multi-valued"))); + am.put("required", "true".equals(a.getAttribute("required"))); + am.put("naming", "true".equals(a.getAttribute("naming"))); + attrs.add(am); + } + cm.put("attributes", attrs); + classes.add(cm); } - cm.put("attributes", attrs); - classes.add(cm); } - } - m.put("classes", classes); - return m; + m.put("classes", classes); + return m; + }); } /** {@code vault.schema}: read the environment's schema into the tree (a vault read, a tree write). */ diff --git a/src/main/java/com/pointblue/dirxml/web/Workspace.java b/src/main/java/com/pointblue/dirxml/web/Workspace.java index 40366233..ca3c774d 100644 --- a/src/main/java/com/pointblue/dirxml/web/Workspace.java +++ b/src/main/java/com/pointblue/dirxml/web/Workspace.java @@ -38,9 +38,11 @@ private Workspace(String id, String branch, Path root, Path tree, Path cases, Pa this.packages = new PackageService(tree, catalog); service.catalog(packages.catalogArg()); this.vault = new VaultService(tree, files); + this.vault.models(service); this.sim = new SimService(tree, SimService.locate(tree, cases)); this.vault.setCasesDir(this.sim.casesDir()); this.prov = new ProvisioningService(tree); + this.prov.models(service); this.credentials = new CredentialService(tree, files); this.events = new EventsService(tree, sim.casesDir(), files); this.traces = new TraceFiles(Path.of(System.getProperty("user.home"), ".idm", "traces", id.replaceAll("[^A-Za-z0-9._-]+", "-"))); diff --git a/src/test/java/com/pointblue/dirxml/web/SharedModelTest.java b/src/test/java/com/pointblue/dirxml/web/SharedModelTest.java new file mode 100644 index 00000000..b1778e5f --- /dev/null +++ b/src/test/java/com/pointblue/dirxml/web/SharedModelTest.java @@ -0,0 +1,67 @@ +package com.pointblue.dirxml.web; + +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertNotSame; +import static org.junit.Assert.assertSame; + +import com.pointblue.dirxml.dev.model.DriverSet; +import java.nio.file.Files; +import java.nio.file.Path; +import java.nio.file.attribute.FileTime; +import org.junit.Rule; +import org.junit.Test; +import org.junit.rules.TemporaryFolder; + +/** + * The schema and provisioning reads share the outline's copy of the tree instead of reading it + * from disk on each request (#177), and still see a change written a moment ago. + */ +public class SharedModelTest { + + @Rule + public TemporaryFolder tmp = new TemporaryFolder(); + + private Path tree() throws Exception { + Path t = tmp.newFolder("tree").toPath(); + Files.writeString(t.resolve("driverset.xml"), "\n"); + Files.createDirectories(t.resolve("drivers/Plain")); + Files.writeString(t.resolve("drivers/Plain/driver.xml"), "\n"); + return t; + } + + @Test + public void readsShareOneModelUntilAFileChanges() throws Exception { + Path tree = tree(); + try (Watcher w = new Watcher(tree, 10_000)) { + TreeService service = new TreeService(tree, w); + DriverSet first = service.withModel(ds -> ds); + assertSame("a second read reuses the model", first, service.withModel(ds -> ds)); + + // a new driver, written without the watcher running: the next read sees it at once + Files.writeString(tree.resolve("driverset.xml"), "" + + "\n"); + Files.setLastModifiedTime(tree.resolve("driverset.xml"), FileTime.fromMillis(System.currentTimeMillis() + 5_000)); + Files.createDirectories(tree.resolve("drivers/Other")); + Files.writeString(tree.resolve("drivers/Other/driver.xml"), "\n"); + DriverSet second = service.withModel(ds -> ds); + assertNotSame(first, second); + assertEquals(2, second.drivers.size()); + } + } + + @Test + public void theServicesReadThroughTheSharedModel() throws Exception { + Path tree = tree(); + try (Watcher w = new Watcher(tree, 10_000)) { + TreeService service = new TreeService(tree, w); + VaultService vault = new VaultService(tree, EnvFiles.single(tree.resolve("no-such-environments.properties"))); + vault.models(service); + ProvisioningService prov = new ProvisioningService(tree); + prov.models(service); + + assertEquals(Boolean.FALSE, vault.appSchema("Plain").get("present")); + assertEquals(Boolean.FALSE, vault.schema().get("present")); + assertEquals("[]", prov.catalog().get("drivers").toString()); + } + } +} From c5f5053390d2e15bce778af9914bccab6cb13d86 Mon Sep 17 00:00:00 2001 From: Jerry Combs Date: Thu, 29 Oct 2026 08:50:56 -0500 Subject: [PATCH 2/2] Pin DirXMLDev 0.59.0; 0.83.0 Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 7 +++++++ pom.xml | 4 ++-- src/main/java/com/pointblue/dirxml/web/Serve.java | 2 +- 3 files changed, 10 insertions(+), 3 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index c171af6a..e7be0524 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,11 +7,18 @@ and this project uses [Semantic Versioning](https://semver.org/spec/v2.0.0.html) ## [Unreleased] +## [0.83.0] - 2026-10-29 + +Needs DirXMLDev 0.59.0. + ### Fixed - A large project no longer runs the server out of memory on the schema and provisioning pages (#177). Each of their requests read and parsed the whole tree again, and a page sends several at once. They now read the tree the outline keeps, read again only when its files change. +- The warning that `hosted.properties` "is readable by other users" no longer shows on a folder + that ignores `chmod`, such as a Windows folder mounted into the container, where its advice + can't be followed (DirXMLDev 0.59.0). ### Changed diff --git a/pom.xml b/pom.xml index f7358f95..fc60eb00 100644 --- a/pom.xml +++ b/pom.xml @@ -6,7 +6,7 @@ com.pointblue.dirxml dirxml-dev-web - 0.82.0 + 0.83.0 MIT License @@ -23,7 +23,7 @@ - 0.58.0 + 0.59.0 1.7.0