Dashboard data file upload limit

[anonymousys99]

The upload limit of data files is limited to 20MB. Our zip files are larger than the limit. Please make the limit configurable.

[mduft]

Sorry for the late reply - vacation season is here :D

I will take care to discuss this with the team.

[mduft]

The reason for this limitation is actually quite deep down in the way that these file updates are processed. These APIs are still built on quite old'ish stuff right now. They are encoded and transferred as part of a JSON payload. The JSON deserializer has a 20MB limit by default.

Right now there is no plan in changing much of that infrastructure. You can adapt JacksonHelper.java (in createObjectMapper) with something like this in your local build:

     result.getFactory().setStreamReadConstraints(StreamReadConstraints.builder().maxStringLength(1024 * 1024 * 256).build());

And then in add-data-file.component.ts change the size limit from 1024 * 1024 * 14 also to 1024 * 1024 * 250. Note the few MB to allow for other than pure file content especially when encoding.

Also, make sure to give the server quite substantially more memory when running like this. The file needs to fit into RAM all at once in this scenario multiple times - the whole thing is not really optimized for this case.

In my testing, with limits increased to 256MB, I was able to upload a ~190MB file just fine with Xmx for the server set to 2G.

Unfortunately to properly integrate this, we'd have to re-write and re-think the whole file transfer mechanism. This upload has been an addition based on framework that is not well-suited for the task. I fear this will take time (we don't have a large team working on this).