Cache for Better Performance - Reflection & LLM

[davin111]

I think we can utilize internal caches for not executing reflection and many logic every time. This is surely optional feature, I think we can add it after the initial release. I'm interested in the implementation.

[mscheong01]

I've done something similar in the past: we could run all the reflection logic at configuration time (when beans are created), store the information somewhere and use it when the proxy is used. It might not have a big impact on the execution time since the main bottleneck will be from the llm apis, but it would save some computing resource.

[mscheong01]

jackson also has a reflection cache

[joowon-dm-snu]

since the main bottleneck will be from the llm apis

LLM API also can be cached with this concept (link)
Although the above library don't support �Kotlin, but it might be helpful to get some ideas.

[mscheong01]

@joowon-dm-snu Thank you for sharing! 🍪
GPTCache seems to implement caches using vector similarity requiring an external vector storage and a cache storage. Since we standardize LLM requests into JVM methods, I think we can try something simpler, similar to Spring caches - by caching the response object using the method call parameter values as cache key.
We could start with local caches and later support external cache storages. Overall, supporting caches for the LLM call itself is a good idea. Thanks!