Background service for chunking and embedding data - #756

Open

PaulKoudelka wants to merge 52 commits into

MindWorkAI:mainfrom

PaulKoudelka:chunk-data

Commits on Apr 10, 2026

added functionality to save/move tokenizer file
PaulKoudelka
committed
0854deb
added functionality to add tokenizer to LLM and embedding models
PaulKoudelka
committed
e07ca37

Commits on Apr 14, 2026

token count now adapts to chosen provider tokenizer
PaulKoudelka
committed
4157340

Commits on Apr 15, 2026

cleanup to ensure uniform style of code
PaulKoudelka
committed
4a8ab90
removed print statements
PaulKoudelka
committed
95fb213

Commits on Apr 23, 2026

added possibility to add tokenizer via plugin and refactored earlier changes
PaulKoudelka
committed
aa554d4

Commits on Apr 24, 2026

Commits on May 8, 2026

background embed
PaulKoudelka
committed
ac677c5

Commits on May 13, 2026

added manual refresh and cleaned up code
PaulKoudelka
committed
637bf97

Commits on May 26, 2026

Merge branch 'main' into chunk-data
PaulKoudelka
committed
00b6c98

Commits on May 27, 2026

current main is now successfully merged into branch
PaulKoudelka
committed
ea87d79

Commits on Jun 10, 2026

Merge branch 'main' into chunk-data
PaulKoudelka
committed
b7b18aa
changed qdrant to qdrant edge
PaulKoudelka
committed
6989cbe

Commits on Jul 1, 2026

Merge branch 'main' into chunk-data
PaulKoudelka
committed
2e06af1

Commits on Jul 28, 2026

added sql integration instead of json files
PaulKoudelka
committed
c59e33e
Added dependencies for sql
PaulKoudelka
committed
2332e83
made file watchers more robust against multiple triggers
PaulKoudelka
committed
95ca748
made the chunking process more robust
PaulKoudelka
committed
1bf9328
improved stability of embedding chunks
PaulKoudelka
committed
759c7f9
added warning and improved error handling
PaulKoudelka
committed
fd2d998

Commits on Jul 29, 2026

made only custom tokenizer visible in chat
PaulKoudelka
committed
2829051
improved chunking pipeline and handling of expert settings
PaulKoudelka
committed
89f7156

Commits on Aug 3, 2026

made the embedding procedure more robust
PaulKoudelka
committed
b3ba945
added metadata for bm25 and also confidence levels
PaulKoudelka
committed
c78a8f0
added retrieval with vector and bm25 search
PaulKoudelka
committed
010f35b

Commits on Aug 4, 2026

Commits on Aug 5, 2026

Added security checks for confidence levels
PaulKoudelka
committed
185a99c

Commits on Aug 10, 2026

fixed tokenization issue for long files
PaulKoudelka
committed
3918b48
ensured prompts fit embedding model
PaulKoudelka
committed
9ab196c
changed DateTime to DateTimeOffset
PaulKoudelka
committed
6985678
made tokenizer concurrent and threadsafe
PaulKoudelka
committed
c8c1e38
fixed the expert settings for datasource_file
PaulKoudelka
committed
f9263b0
ensured consistent maximum token limit
PaulKoudelka
committed
2547701
fixed sqlite info
PaulKoudelka
committed
b4e916e
moved datasource indexing to lower level
PaulKoudelka
committed
202e374

Commits on Aug 11, 2026

ensured data sources can be renamed and made it secure
PaulKoudelka
committed
92e986d
made check for vectordb more robust
PaulKoudelka
committed
987f493

Commits on Aug 12, 2026

made datasources button an icon
PaulKoudelka
committed
c92e83e
disabled images for RAG for now
PaulKoudelka
committed
d95487a
improved code and made sure components are reused if possible
PaulKoudelka
committed
647c428

Commits on Aug 13, 2026

made datasource selection more user friendly
PaulKoudelka
committed
786b106

Commits on Aug 14, 2026

refactored confidence levels and ensured checks work
PaulKoudelka
committed
88999d9
fixed issue where augumentation failed
PaulKoudelka
committed
50aa5c4
replaced embedding_state with index_store as a sqlite name
PaulKoudelka
committed
1a26312
refactored the classes and functions
PaulKoudelka
committed
15afe1c
showed user default values in expert settings for datasources
PaulKoudelka
committed
aafc58b
translated the missing pieces
PaulKoudelka
committed
d883e9d