Like the code in /packages/cletus/src/helpers/files.ts let's create a package that is for breaking down an input stream + a "type" into hierarchy of resources converted into markdown or text. Look at that file to know what I mean.
This library should allow custom type parsers and custom type slicers. Parsers take an input stream and converts it into file parts, where file parts are text and possible page numbers and images that may or may not be summarized or transcribed. Slicers take text and targeting particular token/char limits for a specific model breaks the text down into smaller chunks. If it were a programming language it might try to split by function, erc. Markdown could be by paragraphs or titled sections or something. Importantly there would be options so there is the text split up (so if all were to be concatenated it would equal the original) and there is another piece of text for each section that would be what would be embedded, which in a code file was maybe the method, class, etc. for markdown it could be all previous headers for that content.
I want as many types to be handled as possible for what's available in node.
I want a registry for all these things and a default one which includes everything. Hopefully built in a way that if you don't import the default registry you won't get all the dependencies bundled.
I want to easily be able to check for support.
I want so many options so every possible thing I might want to include in the embeddable text to add context is possible for any of the supported types.
Each parseable type needs to define what the default slicer is. Some parses will handle multiple types and they all will have the same default slicer but can have more specific ones.
This needs to be easily extendable.
A user needs to have a way to know if a type is supported. That means any type that depends on a locally installed cli tool that might need to check physically instead of just relying on the fact that it mapped.
Every inner part of a file should have a unique identifier - ie location. File path > image X > description[0]. That's not a good format, just an example. That way every piece of text has a well known source even if it were buried in a pdf, zip file, etc.
Slicers can also have code to extract resource links. So each resource (the final name of the type that was previously File). So each resource also has links to other resources. Those could be relative resources, or an external url, etc. so you could point all of this at an HTML page and extract all the links to all the other sites and resources and referenced images, etc.
I want to be able to also have a link resolver. A link (text in particular format) to a resource could resolve it to a type and input stream. Could be a url. A file. If it's an HTML result from a url, maybe puppeteer is used (see ginny) to render the full webpage and then that is processed by the html type. Etc.
I want something I can point any resource location and it will be loaded into text and images and I can convert it to embeddable things and it also has links to all the other resources relative to it that I might want to crawl. Another example of that is a code file importing a relative file - that's a link.
Come up with a plan for me to approve.
Like the code in /packages/cletus/src/helpers/files.ts let's create a package that is for breaking down an input stream + a "type" into hierarchy of resources converted into markdown or text. Look at that file to know what I mean.
This library should allow custom type parsers and custom type slicers. Parsers take an input stream and converts it into file parts, where file parts are text and possible page numbers and images that may or may not be summarized or transcribed. Slicers take text and targeting particular token/char limits for a specific model breaks the text down into smaller chunks. If it were a programming language it might try to split by function, erc. Markdown could be by paragraphs or titled sections or something. Importantly there would be options so there is the text split up (so if all were to be concatenated it would equal the original) and there is another piece of text for each section that would be what would be embedded, which in a code file was maybe the method, class, etc. for markdown it could be all previous headers for that content.
I want as many types to be handled as possible for what's available in node.
I want a registry for all these things and a default one which includes everything. Hopefully built in a way that if you don't import the default registry you won't get all the dependencies bundled.
I want to easily be able to check for support.
I want so many options so every possible thing I might want to include in the embeddable text to add context is possible for any of the supported types.
Each parseable type needs to define what the default slicer is. Some parses will handle multiple types and they all will have the same default slicer but can have more specific ones.
This needs to be easily extendable.
A user needs to have a way to know if a type is supported. That means any type that depends on a locally installed cli tool that might need to check physically instead of just relying on the fact that it mapped.
Every inner part of a file should have a unique identifier - ie location. File path > image X > description[0]. That's not a good format, just an example. That way every piece of text has a well known source even if it were buried in a pdf, zip file, etc.
Slicers can also have code to extract resource links. So each resource (the final name of the type that was previously File). So each resource also has links to other resources. Those could be relative resources, or an external url, etc. so you could point all of this at an HTML page and extract all the links to all the other sites and resources and referenced images, etc.
I want to be able to also have a link resolver. A link (text in particular format) to a resource could resolve it to a type and input stream. Could be a url. A file. If it's an HTML result from a url, maybe puppeteer is used (see ginny) to render the full webpage and then that is processed by the html type. Etc.
I want something I can point any resource location and it will be loaded into text and images and I can convert it to embeddable things and it also has links to all the other resources relative to it that I might want to crawl. Another example of that is a code file importing a relative file - that's a link.
Come up with a plan for me to approve.