Adding optional Xapian support

[casouri]

Xapian is a nice indexing and search engine. I added optional Xapian backend support to deft so it can be faster when searching large bodies of notes. It can automatically download the dynamic module I wrote separately that exposes Xapian functionalities. Switching to Xapian backend is very easy, you only need to set deft-use-xapian to t. I've been using my note search package built on top of Xapian (xeft) for a while with great success.

As for the integration, I managed to add support for Xapian with not much code. It has lots of room for improvement, eg, it can load file cache lazily instead of loading all files into cache at the start, which is very slow for enough notes. But for the initial proposal I don't want to make too big of a modification to the codebase.

Anyway, please have a look :-) If you don't feel like adding Xapian support to deft, I totally understand, no hard feelings.

[EFLS]

This looks interesting! I'd like to check this out, but how can I do that? I don't see the deft-use-xapian option in the xeft.el on your Github. What am I missing?

[casouri]

I forgot to add the link to my fork, sorry. Here it is: https://github.com/casouri/deft/tree/xapian Note that the branch is xapian, not master. You can also give Xeft a try to see the potential of Xapian-backend.

To use the fork you need to set deft-xapian-database and set deft-use-xapian to t. I also suggest setting deft-file-limit to like 30 because inserting file summaries for too many results will nullify any speed gained by xapian.

[EFLS]

Thanks for the addition. I'm having trouble using Xapian on my macOS machine. Installed via brew, but then it asks me for the dynamic module. When I say to compile, it says compilation successful, but then returns progn: Cannot open load file: No such file or directory, xapian-lite

Anyway, this might not the place to discuss this issues. I'll have to investigate later.

EDIT: Seems like the compilation didn't actually compile anything. And downloading the module works, but then later gives an error (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64e')) – I guess because it's an M1 mac?

[casouri]

Yes, the pre-compiled binary are for intel Macs. You need to compile the module locally. After compiling, do you see xapian-lite.so under the same directory as deft.el? Does running make PREFIX=/opt/local under that directory compile anything?

[EFLS]

No, doesn't seem to work. This is what I get after make PREFIX=/opt/local/

emacs -q -no-site-file -no-init-file -batch -f batch-byte-compile deft.el

In deft--should-use-xapian:
deft.el:1586:11: Warning: defsubst `deft--should-use-xapian' was used before
    it was defined

In end of data:
deft.el:1656:6: Warning: the function `string-join' is not known to be
    defined.

It does create a deft.elc, but no xapian-lite.so

[casouri]

Ah, I know why it doesn't work now. I pushed a fix. Could you pull and try again? You can either run make xapian-lite.so PREFIX=/opt/local in project root or call deft and let it compile for you. Both should work now.

[EFLS]

Thanks for the follow up. After pulling changes I tried both, and both give the same error:

g++ module/xapian-lite.cc -o xapian-lite.so -shared -fPIC -I/opt/local/include -L/opt/local/lib -lxapian
module/xapian-lite.cc:19:10: fatal error: 'xapian.h' file not found
#include <xapian.h>
         ^~~~~~~~~~
1 error generated.
make: *** [xapian-lite.so] Error 1

I double checked, and brew says xapian is installed.

I tried the xeft repo as well, and it gives the same error after make

[casouri]

I thought homebrew on ARM Mac uses /opt/local, turns out it's actually /opt/homebrew. So you want to use make xapian-lite.so PREFIX= /opt/homebrew.

[EFLS]

That was it! It did compile now, and I'm trying it out. Seems fast indeed!

Could it be that some functionality of Deft is changed because of the Xapian search engine?

I wrote Zetteldeft (https://github.com/efls/zetteldeft), which implements plain text links via the Deft search functions. Following a link consists of searching for a unique ID in the filename. To achieve this, some of the search functions set deft-filter-only-filenames to t. But with Xapian enabled, this setting doesn't seem to be respected, and the Deft search results include both files with the search string in filename as well as file contents.

I haven't looked at the internals of your branch yet, but can you guess why this happens?

[casouri]

Yeah, I didn't add support for that variable. Xapian works by indexing words in a file to a database and querying the database when searching, so it can't search regex nor filename. I think it's easier and more efficient to write some elisp to search for filenames (saving all filenames to a buffer and search in the buffer) than implementing this in xapian.

[EFLS]

Makes sense. Might be important to point out functional differences because of the Xapian engine.

I think it's easier and more efficient to write some elisp to search for filenames (saving all filenames to a buffer and search in the buffer) than implementing this in xapian.

Yeah at some point in the future Zetteldeft should use a non-Deft function to search filenames. Perhaps creating a hash with file paths and filenames IDs or something.

[casouri]

Yeah maybe it isn't that good of an idea to use xapian for deft, since there are quite a few deft features that xapian cannot support: regex, filename-only, plus it's non-trivial to ditch the caching.

[pedro-nonfree]

@casouri I am new to search stuff in emacs, so I am seeing three projects: deft, notdeft and xeft, and I am a bit lost. I am very interested in the usage of xapian (in the way mu-mu4e or xapian-notmuch does to email).

What if we think about the xapian inclusion in deft just for the things that xapian is good, and let everything else a part

If you are planning to do a proof of concept of current deft plus xeft/xapian, I would be interested on trying it and providing feedback, let me know

[casouri]

That already exists in the form of a fork, no? You can see the link to it in my second message.

[pedro-nonfree]

Context: 51 MB of org files, and I am a fan of big files, and I know you are fan of small files, that's what for deft is intended.

[2024-11-04 Mon 13:22:47] $ du -shc --apparent-size ~/org/*.org | grep total
51M	total

I finally could find time to test deft, notdeft and xeft:

• deft collapses on initialization, maybe because it opens too many and too big files.
• notdeft probably collapses on initialization, but fails anyway because is unmaintained? (emacs 30.0.90 2024-08-30, 9.7.10), I can provide more details but maybe does not matter.
• xeft works (good catch to limit by default with 15 elements and allow showing all), but is very simple. but very nice that a library for making xapian applications with xeft is available 🎉 https://github.com/casouri/xapian-lite
• deadgrep probably offtopic, but works very well, of course, due to lack of search engine index, processes everything in the moment like grep does

So I no longer see why deft should incorporate somehow xapian if I cannot even open it