How to index all locally installed packages?

[seagreen]

I'd like to index not just the package I'm working on, but also the rest of the packages in the stack.yaml's snapshot, as well as the packages in extra-deps.

Is there a way to do this, perhaps what haskell-code-explorer.mfix.io does?

[alexwl]

Is there a way to do this, perhaps what haskell-code-explorer.mfix.io does?

In order to index packages for haskell-code-explorer.mfix.io, I wrote a stack script (https://docs.haskellstack.org/en/stable/GUIDE/#script-interpreter) that downloads, builds and indexes a subset of packages from a Stackage snapshot. I plan to release that script (I need to clean it up a bit first).

I'd like to index not just the package I'm working on, but also the rest of the packages in the stack.yaml's snapshot, as well as the packages in extra-deps.

I think the easiest way is to write a simple shell script.

For example, this command downloads the source code of all dependencies of a stack project to PATH:

stack ls dependencies | tr " " - | xargs -i stack unpack {} --to PATH

This command tries to build (and index) each package in PATH with stack using RESOLVER (e.g., lts-13.13):

for p in PATH/*; do (cd "$p" && stack init --resolver=RESOLVER --force && stack build && PATH_TO_HASKELL_CODE_INDEXER -p .); done

(It should work for most packages except for ghc, base, ghc-prim, etc.)

[seagreen]

Sweet, that worked. I've got an example up here: https://github.com/seagreen/hermetic/tree/master/explore

Some thoughts:

• Like you warned it doesn't work for base. How'd you get base indexed for haskell-code-explorer.mfix.io?

• It copies in this premake stack.yaml instead of using stack init:

  resolver: lts-13.0
  allow-newer: true

  The project being indexed uses lts-13.0, so I know that's right without having to run stack init. And without the allow-newer I had a lot of packages fail to build due to a restrictive upper bound on QuickCheck. I'm guessing LTS 13 has a setting in there to override them, which is why they work together in that LTS but don't work when trying to build them individually. I didn't look into it deeply though because since I already know those packages work together I can set allow-newer.

• The application consists of two local packages, hermetic and json-relay. I placed all their dependencies in a deps directory, but decided to index those two in place, because if I copied them to deps it would be easy to let them get out of sync as the project develops. In order to use haskell-code-server with both deps and those two local packages I used this PR: Allow specifying both --packages and --package. #31

  You can see how this worked in action here: https://github.com/seagreen/hermetic/blob/669504a346b2d99f90100527bca10e681ac9e695/explore/serve

[alexwl]

• I used cabal-install (cabal executable) and globally installed GHC to build base. Building base requires certain system libraries and tools that are checked during the configuration step (e.g., cabal configure -f integer-gmp). Some libraries may need to be installed manually.

I'm guessing LTS 13 has a setting in there to override them, which is why they work together in that LTS but don't work when trying to build them individually.

It seems to be the case. For example, LTS 13.0 contains QuickCheck-2.12.6.1 and text-1.2.3.1, and at the same time build component tests of text-1.2.3.1 package depends on QuickCheck >= 2.7 && < 2.11 (the upper bound is too restrictive).

Now, I see that copying stack.yaml with allow-newer:true is a better approach than using stack init command.

• Thanks for the working example!
  I like the new feature (PR Allow specifying both --packages and --package. #31).

[seagreen]

I like the new feature (PR #31).

Glad you like it. Thank you for fixing the PR!

I used cabal-install (cabal executable) and globally installed GHC to build base. Building base requires certain system libraries and tools that are checked during the configuration step (e.g., cabal configure -f integer-gmp). Some libraries may need to be installed manually.

Ah, interesting. I'll take a stab at this sometime and update my example.

[rikvdkleij]

@alexwl

For example, this command downloads the source code of all dependencies of a stack project to PATH:

stack ls dependencies | tr " " - | xargs -i stack unpack {} --to PATH
This command tries to build (and index) each package in PATH with stack using RESOLVER (e.g., lts-13.13):

for p in PATH/*; do (cd "$p" && stack init --resolver=RESOLVER --force && stack build && PATH_TO_HASKELL_CODE_INDEXER -p .); done
(It should work for most packages except for ghc, base, ghc-prim, etc.)

I do not understand this approach. When a project is built, all dependencies are already built. Why do you want to explicitly download all sources and build them again?

[alexwl]

@rikvdkleij

The indexer uses GHC API (http://hackage.haskell.org/package/ghc-8.6.1) to collect information from Haskell AST (Abstract Syntax Tree). The only way to get an AST of a Haskell module is to initialize a GHC API session (https://gitlab.haskell.org/ghc/ghc/wikis/commentary/compiler/api) and to load a source file using setTargets or addTargets function (http://hackage.haskell.org/package/ghc-8.6.1/docs/GHC.html#v:setTargets).

Each already-built dependency is a collection of interface (.hi) files (https://gitlab.haskell.org/ghc/ghc/wikis/commentary/compiler/iface-files) and object files. It is not impossible to get an AST from an interface file (there is a project https://gitlab.haskell.org/ghc/ghc/wikis/hie-files that should fix it in new versions of GHC).