feat: repo map

[yetone]

#122

[yetone]

This PR is still in rapid iteration and is not yet ready for review.

[jmmarotta]

You could probably use the aider queries for treesitter.

[aarnphm]

we are iterating a lot atm, and will consider refactor the query later.

[yetone]

You could probably use the aider queries for treesitter.您可能可以使用树木管理员的辅助查询。

aider's queries return all class/function definitions in the file, but I only need the externally exposed classes/functions (similar to a C header file), so their code doesn't suit my requirements.

[yetone]

Here's an update for everyone: Actually, this PR was completed a long time ago. The functionality tests are all normal and the results are very good, but it consumes a large number of tokens (to test this PR, my Anthropic quota has already been exhausted 😢). We are currently considering how to optimize this aspect.

[SchnozzleCat]

Here's an update for everyone: Actually, this PR was completed a long time ago. The functionality tests are all normal and the results are very good, but it consumes a large number of tokens (to test this PR, my Anthropic quota has already been exhausted 😢). We are currently considering how to optimize this aspect.

Is the consumed token amount similar to what is consumed when using something like Aider (and adding all files in a project)?

[jmmarotta]

Is the consumed token amount similar to what is consumed when using something like Aider (and adding all files in a project)?

The core issue seems to be that the current implementation is using the entire repomap instead of focusing on the most relevant tags. A potential solution would involve finding a way to rank the tags produced by tree-sitter.

In Python ecosystems, libraries like NetworkX (used in aider) can represent these tags as nodes in a graph and then apply algorithms like PageRank to rank them effectively.

[aarnphm]

afaik nvim has binding to python, but it is kinda slow.

Maybe we can do reranking through colbert from rust/c++ or sth.

[jmmarotta]

Pagerank might work better than colbert as it's well-suited for treesitter's tag structure, but using rust/c++ is a great idea.

petgraph might work well

[aarnphm]

petgraph might work well

Thanks for the knowledge sharing, will take a look into this

[yetone]

Is the consumed token amount similar to what is consumed when using something like Aider (and adding all files in a project)?

The core issue seems to be that the current implementation is using the entire repomap instead of focusing on the most relevant tags. A potential solution would involve finding a way to rank the tags produced by tree-sitter.

In Python ecosystems, libraries like NetworkX (used in aider) can represent these tags as nodes in a graph and then apply algorithms like PageRank to rank them effectively.

In my local tests, aider's repo map tends to be inaccurate, bringing in unrelated files while ignoring related ones.

[jmmarotta]

In my local tests, aider's repo map tends to be inaccurate, bringing in unrelated files while ignoring related ones.

PageRank likely overweights common utilities. For a more localized context BFS might be best.