Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow to set a SetLibrarySearchPath in the golang bindings #981

Merged
merged 1 commit into from
Jun 14, 2023

Conversation

mudler
Copy link
Contributor

@mudler mudler commented Jun 14, 2023

This is used to identify the path where all the various implementations are

Describe your changes

This adds SetLibrarySearchPath as model option to search lib path in the specified directory instead of the current work directory.

Issue ticket number and link

#826

Checklist before requesting a review

  • I have performed a self-review of my code.
  • If it is a core feature, I have added thorough tests.
  • I have added thorough documentation for my code.
  • I have tagged PR with relevant project labels. I acknowledge that a PR without labels may be dismissed.
  • If this PR addresses a bug, I have provided both a screenshot/video of the original bug and the working solution.

Demo

Before:

11:59AM DBG Loading model in memory from file: /home/mudler/_git/LocalAI/models/ggml-gpt4all-j                                                                                                                       
Debug: New GPT4ALL                                                                                                                                                                                                          
Debug: Library search path: /tmp/localai/backend_data                                                                                                                                                                       
load_gpt4all_model: error 'Success'                                                                                                                                                                                  
11:59AM DBG [gpt4all] Fails: failed loading mode

After:

gptj_model_load: loading model from '/home/mudler/_git/LocalAI/models/ggml-gpt4all-j' - please wait ...                                                                                                              
gptj_model_load: n_vocab = 50400                                                                                                                                                                                     
gptj_model_load: n_ctx   = 2048                                                                                                                                                                                      
gptj_model_load: n_embd  = 4096
gptj_model_load: n_head  = 16
gptj_model_load: n_layer = 28
gptj_model_load: n_rot   = 64
gptj_model_load: f16     = 2
gptj_model_load: ggml ctx size = 5401.45 MB
gptj_model_load: kv self size  =  896.00 MB
gptj_model_load: ................................... done
gptj_model_load: model size =  3609.38 MB / num tensors = 285
12:01PM DBG [gpt4all] Loads OK
12:01PM DBG Response: {"object":"chat.completion","model":"ggml-gpt4all-j","choices":[{"message":{"role":"assistant","content":"I'm doing well, thank you. How about yourself?"}}],"usage":{"prompt_tokens":0,"comple
tion_tokens":0,"total_tokens":0}}

Steps to Reproduce

This affect only binaries that are not in the same directory within the implementations

Notes

Without this it's impossible to specify a search path to find the various implementations libs.

This is used to identify the path where all the various implementations
are
Copy link
Contributor

@niansa niansa left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks correct to me, from all I can understand.

@mudler mudler merged commit b004c53 into nomic-ai:main Jun 14, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants