Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feature -- LiteLLM integration for Ecologits #61

Merged
merged 8 commits into from
Jul 17, 2024

Conversation

cvarrei
Copy link
Contributor

@cvarrei cvarrei commented Jul 16, 2024

Integration added for the LiteLLM proxy-like library. This library standardises LLM API calls across the OpenAI pattern. This makes it easier to integrate LLMs calling into production.

Done -
I have adapted the documentation, made the LiteLLM tracer, add the corresponding tests, add the instrumentor, and adapted the retrieval of information for calculating the impact to suit LiteLLM without affecting the other tracers.

I have also updated the TOML for dependencies. As LiteLLM relies on all the current main organisations, when initialising litellm in the script, it will initialise all the ecologits instruments. Thus, the user has to install all the other libraries (openai, mistralai, etc.).

Missing - (I'm leaving it open for you to edit, to make it easier for you to adapt the changes to your needs):

  • Updating poetry.lock As you are currently making changes, it will be easier for you to achieve the update with your current changes. The TOML has been modified to make it easier for you.
  • cassettes: I've had several raised errors using it, but my tests without it have been successful. I think it will be easier for you to adapt it to your testing strategy.

Congratulations on this project, it's been great to be able to contribute to it.

Copy link
Member

@adrienbanse adrienbanse left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@cvarrei Many thanks for your PR! 🥳

I made some modifications:

  • I updated LiteLLM
  • I took care of the lock and cassettes (although I had to change the OpenAI model since I don't have credits anymore - @samuelrince you can re-change this if you wish)
  • I integrated your .md doc file in the documentation
  • I removed all the dependencies as I don't think that it's needed (they are in the dependencies of LiteLLM no?) - please check if your OK with this 529794e

I still just have one doubt about line 40 of model_repository.py

Thanks!!

ecologits/model_repository.py Show resolved Hide resolved
@samuelrince samuelrince merged commit 1333f34 into genai-impact:main Jul 17, 2024
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants