Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

test(sdk, api): fix finish reason, chat, audio and completion tests #1038

Open
wants to merge 108 commits into
base: main
Choose a base branch
from

Conversation

justinthelaw
Copy link
Contributor

@justinthelaw justinthelaw commented Sep 17, 2024

Description

Creates and enables vLLM E2E tests that touch each SDK gRPC endpoint at least once with a "happy path" test. Standardizes the new Completions test from vLLM's E2E tests to the LLaMA-CPP-Python E2E tests. Fixes the FinishReason enum typing issues across Completions and ChatCompletions.

BREAKING CHANGES

  • fixes FinishReason to be an enum in both Completions and ChatCompletions protobufs
    • modifies API gRPC handler, typing utils, and helper utils to use a Enum class to define and transform the stub responses
  • fixes object and created field for Completions, using Literal text_completion as defined in the OpenAI API specification

CHANGES

  • adds and enables simple ("happy path") E2E vLLM testing for local environments only (GPU runner is broken, see commit history)
  • adds Completions E2E test to llama-cpp-python and vLLM to catch more potential issues, like FinishReason being the wrong type
  • adds Audio and Completions unit testing
  • condenses tests into a single test file with an ENV parser (default, warning and helper text included) for model name
  • adds more comprehensive Make target for test and typing artifact clean-up
  • improves testing documentation and Makefile targets

Related Issue

Fixes #1037

Relates to #854

Checklist before merging

@justinthelaw justinthelaw added tech-debt Not a feature, but still necessary blocked 🛑 Something needs to happen before this issues is worked labels Sep 17, 2024
@justinthelaw justinthelaw self-assigned this Sep 17, 2024
@justinthelaw justinthelaw requested a review from a team as a code owner September 17, 2024 01:07
@justinthelaw justinthelaw linked an issue Sep 17, 2024 that may be closed by this pull request
Copy link

netlify bot commented Sep 17, 2024

Deploy Preview for leapfrogai-docs canceled.

Name Link
🔨 Latest commit cac0fdb
🔍 Latest deploy log https://app.netlify.com/sites/leapfrogai-docs/deploys/66eca5b299eeee000876a768

@justinthelaw
Copy link
Contributor Author

There seems to be an issue with our GPU runner configurations that is blocking the vLLM E2E tests from running: https://github.com/defenseunicorns/leapfrogai/actions/runs/10894219888/job/30230952202

The upstream UDS Common action is unable to install UDS CLI into the bin directory. It seems like it needs sudo or higher permissions, even though that is not usually required in our CPU (large or regular) runners. As a side note, the vLLM E2E tests run locally.

@justinthelaw justinthelaw marked this pull request as draft September 17, 2024 18:00
@justinthelaw justinthelaw changed the title test(vllm): fix FinishReason, standardize and enable vLLM E2E tests fix(test): fix finish reason, chat and completion e2e tests Sep 18, 2024
@justinthelaw justinthelaw changed the title fix(test): fix finish reason, chat and completion e2e tests fix(sdk, test): fix finish reason, chat and completion e2e tests Sep 18, 2024
@justinthelaw justinthelaw marked this pull request as ready for review September 18, 2024 21:36
@justinthelaw justinthelaw requested review from a team and jalling97 September 18, 2024 22:13
@justinthelaw justinthelaw changed the title fix(sdk, test): fix finish reason, chat and completion e2e tests fix(sdk, test): fix finish reason, chat, audsio and completion tests Sep 19, 2024
@justinthelaw justinthelaw changed the title fix(sdk, test): fix finish reason, chat, audsio and completion tests fix(sdk, test): fix finish reason, chat, audio and completion tests Sep 19, 2024
@justinthelaw justinthelaw changed the title fix(sdk, test): fix finish reason, chat, audio and completion tests test(sdk, api): fix finish reason, chat, audio and completion tests Sep 19, 2024
@justinthelaw justinthelaw marked this pull request as draft September 19, 2024 15:03
@justinthelaw justinthelaw force-pushed the 1037-testvllm-implement-e2e-testing-for-vllm branch from 70f58ad to a1a03c1 Compare September 19, 2024 20:07
@justinthelaw justinthelaw marked this pull request as ready for review September 19, 2024 22:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
api chore tech-debt Not a feature, but still necessary
Projects
None yet
Development

Successfully merging this pull request may close these issues.

chore(vllm): implement e2e testing for vllm
3 participants