Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix compatibility with region inference #531

Merged
merged 2 commits into from
Sep 17, 2024

Conversation

chm10
Copy link
Contributor

@chm10 chm10 commented Sep 15, 2024

Tested with us-east-1 and will solve issue with EU regions too for Claude Sonnet 3.5.

*Issue #508 and #527

Description of changes:
Get the region from BEDROCK_REGION and adapt to cross reference region format.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

@statefb statefb self-requested a review September 17, 2024 00:59
@statefb
Copy link
Contributor

statefb commented Sep 17, 2024

@chm10 Hi, thank you so much for creating PR! LGTM except for CI failure (mypy check). Could you create a commit to fix it? (Need to add two spaces before comment)

region = BEDROCK_REGION.split("-")[0] # Cross-region inference

@statefb
Copy link
Contributor

statefb commented Sep 17, 2024

close #508
close #527

* Two spaces before comment
@statefb
Copy link
Contributor

statefb commented Sep 17, 2024

LGTM!

@statefb statefb merged commit 34228f0 into aws-samples:v1 Sep 17, 2024
6 checks passed
@chm10 chm10 deleted the fix/527-region-inference branch September 17, 2024 01:51
statefb added a commit that referenced this pull request Sep 17, 2024
statefb added a commit that referenced this pull request Sep 17, 2024
@statefb
Copy link
Contributor

statefb commented Sep 17, 2024

@chm10 Sorry, we need to consider following things. Could you re-create a PR including following modification?

  • Only limited regions supports inference profile (currently us and eu only)
    • we can check if the region supports by aws bedrock list-inference-profiles --profile <profile-name>
  • Due to the architecture to routing requests, latency happens (see doc). Could you add enableBedrockCrossRegionInference option to cdk.json with the default value false?
    • Even the value is set to true, if the bedrockRegion is not included in the supported region, model id without cross region inference should be chosen. Also warning log should be added.

Thank you in advance!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants