feat: Add Oracle Cloud Infrastructure (OCI) Generative AI client support#718
feat: Add Oracle Cloud Infrastructure (OCI) Generative AI client support#718fede-kamel wants to merge 29 commits intocohere-ai:mainfrom
Conversation
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
fede-kamel
force-pushed
the
feat/oci-client
branch
from
0e341c1 to
fdebc00
Compare
fede-kamel
force-pushed
the
feat/oci-client
branch
from
a284ea8 to
d7c7ef6
Compare
|
@walterbm-cohere @daniel-cohere @billytrend-cohere Hey maintainers, Friendly bump on this PR - would appreciate your feedback when you have a chance. Happy to address any concerns or make changes as needed. Thanks. |
fede-kamel
force-pushed
the
feat/oci-client
branch
from
49f92cc to
8b9c63d
Compare
|
Addressed Bugbot feedback:
|
fede-kamel
force-pushed
the
feat/oci-client
branch
from
3bf4c54 to
b78c63e
Compare
|
@billytrend-cohere @mkozakov @sanderland @abdullahkady — would appreciate a review on this when you have a moment. This PR has been rebased on the latest What this adds: Oracle Cloud Infrastructure (OCI) Generative AI client support, following the same architectural pattern as the existing Testing: 14 integration tests passing against the OCI Generative AI service. Tested with Command R, Command A, and all embed v3 models. This would bring OCI to parity with the existing Bedrock integration and benefit enterprise customers running Cohere models on Oracle Cloud. Happy to address any feedback. Thank you. |
|
@sanderland Thanks for the approvals on this PR and the others (#717, #698, #697)! What are the next steps to get these merged? |
|
@sanderland quick ping on this one since you approved earlier - could you please take a final look when you have a moment? Thanks! |
|
@sanderland @billytrend-cohere quick follow-up on this OCI PR. It currently has approval and is rebased on latest |
|
@sanderland quick follow-up on this OCI PR. This PR is important for Oracle + Cohere users because it adds first-class support for running Cohere models through Oracle Cloud Infrastructure (OCI) Generative AI while keeping the same Why this is relevant:
Current status:
Could you share the specific remaining blocker(s) or required changes to merge? If review scope is the issue, I can split this into smaller PRs (auth/client scaffolding first, then chat/embed/streaming) to accelerate. |
|
@billytrend-cohere looping you in as well on the OCI PR context above. If there are specific blockers or changes you want before merge, I can address them quickly or split this into smaller PRs to make review easier. |
fede-kamel
force-pushed
the
feat/oci-client
branch
from
1119be8 to
12938af
Compare
fede-kamel
force-pushed
the
feat/oci-client
branch
from
fb5c414 to
b3fcdee
Compare
…issues - Fix OCI pip extras installation by moving from poetry groups to extras - Changed [tool.poetry.group.oci] to [tool.poetry.extras] - This enables 'pip install cohere[oci]' to work correctly - Fix streaming to stop properly after [DONE] signal - Changed 'break' to 'return' in transform_oci_stream_wrapper - Prevents continued chunk processing after stream completion
- Add support for OCI profiles using security_token_file - Load private key properly using oci.signer.load_private_key_from_file - Use SecurityTokenSigner for session-based authentication - This enables use of OCI CLI session tokens for authentication
This commit addresses all copilot feedback and fixes V2 API support: 1. Fixed V2 embed response format - V2 expects embeddings as dict with type keys (float, int8, etc.) - Added is_v2_client parameter to properly detect V2 mode - Updated transform_oci_response_to_cohere to preserve dict structure for V2 2. Fixed V2 streaming format - V2 SDK expects SSE format with "data: " prefix and double newline - Fixed text extraction from OCI V2 events (nested in message.content[0].text) - Added proper content-delta and content-end event types for V2 - Updated transform_oci_stream_wrapper to output correct format based on is_v2 3. Fixed stream [DONE] signal handling - Changed from break to return to stop generator completely - Prevents further chunk processing after [DONE] 4. Added skip decorators with clear explanations - OCI on-demand models don't support multiple embedding types - OCI TEXT_GENERATION models require fine-tuning (not available on-demand) - OCI TEXT_RERANK models require fine-tuning (not available on-demand) 5. Added comprehensive V2 tests - test_embed_v2 with embedding dimension validation - test_embed_with_model_prefix_v2 - test_chat_v2 - test_chat_stream_v2 with text extraction validation All 17 tests now pass with 7 properly documented skips.
- Add comprehensive limitations section to README explaining what's available on OCI on-demand inference vs. what requires fine-tuning - Improve OciClient and OciClientV2 docstrings with: - Clear list of supported APIs - Notes about generate/rerank limitations - V2-specific examples showing dict-based embedding responses - Add checkmarks and clear categorization of available vs. unavailable features - Link to official OCI Generative AI documentation for latest model info
…sion
This commit fixes two issues identified in PR review:
1. V2 response detection overriding passed parameter
- Previously: transform_oci_response_to_cohere() would re-detect V2 from
OCI response apiFormat field, overriding the is_v2 parameter
- Now: Uses the is_v2 parameter passed in (determined from client type)
- Why: The client type (OciClient vs OciClientV2) already determines the
API version, and re-detecting can cause inconsistency
2. Security token file path not expanded before opening
- Previously: Paths like ~/.oci/token would fail because Python's open()
doesn't expand tilde (~) characters
- Now: Uses os.path.expanduser() to expand ~ to user's home directory
- Why: OCI config files commonly use ~ notation for paths
Both fixes maintain backward compatibility and all 17 tests continue to pass.
- Fix authentication priority to prefer API key auth over session-based - Transform V2 content list items type field to uppercase for OCI format - Remove debug logging statements All tests passing (17 passed, 7 skipped as expected)
Support the thinking/reasoning feature for command-a-reasoning-08-2025 on OCI. Transforms Cohere's thinking parameter (type, token_budget) to OCI format and handles thinking content in both non-streaming and streaming responses.
- Remove unused response_mapping and stream_response_mapping dicts - Remove unused transform_oci_stream_response function - Remove unused imports (EmbedResponse, Generation, etc.) - Fix crash when thinking parameter is explicitly None - Fix V2 chat response role not lowercased (ASSISTANT -> assistant) - Fix V2 finish_reason incorrectly lowercased (should stay uppercase) - Add unit tests for thinking=None, role lowercase, and finish_reason
- Fix thinking token_budget → tokenBudget (camelCase for OCI API) - Add V2 response toolCalls → tool_calls conversion for SDK compatibility - Update test for tokenBudget casing - Add test for tool_calls conversion
OCI doesn't provide a generation ID in responses. Previously used modelId which is the model name (e.g. 'cohere.command-r-08-2024'), not a unique generation identifier. Now generates a proper UUID.
- Add validation for direct credentials (user_id requires fingerprint and tenancy_id) - Emit message-end event for V2 streaming before [DONE]
fede-kamel
force-pushed
the
feat/oci-client
branch
from
a9f125f to
9f1e924
Compare
|
Validation on current PR head OCI test results from this branch:
Models exercised in the passing live runs:
Expected skips in live OCI remain limited to service/model availability constraints:
So on this PR head, all runnable OCI tests pass, and the remaining skips are expected OCI capability or region-availability gaps rather than code failures. |
|
Validation update from
The remaining Live models exercised in the passing runs:
The OCI test file was also trimmed to supported scenarios only, so the live runs no longer depend on permanently skipped coverage for unsupported on-demand generation/rerank or region-specific model availability. |
| "embeddings": embeddings, | ||
| "texts": [], # OCI doesn't return texts | ||
| "meta": meta, | ||
| } |
There was a problem hiding this comment.
Missing response_type discriminant in embed response
Medium Severity
The embed response dict is missing the response_type field required by the EmbedResponse discriminated union. EmbedResponse is an Annotated union with UnionMetadata(discriminant="response_type"), discriminating between EmbeddingsFloatsEmbedResponse (expects "embeddings_floats") and EmbeddingsByTypeEmbedResponse (expects "embeddings_by_type"). Without it, discriminated union resolution in _convert_union_type fails and falls through to undiscriminated matching, which is fragile and could break if the SDK's internal resolution logic changes.
| transformed_content.append(item) | ||
| oci_msg["content"] = transformed_content | ||
| else: | ||
| oci_msg["content"] = msg.get("content", []) |
There was a problem hiding this comment.
Content None not defaulting to empty array
Low Severity
When a V2 chat message has content explicitly set to None (e.g., an assistant message with only tool calls), msg.get("content", []) returns None rather than [] because dict.get only uses the default when the key is absent, not when its value is None. This sends "content": null to OCI instead of an empty array, which may be rejected by the OCI API.
2 participants
Overview
I noticed that the Cohere Python SDK has excellent integration with AWS Bedrock through the
BedrockClientimplementation. I wanted to contribute a similar integration for Oracle Cloud Infrastructure (OCI) Generative AI service to provide our customers with the same seamless experience.Motivation
Oracle Cloud Infrastructure offers Cohere's models through our Generative AI service, and many of our enterprise customers use both platforms. This integration follows the same architectural pattern as the existing Bedrock client, ensuring consistency and maintainability.
Implementation
This PR adds comprehensive OCI support with:
Features
~/.oci/config)Architecture
Testing
Documentation
Files Changed
src/cohere/oci_client.py(910 lines) - Main OCI client implementationsrc/cohere/manually_maintained/lazy_oci_deps.py(30 lines) - Lazy OCI SDK loadingtests/test_oci_client.py(393 lines) - Comprehensive integration testsREADME.md- OCI usage documentationpyproject.toml- Optional OCI dependencysrc/cohere/__init__.py- Export OciClient and OciClientV2Test Results
Skipped tests are for OCI service limitations (base models not callable via on-demand inference).
Breaking Changes
None. This is a purely additive feature.
Checklist
Note
Medium Risk
Adds a sizable new OCI transport layer with custom request signing and streaming/event transformations, which is moderately risky due to the complexity of protocol mapping and auth edge cases. Changes are largely additive/optional and should not affect existing non-OCI clients unless imported/used.
Overview
Adds optional Oracle Cloud Infrastructure (OCI) Generative AI support via new
OciClient(v1) andOciClientV2(v2), exposed fromcohere.__init__.Implements OCI request signing and endpoint mapping using
httpxevent hooks, including request/response body translation forembedandchatplus streaming translation to Cohere’s V1/V2 stream event formats (including V2 thinking blocks and usage extraction).Introduces an optional
ocidependency (cohere[oci]) with lazy importing, factors out a sharedStreamerutility used by both AWS and OCI clients, and adds OCI-focused tests plus README documentation covering install, auth methods, supported APIs, and OCI limitations.Written by Cursor Bugbot for commit 14b5c6e. This will update automatically on new commits. Configure here.