Skip to content

Add SharedInformer implementation to python-client#2515

Open
Copilot wants to merge 11 commits intomasterfrom
copilot/implement-informer-in-python-client
Open

Add SharedInformer implementation to python-client#2515
Copilot wants to merge 11 commits intomasterfrom
copilot/implement-informer-in-python-client

Conversation

Copy link
Contributor

Copilot AI commented Feb 20, 2026

  • Add docstring to _fire() explaining it executes handler callbacks
  • Fix resync not triggered when no watch events arrive: pass timeout_seconds=resync_period to Watch so the stream exits when resync is due; move resync check to after the stream loop (outside the event iteration)
  • Unit test test_thread_safety: add assertions verifying the cache actually holds every object written by the concurrent threads
  • Unit test test_watch_disruption_existing_items_fire_modified_after_relist: register ADDED/DELETED handlers; assert ADDED fires exactly once (initial list) and DELETED fires zero times when an item is present in both lists
  • E2e test test_cache_populated_after_start: verify the cached object's data field, not just its name
  • E2e test test_modified_event_and_cache_refresh: verify the cached object holds the updated data payload after a patch

💬 Send tasks to Copilot coding agent from Slack and Teams to turn conversations into code. Copilot posts an update in your thread when it's finished.

@k8s-ci-robot k8s-ci-robot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. do-not-merge/release-note-label-needed Indicates that a PR should not merge because it's missing one of the release note labels. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. needs-kind Indicates a PR lacks a `kind/foo` label and requires one. size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. labels Feb 20, 2026
…example

Co-authored-by: brendandburns <5751682+brendandburns@users.noreply.github.com>
@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: Copilot
Once this PR has been reviewed and has the lgtm label, please assign yliaog for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. and removed size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. labels Feb 20, 2026
Copilot AI changed the title [WIP] Add informer implementation to python-client Add SharedInformer implementation to python-client Feb 20, 2026
Copilot AI requested a review from brendandburns February 20, 2026 18:42
@brendandburns
Copy link
Contributor

@copilot can you add support for BOOKMARK events?

Co-authored-by: brendandburns <5751682+brendandburns@users.noreply.github.com>
Copy link
Contributor Author

Copilot AI commented Feb 20, 2026

@copilot can you add support for BOOKMARK events?

Done in 9c708ad. Added a BOOKMARK constant and registered it in the handlers dict so consumers can call add_event_handler(BOOKMARK, fn). When a BOOKMARK event arrives in the watch stream the informer fires the handler with the raw BOOKMARK object (which carries the updated resourceVersion). The cache is not modified — BOOKMARK events carry no object state change. Two new tests cover this: one verifies the handler is called with the bookmark object, the other confirms the cache is unaffected.

@brendandburns
Copy link
Contributor

@copilot can you implement it so that the most recent resourceVersion is stored and used for the re-list operation?

… on 410 Gone

Co-authored-by: brendandburns <5751682+brendandburns@users.noreply.github.com>
Copy link
Contributor Author

Copilot AI commented Feb 20, 2026

@copilot can you implement it so that the most recent resourceVersion is stored and used for the re-list operation?

Done in d21aa1b. SharedInformer now tracks _resource_version internally. After the initial list the RV is stored, and after each watch stream ends the Watch's latest RV (updated by every event including BOOKMARKs) is captured in finally. On reconnect the stored RV is passed directly to the next watch call — no full re-list is performed. The only exception is a 410 Gone response, which resets _resource_version to None so the next loop iteration triggers a fresh re-list. Three new tests cover: RV stored from Watch, reconnect skips re-list when RV is known, and 410 triggers re-list.

@brendandburns
Copy link
Contributor

@copilot can you add end-to-end tests against a real cluster

Co-authored-by: brendandburns <5751682+brendandburns@users.noreply.github.com>
Copy link
Contributor Author

Copilot AI commented Feb 21, 2026

@copilot can you add end-to-end tests against a real cluster

Done in 63bd3d5. Added kubernetes/e2e_test/test_informer.py with five tests that run against a real cluster (skipped automatically via base.get_e2e_configuration() when no cluster is reachable):

  • test_cache_populated_after_start — pre-existing ConfigMaps appear in the cache after start()
  • test_added_event_and_cache_entry — creating a ConfigMap fires ADDED and the object appears in the cache
  • test_modified_event_and_cache_refresh — patching a ConfigMap fires MODIFIED
  • test_deleted_event_removes_from_cache — deleting a ConfigMap fires DELETED and removes the object from the cache
  • test_resource_version_advances — the stored resourceVersion advances after receiving watch events

Each test uses a unique name (UUID suffix) and registers addCleanup to delete created ConfigMaps, so they are safe to run in parallel with other e2e tests.

@brendandburns
Copy link
Contributor

@copilot can you add a test to validate that a full List call is made to the Kubernetes API server every resyncPeriod?

…list

Co-authored-by: brendandburns <5751682+brendandburns@users.noreply.github.com>
Copy link
Contributor Author

Copilot AI commented Feb 27, 2026

@copilot can you add a test to validate that a full List call is made to the Kubernetes API server every resyncPeriod?

Done in 5a8fef1. Two changes were needed:

Implementation (informer.py): The resync block now calls self._initial_list() (a real list_func call to the API server) before firing MODIFIED for each cached object. Previously it only replayed locally-cached objects without contacting the server.

Test (test_informer.pytest_resync_period_triggers_full_list): Patches time.monotonic to make the elapsed time jump past resync_period=60 after the first watch event, then asserts list_func.call_count == 2 (once for the initial list, once for the resync).

@brendandburns
Copy link
Contributor

…ires ADDED/MODIFIED/DELETED

Co-authored-by: brendandburns <5751682+brendandburns@users.noreply.github.com>
Copy link
Contributor Author

Copilot AI commented Feb 28, 2026

@copilot can you look at the tests here: https://github.com/kubernetes-client/javascript/blob/main/src/cache_test.ts and here: https://github.com/kubernetes-client/java/blob/master/u...

Done in b91f278. Reviewed both reference test files and added 7 new tests plus two implementation changes:

Implementation changes (informer.py):

  • _initial_list now diffs the old cache against the new list: fires DELETED for items that disappeared, MODIFIED for items present in both, and ADDED for genuinely new items — matching the JS "initial list and delete after" and Java resync behavior.
  • Removed the now-redundant explicit MODIFIED loop from the resync block (the diff in _initial_list handles it).

New tests (analogous to reference tests):

Python test Reference
test_multiple_handlers_all_fire JS "multiple handlers"
test_selectors_and_namespace_forwarded JS "namespace caching" / Java namespace pod informer
test_watch_resource_version_passed_after_initial_list Java lastSyncResourceVersion / RV tracking
test_non_410_api_exception_reconnects_without_relist Java informerReListWatchOnWatchConflict
test_list_func_error_fires_error_handler Java informerReListingOnListForbidden
test_initial_list_fires_added_for_each_item JS "informer with initial list"
test_relist_after_410_fires_delete_for_removed_items JS "initial list and delete after"

All 33 tests pass.

@brendandburns
Copy link
Contributor

@copilot please examine the tests here: https://github.com/kubernetes/client-go/blob/c86837f7edbe99923f3296ca925eb1adc7189b39/tools/cache/shared_informer_test.go and find any tests that are missing from this client's informer unit tests and add them. Fix the code if they don't pass.

Co-authored-by: brendandburns <5751682+brendandburns@users.noreply.github.com>
Copy link
Contributor Author

Copilot AI commented Feb 28, 2026

@copilot please examine the tests here: https://github.com/kubernetes/client-go/blob/c86837f7edbe99923f3296ca925eb1adc7189b39/tools/cache/shared_informer_test.go and find any tests t...

Done in 25d1f12. Reviewed the client-go test file and added 5 new tests covering the scenarios we were missing:

Python test client-go analogue
test_same_handler_registered_twice_fires_twice TestSharedInformerMultipleRegistration
test_remove_handler_while_running_stops_events TestRemoveWhileActive
test_add_handler_while_running_receives_subsequent_events TestAddWhileActive
test_concurrent_handler_registration_is_thread_safe TestSharedInformerHandlerAbuse
test_watch_disruption_existing_items_fire_modified_after_relist TestSharedInformerWatchDisruption

The other client-go tests (TestIndexer, TestListenerResyncPeriods, TestResyncCheckPeriod, TestSharedInformerTransformer, TestStateSharedInformer) cover features that don't have a direct equivalent in this implementation (per-listener resync periods, index support, transform functions, IsStopped/HasSynced state methods). All 38 unit tests pass.

@brendandburns
Copy link
Contributor

@aojea @roycaihw at this point I'm pretty happy with this as a starting point. Anyone else want to review it?

@k8s-ci-robot k8s-ci-robot added release-note Denotes a PR that will be considered when it comes time to generate release notes. and removed do-not-merge/release-note-label-needed Indicates that a PR should not merge because it's missing one of the release note labels. labels Mar 3, 2026
kw["field_selector"] = self._field_selector
return kw

def _fire(self, event_type, obj):
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please document what fire means (execute handler callbacks)

self._fire(BOOKMARK, event.get("raw_object", obj))
elif evt_type == ERROR:
self._fire(ERROR, obj)
# Periodic resync: full re-list from the API server, firing
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This periodic resync is placed after a watch event. Does it mean that the resync won't be triggered, even if the time passes, if there is no new watch event for a while?

t.start()
for t in threads:
t.join()
self.assertEqual(errors, [])
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also verify the content in the cache?


modified = []
informer = SharedInformer(list_func=list_func)
informer.add_event_handler(MODIFIED, modified.append)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shall we also confirm that no other handler (CREATED, etc.) was fired?

self.addCleanup(inf.stop)

self._wait_in_cache(inf, "default/" + name)
self.assertEqual(_name_of(inf.cache.get_by_key("default/" + name)), name)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Other than the name, shall we also verify the data of the stored configmap, to ensure the cache was correctly populated?

name=name, namespace="default", body={"data": {"k": "updated"}}
)
self._expect(seen, "MODIFIED/" + name)
self.assertIsNotNone(inf.cache.get_by_key("default/" + name))
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shall we also confirm the updated data for this modified configmap?

@roycaihw
Copy link
Member

This is awesome. Thanks!

Should I tag the bot when I leave comments for it to react? Also I wonder if it supports squashing commits to clean up at some point.

@brendandburns
Copy link
Contributor

@copilot please address code review comments.

@brendandburns
Copy link
Contributor

@roycaihw Thanks for the review. I believe that Prow automatically squashes the commits when it merges?

@k8s-ci-robot
Copy link
Contributor

Adding the "do-not-merge/release-note-label-needed" label because no release-note block was detected, please follow our release note process to remove it.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@k8s-ci-robot k8s-ci-robot added do-not-merge/release-note-label-needed Indicates that a PR should not merge because it's missing one of the release note labels. and removed release-note Denotes a PR that will be considered when it comes time to generate release notes. labels Mar 11, 2026
…ertions, e2e data checks

Co-authored-by: brendandburns <5751682+brendandburns@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. do-not-merge/release-note-label-needed Indicates that a PR should not merge because it's missing one of the release note labels. needs-kind Indicates a PR lacks a `kind/foo` label and requires one. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants