(Improvement) faster metadata schema parsing#745
Draft
mykaul wants to merge 5 commits intoscylladb:masterfrom
Draft
(Improvement) faster metadata schema parsing#745mykaul wants to merge 5 commits intoscylladb:masterfrom
mykaul wants to merge 5 commits intoscylladb:masterfrom
Conversation
mykaul
changed the title
There was a problem hiding this comment.
Pull request overview
This PR aims to reduce CPU time and memory usage during schema metadata refresh by avoiding per-row dict allocations and trimming schema column queries, while also shrinking metadata object overhead.
Changes:
- Introduces an internal
_RowView+_row_factoryand routes schema query result handling through it to reduce per-row allocations. - Adds
__slots__to several metadata model classes and replaces someOrderedDictusages with plaindictto reduce memory overhead. - Narrows the
system_schema.columnsquery inSchemaParserV3to only the fields needed by the parser and refactors_build_table_columnsto classify rows in a single pass.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
This was referenced Mar 14, 2026
Add __slots__ to KeyspaceMetadata, UserType, Aggregate, Function, TableMetadata, TableMetadataV3, TableMetadataDSE68, ColumnMetadata, IndexMetadata, TriggerMetadata, and MaterializedViewMetadata. This reduces per-instance memory overhead by eliminating __dict__ on these frequently-instantiated objects during schema refresh. All attributes previously set as class-level defaults are now initialized in __init__ to satisfy the slots contract.
Python 3.7+ guarantees dict preserves insertion order, making OrderedDict
unnecessary. Replace OrderedDict() with {} in TableMetadata.columns,
TableMetadata.triggers, and MaterializedViewMetadata.columns. Remove the
now-unused OrderedDict import.
….columns Replace SELECT * with an explicit column list for the system_schema.columns query in SchemaParserV3 (inherited by V4). Only the 7 columns actually consumed by the parser are fetched: keyspace_name, table_name, column_name, clustering_order, kind, position, type. This reduces network transfer and deserialization overhead during schema refresh.
Introduce _RowView, a __slots__-based read-only row wrapper that stores data as tuples with a shared column-name-to-index map, and _row_factory that creates these views. Replace dict_factory in _SchemaParser._handle_results and get_column_from_system_local (both reachable from the V4 code path). This eliminates per-row dict allocation during schema parsing. All rows from the same result set share a single index map object. Also refactor SchemaParserV4._build_keyspace_metadata_internal to read from the row without mutating it, since _RowView is read-only. Note: V22-only dict_factory call sites are left unchanged as they do not affect the V3/V4 code path (V3 and V4 fully override _query_all).
…able_columns Rewrite _build_table_columns to classify columns by kind in a single pass instead of iterating col_rows three times with list comprehensions. This also fixes a bug where the third pass filtered on 'clustering_key' instead of 'clustering', causing clustering columns to leak through and get re-processed as regular columns. Additionally, use in-place sort() instead of sorted() to avoid creating intermediate list copies, and append the already-built column_meta object to partition_key/clustering_key instead of re-looking it up from meta.columns by name. Combined benchmark results for the full optimization series (A-F): Row creation + access: 323 ns/row vs 485 ns/row (1.50x faster) _build_table_columns: 9.0 us/table vs 9.9 us/table (1.10x faster) Full pipeline (100 tables x 20 cols): 0.79 ms vs 1.57 ms (1.98x faster) Memory per row: 48 bytes vs 272 bytes (5.7x reduction) __slots__ per instance: 80 bytes (saves ~104 bytes __dict__ overhead)
mykaul
force-pushed
the
improvement/metadata-schema-parsing
branch
from
3a7f4bc to
4f9e4f5
Compare
There was a problem hiding this comment.
Pull request overview
This PR optimizes schema metadata refresh by reducing per-row allocations during schema parsing and narrowing the system_schema.columns select list to only the fields needed for building table/column metadata.
Changes:
- Introduces an internal lightweight row representation (
_RowView+_row_factory) and uses it in schema parser result handling to reduce time/memory overhead. - Reduces the
system_schema.columnsquery to fetch only required columns and refactors_build_table_columnsto classify rows in a single pass. - Adds unit tests for
_RowViewand_row_factorybehavior.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.
| File | Description |
|---|---|
| cassandra/metadata.py | Adds _RowView/_row_factory, switches schema parser row handling away from per-row dicts, tightens system_schema.columns query, and updates metadata classes/docstrings/slots. |
| tests/unit/test_metadata.py | Adds unit tests validating _RowView and _row_factory semantics (getitem/get/contains/read-only/shared index map). |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Reduce the time and memory required to parse schema metadata when refreshing. The biggest item is change select query of system_schema.columns, which is quite big - especially with multiple tables.
Overall perf. results:
Row creation + access: 323 ns/row vs 485 ns/row (1.50x faster)
_build_table_columns: 9.0 us/table vs 9.9 us/table (1.10x faster)
Full pipeline (100 tables x 20 cols): 0.79 ms vs 1.57 ms (1.98x faster) <--- that's not bad, finally in ms improvements.
Memory per row: 48 bytes vs 272 bytes (5.7x reduction)
slots per instance: 80 bytes (saves ~104 bytes dict overhead)
Pre-review checklist
./docs/source/.Fixes:annotations to PR description.