store: Consolidate index creation using CreateIndex and add postponed index support#6434
Open
store: Consolidate index creation using CreateIndex and add postponed index support#6434
Conversation
Move some of the responsibility to the caller to make it clearer what is being used
…able All callers used the name of the table that was passed in, except for the use in `copy.rs`, but copying doesn't change table names, so the src and dst names of tables are the same if they exist in the dst
Also, test some variations of the same index definition
Consolidate index creation into a single `Table::indexes()` method that returns all indexes (time-travel, attribute, aggregate) as `CreateIndex` objects. This replaces the old string-based methods and eliminates the `index_def: Option<IndexList>` parameter threading through the codebase. Key changes: - Add `Table::indexes()` combining time_travel + attribute + aggregate indexes - Add `attr_index_spec()` and `add_attribute_indexes()` structured helpers - Move env var check into `CreateIndex::to_postpone()` so callers need not check - Simplify `Table::as_ddl()` to iterate indexes with postpone filtering - Remove old `create_time_travel_indexes`, `create_attribute_indexes`, `create_postponed_indexes`, `create_aggregate_indexes` string methods - Remove `index_def` parameter from Layout, DeploymentStore, SubgraphStore - Update copy.rs to use `indexes()` + `references_column_not_in()` for new fields - Update prune.rs to use simplified `as_ddl()` without index_def - Update all DDL test constants for new single-line index format
Add a trigger that creates postponed indexes when a subgraph gets within a configurable number of blocks (default 10000) of the chain head. This ensures indexes are in place before the subgraph starts serving queries. The new env var GRAPH_POSTPONE_INDEXES_CREATION_THRESHOLD controls how many blocks before the chain head to trigger index creation. The creation is idempotent (IF NOT EXISTS + CONCURRENTLY) and only attempted once per subgraph run via an AtomicBool guard.
Replace the IndexList-based `recreate_invalid_indexes` call in `start_subgraph()` with a call to `create_postponed_indexes()`. This uses `IF NOT EXISTS` and `CONCURRENTLY` to safely create any missing postponed indexes on every restart, acting as a safety net. Remove the now-unused `IndexList::recreate_invalid_indexes` method.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Currently,
GRAPH_POSTPONE_ATTRIBUTE_INDEX_CREATIONwhen copying subgraphs; but subgraphs whose performance is limited by the speed at which we can write to the database, such as amp subgraphs, will also benefit from deferring index creation.Besides postponing index creation for syncing subgraphs, this PR also refactors how indexes are created for subgraph tables, replacing scattered raw SQL generation with a unified
CreateIndexabstraction.This PR will also be the basis for speeding up
graphman restoreby having it defer index creation until after the data import.Index creation consolidation
Table::indexes()returning all indexes (time-travel, attribute, aggregate) as structuredCreateIndexobjects instead of raw SQL stringsminmax_multi_opsoperator classes and variousWHEREclausesindex_def: Option<IndexList>parameter threading and simplify callers acrosscopy.rs,prune.rs, anddeployment_store.rscreate_indexexample tool for testing index definition parsingPostponed index creation
CreateIndex::to_postpone(), controlled by theGRAPH_POSTPONE_INDEXESenv varGRAPH_POSTPONE_INDEXES_CREATION_THRESHOLD, default 10000)IF NOT EXISTS+CONCURRENTLYTest plan
just test-unit)