Add Impersonate Service Account argument by wintermi · Pull Request #2015 · dataform-co/dataform

wintermi · 2025-09-11T06:43:03Z

This PR adds an --impersonate-service-account argument to the run and test commands, along with the required changes to allow for the impersonation of service accounts without the need to change ADC or call gcloud

This would resolve issue #2000 and would be an alternative to solution than PR #2001

Impersonation could then be achieved by executing:

dataform run --impersonate-service-account=<sSERVICE_ACCT_EMAIL>

…st' commands and the required changes to allow for the impersonation of service accounts without the need to change ADC

google-cla · 2025-09-11T06:43:08Z

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

camilleAmaury · 2025-10-30T10:44:42Z

+1, this would enable to use impersonation in CI rather than giving the rights directly to the CI service account.
There is no way to workaround that currently.

kolina · 2025-11-11T11:28:51Z

/gcbrun

kolina · 2025-11-13T23:33:49Z

Sorry for the late review. A couple of things:

Integration tests are failing, can you take a look at fixing them? Now we have a guide of running them locally
Let's resolve conflicts

cli/api/dbadapters/bigquery.ts

kolina · 2025-11-13T23:17:49Z

cli/api/dbadapters/bigquery.ts

+        clientConfig.authClient = new Impersonated({
+          sourceClient: authClient,
+          targetPrincipal: this.bigQueryCredentials.impersonateServiceAccount,
+          targetScopes: ['https://www.googleapis.com/auth/cloud-platform']


EXTRA_GOOGLE_SCOPES?

Not sure what you would like done here?

I mean using EXTRA_GOOGLE_SCOPES here instead of hard-coding

kolina · 2026-01-02T21:13:06Z

@wintermi, in the current version tests are failing due to linter checks: output

You can check errors using this lint script.

Resolved conflicts: - cli/index.ts: Kept both impersonate-service-account option (fork feature) and job-labels option (upstream feature) - package.json: Updated glob to ^10.5.0, kept google-auth-library dependency - yarn.lock: Regenerated with updated dependencies

wintermi · 2026-01-05T13:31:51Z

Resynced the PR with the latest commit

wintermi · 2026-01-05T14:36:31Z

@wintermi, in the current version tests are failing due to linter checks: output

You can check errors using this lint script.

Fixed the linter issues

wintermi · 2026-01-05T14:37:06Z

@kolina ready for retesting, thanks

kolina · 2026-01-07T12:34:01Z

/gcbrun

kolina · 2026-01-07T12:45:04Z

cli/api/dbadapters/bigquery.ts

+        clientConfig.authClient = new Impersonated({
+          sourceClient: authClient,
+          targetPrincipal: this.bigQueryCredentials.impersonateServiceAccount,
+          targetScopes: ['https://www.googleapis.com/auth/cloud-platform']


I mean using EXTRA_GOOGLE_SCOPES here instead of hard-coding

kolina · 2026-01-07T12:50:20Z

cli/api/dbadapters/bigquery.ts

  }

-  private getClient(projectId?: string) {
+  private async getClient(projectId?: string) {


In #2001 @ashish10alex added support for impersonating a service account through ADC. Will it be enough for your use case or you need an explicit option as well?

If the latter, can you please validate manually that it works?

…ataform-co#2053) (dataform-co#2057)

…2062)

* add custom_attributes struct to metadata & incremental table tests * fix incremental table tests & add view & table test * change custom_attributes to extra_properties

* Proto changes for unit tests definition * Added property that allows disabling unit tests

…empty (dataform-co#2082) * Added logic to fail test compilation if test SQL or expected SQL are empty * Addressed PR comments * Fix pr comments

Bumps [vm2](https://github.com/patriksimek/vm2) from 3.9.19 to 3.10.2. - [Release notes](https://github.com/patriksimek/vm2/releases) - [Commits](patriksimek/vm2@3.9.19...v3.10.2) --- updated-dependencies: - dependency-name: vm2 dependency-version: 3.10.2 dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps [diff](https://github.com/kpdecker/jsdiff) from 4.0.2 to 8.0.3. - [Changelog](https://github.com/kpdecker/jsdiff/blob/master/release-notes.md) - [Commits](kpdecker/jsdiff@v4.0.2...v8.0.3) --- updated-dependencies: - dependency-name: diff dependency-version: 8.0.3 dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

JS API for JiT compilation actions definition (mutually exclusive with AoT properties) and JiT context/result typings.

Implementation of context objects accessible at JiT compilation stage for supported action types. They expose access to adapter/data and context methods, equivalent to AoT context, but with a more restrictive selection of resolvable targets.

Add file_paths field to pass accessible file paths at JiT compilation stage.

Bumps [lodash](https://github.com/lodash/lodash) from 4.17.21 to 4.17.23. - [Release notes](https://github.com/lodash/lodash/releases) - [Commits](lodash/lodash@4.17.21...4.17.23) --- updated-dependencies: - dependency-name: lodash dependency-version: 4.17.23 dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps [diff](https://github.com/kpdecker/jsdiff) from 3.5.0 to 8.0.3. - [Changelog](https://github.com/kpdecker/jsdiff/blob/master/release-notes.md) - [Commits](kpdecker/jsdiff@v3.5.0...v8.0.3) --- updated-dependencies: - dependency-name: diff dependency-version: 8.0.3 dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* JiT Contexts - Add original request & simplify constructor Simplify constructor by explicitly passing and exposing the orignal compilation request (which already contains most of required data). * JiT compilation stage implementation Implement JiT compilation stage in Core: - a new entry point jitCompiler(rpcCallback), which instantiates a single compile method, performing a compilation against a supplied RPC implementation. - table/operations/incremental table context(s) construction and invocation of the JiT code.

* Support both 'yaml' and 'yml' file extensions We can support both formats in Dataform. Native configurations are stored in YAML format, but other configurations eg. from extensions might use YML too. * Support both 'yaml' and 'yml' file extensions We can support both formats in Dataform. Native configurations are stored in YAML format, but other configurations eg. from extensions might use YML too. * Support both 'yaml' and 'yml' file extensions We can support both formats in Dataform. Native configurations are stored in YAML format, but other configurations eg. from extensions might use YML too. * Reorganize imports in compilers_test.ts

dataform-co#2095) Some extensions might use different configuration files, this will allow them to proceed when workflow_settings.yaml is missing.

Bumps [webpack](https://github.com/webpack/webpack) from 5.94.0 to 5.104.1. - [Release notes](https://github.com/webpack/webpack/releases) - [Changelog](https://github.com/webpack/webpack/blob/main/CHANGELOG.md) - [Commits](webpack/webpack@v5.94.0...v5.104.1) --- updated-dependencies: - dependency-name: webpack dependency-version: 5.104.1 dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps [rollup](https://github.com/rollup/rollup) from 2.79.2 to 2.80.0. - [Release notes](https://github.com/rollup/rollup/releases) - [Changelog](https://github.com/rollup/rollup/blob/v2.80.0/CHANGELOG.md) - [Commits](rollup/rollup@v2.79.2...v2.80.0) --- updated-dependencies: - dependency-name: rollup dependency-version: 2.80.0 dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps [underscore](https://github.com/jashkenas/underscore) from 1.13.1 to 1.13.8. - [Commits](jashkenas/underscore@1.13.1...1.13.8) --- updated-dependencies: - dependency-name: underscore dependency-version: 1.13.8 dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

…taform Core and identify bottlenecks (dataform-co#2100) * feat: add --verbose argument to expose steps in stateless installation * chore: make verbose and quiet flags mutually exclusive * feat: use performance timing in node to avoid system clock issues * chore: do not export print * refactor: remove redundant double check * chore: print should pipe to stderr

* feat: Add BigQuery reservation configuration for projects and individual actions. * lint Signed-off-by: Max Ostapenko <1611259+max-ostapenko@users.noreply.github.com> * refactor: rename `bigqueryReservation` to `reservation` across configurations, protos, and related code Signed-off-by: Max Ostapenko <1611259+max-ostapenko@users.noreply.github.com> * docs: clarify BigQuery reservation fallback behavior in protos and CLI Signed-off-by: Max Ostapenko <1611259+max-ostapenko@users.noreply.github.com> * refactor: update default reservation test constant and expand project config assertions Signed-off-by: Max Ostapenko <1611259+max-ostapenko@users.noreply.github.com> * fix test reservation name Signed-off-by: Max Ostapenko <1611259+max-ostapenko@users.noreply.github.com> * added documentation for BigQuery reservation configuration Signed-off-by: Max Ostapenko <1611259+max-ostapenko@users.noreply.github.com> * fix project name Signed-off-by: Max Ostapenko <1611259+max-ostapenko@users.noreply.github.com> * fix reservation location Signed-off-by: Max Ostapenko <1611259+max-ostapenko@users.noreply.github.com> * Add reservation to a profile schema Signed-off-by: Max Ostapenko <1611259+max-ostapenko@users.noreply.github.com> * revert reservation attribute Signed-off-by: Max Ostapenko <1611259+max-ostapenko@users.noreply.github.com> * Note support status --------- Signed-off-by: Max Ostapenko <1611259+max-ostapenko@users.noreply.github.com>

…form-co#2105)

…o#2106) * refactor: split index_test.ts into specialized test files * refactor: decompose monolithic cli/index_test.ts into modular components - Extract shared test utilities into cli/index_test_base.ts - Create dedicated test files for help, init, project, compile, and run commands - Update cli/BUILD to reflect the new modular test structure

Instead of exposing a proto struct, returning (mostly) the same object that got passed in the original key at definition.

…orm-co#2112)

- Enhance error coercion in common/errors/errors.ts - Sync protos/execution.proto and protos/jit.proto with new fields

…st' commands and the required changes to allow for the impersonation of service accounts without the need to change ADC

Resolve conflicts in bigquery adapter imports and getClient method (kept double-quote style and correct indentation), and remove duplicate impersonateServiceAccountOption definition in CLI. Made-with: Cursor

Add the '--impersonate-service-account' argument to the 'run' and 'te…

c3094eb

…st' commands and the required changes to allow for the impersonation of service accounts without the need to change ADC

wintermi requested a review from a team as a code owner September 11, 2025 06:43

wintermi requested review from Ceridan and removed request for a team September 11, 2025 06:43

Ceridan requested review from kolina and removed request for Ceridan November 13, 2025 09:15

kolina requested changes Nov 13, 2025

View reviewed changes

ashish10alex mentioned this pull request Dec 23, 2025

[Dataform CLI] Error when using service account impersonation through ADC #2056

Open

wintermi added 3 commits January 6, 2026 00:41

Make changes as requested

c65768d

Make changes as requested

85a0440

Make changes as requested

a5b4288

wintermi requested a review from kolina January 7, 2026 05:35

kolina reviewed Jan 7, 2026

View reviewed changes

sibyabin and others added 7 commits January 7, 2026 14:59

Fix incremental table recreation when defaultProject not set (issue d…

18a3f7c

…ataform-co#2053) (dataform-co#2057)

bump Dataform version (dataform-co#2061)

7917e9a

Use latest version of secret instead of a specific one. (dataform-co#…

85c09b3

…2062)

Add extra_properties struct to metadata (dataform-co#2055)

8bf72c0

* add custom_attributes struct to metadata & incremental table tests * fix incremental table tests & add view & table test * change custom_attributes to extra_properties

bump dataform version to 3.0.42 (dataform-co#2064)

359e1a2

Proto changes for unit tests definition (dataform-co#2065)

5044e87

* Proto changes for unit tests definition * Added property that allows disabling unit tests

Bump to version 3.0.43 (dataform-co#2066)

e64b7b0

fernst and others added 30 commits February 6, 2026 13:54

Added logic to fail test compilation if test SQL or expected SQL are …

6559fc2

…empty (dataform-co#2082) * Added logic to fail test compilation if test SQL or expected SQL are empty * Addressed PR comments * Fix pr comments

JiT compilation request format (dataform-co#2085)

e4d8a78

JiT compilation support in JS API (dataform-co#2086)

4246ebb

JS API for JiT compilation actions definition (mutually exclusive with AoT properties) and JiT context/result typings.

Release Dataform 3.0.46 (dataform-co#2091)

cc92f4a

JiT contexts (dataform-co#2088)

2918c14

Implementation of context objects accessible at JiT compilation stage for supported action types. They expose access to adapter/data and context methods, equivalent to AoT context, but with a more restrictive selection of resolvable targets.

JiT Compilation protos - Add file_paths (dataform-co#2093)

8724d55

Add file_paths field to pass accessible file paths at JiT compilation stage.

Release Dataform 3.0.47 (dataform-co#2094)

5334037

Allow extensions to populate project config without workflow_settings. (

ebbce70

dataform-co#2095) Some extensions might use different configuration files, this will allow them to proceed when workflow_settings.yaml is missing.

Update the npm package version (dataform-co#2098)

0a8902e

chore: update scripts/run_tests to support additional arguments (data…

94db369

…form-co#2105)

JiT context - decode data (dataform-co#2111)

df8cb18

Instead of exposing a proto struct, returning (mostly) the same object that got passed in the original key at definition.

fix: update internal import linter rule and reorganize imports (dataf…

bdcb1e4

…orm-co#2112)

JiT: protos update to enable execution in CLI (dataform-co#2107)

412d523

- Enhance error coercion in common/errors/errors.ts - Sync protos/execution.proto and protos/jit.proto with new fields

Add the '--impersonate-service-account' argument to the 'run' and 'te…

f98b016

…st' commands and the required changes to allow for the impersonation of service accounts without the need to change ADC

Make changes as requested

030bdb6

Make changes as requested

a910b37

Make changes as requested

0d48014

Merge remote-tracking branch 'origin/main'

763681d

Resolve conflicts in bigquery adapter imports and getClient method (kept double-quote style and correct indentation), and remove duplicate impersonateServiceAccountOption definition in CLI. Made-with: Cursor

Conversation

wintermi commented Sep 11, 2025

Uh oh!

google-cla bot commented Sep 11, 2025

Uh oh!

camilleAmaury commented Oct 30, 2025

Uh oh!

kolina commented Nov 11, 2025

Uh oh!

kolina commented Nov 13, 2025

Uh oh!

Uh oh!

Uh oh!

kolina Nov 13, 2025

Choose a reason for hiding this comment

Uh oh!

wintermi Jan 5, 2026

Choose a reason for hiding this comment

Uh oh!

kolina Jan 7, 2026

Choose a reason for hiding this comment

Uh oh!

kolina commented Jan 2, 2026

Uh oh!

wintermi commented Jan 5, 2026

Uh oh!

wintermi commented Jan 5, 2026

Uh oh!

wintermi commented Jan 5, 2026

Uh oh!

kolina commented Jan 7, 2026

Uh oh!

kolina Jan 7, 2026

Choose a reason for hiding this comment

Uh oh!

kolina Jan 7, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

14 participants