Only send finalState if needed

[taylordowns2000]

Short Description

When a workflow ends with a single leaf step whose dataclip is already persisted, send final_dataclip_id instead of final_state in run:complete to avoid re-uploading state Lightning already has.

Closes #1309 and best served after OpenFn/lightning#4531 is merged.

Implementation Details

• Replaced lastDataclipId with leafDataclipIds[] on RunState, populated in step-complete only for steps with no downstream edges.
• In run-complete, if there's exactly one leaf dataclip that wasn't withheld, send final_dataclip_id; otherwise fall back to sending final_state as before.
• Updated lexicon types, lightning-mock, and tests accordingly.

QA Notes

Branching workflows (multiple leaves) should still send final_state.

• Single-leaf workflows where the dataclip was withheld should also still send final_state.
• Single-leaf, non-withheld workflows should send only final_dataclip_id.

AI Usage

Please disclose whether you've used AI anywhere in this PR (it's cool, we just
want to know!):

• I have used Claude Code
• I have used another model
• I have not used AI

You can read more details in our
Responsible AI Policy

Release branch checklist

Delete this section if this is not a release PR.

If this IS a release branch:

• Run pnpm changeset version from root to bump versions
• Run pnpm install
• Commit the new version numbers
• Run pnpm changeset tag to generate tags
• Push tags git push --tags

Tags may need updating if commits come in after the tags are first generated.

[josephjclark]

@taylordowns2000 I'm going to put out a worker patch today but I'm going to exclude this as we're waiting on the Lightning PR. I'll get this reviewed properly with a clear head tomorrow.

[taylordowns2000]

that's perfect @josephjclark , thank you! if you do get the chance, a review before Thursday morning (730am Paris time) would be great as it would allow @stuartc and I to review your review when we talk about OpenFn/lightning#4531

[josephjclark]

Looks good. Works against lightning main without any complaints or errors, so it's safe to merge and deploy.

I read this as an optimisation @taylordowns2000, nothing more. Does that seem fair, or am I missing something?

[taylordowns2000]

That's correct @josephjclark , but there's a bit of fancy footwork... basically it has to determine (based on its knowledge of the graph) whether or not the finalState for the run is/will be different than the final state for a single step. If you're happy with where that responsibility lies, we're good to go.

(And the lightning change will now track the final state of a run, either simply setting an FK on run or by creating a new dataclip and then setting the FK on run.)

Have added Stu here and we can click merge together tomorrow AM assuming we're happy with the Lightning side.

[josephjclark]

it has to determine (based on its knowledge of the graph) whether or not the finalState for the run is/will be different than the final state for a single step. If you're happy with where that responsibility lies, we're good to go.

yeah this is fine. I think there's an argument that we need to do some Deep Thinking on output state and how we handle this multiple leaf stuff. We're building on a best-guess from 3 (?) years ago, and maybe there are better ways to handle this.

But I'm happy to keep building for now and the solution looks good to me.

You can merge when you like but remember:

• versions need to be bumped before merging (I won't bump versions now because I may be putting out a release today myself)
• tags still need pushing manually because for some reason they don't trigger the action

If you like you can just give me a greenlight when you're ready and I'll handle the release for you