Our new language runtime uses a set of new methods on SyncState to work
with its preferred "full" representation of resource instance objects, but
those are implemented in terms of methods that already existed for the old
runtime's benefit and so we need to deal with some quirks of those existing
methods.
One such quirk is that the operations to write or remove objects also want
to update some resource-level and instance-level metadata as a side-effect,
and we need to carry through that metadata even when we're intending to
completely remove a resource instance object.
To preserve our goal of leaving the existing codepaths untouched for now,
this pushes a little complexity back up into the main caller in the apply
engine, forcing it to call a different method when it knows it has deleted
an object. That new method then only takes the metadata we need and not
an actual resource instance object, so it gels better with the underlying
ModuleState methods it's implemented in terms of.
Hopefully in the long run we'll rethink the state models to not rely on
these hidden side-effects, but that's beyond the scope of our current phase
of work on the new language runtime.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
Previously we had this check in the apply engine because it was making
sure that the decomposed parts of a "replace" don't themselves call for
a replace action, but it's a general rule that RequiresReplace is not
meaningful unless there's both a current and a desired object to compare,
so this moves that check into the "resources" package as a general rule.
A future commit will rely on this once the planning engine correctly
handles "replace" actions by asking the provider to re-plan as if the
old object didn't exist, in order to predict the "create" part of the
replace action.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
The original motivation here was to implement the special case of making
a synthetic plan without calling into the provider when the desired state
is absent and the provider hasn't opted in to planning to destroy objects.
This case needs to be opted in because older providers will crash if asked
to plan with the config value or proposed value set to null.
However, there was already far too much code duplicated across both the
planning engine and the apply engine's "final plan" operation, and so this
seemed like a good prompt to finally resolve those TODO comments and factor
out the shared planning logic into a separate place that both can depend
on.
The new package "resources" is intended to grow into the home for all of
these resource-related operations that each wrap a lower-level provider
call but also perform preparation or followup actions such as making sure
the provider's response is valid according to the requirements of the
plugin protocol.
For now it only includes config validation and planning for managed
resources, but is designed to grow to include other similar operations in
future. The most likely next step is to add a method for applying a
previously-created plan, which would replace a bunch of the code in the
apply engine's implementation of the "ManagedApply" operation.
Currently the way this integrates with the rest of the provider-related
infrastructure is a little awkward. We can hopefully improve on that in
future, but for now the priority was to avoid this becoming yet another
sprawling architecture change that would be hard to review. Once we feel
like we've achieved the "walking skeleton" milestone for the new runtime
we can think about how we want to tidy up the rough edges between the
different components of this new design.
As has become tradition for these new codepaths, this is designed to be
independently testable but not yet actually tested because we want to wait
and see how all of this code settles before we start writing tests that
would then make it harder to refactor as we learn more.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
These both effectively had the behavior of ResourceInstancePrior embedded
in them, reading something from the state and change its address as a
single compound operation.
In the case of ManagedDepose we need to split these up for the
CreateThenDestroy variant of "replace", because we want to make sure the
final plans are valid before we depose anything and we need the prior state
to produce the final plan. (Actually using that will follow in a subsequent
commit.)
This isn't actually necessary for ManageChangeAddr, but splitting it keeps
these two operations consistent in how they interact with the rest of the
operations.
Due to how the existing states.SyncState works we're not actually making
good use of the data flow of these objects right now, but in a future world
where we're no longer using the old state models hopefully the state API
will switch to an approach that's more aligned with how the execgraph
operations are modeled.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
Whenever we plan for a resource instance object to switch from one instance
address to another, we'll use this new op instead of ResourceInstancePrior
to perform the state modification needed for the rename before returning
the updated object.
We combine the rename and the state read into a single operation to ensure
that they can appear to happen atomically as far as the rest of the system
is concerned, so that there's no interim state where either both or neither
of the address bindings are present in the state.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
These were previously using "applying:" as the common prefix, but I found
that confusing in practice because only the "ManagedApply" operation is
_actually_ applying changes.
Instead then we'll identify these trace logs as belonging to the apply
phase as a whole, to try to be a little clearer about what's going on.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
Here we ask the provider to produce a plan based on the finalized
configuration taken from the "desired state", and return it as a final plan
object only if the provider request succeeds and returns something that is
valid under our usual plan consistency rules.
We then ask the provider to apply that plan, and deal with whatever it
returns. The apply part of this is not so complete yet, but there's enough
here to handle the happy path where the operation succeeds and the provider
returns something valid.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
This'll make it clearer which provider we're using to ask each question,
without making a human reader try to work backwards through the execution
graph data flow.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
There is currently a bug in the apply engine that's causing a
self-dependency error during promise resolution, which was a good reminder
that we'd not previously finished connecting all of the parts to be able
to unwind such problems into a useful error message.
Due to how the work is split between the apply engine, the evaluator, and
the execgraph package it takes some awkward back-and-forth to get all of
the needed information together into one place. This compromise aims to
do as little work as possible in the happy path and defer more expensive
analysis until we actually know we're going to report an error message.
In this case we can't really avoid proactively collecting the request IDs
because we don't know ahead of time what (if anything) will be involved in
a promise error, but only when actually generating an error message will
we dig into the original source execution graph to find out what each of
the affected requests was actually trying to do and construct a
human-friendly summary of each one.
This was a bit of a side-quest relative to the current goal of just getting
things basically working with a simple configuration, but this is useful
in figuring out what's going on with the current bug (which will be fixed
in a future commit) and will probably be useful when dealing with future
bugs too.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
During execution graph processing we need to carry addressing information
along with objects so that we can model changes of address as data flow
rather than as modifications to global mutable state.
In previous commits we arranged for other relevant types to track this
information, but the representation of a resource instance object was not
included because that was a messier change to wire in. This commit deals
with that mess: it introduces exec.ResourceInstanceObject to associate a
resource instance object with addressing information, and then adopts that
as the canonical representation of a "resource instance object result"
throughout all of the execution graph operations.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
The first pass of this was mainly just to illustrate how the overall model
of execution graphs might work, using just a few "obvious" opcodes as a
placeholder for the full set.
Now that we're making some progress towards getting this to actually work
as part of a real apply phase this reworks the set of opcodes in a few
different ways:
- Some concepts that were previously handled as special cases in the
execution graph executor are now just regular operations. The general
idea here is that anything that is fallible and/or has externally-visible
side-effects should be modelled as an operation.
- The set of supported operations for managed resources should now be
complete enough to describe all of the different series of steps that
result from different kinds of plan. In particular, this now has the
building blocks needed to implement a "create then destroy" replace
operation, which includes the step of "deposing" the original object
until we've actually destroyed it.
- The new package "exec" contains vocabulary types for data flowing between
operations, and an interface representing the operations themselves.
The types in this package allow us to carry addressing information along
with objects that would not normally carry that directly, which means
we don't need to propagate that address information around the execution
graph through side-channels.
(A notable gap as of this commit is that resource instance objects don't
have address information yet. That'll follow in a subsequent commit just
because it requires some further rejiggering.)
- The interface implemented by the applying engine and provided to the
execution graph now more directly relates to the operations described
in the execution graph, so that it's considerably easier to follow how
the graph built by the planning phase corresponds to calls into that
interface in the applying phase.
- We're starting to consider OpenTofu's own tracking identifiers for
resource instances as separate from how providers refer to their resource
types, just so that the opinions about how those two related can be
more centralized in future, vs. the traditional runtime where the rules
about how those related tend to be spread haphazardly throughout the
codebase, and in particular implementing migration between resource
types was traditionally challenging because it creates a situation where
the desired resource type and the current resource type _intentionally_
differ.
This is a large commit because the execution graph concepts are
cross-cutting between the plan and apply engines, but outside of execgraph
the changes here are mainly focused on just making things compile again
with as little change as possible, and then subsequent commits will get
the other packages into a better working shape again.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
This is just enough to connect all of the apply engine parts together into
a functional whole which can at least attempt to apply a plan. There are
various gaps here that make this not fully work in practice, but the goal
of this commit is to just get all of the components connected together in
their current form and then subsequent commits will gradually build on this
to make it work better.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
This is just some initial sketching of how the applying package might do
its work. Not yet finished, and I'm expecting to change about the
execution graph operations in future commits now that the main plan/apply
flow is plumbed in and so it's easier to test things end-to-end.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
This is just enough to get the plan object and check that it has the
execution graph field populated, and to load the configuration. More to
come in later commits.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
