This is just enough to connect all of the apply engine parts together into
a functional whole which can at least attempt to apply a plan. There are
various gaps here that make this not fully work in practice, but the goal
of this commit is to just get all of the components connected together in
their current form and then subsequent commits will gradually build on this
to make it work better.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
This is just some initial sketching of how the applying package might do
its work. Not yet finished, and I'm expecting to change about the
execution graph operations in future commits now that the main plan/apply
flow is plumbed in and so it's easier to test things end-to-end.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
This is just enough to get the plan object and check that it has the
execution graph field populated, and to load the configuration. More to
come in later commits.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
We previously added a check like this to the Meta.Backend method, but we
use Meta.BackendForLocalPlan instead when we're applying a saved plan, so
we need to make sure the setting gets propagated here too or else the
experimental codepath cannot be entered by the "tofu apply" command.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
This makes explicit the relationships between resource instances so that
they'll be honored automatically when we actually execute the graph.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
The dependencies of a resource instance are based primarily on the marks
in the configuration value, and so we'll decide them as part of the same
function that decides the configuration value.
We then need to propagate that result through all of the intermediate
layers to the code that actually constructs the DesiredResourceInstance
object, at which point it's then available to the planning engine. In a
future commit the planning engine will use it to record the resource
instance dependencies explicitly in the generated execution graph.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
The whole point of this helper is to encapsulate the concern of having
exactly one open/close pair for each distinct provider instance address, so
in order to be effective it must actually save its results in the same
map it was already checking!
This was just an oversight in the original implementation that became more
apparent after having a visualization of execution graphs, since that made
it far more obvious that we were previously planning to open a separate
provider instance per resource instance.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
This should be just enough saved plan file support to help us get an
execution graph saved so that it can be used in a subsequent apply step,
once we implement that.
As usual with the new runtime, the traditional plan file format is mainly
just acting as a vessel to transport the execution graph and so we need to
make it valid enough that the plan loader can load it but don't need to
make it fully consistent. Therefore for now the config snapshot is just
stubbed out as empty, with the assumption that our initial "walking
skeleton" implementation will just use the loose config files on disk for
now.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
In order to produce a valid saved plan file we need to save information
about which provider instance each resource instance belongs to. We're
probably not actually going to use this information in the new
implementation because the same information is captured as part of the
execution graph, but we do at least need to make this valid enough that the
plan file loading code will accept it, because right now (for experimental
purposes) we're smuggling the execution graph through our traditional plan
file format as a sidecar data structure.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
As with the traditional runtime, we need to tolerate invalid plans from
providers that announce that they are implemented with the legacy Terraform
Plugin SDK, because that SDK is incapable of producing valid plans.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
The RPC-based provider client panics if it isn't given a non-nil value for
ProviderMeta. We aren't yet ready to send a "real" ProviderMeta value, but
for now we'll just stub it out as a null value so that we have just enough
to get through the provider client code without crashing.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
The code which tries to ensure that provider clients get closed when they
are no longer needed was not taking into account that a provider client
might not have actually been started at all, due to an error.
It's debatable whether we ought to start this background goroutine in that
case at all, but this is an intentionally-minimal change for now until we
can take the time to think harder about how the call to
reportProviderInstanceClosed should work (and whether it should happen at
all) in that case. This change ensures that it'll still happen in the
same way as it did before.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
In e879e9060f I introduced the manually-written memoization that was
previously here, as a quick fix for a problem I noticed while working on
something else.
However, using sync.Mutex directly here is not correct because it makes it
possible for this function to deadlock if used incorrectly elsewhere in the
system. The "grapheval" utilities are there to avoid exactly that by making
a call fail immediately with an error if a self-dependency problem arises.
Therefore this will now use grapheval.Once to better encapsulate the
memoization. This both ensures that we cooperate well with the rest of the
system using grapheval and requires less boilerplate inside the Value
method because grapheval.Once is already tailored for exactly this sort of
memoization.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
We previously already did some partial skipping of provider reference
validation when there are modules in the tree that failed to load, but we
were still running the main logic that checks whether the providers passed
between module boundaries are consistent.
Because that validation involves checking consistency between different
modules it will tend to generate spurious errors when run against a
partially-loaded module tree. Therefore we must also skip the other
validation steps whenever at least one module failed to load, so that the
reported diagnostics will only describe the problem loading the module and
not produce additional misleading reports describing downstream problems
caused by a module's content being incomplete due to those errors.
This one is kinda awkward to write a test for because it's a very specific
interaction that seems unlikely to arise again, but just to have _some_
representation of this historical problem I wrote a simple test that
intentionally makes one module fail to load and then tests that the result
only contains the diagnostic related to that intentional failure. Before
this fix that test would fail because of the spurious additional errors
produced by the functions we're now skipping.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
We have some test code that includes *addrs.Target values in fmt calls with
the %s and %q verbs, but that type was not actually a fmt.Stringer before.
Go 1.26 is introducing some new checks that cause those uses to cause
failures when building those tests. We could potentially change the tests
to produce the string representation in a different way, but we typically
expect our address types to be "stringable" in this way, so instead we'll
just make Target be a fmt.Stringer, delegating to the String method of
the underlying addrs.Targetable implementation. The addrs.Targetable
interface requires a String method, so all implementations are guaranteed
to support this.
Note that we're not actually using Go 1.26 yet at the time of this commit,
but this is an early fix to make the upgrade easier later. I verified this
using go1.26rc1.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
This removes most of the code previously added in 491969d29d, because we
since learned that the hashicorp/helm provider signals deferral when any
unknown values are present in provider configuration even though in
practice it can sometimes successfully plan changes in spite of those
unknown values.
That therefore made the hashicorp/helm provider behavior worse under this
change than it was before, returning an error when no error was actually
warranted.
The ephemeral resources implementation landed later and was also
interacting with this change, and so this isn't a line-for-line revert of
the original change but still removes everything that was added in support
of handling provider deferral signals so that we'll be able to start fresh
with this later if we find a better way to handle it.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
This is a _very_ early, minimal plumbing of the execution graph builder
into the experimental planning engine.
The goal here is mainly just to prove the idea that the planning engine can
build execution graphs using the execgraph.Builder API we currently have.
This implementation is not complete yet, and also we are expecting to
rework the structure of the planning engine later on anyway so this initial
work focuses on just plumbing it in to what we had as straightforwardly
as possible.
This is enough to get a serialized form of _an_ execution graph included
in the generated plan, though since we don't have the apply engine
implemented we don't actually use it for anything yet.
In subsequent commits we'll continue building out the graph-building logic
and then arrange for the apply phase to unmarshal the saved execution graph
and attempt to execute it, so we can hopefully see a minimal version of all
of this working end-to-end for the first time. But for now, this was mainly
just a proof-of-concept of building an execution graph and capturing it
into its temporary home in the plans.Plan model.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
During the "walking skeleton" phase of our work on the new planning engine
we're intentionally changing other subsystems like the CLI layer as little
as possible and just wiring things together as straightforwardly as
possible so we can find gaps in our new architectural model before we get
too invested in the current experimental shape of it.
With that in mind, this slightly extends the plans.Plan type and its
associated protobuf serialization so that we'll be able to get an opaque
representation of the new runtime's "execution graph" concept from plan
to apply while the CLI layer is still using the traditional plan models.
This is probably not how we'll deal with this in the long run, but it's
intended to be something we can tear back out relatively easily if we
decide either to abandon this "execution graph" idea altogether or once
we reach the point of implementing a new plan model that better suits the
new architecture.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
This was previously dealing okay with the case where a resource doesn't
have the requested instance, but was not handling the situation where the
resource doesn't exist at all yet.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
Most of what we interact with in configgraph is other parts of the
evaluator that automatically get memoized by patterns like OnceValuer, but
the value for a resource instance is always provided by something outside
of the evaluator that won't typically be able to use those mechanisms, and
so the evaluator's ResourceInstance.Value implementation will now provide
memoization on behalf of that external component, to ensure that we end
up with only one value for each resource instance regardless of how that
external component behaves.
In the case of the current planning phase, in particular this means that
we'll now only try to plan each resource instance once, whereas before
we would ask it to make a separate plan for each call to Value.
For now this is just retrofitted in an minimally-invasive way as part of
our "walking skeleton" phase where we're just trying to wire the existing
parts together end-to-end and then decide at the end whether we want to
refactor things more. If this need for general-purpose memoization ends
up appearing in other places too then maybe we'll choose to structure this
a little differently.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
ResultRef[*states.ResourceInstanceObjectFull] is our current canonical
representation of the "final result" of applying changes to a resource
instance, and so it is going to appear a bunch in the plan and apply
engines.
The generic type is clunky to read and write though, so for this one in
particular we'll offer a more concise type alias that we can use for parts
of the API that are representing results from applying.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
