We got this new functionality from an upgrade of the upstream cty library,
but we didn't update the docs to mention it.
The examples in this doc were also evidently generated with a much older
version of OpenTofu's predecessor, because the illustrated output was not
shown as a quoted string. The example output now matches how the current
version of "tofu console" would describe these results.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
Based on discussion so far it seems that we have consensus about merging
the lang/eval and engine-related commits that precede this one as a
starting point for ongoing work in future (hopefully smaller) PRs.
This commit therefore mostly just acknowledges a small amount of unused
code in a way that it won't cause linter noise, since the future work is
likely to make these be used and so it'll be helpful to have them around as
examples to build on. (Which is, after all, the whole point of merging this
as dead code in the first place!)
This does _actually_ remove an old, now-redundant declaration of the
"compiled module instance" interface though, since that eventually ended
up in a different place and I just forgot to clean up this initial form
of it in later refactoring.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
This now seems to more-or-less work, at least as far as the new
compile-and-execute is concerned.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
This covers most of the logic required to turn a source graph into a
compiled graph ready for execution. There's currently only support for one
of the opcodes though, so subsequent commits will sketch those out more
and then add some tests and fix any problems that inevitably exist here but
aren't yet visible because there are no tests.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
Doesn't actually do anything yet. Just sketching how it might be
structured, with a temporary object giving us somewhere to keep track of
the relationships between nodes that we can discard once compilation is
complete, keeping only the information required to execute.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
So far this is mainly just the mechanism for building a graph piecemeal
from multiple callers working together as part of the planning engine.
The end goal is for it to be possible to "compile" an assembled graph into
something that can then be executed, and to be able to marshal/unmarshal an
uncompiled graph to save as part of a plan file, but those other
capabilities will follow in later commits.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
This sketches the main happy path of planning for a desired managed
resource instance, though doesn't include all of the possible variations
that authors can cause with different configuration settings.
The main goal here is just to illustrate how the planning step might
interact with the work done in the eval system. If we decide to go in this
direction then we'll need to bring over all of the complexity from the
traditional runtime into this codepath, or possibly find some way to reuse
that behavior directory from "package tofu" in the short term.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
Previously the PlanGlue methods all took PlanningOracle pointers as one
of their arguments, which is annoying since all of them should end up with
pointers to the same object and it makes it hard for the PlanGlue
implementation to do any work outside of and between the PlanGlue method
calls.
Instead then we'll have DrivePlanning take a function for building a
PlanGlue implementation given a PlanningOracle pointer, and then the
planning engine returns an implementation that binds a planContext to a
PlanningOracle it can then use to do all of its work.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
This is a sketch of starting and configuring provider instances once they
are needed and then closing them once they are no longer needed. The
stubby implementation of data resource instance planning is there mainly
just to illustrate how this is intended to work, but it's not yet complete
enough to actually function.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
This is a similar idea to a sync.WorkGroup except that it tracks the
completion of individual items represented by comparable values, rather
than only a count of expected items to complete.
My intention for this is for the planning engine to use it to track when
all of the expected work for a provider instance or ephemeral resource
instance has been completed, so that it can be closed shortly afterwards
instead of waiting for the entire planning operation to run to completion.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
This completes a previously-missing piece of the "prepareToPlan" result,
tracking which provider instances are relying on each ephemeral resource
instance.
This is important because the planning engine can "see"
resource-instance-to-provider relationships in the state that the eval
system isn't aware of, and so the planning engine must be able to keep
a provider instance open long enough to deal with both config-driven and
state-driven uses of it, which in turn means keeping open any ephemeral
resource instances that those provider instances depend on.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
The eval system gradually reports the "desired" declarations at various
different levels of granularity, and so the planning engine should compare
that with the instances in the previous run state to notice when any
existing resource instance is no longer in the desired state.
This doesn't yet include a real implementation of planning the deletion of
such resource instances. Real planning behaviors depend on us having some
sort of provider instance and ephemeral resource instance manager to be
able to make requests to configured providers, so that will follow in
subsequent commits before we can implement the actual planning behaviors.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
This follows the naming convention from the stdlib packages "slices" and
"maps" of using the verb "Collect" to represent gathering the items from
a sequence into some sort of collection data structure.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
In a new async recursive tree walk we'll gradually traverse the tree of
module calls and announce the desired state objects at each level, so that
the PlanGlue implementation can then compare that with its own record of
the prior state to determine by omission which resource instances are
"orphans" that should be planned for deletion.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
This puts the infrastructure in place to allow us to notify the PlanGlue
implementation about what's declared in the configuration so it can detect
orphaned resource instances by omission, but stops short of actually doing
that because it'll require another concurrent recursive tree walk which
is likely to be complex enough to deserve to live in its own later commit.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
Previously this had the idea that the planning engine would work in two
subphases where first it would deal with the desired state (decided by
evaluating the configuration) and then it would do a second pass to plan
to delete any resource instances that didn't show up in the desired state.
Although that would technically work, we need to keep configured provider
instances open for the whole timespan between planning the first
desired resource instance and the final orphan instance using each
provider, and so splitting into two phases effectively means we'd have
provider instances sitting open but idle waiting for the desired state
subphase to complete to learn if they are needed for the orphan subphase.
This commit stubs out (but does not yet implement) a new approach where
the orphan planning can happen concurrently with the desired state
planning. This works by having the evaluator gradually describe to the
planning engine which module calls, module call instances, resources, and
resource instances _are_ present in the desired state -- as soon as that
information becomes available -- so that the planning engine can notice
by omission what subset of the prior state resource instances are no longer
desired and are therefore orphans.
This means that the planning engine should now have all of the information
it needs to recognize when a provider can be closed: the PlanningOracle
reports which _desired_ resource instances (or placeholders thereof) are
expected to need each provider instance, and the prior state tracks
exactly which provider instance each prior resource instance was associated
with, and so a provider's work is finished once all instances in the union
of those two sets have had their action planned.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
This is a problem that has been on my mind for the past little while as
I've been building this out, so I just wanted to write it down in a place
where others reading this code could find it, to acknowledge that there is
a gap in this new design idea that prevents it from having the similar
provider open/close behavior to the current implementation.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
We already have a few situations where the evaluation phase can produce
an incomplete placeholder for zero or more resource instances that might
appear in a future plan/apply round, and the plan implementation itself
will introduce some more based on replies from provider requests.
Our goal is to always plan as much as possible with the information we
have, including possibly returning errors for partially-evaluated objects
when we're confident that they could possibly become valid in the presence
of more information, and so in some cases we will end up visiting a
resource instance that would not need to be deferred if considered in
isolation but nonetheless its configuration is depending on an outcome of
an action that was already deferred and so it must therefore also be
deferred.
This currently-unused cty mark will allow us to use dynamic analysis to
track when parts of a resource instance whose action was deferred are used
as part of another resource instance. This is superior to a static analysis
approach because it will allow us to notice situations such as when only
one arm of a conditional relies on a deferred result.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
All of this still leads to a TODO panic but we now have a separate method
to implement for each of the resource modes, since the handling of each
is pretty independent of the others.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
This doesn't actually work yet because the main implementation functions
are not written, but this is a sketch of the general layout of things that
the lang/eval package was designed to support: the planning phase first
lets lang/eval drive based on the desired state it discovers gradually in
the configuration, but afterwards it then does followup work for any
resource instances that didn't appear in the desired state but yet are
present in the previous round state.
This overall structure should allow the planning engine to discover all
of the changes that need to be made to resource instances without any
direct knowledge of how the OpenTofu language works and how the module
author chose to describe the desired state as code.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
Although our EvalContext API is primarily focused on what the eval system
needs, it's desirable for higher-level subsystems that wrap the evaluator
to be able to guarantee they are using the same provider and other
dependencies, so we'll expose the EvalContext that each ConfigInstance is
associated with in the public API.
The Providers interface now also grows to include NewConfiguredProvider as
a way to instantiate a configured provider instance. The eval system
doesn't need that itself, but exposing it here means that a caller can be
confident that they are definitely instantiating and configuring the same
provider that the eval system had previously used to validate the
configuration.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
As was already noted in a comment in the PlanningOracle declaration, the
PlanningOracle API is an exported entrypoint usable from outside lang/eval
and so each exported method should establish its own workgraph worker to
make sure it's handling those concerns correctly regardless of how the
caller chooses to use them.
For example, a caller might choose to make concurrent calls into the oracle
from multiple goroutines, which would violate the assumption that each
concurrently-running codepath is associated with a separate workgraph
worker.
Even if a caller is managing their own workgraph workers in the expected
way, this is functionally harmless and relatively cheap.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
This is a building block for use by a planning engine implemented elsewhere
in OpenTofu so that it can be notified (by the completion of a call to the
function) when all resource instances that use a particular provider
instance or ephemeral resource instance have completed their plan-time
evaluation work.
Existing methods of PlanningOracle already expose sets of addresses to
wait for, but this method is separate from those because the planning
engine also needs the dependency information as data to know when an
ephemeral resource instance might need to be left open beyond the part of
the planning process driven by lang/eval in order to plan the delete
actions for any "orphan" resource instances that are in the prior state
but no longer declared in the configuration.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
Continuing to split up some of the source files that have grown long over
ongoing experimentation, this is the start of splitting the config_plan.go
file into smaller parts.
For now only PlanningOracle gets its own file, but other types related to
planning might follow in later commits.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
This was the big known missing piece for making a "real" implementation
of planning using this new package, so that the planning engine can know
what configuration to use when preparing a configured provider instance
for planning requests.
I'm sure there are other missing pieces that I've just not learned about
yet, but this is still a good checkpoint to stop iterating on lang/eval
for a little while and start stubbing out the planning engine.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
The previous approach had the problem that we only had the
CompiledModuleInstance object for a child module briefly in a local
variable while evaluating the child call, which is sufficient for just
evaluation but is not enough to support full config tree traversal for
questions like "what resource instances are declared throughout the
configuration?" and "what configgraph node corresponds to this provider
configuration instance address?"
This moves the separation of concerns around a little to be perhaps more
like how resource instances work, where the ModuleCallInstance.Value
method just wraps a function provided by the "compile" layer which then
takes the config value and compiles the child module instance.
This means that the "compile" layer can then hold on to the
CompiledModuleInstance object for the child module instance as part of
the "glue" in the ModuleCallInstance object and then use it to deal with
the config-tree-traversing methods in its own CompiledModuleInstance
implementation.
The new test of ConfigInstance.PrepareToPlan in lang/eval illustrates that
the system is now finally able to walk the whole configuration tree to find
resource instances and the provider instances they depend on.
(The handling of inheritance of providers between parent and child module
instances is still not working like the current system does, because the
"providers sidechannel" mechanism remains incomplete.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
This file was originally in lang/eval directly and so it had all of the
compile-related functions grouped together to separate them from everything
else in that package.
Now that the "compile" process has its own package we have more room to
have many smaller files instead of one huge file, making this a little
easier to navigate now that the logic has grown.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
Exposing only "ResourceInstancesDeep" on CompiledModuleInstance seemed
attractive at first when it was only partially implemented anyway, but
trying to finish implementing it revealed that the current design couldn't
actually support that API because we don't currently keep the
CompiledModuleInstance object for a child module instance after we've used
it to evaluate the child module instance's output values.
This commit doesn't actually solve that problem, but it does rework the
CompiledModuleInstance API so that the problem of finding and enumerating
child module instances is separated from the problem of finding resource
instances in just the current module instance, and then I'll try to find
a good way to satisfy this new, slightly-generalized API in future commits.
(The ultimate goal here is to be able to enumerate all of the resource
instances throughout the configuration for dependency-tracking purposes,
and to be able to find a specific ProviderInstance object given its
absolute address so that the plan or apply engine can request the
configuration value to configure the provider with.)
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
This introduces the "compiler" changes and the configgraph changes to let
us evaluate the configuration for a particular provider instance.
A future commit will expose this through the public API of lang/eval so
that other systems like the planning engine will be able to ask for the
configuration for whatever provider instance a given resource instance
belongs to.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
This is still not completely true to how current OpenTofu behaves, but it's
at least a little closer and avoids hard-coding a specific fixed provider
address in favor of actually following what's declared in the
configuration.
The system doesn't actually support evaluating the configuration for a
provider instance yet, so this remains just a placeholder for the
relatively simple tests in this package. Support for actually evaluating
the provider configurations will follow in later commits.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
This is a little too busy/complicated for my liking but it's a place to
start and hopefully we'll be able to cut this down a little after we see
how the planning engine implementation turns out.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
The way this all fits together doesn't feel quite right yet since there's
a bunch of wrapping layers that aren't really adding anything except
indirection between the packages, but this is at least a starting point
for child module support.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
This borrows the function table preparation from our previous lang.Scope
API (which this experiment would, if successful, ultimately replace) and
also intentionally implements the exprs.Evalable interface a little
improperly for now in the hope that we'll update our HCL patch for
function inspections to use hcl.StaticCall instead of misusing
hcl.Traversal at some point in the future.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
In other packages we've tried being pragmatic by making the main code
compensate for missing arguments in older tests, so that we don't always
need to update every test across the whole system every time we add a new
feature.
However, that compromise unfortunately tends to lead to non-test code
inadvertently relying on the compensations too, so this is an attempt at
a slightly different compromise where the main code will panic if called
incorrectly but there's an extra helper function for tests to fill in
reasonable "inert" defaults for the fields they leave unpopulated.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
Recent changes in cty have included some new functions for working with
marks more efficiently. This particular function allows us to iterate over
the deep marks on a value without creating a new unmarked copy of the
value and without building an intermediate map of marks.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
This also further highlighted that the marks-related codepaths in cty have
lots of room for performance improvement. In particular, we're spending
a lot of time in cty.Value.UnmarkDeep rebuilding values that actually
don't need to change and could likely get some benefit from rewriting
that to use the recently-added WrangleMarksDeep that is able to avoid
constructing new values for parts of the tree where no marks are changing.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
This stops short of actually implementing the evaluation of a module call,
but gets the compilation behavior in place and the usual boilerplate for
handling multi-instance module calls.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
A big part of the motivation of the design of configgraph vs. package tofu
was to keep each node type self-contained and avoid the need for a big
shared "god object" that everything interacts with, and in turn part of
the reason to be interested in that is because it should makes it actually
practical and useful to write unit tests for these node types.
This commit introduces a few unit tests mainly just as a proof-of-concept
to see if the design has lived up to the ambition. So far that seems to be
true, although I'll reserve judgement until there's at least some coverage
on every on of the node types in this package.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
During early experiments the logic for "compiling" a configs.Module into
a bunch of configgraph node objects was just a bunch of unexported
functions directly in package eval, because it wasn't clear yet what
abstraction would make most sense between those two.
Now that this has solidified a little more this commit is a first attempt
at drawing a line between packages implementing "compilation" for specific
editions of the language (for which there is currently only one, which
I've named "tofu2024" as a placeholder because Terraform's only existing
language was similarly called "TF2021") and the top-level evaluation
package that tries to be as edition-agnostic as possible, although it is
currently still the one responsible for deciding to call into tofu2024.
There doesn't seem much point in abstracting away the decision about what
edition to use until we actually have a plan to introduce a new edition
and therefore have some idea of what the design constraints for that might
turn out to be.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
Completing the reorganization of all of the scope-shape-related decisions
into the "compiler" code instead of package configgraph, this now deals
with the scope that contains the each.key/each.value/count.index local
symbols.
Package configgraph now no longer knows anything about the shape of the
scopes except for a few pragmatic mentions of specific conventions in
error messages where it doesn't seem worth the complexity to try to
organize that differently until we know concretely what problem we're
trying to solve.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
The pattern of having the "compiler" logic in lang/exprs always deal with
building suitable exprs.Valuers and configgraph just evaluating whatever
it was given has been working out in other contexts like the resource to
resource instance journey, so we'll follow that pattern for CheckRule too:
If the evaluation of a CheckRule depends on some value that's generated
by logic in configgraph then configgraph has a callback to ask the
"compiler" layer to dynamically construct zero or more CheckRule objects
that are pre-bound to the relevant value. For checks that only rely on
the global scope, such as output value preconditions, they are still
precompiled into a static slice in the parent object.
This also continues the progress towards all of the "expression scopes"
living with the "compiler" code instead of in configgraph, so that in
future different modules can potentially be bound to differently-shaped
scopes as we continue to evolve the language.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
On reflection it seems inconsistent that the "compile" logic in lang/eval
is responsible for deciding which expressions get evaluated in which scopes
but not for deciding what's actually in the top-level scope, since the
scope contents are something quite likely to vary between language
editions and language experiments.
With that in mind, this separates the idea of the "module instance scope"
from the actual ModuleInstance type, making it instead a wrapper type.
This continues the trend toward the grapheval package being mostly agnostic
to how the surface-level language is designed and instead just modeling
the relationships that are implied by the configuration.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
This introduces the machinery for representing provider configs, provider
instances, and the relationships between resource instances and provider
instances.
Unfortunately there's a lot of accumulated complexity in the rules for
how OpenTofu currently decides which provider configs are available for
use in a module, including lots of implicit behavior for
backward-compatibility, so right now that is stubbed out with just a
hard-coded implementation that puts a "test/foo" provider with local name
"foo" in every module instance. We'll deal with a real implementation of
all that in later commits.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
Previously the way this was dealing with "compiling" resource instances
once we've determined the instance keys was a little inconsistent with the
intended architecture because the configgraph.Resource type needed to know
how to pass various bits of context down into each of the instances, but
everywhere else we've tried to keep the configgraph types pretty agnostic
about such details and let the "compiler" layer worry about that.
Now the entire task of "compiling" a resource instance is delegated to
the "compiler" layer using a callback it provides. This makes things easier
because that layer already has all of the resource-level information it
needs available anyway, and so it can just plug that information directly
into the ResourceInstance fields itself in the same way as it builds
other types like ModuleInstance and Resource.
This also introduces an interface type to represent the callback API from
the ResourceInstance itself back to the compiler layer, which means we can
now split config validation and "get result value" into two methods, which
makes the implementation of each of those methods considerably simpler --
they are both now one-liners, and the data flow from validation to
get result value is encapsulated in the configgraph package where it will
be shared between all phases.
Overall the idea here is to make Resource and ResourceInstance independent
of one another so that, as is the goal for everything in configgraph, it's
more practical to write unit tests that don't need to mock out an entire
world just to test one small unit.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
For annoying historical reasons the various address types we use to talk
about provider configs are inconsistent with how other address types work,
and we have no single address type for talking about a provider _instance_
at all.
As a stopgap to support the experimenting in lang/eval this introduces some
arguably-more-correct address types, which represent things in a more
useful way for a new world where providers aren't such a weird special
case.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
