We have a number of trace attributes that we use across all of our OCI
Distribution-based functionality, so this centralizes their definitions
in package traceattrs.
This intentionally ignores a few additional attribute names that are used
only in the code that interacts with Docker-style credential helpers,
because all of those are used only in a single function and so adding
indirection for those doesn't have enough benefit to offset the cost of
additional indirection.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
We previously added re-exports for some of the functions we'd previously
been importing directly from semconv elsewhere in this codebase. For this
one we'd previously just hard-coded the standardized attribute name, but
for consistency we'll also use a re-export of a semconv function for this
one too.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
Prevously OpenTofu delegated browser launching entirely to the third-party
module github.com/cli/browser, which consists of a number of
platform-specific lists of executable commands to try to run to launch
a web browser.
On Unix systems there is also a de-facto convention of using an environment
variable called BROWSER to explicitly specify what to launch. That variable
can either point directly to a browser, or can point to a script which
implements some more complex policy for choosing a browser, such as
detecting whether the command is running in a GUI context and launching
either a GUI or textmode browser.
The BROWSER variable has been most commonly implemented with similar
treatment to earlier variables like EDITOR and PAGER where it's expected
to be set to just a single command to run, with the URL given as the first
and only argument. There was also an attempt to define a more complex
interpretation of this variable at http://www.catb.org/~esr/BROWSER/ , but
that extended treatment was only implemented in a small amount of software,
and those which implemented it did so slightly inconsistently due to the
specification being ambiguous.
OpenTofu's implementation therefore follows the common simpler convention,
but will silently ignore variable values it cannot use so that OpenTofu
won't fail when run in an environment that has that variable set in a way
that's intended for use by some other software. In that case OpenTofu
will continue to perform the default behavior as implemented in the
third-party library.
Because this convention is Unix-specific, OpenTofu will check for and use
this environment variable only on operating systems that the Go toolchain
considers to be "unix". This means that in particular on Windows systems
OpenTofu will continue to follow the Windows convention of specifying
the default browser via an entry in the Windows Registry.
As usual with this sort of system-integration mechanism it isn't really
viable to test this end-to-end in a portable way, but the main logic is
separated out into testable functions, and I manually tested this on my
own Linux system to verify that it works in a real OpenTofu executable.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
For anything we run as PR checks it should always be straightforward to
recreate any failure locally in a development environment, so instead of
using the golangci/golangci-lint-action GitHub action (whose behavior
is nontrivial and annoying to reproduce outside of GitHub Actions) we'll
just use our existing Makefile target as part of the check run, matching
how we deal with most other similar concerns in this GitHub Actions
workflow.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
This is just a routine upgrade, keeping us up-to-date with the latest
linter changes. This already passes when run on Linux.
We know that currently the lint checks aren't passing on Windows even with
v2.4.0, and this doesn't do anything about that but it does mean that once
we make the linters pass on Windows we'll be dealing with the full set
supported in v2.6.0, rather than potentially having to do this across
two commits.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
Go 1.17 and earlier used a different syntax for build constraint comments,
starting with "+build". Go 1.18 changed this to the modern "go:build" form
as part of standardizing the structure of toolchain directive comments,
and so for a while it was convention to include comments in both styles
to allow building with both old and new Go compilers.
However, Go 1.17 is no longer supported, and regardless of that we only
expect OpenTofu to be built with the specific version we have selected
in "go.mod" and ".go-version" anyway, so we no longer need the legacy form
of these comments: the all supported Go toolchains now support the new
form, which this commit retains.
golangci-lint v2.6.0 includes a check for this legacy form, so removing
this will also allow us to upgrade to a newer version of that linter
aggregator in a future commit.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
OpenTelemetry has various Go packages split across several Go modules that
often need to be carefully upgraded together. And in particular, we are
using the "semconv" package in conjunction with the OpenTelemetry SDK's
"resource" package in a way that requires that they both agree on which
version of the OpenTelemetry Semantic Conventions are being followed.
To help avoid "dependency hell" situations when upgrading, this centralizes
all of our direct calls into the OpenTelemetry SDK and tracing API into
packages under internal/tracing, by exposing a few thin wrapper functions
that other packages can use to access the same functionality indirectly.
We only use a relatively small subset of the OpenTelemetry library surface
area, so we don't need too many of these reexports and they should not
represent a significant additional maintenance burden.
For the semconv and resource interaction in particular this also factors
that out into a separate helper function with a unit test, so we should
notice quickly whenever they become misaligned. This complements the
end-to-end test previously added in opentofu/opentofu#3447 to give us
faster feedback about this particular problem, while the end-to-end test
has the broader scope of making sure there aren't any errors at all when
initializing OpenTelemetry tracing.
Finally, this also replaces the constants we previously had in package
traceaddrs with functions that return attribute.KeyValue values directly.
This matches the API style used by the OpenTelemetry semconv packages, and
makes the calls to these helpers from elsewhere in the system a little
more concise.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
We got this new functionality from an upgrade of the upstream cty library,
but we didn't update the docs to mention it.
The examples in this doc were also evidently generated with a much older
version of OpenTofu's predecessor, because the illustrated output was not
shown as a quoted string. The example output now matches how the current
version of "tofu console" would describe these results.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
Based on discussion so far it seems that we have consensus about merging
the lang/eval and engine-related commits that precede this one as a
starting point for ongoing work in future (hopefully smaller) PRs.
This commit therefore mostly just acknowledges a small amount of unused
code in a way that it won't cause linter noise, since the future work is
likely to make these be used and so it'll be helpful to have them around as
examples to build on. (Which is, after all, the whole point of merging this
as dead code in the first place!)
This does _actually_ remove an old, now-redundant declaration of the
"compiled module instance" interface though, since that eventually ended
up in a different place and I just forgot to clean up this initial form
of it in later refactoring.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
This now seems to more-or-less work, at least as far as the new
compile-and-execute is concerned.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
This covers most of the logic required to turn a source graph into a
compiled graph ready for execution. There's currently only support for one
of the opcodes though, so subsequent commits will sketch those out more
and then add some tests and fix any problems that inevitably exist here but
aren't yet visible because there are no tests.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
Doesn't actually do anything yet. Just sketching how it might be
structured, with a temporary object giving us somewhere to keep track of
the relationships between nodes that we can discard once compilation is
complete, keeping only the information required to execute.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
So far this is mainly just the mechanism for building a graph piecemeal
from multiple callers working together as part of the planning engine.
The end goal is for it to be possible to "compile" an assembled graph into
something that can then be executed, and to be able to marshal/unmarshal an
uncompiled graph to save as part of a plan file, but those other
capabilities will follow in later commits.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
This sketches the main happy path of planning for a desired managed
resource instance, though doesn't include all of the possible variations
that authors can cause with different configuration settings.
The main goal here is just to illustrate how the planning step might
interact with the work done in the eval system. If we decide to go in this
direction then we'll need to bring over all of the complexity from the
traditional runtime into this codepath, or possibly find some way to reuse
that behavior directory from "package tofu" in the short term.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
Previously the PlanGlue methods all took PlanningOracle pointers as one
of their arguments, which is annoying since all of them should end up with
pointers to the same object and it makes it hard for the PlanGlue
implementation to do any work outside of and between the PlanGlue method
calls.
Instead then we'll have DrivePlanning take a function for building a
PlanGlue implementation given a PlanningOracle pointer, and then the
planning engine returns an implementation that binds a planContext to a
PlanningOracle it can then use to do all of its work.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
This is a sketch of starting and configuring provider instances once they
are needed and then closing them once they are no longer needed. The
stubby implementation of data resource instance planning is there mainly
just to illustrate how this is intended to work, but it's not yet complete
enough to actually function.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
This is a similar idea to a sync.WorkGroup except that it tracks the
completion of individual items represented by comparable values, rather
than only a count of expected items to complete.
My intention for this is for the planning engine to use it to track when
all of the expected work for a provider instance or ephemeral resource
instance has been completed, so that it can be closed shortly afterwards
instead of waiting for the entire planning operation to run to completion.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
This completes a previously-missing piece of the "prepareToPlan" result,
tracking which provider instances are relying on each ephemeral resource
instance.
This is important because the planning engine can "see"
resource-instance-to-provider relationships in the state that the eval
system isn't aware of, and so the planning engine must be able to keep
a provider instance open long enough to deal with both config-driven and
state-driven uses of it, which in turn means keeping open any ephemeral
resource instances that those provider instances depend on.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
The eval system gradually reports the "desired" declarations at various
different levels of granularity, and so the planning engine should compare
that with the instances in the previous run state to notice when any
existing resource instance is no longer in the desired state.
This doesn't yet include a real implementation of planning the deletion of
such resource instances. Real planning behaviors depend on us having some
sort of provider instance and ephemeral resource instance manager to be
able to make requests to configured providers, so that will follow in
subsequent commits before we can implement the actual planning behaviors.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
This follows the naming convention from the stdlib packages "slices" and
"maps" of using the verb "Collect" to represent gathering the items from
a sequence into some sort of collection data structure.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
In a new async recursive tree walk we'll gradually traverse the tree of
module calls and announce the desired state objects at each level, so that
the PlanGlue implementation can then compare that with its own record of
the prior state to determine by omission which resource instances are
"orphans" that should be planned for deletion.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
This puts the infrastructure in place to allow us to notify the PlanGlue
implementation about what's declared in the configuration so it can detect
orphaned resource instances by omission, but stops short of actually doing
that because it'll require another concurrent recursive tree walk which
is likely to be complex enough to deserve to live in its own later commit.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
Previously this had the idea that the planning engine would work in two
subphases where first it would deal with the desired state (decided by
evaluating the configuration) and then it would do a second pass to plan
to delete any resource instances that didn't show up in the desired state.
Although that would technically work, we need to keep configured provider
instances open for the whole timespan between planning the first
desired resource instance and the final orphan instance using each
provider, and so splitting into two phases effectively means we'd have
provider instances sitting open but idle waiting for the desired state
subphase to complete to learn if they are needed for the orphan subphase.
This commit stubs out (but does not yet implement) a new approach where
the orphan planning can happen concurrently with the desired state
planning. This works by having the evaluator gradually describe to the
planning engine which module calls, module call instances, resources, and
resource instances _are_ present in the desired state -- as soon as that
information becomes available -- so that the planning engine can notice
by omission what subset of the prior state resource instances are no longer
desired and are therefore orphans.
This means that the planning engine should now have all of the information
it needs to recognize when a provider can be closed: the PlanningOracle
reports which _desired_ resource instances (or placeholders thereof) are
expected to need each provider instance, and the prior state tracks
exactly which provider instance each prior resource instance was associated
with, and so a provider's work is finished once all instances in the union
of those two sets have had their action planned.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
This is a problem that has been on my mind for the past little while as
I've been building this out, so I just wanted to write it down in a place
where others reading this code could find it, to acknowledge that there is
a gap in this new design idea that prevents it from having the similar
provider open/close behavior to the current implementation.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
We already have a few situations where the evaluation phase can produce
an incomplete placeholder for zero or more resource instances that might
appear in a future plan/apply round, and the plan implementation itself
will introduce some more based on replies from provider requests.
Our goal is to always plan as much as possible with the information we
have, including possibly returning errors for partially-evaluated objects
when we're confident that they could possibly become valid in the presence
of more information, and so in some cases we will end up visiting a
resource instance that would not need to be deferred if considered in
isolation but nonetheless its configuration is depending on an outcome of
an action that was already deferred and so it must therefore also be
deferred.
This currently-unused cty mark will allow us to use dynamic analysis to
track when parts of a resource instance whose action was deferred are used
as part of another resource instance. This is superior to a static analysis
approach because it will allow us to notice situations such as when only
one arm of a conditional relies on a deferred result.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
All of this still leads to a TODO panic but we now have a separate method
to implement for each of the resource modes, since the handling of each
is pretty independent of the others.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
This doesn't actually work yet because the main implementation functions
are not written, but this is a sketch of the general layout of things that
the lang/eval package was designed to support: the planning phase first
lets lang/eval drive based on the desired state it discovers gradually in
the configuration, but afterwards it then does followup work for any
resource instances that didn't appear in the desired state but yet are
present in the previous round state.
This overall structure should allow the planning engine to discover all
of the changes that need to be made to resource instances without any
direct knowledge of how the OpenTofu language works and how the module
author chose to describe the desired state as code.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
Although our EvalContext API is primarily focused on what the eval system
needs, it's desirable for higher-level subsystems that wrap the evaluator
to be able to guarantee they are using the same provider and other
dependencies, so we'll expose the EvalContext that each ConfigInstance is
associated with in the public API.
The Providers interface now also grows to include NewConfiguredProvider as
a way to instantiate a configured provider instance. The eval system
doesn't need that itself, but exposing it here means that a caller can be
confident that they are definitely instantiating and configuring the same
provider that the eval system had previously used to validate the
configuration.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
As was already noted in a comment in the PlanningOracle declaration, the
PlanningOracle API is an exported entrypoint usable from outside lang/eval
and so each exported method should establish its own workgraph worker to
make sure it's handling those concerns correctly regardless of how the
caller chooses to use them.
For example, a caller might choose to make concurrent calls into the oracle
from multiple goroutines, which would violate the assumption that each
concurrently-running codepath is associated with a separate workgraph
worker.
Even if a caller is managing their own workgraph workers in the expected
way, this is functionally harmless and relatively cheap.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
This is a building block for use by a planning engine implemented elsewhere
in OpenTofu so that it can be notified (by the completion of a call to the
function) when all resource instances that use a particular provider
instance or ephemeral resource instance have completed their plan-time
evaluation work.
Existing methods of PlanningOracle already expose sets of addresses to
wait for, but this method is separate from those because the planning
engine also needs the dependency information as data to know when an
ephemeral resource instance might need to be left open beyond the part of
the planning process driven by lang/eval in order to plan the delete
actions for any "orphan" resource instances that are in the prior state
but no longer declared in the configuration.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
Continuing to split up some of the source files that have grown long over
ongoing experimentation, this is the start of splitting the config_plan.go
file into smaller parts.
For now only PlanningOracle gets its own file, but other types related to
planning might follow in later commits.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
This was the big known missing piece for making a "real" implementation
of planning using this new package, so that the planning engine can know
what configuration to use when preparing a configured provider instance
for planning requests.
I'm sure there are other missing pieces that I've just not learned about
yet, but this is still a good checkpoint to stop iterating on lang/eval
for a little while and start stubbing out the planning engine.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
The previous approach had the problem that we only had the
CompiledModuleInstance object for a child module briefly in a local
variable while evaluating the child call, which is sufficient for just
evaluation but is not enough to support full config tree traversal for
questions like "what resource instances are declared throughout the
configuration?" and "what configgraph node corresponds to this provider
configuration instance address?"
This moves the separation of concerns around a little to be perhaps more
like how resource instances work, where the ModuleCallInstance.Value
method just wraps a function provided by the "compile" layer which then
takes the config value and compiles the child module instance.
This means that the "compile" layer can then hold on to the
CompiledModuleInstance object for the child module instance as part of
the "glue" in the ModuleCallInstance object and then use it to deal with
the config-tree-traversing methods in its own CompiledModuleInstance
implementation.
The new test of ConfigInstance.PrepareToPlan in lang/eval illustrates that
the system is now finally able to walk the whole configuration tree to find
resource instances and the provider instances they depend on.
(The handling of inheritance of providers between parent and child module
instances is still not working like the current system does, because the
"providers sidechannel" mechanism remains incomplete.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
This file was originally in lang/eval directly and so it had all of the
compile-related functions grouped together to separate them from everything
else in that package.
Now that the "compile" process has its own package we have more room to
have many smaller files instead of one huge file, making this a little
easier to navigate now that the logic has grown.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
Exposing only "ResourceInstancesDeep" on CompiledModuleInstance seemed
attractive at first when it was only partially implemented anyway, but
trying to finish implementing it revealed that the current design couldn't
actually support that API because we don't currently keep the
CompiledModuleInstance object for a child module instance after we've used
it to evaluate the child module instance's output values.
This commit doesn't actually solve that problem, but it does rework the
CompiledModuleInstance API so that the problem of finding and enumerating
child module instances is separated from the problem of finding resource
instances in just the current module instance, and then I'll try to find
a good way to satisfy this new, slightly-generalized API in future commits.
(The ultimate goal here is to be able to enumerate all of the resource
instances throughout the configuration for dependency-tracking purposes,
and to be able to find a specific ProviderInstance object given its
absolute address so that the plan or apply engine can request the
configuration value to configure the provider with.)
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
This introduces the "compiler" changes and the configgraph changes to let
us evaluate the configuration for a particular provider instance.
A future commit will expose this through the public API of lang/eval so
that other systems like the planning engine will be able to ask for the
configuration for whatever provider instance a given resource instance
belongs to.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
This is still not completely true to how current OpenTofu behaves, but it's
at least a little closer and avoids hard-coding a specific fixed provider
address in favor of actually following what's declared in the
configuration.
The system doesn't actually support evaluating the configuration for a
provider instance yet, so this remains just a placeholder for the
relatively simple tests in this package. Support for actually evaluating
the provider configurations will follow in later commits.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
