This completes some of the missing connections for contexts in the provider
source codepaths by introducing context.Context parameters and wiring them
through so we can eliminate a few more context.TODO() placeholders.
For consistency's sake this adds context.Context to all four of the
getproviders.Source implementations that directly interact with stuff
outside of OpenTofu (network services or filesystem), even though not
all of them currently make use of it, just because interactions with
outside stuff tends to encourage cross-cutting concerns like logging and
tracing and so this ensures we have contexts propagated in there for such
future uses.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
Previously we were using a third-party library, but that doesn't have any
support for passing context.Context through its API and so isn't suitable
for our goals of adding OpenTelemetry tracing for all outgoing network
requests.
We now have our own fork that is updated to use context.Context. It also
has a slightly reduced scope no longer including various details that
are tightly-coupled to our cliconfig mechanism and so better placed in the
main OpenTofu codebase so we can evolve it in future without making
lockstep library releases.
The "registry-address" library also uses svchost and uses some of its types
in its public API, so this also incorporates v2 of that library that is
updated to use our own svchost module.
Unfortunately this commit is a mix of mechanical updates to the new
libraries and some new code dealing with the functionality that is removed
in our fork of svchost. The new code is primarily in the "svcauthconfig"
package, which is similar in purpose "ociauthconfig" but for OpenTofu's
own auth mechanism instead of the OCI Distribution protocol's auth
mechanism.
This includes some additional plumbing of context.Context where it was
possible to do so without broad changes to files that would not otherwise
have been included in this commit, but there are a few leftover spots that
are context.TODO() which we'll address separately in later commits.
This removes the temporary workaround from d079da6e9e, since we are now
able to plumb the OpenTelemetry span tree all the way to the service
discovery requests.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
Continuing our work to gradually plumb context.Context to everywhere that
we want to generate OpenTelemetry traces, this completes the call path
for most (but not all) of the gRPC requests to provider plugins, so that
we can add OpenTelemetry trace instrumentation in a future commit.
Unfortunately there are still a few providers.Interface callers left in
functions that don't have context.Context plumbed to them yet, and so
those are temporarily stubbed as context.TODO() here so we can more easily
find and complete them later.
The two gRPC implementations of providers.Interface were previously making
provider requests using a single context.Context established at the time
the provider process was started, but that isn't an appropriate context
to use for per-request concerns like tracing, so that context is now
unused and could potentially be removed in a future commit, but this change
already got pretty large and so I intend to deal with that separately
later.
This now exposes the gRPC provider calls to potential context cancellation
that they would previously observe only indirectly though the Stop method.
Since Stop is primarily used for graceful shutdown of ApplyResourceChange,
the changes here explicitly disconnect the cancellation signal for
ApplyResourceChange in particular, while letting the others get canceled
in the normal way since they are expected to be free of significant
side-effects. In future work we could consider removing Stop from the
internal API entirely and keeping it only as an implementation detail of
the gRPC implementation of this interface, with ApplyResourceChange
directly reacting to context cancellation and sending the gRPC Stop call
itself, but again that's too much change for this already-large commit.
The internal/legacy package currently contains some legacy code preserved
for the benefit of the backends, and unfortunately contains more than is
strictly necessary to support those callers, and so there was some dead
code there that also needed updating. provider_mock.go is removed entirely
because it's just an older copy of the similar file in package tofu. The
few calls to providers in schemas.go are updated to use
context.Background() rather than context.TODO() because we have no
intention of plumbing context.Context into that legacy code, and will
hopefully just delete it wholesale one day.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
The primary reason for this change is that registry.NewClient was
originally imposing its own decision about service discovery request
policy on every other user of the shared disco.Disco object by modifying
it directly.
We have been moving towards using a dependency inversion style where
package main is responsible for deciding how everything should be
configured based on global CLI arguments, environment variables, and the
CLI configuration, and so this commit moves to using that model for the
HTTP clients used by the module and provider registry client code.
This also makes explicit what was previously hidden away: that all service
discovery requests are made using the same HTTP client policy as for
requests to module registries, even if the service being discovered is not
a registry. This doesn't seem to have been the intention of the code as
previously written, but was still its ultimate effect: there is only one
disco.Disco object shared across all discovery callers and so changing its
configuration in any way changes it for everyone.
This initial rework is certainly not perfect: these components were not
originally designed to work in this way and there are lots of existing
test cases relying on them working the old way, and so this is a compromise
to get the behavior we now need (using consistent HTTP client settings
across all callers) without disrupting too much existing code.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
As part of our ongoing work to plumb cross-cutting concerns like tracing
spans into the core language runtime, here we change the exported API
of the context.Schemas method to take a context.Context, and trivially
update all of the callers to pass in a suitable context.
Earlier work on this means that we don't have fix up too many call stack
levels before we already have a suitable context.Context value to use.
The Schemas method doesn't yet make any use of its new context.Context, but
that will follow in subsequent PRs.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
This adds a new context.Context argument to the Backend.DeleteWorkspace
method, updates all of the implementations to match, and then updates all
of the callers to pass in a context.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
This adds a new context.Context argument to the Backend.StateMgr method,
updates all of the implementations to match, and then updates all of the
callers to pass in a context.
A small number of callers don't yet have context plumbed to them so those
use context.TODO() as a placeholder for now, so we can more easily find
and fix them in later commits once we have contexts more thoroughly
plumbed.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
This adds a new context.Context argument to the Backend.Workspaces method,
updates all of the implementations to match, and then updates all of the
callers to pass in a context.
A small number of callers don't yet have context plumbed to them so those
use context.TODO() as a placeholder for now, so we can more easily find
and fix them in later commits once we have contexts more thoroughly
plumbed.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
This adds a new context.Context argument to the Backend.Configure method,
updates all of the implementations to match, and then updates all of the
callers to pass in a context.
A small number of callers don't yet have context plumbed to them so those
use context.TODO() as a placeholder for now, so we can more easily find
and fix them in later commits once we have contexts more thoroughly
plumbed.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
This test uses a temporary file as an overridden CLI configuration to
force using a specific plugin cache directory, but temporary file paths
contain backslashes on Windows and the CLI configuration syntax is HCL
so would require backslashes to be escaped.
Since Windows will accept forward-slash paths as a supported variation,
and we typically recommend that folks write paths that way in HCL for
portability anyway, this uses filepath.ToSlash to force consistent use
of slashes on all platforms. It would also have been reasonable to use
a %q format verb to use Go's string quoting syntax, but that's not the
way we typically recommend folks hand-write their CLI configurations and
so this way is just-so-slightly more "realistic".
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
Our handling of Docker-style configuration files as an authentication
source is intentionally a bit of a compromise because various other
software reads and writes these files despite there being no single
standard for the format, and unfortunately different software makes
different tradeoffs when the configuration is ambiguous.
One oddity that we didn't notice originally is that the "login" command of
some programs will respect the "credsStore" property for storing new
credentials using a helper program instead of storing them in cleartext
in the config file BUT will still create an empty entry in the "auths"
property for whatever domain the operator logged into. Our logic wasn't
built to tolerate an "auths" entry without an "auth" property inside it
and so we would then fail to select credentials correctly for the affected
domain.
This commit makes our handling a little more resilient against
oddly-generated configuration files by silently ignoring all three of the
following oddities:
- An "auths" entry that has no "auth" property, as described above.
- An "auths" entry that is JSON null, which would previously cause OpenTofu
to crash with a null pointer dereference.
- An "auths" entry with "auth" set to an empty string, since generating
that instead of omitting the property entirely is a relatively common
mistake when using Go's encoding/json library and forgetting to add
the special "omitempty" tag to the corresponding struct field.
(An empty string is never a valid value for this property because it's
supposed to be the base64 encoding of a string like "username:password",
and so it should always at least contain a base64-encoded colon.)
Since there is no plausible valid meaning of any of these odd constructions
we prefer to just silently ignore them without any errors and without
generating any distracting log noise, which seems to match how other
software like Docker CLI and ORAS CLI handles them.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
The "tofu show" command has historically been difficult to extend to meet
new use-cases, such as showing the current configuration without creating
a plan, because it was designed to take zero or one arguments and then try
to guess what the one specified argument was intended to mean.
This commit introduces a new style where the type of object to inspect is
specified using command line option syntax, using one of two
mutually-exclusive options:
-state Show the latest state snapshot.
-plan=FILE Show the plan from the given saved plan file.
We expect that a future commit will extend this with a new "-config" option
to inspect the configuration rooted in the current working directory, and
possibly with "-module=DIR" to shallowly inspect a single module without
necessarily having to fully initialize it with all of its dependencies
first. However, both of those use-cases (and any others) are not in scope
for this commit, which is focused only on refactoring to make those future
use-cases possible.
The old mode of specifying neither option and providing zero or one
positional arguments is still supported for backward compatibility.
Notably, the legacy style is the only way to access the legacy behavior of
inspecting a specific state snapshot file from the local filesystem, which
has not often been used since Terraform v0.9 as we've moved away
from manual management of state files to the structure of state backends.
Those who _do_ still need that old behavior can still access it in the
old way, but there will be no new-style equivalent of it unless we learn
of a compelling use case for it.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
Following the lead of similar earlier work on testing the installation of
provider packages from OCI repositories, this new test exercises the new
OCI-based module source address syntax in an end-to-end fashion by directly
running "tofu init".
For the reasons described inline, this test uses a local test server as its
target OCI Registry and therefore needs to rely on a Go standard library
feature for overriding the trusted TLS certs which only works on Unix
systems other than macOS, and therefore this test will only run when the
e2etest suite is run on Linux systems. This matches the same compromise we
previously made for the provider installation flavor of this test, with
the same assumption that our module installer isn't doing anything
particularly platform-specific and that we're doing this in e2etest only
because that's an effective way to test that "package main" is wiring all
of the internal components together correctly.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
We previously had two tests of how the module installer responds to
cancellation (e.g. SIGINT) which were flakey because they tried to rely
on the cancellation being detected at some arbitrary point before the
module installer attempted to make a request, which isn't guaranteed in
practice because our interrupt mechanism only aims to cause OpenTofu to
exit "soon", with no guarantee about how much ongoing progress it will
make before it does.
To make these tests more robust, we'll now instead tell the module
installer to install from a real HTTP server that is intentionally designed
to stall the client by accepting its request but then just leaving the
connection open without responding.
This means that we can now test the more realistic situation of the cancel
signal being triggered after a slow request is already in progress, and
be sure that we're definitely sending the cancel signal at a moment that
matches that intention.
This is similar to a strategy we previously took to improve the reliability
of the tests for cancellation of the _provider_ installer, in
TestInit_cancelProviders. However, our provider installer version of this
used an intentionally-stalling implementation of getproviders.Source
instead of running a real server because the provider installer is designed
to support configurable installation methods, while the module installer
is not: its policy about what module source types are accepted is
hard-coded in package getproviders, at least for now.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
This continues our work to follow the dependency inversion style for the
"package fetcher" component of the module installer.
Mimicking the existing pattern for providers, package main is now
responsible for instantiating the PackageFetcher and providing it to
the "command" package as a field of command.Meta.
We could potentially go further here and follow dependency inversion style
for _all_ of the special clients needed by the various go-getter getters,
but our primary concern for now is preparing to add a new "getter" for
installation from an OCI Distribution repository, and so we'll leave the
other already-working code unchanged to reduce the risk of this initial
work.
Future commits will actually wire in the implementation details for OCI
Repository access. This commit focuses only on plumbing the necessary
objects through the API layers.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
Earlier work started to reshape this API to follow the dependency inversion
style, but didn't go so far as treating the package fetcher as an argument
because so far it hasn't offered any customizable policy anyway.
In future commits we will be introducing some policy arguments for the
package fetcher, and so this is some preparation work where we move the
responsibility for calling getmodules.NewPackageFetcher() out into the
caller of initwd.NewModuleInstaller().
This changes the API consumed by a bunch of unit testing helpers, so
splitting this out into its own commit should hopefully make future
commits more focused. The module installer now explicitly supports being
instantiated without a registry client or a remote package fetcher and
will in that case return an error if it's asked to install from anywhere
other than local relative directories. Most of our existing tests are
comfortable running under that constraint and so will not need any further
work in later commits that will change the signature of
getmodules.NewPackageFetcher.
However, a couple tests in package initwd _itself_ were making use of the
esoteric legacy support for treating an absolute filesystem path as a funny
sort of remote source, and so for now those will instantiate their own
package fetcher. Future commits that change the NewPackageFetcher signature
will need to offer a concession for those two tests.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
As discussed in opentofu/opentofu#2656, this consolidates the two concerns
of the PackageAuthentication interface into a single function that deals
both with package authentication _and_ with reporting all of the package
hashes that were used to make the authentication decision.
This means that any .zip archive that OpenTofu directly verifies during
installation can now have its hash recorded in the dependency lock file
even if that package didn't come from the provider's origin registry, which
is beneficial when the first installation of a provider comes from a
secondary ("mirror") source because it creates an additional hook by which
that dependency lock file entry can be "upgraded" to be complete in a
future "tofu init" run against the origin registry, or by the
"tofu providers lock" command.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
Most of the OCI registry interactions are unit tested in the most relevant
packages, but the overall system will only work correctly if all of the
components are correctly wired together by "package main", and that's one
part of the system that needs to be tested concretely rather than via
test doubles.
Therefore this adds an end-to-end test in our existing e2etest package
that runs "tofu init" with a CLI configuration that forces using an OCI
mirror with a TLS server provided locally by our test program. It exercises
the main happy path of provider installation in the same way that an
end-user would interact with it, to help avoid accidentally regressing
the interactions between these packages in future versions.
Unfortunately the technique this test uses to force the OpenTofu CLI
binary to trust the test server doesn't work on macOS or Windows and so
for now this test is Linux-specific. That's certainly non-ideal, but
pragmatic since we'll be relying mainly on the platform-agnostic unit tests
to cover this behavior, and we're unlikely to ever stop running the
e2etests on Linux as part of our pull request checks so even those
developing on macOS or Windows can still notice if this test becomes
broken before merging a change.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
It's now valid to include an oci_mirror block in the provider_installation
block in the CLI configuration, specifying that OpenTofu should try to
install providers from OCI repositories based on a template that maps
from OpenTofu-style provider source addresses into OCI repository
addresses.
The getproviders.Source implementation for this was added in a previous
commit, so this is mainly just wiring it up to the cliconfig layer and
the dependency wiring code in package main.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
