We have a number of trace attributes that we use across all of our OCI
Distribution-based functionality, so this centralizes their definitions
in package traceattrs.
This intentionally ignores a few additional attribute names that are used
only in the code that interacts with Docker-style credential helpers,
because all of those are used only in a single function and so adding
indirection for those doesn't have enough benefit to offset the cost of
additional indirection.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
We previously added re-exports for some of the functions we'd previously
been importing directly from semconv elsewhere in this codebase. For this
one we'd previously just hard-coded the standardized attribute name, but
for consistency we'll also use a re-export of a semconv function for this
one too.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
OpenTelemetry has various Go packages split across several Go modules that
often need to be carefully upgraded together. And in particular, we are
using the "semconv" package in conjunction with the OpenTelemetry SDK's
"resource" package in a way that requires that they both agree on which
version of the OpenTelemetry Semantic Conventions are being followed.
To help avoid "dependency hell" situations when upgrading, this centralizes
all of our direct calls into the OpenTelemetry SDK and tracing API into
packages under internal/tracing, by exposing a few thin wrapper functions
that other packages can use to access the same functionality indirectly.
We only use a relatively small subset of the OpenTelemetry library surface
area, so we don't need too many of these reexports and they should not
represent a significant additional maintenance burden.
For the semconv and resource interaction in particular this also factors
that out into a separate helper function with a unit test, so we should
notice quickly whenever they become misaligned. This complements the
end-to-end test previously added in opentofu/opentofu#3447 to give us
faster feedback about this particular problem, while the end-to-end test
has the broader scope of making sure there aren't any errors at all when
initializing OpenTelemetry tracing.
Finally, this also replaces the constants we previously had in package
traceaddrs with functions that return attribute.KeyValue values directly.
This matches the API style used by the OpenTelemetry semconv packages, and
makes the calls to these helpers from elsewhere in the system a little
more concise.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
The new version has slightly changed the structure of returned error
messages, so this includes some small compensating changes to avoid the
messages redundantly mentioning the same tag or digest twice.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
This adds more detailed OTel trace spans to our various different
interactions with OCI repositories, which is helpful to understand the
time spent in each of the various sequential steps involved in resolving
an OCI artifact.
OTel's centrally-maintained conventions for attribute names currently only
have a standard for reporting a manifest digest, so we'll use that where
it's appropriate but use our own "opentofu.oci."-prefixed attribute names
for everything else for now. Hopefully the upstream standard will be
broadened later to include some additional concepts, at which point we
can switch over to the standardized attribute names.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
This connects up various work done in earlier commits so that it's now
possible to install modules from source addresses that start with "oci:",
which will each get interpreted as a reference to an artifacts in an OCI
Distribution repository.
For the first time we now have a getter that needs to be configured
dynamically based on the CLI configuration, so this slightly reworks the
"reusingGetter" type so that each instance has its own map of getters
that's based on the statically-configured one. Currently "oci" is the only
getter that needs this dynamic configuration, but perhaps in future we'll
adopt a similar dependency inversion style for some of the other getters
so that we can centralize concerns such as allowing operators to configure
additional TLS certificates for OpenTofu to trust.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
Originally reusingGetter was just a map directly, since it had no other
data to track beyond "remembering" what it had previously installed.
In a future commit we'll be adding a dynamically-constructed set of
"getters" so that we can configure the new "OCI" getter dynamically based
on the active CLI configuration, so we'll need this to be a struct type
instead.
Since we are now able to do so, this commit introduces a mutex to make it
safe to call the reusingGetter concurrently from multiple goroutines,
although this initial implementation is intentionally simplistic and just
forces all of the install requests to be handled sequentially, since we
don't _actually_ currently attempt concurrent module installs and so this
addition is primarily for robustness against incorrect use.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
This is a new implementation of go-getter's "Getter" interface intended
to support installing OpenTofu module packages from OCI Distribution
repositories.
Since this is currently only intended for OpenTofu's use it makes some
simplifying assumptions that would not be acceptable for an upstream
getter, but are okay for the limited way that OpenTofu's module installer
uses go-getter, which is already intentionally constrained and hidden
behind a simpler API so we can treat go-getter as purely an implementation
detail.
This commit only introduces the getter, without actually registering it
as available for use in the module package fetcher used by "tofu init",
and so this is effectively just a bunch of dead code. A later commit will
wire this in properly and introduce an end-to-end test to demonstrate that
it's properly integrated.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
This continues our work to follow the dependency inversion style for the
"package fetcher" component of the module installer.
Mimicking the existing pattern for providers, package main is now
responsible for instantiating the PackageFetcher and providing it to
the "command" package as a field of command.Meta.
We could potentially go further here and follow dependency inversion style
for _all_ of the special clients needed by the various go-getter getters,
but our primary concern for now is preparing to add a new "getter" for
installation from an OCI Distribution repository, and so we'll leave the
other already-working code unchanged to reduce the risk of this initial
work.
Future commits will actually wire in the implementation details for OCI
Repository access. This commit focuses only on plumbing the necessary
objects through the API layers.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
* Rename module name from "github.com/hashicorp/terraform" to "github.com/placeholderplaceholderplaceholder/opentf".
Signed-off-by: Jakub Martin <kubam@spacelift.io>
* Gofmt.
Signed-off-by: Jakub Martin <kubam@spacelift.io>
* Regenerate protobuf.
Signed-off-by: Jakub Martin <kubam@spacelift.io>
* Fix comments.
Signed-off-by: Jakub Martin <kubam@spacelift.io>
* Undo issue and pull request link changes.
Signed-off-by: Jakub Martin <kubam@spacelift.io>
* Undo comment changes.
Signed-off-by: Jakub Martin <kubam@spacelift.io>
* Fix comment.
Signed-off-by: Jakub Martin <kubam@spacelift.io>
* Undo some link changes.
Signed-off-by: Jakub Martin <kubam@spacelift.io>
* make generate && make protobuf
Signed-off-by: Jakub Martin <kubam@spacelift.io>
---------
Signed-off-by: Jakub Martin <kubam@spacelift.io>
There was an unintended regression in go-getter v1.5.9's GitGetter which
caused us to temporarily fork that particular getter into Terraform to
expedite a fix. However, upstream v1.5.10 now includes a
functionally-equivalent fix and so we can heal that fork by upgrading.
We'd also neglected to update the Module Sources docs when upgrading to
go-getter v1.5.9 originally and so we were missing documentation about the
new "depth" argument to enable shadow cloning, which I've added
retroactively here along with documenting its restriction of only
supporting named refs.
This new go-getter release also introduces a new credentials-passing
method for the Google Cloud Storage getter, and so we must incorporate
that into the Terraform-level documentation about module sources.
Earlier versions of this code allowed "ref" to take any value that would
be accepted by "git checkout" as a valid target of a symbolic ref. We
inadvertently accepted a breaking change to upstream go-getter that broke
that as part of introducing a shallow clone optimization, because shallow
clone requires selecting a single branch.
To restore the previous capabilities while retaining the "depth" argument,
here we accept a compromise where "ref" has the stronger requirement of
being a valid named ref in the remote repository if and only if "depth"
is set to a value greater than zero. If depth isn't set or is less than
one, we will do the old behavior of just cloning all of the refs in the
remote repository in full and then switching to refer to the selected
branch, tag, or naked commit ID as a separate step.
This includes a heuristic to generate an additional error message hint if
we get an error from "git clone" and it looks like the user might've been
trying to use "depth" and "ref=COMMIT" together. We can't recognize that
error accurately because it's only reported as human-oriented git command
output, but this heuristic should hopefully minimize situations where we
show it inappropriately.
For now this is a change in the Terraform repository directly, so that we
can expedite the fix to an already-reported regression. After this is
released I tend to also submit a similar set of changes to upstream
go-getter, at which point we can revert Terraform to using the upstream
getter.GitGetter instead of our own local fork.
This is a pragmatic temporary solution to allow us to more quickly resolve
an upstream regression in go-getter locally within Terraform, so that the
work to upstream it for other callers can happen asynchronously and with
less time pressure.
This commit doesn't yet include any changes to address the bug, and
instead aims to be functionally equivalent to getter.GitGetter. A
subsequent commit will then address the regression, so that the diff of
that commit will be easier to apply later to the upstream to get the same
effect there.
Earlier work to make "terraform init" interruptible made the getproviders
package context-aware in order to allow provider installation to be cancelled.
Here we make a similar change for module installation, which is now also
cancellable with SIGINT. This involves plumbing context through initwd and
getmodules. Functions which can make network requests now include a context
parameter whose cancellation cancels those requests.
Since the module installation code is shared, "terraform get" is now
also interruptible during module installation.
It's been a long while since we gave close attention to the codepaths for
module source address parsing and external module package installation.
Due to their age, these codepaths often diverged from our modern practices
such as representing address types in the addrs package, and encapsulating
package installation details only in a particular location.
In particular, this refactor makes source address parsing a separate step
from module installation, which therefore makes the result of that parsing
available to other Terraform subsystems which work with the configuration
representation objects.
This also presented the opportunity to better encapsulate our use of
go-getter into a new package "getmodules" (echoing "getproviders"), which
is intended to be the only part of Terraform that directly interacts with
go-getter.
This is largely just a refactor of the existing functionality into a new
code organization, but there is one notable change in behavior here: the
source address parsing now happens during configuration loading rather
than module installation, which may cause errors about invalid addresses
to be returned in different situations than before. That counts as
backward compatible because we only promise to remain compatible with
configurations that are _valid_, which means that they can be initialized,
planned, and applied without any errors. This doesn't introduce any new
error cases, and instead just makes a pre-existing error case be detected
earlier.
Our module registry client is still using its own special module address
type from registry/regsrc for now, with a small shim from the new
addrs.ModuleSourceRegistry type. Hopefully in a later commit we'll also
rework the registry client to work with the new address type, but this
commit is already big enough as it is.
This new package aims to encapsulate all of our interactions with
go-getter to fetch remote module packages, to ensure that the rest of
Terraform will only use the small subset of go-getter functionality that
our modern module installer uses.
In older versions of Terraform, go-getter was the entire implementation
of module installation, but along the way we found that several aspects of
its design are poor fit for Terraform's needs, and so now we're using it
as just an implementation detail of Terraform's handling of remote module
packages only, hiding it behind this wrapper API which exposes only
the services that our module installer needs.
This new package isn't actually used yet, but in a later commit we will
change all of the other callers to go-getter to only work indirectly
through this package, so that this will be the only package that actually
imports the go-getter packages.
