OpenTelemetry has various Go packages split across several Go modules that
often need to be carefully upgraded together. And in particular, we are
using the "semconv" package in conjunction with the OpenTelemetry SDK's
"resource" package in a way that requires that they both agree on which
version of the OpenTelemetry Semantic Conventions are being followed.
To help avoid "dependency hell" situations when upgrading, this centralizes
all of our direct calls into the OpenTelemetry SDK and tracing API into
packages under internal/tracing, by exposing a few thin wrapper functions
that other packages can use to access the same functionality indirectly.
We only use a relatively small subset of the OpenTelemetry library surface
area, so we don't need too many of these reexports and they should not
represent a significant additional maintenance burden.
For the semconv and resource interaction in particular this also factors
that out into a separate helper function with a unit test, so we should
notice quickly whenever they become misaligned. This complements the
end-to-end test previously added in opentofu/opentofu#3447 to give us
faster feedback about this particular problem, while the end-to-end test
has the broader scope of making sure there aren't any errors at all when
initializing OpenTelemetry tracing.
Finally, this also replaces the constants we previously had in package
traceaddrs with functions that return attribute.KeyValue values directly.
This matches the API style used by the OpenTelemetry semconv packages, and
makes the calls to these helpers from elsewhere in the system a little
more concise.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
This completes some of the missing connections for contexts in the provider
source codepaths by introducing context.Context parameters and wiring them
through so we can eliminate a few more context.TODO() placeholders.
For consistency's sake this adds context.Context to all four of the
getproviders.Source implementations that directly interact with stuff
outside of OpenTofu (network services or filesystem), even though not
all of them currently make use of it, just because interactions with
outside stuff tends to encourage cross-cutting concerns like logging and
tracing and so this ensures we have contexts propagated in there for such
future uses.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
This adds more detailed OTel trace spans to our various different
interactions with OCI repositories, which is helpful to understand the
time spent in each of the various sequential steps involved in resolving
an OCI artifact.
OTel's centrally-maintained conventions for attribute names currently only
have a standard for reporting a manifest digest, so we'll use that where
it's appropriate but use our own "opentofu.oci."-prefixed attribute names
for everything else for now. Hopefully the upstream standard will be
broadened later to include some additional concepts, at which point we
can switch over to the standardized attribute names.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
Because the CLI Configuration allows specifying multiple provider
installation methods, we use getproviders.MultiSource to aggregate together
all of their results for a particular provider and then select the newest
available version from across all of the methods.
Different sources might offer a different subset of providers, and so
MultiSource ignores getproviders.ErrProviderNotFound errors from
Source.AvailableVersions unless _all_ sources return that error, which
therefore avoids the need for the operator to laboriously configure
include/exclude rules to constrain which real sources get queried for
each provider.
The OCIRegistryMirrorSource implements AvailableVersions by asking the
repository to list the tags it has available, and therefore this
source's implementation of the "provider not found" signal is based on
the registry returning a "not found" error from that specific request.
However, we were previously only handling the error code that ORAS-Go uses
for its local repository implementations, like the in-memory store or the
OCI layout store. For remote registries using the OCI Distribution protocol
ORAS-Go uses a different error type that is able to give more detail about
the server's response, and so we also need to treat some instances of
_that_ error type as representing "provider not found".
The OCI Distribution protocol does not require that a registry report a
JSON representation of the details of an error, but defines an optional
means of doing so that can give more detail than just the HTTP error
code. Therefore we use the OCI Distribution-specific error details if
possible, but fall back on just using the HTTP status code for servers
that choose not to return machine-readable error details. This compromise
is intended to allow us to report errors _other than_ "provider not found"
explicitly when possible, but to err on the side of allowing other sources
a chance to succeed if the situation is ambiguous. The ORAS-Go library
generates log lines for each request in our output to help with detailed
debugging in case we misclassify an error, but we'd prefer to give the
operator direct feedback about a configuration error if possible.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
The initial design of this attempted to maximize our flexibility to make
backward-compatible changes to the manifest structure for providers in
future OpenTofu releases, by expecting the "layers" entry describing the
provider package to include a specific artifactType and ignoring any
layers with different artifactType values.
However, general-purpose tools for constructing image manifests don't tend
to allow setting the artifactType in a layer descriptor, so we'll drop
that requirement as a measure of pragmatism.
This implies that the only possible extension we could make for _layers
in particular_ in a future release is to support an additional archive
format that would presumably then be used in much the same way as our
current archive/zip usage: by extracting the archive into the target
directory. This ability to introduce new formats is the most likely future
evolution we identified while designing this, with all other evolutions
being more speculative/theoretical.
This does still retain various other extension points for future additions,
including but not limited to:
- Introducing an alternative artifactType for the image manifest _itself_,
rather than for the layers within it, and then having the later version
of OpenTofu prefer to choose a manifest with the newer artifactType while
still supporting the old one as a fallback.
- Using the "config" property of the manifest to introduce arbitrary
additional metadata that future versions might need, and using the
config descriptor's own mediaType to recognize when those new additions
are present.
Therefore this seems like a reasonable compromise to make it easier to
assemble OCI manifests for OpenTofu provider packages using general-purpose
tools like ORAS CLI, rather than requiring OpenTofu-specific tools.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
It's now valid to include an oci_mirror block in the provider_installation
block in the CLI configuration, specifying that OpenTofu should try to
install providers from OCI repositories based on a template that maps
from OpenTofu-style provider source addresses into OCI repository
addresses.
The getproviders.Source implementation for this was added in a previous
commit, so this is mainly just wiring it up to the cliconfig layer and
the dependency wiring code in package main.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
In practical testing I've found that some OCI registry implementations
don't return the artifactType field in their response to a tag resolution
request.
Unfortunately that behavior is ambiguous with what happens when a server
that _does_ support the artifact type describes a manifest that lacks one,
as is true for traditional container images and Helm chart artifacts.
Therefore we must unfortunately sacrifice our specialized error message
that tried to hint about accidentally selecting a container image, and
in that case we'll now return a more generic error about the manifest
having the wrong media type because that's the best we can do with the
information that an OCI registry is _required_ to return from tag
resolution.
Perhaps we'll try to improve on this in a different way later if we find
that folks are often getting confused by this, but for now we'll accept the
slightly-worse error message in this hopefully-rare case so we can
prioritize getting the happy path finished and shipped.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
This is a new implementation of getproviders.Source that has a similar
purpose to HTTPMirrorSource but uses the OCI Distribution protocol instead
of the OpenTofu-specific Provider Mirror Protocol.
It translates OpenTofu-style provider source addresses into OCI repository
addresses using an externally-defined rule and then discovers provider
package metadata using the tags and associated manifests in the selected
repository.
Signed-off-by: Martin Atkins <mart@degeneration.co.uk>
