[Dwarf-discuss] Enhancement: DWARF Extension Registry

[Cary Coutant]

Added as new issue 231110.1 <https://dwarfstd.org/issues/231110.1.html>.

-cary


On Fri, Nov 10, 2023 at 1:52 PM Ben Woodard <woodard@redhat.com> wrote:

> DWARF Extension Registry
> Background
>
> The DWARF standard has always had the wisdom to acknowledge the need for
> Vendor Extensibility. Section 1.3.13 describes this policy. For the
> producers it explicitly reserves some of the valid values for encoding
> various constructs and promises not to use those values in future versions
> of the standard. Consumers are given the freedom and expected to skip over
> data that they do not recognize.
>
> The original intent of these vendor extensions was that they would be a
> private agreement between a particular producer and a cooperating consumer.
> However, the range of tools expanded beyond a toolchain provided by a
> single vendor and so the concept of a private agreement between a specific
> producer and a cooperating consumer came to be regarded as a registry of
> vendor extensions that every tool must be aware of for the sake of
> compatibility.
>
> Because compatibility between tools was prioritized, the extensions in
> these de facto registries were effectively considered permanent unofficial
> DWARF encodings. This presumptive permanence is in spite of the fact that
> in some cases:
>
>    -
>
>    Producers that made use of these extensions were never implemented or
>    publicly released.
>    -
>
>    Producers that made use of these extensions are so obsolete that not
>    only does the tool chain no longer exist but the hardware that this
>    toolchain was designed for no longer exists except as historical artifacts.
>    -
>
>    Official versions of DWARF encodings that accomplish the same thing
>    have been added to the standard making the vendor extension obsolete.
>
>
> The accumulation of these vendor extensions over the years has led to
> problems with some DWARF constructs where they are running out of available
> encodings that do not overlap with the ones used by or reserved by the
> standard. In particular the range of vendor vendor extension encoding is
> running particularly short in the range of DW_OP encoding. This has led to
> several informal proposals within the vendor community along the lines of
> an official proposal 230324.1 Expression Operation Vendor Extensibility
> Opcode https://dwarfstd.org/issues/230324.1.html The basic idea in most
> of these proposals is to extend the range of available vendor extension
> encodings. While DWARF expression operations, Section 2.5, has the most
> extreme shortage of vendor encodings, other DWARF constructs are also
> running out of usable encodings.
> Overview
>
> This proposal is an alternative to 230324.1 and similar informal
> approaches to increase the available encoding space. It does this in two
> ways.
>
> First it makes it explicit that Vendor Extensions can only be interpreted
> in the context of a particular producer and a DWARF standard version. This
> way obsolete vendor extensions can be retired when a new version of the
> standard is released. It also begins the process of replacing the
> expectation for consumers that they can simply ignore encodings that they
> do not understand with the expectation they will need to have producer and
> version awareness when interpreting encodings. This is because tools chains
> are no longer developed by a close knit group of developers who all work
> for a single vendor. Tools are now developed by diverse distributed teams
> and users need and rightly expect interoperability.
>
> Because the bulk of tool development is no longer done by vendors who
> build a vertically integrated tool chain but rather is done by looser
> confederation of projects that need to cooperate to achieve compatibility,
> the term “vendor” is replaced with more generic terms when referring to
> these extensions.
>
> The second big change is it seeks to foster compatibility and
> collaboration by taking the collection of informal registries such as
> https://sourceware.org/elfutils/DwarfExtensions and
> https://llvm.org/docs/SourceLevelDebugging.html#new-dwarf-tags and
> collects them and brings them under the aegis of the DWARF standards body.
> To facilitate innovation and quickly meet producer’s and consumer’s needs
> the expectation is that registering a new producer specific construct and
> encoding would be a simple, light weight or even partially automated task.
> It would not be at all like amending the standard. The registry would be
> published on the DWARF standard’s web site so that consumers can easily
> find it. Each extension would be listed along with the producers that emit
> it and the range of DWARF standard versions that it applies to. When new
> DWARF standard versions are released, each extension will be reconsidered
> to see if it continues to add utility in the context of the new version of
> the DWARF standard. Extensions that persist over several versions of the
> DWARF standard probably should be considered for inclusion into the
> official standard.
> Proposed Changes Section 1.3.13
>
> Section 1.3.13 currently named “Vendor Extensibility” shall be renamed to
> “Extensibility” to reflect the fact that cooperating tools using these
> extensions are no longer being developed by vendors who control the entire
> tool chain. Likewise the following paragraphs should change to:
>
> This document does not attempt to cover all interesting languages or even
> to cover all of the possible debugging information needs for its primary
> target languages. Therefore, the document provides tool developers a way to
> define their own debugging information tags, attributes, base type
> encodings, location operations, language names, calling conventions and
> call frame instructions by reserving a subset of the valid values for these
> constructs for tool developer specific additions and defining related
> naming conventions. Producers may also use debugging information entries
> and attributes defined here in new situations. Future versions of this
> document will not use names or values reserved for these specific additions
> but the meanings of these encoding may change from DWARF version to
> version. Therefore the values of these extensions must be interpreted in
> the context of the DWARF standard versions for which they apply. All names
> and values not explicitly reserved for producer specific additions,
> however, are reserved for future versions of this document.
>
> Where this specification provides a means for describing the source
> language, implementers are expected to adhere to that specification. For
> language features that are not supported, implementers may use existing
> attributes in novel ways or add producer-defined attributes.
>
> Implementers who make extensions are strongly encouraged to design them to
> be compatible with this specification in the absence of those extensions.
> Implementers should also register them at <url> so that they can be
> included in the database of known extensions which can be found at <url>.
> This will allow consumers that they are not directly cooperating with to be
> aware of their extensions and know how to interpret them.
>
> While the DWARF format is organized so that a consumer can skip over data
> which it does not recognize and this may allow a consumer to read and
> process files generated according to a later version of this standard or
> which contain producer specific extensions, albeit possibly in a degraded
> manner, this has been found to reduce the tool compatibility which users
> have come to expect. Consumers should therefore refer to <url> when
> encountering a producer specific construct that they are not familiar with.
> Section 6.1.1.2 Structure of the Name Index
>
> The paragraph:
>
> A producer may define additional vendor-specific attributes, and a
> consumer will be able to ignore and skip over any attributes it is not
> prepared to handle.
>
> Shall be replaced by:
>
> A producer may define additional producer-specific attributes, and a
> consumer should be able to ignore and skip over any attributes it is not
> prepared to handle. However, to achieve full compatibility a consumer
> should refer to <url> where all known producer specific attributes that
> apply to the Name Index are enumerated.
> Section 6.2.4 The Line Number Program Header
>
> In the definition of “opcode base”, the phrase “vendor-specific
> extensions” should be replaced with “producer-specific extensions” in all
> three places where it is used.
> Section 6.2.4.2 Vendor-defined Content Descriptions
>
> The name of the section should be changed to “Producer-defined Content
> Descriptions” and the first word of the first sentence should also be
> changed to match.
>
> The non-normative text following the section should be changed to:
>
> If a consumer encounters a producer-defined content type that it does not
> understand, it should skip the content data as though it were not present.
> However, to achieve full compatibility the consumer’s developer should
> refer to <url> where all known producer-defined line number content
> descriptors are enumerated.
> Section 6.3.1 Macro Information Header
>
> In the second paragraph of item 4 the phrase “Vendor extension entry
> opcodes” should be replaced with “Producer specific entry opcodes”.
> Section 7.1 Vendor Extensibility
>
> The name of the section should be changed from “Vendor Extensibility” to
> “Extensibility”.
>
> Throughout section 7.1 the phrase “vendor specific” should be replaced
> with “producer specific”.
>
> In the second paragraph, the sentence beginning with “Vendors may use
> values” should be replaced with “Producers may use values”.
>
> The paragraph and list of points at the end of the section should be
> replaced with:
>
> Producer defined tags, attributes, base type encodings, location atoms,
> language names, line number actions, calling conventions and call frame
> instructions, historically have used the form prefix_vendor_id_name, where
> vendor_id is some identifying character sequence chosen so as to avoid
> conflicts with other vendors. This established convention can continue in
> the cases where there is historical precedent or where there is a vendor
> but it can also be the name of the producer's project when there is no
> specific vendor.
>
> To ensure that extensions added by one producer may be safely ignored by
> consumers that do not understand those extensions, the following rules must
> be followed:
>
>    1.
>
>    Producer defined extensions must be evaluated in the context of a
>    DWARF standard version.
>    2.
>
>    New attributes are added in such a way that a debugger may recognize
>    the format of a new attribute value without knowing the content of that
>    attribute value.
>    3.
>
>    The semantics of any new attributes do not alter the semantics of
>    previously existing attributes.
>    4.
>
>    The semantics of any new tags do not conflict with the semantics of
>    previously existing tags.
>    5.
>
>    New forms of attribute value are not added.
>
> To ensure full tool intercompatibility, producer defined DWARF extensions
> should be registered with the DWARF standards body at <URL>.
> Section 7.32 Type Signature Computation
>
> In the 3rd paragraph of point 4 replace “vendor-specific attributes” with
> “producer-specific attributes”.
> Appendix A
>
> In the first paragraph, replace “and new, vendor-defined ones” with “and
> producer defined ones”
> Appendix D.7.3
>
> In the text between the examples replace “vendor-specific” with
> “producer-specific” in both cases.
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.dwarfstd.org/pipermail/dwarf-discuss/attachments/20231124/232ec607/attachment-0001.htm>