[Dwarf-discuss] Enhancement: DWARF Extension Registry

[Ben Woodard]

DWARF Extension Registry


  Background

The DWARF standard has always had the wisdom to acknowledge the need for 
Vendor Extensibility. Section 1.3.13 describes this policy. For the 
producers it explicitly reserves some of the valid values for encoding 
various constructs and promises not to use those values in future 
versions of the standard. Consumers are given the freedom and expected 
to skip over data that they do not recognize.

The original intent of these vendor extensions was that they would be a 
private agreement between a particular producer and a cooperating 
consumer. However, the range of tools expanded beyond a toolchain 
provided by a single vendor and so the concept of a private agreement 
between a specific producer and a cooperating consumer came to be 
regarded as a registry of vendor extensions that every tool must be 
aware of for the sake of compatibility.

Because compatibility between tools was prioritized, the extensions in 
these de facto registries were effectively considered permanent 
unofficial DWARF encodings. This presumptive permanence is in spite of 
the fact that in some cases:

  *

    Producers that made use of these extensions were never implemented
    or publicly released.

  *

    Producers that made use of these extensions are so obsolete that not
    only does the tool chain no longer exist but the hardware that this
    toolchain was designed for no longer exists except as historical
    artifacts.

  *

    Official versions of DWARF encodings that accomplish the same thing
    have been added to the standard making the vendor extension obsolete.


The accumulation of these vendor extensions over the years has led to 
problems with some DWARF constructs where they are running out of 
available encodings that do not overlap with the ones used by or 
reserved by the standard. In particular the range of vendor vendor 
extension encoding is running particularly short in the range of DW_OP 
encoding. This has led to several informal proposals within the vendor 
community along the lines of an official proposal 230324.1 Expression 
Operation Vendor Extensibility Opcode 
https://dwarfstd.org/issues/230324.1.html 
<https://dwarfstd.org/issues/230324.1.html>The basic idea in most of 
these proposals is to extend the range of available vendor extension 
encodings. While DWARF expression operations, Section 2.5, has the most 
extreme shortage of vendor encodings, other DWARF constructs are also 
running out of usable encodings.


  Overview

This proposal is an alternative to 230324.1 and similar informal 
approaches to increase the available encoding space. It does this in two 
ways.

First it makes it explicit that Vendor Extensions can only be 
interpreted in the context of a particular producer and a DWARF standard 
version. This way obsolete vendor extensions can be retired when a new 
version of the standard is released. It also begins the process of 
replacing the expectation for consumers that they can simply ignore 
encodings that they do not understand with the expectation they will 
need to have producer and version awareness when interpreting encodings. 
This is because tools chains are no longer developed by a close knit 
group of developers who all work for a single vendor. Tools are now 
developed by diverse distributed teams and users need and rightly expect 
interoperability.

Because the bulk of tool development is no longer done by vendors who 
build a vertically integrated tool chain but rather is done by looser 
confederation of projects that need to cooperate to achieve 
compatibility, the term “vendor” is replaced with more generic terms 
when referring to these extensions.

The second big change is it seeks to foster compatibility and 
collaboration by taking the collection of informal registries such as 
https://sourceware.org/elfutils/DwarfExtensions 
<https://sourceware.org/elfutils/DwarfExtensions>and 
https://llvm.org/docs/SourceLevelDebugging.html#new-dwarf-tags 
<https://llvm.org/docs/SourceLevelDebugging.html#new-dwarf-tags>and 
collects them and brings them under the aegis of the DWARF standards 
body. To facilitate innovation and quickly meet producer’s and 
consumer’s needs the expectation is that registering a new producer 
specific construct and encoding would be a simple, light weight or even 
partially automated task. It would not be at all like amending the 
standard. The registry would be published on the DWARF standard’s web 
site so that consumers can easily find it. Each extension would be 
listed along with the producers that emit it and the range of DWARF 
standard versions that it applies to. When new DWARF standard versions 
are released, each extension will be reconsidered to see if it continues 
to add utility in the context of the new version of the DWARF standard. 
Extensions that persist over several versions of the DWARF standard 
probably should be considered for inclusion into the official standard.


  Proposed Changes


    Section 1.3.13

Section 1.3.13 currently named “Vendor Extensibility” shall be renamed 
to “Extensibility” to reflect the fact that cooperating tools using 
these extensions are no longer being developed by vendors who control 
the entire tool chain. Likewise the following paragraphs should change to:

This document does not attempt to cover all interesting languages or 
even to cover all of the possible debugging information needs for its 
primary target languages. Therefore, the document provides tool 
developers a way to define their own debugging information tags, 
attributes, base type encodings, location operations, language names, 
calling conventions and call frame instructions by reserving a subset of 
the valid values for these constructs for tool developer specific 
additions and defining related naming conventions. Producers may also 
use debugging information entries and attributes defined here in new 
situations. Future versions of this document will not use names or 
values reserved for these specific additions but the meanings of these 
encoding may change from DWARF version to version. Therefore the values 
of these extensions must be interpreted in the context of the DWARF 
standard versions for which they apply. All names and values not 
explicitly reserved for producer specific additions, however, are 
reserved for future versions of this document.

Where this specification provides a means for describing the source 
language, implementers are expected to adhere to that specification. For 
language features that are not supported, implementers may use existing 
attributes in novel ways or add producer-defined attributes.

Implementers who make extensions are strongly encouraged to design them 
to be compatible with this specification in the absence of those 
extensions. Implementers should also register them at <url> so that they 
can be included in the database of known extensions which can be found 
at <url>. This will allow consumers that they are not directly 
cooperating with to be aware of their extensions and know how to 
interpret them.

While the DWARF format is organized so that a consumer can skip over 
data which it does not recognize and this may allow a consumer to read 
and process files generated according to a later version of this 
standard or which contain producer specific extensions, albeit possibly 
in a degraded manner, this has been found to reduce the tool 
compatibility which users have come to expect. Consumers should 
therefore refer to <url> when encountering a producer specific construct 
that they are not familiar with.


    Section 6.1.1.2 Structure of the Name Index

The paragraph:

A producer may define additional vendor-specific attributes, and a 
consumer will be able to ignore and skip over any attributes it is not 
prepared to handle.

Shall be replaced by:

A producer may define additional producer-specific attributes, and a 
consumer should be able to ignore and skip over any attributes it is not 
prepared to handle. However, to achieve full compatibility a consumer 
should refer to <url> where all known producer specific attributes that 
apply to the Name Index are enumerated.


    Section 6.2.4 The Line Number Program Header

In the definition of “opcode base”, the phrase “vendor-specific 
extensions” should be replaced with “producer-specific extensions” in 
all three places where it is used.


      Section 6.2.4.2 Vendor-defined Content Descriptions

The name of the section should be changed to “Producer-defined Content 
Descriptions” and the first word of the first sentence should also be 
changed to match.

The non-normative text following the section should be changed to:

If a consumer encounters a producer-defined content type that it does 
not understand, it should skip the content data as though it were not 
present. However, to achieve full compatibility the consumer’s developer 
should refer to <url> where all known producer-defined line number 
content descriptors are enumerated.


    Section 6.3.1 Macro Information Header

In the second paragraph of item 4 the phrase “Vendor extension entry 
opcodes” should be replaced with “Producer specific entry opcodes”.


    Section 7.1 Vendor Extensibility

The name of the section should be changed from “Vendor Extensibility” to 
“Extensibility”.

Throughout section 7.1 the phrase “vendor specific” should be replaced 
with “producer specific”.

In the second paragraph, the sentence beginning with “Vendors may use 
values” should be replaced with “Producers may use values”.

The paragraph and list of points at the end of the section should be 
replaced with:

Producer defined tags, attributes, base type encodings, location atoms, 
language names, line number actions, calling conventions and call frame 
instructions, historically have used the form prefix_vendor_id_name, 
where vendor_id is some identifying character sequence chosen so as to 
avoid conflicts with other vendors. This established convention can 
continue in the cases where there is historical precedent or where there 
is a vendor but it can also be the name of the producer's project when 
there is no specific vendor.

To ensure that extensions added by one producer may be safely ignored by 
consumers that do not understand those extensions, the following rules 
must be followed:

 1.

    Producer defined extensions must be evaluated in the context of a
    DWARF standard version.

 2.

    New attributes are added in such a way that a debugger may recognize
    the format of a new attribute value without knowing the content of
    that attribute value.

 3.

    The semantics of any new attributes do not alter the semantics of
    previously existing attributes.

 4.

    The semantics of any new tags do not conflict with the semantics of
    previously existing tags.

 5.

    New forms of attribute value are not added.

To ensure full tool intercompatibility, producer defined DWARF 
extensions should be registered with the DWARF standards body at <URL>.


    Section 7.32 Type Signature Computation

In the 3rd paragraph of point 4 replace “vendor-specific attributes” 
with “producer-specific attributes”.


    Appendix A

In the first paragraph, replace “and new, vendor-defined ones” with “and 
producer defined ones”


    Appendix D.7.3

In the text between the examples replace “vendor-specific” with 
“producer-specific” in both cases.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.dwarfstd.org/pipermail/dwarf-discuss/attachments/20231110/9d9b8a12/attachment-0001.htm>