[Dwarf-discuss] Enhancement: DWARF Extension Registry
Ben Woodard
woodard@redhat.com
Fri Dec 1 13:24:30 GMT 2023
Sorry that it has taken me so long to get back around to this. Holidays
vacation and other work.
On 11/15/23 11:19, Ron Brender wrote:
> Ben,
>
> This is a good first proposal for a DWARF Extension Registry.
> Following are a few initial reactions...
>
> 1. I don't understand why this proposal is an alternative to 230324.1.
> That proposal defines a way to get more numbers. This
> proposal describes how to manage use of those numbers. Seems like both
> proposals are necessary.
That is a fair point. I can see it both ways.
My reasoning is that the reason why we are running out of vendor defined
space is that within in the various vendor spaces the encoding space is
consumed by legacy extensions that:
1) were never implemented publicly
2) were implemented but are no longer in use because the compilers that
generated them have been abandoned
3) were in use but have been incorporated into the standard version of
DWARF.
I feel like clearing those out by drawing a line in the sand and saying
that extensions which existed in previous versions of DWARF do not
necessarily mean the same thing once the new version of DWARF is
released, should clear out the legacy cruft such that there should be
sufficient encoding space for new producer extensions.
I'm personally not a big fan of 230324.1 or at best I'm kind of
ambivalent toward it.
1) As long as they have at least one number left in the encoding space
they can do this already. They just take the last value and define it as
an extension operator. I don't see why this must be in the standard itself.
2) I do not like the idea of having a practically infinite extension
space. I will admit that it is not the strongest argument but I believe
that having a limited producer encoding space encourages the authors of
producers to work through the standardization process and standardize
their extension rather than just accumulating them in the producer
defined encoding space.
3) I also feel like intentionally requiring the producer defined
extensions to be reconsidered with every major version release of the
standard helps keep the standardization process moving along.
>
> 2. The proposed text seems to randomly replace "vender-defined" with
> "producer-defined", "producer defined" or "producer specific"
> (curiously omitting "producer-specific" :-). Use one rendition
> consistently, which I think should be "producer-defined".
I will fix this and submit a revised version.
>
> 3. Whatever URLs are part of the registration process should be
> mentioned in only one place. Section 7.1 is an appropriate place.
> References to 7.1 can be included elsewhere if really needed.
I will fix this in a revised version.
>
> 4. Re 1.3.13. I can't tell if you actually changed anything in the
> first two paragraphs.
I'll try to make the change more obvious in the next version.
>
> 5. Re 1.3.13: Blaming "reduced tool compatibility" on the
> skipability of unknown constructs seems a huge and unjustified claim.
> I don't buy it.
In most cases, I can see your point but in this case I cannot. To me
this seems obvious.
Of course a consumer should be able to skip over something that it
doesn't understand. However, if a consumer wants to achieve full
compatibility with the producer, it needs to be able to interpret those
constructs. It can't skip over them. That seems unquestionably obvious
to me. So that the consumer's author can write the code that allows the
consumer to interpret these producer specific constructs, the text
refers the consumer to the registry.
>
> 6. Re 6.1.1.2 and 6.2.4.2 <http://6.2.4.2>: Why single these out? This
> is no different than any other producer-defined extension and should
> be covered adequately by the general discussion in 7.1
There was specific text in those sections which seemed to need to be
changed.
>
> 7. Re 7.1: There seems to be text missing between "where there is a
> vendor" and "but it can also".
>
> By tieing a producer-defined extension to a particular version of
> DWARF, does that mean that when a new version of DWARF comes out that
> producer must re-register the extension to keep "ownership" of the code?
I would say that as part of the new DWARF version release process,
asking the producer developers who have registered extensions if those
extensions are still meaningful and useful in the context of the new
DWARF version.
If so, then it gets carried over. However, If some particular extension
continues to be useful across many versions of DWARF and producers are
still actively using it, then that extension should be considered for
standardization.
If the extension has been standardized as part of the new DWARF version
then that producer specific encoding can be reused in the context of the
new DWARF version. In other words, producers should prefer the
standardized way of expressing a concept over a producer specific way.
> Anything short of permanent registration seem contrary to our general
> backward compatible mindset. I think associating an extension with a
> DWARF version is probably not a good idea.
Backward compatibility is a worthy goal but if no producer is going to
generate that producer specific extension in the context of that DWARF
version, then it just consumes encoding space. Compatibility between
tools and specifically backward compatibility needs to be more than a
theoretical idea. There needs to be an extant producer that will
generate DWARF with those extensions that can the consumer can be tested
against. When you look at:
https://sourceware.org/elfutils/DwarfExtensions There are far too many
things like:
DW_TAG_MIPS_loop 0x4081 Never implemented, mips_extensions
Nothing Constant only
DW_AT_sf_names 0x2101 DWARF1 only? Unknown Constant only
DW_AT_src_info 0x2102 DWARF1 only? Unknown Constant only
DW_AT_mac_info 0x2103 DWARF1 only? Unknown Constant only
DW_AT_src_coords 0x2104 DWARF1 only? Unknown Constant only
DW_AT_body_begin 0x2105 DWARF1 only? Unknown Constant only
DW_AT_body_end 0x2106 DWARF1 only? Unknown Constant only
We don't know what it does, we don't know of any producer which
generates it. No consumer can do anything with it. It is just taking up
space. No producer is going to emit those in the context of DWARF6,
let's reuse the encodings.
And there are things like:
DW_AT_GNU_guarded_by 0x2108 GNU ThreadSafetyAnnotations Not
implemented Constant only
DW_AT_GNU_pt_guarded_by 0x2109 GNU ThreadSafetyAnnotations
Not implemented Constant only
DW_AT_GNU_guarded 0x210a GNU ThreadSafetyAnnotations Not
implemented Constant only
DW_AT_GNU_pt_guarded 0x210b GNU ThreadSafetyAnnotations Not
implemented Constant only
They never got implemented. No producer actually uses them, no consumer
knows what to do with them.
Then there are things like:
DW_AT_GNU_call_site_value 0x2111 GNU call site DWARF5 proposal
GCC Recognized in readelf
DW_AT_GNU_call_site_data_value 0x2112 GNU call site DWARF5
proposal GCC Recognized in readelf
DW_AT_GNU_call_site_target 0x2113 GNU call site DWARF5 proposal
GCC Recognized in readelf
DW_AT_GNU_call_site_target_clobbered 0x2114 GNU call site DWARF5
proposal GCC Recognized in readelf
Presumably, if I do gcc -gdwarf-4 then gcc will use these vendor
specific encodings rather than the DWARF5 standardized versions.
However, GCC should never emit those when it is emitting DWARF5 or
later. This seems like a legitimate case where a consumer may want to
implement backward compatibility. Those attribute encoding can be tied
to DWARF2-4 and then those encodings can be reused in the context of DWARF6.
>
> If extensions are registered, is there any remaining need for
> "vender_id"/"vender_id_name" in names. Uniqueness of codes is assured
> by the
> requirement to register, right? So instead of some kind of
> vendor-related text we could just use "ext_" to keep names separate
> from the standard stuff.
That is a good point. The registration process can log the producers
which make use of the extension.
>
> 8. This is perhaps enough of a proposal to describe changes to the
> DWARF document itself. But I would like to see the proposal include
> more about the registration contents proper. In particular, I think
> there needs to be a requirement that the registration include a
> complete specification of the extension. The goal is to allow
> (encourage) other consumers to accept the same extension if
> appropriate but also for other producers to adopt the same extension
> if suitable.
>
> In that connection, there probably needs to be some kind of statement
> that any and all information included in a registration must be free
> of any restrictions (patent, copyright, trademark, whatever) and
> freely reusable by others.
Sounds like a good idea to me. I just think that the registration
process should not be so onerous that it inhibits disclosure and
innovation. For example there was an idea behind:
DW_AT_GNU_guarded_by 0x2108 GNU ThreadSafetyAnnotations Not
implemented Constant only
DW_AT_GNU_pt_guarded_by 0x2109 GNU ThreadSafetyAnnotations
Not implemented Constant only
DW_AT_GNU_guarded 0x210a GNU ThreadSafetyAnnotations Not
implemented Constant only
DW_AT_GNU_pt_guarded 0x210b GNU ThreadSafetyAnnotations Not
implemented Constant only
DW_AT_GNU_locks_excluded 0x210c GNU ThreadSafetyAnnotations
Not implemented Constant only
DW_AT_GNU_exclusive_locks_required 0x210d GNU
ThreadSafetyAnnotations Not implemented Constant only
DW_AT_GNU_shared_locks_required 0x210e GNU
ThreadSafetyAnnotations Not implemented Constant only
It might have been a good one, it might have been a bad one; I don't
know I wasn't involved. Maybe someone even did a prototype
implementation that never was accepted into a mainline release. I think
it is still worthwhile to allow an eager compiler developer to register
those attributes do their prototype implementation in their own reserved
sandbox. If it catches on, then great; they can provide more information
so other tools can implement it. If the idea gets abandoned then the
encodings only exist for that release of the DWARF version and the
encodings can be reused in the next DWARF cycle.
>
> That is enough for now,
> Ron
>
>
>
>
>
>
>
> On Fri, Nov 10, 2023 at 4:52 PM Ben Woodard via Dwarf-discuss
> <dwarf-discuss@lists.dwarfstd.org> wrote:
>
> DWARF Extension Registry
>
>
> Background
>
> The DWARF standard has always had the wisdom to acknowledge the
> need for Vendor Extensibility. Section 1.3.13 describes this
> policy. For the producers it explicitly reserves some of the valid
> values for encoding various constructs and promises not to use
> those values in future versions of the standard. Consumers are
> given the freedom and expected to skip over data that they do not
> recognize.
>
> The original intent of these vendor extensions was that they would
> be a private agreement between a particular producer and a
> cooperating consumer. However, the range of tools expanded beyond
> a toolchain provided by a single vendor and so the concept of a
> private agreement between a specific producer and a cooperating
> consumer came to be regarded as a registry of vendor extensions
> that every tool must be aware of for the sake of compatibility.
>
> Because compatibility between tools was prioritized, the
> extensions in these de facto registries were effectively
> considered permanent unofficial DWARF encodings. This presumptive
> permanence is in spite of the fact that in some cases:
>
> *
>
> Producers that made use of these extensions were never
> implemented or publicly released.
>
> *
>
> Producers that made use of these extensions are so obsolete
> that not only does the tool chain no longer exist but the
> hardware that this toolchain was designed for no longer exists
> except as historical artifacts.
>
> *
>
> Official versions of DWARF encodings that accomplish the same
> thing have been added to the standard making the vendor
> extension obsolete.
>
>
> The accumulation of these vendor extensions over the years has led
> to problems with some DWARF constructs where they are running out
> of available encodings that do not overlap with the ones used by
> or reserved by the standard. In particular the range of vendor
> vendor extension encoding is running particularly short in the
> range of DW_OP encoding. This has led to several informal
> proposals within the vendor community along the lines of an
> official proposal 230324.1 Expression Operation Vendor
> Extensibility Opcode https://dwarfstd.org/issues/230324.1.html
> <https://dwarfstd.org/issues/230324.1.html>The basic idea in most
> of these proposals is to extend the range of available vendor
> extension encodings. While DWARF expression operations, Section
> 2.5, has the most extreme shortage of vendor encodings, other
> DWARF constructs are also running out of usable encodings.
>
>
> Overview
>
> This proposal is an alternative to 230324.1 and similar informal
> approaches to increase the available encoding space. It does this
> in two ways.
>
> First it makes it explicit that Vendor Extensions can only be
> interpreted in the context of a particular producer and a DWARF
> standard version. This way obsolete vendor extensions can be
> retired when a new version of the standard is released. It also
> begins the process of replacing the expectation for consumers that
> they can simply ignore encodings that they do not understand with
> the expectation they will need to have producer and version
> awareness when interpreting encodings. This is because tools
> chains are no longer developed by a close knit group of developers
> who all work for a single vendor. Tools are now developed by
> diverse distributed teams and users need and rightly expect
> interoperability.
>
> Because the bulk of tool development is no longer done by vendors
> who build a vertically integrated tool chain but rather is done by
> looser confederation of projects that need to cooperate to achieve
> compatibility, the term “vendor” is replaced with more generic
> terms when referring to these extensions.
>
> The second big change is it seeks to foster compatibility and
> collaboration by taking the collection of informal registries such
> as https://sourceware.org/elfutils/DwarfExtensions
> <https://sourceware.org/elfutils/DwarfExtensions>and
> https://llvm.org/docs/SourceLevelDebugging.html#new-dwarf-tags
> <https://llvm.org/docs/SourceLevelDebugging.html#new-dwarf-tags>and
> collects them and brings them under the aegis of the DWARF
> standards body. To facilitate innovation and quickly meet
> producer’s and consumer’s needs the expectation is that
> registering a new producer specific construct and encoding would
> be a simple, light weight or even partially automated task. It
> would not be at all like amending the standard. The registry would
> be published on the DWARF standard’s web site so that consumers
> can easily find it. Each extension would be listed along with the
> producers that emit it and the range of DWARF standard versions
> that it applies to. When new DWARF standard versions are released,
> each extension will be reconsidered to see if it continues to add
> utility in the context of the new version of the DWARF standard.
> Extensions that persist over several versions of the DWARF
> standard probably should be considered for inclusion into the
> official standard.
>
>
> Proposed Changes
>
>
> Section 1.3.13
>
> Section 1.3.13 currently named “Vendor Extensibility” shall be
> renamed to “Extensibility” to reflect the fact that cooperating
> tools using these extensions are no longer being developed by
> vendors who control the entire tool chain. Likewise the following
> paragraphs should change to:
>
> This document does not attempt to cover all interesting languages
> or even to cover all of the possible debugging information needs
> for its primary target languages. Therefore, the document provides
> tool developers a way to define their own debugging information
> tags, attributes, base type encodings, location operations,
> language names, calling conventions and call frame instructions by
> reserving a subset of the valid values for these constructs for
> tool developer specific additions and defining related naming
> conventions. Producers may also use debugging information entries
> and attributes defined here in new situations. Future versions of
> this document will not use names or values reserved for these
> specific additions but the meanings of these encoding may change
> from DWARF version to version. Therefore the values of these
> extensions must be interpreted in the context of the DWARF
> standard versions for which they apply. All names and values not
> explicitly reserved for producer specific additions, however, are
> reserved for future versions of this document.
>
> Where this specification provides a means for describing the
> source language, implementers are expected to adhere to that
> specification. For language features that are not supported,
> implementers may use existing attributes in novel ways or add
> producer-defined attributes.
>
> Implementers who make extensions are strongly encouraged to design
> them to be compatible with this specification in the absence of
> those extensions. Implementers should also register them at <url>
> so that they can be included in the database of known extensions
> which can be found at <url>. This will allow consumers that they
> are not directly cooperating with to be aware of their extensions
> and know how to interpret them.
>
> While the DWARF format is organized so that a consumer can skip
> over data which it does not recognize and this may allow a
> consumer to read and process files generated according to a later
> version of this standard or which contain producer specific
> extensions, albeit possibly in a degraded manner, this has been
> found to reduce the tool compatibility which users have come to
> expect. Consumers should therefore refer to <url> when
> encountering a producer specific construct that they are not
> familiar with.
>
>
> Section 6.1.1.2 Structure of the Name Index
>
> The paragraph:
>
> A producer may define additional vendor-specific attributes, and a
> consumer will be able to ignore and skip over any attributes it is
> not prepared to handle.
>
> Shall be replaced by:
>
> A producer may define additional producer-specific attributes, and
> a consumer should be able to ignore and skip over any attributes
> it is not prepared to handle. However, to achieve full
> compatibility a consumer should refer to <url> where all known
> producer specific attributes that apply to the Name Index are
> enumerated.
>
>
> Section 6.2.4 The Line Number Program Header
>
> In the definition of “opcode base”, the phrase “vendor-specific
> extensions” should be replaced with “producer-specific extensions”
> in all three places where it is used.
>
>
> Section 6.2.4.2 Vendor-defined Content Descriptions
>
> The name of the section should be changed to “Producer-defined
> Content Descriptions” and the first word of the first sentence
> should also be changed to match.
>
> The non-normative text following the section should be changed to:
>
> If a consumer encounters a producer-defined content type that it
> does not understand, it should skip the content data as though it
> were not present. However, to achieve full compatibility the
> consumer’s developer should refer to <url> where all known
> producer-defined line number content descriptors are enumerated.
>
>
> Section 6.3.1 Macro Information Header
>
> In the second paragraph of item 4 the phrase “Vendor extension
> entry opcodes” should be replaced with “Producer specific entry
> opcodes”.
>
>
> Section 7.1 Vendor Extensibility
>
> The name of the section should be changed from “Vendor
> Extensibility” to “Extensibility”.
>
> Throughout section 7.1 the phrase “vendor specific” should be
> replaced with “producer specific”.
>
> In the second paragraph, the sentence beginning with “Vendors may
> use values” should be replaced with “Producers may use values”.
>
> The paragraph and list of points at the end of the section should
> be replaced with:
>
> Producer defined tags, attributes, base type encodings, location
> atoms, language names, line number actions, calling conventions
> and call frame instructions, historically have used the form
> prefix_vendor_id_name, where vendor_id is some identifying
> character sequence chosen so as to avoid conflicts with other
> vendors. This established convention can continue in the cases
> where there is historical precedent or where there is a vendor but
> it can also be the name of the producer's project when there is no
> specific vendor.
>
> To ensure that extensions added by one producer may be safely
> ignored by consumers that do not understand those extensions, the
> following rules must be followed:
>
> 1.
>
> Producer defined extensions must be evaluated in the context
> of a DWARF standard version.
>
> 2.
>
> New attributes are added in such a way that a debugger may
> recognize the format of a new attribute value without knowing
> the content of that attribute value.
>
> 3.
>
> The semantics of any new attributes do not alter the semantics
> of previously existing attributes.
>
> 4.
>
> The semantics of any new tags do not conflict with the
> semantics of previously existing tags.
>
> 5.
>
> New forms of attribute value are not added.
>
> To ensure full tool intercompatibility, producer defined DWARF
> extensions should be registered with the DWARF standards body at
> <URL>.
>
>
> Section 7.32 Type Signature Computation
>
> In the 3rd paragraph of point 4 replace “vendor-specific
> attributes” with “producer-specific attributes”.
>
>
> Appendix A
>
> In the first paragraph, replace “and new, vendor-defined ones”
> with “and producer defined ones”
>
>
> Appendix D.7.3
>
> In the text between the examples replace “vendor-specific” with
> “producer-specific” in both cases.
>
> --
> Dwarf-discuss mailing list
> Dwarf-discuss@lists.dwarfstd.org
> https://lists.dwarfstd.org/mailman/listinfo/dwarf-discuss
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.dwarfstd.org/pipermail/dwarf-discuss/attachments/20231201/ecc4a622/attachment-0001.htm>
More information about the Dwarf-discuss
mailing list