[Dwarf-Discuss] DWP mixed (DWARFv4/pre-standard + DWARFv5) content

[Pavel Labath]

On Tue, 25 Feb 2020 at 21:36, David Blaikie <dblaikie at gmail.com> wrote:
>
>
>
> On Tue, Feb 25, 2020 at 11:41 AM Pavel Labath <labath at google.com> wrote:
>>
>> Yeah, this sounds tricky, but it is actually good timing, because I
>> was just about to start working on DWP v5 in lldb. I was hoping that
>> would be an easy ride, but it looks like things will get complicated.
>>
>> On Tue, 25 Feb 2020 at 18:53, David Blaikie <dblaikie at gmail.com> wrote:
>> >
>> > (please add anyone who has a vested interest in Split DWARF in general and dwp in particular)
>> >
>> > tl;dr:
>> > How should DWARFv4 and DWARFv5 coexist in a DWP file:
>> > 1) Not at all (invalid/unsupported)
>> > 2) Single index table where the section indexes are subjective (look at the version of the referenced CU/TU to decide what a column number means (eg: 8 is DW_SECT_MACRO in v4, DW_SECT_RNGLISTS in v5)
>> > 3) Emit multiple indexes (cu_index and tu_index each would contain two indexes) - a v4 (well, index version 2) and a v5 (index version 5) index.
>>
>> What about option 4)? :P Have the numbers be interpreted relative to
>> the debug_cu_index version number?
>> The idea would be that a DWARF v4 (dwp tool seems to use version
>> number 2 for some reason)
>
>
> Yeah - the version 2 comes from the pre-standard development iterations ( https://gcc.gnu.org/wiki/DebugFissionDWP documents version 2/mentions the existence of version 1)
>
>>
>> debug_cu_index would keep using the existing
>> pre-standard scheme, and it would not be able to refer to v5 units. In
>> a v5 debug_cu_index (*), we would use the official DWARF v5 scheme,
>> *but* we would, as an extension, support referring to the pre-standard
>> sections via custom DW_SECT constants (with some high numbers to avoid
>> conflicting with future standard versions).
>
>
> Yeah, extending the v5 format might be a good idea. I'm open to that.
>
> Pity that there wasn't a standard extension space in the DWP spec to make that smoother. The existence of macinfo and macro in the pre-standard version perhaps could've been a hint that non-standard extensions might be needed (debug_macro being a non-standard extension itself).

Yes, the lack of an official extension space is unfortunate, but I
don't think this needs to be a blocker -- the spec also doesn't
include an DW_FORM extension space, but that hasn't stopped people
from inventing new forms. I would say that the situation is much
better here actually -- it's quite easy for a consumer to ignore an
unknown DW_SECT constant, whereas an unknown DW_FORM pretty much means
you need to throw away the whole unit.

Thinking about this further, the lack of a DW_FORM extension space is
probably a reason for not introducing a DW_SECT extension space --
it's hard to imagine how someone could meaningfully add a new section
without a new form to refer to it.

The main question on my mind now is, what is the likely future
direction of the DWARF spec -- if say DWARF v6 adds a new section, how
will it handle mixed v5+v6 debug_cu_indexes? I don't think it will
want to make that unsupported (1). I also doubt it will resort to
"subjective" DW_SECT constants, as it is easy to ensure the official
DW_SECT constants do not overlap.

The option (3) sounds plausible --  I do appreciate the cleanliness of
having a v5 index containing v5 units only, and follow it with a v6
index with v6 units. But I am not sure if that's the direction I would
take, because I also very much like the idea of having just a single
index table to access all units.

>
>>
>> (*) The v5 cu_index would be used whenever we have at least one v5
>> dwarf unit. Ignoring the section number mismatches, I think this is a
>> reasonable approach, and it is the one I would make if I was in the
>> shoes of the DWARF v6 committee and needed to make the debug_cu_index
>> cover mixed-version units
>>
>>
>> Speaking of DWARF v6, I think the debug_cu/tu_index sections are in
>> need of some redesign anyway. For example, it's not clear to me how
>> these sections would work in presence of DWARF64 -- all section
>> offsets are hardcoded to 4 bytes, and we don't even have the
>> initial_length mechanism to select bigger sizes.
>
>
> Yep - and if we're enumerating things to keep in mind for v6 here: probably collapse the tu_index and cu_index, just index all the things together in one index, probably? They're all indexed by signatures and they're all in the same section now.

I've been wondering about that too. The way I've explained this for
myself is that these are actually two different keys they are indexing
over -- type units have a "type signature", whereas compile units have
a "dwo id". It's always clear from the context which one are you
referring to so it's possible to have a compile unit and a type unit
with the same "id" coexist -- which would be trickier if they were in
the same index.

>
> Good point about DWARF64 there. (I feel like, if possible, all the DWARF contribution headers should be "DWARF32/64 length, then version, then stuff..." - but I guess that might mean needing to rename some sections again... (I think debug_macro missed this too - it has no length, just a version).

I can see how debug_[ct]u_index could be considered special as (I
think) there's only ever supposed to be a single contribution in them,
and we could invent a different way to signal DWARF64 there, but the
lack of the initial_length field in debug_macro sounds like a big
omission. I would very much like to clean this up, though I dread the
idea of introducing yet another macro section (debug_maclists anyone ?
:P)

pl