[Dwarf-discuss] Seeking a test program with a >4GB .debug_info section

Mon Apr 24 19:10:46 GMT 2023

On Fri, Apr 21, 2023 at 2:16 PM John DelSignore <JDelSignore@perforce.com>
wrote:

>
> On 4/21/23 16:36, David Blaikie wrote:
>
> On Fri, Apr 21, 2023 at 12:44 PM John DelSignore <JDelSignore@perforce.com>
> wrote:
>
>> Well, it took a long time to compile 5 CUs that contained your test code,
>> and things were looking promising, but the link failed:
>>
>> rocm2 42 04/21 15:14 /build/jdelsign/fatty % make
>> g++ -g -c fatty4.cxx -o fatty4.o
>> g++ -g -c fatty5.cxx -o fatty5.o
>> g++ -g -o fatty fatty.o fatty2.o fatty3.o fatty4.o fatty5.o
>> fatty5.o:(.debug_aranges+0x6): relocation truncated to fit: R_X86_64_32
>> against `.debug_info'
>> collect2: error: ld returned 1 exit status
>> make: *** [Makefile:5: fatty] Error 1
>> rocm2 43 04/21 15:39 /build/jdelsign/fatty %
>>
>> I guess I'm now in favor of the proposal to get rid of .debug_aranges. :-)
>>
> I guess, backing up - what's your goal/what're you trying to do with DWARF
> over 4GB?
>
> We have a TotalView user that has a gigantic executable (~9GB) and the
> .debug_info section alone is 4.9GB. A few of the other sections are about
> 1GB, but under the 32-bit limit. That's about all I know, because the user
> is at a secure location and cannot share the executable or other details.
>
> TotalView had a built-in limit of 4GB for DWARF sections. Recently, I
> increased TV's DWARF section sizes to 64-bits to handle this user's code,
> but I'm sure there are other changes that are needed to properly handle a
> .debug_info section that is that big. So, I'm looking for a test case to
> feed to TotalView.
>
Makes sense

> I changed "fatty" to compile with clang++ (instead of g++), and that
> worked. I have the following section sizes:
>
> rocm2 55 04/21 16:40 /build/jdelsign/fatty % readelf -SW fatty|grep debug
>   [30] .debug_abbrev     PROGBITS        0000000000000000 002948 000670
> 00      0   0  1
>   [31] .debug_info       PROGBITS        0000000000000000 002fb8 15e82990e
> 00      0   0  1
>   [32] .debug_str_offsets PROGBITS        0000000000000000 15e82c8c6
> c0db3e8 00      0   0  1
>   [33] .debug_str        PROGBITS        0000000000000000 16a907cae
> 217ee2aa 01  MS  0   0  1
>   [34] .debug_addr       PROGBITS        0000000000000000 18c0f5f58 000050
> 00      0   0  1
>   [35] .debug_line       PROGBITS        0000000000000000 18c0f5fa8 0001dd
> 00      0   0  1
>   [36] .debug_line_str   PROGBITS        0000000000000000 18c0f6185 00004c
> 01  MS  0   0  1
> rocm2 56 04/21 16:41 /build/jdelsign/fatty %
>
> That looks like a decent test case (I haven't tried TotalView on it yet),
> so thanks for the suggestion.
>
>
> You do have to use DWARF64 for a .debug_info section over 4GB - for any
> section-relative reference in that section, such as cross-CU references
> (sec_offset), or aranges or debug_names I think, etc. Because with DWARF32
> the section references are 32bit, so can't exceed 4GB.
>
> I suspected that the application might contain 64-bit DWARF, so I asked
> the user dump/grep the .debug_info section and oddly enough all of the CUs
> are 32-bit DWARF. I have no idea if they are damaged. The CUs must not
> reference any DIEs outside their own CU.
>
Yeah, I guess if you don't use sec_offset (usually only appears with LTO,
at least that's the case with clang) and don't use aranges - there's no
need to refer to the absolute offset of a thing in .debug_info, so you can
exceed the 32 bit limit in total .debug_info size without needing DWARF64.

>
> (also, with a different example, you'd get string data over 4GB, which
> you'd also need DWARF64 for, or in DWARFv5 (with some dispensations from
> DWARFv6) you could use DWARF64 for .debug_str_offsets (assuming all string
> references were strx forms) without using DWARF64 for everything else)
>
> Understood. Luckily, my test program's string table made it with some room
> to spare.
>
>
> In Split DWARF if each contribution is <4GB you only need a 64 bit
> cu/tu_index and 64 bit str_offsets (& you could even do that selectively,
> only using DWARF64 str_offsets for contributions that need 64 bit offsets),
> since nothing else references across whole sections - which makes it much
> more scalable/easier to solve.
>
> Yes, I asked if they'd be willing to consider Split DWARF, but I haven't
> received an answer yet. I think it would save considerable link time and
> disk space, but they may have reasons why Split DWARF would not work for
> them.
>
> BTW, given that "gold" is deprecated, is there a linker than can build a
> decent ".gdb_index" to use with the Split DWARF? Without a decent index,
> it's hard to get same-day service out of the debugger with these giant
> executables.
>
lld has support for .gdb_index from .debug_gnu_pubnames/pubtypes - that's
what we use at Google. (& for Split DWARF, as far as I recall/understand,
but don't have a concrete example these days - gdb /requires/ an index for
correctness, it'll assume that a lookup failure is authoritative - it won't
actually load in all the .dwo/dwp content in the case of a missing index -
though, I might be wrong, or maybe it's changed, because, as I said, don't
have a clear example that demonstrates this failure at the moment)

> Also, another issue is that even if you have simple/small bits of DWARF32
> (in some precompiled library, etc), if your total program exceeds 64 bit,
> you may not be able link the program because that DWARF32 might end up
> being put at the end of the section, and so the debug_aranges, for
> instance, needs to record the offset of the CU in 32 bits but can't. So
> there's various discussions about linkers being able to sort the DWARF32
> contributions earlier/first before the DWARF64 contributions. Then you
> could still link DWARF32 precompiled libraries into huge programs that
> exceed the DWARF32 limits. I can go find some links to those
> threads/discussions if you need them, I think some happened in the LLVM
> open source community.
>
> Looks like clang dodged all of those problems somehow:
>
> rocm2 60 04/21 16:58 /build/jdelsign/fatty % llvm-dwarfdump fatty | grep
> "Compile Unit"
> 0x00000000: Compile Unit: length = 0x461a1e96, format = DWARF32, version =
> 0x0005, unit_type = DW_UT_compile, abbr_offset = 0x0000, addr_size = 0x08
> (next unit at 0x461a1e9a)
> 0x461a1e9a: Compile Unit: length = 0x461a1e99, format = DWARF32, version =
> 0x0005, unit_type = DW_UT_compile, abbr_offset = 0x0148, addr_size = 0x08
> (next unit at 0x8c343d37)
> 0x8c343d37: Compile Unit: length = 0x461a1e99, format = DWARF32, version =
> 0x0005, unit_type = DW_UT_compile, abbr_offset = 0x0292, addr_size = 0x08
> (next unit at 0xd24e5bd4)
> 0xd24e5bd4: Compile Unit: length = 0x461a1e99, format = DWARF32, version =
> 0x0005, unit_type = DW_UT_compile, abbr_offset = 0x03dc, addr_size = 0x08
> (next unit at 0x118687a71)
> 0x118687a71: Compile Unit: length = 0x461a1e99, format = DWARF32, version
> = 0x0005, unit_type = DW_UT_compile, abbr_offset = 0x0526, addr_size = 0x08
> (next unit at 0x15e82990e)
> rocm2 61 04/21 17:13 /build/jdelsign/fatty %
>
> This should give the debuggers something to chew on for a while.
>

Cool cool :)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.dwarfstd.org/pipermail/dwarf-discuss/attachments/20230424/4aad0092/attachment-0001.htm>