[Dwarf-Discuss] debug_aranges use and overhead

[Samy Al Bahra]

Hi Greg,

We currently use a similar format for server-side symbolication where
static analysis is not being performed (for example, when we ingest core
dumps / minidump directly on the backend) with similar design goals, it was
inspired by Breakpad's sym and includes similar extensions for inline
functions. GSym looks terrific for that use-case and we'll definitely take
a look if it means we can use an open standard. I'll keep you posted and
thanks for the reference!

For our server-side debugger, it is run on core dumps or live processes and
we require full debug information as our debugger will do pointer chasing
and static analysis such as:
- Alias detection to the faulting address.
- Stale pointer detection (marries allocator analysis along with DWARF type
information).
- Memory allocator integrity scanning and metadata collection.
- Variable resolution, pretty-printing.
- And a bunch of other things (malware detection including things like
executable modification, ROP chain detection, etc...).

Gsym does not have the necessary information for this. The use-case here is
for servers and appliances that ship with debug information, the analysis
is meant to be done on the servers efficiently or offloaded to dedicated
core dump processing servers (customers prefer the former, it's a much
simpler setup on their end operationally). In many cases, the debugger is
on the fast path before the software running on the appliance is restarted.

On Fri, Mar 19, 2021 at 2:15 PM Greg Clayton <clayborg at gmail.com> wrote:

>
>
> On Mar 19, 2021, at 9:33 AM, Samy Al Bahra via Dwarf-Discuss <
> dwarf-discuss at lists.dwarfstd.org> wrote:
>
> Hi David,
>
> Sorry I'm a bit late to the game. On the value of having .debug_aranges
> and the performance impact:
>
> Our debugger was designed for performance and does end to end lazy
> evaluation (down to the DIE). This is quite old (excuse the formatting) but
> numbers are here:
> https://engineering.backtrace.io/2014-09-15-bt-lightweight-backtrace-tool/
> , search for "Chromium".  This is something other debuggers can take
> advantage of if they run in a non-interactive / batch mode (think bulk
> processing of millions - billions of dumps a month) and is generally useful
> when folks are iterating in development (fast feedback for crashes while
> having some background indexing work going on). I'm also happy to run
> benchmarks for you with and without .debug_aranges on top of our debugger
> if it'll be useful.
>
> One of the crucial optimizations we made is incremental indexing on top of
> .debug_aranges based on PC values (+ complexities Greg mentions later in
> the thread). In cases where we lack this, we use our own persistent cache
> which introduces unnecessary complexity. Now I am considering going as far
> as adding a multi-threaded indexer for cases where a persistent cache /
> build system modifications aren't an option (work to begin in the next week
> or two).
>
>
>
> For symbolicating in bulk, millions - billions of dumps a month, I would
> highly suggest you check out converting your DWARF to GSYM. This file
> format was created because of the inefficiencies of using DWARF for
> symbolication when done at a large scale on servers. The file format is
> designed to be mapped into memory and used as is. It is many times smaller
> than DWARF and the lookup speeds are orders of magnitude faster. The file
> format is designed to be mmap'ed into one or more processes as shared
> memory so that if your symbolication servers can route the symbolication
> requests for the same GSYM file to the same server then pages from that
> GSYM file can be hot in the file cache and speed up requests event further.
>
> Features of the GSYM file format:
> - Address table is sorted for fastest lookups and doesn't store full 64
> bit addresses, just offsets from a base address from the file header. This
> allows you to search for your address and touch as few pages as possible
> when doing the lookup. You are also only searching addresses when doing the
> search, no extra data. The offsets to the data for each address are in a
> separate table that follows the address table.
> - Single uniqued file table to store source file paths for any data that
> is references within the GSYM file (like the source files for the line
> tables). This means all compile unit file tables from the DWARF, which each
> have their own file tables in each prologue of the info in .debug_line, are
> uniqued into a single file table which reduces GSYM file size. The file
> table also splits all paths up into directory + basename which allows all
> files in the same directory to shared the same string in the string table
> for the directory path.
> - All function info is stored in one contiguous blob of data and includes:
>   - full line table for the function only (no more parsing the entire line
> table of a compile unit just to get the lines for a function)
>   - inline call stack information to unwind inline call stacks
>
> The GSYM file format description is contained in the header files in the
> headerdoc. The GSYM file format is open sourced into llvm.org and the
> code can be found:
>
> Library code:
>
>   llvm/include/llvm/DebugInfo/GSYM
>   llvm/lib/DebugInfo/GSYM
>
> Tool code:
>
>   llvm/tools/llvm-gsymutil
>
>
> If you build llvm-gsymutil you can convert any DWARF to GSYM using:
>
>   $ llvm-gsymutil --convert a.out -o a.out.gsym
>
> We have seen GSYM files up to 20x smaller than the DWARF file, which to be
> fair often includes all other sections (.text, .data, etc).
>
> If you want to see exactly what is encoded in the GSYM file you can dump
> the entire file using:
>
>   $ llvm-gsymutil -o a.out.gsym
>
> This will give you a great idea of exactly what is encoded in the GSYM
> file as the dump accurately mirrors the exact contents of the file.
>
> Then lookups can be done with a the tool:
>
>   $ llvm-gsymutil --address 0x1000 a.out.gsym
>
> Or by linking against the LLVM library code and doing it with a few lines
> of code:
>
>   llvm::Expected<GsymReader> Gsym = GsymReader::openFile(GSYMPath);
>   if (!Gsym)
>     error(GSYMPath, Gsym.takeError());
>
>
>   if (llvm::Expected<LookupResult> Result = Gsym->lookup(Addr))
>     OS << Result.get();
>   else
>     error(Result.takeError());
>
> The LookupResult object has all of the information including function +
> source file + source line for all frames from the inline functions and
> concrete functions from the one address you looked up.
>
> Anyone interested in learning more can feel free to message me directly
> offline for more information or support on GSYM.
>
> Greg Clayton
>
> .debug_aranges would provide a lot of value to our users.
>
> On Thu, Mar 11, 2021 at 3:48 PM David Blaikie via Dwarf-Discuss <
> dwarf-discuss at lists.dwarfstd.org> wrote:
>
>> On Thu, Mar 11, 2021 at 5:48 AM <paul.robinson at sony.com> wrote:
>>
>>> Hopefully not to side-track things too much... maybe wants its own
>>> thread, if there's more to debate here.
>>>
>>
>> Yeah, how about we spin it off into another thread (done here)
>>
>>
>>> >> For the case you suggested where it would be useful to keep the range
>>> >> list for the CU in the .o file, I think .debug_aranges is what you're
>>> >> looking for.
>>> >
>>> > aranges has been off by default in LLVM for a while - it adds a lot of
>>> > overhead (doesn't have all the nice rnglist encodings for instance -
>>> > nor can it use debug_addr, and if it did it'd still be duplicate with
>>> > the CU ranges wherever they were).
>>>
>>> Did you want to file an issue to improve how .debug_aranges works?
>>>
>>
>> I don't currently understand the value it provides, and I at least don't
>> have a use case for it, so I'm not sure I'd be the best person to
>> advocate/drive that work.
>>
>> Complaining that it duplicates CU ranges is missing the point, though;
>>> it's an index, like .debug_names, of course it duplicates other info.
>>> If you want to suggest an improved index, like we did with .debug_names,
>>> that would be great too.
>>>
>>
>> .debug_names is quite different though - it collects information from
>> across the DIE tree - information that is expensive to otherwise gather
>> (walking the whole DIE tree).
>>
>> .debug_aranges is not like that for most producers (producers that do
>> include the address ranges on the CU DIE) - the data is readily available
>> immediately on the CU. That does involve reading some of .debug_abbrev, and
>> interpreting a handful of attributes - but at least for the use cases I'm
>> aware of, that overhead isn't worth the size increase.
>>
>> Do you have numbers on the benefits of .debug_aranges compared to parsing
>> the ranges from CU DIEs?
>>
>> (one possible issue: the CU doesn't /have/ to contain low/high/ranges if
>> its children DIEs contain addresses - having that as a guarantee, or some
>> preferred way of encoding zero length (high/low of 0 would be acceptable, I
>> guess) would be nice & make it cheap to skip over CUs that don't have any
>> address ranges)
>>
>> Roughly, a modern debug_aranges to me would look something like:
>>
>> <length>
>> <version>
>> <CU sec_offset>
>> <addr_base>
>> <rnglist sec_offset>
>>
>> So it could fully re-use the rnglist encoding. If this was going to be as
>> compact as possible, it'd need to be configurable which encodings it uses -
>> ranges V high/low, addrx V addr - at which point it'd probably look like a
>> small DIE with an inline abbrev (similar to the way DWARFv5 encodes the
>> file and directory entries now, and how debug_names is self-describing) -
>> at which point it looks to me a lot like parsing the CU DIEs.
>>
>> _______________________________________________
>> Dwarf-Discuss mailing list
>> Dwarf-Discuss at lists.dwarfstd.org
>> http://lists.dwarfstd.org/listinfo.cgi/dwarf-discuss-dwarfstd.org
>>
>
>
> --
> Samy Al Bahra [http://repnop.org]
> _______________________________________________
> Dwarf-Discuss mailing list
> Dwarf-Discuss at lists.dwarfstd.org
> http://lists.dwarfstd.org/listinfo.cgi/dwarf-discuss-dwarfstd.org
>
>
>

-- 
Samy Al Bahra [http://repnop.org]
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.dwarfstd.org/pipermail/dwarf-discuss-dwarfstd.org/attachments/20210319/5eec8b44/attachment-0001.html>