Feature #18559: Allocation tracing: Objects created by the parser are attributed to Kernel.require

• Status: Closed
• Assignee: -
• Target version: -

[byroot (Jean Boussier)]

Marking this as a feature, because I think it should be improved but can hardly be considered a bug.

Repro

Consider the following script:

# /tmp/allocation-source.rb
require 'objspace'
require 'tmpdir'

source = File.join(Dir.tmpdir, "foo.rb")
File.write(source, <<~RUBY)
  # frozen_string_literal: true
  class Foo
    def plop
      "fizz"
    end
  end
RUBY

ObjectSpace.trace_object_allocations_start

GC.start
gen = GC.count
require(source)
ObjectSpace.dump_all(output: $stdout, since: gen)

Expected behavior

I'd expect the ObjectSpace.dump_all output to attribute all new objects, including T_IMEMO etc, to foo.rb

Actual behavior

They are attributed to the source file that called Kernel.require (so with --disable-gems):

{"address":"0x11acaec78", "type":"CLASS", "class":"0x11acaebb0", "superclass":"0x10fa4a848", "name":"Foo", "references":["0x10fa4a848", "0x11acaea98", "0x11acaf790"], "file":"/var/folders/vy/srfpq1vn6hv5r6bzkvcw13y80000gn/T/foo.rb", "line":2, "generation":1, "memsize":544, "flags":{"wb_protected":true}}
{"address":"0x11acaeca0", "type":"IMEMO", "class":"0x8", "imemo_type":"cref", "references":["0x10fa4a848"], "file":"/tmp/allocation-source.rb", "line":19, "method":"require", "generation":1, "memsize":40, "flags":{"wb_protected":true}}
{"address":"0x11acaecc8", "type":"STRING", "class":"0x10fa42418", "frozen":true, "embedded":true, "fstring":true, "bytesize":4, "value":"fizz", "encoding":"UTF-8", "file":"/tmp/allocation-source.rb", "line":19, "method":"require", "generation":1, "memsize":40, "flags":{"wb_protected":true}}
{"address":"0x11acaecf0", "type":"ARRAY", "class":"0x10fa28f68", "frozen":true, "length":2, "embedded":true, "references":["0x11acaff88", "0x11acaf240"], "file":"/tmp/allocation-source.rb", "line":19, "method":"require", "generation":1, "memsize":40, "flags":{"wb_protected":true}}
{"address":"0x11acaed18", "type":"IMEMO", "imemo_type":"iseq", "references":["0x11acaecc8", "0x11acaf600", "0x11acaf600", "0x11acaecf0"], "file":"/tmp/allocation-source.rb", "line":19, "method":"require", "generation":1, "memsize":416, "flags":{"wb_protected":true}}
{"address":"0x11acaf1a0", "type":"ARRAY", "class":"0x10fa28f68", "frozen":true, "length":2, "embedded":true, "references":["0x11acaff88", "0x11acaf240"], "file":"/tmp/allocation-source.rb", "line":19, "method":"require", "generation":1, "memsize":40, "flags":{"wb_protected":true}}
{"address":"0x11acaf1c8", "type":"IMEMO", "imemo_type":"iseq", "references":["0x11acaed18", "0x11acaf1f0", "0x11acaf1f0", "0x11acaf1a0", "0x11acaf290"], "file":"/tmp/allocation-source.rb", "line":19, "method":"require", "generation":1, "memsize":456, "flags":{"wb_protected":true}}
{"address":"0x11acaf1f0", "type":"STRING", "class":"0x10fa42418", "frozen":true, "embedded":true, "fstring":true, "bytesize":11, "value":"<class:Foo>", "file":"/tmp/allocation-source.rb", "line":19, "method":"require", "generation":1, "memsize":40, "flags":{"wb_protected":true}}
{"address":"0x11acaf218", "type":"ARRAY", "class":"0x10fa28f68", "frozen":true, "length":2, "embedded":true, "references":["0x11acaff88", "0x11acaf240"], "file":"/tmp/allocation-source.rb", "line":19, "method":"require", "generation":1, "memsize":40, "flags":{"wb_protected":true}}
{"address":"0x11acaf240", "type":"STRING", "class":"0x10fa42418", "frozen":true, "fstring":true, "bytesize":63, "value":"/private/var/folders/vy/srfpq1vn6hv5r6bzkvcw13y80000gn/T/foo.rb", "encoding":"UTF-8", "file":"/tmp/allocation-source.rb", "line":19, "method":"require", "generation":1, "memsize":104, "flags":{"wb_protected":true}}
....

Why is it a problem?

This behavior makes it impossible to properly analyze which part of an application use the most memory. For instance when using heap-profiler on an app using Bootsnap, all objects created as a result of loading source file are attributed to bootsnap:

retained memory by gem
-----------------------------------
 351.64 MB  bootsnap-1.10.2

If this behaved as I expect, heap-profiler would be able to report how much each gem contribute to the app RAM usage.

[byroot (Jean Boussier)]

• Description updated (diff)

[byroot (Jean Boussier)]

So I tried adding a frame before the ISeq is compiled: https://github.com/ruby/ruby/pull/5998

It somewhat works for regular require but:

• It's awkward to put an uncompiled ISeq in the stack, there is even an assert to prevent this.
• It doesn't work when loading the ISeq with load_from_binary because the source location is inside the ISeq. We could re-order the iseq fields so it's the first loaded element, but still very awkward.

So I tried a much simpler, ad hoc solution which is to store an override on the execution context before starting the compilation: https://github.com/ruby/ruby/pull/6057. If present TracePoint expose that. It's not very clean conceptually, but seem much simpler and safer.

@ko1 (Koichi Sasada) expressed some reservations about it, but I don't see any other solution. Opinions welcome.

[byroot (Jean Boussier)]

@ko1 (Koichi Sasada): Because the precise implementation for it is hard, I like foo.rb:0. I will ask the original poster if the lineno is really important

The lineno isn't essential, if you think it's too much, I'll still be happy with lineno=0 for objects created by the parser/compiler.

However my current PR seem to be able to provide the correct line number just fine, but maybe as you suggest it adds lots of overhead? Up to you to decide.

[byroot (Jean Boussier)]

I had a quick chat with @ko1 (Koichi Sasada), he plans to try a better implementation after RubyKaigi (mid-september).

I'll hold on this.

[byroot (Jean Boussier)]

• Status changed from Open to Closed

This was implemented by @ko1 (Koichi Sasada) in e35c528d721d209ed8531b10b46c2ac725ea7bf5

I tested it on our system it has the desired effect. Thank you!