macOS High Sierra and “fork” compatibility
This was originally discussed on the issue tracker for Puma (https://github.com/puma/puma/issues/1421), however, it is possible that it would make more sense for inclusion in the Ruby implementation itself.
macOS High Sierra has changed the behaviour of the fork syscall such that initialising Objective-C APIs in forked processes are treated as errors. (see http://sealiesoftware.com/blog/archive/2017/6/5/Objective-C_and_fork_in_macOS_1013.html for more details)
This means that many applications which use forking to process concurrently will forcibly crash if the forked process calls out to any Objective-C library when Objective-C was not already initialised in the host process. This includes Puma, Unicorn, iodine and Passenger.
A workaround I proposed for Puma was to implicitly load the Objective-C runtime before performing any forks (https://github.com/puma/puma/issues/1421#issuecomment-332650703). This causes forked processes using other Objective-C APIs to not crash.
The workaround (specific to Puma’s DSL) was:
# Work around macOS 10.13 and later being very picky about # `fork` usage and interactions with Objective-C code # see: <https://github.com/puma/puma/issues/1421> if /darwin/ =~ RUBY_PLATFORM before_fork do require 'fiddle' # Dynamically load Foundation.framework, ~implicitly~ initialising # the Objective-C runtime before any forking happens in Puma Fiddle.dlopen '/System/Library/Frameworks/Foundation.framework/Foundation' end end
A similar fix has now been included in Passenger (https://github.com/phusion/passenger/blob/2a55a84e5de721d8bd806a8fea0bcedf27583c29/src/ruby_supportlib/phusion_passenger/loader_shared_helpers.rb#L84-L105).
It was, however, proposed that it might make more sense for Ruby on macOS High Sierra and onward to implicitly initialise the Objective-C framework itself, so that forked processes work roughly as expected even if they intend to use Objective-C APIs.
I understand that this is a heavy-handed move, but it seems to me that this relatively common technique will remain broken in Ruby unless everyone deploys a workaround (iodine has already expressed disinterest in doing so) or Ruby adopts one at the higher level.
This issue is also applicable to all Ruby versions which support fork and run on macOS High Sierra.
Thank you for your time. :)
#1 [ruby-core:83243] Updated by hongli (Hongli Lai) 6 days ago
Perhaps it helps if I further clarify this issue for those who are unfamiliar with it.
As you probably already know, forking (but without exec'ing) in a multithreaded environment is inherently dangerous and the environment must be carefully written to support such a thing. Apple's Objective-C libraries have traditionally not supported being called in a forked (but not exec'd) child process at all, but since High Sierra 10.13 they've tried to add limited support for this. However in doing so they've also defined rules on what is not allowed after forking. One of the rules state that it is not allowed to call the
initialize function of certain Objective-C classes after forking; that may only happen before forking.
Makes sense so far. The problem occurs because of a combination of three things:
- Ruby itself is not linked to any Objective-C libraries, and so does not initialize Objective-C classes by itself.
- The user may use gems that do link to Objective-C libraries. Due to how these gems are used, it can occur that these gems end up calling Objective-C initializers after the app server has forked.
- The new Apple-enforced rule checks then abort the process with a warning like this:
objc: +[__NSPlaceholderDictionary initialize] may have been in progress in another thread when fork() was called. objc: +[__NSPlaceholderDictionary initialize] may have been in progress in another thread when fork() was called. We cannot safely call it or ignore it in the fork() child process. Crashing instead. Set a breakpoint on objc_initializeAfterForkError to debug.
By itself, Apple's error check makes sense. Forking is dangerous. But all these factors combined make less sense. Adding a workaround in Ruby (in the form of ensuring that Objective-C initializers are called before forking) will at least ensure that we return to pre-High Sierra behavior.
Adding a workaround inside Ruby would only be a part of the whole solution. In order to fully fix the problem, cooperation from the wider Ruby community is required: all the gem authors will also have to ensure that native libraries don't spawn any threads until the app server has forked. But I do believe that having a workaround inside Ruby is an essential part of the entire fix, because updating all gems takes a lot of time and effort.
#3 [ruby-core:83250] Updated by myst (Boaz Segev) 5 days ago
I should point out that
"Foundation.framework/Foundation" seems to be enough.
This is the C code I'm currently considering for the iodine Ruby server.
#ifdef __APPLE__ void *obj_c_runtime = dlopen("Foundation.framework/Foundation", RTLD_LAZY); /* iodine runs here and `fork` might be called */ dlclose(obj_c_runtime); #else /* iodine runs here and `fork` might be called */ #endif
__APPLE__directive will only work on
clangand intel compilers (other compilers might be supported, but I don't know).
The code doesn't check for errors, because a quite failure doesn't effect behavior. If Objective-C isn't available, than forking will not be effected.
The code doesn't check for High Sierra specifically and will load Objective-C on all macOS versions (anything over 10.9).
The memory increase was from 4388Kb (Ruby VM) to ~4560Kb (compared using Fiddle, not a direct C call), which seems insignificant on the larger scale of things.
Baseline: loading the Objective-C using Fiddle caused a 3.9% increase in memory consumption relative to the basic the VM core.
Loading the Socket Standard library (
require 'socket') caused a 16% increase in memory consumption relative to the basic the VM core.
Loading Sinatra caused more than a 196% increase in memory consumption relative to the basic the VM core.
Loading Rails caused more than a 264% increase in memory consumption relative to the basic the VM core.
It's my understanding that a simple patch will load the Objective-C library within the
ruby_init() function and ignore any errors in loading the library.
A more complex approach will lazy load the library only when
Process#fork is called... but this might effect C extensions that might call
fork directly and it might also effect the use of Ruby as a library (when
ruby_init() is called within a C application that uses Ruby internally).
Personally, I'm both lazy and paranoid and I would assume that the simple approach would both work better and fail better (expose issues earlier rather than later).
#4 Updated by nobu (Nobuyoshi Nakada) 5 days ago
- Status changed from Open to Feedback
Foundation framework instead of
CoreFoundation framework work?
diff --git a/configure.in b/configure.in index bb7cc4daa5..4a2243afa1 100644 --- a/configure.in +++ b/configure.in @@ -4144,8 +4144,8 @@ AS_CASE(["$target_os"], ], [darwin*], [ RUBY_APPEND_OPTION(CFLAGS, -pipe) - RUBY_APPEND_OPTION(XLDFLAGS, [-framework CoreFoundation]) - RUBY_APPEND_OPTION(LIBRUBYARG_STATIC, [-framework CoreFoundation]) + RUBY_APPEND_OPTION(XLDFLAGS, [-framework Foundation]) + RUBY_APPEND_OPTION(LIBRUBYARG_STATIC, [-framework Foundation]) ], [osf*], [ if test "$GCC" != "yes" ; then
diff --git a/configure.ac b/configure.ac index e6cd95cc9d..6cd99e6ef2 100644 --- a/configure.ac +++ b/configure.ac @@ -4116,8 +4116,8 @@ AS_CASE(["$target_os"], ], [darwin*], [ RUBY_APPEND_OPTION(CFLAGS, -pipe) - RUBY_APPEND_OPTION(XLDFLAGS, [-framework CoreFoundation]) - RUBY_APPEND_OPTION(LIBRUBYARG_STATIC, [-framework CoreFoundation]) + RUBY_APPEND_OPTION(XLDFLAGS, [-framework Foundation]) + RUBY_APPEND_OPTION(LIBRUBYARG_STATIC, [-framework Foundation]) ], [osf*], [ AS_IF([test "$GCC" != "yes" ], [
#6 [ruby-core:83260] Updated by hongli (Hongli Lai) 5 days ago
nobu (Nobuyoshi Nakada) wrote:
Foundationframework instead of
According to my tests on High Sierra, linking to Foundation helps, but linking to CoreFoundation does not. CoreFoundation does not (at load time) initialize any Objective-C classes. Here are test scripts which simulate the issue (and a working fix and a non-working fix) on High Sierra:
# This script simulates a child process that initializes ObjC classes, # and should crash. require 'fiddle' def init_objc_classes Fiddle.dlopen('/System/Library/Frameworks/Foundation.framework/Foundation') end pid = fork do init_objc_classes end Process.waitpid(pid)
$ ruby broken.rb objc: +[__NSPlaceholderDictionary initialize] may have been in progress in another thread when fork() was called. objc: +[__NSPlaceholderDictionary initialize] may have been in progress in another thread when fork() was called. We cannot safely call it or ignore it in the fork() child process. Crashing instead. Set a breakpoint on objc_initializeAfterForkError to debug.
# This script simulates a process that initializes ObjC classes before, # forking, and should NOT crash. require 'fiddle' def init_objc_classes Fiddle.dlopen('/System/Library/Frameworks/Foundation.framework/Foundation') end init_objc_classes pid = fork do # The following does nothing because ObjC classes # are already initialized before forking. init_objc_classes end Process.waitpid(pid)
$ ruby fixed.rb (no crash)
# This script simulates a process that tries to initialize ObjC classes before, # forking, and still crashes because the method doesn't actually work. This # simulates nobu's proposed fix in https://bugs.ruby-lang.org/issues/14009#note-4 require 'fiddle' def invoke_nobu_fix Fiddle.dlopen('/System/Library/Frameworks/CoreFoundation.framework/CoreFoundation') end def init_objc_classes Fiddle.dlopen('/System/Library/Frameworks/Foundation.framework/Foundation') end invoke_nobu_fix pid = fork do init_objc_classes end Process.waitpid(pid)
$ ruby nobu-fix.rb objc: +[__NSPlaceholderDictionary initialize] may have been in progress in another thread when fork() was called. objc: +[__NSPlaceholderDictionary initialize] may have been in progress in another thread when fork() was called. We cannot safely call it or ignore it in the fork() child process. Crashing instead. Set a breakpoint on objc_initializeAfterForkError to debug.
And here is a sanity check to verify that loading CoreFoundation in fact does not initialize prohibited ObjC classes:
# This script should not crash in the child. It should verify # that CoreFoundation does not initialize any Objective-C # classes. require 'fiddle' def load_core_foundation Fiddle.dlopen('/System/Library/Frameworks/CoreFoundation.framework/CoreFoundation') end load_core_foundation pid = fork do load_core_foundation end Process.waitpid(pid)
$ ruby corefoundation-sanity-check.rb (no crash)
#8 [ruby-core:83265] Updated by myst (Boaz Segev) 5 days ago
Please ignore the original comment.
I tested linkage in Sierra (not High Sierra), the
-framework instruction is honored even when no symbols are used.
The linker approach seems safe to use in macOS. I wish I could test it on High Sierra.
nobu (Nobuyoshi Nakada) , I'm wondering if some linkers might optimize this instruction away...?
My question is whether or not an optimized linker might ignore the instruction after noticing that no symbols from the shared library are used.
If this is the case, than the linker approach will work on some linkers and not on others.
I think the documentation I am referencing (the documentation for
ld) is the same for both static and dynamic libraries (but maybe it's only how static libraries are resolved):
Add archive file archive to the list of files to link. This option may be used any number of times. ld will search its path-list for occurrences of libarchive.a for every archive specified. On systems which support shared libraries, ld may also search for libraries with extensions other than .a. Specifically, on ELF and SunOS systems, ld will search a directory for a library with an extension of .so before searching for one with an extension of .a. By convention, a .so extension indicates a shared library. The linker will search an archive only once, at the location where it is specified on the command line. If the archive defines a symbol which was undefined in some object which appeared before the archive on the command line, the linker will include the appropriate file(s) from the archive. However, an undefined symbol in an object appearing later on the command line will not cause the linker to search the archive again. See the -( option for a way to force the linker to search archives multiple times. You may list the same archive multiple times on the command line. This type of archive searching is standard for Unix linkers. However, if you are using ld on AIX, note that it is different from the behaviour of the AIX linker.
Because Ruby doesn't have any undefined symbols that are defined in the Foundation library, this library might be ignored and the patch will fail.
#10 [ruby-core:83268] Updated by ticky (Jessica Stokes) 5 days ago
Alright, I’ve checked it out, and can confirm that the patch nobu (Nobuyoshi Nakada) suggested does indeed work around the original issue I encountered in Puma, with my mitigations removed from our application code.
I can also confirm that with that patch applied, none of the code snippets supplied by hongli (Hongli Lai) fail on High Sierra, which means this definitely works around the problem. :)
#12 [ruby-core:83271] Updated by hongli (Hongli Lai) 5 days ago
Regarding nobu's comment on whether linkers can ignore "useless" library links (comment #8): yes they can, but both on GNU/Linux systems as well as on macOS, this behavior is at present not the default.
On GNU systems, --as-needed lets the GNU linker skip linking unused libraries. There is one blog article out there that claims that
--as-needed has been made the default on newer GNU systems, but I've not personally verified this, nor have I found any other sources so far.
macOS's linker has
-dead_strip_dylibs which does the same.
So we don't have to worry about this issue.
#15 Updated by nobu (Nobuyoshi Nakada) 4 days ago
- Status changed from Feedback to Closed
#18 [ruby-core:83303] Updated by myst (Boaz Segev) 2 days ago
I'm happy this was merged to the Ruby trunk.
Just an idea, environment variable DYLD_INSERT_LIBRARIES=/System/Library/Frameworks/Foundation.framework/Versions/Current/Foundation doesn't work?
Interesting idea :)
It would probably work exactly like the patch, except it will be user controlled instead of persistent.