Bug #18947: Unexpected Errno::ENAMETOOLONG on Windows

• Status: Open
• Assignee: windows
• Target version: -
• ruby -v: ruby 3.1.2p20 (2022-04-12 revision 4491bb740a) [x64-mingw-ucrt]
• Backport: 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN

[inversion (Yura Babak)]

On Windows 10, I am working on a script to copy a complex folder structure.

Pathname and FileUtils work fine for me until there is a folder with a very long path (>260 chars).

Normally you cannot access such a folder with Ruby.
The next operations will raise Errno::ENOENT

Pathname.new(300_chars_path).children
FileUtils.mkpath(300_chars_path)

But there is a way in Windows to remove the MAX_PATH limitation.
You can find a small .reg file in this article:
https://docs.microsoft.com/en-us/windows/win32/fileio/maximum-file-path-limitation?tabs=registry

After changing this system option, things start to work strangely in Ruby.

This will now raise Errno::ENAMETOOLONG:

Pathname.new(300_chars_path).children

But at the same time, you can create a folder with such a long path and write-read a file in it

FileUtils.mkpath(300_chars_path)
file = Pathname.new(300_chars_path+'/file.txt')
file.write 'oooooooooo'
puts Pathname.new(300_chars_path+'/file.txt').read

So you can work with individual items but attempts to list such folders' content fail (.children, .glob, .copy, etc).
In my case, deep .glob is broken for all the parent folders of that deep long-path folder ((

The only way I found for listing is

require 'win32ole'
fso = WIN32OLE.new 'Scripting.FileSystemObject'
for file in fso.GetFolder(300_chars_path).files
    file.name
    file.path.length
end

But using this workaround breaks all my code workflow built on top of Pathname and FileUtils ((.

So for me, it looks like some operations with long-path folders are not working just because in Ruby there is a check for the path length and not a real operation problem. And in some places (see .mkpath) there is no such check and all works fine.

Also notice that other applications on Windows have no problems with long-path folders (like Total Commander).

Please consider reviewing if we really need to raise Errno::ENAMETOOLONG if the LongPathsEnabled option is enabled in the Windows registry.

[austin (Austin Ziegler)]

inversion (Yura Babak) wrote:

Pathname and FileUtils work fine for me until there is a folder with a very long path (>260 chars).

…
But there is a way in Windows to remove the MAX_PATH limitation.
You can find a small .reg file in this article:
https://docs.microsoft.com/en-us/windows/win32/fileio/maximum-file-path-limitation?tabs=registry

It’s been a long time since I’ve done anything in Windows, but the article you posted indicates that there are two conditions that must be met for the long path support to be enabled:

• The registry key Computer\HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\FileSystem\LongPathsEnabled must exist and be set to 1.
• The application manifest must also include the longPathAware element.

If Ruby does not have an application manifest (I don’t know whether it does or not) with this element, then long path support is disabled. If the registry key is not set, then long path support is disabled. Thus, even if Ruby were to have an embedded application manifest with this key, then we’d want to have a runtime API so that scripts could determine whether or not they can safely use long paths.

There is a way to handle long paths on Windows whether or not this registry key is enabled: prefix the drive root with \\?\ (which would be '\\\\?\\' as ` is an escape character). It means you can’t work with relative paths and you must always use \ as the file separator, but it always works and definitely did when I was using Ruby on Windows back in 2004–2011.

[inversion (Yura Babak)]

austin (Austin Ziegler) wrote in #note-1:

If Ruby does not have an application manifest (I don’t know whether it does or not)

As I understand it was added here — [Win32] long path name support [Bug #12551]

There is a way to handle long paths on Windows whether or not this registry key is enabled: prefix the drive root with \\?\ (which would be '\\\\?\\' as ` is an escape character). It means you can’t work with relative paths and you must always use \ as the file separator, but it always works and definitely did when I was using Ruby on Windows back in 2004–2011.

It doesn't work for me, still .children raises an expectation for the existing folder:
<internal:dir>:98:in 'open': Filename too long @ dir_initialize - \\\\?\\D:\\very_loo…oong_path (Errno::ENAMETOOLONG)

[inversion (Yura Babak)]

Today rechecked that for the latest
ruby 3.2.2 (2023-03-30 revision e51014f9c0) [x64-mingw-ucrt]
still we have same problems (

[hsbt (Hiroshi SHIBATA)]

• Assignee set to windows

[hsbt (Hiroshi SHIBATA)]

There is another way that does not need the registry key. Go's runtime sets the undocumented IsLongPathAwareProcess bit (0x80 at PEB offset 3) at startup, which RtlAreLongPathsEnabled reads, so long paths work per-process regardless of LongPathsEnabled (os_windows.go, Windows 10.0.15063+).

I tried it on my machine with LongPathsEnabled=0 and 260+ character paths worked for File, Dir, Dir.glob, require, and even Dir.chdir, which \\?\ cannot handle.

Being undocumented is the obvious catch (golang/go#66560), and the 255 character limit per component stays.

[hsbt (Hiroshi SHIBATA)]

• Is duplicate of Bug #18923: Dir.glob Errno::ENAMETOOLONG - Caused by outdated logic in open_dir_handle (win32.c) added

[hsbt (Hiroshi SHIBATA)]

https://github.com/ruby/ruby/pull/18088

[hsbt (Hiroshi SHIBATA)]

Two questions before this goes further.

• The bit is set in rb_w32_sysinit, so only hosts that call ruby_sysinit get it, which effectively means ruby.exe. Embedders that do their own initialization keep the 260 character limit. Should I move it somewhere every embedder reaches?
• IsLongPathAwareProcess is undocumented. golang has set it since 1.23, but Microsoft's Go maintainers proposed removing it. The alternative is \\?\ prefixing at about 15 call sites, which still cannot fix Dir.chdir or require.

[hsbt (Hiroshi SHIBATA)]

One more limitation besides the ones above: spawning a child process from a 260+ character working directory still fails, since CreateProcessW keeps the MAX_PATH limit on lpCurrentDirectory.

I also audited every fixed-size buffer that receives a path from a Win32 API, since long results now actually flow into them. The relevant paths all allocate dynamically by probing first, such as getcwd, Dir.glob, and GetFullPathNameW in expand_path. WIN32_FIND_DATAW.cFileName stays MAX_PATH but holds a single component, which NTFS caps at 255 characters. The remaining fixed buffers feed the executable search in dln_find_1, which safely rejects oversized candidates as not found. I exercised File, Dir, Dir.glob, require, and Dir.chdir with 420 to 625 character paths and saw no crashes.

[matz (Yukihiro Matsumoto)]

Please go ahead with this approach. Relying on an undocumented bit is not ideal, but the failure mode decides it for me: if the bit ever stops working, we are back to today's 260 character limit, not broken. That is a cheap downside for fixing a four year old problem that the \\?\ alternative cannot fully fix anyway.

I leave the embedder question to you. Whichever way you go, please document the limitations you found in #note-9: the 255 character limit per path component, and that spawning a child process from a working directory longer than 260 characters still fails.

Matz.