Bug #1685

Some windows unicode path issues remain

Added by B Kelly almost 5 years ago. Updated 11 days ago.

[ruby-core:24010]
Status:Closed
Priority:Normal
Assignee:Usaku NAKAMURA
Category:M17N
Target version:next minor
ruby -v:ruby 1.9.2dev (2009-06-24) [i386-mswin32_71] Backport:

Description

=begin
Hi,

I see some nice progress has been made in unicode path
handling on windows.

The following tests are not exhaustive, but do reveal some
remaining issues.

Everything below "NOT WORKING" fails in one way or another.

Regards,

Bill

 # encoding: UTF-8

 # Test unicode path/dir handling on windows

 require 'test/unit'

 class TestUnicodeFilenamesAndPaths < Test::Unit::TestCase
   def setup
     tmpdir = ENV['TEMP'] || "C:/TEMP"
     Dir.chdir tmpdir
     puts Dir.pwd
     testdir = "ruby_unicode_test"
     Dir.mkdir testdir unless test ?d, testdir
     Dir.chdir testdir
     puts Dir.pwd
   end

   def test_unicode_paths
     fname_resume = "R\xC3\xA9sum\xC3\xA9".force_encoding("UTF-8")
     fname_chinese = "\u52ec\u52ee\u52f1\u52f2.txt"
     dname_chinese = "\u52ec\u52ee\u52f1\u52f2"

     assert_equal( "UTF-8", fname_resume.encoding.name )
     File.open(fname_resume, "w") {|io| io.puts "Hello, World"}

     assert_equal( "UTF-8", fname_chinese.encoding.name )
     File.open(fname_chinese, "w") {|io| io.puts "Hello, World"}

     dat = File.read(fname_chinese)
     assert_equal( "Hello, World\n", dat )

     files = Dir["*"]
     assert( files.include? fname_resume )
     assert( files.include? fname_chinese )

 # NOT WORKING:
     Dir.rmdir dname_chinese rescue nil
     Dir.mkdir dname_chinese
     test ?d, dname_chinese
     Dir.chdir dname_chinese
     cwd = Dir.pwd
     assert( cwd[(-dname_chinese.length)..-1] == dname_chinese )
     Dir.chdir ".."

     x = File.stat(fname_resume)
     x = File.stat(fname_chinese)
     x = File.stat(dname_chinese)

     assert( File.exist? fname_resume )
     assert( File.exist? fname_chinese )
     assert( test(?f, fname_resume) )
     assert( test(?f, fname_chinese) )

     files = Dir[fname_resume]
     assert_equal( fname_resume, files.first )
     files = Dir[fname_chinese]
     assert_equal( fname_chinese, files.first )
     files = Dir[dname_chinese]
     assert_equal( dname_chinese, files.first )
   end  
 end
=end

spatulasnout-unicode-mkdir-diffs.txt Magnifier - Diffs for win32 unicode support for Dir.mkdir (3.56 KB) B Kelly, 03/25/2010 10:13 AM

test_io_unicode_paths.rb Magnifier - new file: bootstraptest/test_io_unicode_paths.rb (925 Bytes) B Kelly, 03/25/2010 10:13 AM


Related issues

Related to ruby-trunk - Feature #2255: unicode parameters cannot be passed to ruby Assigned 10/22/2009
Related to ruby-trunk - Bug #2332: Ruby doesn't run properly from unicode folder on windows Closed 11/04/2009
Related to ruby-trunk - Bug #1771: system()/popen()/popen3() & windows & unicode is not working Closed 07/13/2009
Duplicated by ruby-trunk - Bug #2137: Dir.glob does not support unicode on Windows Closed 09/23/2009

Associated revisions

Revision 27570
Added by Usaku NAKAMURA almost 4 years ago

  • merge some patches from win32-uncode-test branch.
    see #1685.

  • file.c, include/ruby/intern.h (rbstrencode_ospath): new function
    to convert encoding for pathname.

  • win32.c, include/ruby/win32.h (rbw32ulink, rbw32urename,
    rbw32ustati64, rbw32uopen, rbw32uutime, rbw32uchdir,
    rbw32umkdir, rbw32urmdir, rbw32uunlink): new functions to
    accept UTF-8 path.

  • win32/win32.c (rbw32opendir, link, rbw32stati64, rbw32utime,
    rbw32unlink): use WCHAR path internally.

  • file.c (rbstat, eaccess, accessinternal, rbfilesftype,
    chmod
    internal, rbfilechmod, rbfilechown, utimeinternal,
    rb
    fileslink, unlinkinternal, rbfilesrename): use UTF-8 version
    functions on Win32.

  • file.c (apply2files, rbstat, rbfileslstat, rbfilesymlinkp,
    rb
    filereadablep, rbfilewritablep, rbfileexecutablep,
    check3rdbyte, rbfileidenticalp, rbfilechmod, rbfilechown,
    rb
    fileslink, rbfilessymlink, rbfilesrename): call
    rbstrencode_ospath() before passing the path to system.

  • io.c (rb_sysopen): ditto.

  • dir.c (dirchdir, dirsmkdir, dirs_rmdir): ditto.

History

#1 Updated by Yuki Sonoda almost 5 years ago

  • Status changed from Open to Assigned
  • Assignee set to Usaku NAKAMURA

=begin

=end

#2 Updated by Usaku NAKAMURA over 4 years ago

  • Priority changed from Normal to High

=begin

=end

#3 Updated by Vit Ondruch about 4 years ago

=begin
Is there any progress regarding this issue(s)?
=end

#4 Updated by B Kelly about 4 years ago

=begin
Hi,

I'll be needing win32 unicode path support for my current project, so I would like to try to tackle the remaining issues.

I started with a relatively easy one, Dir.mkdir

For Dir.mkdir, I took an approach similar to what was already in place for rbsysopen(), which is that it tries to call w32convtoutf16() on the path, and if it succeeds calls the new rbw32wmkdir() with the wide path; otherwise it falls back to calling the old rbw32mkdir().

Attached files should include the diffs, and a new file adding a bootstrap test for unicode paths. (The tests currently fail, because they need a working unicode stat and unlink in order to function.)

I'm planning to attempt File.stat next, but I have some questions about it so I'll post separately.

Regards,

Bill

=end

#5 Updated by Vit Ondruch about 4 years ago

=begin
Hello Bill,

Are you aware of win32unicodebranch? Its not up-to-date as far as I know, but there is lot of Unicode functionality covered. There is missing mainly Dir.glob functionality.

Vit
=end

#6 Updated by B Kelly about 4 years ago

=begin
Hi Vit,

Thanks. Wow. Good to know.

win32/win32.c:rbw32uchown(const char *path, int owner, int group)
win32/win32.c:rbw32ulink(const char *from, const char *to)
win32/win32.c:rbw32urename(const char *from, const char *to)
win32/win32.c:rbw32ustati64(const char *path, struct stati64 *st)
win32/win32.c:rbw32uaccess(const char *path, int mode)
win32/win32.c:rbw32uopen(const char *file, int oflag, ...)
win32/win32.c:rbw32uutime(const char *path, const struct utimbuf *times)
win32/win32.c:rbw32utime(const char *path, const struct utimbuf *times)
win32/win32.c:rbw32uchdir(const char *path)
win32/win32.c:rbw32umkdir(const char *path, int mode)
win32/win32.c:rbw32urmdir(const char *path)
win32/win32.c:rbw32uunlink(const char *path)
win32/win32.c:rbw32unlink(const char *path)
win32/win32.c:rbw32uchmod(const char *path, int mode)

And it looks like a much cleaner implementation than what
is currently in the 1.9.2 trunk. No more 'wchar' in
sysopenstruct, no more #ifdef _WIN32 surrounding
w32
convtoutf16 logic, just some defines at the top:

dir.c:#define chdir(p) rbw32uchdir(p)
dir.c:#define mkdir(p, m) rbw32umkdir(p, m)
dir.c:#define rmdir(p) rbw32urmdir(p)
file.c:#define STAT(p, s) rbw32ustati64(p, s)
file.c:#define lstat(p, s) rbw32ustati64(p, s)
file.c:#define access(p, m) rbw32uaccess(p, m)
file.c:#define chmod(p, m) rbw32uchmod(p, m)
file.c:#define chown(p, o, g) rbw32uchown(p, o, g)
file.c:#define utime(p, t) rbw32uutime(p, t)
file.c:#define link(f, t) rbw32ulink(f, t)
file.c:#define unlink(p) rbw32uunlink(p)
file.c:#define rename(f, t) rbw32urename(f, t)
io.c:#define open rbw32uopen

I wonder if there is a reason this should not be merged
into trunk ASAP?

Regards,

Bill

=end

#7 Updated by Usaku NAKAMURA about 4 years ago

=begin
Hello,

In message " [Bug #1685] Some windows unicode path issues remain"
on Mar.25,2010 19:10:35, redmine@ruby-lang.org wrote:

I wonder if there is a reason this should not be merged
into trunk ASAP?

Because I'm too busy to test this branch well :(

Endoh-san says that the feature freeze is March 31.
Then, it is necessary to complete merging it until then,
if we want to include it in 1.9.2 release...

win32-unicode-branch has not contained the globbing features
yet, as Vit pointed in ruby-core:28977.
However, because it relates to the command line interpretation,
it might be difficult to implement until March 31.
Should we wait until all functions are covered, or merge the
current one?

Summary:
(1) need the decision whether merging it or not
(2) need testers :)
(3) need the worker(s) to make the patch to trunk

Regards,
--
U.Nakamura usa@garbagecollect.jp

=end

#8 Updated by Vit Ondruch about 4 years ago

=begin
For me, it would be helpful to merge what we have now. I am not aware of any problematic parts with methods which are already implemented. However, I am aware that this can lead in confusion when not everything will work with unicode :(

Vit
=end

#9 Updated by B Kelly about 4 years ago

=begin
Hi,

U.Nakamura wrote:

Hello,

In message " [Bug #1685] Some windows unicode path issues remain"
on Mar.25,2010 19:10:35, redmine@ruby-lang.org wrote:

I wonder if there is a reason this should not be merged
into trunk ASAP?

Because I'm too busy to test this branch well :(

Endoh-san says that the feature freeze is March 31.
Then, it is necessary to complete merging it until then,
if we want to include it in 1.9.2 release...

win32-unicode-branch has not contained the globbing features
yet, as Vit pointed in ruby-core:28977.
However, because it relates to the command line interpretation,
it might be difficult to implement until March 31.

I understand how this might be considered a 'feature', but
I think it is also possible to consider it a bug fix.

1.9.1 was supposed to support unicode path on win32, but
this was deferred to 1.9.2.

Nevertheless, I quote matz from November, 2008:

Yukihiro Matsumoto wrote:

Hi,

In message "Re: Re: 1.9, encoding & win32 wide char support"
on Wed, 26 Nov 2008 12:26:53 +0900, "Bill Kelly" billk@cts.com writes:

|> Does anyone have information as to the current status of
|> adding Unicode-savvy path handling to 1.9 ruby?
|
|Ugh. Sorry, I mean of course: Unicode-savvy path handling
|on win32 ruby 1.9.

Every path encoding is UTF-8 and converted to UTF-16 internally. If
there's something still use *A functions, it will eventually replaced
by *W functions. In short, if you're using UTF-8 for your program
encoding, you should not see any problem (if you do, it's a bug).

                                                  matz.

I don't know if matz has changed his mind, but; personally I would
like to consider it a bug that ruby 1.9.x fails for unicode paths
on windows.

Should we wait until all functions are covered, or merge the
current one?

Summary:
(1) need the decision whether merging it or not
(2) need testers :)
(3) need the worker(s) to make the patch to trunk

(1) Please, yes. Let us merge. 93.75% is better than current 6.25% coverage.
(2) I hope to contribute unicode_path unit-tests. (such as in bootstraptest/)
(3) I would like to contribute to the patch if my efforts can be useful.
(diffs on io.c, file.c, and dir.c look pretty straightforward.)
(diffs on win32/win32.c look more difficult, but I can attempt.)

Regards,

Bill

=end

#10 Updated by Usaku NAKAMURA about 4 years ago

=begin
Hello,

In message " Re: [Bug #1685] Some windows unicode path issues remain"
on Mar.28,2010 16:51:26, billk@cts.com wrote:

I understand how this might be considered a 'feature', but
I think it is also possible to consider it a bug fix.

Hmm, it has a point in it.
The branch manager should judge whether this change is a bug fix or
feature change.
How do you think, Yugui-san?

Summary:
(1) need the decision whether merging it or not
(2) need testers :)
(3) need the worker(s) to make the patch to trunk

(1) Please, yes. Let us merge. 93.75% is better than current 6.25% coverage.
(2) I hope to contribute unicode_path unit-tests. (such as in bootstraptest/)
(3) I would like to contribute to the patch if my efforts can be useful.
(diffs on io.c, file.c, and dir.c look pretty straightforward.)
(diffs on win32/win32.c look more difficult, but I can attempt.)

I'm very glad to hear your offer of cooperation.
Thank you!

Regards,
--
U.Nakamura usa@garbagecollect.jp

=end

#11 Updated by B Kelly about 4 years ago

=begin
U.Nakamura wrote:

Hello,

In message " Re: [Bug #1685] Some windows unicode path issues remain"
on Mar.28,2010 16:51:26, billk@cts.com wrote:

I understand how this might be considered a 'feature', but
I think it is also possible to consider it a bug fix.

Hmm, it has a point in it.
The branch manager should judge whether this change is a bug fix or
feature change.
How do you think, Yugui-san?

Any word?

Regards,

Bill

=end

#12 Updated by Yuki Sonoda about 4 years ago

=begin

The branch manager should judge whether this change is a bug fix or
feature change.
How do you think, Yugui-san?

It's a bug fix.
=end

#13 Updated by B Kelly about 4 years ago

=begin
Yuki Sonoda wrote:

Issue #1685 has been updated by Yuki Sonoda.

The branch manager should judge whether this change is a bug fix or
feature change.
How do you think, Yugui-san?

It's a bug fix.

Wonderful news!

Thank you,

Bill

=end

#14 Updated by B Kelly almost 4 years ago

=begin
Hi,

Bill Kelly wrote:

Yuki Sonoda wrote:

Issue #1685 has been updated by Yuki Sonoda.

The branch manager should judge whether this change is a bug fix or
feature change.
How do you think, Yugui-san?

It's a bug fix.

Wonderful news!

In order to avoid duplication of effort, I wanted to inquire
whether anyone else may currently be working on Windows
Unicode related code?

U.Nakamura wrote:

(2) need testers :)
(3) need the worker(s) to make the patch to trunk

If there is no conflict with others' work, I would like to
attempt merging the win32-unicode branch into trunk within
the next week or two.

Regards,

Bill

=end

#15 Updated by Usaku NAKAMURA almost 4 years ago

=begin
Hello,

In message " Re: [Bug #1685] Some windows unicode path issues remain"
on Apr.30,2010 08:12:33, billk@cts.com wrote:
| In order to avoid duplication of effort, I wanted to inquire
| whether anyone else may currently be working on Windows
| Unicode related code?
|
|
| U.Nakamura wrote:
| >
| > (2) need testers :)
| > (3) need the worker(s) to make the patch to trunk
|
|
| If there is no conflict with others' work, I would like to
| attempt merging the win32-unicode branch into trunk within
| the next week or two.

Ah, I've merged most parts of win32-unicode-test branch because
the time limit of code freeze (Apr.30) has come.
# See r27570

Of course, test cases and bug reports are welcomed.

Regards
--
U.Nakamura usa@garbagecollect.jp

=end

#16 Updated by B Kelly almost 4 years ago

=begin
U.Nakamura wrote:

Hello,

In message " Re: [Bug #1685] Some windows unicode path issues remain"
on Apr.30,2010 08:12:33, billk@cts.com wrote:
|
| If there is no conflict with others' work, I would like to
| attempt merging the win32-unicode branch into trunk within
| the next week or two.

Ah, I've merged most parts of win32-unicode-test branch because
the time limit of code freeze (Apr.30) has come.

See r27570

Oh! Thank you very much!

(I had thought the code freeze applied to new features, rather
than bug fixes.)

Of course, test cases and bug reports are welcomed.

My initial attempt at a bootstraptest for unicode path
support is failing.

It is incomplete, but I uploaded the current version:

http://redmine.ruby-lang.org/attachments/download/910

It is failing at:

DNAMECHINESE = "\u52ec\u52ee\u52f1\u52f2"
Dir.mkdir DNAME
CHINESE
test(?d, DNAME_CHINESE) or raise "test ?d fail"

It seems rb_stat in file.c calls stat(), but stat does
not map to the unicode version.

win32.h:

#define stat(path,st) rbw32stat(path,st)

file.c:

static int
rb_stat(VALUE file, struct stat *st)
{
VALUE tmp;

 rb_secure(2);
 tmp = rb_check_convert_type(file, T_FILE, "IO", "to_io");
 if (!NIL_P(tmp)) {
rb_io_t *fptr;

GetOpenFile(tmp, fptr);
return fstat(fptr->fd, st);
 }
 FilePathValue(file);
 file = rb_str_encode_ospath(file);
 return stat(StringValueCStr(file), st);

}

Regards,

Bill

=end

#17 Updated by Usaku NAKAMURA almost 4 years ago

=begin
Hello,

In message " Re: [Bug #1685] Some windows unicode path issues remain"
on May.05,2010 15:35:11, billk@cts.com wrote:
| My initial attempt at a bootstraptest for unicode path
| support is failing.
|
| It is incomplete, but I uploaded the current version:
|
| http://redmine.ruby-lang.org/attachments/download/910
|
| It is failing at:
|
| DNAMECHINESE = "\u52ec\u52ee\u52f1\u52f2"
| Dir.mkdir DNAME
CHINESE
| test(?d, DNAMECHINESE) or raise "test ?d fail"
|
|
| It seems rb
stat in file.c calls stat(), but stat does
| not map to the unicode version.

Oops, thank you!

Regards
--
U.Nakamura usa@garbagecollect.jp

=end

#18 Updated by B Kelly almost 4 years ago

=begin
U.Nakamura wrote:

In message " Re: [Bug #1685] Some windows unicode path issues remain"
on May.05,2010 15:35:11, billk@cts.com wrote:
|
| It seems rb_stat in file.c calls stat(), but stat does
| not map to the unicode version.

Oops, thank you!

Thanks, the test gets much further now.

It now fails at the last line:

Dir.chdir DNAMECHINESE
cwd = Dir.pwd
( cwd[(-DNAME
CHINESE.length)..-1] == DNAME_CHINESE ) or raise "cwd check fail"

Currently there was only rbw32getcwd. I have added a unicode
rbw32ugetcwd:

 Index: include/ruby/win32.h
 ===================================================================
 --- include/ruby/win32.h   (revision 27644)
 +++ include/ruby/win32.h   (working copy)
 @@ -254,6 +254,7 @@
  extern struct servent  *WSAAPI rb_w32_getservbyport(int, const char *);
  extern int    rb_w32_socketpair(int, int, int, int *);
  extern char * rb_w32_getcwd(char *, int);
 +extern char * rb_w32_ugetcwd(char *, int);
  extern char * rb_w32_getenv(const char *);
  extern int    rb_w32_rename(const char *, const char *);
  extern int    rb_w32_urename(const char *, const char *);
 @@ -611,7 +612,7 @@
  #define get_osfhandle(h)  rb_w32_get_osfhandle(h)

  #undef getcwd
 -#define getcwd(b, s)      rb_w32_getcwd(b, s)
 +#define getcwd(b, s)      rb_w32_ugetcwd(b, s)

  #undef getenv
  #define getenv(n)     rb_w32_getenv(n)
 Index: win32/win32.c
 ===================================================================
 --- win32/win32.c  (revision 27644)
 +++ win32/win32.c  (working copy)
 @@ -3692,6 +3692,57 @@
      return p;
  }

 +char *
 +rb_w32_ugetcwd(char *buffer, int size)
 +{
 +    char *p;
 +    WCHAR *wp;
 +    long len, wlen;
 +
 +    wlen = GetCurrentDirectoryW(0, NULL);  // wlen includes null terminating character
 +    if (!wlen) {
 +  errno = map_errno(GetLastError());
 +  return NULL;
 +    }
 +
 +    wp = malloc(wlen * sizeof(WCHAR));
 +    if (!wp) {
 +  errno = ENOMEM;
 +  return NULL;
 +    }
 +
 +    if (!GetCurrentDirectoryW(wlen, wp)) {
 +  errno = map_errno(GetLastError());
 +  free(wp);
 +        return NULL;
 +    }
 +
 +    p = wstr_to_utf8(wp, &len);
 +    free(wp);
 +    len += 1;  // len now includes null terminating character
 +
 +    if (!p) {
 +  errno = ENOMEM;
 +  return NULL;
 +    }
 +
 +    if (buffer) {
 +  if (size < len) {
 +      free(p);
 +      errno = ERANGE;
 +      return NULL;
 +  }
 +
 +  memcpy(buffer, p, len);
 +  free(p);
 +  p = buffer;
 +    }
 +
 +    translate_char(p, '\\', '/');
 +
 +    return p;
 +}
 +
  int
  chown(const char *path, int owner, int group)
  {

This works, in terms of returning a UTF-8 path string; however,
rbdirgetwd calls rbencassociate(cwd, rbfilesystemencoding())
on the result, associating the WINDOWS-1252 encoding instead of
UTF-8.

So, I would like to ask: is there a reason
encsetfilesystem_encoding() should not return UTF-8 now for
Windows?

static int
encsetfilesystemencoding(void)
{
int idx;
#if defined NO
LOCALECHARMAP
idx = rb
enctoindex(rbdefaultexternalencoding());
#elif defined _WIN32 || defined _
CYGWIN__
char cp[sizeof(int) * 8 / 3 + 4];
snprintf(cp, sizeof cp, "CP%d", AreFileApisANSI() ? GetACP() : GetOEMCP());
idx = rbencfindindex(cp);
if (idx < 0) idx = rb
ascii8bitencindex();
#else
idx = rb
enctoindex(rbdefaultexternal_encoding());
#endif

 enc_alias_internal("filesystem", idx);
 return idx;

}

It seems strange that it still selects non-unicode encodings.


Also, my bootstraptest encountered one more problem. The mktmpdir
can't delete the unicode directory entries created by my test:

P:/code/ruby-svn/trunk/lib/fileutils.rb:1307:in unlink': Invalid argument - C:/temp/bootstraptest20100505-1016-1lvss6a.tmpwd/???? (Errno::EINVAL)
from P:/code/ruby-svn/trunk/lib/fileutils.rb:1307:in
block in removefile'
from P:/code/ruby-svn/trunk/lib/fileutils.rb:1315:in platform_support'
from P:/code/ruby-svn/trunk/lib/fileutils.rb:1306:in
remove
file'
from P:/code/ruby-svn/trunk/lib/fileutils.rb:1295:in remove'
from P:/code/ruby-svn/trunk/lib/fileutils.rb:761:in
block in removeentry'
from P:/code/ruby-svn/trunk/lib/fileutils.rb:1345:in block (2 levels) in postorder_traverse'
from P:/code/ruby-svn/trunk/lib/fileutils.rb:1349:in
postorder
traverse'
from P:/code/ruby-svn/trunk/lib/fileutils.rb:1344:in block in postorder_traverse'
from P:/code/ruby-svn/trunk/lib/fileutils.rb:1343:in
each'
from P:/code/ruby-svn/trunk/lib/fileutils.rb:1343:in postorder_traverse'
from P:/code/ruby-svn/trunk/lib/fileutils.rb:759:in
removeentry'
from P:/code/ruby-svn/trunk/lib/fileutils.rb:688:in remove_entry_secure'
from P:/code/ruby-svn/trunk/lib/tmpdir.rb:85:in
ensure in mktmpdir'
from P:/code/ruby-svn/trunk/lib/tmpdir.rb:85:in mktmpdir'
from ./bootstraptest/runner.rb:375:in
in
temporaryworkingdirectory'
from ./bootstraptest/runner.rb:126:in main'
from ./bootstraptest/runner.rb:398:in
'

I don't have a patch for this yet. However, it looks like
in win32.c, routines such as rbw32opendir and rbw32readdirwithenc
are already using WCHAR internally!

For example:

DIR *
rbw32opendir(const char *filename)
{
struct stati64 sbuf;
WIN32FINDDATAW fd;
HANDLE fh;
WCHAR *wpath;

 if (!(wpath = filecp_to_wstr(filename, NULL)))
return NULL;

... so it seems if filesystem encoding were considered UTF-8
instead of WINDOWS-1252, then opendir might just work.

Similarly (somewhat) with rbw32readdirwithenc. (At least,
it does call readdir_internal, which uses WCHAR.)

So I think these are very close to working UTF-8, but, again,
I don't understand why encsetfilesystem_encoding() uses
WINDOWS-1252 still.

Thanks,

Regards,

Bill

=end

#19 Updated by Usaku NAKAMURA almost 4 years ago

=begin
Hello,

In message " Re: [Bug #1685] Some windows unicode path issues remain"
on May.06,2010 19:39:27, billk@cts.com wrote:

This works, in terms of returning a UTF-8 path string; however,
rbdirgetwd calls rbencassociate(cwd, rbfilesystemencoding())
on the result, associating the WINDOWS-1252 encoding instead of
UTF-8.

So, I would like to ask: is there a reason
encsetfilesystem_encoding() should not return UTF-8 now for
Windows?

For compatibility.

I will not change filesystem encoding in Windows in 1.9 series.
In all methods which returns filenames, the default encoding
of returned value must be filesystem encoding.
So, if someone want to get filename with another encoding, he/she
should specify the encoding by some way.
Of course, it is necessary to decide the "some way" of each
methods.

Also, my bootstraptest encountered one more problem. The mktmpdir
can't delete the unicode directory entries created by my test:

Yes, I know it.
This is the problem of globbing.
I've already decided to solve this problem 1.9.3 or later.

Regards,
--
U.Nakamura usa@garbagecollect.jp

=end

#20 Updated by B Kelly almost 4 years ago

=begin
Hi,

U.Nakamura wrote:

In message " Re: [Bug #1685] Some windows unicode path issues remain"
on May.06,2010 19:39:27, billk@cts.com wrote:

This works, in terms of returning a UTF-8 path string; however,
rbdirgetwd calls rbencassociate(cwd, rbfilesystemencoding())
on the result, associating the WINDOWS-1252 encoding instead of
UTF-8.

So, I would like to ask: is there a reason
encsetfilesystem_encoding() should not return UTF-8 now for
Windows?

For compatibility.

I will not change filesystem encoding in Windows in 1.9 series.
In all methods which returns filenames, the default encoding
of returned value must be filesystem encoding.
So, if someone want to get filename with another encoding, he/she
should specify the encoding by some way.
Of course, it is necessary to decide the "some way" of each
methods.

Ah.

So my rbw32ugetcwd patch is not very useful, at present,
since there is no "some way" to specify the encoding via
Dir.pwd.

May I suggest a new command line flag for this purpose:

ruby --DEARGODWORKWITHUTF8DAMN_IT

;)

Well then, this becomes a philosophical question at this point,
but in an effort to better understand, I am wondering:

How does it break compatibility, if we allow filesystem encoding
to become UTF-8 when rbdefaultexternal_encoding is UTF-8?

Do we have evidence that anyone has written scripts that will
break in such a case? (And if so, can we agree to summon the
fleas of a thousand camels to infest their undergarments?)

Also, my bootstraptest encountered one more problem. The mktmpdir
can't delete the unicode directory entries created by my test:

Yes, I know it.
This is the problem of globbing.
I've already decided to solve this problem 1.9.3 or later.

OK.

I admit I don't understand why it's considered a globbing problem.
Does the UTF-8 support somehow make the globbing more difficult?
I thought it was just the same situation as above: a filesystem
encoding problem?

Regards,

Bill

=end

#21 Updated by Usaku NAKAMURA almost 4 years ago

=begin
Hello,

In message " Re: [Bug #1685] Some windows unicode path issues remain"
on May.06,2010 21:38:19, billk@cts.com wrote:

For compatibility.

I will not change filesystem encoding in Windows in 1.9 series.
In all methods which returns filenames, the default encoding
of returned value must be filesystem encoding.
So, if someone want to get filename with another encoding, he/she
should specify the encoding by some way.
Of course, it is necessary to decide the "some way" of each
methods.

Ah.

So my rbw32ugetcwd patch is not very useful, at present,
since there is no "some way" to specify the encoding via
Dir.pwd.

Unfortunately...

Well then, this becomes a philosophical question at this point,
but in an effort to better understand, I am wondering:

How does it break compatibility, if we allow filesystem encoding
to become UTF-8 when rbdefaultexternal_encoding is UTF-8?

You should advocate using defaultinternal instead of
default
external :)
It's acceptable for me.

I admit I don't understand why it's considered a globbing problem.

FileUtils uses Dir.entries to get filenames to remove.

Does the UTF-8 support somehow make the globbing more difficult?
I thought it was just the same situation as above: a filesystem
encoding problem?

Yes, you are right.

Regards,
--
U.Nakamura usa@garbagecollect.jp

=end

#22 Updated by B Kelly almost 4 years ago

=begin
Hi,

U.Nakamura wrote:

How does it break compatibility, if we allow filesystem encoding
to become UTF-8 when rbdefaultexternal_encoding is UTF-8?

You should advocate using defaultinternal instead of
default
external :)
It's acceptable for me.

Ah, thanks, default_internal does make more sense. :)

Regarding advocacy: apart from yourself, who are the people
who need to comment on this? Is this a question for Matz?
Or... ?

Thanks,

Bill

=end

#23 Updated by Usaku NAKAMURA almost 4 years ago

=begin
Hello,

In message " Re: [Bug #1685] Some windows unicode path issues remain"
on May.07,2010 06:11:00, billk@cts.com wrote:

Regarding advocacy: apart from yourself, who are the people
who need to comment on this? Is this a question for Matz?
Or... ?

I assume,

1) matz: the grand designer of Ruby
2) naruse: an authority of Ruby M17N
3) me: main maintainer of Ruby on Windows
4) all users of Ruby, of course, especially people using
non-unicode (multibyte) environment

Regards,
--
U.Nakamura usa@garbagecollect.jp

=end

#24 Updated by B Kelly almost 4 years ago

=begin
Hi,

U.Nakamura wrote:

I assume,

1) matz: the grand designer of Ruby
2) naruse: an authority of Ruby M17N
3) me: main maintainer of Ruby on Windows
4) all users of Ruby, of course, especially people using
non-unicode (multibyte) environment

For #1 (sorry, I can't resist :)

http://blade.nagaokaut.ac.jp/cgi-bin/scat.rb/ruby/ruby-core/20110

For #4, wouldn't we expect people using a non-unicode environment
to not set their default_internal encoding to UTF-8 ? So I
would think they would not be affected.


As an aside, from the point of view of writing ruby software to
be installed on arbitrary user's machines, I don't think there's
any such thing as a non-unicode environment anymore.

The minute any of my users (even if they are English speaking)
downloads a file from their web browser called
私のホバークラフトは鰻でいっぱいです.mpg
my application which uses Dir.entries to locate and catalog
media files, is now broken on their system.

(Of course, since I distribute the Ruby interpreter with my
application, I have the luxury of working around the problem,
by installing a non-standard Ruby. But I still believe it's
important for standard Ruby to have full Unicode support on
Windows.)

Regards,

Bill

=end

#25 Updated by B Kelly almost 4 years ago

=begin
Hi,

Bill Kelly wrote:

1) matz: the grand designer of Ruby
2) naruse: an authority of Ruby M17N
3) me: main maintainer of Ruby on Windows
4) all users of Ruby, of course, especially people using
non-unicode (multibyte) environment

For #4, wouldn't we expect people using a non-unicode environment
to not set their default_internal encoding to UTF-8 ? So I
would think they would not be affected.

Noticed an interesting ChangeLog entry from yesterday on
ruby19_2 branch:

Mon May 17 11:09:58 2010 NAKAMURA Usaku usa@ruby-lang.org

     merge from trunk (r27856, r27857)

     * lib/fileutils.rb (FileUtils::Entry_#entries): returns pathname in
       UTF-8 on Windows to allow FileUtils accessing all pathnames
       internally.

Index: lib/fileutils.rb
===================================================================
--- lib/fileutils.rb (revision 27657)
+++ lib/fileutils.rb (working copy)
@@ -1176,7 +1176,9 @@
end

  def entries
  • Dir.entries(path())\
  • opts = {}
  • opts[:encoding] = "UTF-8" if /mswin|mignw/ =~ RUBY_PLATFORM
  •  Dir.entries(path(), opts)\
        .reject {|n| n == '.' or n == '..' }\
        .map {|n| Entry_.new(prefix(), join(rel(), n.untaint)) }
    

    end

    Would this approach also be considered for Dir.pwd:

    result = Dir.pwd(:encoding => "UTF-8")

    ?

    If so, I already have the rbw32ugetcwd implementation (presented
    in ).

    I would be happy to provide a patch for Dir.pwd if this is
    acceptable.

    Regards,

    Bill

=end

#26 Updated by Usaku NAKAMURA almost 4 years ago

=begin
Hello,

In message " Re: [Bug #1685] Some windows unicode path issues remain"
on May.18,2010 19:30:53, billk@cts.com wrote:

Noticed an interesting ChangeLog entry from yesterday on
ruby19_2 branch:

Mon May 17 11:09:58 2010 NAKAMURA Usaku usa@ruby-lang.org

    merge from trunk (r27856, r27857)

    * lib/fileutils.rb (FileUtils::Entry_#entries): returns pathname in
      UTF-8 on Windows to allow FileUtils accessing all pathnames
      internally.

In this case, Dir.entries already has its own "some way".
So I can use it.

Would this approach also be considered for Dir.pwd:

result = Dir.pwd(:encoding => "UTF-8")

?

This might be a moot point.
For instance, there might be an insistence that Dir.pwd
should accept only the encoding because there is no possiblity
that it takes other arguments.

Regards,
--
U.Nakamura usa@garbagecollect.jp

=end

#27 Updated by B Kelly almost 4 years ago

=begin
Hi,

U.Nakamura wrote:

In message " Re: [Bug #1685] Some windows unicode path issues remain"
on May.18,2010 19:30:53, billk@cts.com wrote:

Would this approach also be considered for Dir.pwd:

result = Dir.pwd(:encoding => "UTF-8")

?

This might be a moot point.
For instance, there might be an insistence that Dir.pwd
should accept only the encoding because there is no possiblity
that it takes other arguments.

Any solution would be fine with me. :)

Thanks to your finding a solution for Dir.entries, it seems
we are approaching nearly 100% unicode path capability for
win32!

Do you anticipate it will be difficult to reach a decision
regarding:

result = Dir.pwd(:encoding => "UTF-8")
vs.
result = Dir.pwd("UTF-8")
vs.
(some other way)

?

Thanks,

Regards,

Bill

=end

#28 Updated by Usaku NAKAMURA almost 4 years ago

  • Category changed from core to M17N
  • Priority changed from High to Normal
  • Target version changed from 1.9.2 to 2.0.0

=begin

=end

#29 Updated by Yusuke Endoh over 1 year ago

  • Description updated (diff)

Usa-san, what's the status?

Yusuke Endoh mame@tsg.ne.jp

#30 Updated by Yusuke Endoh about 1 year ago

  • Target version changed from 2.0.0 to next minor

Usa-san, what's the status?

Yusuke Endoh mame@tsg.ne.jp

#31 Updated by Thomas Thomassen 16 days ago

B Kelly wrote:

=begin
Thanks to your finding a solution for Dir.entries, it seems
we are approaching nearly 100% unicode path capability for
win32!
=end

In Ruby 2.0 there appear to still be several issues with Ruby and Unicode characters in filenames. Dir.entries fail, load and require fail. FILE has the wrong encoding. I see some things slated for Ruby 2.2, but not everything.

#32 Updated by Martin Dürst 15 days ago

On 2014/04/03 23:07, thomas@thomthom.net wrote:

Issue #1685 has been updated by Thomas Thomassen.

In Ruby 2.0 there appear to still be several issues with Ruby and Unicode characters in filenames. Dir.entries fail, load and require fail. FILE has the wrong encoding. I see some things slated for Ruby 2.2, but not everything.

If you know of anything that's not yet in Ruby 2.2, please tell us, best
by opening a bug for each issue.

Regards, Martin.


Bug #1685: Some windows unicode path issues remain
https://bugs.ruby-lang.org/issues/1685#change-46061

  • Author: B Kelly
  • Status: Assigned
  • Priority: Normal
  • Assignee: Usaku NAKAMURA
  • Category: M17N
  • Target version: next minor
  • ruby -v: ruby 1.9.2dev (2009-06-24) [i386-mswin32_71]

* Backport:

=begin
Hi,

I see some nice progress has been made in unicode path
handling on windows.

The following tests are not exhaustive, but do reveal some
remaining issues.

Everything below "NOT WORKING" fails in one way or another.

Regards,

Bill

  # encoding: UTF-8

  # Test unicode path/dir handling on windows

  require 'test/unit'

  class TestUnicodeFilenamesAndPaths < Test::Unit::TestCase
    def setup
      tmpdir = ENV['TEMP'] || "C:/TEMP"
      Dir.chdir tmpdir
      puts Dir.pwd
      testdir = "ruby_unicode_test"
      Dir.mkdir testdir unless test ?d, testdir
      Dir.chdir testdir
      puts Dir.pwd
    end

    def test_unicode_paths
      fname_resume = "R\xC3\xA9sum\xC3\xA9".force_encoding("UTF-8")
      fname_chinese = "\u52ec\u52ee\u52f1\u52f2.txt"
      dname_chinese = "\u52ec\u52ee\u52f1\u52f2"

      assert_equal( "UTF-8", fname_resume.encoding.name )
      File.open(fname_resume, "w") {|io| io.puts "Hello, World"}

      assert_equal( "UTF-8", fname_chinese.encoding.name )
      File.open(fname_chinese, "w") {|io| io.puts "Hello, World"}

      dat = File.read(fname_chinese)
      assert_equal( "Hello, World\n", dat )

      files = Dir["*"]
      assert( files.include? fname_resume )
      assert( files.include? fname_chinese )

  # NOT WORKING:
      Dir.rmdir dname_chinese rescue nil
      Dir.mkdir dname_chinese
      test ?d, dname_chinese
      Dir.chdir dname_chinese
      cwd = Dir.pwd
      assert( cwd[(-dname_chinese.length)..-1] == dname_chinese )
      Dir.chdir ".."

      x = File.stat(fname_resume)
      x = File.stat(fname_chinese)
      x = File.stat(dname_chinese)

      assert( File.exist? fname_resume )
      assert( File.exist? fname_chinese )
      assert( test(?f, fname_resume) )
      assert( test(?f, fname_chinese) )

      files = Dir[fname_resume]
      assert_equal( fname_resume, files.first )
      files = Dir[fname_chinese]
      assert_equal( fname_chinese, files.first )
      files = Dir[dname_chinese]
      assert_equal( dname_chinese, files.first )
    end
  end
=end


---Files--------------------------------
spatulasnout-unicode-mkdir-diffs.txt (3.56 KB)
test_io_unicode_paths.rb (925 Bytes)


#33 Updated by Thomas Thomassen 15 days ago

Martin Dürst wrote:

If you know of anything that's not yet in Ruby 2.2, please tell us, best
by opening a bug for each issue.

I've been setting up tests and running them through Ruby 2.2 I find some are fixed but there is still several issues related to file handling. We'll be filing issues for what we have uncovered.

#34 Updated by Usaku NAKAMURA 11 days ago

  • Status changed from Assigned to Closed

This ticket is too old and too various problems.
Now Thomas investigates many things and is making some new tickets. (Thank you!)
Please refer to them from now on.

Also available in: Atom PDF