[master] UnicodeDecodeError when processing paths that have the non-ascii characters

I can reproduce this bug. It also occurs if the folder name includes a "ü", a "ö" or a "ß". I tested it initializing the folder as a "standalone tree".

Could you please reproduce this again, then upload your .bzr.log file? The log file location can be found from "About" in Tortoise, or by executing "bzr --version"

Initialized D:\TEMP\ä and tried to commit. Log output attached.

It also occurs with other characters:

á

for example

i think, any character not belonging tho the standard ascii will throw this error

never tried with a folder named in chinese

I'll do and if the error happens i'll report it too

Ok, I created the following folder:

K:\天龙

If I try to create a repository there via TortoiseBZR I can't, TortoiseBZR shows no option.

I restarted TortoiseBZR and tried again with no luck.

I think the problem is TortoiseBZR can't read some file/folder names, maybe the ones in unicode or other character encodings?

I'll try vía command line and say if it works.

Trying to create a repository inside this folder
K:\天龙
fails from command line too.

I think i need to make also a bug for this commandline error too.

I think the problem relies in the fact Windows GUI is not using ASCII codepage but a different one, not exactly sure what is on Windows XP, but for sure is not ASCII.

Tthen, the error could be we are sending a wrong parameter vía the Windows GUI to bazaar tools when we try to launch them with TortoiseBZR... sending the path from Windows codepage and Bazaar trying to read it in ASCII codepage messing up the non-ASCII characters...

Maybe this link could be useful to resolve this issue:
http://docs.python.org/whatsnew/2.3.html#pep-277-unicode-file-name-support-for-windows-nt

And this other one:
http://mail.python.org/pipermail/python-dev/2003-February/033310.html

Attached patch is decode filenames with locale.getpreferredencoding().
Please try it, anyone.

Sorry, a little question, how to put this patch into an existing windows bazaar? there's no scripts folder here

Looking at the bzr log, it seems that 'bzr init' works fine:

0.437 creating repository in file:///D:/TEMP/%C3%A4/.bzr/.

^- that is a valid URL (using UTF-8 + urlquoting)

v- Hmm.. this is a UTF-8 path. I probably would have expected Unicode, but potentially this has to do with us writing to the log file.

0.672 opening working tree 'D:/TEMP/\xc3\xa4'
0.969 opening working tree 'D:/TEMP/\xc3\xa4'
1.062 return code 0
1533.813 opening working tree 'D:/TEMP/\xc3\xa4'
1533.813 opening working tree 'D:/TEMP/\xc3\xa4'

^- 1500 seconds

Looking at the second half:
  File "tbzrcommand.py", line 177, in invoke_command
  File "D:/ohjelmistot/Bazaar/plugins\qbzr\lib\commands.py", line 125, in run
  File "D:/ohjelmistot/Bazaar/plugins\qbzr\lib\commands.py", line 112, in run
  File "D:/ohjelmistot/Bazaar/plugins\qbzr\lib\commands.py", line 141, in run
  File "D:/ohjelmistot/Bazaar/plugins\qbzr\lib\commands.py", line 308, in _qbzr_run
  File "bzrlib\builtins.pyo", line 70, in tree_files
  File "bzrlib\builtins.pyo", line 119, in internal_tree_files
  File "bzrlib\osutils.pyo", line 297, in _win32_realpath

^- It would appear that "_qbzr_run" is passing a plain "str" to tree_files, rather than passing in a Unicode string, and that is causing us to try to cast it into Unicode.

I think the correct fix would be in "qbzr\lib\commands.py" to have it properly handle non-ascii, and pass it down appropriately. It seemed to get it right for "init" so it seems to just need to be updated for whatever this next step is. (My guess is 'status')

bzrlib.commands.main() decodes sys.argv like:
        user_encoding = osutils.get_user_encoding()
        argv = [a.decode(user_encoding) for a in argv[1:]]

This patch follows the way.

But, Windows locale encoding is not support all of unicode chars.
To support Unicode filenames, I can think following ways.
 1. Use Py3k ;-)
 2. Use ctypes.GetCommandLineW()
 3. Use urlencode for filenames. ( file:///D:/%uXXXX...)
 4. pass filenames with stdin instead of commandline arguments.

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

INADA Naoki wrote:
> bzrlib.commands.main() decodes sys.argv like:
> user_encoding = osutils.get_user_encoding()
> argv = [a.decode(user_encoding) for a in argv[1:]]
>
> This patch follows the way.
>
>
> But, Windows locale encoding is not support all of unicode chars.
> To support Unicode filenames, I can think following ways.
> 1. Use Py3k ;-)
> 2. Use ctypes.GetCommandLineW()
> 3. Use urlencode for filenames. ( file:///D:/%uXXXX...)
> 4. pass filenames with stdin instead of commandline arguments.
>
> ** Attachment added: "decode filenames with get_user_encoding()"
> http://launchpadlibrarian.net/27783268/fenc2.patch
>

The way we get the command line has changed in recent versions. We now
use GetCommandLineW function (IIRC), and we expect the arguments passed
to main() are going to be Unicode arguments.

Also, I don't know what qbzr() is doing, but it should be calling
run_bzr() not main(). (main would override the terminal settings, etc.)
And run_bzr() has always been defined as taking Unicode strings.

John
=:->
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Cygwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAkoxaBsACgkQJdeBCYSNAAPykQCgh1Cxdera/CItIqERHWdorz7Z
UoUAnA+w7kA6JG0UWdYYsBbNIdVsdQfe
=Yb3V
-----END PGP SIGNATURE-----

John A Meinel пишет:
> INADA Naoki wrote:
>> bzrlib.commands.main() decodes sys.argv like:
>> user_encoding = osutils.get_user_encoding()
>> argv = [a.decode(user_encoding) for a in argv[1:]]
>
>> This patch follows the way.
>
>
>> But, Windows locale encoding is not support all of unicode chars.
>> To support Unicode filenames, I can think following ways.
>> 1. Use Py3k ;-)
>> 2. Use ctypes.GetCommandLineW()
>> 3. Use urlencode for filenames. ( file:///D:/%uXXXX...)
>> 4. pass filenames with stdin instead of commandline arguments.
>
>> ** Attachment added: "decode filenames with get_user_encoding()"
>> http://launchpadlibrarian.net/27783268/fenc2.patch
>
>
> The way we get the command line has changed in recent versions. We now
> use GetCommandLineW function (IIRC), and we expect the arguments passed
> to main() are going to be Unicode arguments.
>
> Also, I don't know what qbzr() is doing, but it should be calling
> run_bzr() not main(). (main would override the terminal settings, etc.)
> And run_bzr() has always been defined as taking Unicode strings.

I'm not sure which one part of qbzr is under suspicious, but here is how
we working with bzrlib from qsubprocess:

    def run(self, cmd):
        argv = [p.decode('utf8') for p in shlex.split(cmd.encode('utf8'))]
        commands.run_bzr(argv)

>
> John
> =:->

On Thu, Jun 11, 2009 at 12:19 AM, cablop <email address hidden> wrote:

> It also occurs with other characters:
>
> á
>
> for example

That one is especially nasty because it represents a non-breaking space in
some codepages.

English speaking Windows XP users are probably using codepage 437.

>> But, Windows locale encoding is not support all of unicode chars.
>> To support Unicode filenames, I can think following ways.
>>  1. Use Py3k ;-)
>>  2. Use ctypes.GetCommandLineW()
>>  3. Use urlencode for filenames. ( file:///D:/%uXXXX...)
>>  4. pass filenames with stdin instead of commandline arguments.
>>
>> ** Attachment added: "decode filenames with get_user_encoding()"
>>    http://launchpadlibrarian.net/27783268/fenc2.patch
>>
>
> The way we get the command line has changed in recent versions. We now
> use GetCommandLineW function (IIRC), and we expect the arguments passed
> to main() are going to be Unicode arguments.

I read Python source code and found that py2k uses main(int, char**) and py3k
uses wmain(int, wchar_t**). I couldn't find GetCommandLineW.
So I think there are no way to get sys.argv in unicode on py2k.

http://code.activestate.com/recipes/436187/

Maybe, using GetCommandLineW and CommandLineToArgvW is good enough.

INADA Naoki пишет:
> http://code.activestate.com/recipes/436187/
>
> Maybe, using GetCommandLineW and CommandLineToArgvW is good enough.

Look at this code:
http://bazaar.launchpad.net/%7Ebzr/bzr/trunk/annotate/head%3A/bzrlib/win32utils.py#L493

It will be released in 1.16.

> Look at this code:
> http://bazaar.launchpad.net/%7Ebzr/bzr/trunk/annotate/head%3A/bzrlib/win32utils.py#L493
>
> It will be released in 1.16.

Very nice!!!

is there a way to test it before being launched?

cause i don't know how to compile python into executable, please don't address me to the download code and compile option...

but in my opinion it could be good if we can test it before the real relese

This is my way.

1. Install bzr 1.15 standalone with tbzr.
2. Install Python
3. Install bzr (any version) to Python from source.
4. Install PyQt4.5
5. Install qbzr
6. bzr branch lp:tortoisebzr
7. Prepare accordance with tortoisebzr/README (ex set envvar)
8. Kill tbzrcachew.exe from taskmanager.
9. python scripts/tbzrcache.py

You doun't have to install bzr standalone. Alternative way is:
1. Compile (or get from somwhere) tbzrshellext_x86.dll
2. regsvr (see README)

I tried it with bzr.dev and it looks work good.

--- scripts/tbzrcommand.py 2009-05-27 09:04:00 +0000
+++ scripts/tbzrcommand.py 2009-06-14 16:12:47 +0000
@@ -172,7 +172,7 @@
                     argv0 = os.path.join(sys.prefix, "scripts", "bzr")
                 if not os.path.isfile(argv0):
                     raise RuntimeError, "Can't find a bzr to execute"
- sys.argv[0] = argv0
+ sys.argv = [argv0]
         # finally get down to business...
         instance = klass()
         instance.run(**kw)
@@ -255,4 +255,6 @@

 if __name__=='__main__':
+ import bzrlib.win32utils
+ sys.argv[1:] = bzrlib.win32utils.get_unicode_argv()
     main()

i'll give it a try... or just wait till the next release...

Is critical higher than high?

i expect that :) if tortoisebzr won't solve this issue just people who write in ascii is able to use it, and the majority can't use it

Some people choose bzr against hg and git because bzr handles
filenames as unicode like svn.

I hope this bug is fixed and released in bzr-1.16 Windows standalone exe
package if possible.

INADA Naoki пишет:
> Some people choose bzr against hg and git because bzr handles
> filenames as unicode like svn.

Yes, bzr itself handles filenames as unicodes.
But this bug is about TortoiseBzr: shell integration into Windows
Explorer, and it has bugs here. Bugs of TortoiseBzr is not directly
related to bzr command-line client itself.

> I hope this bug is fixed and released in bzr-1.16 Windows standalone exe
> package if possible.

I can't promise it will be fixed in released with 1.16, because 1.16 is
already released and because main developer of TortoiseBzr does not work
on its codebase right now.

As short term solution I'd like to suggest you using bzr explorer plugin
(https://launchpad.net/bzr-explorer) as good GUI for Bazaar. It working
fine with unicode, and using QBzr under the hood (as TortoiseBzr does).

I'm not exactly sure if this bug is not related to the commandline, i tried to use the command line to create a repository inside a directory with chinese characters, it fails!

so maybe this bug is in bzr too from the command-line...

cablop пишет:
> I'm not exactly sure if this bug is not related to the commandline, i
> tried to use the command line to create a repository inside a directory
> with chinese characters, it fails!

What version of bzr you are using? As I said in this thread before, I've fixed some unicode problem
in command-line client, and in bzr 1.16 it should works much much better.

> so maybe this bug is in bzr too from the command-line...

To know for sure it will be better to show the error and provide relevant part of .bzr.log with log
of error (there should be python traceback).

INADA, I've slightly reworked your patch, please check my branch lp:~tortoisebzr-developers/tortoisebzr/bug-335362 and say is it work for you. If it fixes problem I'll merge it to trunk and I hope bzr 1.17 will include this fix.

Thank you, Alexander!
I check on Python 2.6.2 + bzr 1.16.1 + QBzr 1.11 + Your TortoiseBzr.
Some commands are fixed to work in Japanese directory (qlog, qbrowse, etc.)
Although some commands still broken, it makes better certainly.

INADA Naoki пишет:
> Thank you, Alexander!
> I check on Python 2.6.2 + bzr 1.16.1 + QBzr 1.11 + Your TortoiseBzr.
> Some commands are fixed to work in Japanese directory (qlog, qbrowse, etc.)
> Although some commands still broken, it makes better certainly.

What is not work? Do you have some log of errors?
Is it work with your patch?

> What is not work? Do you have some log of errors?
> Is it work with your patch?
No. Bugs I found are all qbzr's problem.

1. qinit
qinit can't start in non-ascii directory.
This is qbzr's bug. I'll report it to qbzr with patch.

2. qadd
qadd don't add files.
This may be also qbzr's bug.
Same problem happen when I start qadd in cmd.

--
Naoki INADA <email address hidden>

INADA Naoki пишет:
>> What is not work? Do you have some log of errors?
>> Is it work with your patch?
> No. Bugs I found are all qbzr's problem.
>
> 1. qinit
> qinit can't start in non-ascii directory.
> This is qbzr's bug. I'll report it to qbzr with patch.
>
> 2. qadd
> qadd don't add files.
> This may be also qbzr's bug.
> Same problem happen when I start qadd in cmd.

Please report this as well, and provide some examples of your filenames.
Even though I can't read Japanese, but I can copy-paste them and use for
testing. So the more info you will provide the easier will be to fix
these problems.

INADA Naoki пишет:
> Thank you, Alexander!
> I check on Python 2.6.2 + bzr 1.16.1 + QBzr 1.11 + Your TortoiseBzr.
> Some commands are fixed to work in Japanese directory (qlog, qbrowse, etc.)
> Although some commands still broken, it makes better certainly.

OK, I've pushed my fix to the trunk.

I hope John Meinel using trunk branch of TBZR to build official windows
installer. If so, fixed version of TBZR will be released as part of bzr
1.17.

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Alexander Belchenko wrote:
> INADA Naoki пишет:
>> Thank you, Alexander!
>> I check on Python 2.6.2 + bzr 1.16.1 + QBzr 1.11 + Your TortoiseBzr.
>> Some commands are fixed to work in Japanese directory (qlog, qbrowse, etc.)
>> Although some commands still broken, it makes better certainly.
>
> OK, I've pushed my fix to the trunk.
>
> I hope John Meinel using trunk branch of TBZR to build official windows
> installer. If so, fixed version of TBZR will be released as part of bzr
> 1.17.
>

I'm building a 1.16.1-2 if you want to try it out.

This should include an upgrade to python 2.5.4 (from 2.5.2), of PyQt to
4.4.3, and TortoiseBzr to revno 141.

John
=:->

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Cygwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAkpKHMkACgkQJdeBCYSNAAMVfgCg1HqMisqoL6KsEmKJ8BQePmTY
nLgAoLqb4ERtqkkT7BfayKj1QOZvXLTX
=q3zc
-----END PGP SIGNATURE-----

Hi John.

> I'm building a 1.16.1-2 if you want to try it out.
>
> This should include an upgrade to python 2.5.4 (from 2.5.2), of PyQt to
> 4.4.3, and TortoiseBzr to revno 141.
>
> John
> =:->

Thank you for your nice proposition.
But I found some bugs relating to non-ascii paths remain in QBzr.
(#394395, #393714).
After they are fixed, releasing new build is helps many non-ascii path
users to test it.

--
Naoki INADA <email address hidden>

INADA Naoki пишет:
> Hi John.
>
>> I'm building a 1.16.1-2 if you want to try it out.
>>
>> This should include an upgrade to python 2.5.4 (from 2.5.2), of PyQt to
>> 4.4.3, and TortoiseBzr to revno 141.
>>
>> John
>> =:->
>
> Thank you for your nice proposition.
> But I found some bugs relating to non-ascii paths remain in QBzr.
> (#394395, #393714).
> After they are fixed, releasing new build is helps many non-ascii path
> users to test it.

Naoki, anyone can upgrade QBzr without problems, but building new
installer with TortoiseBzr is a problem. So, right now anyone can start
use new version of TBZR + QBzr trunk with your fixes.

I'm planning to release QBzr 0.12 in sync with bzr 1.17, so all required
fixes will be available in official bzr 1.17 build.

For asian character testing you can try

天龙 (this is my chinese name)

If you are using XP then you can't create a repository inside that folder, not even from command line.

I see.
Setuping TortoiseBZR from source is not easy and setupping qbzr from source is
 just "cd /path/to/plugins; bzr branch lp:qbzr".

If John build bzr-1.16.1-2, I spread it in Japan.

--
Naoki INADA <email address hidden>

I can init from TBZR in "あいう" (Japanese, my codepage).
But I can't init in '天龙' (not in my codepage)

Following 2 patches fix it:

1. tbzrlib/vcspath.py
@@ -82,7 +82,7 @@
             # *sob* - I can't make stat() give me a float on Windows, even
             # after calling os.stat_float_times(True)
             self._attr, ctime, atime, self._mtime, size = \
- win32file.GetFileAttributesEx(self.path)
+ win32file.GetFileAttributesExW(self.path)
         except (win32file.error, os.error):
             self._attr = None
             self._mtime = None

2. bzrlib/win32util.py

 def set_file_attr_hidden(path):
     """Set file attributes to hidden if possible"""
     if has_win32file:
- win32file.SetFileAttributes(path, win32file.FILE_ATTRIBUTE_HIDDEN)
+ win32file.SetFileAttributesW(path, win32file.FILE_ATTRIBUTE_HIDDEN)

--
Naoki INADA <email address hidden>

天龙

This is Simplified Chinese, i don't know if solving the problem for Simplified Chinese can also fix it for Traditional Chinese

I can't switch my system to Chinese, cause my base system is english... maybe a problem is related to that command line of windows... it is not unicode...

and i can get an unicode command line alternative of cmd...

> This is Simplified Chinese, i don't know if solving the problem for
> Simplified Chinese can also fix it for Traditional Chinese

Ok, Maybe. My patch is use "-W" API. It handles path as utf-16 instead
of codepage.
So I can handle Simplified Chinese name in my Japanese codepage.

> I can't switch my system to Chinese, cause my base system is english...
> maybe a problem is related to that command line of windows... it is not
> unicode...
>
> and i can get an unicode command line alternative of cmd...

CommandLine can output unicode with "WriteConsoleW" API.
Please see below:
http://bitbucket.org/methane/hg-fixutf8-jp/src/50b7e27c2821/win32helper.py

bzrlib.win32helper can wrap stderr with that API.

--
Naoki INADA <email address hidden>

> bzrlib.win32helper can wrap stderr with that API.
I made error.
bzrlib.win32helper doesn't wrap stderr.

I'll try to make the wrapper for bzrlib, in a week.

--
Naoki INADA <email address hidden>

INADA Naoki пишет:
>> bzrlib.win32helper can wrap stderr with that API.
> I made error.
> bzrlib.win32helper doesn't wrap stderr.
>
> I'll try to make the wrapper for bzrlib, in a week.

Naoki, there is no bzrlib.win32helper.
There is bzrlib.win32utils, I suspect you're talking about latter.

Sorry, I was bit confused.

win32helper module is in hg-fixutf8. hg-fixutf8 helps mercurial to
handle unicode pathes and
use unicode console. It doesn't help bzr obviously.

bzrlib.win32utils doesn't wraps stderr with WriteConsoleW() currently.
But console output doesn't relate to this bug and TBZR.

--
Naoki INADA <email address hidden>

INADA Naoki пишет:
> I can init from TBZR in "あいう" (Japanese, my codepage).
> But I can't init in '天龙' (not in my codepage)
>
> Following 2 patches fix it:
>
> 1. tbzrlib/vcspath.py
> @@ -82,7 +82,7 @@
> # *sob* - I can't make stat() give me a float on Windows, even
> # after calling os.stat_float_times(True)
> self._attr, ctime, atime, self._mtime, size = \
> - win32file.GetFileAttributesEx(self.path)
> + win32file.GetFileAttributesExW(self.path)
> except (win32file.error, os.error):
> self._attr = None
> self._mtime = None

This fix is pushed to TBZR trunk.

> 2. bzrlib/win32util.py
>
> def set_file_attr_hidden(path):
> """Set file attributes to hidden if possible"""
> if has_win32file:
> - win32file.SetFileAttributes(path, win32file.FILE_ATTRIBUTE_HIDDEN)
> + win32file.SetFileAttributesW(path, win32file.FILE_ATTRIBUTE_HIDDEN)

My version of the patch for set_file_attr_hidden() has sent to ML:
http://permalink.gmane.org/gmane.comp.version-control.bazaar-ng.general/59676

Thank you.