----------------------------------------------------------------------------------
@MSGID: 2@dont-email.me> ab3ced75
@REPLY: <MR2dnWJkR5f55mb5nZ2dnZfqnPudnZ2d@giganews.com>
f2a92e96
@REPLYADDR Michael Soyka <mssr953@gmail.com>
@REPLYTO 2:5075/128 Michael Soyka
@CHRS: CP866 2
@RFC: 1 0
@RFC-Message-ID: 2@dont-email.me>
@RFC-References:
<53eaef44-31be-4cd5-98a4-b535c4b1b1f6n@googlegroups.com> <87jzt3c5i4.fsf@pointsman.de> 1@dont-email.me>
<8734zpcygj.fsf@pointsman.de> 1@dont-email.me>
<MR2dnWJkR5f55mb5nZ2dnZfqnPudnZ2d@giganews.com>
@TZUTC: -0400
@PID: Mozilla/5.0 (Windows NT 10.0; Win64; x64;
rv:102.0) Gecko/20100101 Thunderbird/102.15.0
@TID: FIDOGATE-5.12-ge4e8b94
On 09/08/2023 3:52 PM, Robert Heller wrote:
> At Fri, 8 Sep 2023 14:49:48 -0400 Michael Soyka <
mssr953@gmail.com> wrote:
>
>>
>> On 09/07/2023 8:15 PM, Rolf Ade wrote:
>>>
>>> Michael Soyka writes:
>>>> On 09/06/2023 6:16 PM, Rolf Ade wrote:
>>>>> Michael Soyka <mssr953@gmail.com> writes:
>>>>>> I`m using the Magicsplat distribution of tcl 8.6.12 on a Windows 10
>>>>>> [...]
>>>>>> filenames came to include emoji characters.
>>>>>>
>>>>>> Now to the problem. When I try to access these files using Tcl, I get
>>>>>> what I consider to be nonsensical errors. For example, the "open"
>>>>>> command fails with the message "filename is invalid on this platform",
>>>>>> even though the file does exist. On the other hand, various "file"
>>>>>> commands that also take a filename argument, such as "exists" and
>>>>>> "size", return "no such file or directory". Again, the file certainly
>>>>>> does exist.
>>>>> You haven`t shown us how you call that commands in Tcl, with the
>>>>> emoji
>>>>> literal in the source code or escaped as \\Uxxxxx, for example and what
>>>>> encoding your source file has.
>>>>
>>>> The filenames were obtained using the "glob" command. The files
>>>> themselves were created, I believe, by others using a mail client on
>>>> Windows.
>>>>> Since Tcl 8.6.10 I think and for sure with the upcomming Tcl 9 there
>>>>> is
>>>>> no problem in handling such filenames (with unicode code points in
>>>>> proper utf.8 in it as emojis).
>>>>> See for example:
>>>>> # The following is: a\\U1f972
>>>>> set fd [open a???°?,???,???,?? w+]
>>>>> # \\U1f926
>>>>> puts $fd ???°?,???,?¤?,??
>>>>> close $fd
>>>>> set fd [open a???°?,???,???,??]
>>>>> puts [read $fd]
>>>>> close $fd
>>>>> This script works for me on linux with 8.6.10, 8.6.13 and 9. Though
>>>>> this
>>>>> is on linux.
>>>>>
>>>>>> I haven`t been able to construct such a filename using Tcl commands.
>>>>>> Instead, I`ve used "glob" to get the filename from the filesystem
>>>>>> (NTFS) and used the result as the argument for "open" and "file".
>>>>> So you can construct the filenames with results of Tcl commands and
>>>>> successfully open the files?
>>>>
>>>> The only reason I tried to create a file that includes emoji
>>>> characters in its name was to investigate the contradictory responses
>>>> I was getting from the "open" and "file" commands.
>>>>
>>>> However, that`s not the primary issue I tried to raise so I`ll try to
>>>> be more specific.
>>>>
>>>> I was given a collection of files on a thumb drive. One of the files
>>>> contains a sequence of three emoji characters in its name: "two
>>>> hearts", "revolving hearts" and "two hearts". The corresponding
>>>> unicode values are \\U01F495 and \\U01F49E. One of the reasons I
>>>> believe this is based on the following code:
>>>>
>>>> proc DisplayCharCodes {string} {
>>>> foreach c [split $string {}] {
>>>> puts [format {%s: %#x} $c [scan $c %c]]
>>>> }
>>>> }
>>>> set fileList [glob -type f *.eml]
>>>> set filename [lindex $fileList 1]
>>>> DisplayCharCodes $filename
>>>>
>>>> which outputs the following:
>>>>
>>>> N: 0x4e
>>>> E: 0x45
>>>> X: 0x58
>>>> T: 0x54
>>>> : 0x20
>>>> S: 0x53
>>>> A: 0x41
>>>> T: 0x54
>>>> .: 0x2e
>>>> : 0x20
>>>> 2: 0x32
>>>> _: 0x5f
>>>> 1: 0x31
>>>> 5: 0x35
>>>> _: 0x5f
>>>> : 0x20
>>>> F: 0x46
>>>> A: 0x41
>>>> D: 0x44
>>>> E: 0x45
>>>> D: 0x44
>>>> : 0x20
>>>> L: 0x4c
>>>> O: 0x4f
>>>> V: 0x56
>>>> E: 0x45
>>>> R: 0x52
>>>> S: 0x53
>>>> : 0x20
>>>> T: 0x54
>>>> O: 0x4f
>>>> U: 0x55
>>>> R: 0x52
>>>> : 0x20
>>>> i: 0x69
>>>> n: 0x6e
>>>> : 0x20
>>>> P: 0x50
>>>> R: 0x52
>>>> O: 0x4f
>>>> V: 0x56
>>>> I: 0x49
>>>> D: 0x44
>>>> E: 0x45
>>>> N: 0x4e
>>>> C: 0x43
>>>> E: 0x45
>>>> !: 0x21
>>>> : 0x20
>>>> ???°?,???,?`?,?o: 0x1f495
>>>> ???°?,???,?`?,??: 0x1f49e
>>>> ???°?,???,?`?,?o: 0x1f495
>
> Noticing that these are 16-bit characters...
This doesn`t look what I see in my posts which are the characters
themselves.
>
>>>> .: 0x2e
>>>> e: 0x65
>>>> m: 0x6d
>>>> l: 0x6c
>>>>
>>>> Given the above, this is what "open" returns:
>>>>
>>>> % open $filename r
>>>> couldn`t open "NEXT SAT. 2_15_ FADED LOVERS TOUR in PROVIDENCE!
>>>> ???°?,???,?`?,?o???°?,???,?`?,?????°?,???,?`?,?o.eml": filename is
invalid on this platform
>>>>
>>>> and the response of "file exists $filename" is zero.
>>>>
>>>> So I`m looking for a reason behind this inconsistent and, in my mind,
>>>> nonsensical behavior. Is it a Windows issue, a Tcl issue, a little of
>>>> both and/or something else?
>>>>
>>>> I hope the above clarifies my problem.
>>>
>>> Thanks.
>>>
>>> Yes, typically you should be able to use any file name returned by glob
>>> as argument for open or file exists. There is an exception of that rule
>>> (what Harald mattered) and that may be in place here.
>>>
>>> Can you open the file in question with the file explorer? Perhaps you
>>> can truncate it and provide it as download somewhere (in the hope that
>>> the "strangeness" of the file name survives this actions, which is not a
>>> given)?
>>
>> Yes, using Windows Explorer I can open the file with Vim and open the
>> file with Outlook. I can also rename the file, deleting the 3 emoji
>> characters, and open it using the Tcl commands "glob" and "open".
>>
>>>
>>> The one known scenario which shows what you describe (you can`t open a
>>> filename you got from glob) is: the file names are written in another
>>> encoding then what the system use for its filenames. Though, in what you
>>> presented as results of your own investigations I cannot see indication
>>> that this is the case here.
>>>
>>> But perhaps it`s in fact a strangeness of the used windows APIs (or how
>>> they are used). At least you are right in saying this is strange and
>>> need an explanation. If it`s not the thing from above.
>>
>> I`ve since copied the files from the same thumb drive onto my linux
>> system and retried the "glob" and "open" commands using 8.6.10- it all
>> works. My Windows version is 8.6.12, a later version, so it appears
>> that my problems are peculiar to Windows.
>
> What does DisplayCharCodes display under Linux? Do the emoji chars display as
> 16-bit chars or two 8-bit characters?
I see the emoji characters themselves followed by the 24-bit value.
Just to be clear though, by "emoji characters themselves" I mean as they
are displayed as in my earlier post, not as they are displayed above.
If I pipe the filename into a file and octal dump the file, I see these
byte values (octal) where the emoji characters are:
360 237 222 225 360 237 222 236 360 237 222 225
which looks like UTF-8 encoding to me. Microsoft claims it uses UTF-16
for its filenames so I`d guess the end-result is the same.
>
>>
>> Thanks for your continuing interest- it`s helped motivate me to look
>> deeper into the problem.
>>
>> -mike
>>
>>>
>>> rolf
>>
>>
>>
>
--- Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:102.0) Gecko/20100101
Thunderbird/102.15.0
* Origin: self (2:5075/128)
SEEN-BY: 5001/100 5005/49 5015/255 5019/40 5020/715
848 1042 4441 12000
SEEN-BY: 5030/49 1081 5058/104 5075/128
@PATH: 5075/128 5020/1042 4441