So I have my music collection on an external NTFS volume connected to my Slim Center (music server) machine.

On my Mac I have some NTFS driver installed so that I can write to NTFS volumes, because Mac OS natively supports reading only.

I just noticed that I cannot see all my files/folders when the drive is connected to my Mac. Based on one folder and its contents so far, it looks like the Mac is not seeing anything that's using 8.3 filenames. Example below...

In Windows Explorer I see the following path/filename when browsing:
S:\Master Music Collection\Music Library\Artists M-Z\Susana Félix\2007 - Pulsação\Susana Félix - Pulsação - 05 - Flutuo.mp3

In Slim Center if I navigate to that track and check its information, the path displayed is:
S:\Master Music Collection\Music Library\Artists M-Z\Susana Félix\2007-P~1\SU69A1~1.MP3

When I check other album folders for the same artist in Slim Center, they do not appear in the 8.3 format and they DO show up properly on the Mac.

There are about 200-250 folders currently missing when I view that drive on my Mac, but I don't know what they are yet. Is there a tool somewhere that will identify folders/files using this type of filename and optionally rename them to their full filename equivalents?

I believe this filename issue was caused by that POS software Songbird and its Lyrics plugin. It also changed the Genre on a ton of my tracks to ID3v1 numerical values.

Incidentally, I've tried 3 different drivers on Mac OS: Built-in NTFS (read-only), NTFS-3g (via MacFUSE, read/write) and Paragon NTFS for Mac OS X (read/write).

Browsing through Finder actually shows a blank document icon for the folders it can't read. Browsing with PathFinder (my default) simply doesn't show the problem folders at all (which if why I didn't see them originally).

I would imagine that this has to do with either the "ç" or the "ã" (or both). Can you create other directories and files on the filesystem with those characters and see what happens?

What happens if you look at it from the command line BTW?

It's not the accents in the name. NTFS stores its filenames in unicode, so one would assume these accents should work.

But to remove assumptions however, I've just been doing some testing re-creating folders with the same accents - and in fact the exact same name(s).

The folder I mentioned along with some others that fail, also fail when browsing over the network. Not necessarily in the same way though. So I've done testing over the network as well.

Over on the Windows side I've created some new folders with the same names using Explorer.

Browse them on the Mac with the drive connected directly or over the network and it all works fine.

If I COPY the original problem folders then the copies show the same issues. Therefore, in the case mentioned specifically above, if I create a new folder but copy in the old files, I can see the folder, but the files won't show up.

I have now set XP, by making an edit to the registry, to not create 8.3 names, which is a feature to support legacy 16-bit apps for the most part. This will hopefully prevent any new problem filenames and is also supposed to increase disk performance in some cases). However, I can't find a way to purge/delete from the disk all 8.3 aliases. I think that would solve the problem here.

One of the other folders that shows this issue is named "M.I.A." and neither it nor its sub-folders/files have any accents.

And what is "Language for non-Unicode applications" (or "System default codepage", depending on your Windows version) set to? Is there, on the Mac, any sort of setting for codepage for Windows filesystems? (Although that'd still be a bit weird, as the on-disk filename format in NTFS is UTF-16, i.e. not codepage-dependent.)

Peter

That's another interesting bit of evidence, when you consider that "M.I.A." is not a valid 8.3 filename (multiple dots weren't allowed).

Peter

From the Windows command line, the new folders I create look "mostly" right. The accent over the "a" in "Pulsação" doesn't show up, so it reads "Pulsaçao" - but I can cd to them properly. Seems like a display limitation/quirk of the command line, while the filesystem has the correct information.

The original problem folder however, shows up as "Pulsac,ao" and I can't figure out how to cd to it at all.

When I create new folders everything works beautifully. Windows XP doesn't need any special settings to support the basic languages, which includes all of the Western European, to which those accents/characters belong.

It's most definitely a problem with the 8.3 names, which as mentioned, aren't an issue on newly created files/folders in Explorer (at least not after disabling 8.3 alias creation in the registry).

Let's ignore the "M.I.A." example though. That may be a special case as I can no longer access it even under Windows. That folder had an odd character instead of the last period and I renamed it earlier today to "fix" it. Now it can't be opened under Windows and doesn't show up at all under Mac OS. Nice.

Ultimately, your problem is that you're using a closed filesystem for interaccess, and all of the non-Windows drivers are based on guesswork.

You'd probably be better off using an open filesystem and having a third-party Windows driver. That said, you don't have a lot of choices if you want native Unicode filename support. Possibly zero.

The problem isn't necessarily trying to access a closed filesystem with a driver that's been created by reverse engineering. I think the problem is that there are a lot of crappy Windows programs that can really screw things up on Microsoft's somewhat crappy filesystem, period.

The NTFS driver built into the system and the Paragon version I'm trying now seem to do very well reading all files/folders that have not been messed up in this way. That is to say, ones using only long filenames without any backward compatibility kludges.

The problem files are also an issue with network browsing from the Mac. I'm going to see what happens when I browse them from another Windows PC.

As an aside, does anyone know how to delete a "stuck" folder in Windows? That MIA folder brings up a message saying "refers to a location that is unavailable..." and I can't do anything with it, including cd from the command prompt.

Maybe your filesystem is corrupted.

I would try putting the problem files on a USB stick, then copy them to your OS X filesystem, and then write them to the NTFS drive from OS X and see what happens.

As far as I know, it's not. No errors come up in Windows except for that one M.I.A. file not being able to be deleted. MIght be because it ends in a period which is valid for NTFS but I think Win32 has a problem with it (Explorer will never allow a period to end a filename or folder name when renaming).

The issue with copying files to and from USB sticks is that I have no idea exactly which files/folders are affected. I know a few of them, but there are at least 6000 files affected. That's the difference in file counts between Slim Center and what iTunes reported on my Mac after a drag and drop of my entire music hierarchy.

Um, you can probably figure out some way to parse "dir /s /x z:\".

You may also be interested in: "fsutil behavior set disable8dot3 1". (You have to reboot after setting this.)

Sadly, while fsutil gives you a way to set shortnames ("fsutil file setshortname z:\longfilename.txt longfi~1.txt"), it does not provide a way to erase existing ones. However, if you rename a file with a long filename with an 8.3 filename, it purges the saved short filename. Then you can rename it back. So you should be able to implement this pseudocode:


Of course, you'll want to disable8dot3 first.

The registry edit I made is the same thing as the fsutil command you posted Bitt.

But, thanks for the renaming tip. It lead me to discover some more information.

Arrrgh.

What appears to be the main issue is that the filenames are in fact NOT using the correct unicode characters. The cedilla for instance was made up of two characters in the problem names. If I copy and paste this to Mac OS it looks fine. But isn't the same thing as when I type it properly.

So for instance can type a filename into a mail message with the correct cedilla "ç" and then paste the Windows name directly under it. Both look identical. Only the one I typed can be pasted back to Windows and work correctly.

What I'm doing to fix the mp3 filenames is running a process through MP3Tag to rename the files based on the tag text. It handles the unicode properly so far and omits invalid characters. Easier than writing a script to recursively traverse my filesystem.

The same problem existed for "Maxïmo Park" while "Tïesto" was working fine. Looking carefully you can actually see (in Explorer) than the umlaut above the i in Maxïmo was slightly offset to the right. Two characters. One i and one umlaut. Weird.

The 8.3 might have been a wild goose chase after all, and only an issue in representing the long names because of the oddball characters that were within them.

I'm still going to have to do some manual searching for any folder names affected, because the tag-to-filename process is only touching the files.

Ah! Good spot. There are two (common) ways of writing accented characters in Unicode, "NFC" (normal form composed), where you use (say) a c-cedilla character, and "NFD" (normal form decomposed), where you use the normal c character followed by a "composing cedilla". The MacOS API is defined to use UTF-8 NFD, Linux (and probably Slimserver) use UTF-8 NFC, and Windows and thus NTFS use UTF-16 NFC. It looks like something along the way has managed the UTF-8/UTF-16 conversion but botched the NFC/NFD conversion.

Googling all this produces http://macntfs-3g.blogspot.com/2009/02/ntfs-3g-200921-update-1.html (last comment).

Peter

The NFC/NFD normalization does sound plausible, but I can't for the life of me think of why I didn't see this until now.

But I did just discover the culprit. At least I'm next to positive. MacDrive. A Windows driver for reading Mac disks.

Anyway, from my Mac, I put a file named "Pulsação" onto a thumbdrive formatted HFS+ When I view that folder in Windows, sure enough, the cedilla is next to the c, not directly under it.

I can in fact create another folder in Windows with the correct filename and it will happily sit along-side the original one. Because as far as Windows is concerned, they're different.

Move the thumbdrive back to my Mac and the folder names clash and one is now appended with ~1.

I did some copying/moving of folders to and from my NTFS drive under Windows and didn't run into any problems on the Mac however. So while I seem to have discovered that MacDrive is showing the folder names incorrectly when a cedilla is present, it doesn't cause an issue when using the HFS+ format on the Mac, regardless of where the file is created/named. I suspect the issue is only when viewing that file on an NTFS volume on the Mac - which I'll try in a moment by bringing the other drive over again.

Ideally I could just use HFS+ for my large media disk in Windows, but MacDrive has proven to me time and time again that it's unreliable by causing simple folder browsing to hang Explorer on my music drive.

So it seems like Songbird isn't responsible for this issue. It was a coincidence that I first noticed it on songs which it did mess up tags for (numerical genres, adding ID3v1 and forcing ID3v2.4 instead of preserving the previous 2.3)

Confirmed. The problem becomes visible on an NTFS disk, on the Mac, when the files are copied in Windows while MacDrive is interpreting the originals.

Summary:
1. MacDrive in Windows misrepresents c-cedilla and i-umlaut (and others) when reading Mac HFS+ disks.
2. Copying operations that end on an HFS+ disk in Windows using MacDrive are OK on the Mac. It seems to be ok at translating back from the misrepresented accents. Or Mac OS simply understands the messed up names when it reads the HFS+ volume.
3. Copying operations that end on an NTFS disk in Windows using MacDrive will prevent Mac OS from seeing the files with the misinterpreted accents. It doesn't matter if you're using the built-in NTFS read-only driver, Paragon NTFS or NTFS-3g.

I'm sure you're more knowledgeable about this than I am, and I'm sure I'm showing my ignorance, but isn't it possible to support both at the same time? I was under the impression that, for example, c-cedilla was one character and "c" plus a composing cedilla was two characters, and that the two constructs are completely disjoint.

I guess my question can be summarized as "is a string containing both c-cedilla and a "c" plus composing cedilla couplet invalid?". That is, it seems to me that "<U+00E7><U+0063><U+0327>" should be valid and be rendered as "çç".

The two constructs are different, but semantically equivalent. The MacOS people adopted the attitude that having both forms coexisting was a recipe for confusion, and all the MacOS file APIs normalise to NFD before writing to disk. An HFS+ driver for Windows that allows NFC filenames that are not NFD onto the disk, is creating broken filesystems, analogously to (say) a FAT32 driver for Linux that allowed both upper- and lower-case versions of the same name onto the disk.


It's a perfectly valid Unicode string, but it's not a valid on-disk HFS+ filename.

Peter

Gotcha. That makes sense. Well, as much sense as arbitrarily limiting it makes sense. And as much sense as being able to create two distinct semantically equivalent strings makes sense.

My only hurdle right now is still trying to delete that MIA folder. I have yet to try third-party software, but so far chkdsk didn't produce any results (by fixing an error that may have been preventing accessing that folder).

DONE!

I found this program; http://www.purgeie.com/delinv/dldelinv.htm

By searching Google for "delete stubborn file windows xp"

It's one butt-ugly UI, but it worked to rename the folder. The contents of the folder were still A-OK.

The issue was that Windows supports more funkiness within filenames than Explorer does. Typical Microshaft.

Funkiness and Microsoft, in the same post.

Yep. Somehow, some tool managed to create a folder with a space at the end of the name. IE, C:\Dir1 \File1

Lots of other programs barfed on this, and it took a bit of work to fix.

I once moved a network share to a new server, so at the old address, I created a directory named "moved to &#8260;&#8260;newserver&#8260;share", where the slashes are "fraction slashes" (U+2044). A huge number of programs choked on that.

BTDT, sort of. Exact Audio Copy will cheerfully create directory paths/filenames whose length is longer than Microsoft allows (what is it, 255 characters?) and Windows Explorer won't process them, although I seem to recall that MP3TS would.

tanstaafl.

Just wanted to post a follow-up here with more information on the messed up filename situation with regards to accented characters. I still haven't found an automated way to make corrections, but I know how to find any problem files now, even if searching for one problem type at a time.

My conversion of tag to filename fixed a lot of the problems but unfortunately, it didn't take care of everything - likely because the problem characters also existed in the tags.

I did find a command-line linux program called convmv which sounded promising, but I wasn't able to get it to do anything useful at all. As is sometimes the case, the author seems to have been hyper-focused on his own need for creating the program that he fails to provide decent documentation or examples for everyone else.

Anyway, the filename is caused because instead of the accented characters being made up of single unicode entities, they're using multiple. They're made up of Unicode Combining Characters.

A list of Unicode Combining Characters can be found on this US government page funnily enough. "MARC standards" - they're at the bottom of the list.

You can see that each combining character is two bytes itself. These get added to a one or two byte letter to create a visual representation of the accented character.

These should NOT be used in filenames as far as I can tell. They won't zip nor work with plenty of software because the characters returned aren't what's expected. This is why iTunes fails to read these files.

To help me locate affected filenames I've made a file containing all these characters (using a hex editor) and labeled them accordingly. I load this file into Notepad and then copy the combining character which I then use to paste into a file search field. Windows will happily find the files using the combining characters. Then I make sure the tags are OK and then I perform either a manual or tag-to-filename rename.

I've attached the combining_characters.txt file to this post. MOstly just in case I need it again and lose my local copy.

I've also now found at least one source of the problem. An iTunes script for populating the Track name from file name will create these combining characters after reading the otherwise correct filenames.

That's not a complete list, only the most common ones. Everything between U+0300 and U+036F inclusive is a composing diacritic. (There are other combining characters for other scripts, but all the ones for Latin/Greek/Cyrillic are in U+0300..U+036F.)

Peter

I know it's not a complete list. Neither the link I posted, nor the file I attached. I was only including (in the attached file) the ones that might have been in my music, which would only include Latin. Though I did toss in a few extra ones as a sanity check/search.

Oh, OK. I suppose all I'm saying then is that if you ever get round to writing a program or script to find these things, the script's life can be made slightly easier if it knows that all characters in that range, and none outside it, are the ones it should be flagging-up.

Peter