I’m encountering a strange, sporadic error in FileManager.replaceItemAt(_:withItemAt:) when trying to update files that happen to be stored in cloud containers such as iCloud Drive or Dropbox. Here’s my setup:
-
I have an NSDocument-based app which uses a zip file format (although the error can be reproduced using any kind of file).
-
In my
NSDocument.writeToURL:implementation, I do the following:
-
Create a temp folder using
FileManager.url(for: .itemReplacementDirectory, in: .userDomainMask, appropriateFor: fileURL, create: true). -
Copy the original zip file into the temp directory.
-
Update the zip file in the temp directory.
-
Move the updated zip file into place by moving it from the temp directory to the original location using
FileManager.replaceItemAt(_:withItemAt:).
This all works perfectly - most of the time. However, very occasionally I receive a save error caused by replaceItemAt(_withItemAt:) failing. Saving can work fine for hundreds of times, but then, once in a while, I’ll receive an “operation not permitted” error in replaceItemAt.
I have narrowed the issue down and found that it only occurs when the original file is in a cloud container - when FileManager.isUbiquitousItem(at:) returns true for the original fileURL I am trying to replace. (e.g. Because the user has placed the file in iCloud Drive.) Although strangely, the permissions issue seems to be with the temp file rather than with the original (if I try copying or deleting the temp file after this error occurs, I’m not allowed; I am allowed to delete the original though - not that I’d want to of course).
Here’s an example of the error thrown by replaceItemAt:
Error Domain=NSCocoaErrorDomain Code=513 "You don’t have permission to save the file “test-file.txt” in the folder “Dropbox”." UserInfo={NSFileBackupItemLeftBehindLocationKey=file:///var/folders/mt/0snrr8fx7270rm0b14ll5k500000gn/T/TemporaryItems/NSIRD_TempFolderBug_y3UvzP/test-file.txt, NSFileOriginalItemLocationKey=file:///var/folders/mt/0snrr8fx7270rm0b14ll5k500000gn/T/TemporaryItems/NSIRD_TempFolderBug_y3UvzP/test-file.txt, NSURL=file:///Users/username/Library/CloudStorage/Dropbox/test-file.txt, NSFileNewItemLocationKey=file:///Users/username/Library/CloudStorage/Dropbox/test-file.txt, NSUnderlyingError=0xb1e22ff90 {Error Domain=NSCocoaErrorDomain Code=513 "You don’t have permission to save the file “test-file.txt” in the folder “NSIRD_TempFolderBug_y3UvzP”." UserInfo={NSURL=file:///var/folders/mt/0snrr8fx7270rm0b14ll5k500000gn/T/TemporaryItems/NSIRD_TempFolderBug_y3UvzP/test-file.txt, NSFilePath=/var/folders/mt/0snrr8fx7270rm0b14ll5k500000gn/T/TemporaryItems/NSIRD_TempFolderBug_y3UvzP/test-file.txt, NSUnderlyingError=0xb1e22ffc0 {Error Domain=NSPOSIXErrorDomain Code=1 "Operation not permitted"}}}}
And here’s some very simple sample code that reproduces the issue in a test app:
// Ask user to choose this via a save panel.
var savingURL: URL? {
didSet {
setUpSpamSave()
}
}
var spamSaveTimer: Timer?
// Set up a timer to save the file every 0.2 seconds so that we can see the sporadic save problem quickly.
func setUpSpamSave() {
spamSaveTimer?.invalidate()
let timer = Timer(fire: Date(), interval: 0.2, repeats: true) { [weak self] _ in
self?.spamSave()
}
spamSaveTimer = timer
RunLoop.main.add(timer, forMode: .default)
}
func spamSave() {
guard let savingURL else { return }
let fileManager = FileManager.default
// Create a new file in a temp folder.
guard let replacementDirURL = try? fileManager.url(for: .itemReplacementDirectory, in: .userDomainMask, appropriateFor: savingURL, create: true) else {
return
}
let tempURL = replacementDirURL.appendingPathComponent(savingURL.lastPathComponent)
guard (try? "Dummy text".write(to: tempURL, atomically: false, encoding: .utf8)) != nil else {
return
}
do {
// Use replaceItemAt to safely move the new file into place.
_ = try fileManager.replaceItemAt(savingURL, withItemAt: tempURL)
print("save succeeded!")
try? fileManager.removeItem(at: replacementDirURL) // Clean up.
} catch {
print("save failed with error: \(error)")
// Note: if we try to remove replaceDirURL here or do anything with tempURL we will be refused permission.
NSAlert(error: error).runModal()
}
}
If you run this code and set savingURL to a location in a non-cloud container such as your ~/Documents directory, it will run forever, resaving the file over and over again without any problems.
But if you run the code and set savingURL to a location in a cloud container, such as in an iCloud Drive folder, it will work fine for a while, but after a few minutes - after maybe 100 saves, maybe 500 - it will throw a permissions error in replaceItemAt.
(Note that my real app has all the save code wrapped in file coordination via NSDocument methods, so I don’t believe file coordination to be the problem.)
What am I doing wrong here? How do I avoid this error? Thanks in advance for any suggestions.
I hadn’t filed a bug report yet because I had assumed it was something I was doing wrong, given that using replaceItem and a temporary folder is presumably a common pattern. I’ll file a report tomorrow - I’m following the iCloud Drive profile instructions you linked to and am now waiting the 24 hours they say I need to wait before I can get the sysdiagnose. Once I have that, I’ll file the report along with a sample project.
Perfect, thank you.
...using replaceItem and a temporary folder is presumably a common pattern
I didn't get into it above, but there very likely are some nuances/details involved that are a contributing factor. For example, I suspect this doesn't happen if you start with a security-scoped bookmark, which you resolve to a bookmark before each save. I think NSDocument's default implementation also writes out a new file before each save, which means it's always starting with a "new" URL. That doesn't mean there's anything "wrong" with what you're doing, but that's probably why this isn't more widespread.
With a bit of refactoring, I probably could retry the save. In my app, this is all done inside my NSDocument’s writeToURL method. I use my own drop-in replacement for FileWrapper (you helped me with some of the finer points of FileWrapper a few years ago).
Fabulous! Always good to hear when my ideas have worked out!
A potential problem with the re-save approach is that my save usually works by copying the zip file at the original location to a temporary location, updating it there, and then moving it into place using replaceItemAt.
Just to clarify, are you:
a) Copying the file once, modifying it over time, then copying that file back for each save.
b) Copying the file prior to each save operation.
I suspect you're doing "a" (and it's probably what I would do), but if you're doing "b”, then that changes things a bit.
Assuming you're starting with "a", then my intuition would be to:
-
Commit your change to your temp file.
-
Clone that file into a new temp file.
-
Use that new temp file as the source for your save.
There are a few advantages to this:
-
It may avoid the immediate issue here, since you'll always be replacing with a "new" file object.
-
If anything goes wrong, you can retry the save by restarting with a clean clone.
-
It can be a useful architecture to build on for other edge cases.
Expanding on that last point, one of the issues you can run into is cases where the files involved are large and the final save destination is VERY slow (like an SMB drive). Putting that in concrete terms, let’s say you want to autosave every 1s, but the save destination is going to take 5s-10s to complete the save. Here is one way to handle that:
-
Your app copies from the destination to your local storage. This becomes your "working" copy that you modify.
-
Your app autosaves to local storage every ~1s.
-
Your app pushes that initial save data to the final target.
-
Every time the final save finishes, it starts a new save using the most recent save.
In other words, your app can rely on its "standard" set of 1s autosaves, but you're actually only saving to the final target every 5-10s (as the previous save finishes).
One final point here— if you're working with package and large file counts, directory cloning may provide a significant performance benefit. The man pages warn against cloning directories, but this forum post explains what the actual risks are and when it's a reasonable option.
I wonder, though— given that the original file has in fact been replaced by the temp file despite the error, can I not just check for this and ignore the error if the file seems to have been replaced after all? E.g.: ... Is there something wrong with this approach?
That is a REALLY tough call. The problem here is that your visibility into the exact cause of the error is limited, so while it's certainly safe in the particular case, it's hard to be sure that you're ACTUALLY dealing with this exact case. Even worse, my concern here would be the proliferation of edge cases, both in terms of what's out there "today" and in terms of future configurations/changes.
My own instincts would be to redo the entire save, but if you want to do this, I would do two things:
-
Look at the NSError object you're getting back "in detail" so you can identify as "specific" a failure as possible. Notably, I think you can use NSUnderlyingErrorKey to pull an NSError object for the lower-level error, so I'd look at that object (and possibly any underlying NSError), not just catching the "fail".
-
...then an ID check to confirm you're "right" about the failure. I'd even check things like file size and possibly times so that you're "sure" everything looks the way you'd expect. Most of that metadata is collected with a single syscall, so it gets you a little extra safety without actually making things slower.
The goal is to fingerprint a particular failure you consider "safe", not just trusting the ID change. Having said that, I'd also be tempted to expand the check in #2 and run it against all files, not just the cases where you got an error. Done properly, there’s minimal performance cost, and there are worse things an app can do than double-checking its saves.
__
Kevin Elliott
DTS Engineer, CoreOS/Hardware
