program won't end (resolved) & java.nio.file.AccessDeniedException on Windows #156
Comments
|
Thanks for reporting this. The Fixed now. I will do a release within the next hour or so. |
|
Released with v0.10.7. Should be on maven central soon. |
|
I've just tried v0.10.7, the program did end, thanks~ but if I run the program a second time (after the first time finished successfully), I got this: |
|
That error is basically saying that it failed to delete the log Reading this thread it looks like it's a Windows permission issue. I don't have Windows. Could you please do me a favour and let me know if this program works for you? I wonder if you get the same import java.nio.file.{Files, Paths}
object Test extends App {
val folderPath = Paths.get("test")
val filePath = folderPath.resolve("0.log")
Files.createDirectories(folderPath)
Files.createFile(filePath)
assert(Files.exists(folderPath), "folder does not exists.")
assert(Files.exists(filePath), "file does not exists.")
Files.delete(filePath)
Files.delete(folderPath)
assert(Files.notExists(folderPath), "folder did not get deleted.")
}
|
|
@javadev do you also see the above |
|
the program exit with 0, all asserts pass |
|
I go to delete the
|
|
The log file number increments each time you restart the database or if the file gets flushed due to it being full. So you will see But you might be right about it trying to delete a file that doesn't exist. I will write some more random tests to see if I can somehow re-create the issue. If by any chance if you still have the database folder ( If not that's ok. Will eventually figure out the cause :) |
|
No problems running it on my Mac. Just so that I can run it on the right machine what version of Windows, Java and Scala are you running? |
|
java: jdk 1.8.0_152 I could repeat the test on Windows 10 later today |
|
full log if it helps~ notice that:
|
|
Thank you so much. The full logs helps a lot. Can you please do another debug run for me by adding "Thread.sleep(2000)" at the end of your code and see if you still see the exception? def main(args: Array[String]): Unit = {
val map1: Map[String, String, Nothing, ApiIO] = persistent.Map[String, String, Nothing, ApiIO]("map1").get
map1.put("abc" -> "def", "123" -> "456").get
map1.foreach(println)
.materialize.get
map1.close().get
println("done")
Thread.sleep(2000) //sleep for 2 seconds. after adding this here can you still see the exception?
}Another one by commenting out |
|
still see the exception, modified as: def main(args: Array[String]): Unit = {
val map1: Map[String, String, Nothing, ApiIO] = persistent.Map[String, String, Nothing, ApiIO]("map1").get
map1.put("abc" -> "def", "123" -> "456").get
map1.foreach(println)
.materialize.get
println("sleep")
Thread.sleep(2000)
println("close")
map1.close().get
println("done")
}same result commenting out |
|
Yep the background compaction thread throws that exception. But the data is still safe as it's reverts back to previous stable state. |
|
I found a discussion on SO. It looks like we are trying to delete file which was not closed. |
|
The files are always closed before deleting. |
simerplaha
changed the title
simerplaha
added
good first issue
|
Labeled this as Suggestion on debugging
|
|
I also encountered this problem with Java, running on Windows 10. Map<Integer, Integer, Void> map = MapConfig.functionsOff(Paths.get("data"), intSerializer(), intSerializer()).get();
System.out.println(map.get(42).get());Strangely, this works for the first run after storing, but all subsequent program runs fail. This problem is severe, as the code is crashing already on the MapConfig get() call, which leads to termination of the whole program. Maybe I can help to fix it, but I have no experience with Scala. |
|
@Ectras thank you for reporting this. I will try to replicate the error during this weekend. This issue is high priority now. |
|
Turns out on Windows we cannot delete memory-mapped files without clearing the in-memory The following test passes on Mac but fails on Windows with the exception you reported.
import java.io.IOException;
import java.nio.MappedByteBuffer;
import java.nio.channels.FileChannel;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.Paths;
import java.nio.file.StandardOpenOption;
public class MemoryMapFileDeleteTest {
public static void main(String[] args) throws IOException {
Path path = Paths.get("myFile.txt");
FileChannel channel = FileChannel.open(path, StandardOpenOption.READ, StandardOpenOption.WRITE, StandardOpenOption.CREATE_NEW);
MappedByteBuffer buff = channel.map(FileChannel.MapMode.PRIVATE, 0, 4000);
channel.close();
buff.force();
Files.delete(path); // throws java.nio.file.AccessDeniedException
}
}Solution is to delete memory-mapped files only after Temporarily you can create a database with memory-mapped disabled. import swaydb.data.config.IOStrategy;
import swaydb.data.config.MMAP;
import swaydb.data.config.SegmentConfig;
import swaydb.java.persistent.MapConfig;
import java.nio.file.Paths;
import java.util.Collections;
import static swaydb.java.serializers.Default.intSerializer;
public class MMAPDisabledExample {
public static void main(String[] args) {
Map<Integer, Integer, Void> map =
MapConfig
.functionsOff(Paths.get("data"), intSerializer(), intSerializer())
.setMmapAppendix(false)
.setMmapMaps(false)
.setSegmentConfig(defaultSegmentConfig())
.get();
map.put(42, 42);
System.out.println(map.get(42).get());
}
/**
* Builds a default SegmentConfig with MMAP disabled.
*
* TODO - Make it easier to build SegmentConfig from default SegmentConfig
* for overriding single fields like MMAP.
*/
private static SegmentConfig defaultSegmentConfig() {
return
SegmentConfig
.builder()
.cacheSegmentBlocksOnCreate(true)
.deleteSegmentsEventually(true)
.pushForward(true)
.mmap(MMAP.disabled()) //disable MMAP for Segment files.
.minSegmentSize(StorageUnits.mb(2))
.maxKeyValuesPerSegment(Integer.MAX_VALUE)
.ioStrategy(
ioStrategy -> {
if (ioStrategy.isOpenResource() || ioStrategy.isReadDataOverview()) {
return new IOStrategy.SynchronisedIO(true);
} else {
return new IOStrategy.SynchronisedIO(false);
}
}
)
.compression(info -> Collections.emptyList());
}
} |
|
Thank you for your quick reply! |
|
@Ectras just did a quick release of 0.14.4 with two important bug-fixes (unrelated to this issue which will be fixed in 0.14.5). Disabling memory-mapped files from Java is also simpler now. This following example is in Java.examples repo. import swaydb.data.config.MMAP;
import swaydb.java.Map;
import swaydb.java.persistent.MapConfig;
import swaydb.persistent.DefaultConfigs;
import java.nio.file.Paths;
import static swaydb.java.serializers.Default.intSerializer;
public class MMAPDisabledExample {
public static void main(String[] args) {
Map<Integer, Integer, Void> map =
MapConfig
.functionsOff(Paths.get("data"), intSerializer(), intSerializer())
.setMmapAppendix(false)
.setMmapMaps(false)
.setSegmentConfig(DefaultConfigs.segmentConfig(true).copyWithMmap(MMAP.disabled()))
.get();
map.put(42, 42);
System.out.println(map.get(42).get());
}
} |
…d files only after all the ByteBuffers are cleared.
|
Just released v0.14.5 which resolves majority issues that were indirectly linked to this issue. This issue is still in progress. |
|
This issue is now fixed in 0.14.6 release. All test-cases are passing on Windows 10. Please try re-running your code. @Ectras you can now run your code without disabling mmap config. Map<Integer, Integer, Void> map =
MapConfig
.functionsOff(Paths.get("data"), intSerializer(), intSerializer())
.get();
map.put(42, 42);
System.out.println(map.get(42).get());As mentioned before, the problem was that Windows does not allow deleting memory-mapped files unless the file's in-memory SolutionWe now have a boolean You don't really have to learn these configurations at all because the default instances are already pre-configured and I will document these configurations on the website in detail but here is a basic overview if you are interested. deleteAfterCleanIn the following snippets val mmap =
MMAP.Enabled(
deleteAfterClean = OperatingSystem.isWindows, //deleteAfterClean if OS is Windows
forceSave = ForceSave.Disabled //disable ForceSave
)Java code looks like MMAP.Enabled enabled =
MMAP.enabled(
OperatingSystem.isWindows(),
ForceSave.disabled()
);ForceSave
What is force?
We can configure //tells SwayDB to invoke forceSave before closing a file
val beforeClose =
ForceSave.BeforeClose(
enableBeforeCopy = true, //also enable before copying a file
enableForReadOnlyMode = false, //do not force save if the file is in read-only mode
logBenchmark = true //log out time take to execute force
)
//tells SwayDB to invoke forceSave before cleaning a memory-mapped file's MappedByteBuffer
val beforeClean =
ForceSave.BeforeClean(
enableBeforeCopy = true,
enableForReadOnlyMode = true,
logBenchmark = true
)
//tells SwayDB to invoke forceSave before copying a file. This occurs during compaction
//where a file can simply be copied into another level without merge when there are no overlapping keys.
val beforeCopy =
ForceSave.BeforeCopy(
enableForReadOnlyMode = true,
logBenchmark = true
)
//disables force save.
val disabled =
ForceSave.Disabled |
|
Closing this as it is resolved. Please reopen if needed. |
the following program will print
done, but won't end normally.The text was updated successfully, but these errors were encountered: