Miner stops reading exFat disks if a new block comes in before all drives are read.

[Leozolt]

When a new block is announced while the miner is reading the disks, some drives (the drives that did not finish yet) do not get read in the next blocks. If it happens very early in the read process more drives are left out on the next blocks. (The drives that did not finish seem to be left out in the next blocks and the miner doesn't recover from it so the problem gets worse over time until no drives are read at all)

Release 1.7.16

mining.conf.txt
creepMiner_20180315_140131.165603.log

Using Ext4, ntfs and exFat

maybe the issue is similar to #177

[loppefaaret]

yeah, your "/media/leozolt/Burst1" does not finish after round 818 - perhaps verify that the drive is not dying ?
does it work if you restart the miner ?

[Leozolt]

Yes restarting the miner helps until a fast block comes in before the reading finishes again. Removed ../Burst1 from the config & need some fast block now to verify. It's not only ../Burst1 but it's usually the first drive to fail being read, as it is always the slowest 10TB exFat. Ok without ../Burst1 it is ../Burst3 failing first (which is the second exFat). Trying to also remove ../Burst3 now

[Leozolt]

The miner doesn't hang on fast blocks without the exFat drives. Any idea what could cause this behavior (Ubuntu 16.04 with exfat-fuse 1.2.3-1)
Update:
Running 2 instances of the miner with the exFat drives submitting to the other instance on the same machine. The instance of the miner with 2 exFat formated drives hangs/stops mining, Instance with the other mixed format ext4/ntfs keeps running. Overall the performance of the server instance gets worse reading performance but one of the exFat drives gets faster in the sepearte instance... until a new block comes in while the readers are working on exfat.
creepMiner_20180316_174625.972998.log
The difference i see between 1 instance reading all the files and 2 miners one instance with exfat, is that if i mine with all the drives in one instance, the miner first fails and only reads 86% and continues only reding 86%, then the next time it gets caught reading it misses out the next drive until it only reads one drive so fast that it only totally fails on 1~12s blocks, while with 2 instances, the remaining drives keep mining.

[Leozolt]

For the people that like screenshots: the instance in focus always shows 92,39%, updates the block info but does not read the exfat disks.

[Leozolt]

maybe exfat-fuse driver needs a special "reset current read-command"? That is higher than my skill level sadly.. I agree that it could possibly be a machine specific or host system specific issue but maybe soemone here has a hint how to solve this (whitout replotting on ext4)((I would need to truncate the plots t make them fit by copying on the same size ext4))
So if someone can point out a logfile on ubuntu 16.04 to look after that possibly could help resolve this behavior, that would be really helpfull.

[Leozolt]

Partially resolved the failure by running all non exFat drives in one (server) instance and running the 2 other drives using exFat on individual instances for each drive. First 1.7.16-2 reads, then 1.7.16-3 reads. no failure. Weird

[Leozolt]

Maybe it is something with exFat driver (exfat-fuse 1.2.3-1 on ubuntu 16.04) not supporting more than x threads from one instance? But that sould have been noticed by exfat users...

[loppefaaret]

when the disks fail to complete the read, do you have any kernel messages about read errors?
when a disk is in failmode, are you able to work on the filesystem, such as ls /media/faileddisk/plots and/or touch /media/faileddisk/test.file && rm /media/faileddisk/test.file ?

[Leozolt]

working on the filesystem is abnormally slow, got a touch test pass through after a fast block came in an the miner stuck on 11% read (some disks got read after) but no ls later on the drive i probed
.

[nitr8]

@Leozolt - please try the new 1.8.0 build, i cannot reproduce this issue with the dev build or 1.8.0

[kattycris]

i have the same problem, i test miner 1.7 and 1.8 and not fix.....