I have large files that I need to copy to my USB 3.0 external drive. It's a HDD with enclosure.
The problem is the copy speed. When the copy start it start at 110+ MBps, but after few seconds it goes down to 15-20mbps and stays there.
I notice, if I copy single file, wait 15-30 seconds then copy another files the speed is 110+, but whenever I try to copy multiple files it slows down.
After some research, on a source it says, it something that relates to copy buffer. At the start copy buffer is empty, so the copy speed is quite fast.. but later.. it slow down as the copy buffer is full.. but that source does not says anything about how to clear/empty the buffer :(
Is there any work around to copy all the files at same speed, or empty the buffer before copying each file?
here is the screenshot of fast copy:
Here is the screenshot of slow copy:
any help would be highly appreciated..
Notes: It would be easier to understand the behavior if you watch the videos below:
3 Answers
You can't beat the laws of physics, so unless you fix the bottleneck you won't be able to see the initial speeds you see.
There are some things you can do to improve things (but no free magic bullet) -
Defragment the drive periodically. If the drive us fragmented the files need to be scattered and written in the gaps, meaning more disk repositioning.
Ensure the drive is USB3, the port is USB3 and the drivers support USB3.
If you have money, replace the HDD with an SSD (and you can ignore fragmentation.
There is other software which can limit the speeds. This will give you more consistent speeds, bit slower.
You may be able to get better performance by changing the filesystem. NTFS will likely improve your throughput. Increading the block size will speed things up as well.
You're confused about how the buffer works. A buffer is an in-memory waiting room for data to be written. It's useful because making a lot of small writes would be very inefficient. It's better to wait for a short period of time for more writes to queue up and then perform them in a single batch ("flush" the buffer). So operating system buffers all writes to improve performance in these scenarios. The buffer will always get flushed eventually, either when it fills up or after some time, but until it happens, the write isn't complete. (By the way, that's why you should always safely remove USB devices instead of just unplugging them - safe removal flushes buffers.)
What you're observing is the buffer filling up initially with high speed. This data isn't written to disk yet, because disk is slower than the buffer and can't keep up. The OS is simultaneously flushing the buffer to disk and buffering more data to be written, but buffer has fixed size, so buffering slows down to the speed at which OS can free up buffer space by flushing. At this point you're observing disk's real write speed (possibly bottlenecked). Finally, everything is buffered and the progress window disappears, but OS actually keeps flushing the buffer in background until it's empty.
The "slow" speed is disk's true speed. The "fast" speed is just speed of the buffer, but data isn't really written to disk at that speed - it's just being queued to be slowly written.
2Here are the few core things that are making it slow (Tried not to use much technical terms below):
Your source disk's (the disk from which you copy) read speed will be affected if you read multiple files in parallel which will affect the overall transfer rate.
Your CPU will be doing more work because of handling multiple files (in conjunction with IO controller) which reduces the performance and hence it affects your transfer rate
When you send multiple files to the destination disk (external disk) it has to write multiple files and each file's blocks (internal split up of a file on disk) will be scattered across disk and written and disk has to spin for each file block back and forth causing a delay in writing the data. If it had been a single file and if you had contiguous memory space in the disk (if lucky) then this would be really fast. Even if you don't have contiguous memory location, handling a single file and spinning only to store that file will be the sole work of the disk (theoretically) - Your disk's RPM is the factor here.
