Processing huge files
#4
(08-24-2022, 10:13 PM)mnrvovrfc Wrote:
(08-22-2022, 05:08 PM)mdijkens Wrote: I sometimes have to process huge (100GB+) csv files.
:
Seriously, I give up.

The thing is that when I try to copy a 4GB file into an USB v3.0 external disk it takes more than half-hour! It seems more ornerous doing it on Linux than on Windows. How long does it take for you to copy a file that large from one disk to another? Or do you need to copy it?

Thank you for this program, at any rate. :tu:

Did not see the yellow text at first...
Just copying with this block based approach is mostly limited by the write speed of the USB stick. 
I have a relatively fast one that reaches around 250MB/s.
But on PCIe thunderbolt this same routine reaches 3GB/s
45y and 2M lines of MBASIC>BASICA>QBASIC>QBX>QB64 experience
Reply


Messages In This Thread
Processing huge files - by mdijkens - 08-22-2022, 05:08 PM
RE: Processing huge files - by mnrvovrfc - 08-24-2022, 10:13 PM
RE: Processing huge files - by mdijkens - 08-24-2022, 10:44 PM
RE: Processing huge files - by mdijkens - 08-24-2022, 10:38 PM



Users browsing this thread: 2 Guest(s)