I want to write a Go program similar to a benchmark that tests different
ciphers for their speed.
The input data ranges to files up to 4GB, which means reading all input
into the RAM at once won't work on the most machines.
Thus, I'm in need for an alternate method to encrypt these big files with
as few I/O throughput as possible, so that the drives don't limit the
cipher implementations.
Another thing I want to achieve is parallelism - which is quite hard when
only a part of the file is in memory.
I already thought of various ways, but none of these seem compelling to me:
1) Read the whole file into the RAM and write the encrypted file
step-by-step using the Writer interface and the cipher.Encrypt() functions.
This won't work on most machines and definitely not on 32bit ones, but it
would be fast enough so that the drives don't limit the cipher's speed.
Due to the whole file being in RAM, it can be split up to achieve
parallelism quite easily.
2) Read the input file step-by-step and write the encrypted file
step-by-step, so that each input is encrypted and written before the next
input comes.
Minimal stress to the RAM, but sequential and thus most likely slower and a
bit more stressful for the drive.
Also, parallelism is afaik not possible, because writing to the output file
needs to be sequential.
3) Buffer some of the input file (~256MB) and write the encrypted file
step-by-step when the next part of the input is converted (puzzle them
together after parallel encryption).
Easier on the drive than 2), easier on the RAM than 1), but most likely
quite hard to do and sync up, slower threads might cause the RAM to fill up
too fast.
I would really appreciate ideas and tips on how to handle this issue, as
I'm pretty much out of them by now... :)
--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.