FAQ
Dear golang-nuts community,

I want to write a Go program similar to a benchmark that tests different
ciphers for their speed.
The input data ranges to files up to 4GB, which means reading all input
into the RAM at once won't work on the most machines.
Thus, I'm in need for an alternate method to encrypt these big files with
as few I/O throughput as possible, so that the drives don't limit the
cipher implementations.
Another thing I want to achieve is parallelism - which is quite hard when
only a part of the file is in memory.
I already thought of various ways, but none of these seem compelling to me:

1) Read the whole file into the RAM and write the encrypted file
step-by-step using the Writer interface and the cipher.Encrypt() functions.
This won't work on most machines and definitely not on 32bit ones, but it
would be fast enough so that the drives don't limit the cipher's speed.
Due to the whole file being in RAM, it can be split up to achieve
parallelism quite easily.

2) Read the input file step-by-step and write the encrypted file
step-by-step, so that each input is encrypted and written before the next
input comes.
Minimal stress to the RAM, but sequential and thus most likely slower and a
bit more stressful for the drive.
Also, parallelism is afaik not possible, because writing to the output file
needs to be sequential.

3) Buffer some of the input file (~256MB) and write the encrypted file
step-by-step when the next part of the input is converted (puzzle them
together after parallel encryption).
Easier on the drive than 2), easier on the RAM than 1), but most likely
quite hard to do and sync up, slower threads might cause the RAM to fill up
too fast.

I would really appreciate ideas and tips on how to handle this issue, as
I'm pretty much out of them by now... :)

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Search Discussions

  • Dmitry Chestnykh at Feb 20, 2013 at 10:11 am

    On Wednesday, February 20, 2013 11:04:35 AM UTC+1, decipher...@gmail.com wrote:

    I want to write a Go program similar to a benchmark that tests different
    ciphers for their speed.
    If you want to write a program to benchmark ciphers for speed, there's no
    need to encrypt actual files and deal with I/O.

    -Dmitry

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Decipheredgaming at Feb 20, 2013 at 10:16 am
    Well, all ciphers are supposed to start with the same pseudorandom
    plaintext, that's why I created the files.
    Otherwise, one might think that a certain plaintext was "easier" than
    another.

    Am Mittwoch, 20. Februar 2013 11:11:27 UTC+1 schrieb Dmitry Chestnykh:
    On Wednesday, February 20, 2013 11:04:35 AM UTC+1, decipher...@gmail.comwrote:
    I want to write a Go program similar to a benchmark that tests different
    ciphers for their speed.
    If you want to write a program to benchmark ciphers for speed, there's no
    need to encrypt actual files and deal with I/O.

    -Dmitry
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • David DENG at Feb 20, 2013 at 10:21 am
    You can specify a same seed for the random generator before each cipher
    testing. The same plaintext will be generated. Since the random is pseudo.

    David

    On Wednesday, February 20, 2013 6:16:09 PM UTC+8, decipher...@gmail.com
    wrote:
    Well, all ciphers are supposed to start with the same pseudorandom
    plaintext, that's why I created the files.
    Otherwise, one might think that a certain plaintext was "easier" than
    another.

    Am Mittwoch, 20. Februar 2013 11:11:27 UTC+1 schrieb Dmitry Chestnykh:
    On Wednesday, February 20, 2013 11:04:35 AM UTC+1, decipher...@gmail.comwrote:
    I want to write a Go program similar to a benchmark that tests different
    ciphers for their speed.
    If you want to write a program to benchmark ciphers for speed, there's no
    need to encrypt actual files and deal with I/O.

    -Dmitry
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Dmitry Chestnykh at Feb 20, 2013 at 10:32 am

    On Wednesday, February 20, 2013 11:16:09 AM UTC+1, decipher...@gmail.com wrote:

    Well, all ciphers are supposed to start with the same pseudorandom
    plaintext, that's why I created the files.
    Otherwise, one might think that a certain plaintext was "easier" than
    another.
    What's the reason for starting with pseudorandom plaintext as opposed to
    the plaintext consisting of all zeroes?

    Check out how benchmarking is done in the standard library:
    http://tip.golang.org/src/pkg/crypto/rc4/rc4_test.go#L94

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Patrick Mylund Nielsen at Feb 20, 2013 at 10:37 am
    If you find that that's the case, you've likely found side-channel/timing
    vulnerabilities. In general, encrypting the same block or set of blocks
    repeatedly is a fine indicator.

    On Wed, Feb 20, 2013 at 11:16 AM, wrote:

    Well, all ciphers are supposed to start with the same pseudorandom
    plaintext, that's why I created the files.
    Otherwise, one might think that a certain plaintext was "easier" than
    another.

    Am Mittwoch, 20. Februar 2013 11:11:27 UTC+1 schrieb Dmitry Chestnykh:
    On Wednesday, February 20, 2013 11:04:35 AM UTC+1, decipher...@gmail.comwrote:
    I want to write a Go program similar to a benchmark that tests different
    ciphers for their speed.
    If you want to write a program to benchmark ciphers for speed, there's no
    need to encrypt actual files and deal with I/O.

    -Dmitry
    --
    You received this message because you are subscribed to the Google Groups
    "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Volker Dobler at Feb 20, 2013 at 10:58 am

    Am Mittwoch, 20. Februar 2013 11:16:09 UTC+1 schrieb decipher...@gmail.com:
    Well, all ciphers are supposed to start with the same pseudorandom
    plaintext, that's why I created the files.
    Otherwise, one might think that a certain plaintext was "easier" than
    another.
    1. No sensible ciphers has "easy" plaintexts (at least non known ones).

    2. Most ciphers are block ciphers. Pushing in 4GB just repeats encrypting a
    small
    block 4 times more often than encrypting 1 GB.

    You might want to rethink your experimental setup.

    V.

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Decipheredgaming at Feb 20, 2013 at 11:21 am
    You're right on the first one, yeah, but pseudorandom files were the first
    I thought of when creating fixed-size files.
    But your second argument isn't quite applicable here, as I want to compare
    stream ciphers, block ciphers and asymmetric ciphers (RSA).
    And to allow a fair comparison, I'll need to measure those values for each
    cipher - whether it is quite predictable or not.

    Am Mittwoch, 20. Februar 2013 11:58:05 UTC+1 schrieb Volker Dobler:


    Am Mittwoch, 20. Februar 2013 11:16:09 UTC+1 schrieb decipher...@gmail.com
    :
    Well, all ciphers are supposed to start with the same pseudorandom
    plaintext, that's why I created the files.
    Otherwise, one might think that a certain plaintext was "easier" than
    another.
    1. No sensible ciphers has "easy" plaintexts (at least non known ones).

    2. Most ciphers are block ciphers. Pushing in 4GB just repeats encrypting
    a small
    block 4 times more often than encrypting 1 GB.

    You might want to rethink your experimental setup.

    V.
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Niklas Schnelle at Feb 20, 2013 at 11:46 am
    Assymetric ciphers are almost never used to encrypt more than a hash of
    maybe 256 bit. It might not even be safe fro arbitary large inputs.

    On Wednesday, February 20, 2013 12:21:55 PM UTC+1, decipher...@gmail.com
    wrote:
    You're right on the first one, yeah, but pseudorandom files were the first
    I thought of when creating fixed-size files.
    But your second argument isn't quite applicable here, as I want to compare
    stream ciphers, block ciphers and asymmetric ciphers (RSA).
    And to allow a fair comparison, I'll need to measure those values for each
    cipher - whether it is quite predictable or not.

    Am Mittwoch, 20. Februar 2013 11:58:05 UTC+1 schrieb Volker Dobler:


    Am Mittwoch, 20. Februar 2013 11:16:09 UTC+1 schrieb
    decipher...@gmail.com:
    Well, all ciphers are supposed to start with the same pseudorandom
    plaintext, that's why I created the files.
    Otherwise, one might think that a certain plaintext was "easier" than
    another.
    1. No sensible ciphers has "easy" plaintexts (at least non known ones).

    2. Most ciphers are block ciphers. Pushing in 4GB just repeats encrypting
    a small
    block 4 times more often than encrypting 1 GB.

    You might want to rethink your experimental setup.

    V.
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Decipheredgaming at Feb 20, 2013 at 11:52 am
    Well, what about PGP, which is still considered to be the standard for mail
    encryption?
    Or public-key usage for SSH connections, which are used to securely
    administrate servers?

    However, this is not the place for discussing this, as it would be quite
    off topic.

    Am Mittwoch, 20. Februar 2013 12:46:50 UTC+1 schrieb Niklas Schnelle:
    Assymetric ciphers are almost never used to encrypt more than a hash of
    maybe 256 bit. It might not even be safe fro arbitary large inputs.
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Patrick Mylund Nielsen at Feb 20, 2013 at 12:09 pm
    Most public-key-based constructions use asymmetric ciphers, e.g. RSA, to
    share a master key for a symmetrical algorithm, e.g. AES, that they then
    use for the remainder of the session (or for some amount of traffic.) In
    the case of SSH, a secure connection is established regardless of whether
    you're using public-key authentication or not.

    If you want to do something like encrypt a blob of data using RSA, I'd
    recommend looking at NaCl (http://nacl.cr.yp.to/) in go.crypto, KeyCzar, or
    the OpenPGP package, also in go.crypto, rather than implement your own
    construction. (NaCl in particular sounds like what you might be looking
    for.) It's too easy to get wrong.

    If you're looking for hash functions and ciphers that are fast in software,
    check out BLAKE2, SipHash and Salsa20 in addition to AES and RSA.

    On Wed, Feb 20, 2013 at 12:52 PM, wrote:

    Well, what about PGP, which is still considered to be the standard for
    mail encryption?
    Or public-key usage for SSH connections, which are used to securely
    administrate servers?

    However, this is not the place for discussing this, as it would be quite
    off topic.

    Am Mittwoch, 20. Februar 2013 12:46:50 UTC+1 schrieb Niklas Schnelle:
    Assymetric ciphers are almost never used to encrypt more than a hash of
    maybe 256 bit. It might not even be safe fro arbitary large inputs.
    --
    You received this message because you are subscribed to the Google Groups
    "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupgolang-nuts @
categoriesgo
postedFeb 20, '13 at 10:04a
activeFeb 20, '13 at 12:09p
posts11
users6
websitegolang.org

People

Translate

site design / logo © 2022 Grokbase