Grokbase Groups Kafka users June 2014
FAQ
Hi,I wanted to re-ignite the discussion around Apache Kafka Security. This
is a huge bottleneck (non-starter in some cases) for a lot of organizations
(due to regulatory, compliance and other requirements). Below are my
suggestions for specific changes in Kafka to accommodate security
requirements. This comes from what folks are doing "in the wild" to
workaround and implement security with Kafka as it is today and also what I
have discovered from organizations about their blockers. It also picks up
from the wiki (which I should have time to update later in the week based
on the below and feedback from the thread).

1) Transport Layer Security (i.e. SSL)

This also includes client authentication in addition to in-transit security
layer. This work has been picked up here
https://issues.apache.org/jira/browse/KAFKA-1477 and do appreciate any
thoughts, comments, feedback, tomatoes, whatever for this patch. It is a
pickup from the fork of the work first done here
https://github.com/relango/kafka/tree/kafka_security.

2) Data encryption at rest.

This is very important and something that can be facilitated within the
wire protocol. It requires an additional map data structure for the
"encrypted [data encryption key]". With this map (either in your object or
in the wire protocol) you can store the dynamically generated symmetric key
(for each message) and then encrypt the data using that dynamically
generated key. You then encrypt the encryption key using each public key
for whom is expected to be able to decrypt the encryption key to then
decrypt the message. For each public key encrypted symmetric key (which is
now the "encrypted [data encryption key]" along with which public key it
was encrypted with for (so a map of [publicKey] =
encryptedDataEncryptionKey) as a chain. Other patterns can be implemented
but this is a pretty standard digital enveloping [0] pattern with only 1
field added. Other patterns should be able to use that field to-do their
implementation too.

3) Non-repudiation and long term non-repudiation.

Non-repudiation is proving data hasn't changed. This is often (if not
always) done with x509 public certificates (chained to a certificate
authority).

Long term non-repudiation is what happens when the certificates of the
certificate authority are expired (or revoked) and everything ever signed
(ever) with that certificate's public key then becomes "no longer provable
as ever being authentic". That is where RFC3126 [1] and RFC3161 [2] come
in (or worm drives [hardware], etc).

For either (or both) of these it is an operation of the encryptor to
sign/hash the data (with or without third party trusted timestap of the
signing event) and encrypt that with their own private key and distribute
the results (before and after encrypting if required) along with their
public key. This structure is a bit more complex but feasible, it is a map
of digital signature formats and the chain of dig sig attestations. The
map's key being the method (i.e. CRC32, PKCS7 [3], XmlDigSig [4]) and then
a list of map where that key is "purpose" of signature (what your attesting
too). As a sibling field to the list another field for "the attester" as
bytes (e.g. their PKCS12 [5] for the map of PKCS7 signatures).

4) Authorization

We should have a policy of "404" for data, topics, partitions (etc) if
authenticated connections do not have access. In "secure mode" any non
authenticated connections should get a "404" type message on everything.
Knowing "something is there" is a security risk in many uses cases. So if
you don't have access you don't even see it. Baking "that" into Kafka
along with some interface for entitlement (access management) systems
(pretty standard) is all that I think needs to be done to the core project.
  I want to tackle item later in the year after summer after the other three
are complete.

I look forward to thoughts on this and anyone else interested in working
with us on these items.

[0]
http://www.emc.com/emc-plus/rsa-labs/standards-initiatives/what-is-a-digital-envelope.htm
[1] http://tools.ietf.org/html/rfc3126
[2] http://tools.ietf.org/html/rfc3161
[3]
http://www.emc.com/emc-plus/rsa-labs/standards-initiatives/pkcs-7-cryptographic-message-syntax-standar.htm
[4] http://en.wikipedia.org/wiki/XML_Signature
[5] http://en.wikipedia.org/wiki/PKCS_12

/*******************************************
  Joe Stein
  Founder, Principal Consultant
  Big Data Open Source Security LLC
  http://www.stealth.ly
  Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop>
********************************************/

Search Discussions

  • Robert Rodgers at Jun 3, 2014 at 8:03 pm
    ... client specific presented information, signed in some way, listing topic permissions. read, write, list.

    TLS lends itself to client certificates.

    On Jun 3, 2014, at 12:57 PM, Joe Stein wrote:

    4) Authorization

    We should have a policy of "404" for data, topics, partitions (etc) if
    authenticated connections do not have access. In "secure mode" any non
    authenticated connections should get a "404" type message on everything.
    Knowing "something is there" is a security risk in many uses cases. So if
    you don't have access you don't even see it. Baking "that" into Kafka
    along with some interface for entitlement (access management) systems
    (pretty standard) is all that I think needs to be done to the core project.
    I want to tackle item later in the year after summer after the other three
    are complete.
  • Todd Palino at Jun 3, 2014 at 10:12 pm
    I think that¹s one option. What I would offer here is that we need to
    separate out the concepts of authorization and authentication.
    Authentication should definitely be modular, so that we can plug in
    appropriate schemes depending on the organization. For example, you may
    want client certificates, I may want radius, and someone else is going to
    want LDAP.

    Authorization is the other piece that¹s needed, and that could be
    internal. Since what you¹re authorizing (topic name, read or write, may
    rate limiting) is specific to the application, it may not make sense to
    modularize it.

    -Todd
    On 6/3/14, 1:03 PM, "Robert Rodgers" wrote:

    ... client specific presented information, signed in some way, listing
    topic permissions. read, write, list.

    TLS lends itself to client certificates.

    On Jun 3, 2014, at 12:57 PM, Joe Stein wrote:

    4) Authorization

    We should have a policy of "404" for data, topics, partitions (etc) if
    authenticated connections do not have access. In "secure mode" any non
    authenticated connections should get a "404" type message on everything.
    Knowing "something is there" is a security risk in many uses cases. So
    if
    you don't have access you don't even see it. Baking "that" into Kafka
    along with some interface for entitlement (access management) systems
    (pretty standard) is all that I think needs to be done to the core
    project.
    I want to tackle item later in the year after summer after the other
    three
    are complete.
  • Joe Stein at Jun 4, 2014 at 10:44 am
    Hey Todd, I think you are right on both points.

    Maybe instead of modularizing authorization we could instead support some
    feature like being able to associate "labels" for the application specific
    items (topic name, reads/writes, delete topic, change config, rate
    limiting, etc) and then accept a file or something with a list
    entitlements (where name is the "label" and value is some class that gets
    run with the ability to extend your own or integrate with some other
    system). e.g. you associate with a topic named "hasCreditCardDataInIt" a
    label == "PCIDSS". Your config would be "PCIDSS" = "CheckForPCIDSS" and
    in CheckForPCIDSS code you could do functions like "verify the topic is
    going over an encrypted channel return true else false", etc, whatever.

    - Joe Stein

    On Tue, Jun 3, 2014 at 6:12 PM, Todd Palino wrote:

    I think that¹s one option. What I would offer here is that we need to
    separate out the concepts of authorization and authentication.
    Authentication should definitely be modular, so that we can plug in
    appropriate schemes depending on the organization. For example, you may
    want client certificates, I may want radius, and someone else is going to
    want LDAP.

    Authorization is the other piece that¹s needed, and that could be
    internal. Since what you¹re authorizing (topic name, read or write, may
    rate limiting) is specific to the application, it may not make sense to
    modularize it.

    -Todd
    On 6/3/14, 1:03 PM, "Robert Rodgers" wrote:

    ... client specific presented information, signed in some way, listing
    topic permissions. read, write, list.

    TLS lends itself to client certificates.

    On Jun 3, 2014, at 12:57 PM, Joe Stein wrote:

    4) Authorization

    We should have a policy of "404" for data, topics, partitions (etc) if
    authenticated connections do not have access. In "secure mode" any non
    authenticated connections should get a "404" type message on everything.
    Knowing "something is there" is a security risk in many uses cases. So
    if
    you don't have access you don't even see it. Baking "that" into Kafka
    along with some interface for entitlement (access management) systems
    (pretty standard) is all that I think needs to be done to the core
    project.
    I want to tackle item later in the year after summer after the other
    three
    are complete.
  • Jay Kreps at Jun 4, 2014 at 5:11 pm
    Hey Joe,

    Thanks for kicking this discussion off! I totally agree that for something
    that acts as a central message broker security is critical feature. I think
    a number of people have been interested in this topic and several people
    have put effort into special purpose security efforts.

    Since most the LinkedIn folks are working on the consumer right now I think
    this would be a great project for any other interested people to take on.
    There are some challenges in doing these things distributed but it can also
    be a lot of fun.

    I think a good first step would be to get a written plan we can all agree
    on for how things should work. Then we can break things down into chunks
    that can be done independently while still aiming at a good end state.

    I had tried to write up some notes that summarized at least the thoughts I
    had had on security:
    https://cwiki.apache.org/confluence/display/KAFKA/Security

    What do you think of that?

    One assumption I had (which may be incorrect) is that although we want all
    the things in your list, the two most pressing would be authentication and
    authorization, and that was all that write up covered. You have more
    experience in this domain, so I wonder how you would prioritize?

    Those notes are really sketchy, so I think the first goal I would have
    would be to get to a real spec we can all agree on and discuss. A lot of
    the security stuff has a high human interaction element and needs to work
    in pretty different domains and different companies so getting this kind of
    review is important.

    -Jay

    On Tue, Jun 3, 2014 at 12:57 PM, Joe Stein wrote:

    Hi,I wanted to re-ignite the discussion around Apache Kafka Security. This
    is a huge bottleneck (non-starter in some cases) for a lot of organizations
    (due to regulatory, compliance and other requirements). Below are my
    suggestions for specific changes in Kafka to accommodate security
    requirements. This comes from what folks are doing "in the wild" to
    workaround and implement security with Kafka as it is today and also what I
    have discovered from organizations about their blockers. It also picks up
    from the wiki (which I should have time to update later in the week based
    on the below and feedback from the thread).

    1) Transport Layer Security (i.e. SSL)

    This also includes client authentication in addition to in-transit security
    layer. This work has been picked up here
    https://issues.apache.org/jira/browse/KAFKA-1477 and do appreciate any
    thoughts, comments, feedback, tomatoes, whatever for this patch. It is a
    pickup from the fork of the work first done here
    https://github.com/relango/kafka/tree/kafka_security.

    2) Data encryption at rest.

    This is very important and something that can be facilitated within the
    wire protocol. It requires an additional map data structure for the
    "encrypted [data encryption key]". With this map (either in your object or
    in the wire protocol) you can store the dynamically generated symmetric key
    (for each message) and then encrypt the data using that dynamically
    generated key. You then encrypt the encryption key using each public key
    for whom is expected to be able to decrypt the encryption key to then
    decrypt the message. For each public key encrypted symmetric key (which is
    now the "encrypted [data encryption key]" along with which public key it
    was encrypted with for (so a map of [publicKey] =
    encryptedDataEncryptionKey) as a chain. Other patterns can be implemented
    but this is a pretty standard digital enveloping [0] pattern with only 1
    field added. Other patterns should be able to use that field to-do their
    implementation too.

    3) Non-repudiation and long term non-repudiation.

    Non-repudiation is proving data hasn't changed. This is often (if not
    always) done with x509 public certificates (chained to a certificate
    authority).

    Long term non-repudiation is what happens when the certificates of the
    certificate authority are expired (or revoked) and everything ever signed
    (ever) with that certificate's public key then becomes "no longer provable
    as ever being authentic". That is where RFC3126 [1] and RFC3161 [2] come
    in (or worm drives [hardware], etc).

    For either (or both) of these it is an operation of the encryptor to
    sign/hash the data (with or without third party trusted timestap of the
    signing event) and encrypt that with their own private key and distribute
    the results (before and after encrypting if required) along with their
    public key. This structure is a bit more complex but feasible, it is a map
    of digital signature formats and the chain of dig sig attestations. The
    map's key being the method (i.e. CRC32, PKCS7 [3], XmlDigSig [4]) and then
    a list of map where that key is "purpose" of signature (what your attesting
    too). As a sibling field to the list another field for "the attester" as
    bytes (e.g. their PKCS12 [5] for the map of PKCS7 signatures).

    4) Authorization

    We should have a policy of "404" for data, topics, partitions (etc) if
    authenticated connections do not have access. In "secure mode" any non
    authenticated connections should get a "404" type message on everything.
    Knowing "something is there" is a security risk in many uses cases. So if
    you don't have access you don't even see it. Baking "that" into Kafka
    along with some interface for entitlement (access management) systems
    (pretty standard) is all that I think needs to be done to the core project.
    I want to tackle item later in the year after summer after the other three
    are complete.

    I look forward to thoughts on this and anyone else interested in working
    with us on these items.

    [0]

    http://www.emc.com/emc-plus/rsa-labs/standards-initiatives/what-is-a-digital-envelope.htm
    [1] http://tools.ietf.org/html/rfc3126
    [2] http://tools.ietf.org/html/rfc3161
    [3]

    http://www.emc.com/emc-plus/rsa-labs/standards-initiatives/pkcs-7-cryptographic-message-syntax-standar.htm
    [4] http://en.wikipedia.org/wiki/XML_Signature
    [5] http://en.wikipedia.org/wiki/PKCS_12

    /*******************************************
    Joe Stein
    Founder, Principal Consultant
    Big Data Open Source Security LLC
    http://www.stealth.ly
    Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop>
    ********************************************/
  • Joe Stein at Jun 5, 2014 at 12:57 am
    I like the idea of working on the spec and prioritizing. I will update the
    wiki.

    - Joestein

    On Wed, Jun 4, 2014 at 1:11 PM, Jay Kreps wrote:

    Hey Joe,

    Thanks for kicking this discussion off! I totally agree that for something
    that acts as a central message broker security is critical feature. I think
    a number of people have been interested in this topic and several people
    have put effort into special purpose security efforts.

    Since most the LinkedIn folks are working on the consumer right now I think
    this would be a great project for any other interested people to take on.
    There are some challenges in doing these things distributed but it can also
    be a lot of fun.

    I think a good first step would be to get a written plan we can all agree
    on for how things should work. Then we can break things down into chunks
    that can be done independently while still aiming at a good end state.

    I had tried to write up some notes that summarized at least the thoughts I
    had had on security:
    https://cwiki.apache.org/confluence/display/KAFKA/Security

    What do you think of that?

    One assumption I had (which may be incorrect) is that although we want all
    the things in your list, the two most pressing would be authentication and
    authorization, and that was all that write up covered. You have more
    experience in this domain, so I wonder how you would prioritize?

    Those notes are really sketchy, so I think the first goal I would have
    would be to get to a real spec we can all agree on and discuss. A lot of
    the security stuff has a high human interaction element and needs to work
    in pretty different domains and different companies so getting this kind of
    review is important.

    -Jay

    On Tue, Jun 3, 2014 at 12:57 PM, Joe Stein wrote:

    Hi,I wanted to re-ignite the discussion around Apache Kafka Security. This
    is a huge bottleneck (non-starter in some cases) for a lot of
    organizations
    (due to regulatory, compliance and other requirements). Below are my
    suggestions for specific changes in Kafka to accommodate security
    requirements. This comes from what folks are doing "in the wild" to
    workaround and implement security with Kafka as it is today and also what I
    have discovered from organizations about their blockers. It also picks up
    from the wiki (which I should have time to update later in the week based
    on the below and feedback from the thread).

    1) Transport Layer Security (i.e. SSL)

    This also includes client authentication in addition to in-transit security
    layer. This work has been picked up here
    https://issues.apache.org/jira/browse/KAFKA-1477 and do appreciate any
    thoughts, comments, feedback, tomatoes, whatever for this patch. It is a
    pickup from the fork of the work first done here
    https://github.com/relango/kafka/tree/kafka_security.

    2) Data encryption at rest.

    This is very important and something that can be facilitated within the
    wire protocol. It requires an additional map data structure for the
    "encrypted [data encryption key]". With this map (either in your object or
    in the wire protocol) you can store the dynamically generated symmetric key
    (for each message) and then encrypt the data using that dynamically
    generated key. You then encrypt the encryption key using each public key
    for whom is expected to be able to decrypt the encryption key to then
    decrypt the message. For each public key encrypted symmetric key (which is
    now the "encrypted [data encryption key]" along with which public key it
    was encrypted with for (so a map of [publicKey] =
    encryptedDataEncryptionKey) as a chain. Other patterns can be
    implemented
    but this is a pretty standard digital enveloping [0] pattern with only 1
    field added. Other patterns should be able to use that field to-do their
    implementation too.

    3) Non-repudiation and long term non-repudiation.

    Non-repudiation is proving data hasn't changed. This is often (if not
    always) done with x509 public certificates (chained to a certificate
    authority).

    Long term non-repudiation is what happens when the certificates of the
    certificate authority are expired (or revoked) and everything ever signed
    (ever) with that certificate's public key then becomes "no longer provable
    as ever being authentic". That is where RFC3126 [1] and RFC3161 [2] come
    in (or worm drives [hardware], etc).

    For either (or both) of these it is an operation of the encryptor to
    sign/hash the data (with or without third party trusted timestap of the
    signing event) and encrypt that with their own private key and distribute
    the results (before and after encrypting if required) along with their
    public key. This structure is a bit more complex but feasible, it is a map
    of digital signature formats and the chain of dig sig attestations. The
    map's key being the method (i.e. CRC32, PKCS7 [3], XmlDigSig [4]) and then
    a list of map where that key is "purpose" of signature (what your attesting
    too). As a sibling field to the list another field for "the attester" as
    bytes (e.g. their PKCS12 [5] for the map of PKCS7 signatures).

    4) Authorization

    We should have a policy of "404" for data, topics, partitions (etc) if
    authenticated connections do not have access. In "secure mode" any non
    authenticated connections should get a "404" type message on everything.
    Knowing "something is there" is a security risk in many uses cases. So if
    you don't have access you don't even see it. Baking "that" into Kafka
    along with some interface for entitlement (access management) systems
    (pretty standard) is all that I think needs to be done to the core project.
    I want to tackle item later in the year after summer after the other three
    are complete.

    I look forward to thoughts on this and anyone else interested in working
    with us on these items.

    [0]

    http://www.emc.com/emc-plus/rsa-labs/standards-initiatives/what-is-a-digital-envelope.htm
    [1] http://tools.ietf.org/html/rfc3126
    [2] http://tools.ietf.org/html/rfc3161
    [3]

    http://www.emc.com/emc-plus/rsa-labs/standards-initiatives/pkcs-7-cryptographic-message-syntax-standar.htm
    [4] http://en.wikipedia.org/wiki/XML_Signature
    [5] http://en.wikipedia.org/wiki/PKCS_12

    /*******************************************
    Joe Stein
    Founder, Principal Consultant
    Big Data Open Source Security LLC
    http://www.stealth.ly
    Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop>
    ********************************************/
  • Rajasekar Elango at Jun 5, 2014 at 3:56 pm
    Hi Jay,

    Thanks for putting together a spec for security.

    Joe,

    Looks "Securing zookeeper.." part has been deleted from assumptions
    section. communication with zookeeper need to be secured as well to make
    entire kafka cluster secure. It may or may not require changes to kafka.
    But it's good to have it in spec.

    I could not find a link to edit the page after login into wiki. Do I need
    any special permission to make edits?

    Thanks,
    Raja.

    On Wed, Jun 4, 2014 at 8:57 PM, Joe Stein wrote:

    I like the idea of working on the spec and prioritizing. I will update the
    wiki.

    - Joestein

    On Wed, Jun 4, 2014 at 1:11 PM, Jay Kreps wrote:

    Hey Joe,

    Thanks for kicking this discussion off! I totally agree that for something
    that acts as a central message broker security is critical feature. I think
    a number of people have been interested in this topic and several people
    have put effort into special purpose security efforts.

    Since most the LinkedIn folks are working on the consumer right now I think
    this would be a great project for any other interested people to take on.
    There are some challenges in doing these things distributed but it can also
    be a lot of fun.

    I think a good first step would be to get a written plan we can all agree
    on for how things should work. Then we can break things down into chunks
    that can be done independently while still aiming at a good end state.

    I had tried to write up some notes that summarized at least the thoughts I
    had had on security:
    https://cwiki.apache.org/confluence/display/KAFKA/Security

    What do you think of that?

    One assumption I had (which may be incorrect) is that although we want all
    the things in your list, the two most pressing would be authentication and
    authorization, and that was all that write up covered. You have more
    experience in this domain, so I wonder how you would prioritize?

    Those notes are really sketchy, so I think the first goal I would have
    would be to get to a real spec we can all agree on and discuss. A lot of
    the security stuff has a high human interaction element and needs to work
    in pretty different domains and different companies so getting this kind of
    review is important.

    -Jay

    On Tue, Jun 3, 2014 at 12:57 PM, Joe Stein wrote:

    Hi,I wanted to re-ignite the discussion around Apache Kafka Security. This
    is a huge bottleneck (non-starter in some cases) for a lot of
    organizations
    (due to regulatory, compliance and other requirements). Below are my
    suggestions for specific changes in Kafka to accommodate security
    requirements. This comes from what folks are doing "in the wild" to
    workaround and implement security with Kafka as it is today and also what I
    have discovered from organizations about their blockers. It also picks
    up
    from the wiki (which I should have time to update later in the week
    based
    on the below and feedback from the thread).

    1) Transport Layer Security (i.e. SSL)

    This also includes client authentication in addition to in-transit security
    layer. This work has been picked up here
    https://issues.apache.org/jira/browse/KAFKA-1477 and do appreciate any
    thoughts, comments, feedback, tomatoes, whatever for this patch. It
    is a
    pickup from the fork of the work first done here
    https://github.com/relango/kafka/tree/kafka_security.

    2) Data encryption at rest.

    This is very important and something that can be facilitated within the
    wire protocol. It requires an additional map data structure for the
    "encrypted [data encryption key]". With this map (either in your object or
    in the wire protocol) you can store the dynamically generated symmetric key
    (for each message) and then encrypt the data using that dynamically
    generated key. You then encrypt the encryption key using each public
    key
    for whom is expected to be able to decrypt the encryption key to then
    decrypt the message. For each public key encrypted symmetric key
    (which
    is
    now the "encrypted [data encryption key]" along with which public key
    it
    was encrypted with for (so a map of [publicKey] =
    encryptedDataEncryptionKey) as a chain. Other patterns can be
    implemented
    but this is a pretty standard digital enveloping [0] pattern with only
    1
    field added. Other patterns should be able to use that field to-do
    their
    implementation too.

    3) Non-repudiation and long term non-repudiation.

    Non-repudiation is proving data hasn't changed. This is often (if not
    always) done with x509 public certificates (chained to a certificate
    authority).

    Long term non-repudiation is what happens when the certificates of the
    certificate authority are expired (or revoked) and everything ever
    signed
    (ever) with that certificate's public key then becomes "no longer provable
    as ever being authentic". That is where RFC3126 [1] and RFC3161 [2]
    come
    in (or worm drives [hardware], etc).

    For either (or both) of these it is an operation of the encryptor to
    sign/hash the data (with or without third party trusted timestap of the
    signing event) and encrypt that with their own private key and
    distribute
    the results (before and after encrypting if required) along with their
    public key. This structure is a bit more complex but feasible, it is a map
    of digital signature formats and the chain of dig sig attestations.
    The
    map's key being the method (i.e. CRC32, PKCS7 [3], XmlDigSig [4]) and then
    a list of map where that key is "purpose" of signature (what your attesting
    too). As a sibling field to the list another field for "the attester"
    as
    bytes (e.g. their PKCS12 [5] for the map of PKCS7 signatures).

    4) Authorization

    We should have a policy of "404" for data, topics, partitions (etc) if
    authenticated connections do not have access. In "secure mode" any non
    authenticated connections should get a "404" type message on
    everything.
    Knowing "something is there" is a security risk in many uses cases. So if
    you don't have access you don't even see it. Baking "that" into Kafka
    along with some interface for entitlement (access management) systems
    (pretty standard) is all that I think needs to be done to the core project.
    I want to tackle item later in the year after summer after the other three
    are complete.

    I look forward to thoughts on this and anyone else interested in
    working
    with us on these items.

    [0]
    http://www.emc.com/emc-plus/rsa-labs/standards-initiatives/what-is-a-digital-envelope.htm
    http://www.emc.com/emc-plus/rsa-labs/standards-initiatives/pkcs-7-cryptographic-message-syntax-standar.htm
    [4] http://en.wikipedia.org/wiki/XML_Signature
    [5] http://en.wikipedia.org/wiki/PKCS_12

    /*******************************************
    Joe Stein
    Founder, Principal Consultant
    Big Data Open Source Security LLC
    http://www.stealth.ly
    Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop>
    ********************************************/


    --
    Thanks,
    Raja.
  • Joe Stein at Jun 5, 2014 at 4:02 pm
    Raja, you need to sign an ICLA http://www.apache.org/licenses/icla.txt once
    that is on file your user can get permed to contribute.

    I think securing communication to "offset & broker management source" which
    can be a zookeeper implementation is important. I will elaborate more on
    that with the other edits I have for the page.

    /*******************************************
      Joe Stein
      Founder, Principal Consultant
      Big Data Open Source Security LLC
      http://www.stealth.ly
      Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop>
    ********************************************/

    On Thu, Jun 5, 2014 at 11:56 AM, Rajasekar Elango wrote:

    Hi Jay,

    Thanks for putting together a spec for security.

    Joe,

    Looks "Securing zookeeper.." part has been deleted from assumptions
    section. communication with zookeeper need to be secured as well to make
    entire kafka cluster secure. It may or may not require changes to kafka.
    But it's good to have it in spec.

    I could not find a link to edit the page after login into wiki. Do I need
    any special permission to make edits?

    Thanks,
    Raja.

    On Wed, Jun 4, 2014 at 8:57 PM, Joe Stein wrote:

    I like the idea of working on the spec and prioritizing. I will update the
    wiki.

    - Joestein

    On Wed, Jun 4, 2014 at 1:11 PM, Jay Kreps wrote:

    Hey Joe,

    Thanks for kicking this discussion off! I totally agree that for something
    that acts as a central message broker security is critical feature. I think
    a number of people have been interested in this topic and several
    people
    have put effort into special purpose security efforts.

    Since most the LinkedIn folks are working on the consumer right now I think
    this would be a great project for any other interested people to take
    on.
    There are some challenges in doing these things distributed but it can also
    be a lot of fun.

    I think a good first step would be to get a written plan we can all
    agree
    on for how things should work. Then we can break things down into
    chunks
    that can be done independently while still aiming at a good end state.

    I had tried to write up some notes that summarized at least the
    thoughts
    I
    had had on security:
    https://cwiki.apache.org/confluence/display/KAFKA/Security

    What do you think of that?

    One assumption I had (which may be incorrect) is that although we want all
    the things in your list, the two most pressing would be authentication and
    authorization, and that was all that write up covered. You have more
    experience in this domain, so I wonder how you would prioritize?

    Those notes are really sketchy, so I think the first goal I would have
    would be to get to a real spec we can all agree on and discuss. A lot
    of
    the security stuff has a high human interaction element and needs to
    work
    in pretty different domains and different companies so getting this
    kind
    of
    review is important.

    -Jay

    On Tue, Jun 3, 2014 at 12:57 PM, Joe Stein wrote:

    Hi,I wanted to re-ignite the discussion around Apache Kafka Security. This
    is a huge bottleneck (non-starter in some cases) for a lot of
    organizations
    (due to regulatory, compliance and other requirements). Below are my
    suggestions for specific changes in Kafka to accommodate security
    requirements. This comes from what folks are doing "in the wild" to
    workaround and implement security with Kafka as it is today and also what I
    have discovered from organizations about their blockers. It also
    picks
    up
    from the wiki (which I should have time to update later in the week
    based
    on the below and feedback from the thread).

    1) Transport Layer Security (i.e. SSL)

    This also includes client authentication in addition to in-transit security
    layer. This work has been picked up here
    https://issues.apache.org/jira/browse/KAFKA-1477 and do appreciate
    any
    thoughts, comments, feedback, tomatoes, whatever for this patch. It
    is a
    pickup from the fork of the work first done here
    https://github.com/relango/kafka/tree/kafka_security.

    2) Data encryption at rest.

    This is very important and something that can be facilitated within
    the
    wire protocol. It requires an additional map data structure for the
    "encrypted [data encryption key]". With this map (either in your
    object
    or
    in the wire protocol) you can store the dynamically generated
    symmetric
    key
    (for each message) and then encrypt the data using that dynamically
    generated key. You then encrypt the encryption key using each public
    key
    for whom is expected to be able to decrypt the encryption key to then
    decrypt the message. For each public key encrypted symmetric key
    (which
    is
    now the "encrypted [data encryption key]" along with which public key
    it
    was encrypted with for (so a map of [publicKey] =
    encryptedDataEncryptionKey) as a chain. Other patterns can be
    implemented
    but this is a pretty standard digital enveloping [0] pattern with
    only
    1
    field added. Other patterns should be able to use that field to-do
    their
    implementation too.

    3) Non-repudiation and long term non-repudiation.

    Non-repudiation is proving data hasn't changed. This is often (if
    not
    always) done with x509 public certificates (chained to a certificate
    authority).

    Long term non-repudiation is what happens when the certificates of
    the
    certificate authority are expired (or revoked) and everything ever
    signed
    (ever) with that certificate's public key then becomes "no longer provable
    as ever being authentic". That is where RFC3126 [1] and RFC3161 [2]
    come
    in (or worm drives [hardware], etc).

    For either (or both) of these it is an operation of the encryptor to
    sign/hash the data (with or without third party trusted timestap of
    the
    signing event) and encrypt that with their own private key and
    distribute
    the results (before and after encrypting if required) along with
    their
    public key. This structure is a bit more complex but feasible, it is
    a
    map
    of digital signature formats and the chain of dig sig attestations.
    The
    map's key being the method (i.e. CRC32, PKCS7 [3], XmlDigSig [4]) and then
    a list of map where that key is "purpose" of signature (what your attesting
    too). As a sibling field to the list another field for "the
    attester"
    as
    bytes (e.g. their PKCS12 [5] for the map of PKCS7 signatures).

    4) Authorization

    We should have a policy of "404" for data, topics, partitions (etc)
    if
    authenticated connections do not have access. In "secure mode" any
    non
    authenticated connections should get a "404" type message on
    everything.
    Knowing "something is there" is a security risk in many uses cases.
    So
    if
    you don't have access you don't even see it. Baking "that" into
    Kafka
    along with some interface for entitlement (access management) systems
    (pretty standard) is all that I think needs to be done to the core project.
    I want to tackle item later in the year after summer after the other three
    are complete.

    I look forward to thoughts on this and anyone else interested in
    working
    with us on these items.

    [0]
    http://www.emc.com/emc-plus/rsa-labs/standards-initiatives/what-is-a-digital-envelope.htm
    http://www.emc.com/emc-plus/rsa-labs/standards-initiatives/pkcs-7-cryptographic-message-syntax-standar.htm
    [4] http://en.wikipedia.org/wiki/XML_Signature
    [5] http://en.wikipedia.org/wiki/PKCS_12

    /*******************************************
    Joe Stein
    Founder, Principal Consultant
    Big Data Open Source Security LLC
    http://www.stealth.ly
    Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop>
    ********************************************/


    --
    Thanks,
    Raja.
  • Jay Kreps at Jun 5, 2014 at 9:02 pm
    Hey Joe,

    I don't really understand the sections you added to the wiki. Can you
    clarify them?

    Is non-repudiation what SASL would call integrity checks? If so don't SSL
    and and many of the SASL schemes already support this as well as
    on-the-wire encryption?

    Or are you proposing an on-disk encryption scheme? Is this actually needed?
    Isn't a on-the-wire encryption when combined with mutual authentication and
    permissions sufficient for most uses?

    On-disk encryption seems unnecessary because if an attacker can get root on
    the kafka boxes it can potentially modify Kafka to do anything he or she
    wants with data. So this seems to break any security model.

    I understand the problem of a large organization not really having a
    trusted network and wanting to secure data transfer and limit and audit
    data access. The uses for these other things I don't totally understand.

    Also it would be worth understanding the state of other messaging and
    storage systems (Hadoop, dbs, etc). What features do they support. I think
    there is a sense in which you don't have to run faster than the bear, but
    only faster then your friends. :-)

    -Jay

    On Wed, Jun 4, 2014 at 5:57 PM, Joe Stein wrote:

    I like the idea of working on the spec and prioritizing. I will update the
    wiki.

    - Joestein

    On Wed, Jun 4, 2014 at 1:11 PM, Jay Kreps wrote:

    Hey Joe,

    Thanks for kicking this discussion off! I totally agree that for something
    that acts as a central message broker security is critical feature. I think
    a number of people have been interested in this topic and several people
    have put effort into special purpose security efforts.

    Since most the LinkedIn folks are working on the consumer right now I think
    this would be a great project for any other interested people to take on.
    There are some challenges in doing these things distributed but it can also
    be a lot of fun.

    I think a good first step would be to get a written plan we can all agree
    on for how things should work. Then we can break things down into chunks
    that can be done independently while still aiming at a good end state.

    I had tried to write up some notes that summarized at least the thoughts I
    had had on security:
    https://cwiki.apache.org/confluence/display/KAFKA/Security

    What do you think of that?

    One assumption I had (which may be incorrect) is that although we want all
    the things in your list, the two most pressing would be authentication and
    authorization, and that was all that write up covered. You have more
    experience in this domain, so I wonder how you would prioritize?

    Those notes are really sketchy, so I think the first goal I would have
    would be to get to a real spec we can all agree on and discuss. A lot of
    the security stuff has a high human interaction element and needs to work
    in pretty different domains and different companies so getting this kind of
    review is important.

    -Jay

    On Tue, Jun 3, 2014 at 12:57 PM, Joe Stein wrote:

    Hi,I wanted to re-ignite the discussion around Apache Kafka Security. This
    is a huge bottleneck (non-starter in some cases) for a lot of
    organizations
    (due to regulatory, compliance and other requirements). Below are my
    suggestions for specific changes in Kafka to accommodate security
    requirements. This comes from what folks are doing "in the wild" to
    workaround and implement security with Kafka as it is today and also what I
    have discovered from organizations about their blockers. It also picks
    up
    from the wiki (which I should have time to update later in the week
    based
    on the below and feedback from the thread).

    1) Transport Layer Security (i.e. SSL)

    This also includes client authentication in addition to in-transit security
    layer. This work has been picked up here
    https://issues.apache.org/jira/browse/KAFKA-1477 and do appreciate any
    thoughts, comments, feedback, tomatoes, whatever for this patch. It
    is a
    pickup from the fork of the work first done here
    https://github.com/relango/kafka/tree/kafka_security.

    2) Data encryption at rest.

    This is very important and something that can be facilitated within the
    wire protocol. It requires an additional map data structure for the
    "encrypted [data encryption key]". With this map (either in your object or
    in the wire protocol) you can store the dynamically generated symmetric key
    (for each message) and then encrypt the data using that dynamically
    generated key. You then encrypt the encryption key using each public
    key
    for whom is expected to be able to decrypt the encryption key to then
    decrypt the message. For each public key encrypted symmetric key
    (which
    is
    now the "encrypted [data encryption key]" along with which public key
    it
    was encrypted with for (so a map of [publicKey] =
    encryptedDataEncryptionKey) as a chain. Other patterns can be
    implemented
    but this is a pretty standard digital enveloping [0] pattern with only
    1
    field added. Other patterns should be able to use that field to-do
    their
    implementation too.

    3) Non-repudiation and long term non-repudiation.

    Non-repudiation is proving data hasn't changed. This is often (if not
    always) done with x509 public certificates (chained to a certificate
    authority).

    Long term non-repudiation is what happens when the certificates of the
    certificate authority are expired (or revoked) and everything ever
    signed
    (ever) with that certificate's public key then becomes "no longer provable
    as ever being authentic". That is where RFC3126 [1] and RFC3161 [2]
    come
    in (or worm drives [hardware], etc).

    For either (or both) of these it is an operation of the encryptor to
    sign/hash the data (with or without third party trusted timestap of the
    signing event) and encrypt that with their own private key and
    distribute
    the results (before and after encrypting if required) along with their
    public key. This structure is a bit more complex but feasible, it is a map
    of digital signature formats and the chain of dig sig attestations.
    The
    map's key being the method (i.e. CRC32, PKCS7 [3], XmlDigSig [4]) and then
    a list of map where that key is "purpose" of signature (what your attesting
    too). As a sibling field to the list another field for "the attester"
    as
    bytes (e.g. their PKCS12 [5] for the map of PKCS7 signatures).

    4) Authorization

    We should have a policy of "404" for data, topics, partitions (etc) if
    authenticated connections do not have access. In "secure mode" any non
    authenticated connections should get a "404" type message on
    everything.
    Knowing "something is there" is a security risk in many uses cases. So if
    you don't have access you don't even see it. Baking "that" into Kafka
    along with some interface for entitlement (access management) systems
    (pretty standard) is all that I think needs to be done to the core project.
    I want to tackle item later in the year after summer after the other three
    are complete.

    I look forward to thoughts on this and anyone else interested in
    working
    with us on these items.

    [0]
    http://www.emc.com/emc-plus/rsa-labs/standards-initiatives/what-is-a-digital-envelope.htm
    http://www.emc.com/emc-plus/rsa-labs/standards-initiatives/pkcs-7-cryptographic-message-syntax-standar.htm
    [4] http://en.wikipedia.org/wiki/XML_Signature
    [5] http://en.wikipedia.org/wiki/PKCS_12

    /*******************************************
    Joe Stein
    Founder, Principal Consultant
    Big Data Open Source Security LLC
    http://www.stealth.ly
    Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop>
    ********************************************/
  • Todd Palino at Jun 5, 2014 at 9:06 pm
    No, at-rest encryption is definitely important. When you start talking
    about data that is used for financial reporting, restricting access to it
    (both modification and visibility) is a critical component.

    -Todd

    On 6/5/14, 2:01 PM, "Jay Kreps" wrote:

    Hey Joe,

    I don't really understand the sections you added to the wiki. Can you
    clarify them?

    Is non-repudiation what SASL would call integrity checks? If so don't SSL
    and and many of the SASL schemes already support this as well as
    on-the-wire encryption?

    Or are you proposing an on-disk encryption scheme? Is this actually
    needed?
    Isn't a on-the-wire encryption when combined with mutual authentication
    and
    permissions sufficient for most uses?

    On-disk encryption seems unnecessary because if an attacker can get root
    on
    the kafka boxes it can potentially modify Kafka to do anything he or she
    wants with data. So this seems to break any security model.

    I understand the problem of a large organization not really having a
    trusted network and wanting to secure data transfer and limit and audit
    data access. The uses for these other things I don't totally understand.

    Also it would be worth understanding the state of other messaging and
    storage systems (Hadoop, dbs, etc). What features do they support. I think
    there is a sense in which you don't have to run faster than the bear, but
    only faster then your friends. :-)

    -Jay

    On Wed, Jun 4, 2014 at 5:57 PM, Joe Stein wrote:

    I like the idea of working on the spec and prioritizing. I will update
    the
    wiki.

    - Joestein

    On Wed, Jun 4, 2014 at 1:11 PM, Jay Kreps wrote:

    Hey Joe,

    Thanks for kicking this discussion off! I totally agree that for something
    that acts as a central message broker security is critical feature. I think
    a number of people have been interested in this topic and several people
    have put effort into special purpose security efforts.

    Since most the LinkedIn folks are working on the consumer right now I think
    this would be a great project for any other interested people to take on.
    There are some challenges in doing these things distributed but it can also
    be a lot of fun.

    I think a good first step would be to get a written plan we can all agree
    on for how things should work. Then we can break things down into chunks
    that can be done independently while still aiming at a good end state.

    I had tried to write up some notes that summarized at least the
    thoughts
    I
    had had on security:
    https://cwiki.apache.org/confluence/display/KAFKA/Security

    What do you think of that?

    One assumption I had (which may be incorrect) is that although we want all
    the things in your list, the two most pressing would be authentication and
    authorization, and that was all that write up covered. You have more
    experience in this domain, so I wonder how you would prioritize?

    Those notes are really sketchy, so I think the first goal I would have
    would be to get to a real spec we can all agree on and discuss. A lot of
    the security stuff has a high human interaction element and needs to work
    in pretty different domains and different companies so getting this
    kind
    of
    review is important.

    -Jay


    On Tue, Jun 3, 2014 at 12:57 PM, Joe Stein <joe.stein@stealth.ly>
    wrote:
    Hi,I wanted to re-ignite the discussion around Apache Kafka
    Security.
    This
    is a huge bottleneck (non-starter in some cases) for a lot of
    organizations
    (due to regulatory, compliance and other requirements). Below are my
    suggestions for specific changes in Kafka to accommodate security
    requirements. This comes from what folks are doing "in the wild" to
    workaround and implement security with Kafka as it is today and also what I
    have discovered from organizations about their blockers. It also
    picks
    up
    from the wiki (which I should have time to update later in the week
    based
    on the below and feedback from the thread).

    1) Transport Layer Security (i.e. SSL)

    This also includes client authentication in addition to in-transit security
    layer. This work has been picked up here
    https://issues.apache.org/jira/browse/KAFKA-1477 and do appreciate
    any
    thoughts, comments, feedback, tomatoes, whatever for this patch. It
    is a
    pickup from the fork of the work first done here
    https://github.com/relango/kafka/tree/kafka_security.

    2) Data encryption at rest.

    This is very important and something that can be facilitated within
    the
    wire protocol. It requires an additional map data structure for the
    "encrypted [data encryption key]". With this map (either in your
    object
    or
    in the wire protocol) you can store the dynamically generated
    symmetric
    key
    (for each message) and then encrypt the data using that dynamically
    generated key. You then encrypt the encryption key using each
    public
    key
    for whom is expected to be able to decrypt the encryption key to
    then
    decrypt the message. For each public key encrypted symmetric key
    (which
    is
    now the "encrypted [data encryption key]" along with which public
    key
    it
    was encrypted with for (so a map of [publicKey] =
    encryptedDataEncryptionKey) as a chain. Other patterns can be
    implemented
    but this is a pretty standard digital enveloping [0] pattern with
    only
    1
    field added. Other patterns should be able to use that field to-do
    their
    implementation too.

    3) Non-repudiation and long term non-repudiation.

    Non-repudiation is proving data hasn't changed. This is often (if
    not
    always) done with x509 public certificates (chained to a certificate
    authority).

    Long term non-repudiation is what happens when the certificates of
    the
    certificate authority are expired (or revoked) and everything ever
    signed
    (ever) with that certificate's public key then becomes "no longer provable
    as ever being authentic". That is where RFC3126 [1] and RFC3161 [2]
    come
    in (or worm drives [hardware], etc).

    For either (or both) of these it is an operation of the encryptor to
    sign/hash the data (with or without third party trusted timestap of
    the
    signing event) and encrypt that with their own private key and
    distribute
    the results (before and after encrypting if required) along with
    their
    public key. This structure is a bit more complex but feasible, it
    is a
    map
    of digital signature formats and the chain of dig sig attestations.
    The
    map's key being the method (i.e. CRC32, PKCS7 [3], XmlDigSig [4])
    and
    then
    a list of map where that key is "purpose" of signature (what your attesting
    too). As a sibling field to the list another field for "the
    attester"
    as
    bytes (e.g. their PKCS12 [5] for the map of PKCS7 signatures).

    4) Authorization

    We should have a policy of "404" for data, topics, partitions (etc)
    if
    authenticated connections do not have access. In "secure mode" any
    non
    authenticated connections should get a "404" type message on
    everything.
    Knowing "something is there" is a security risk in many uses cases.
    So
    if
    you don't have access you don't even see it. Baking "that" into
    Kafka
    along with some interface for entitlement (access management)
    systems
    (pretty standard) is all that I think needs to be done to the core project.
    I want to tackle item later in the year after summer after the
    other
    three
    are complete.

    I look forward to thoughts on this and anyone else interested in
    working
    with us on these items.

    [0]
    http://www.emc.com/emc-plus/rsa-labs/standards-initiatives/what-is-a-digi
    tal-envelope.htm
    http://www.emc.com/emc-plus/rsa-labs/standards-initiatives/pkcs-7-cryptog
    raphic-message-syntax-standar.htm
    [4] http://en.wikipedia.org/wiki/XML_Signature
    [5] http://en.wikipedia.org/wiki/PKCS_12

    /*******************************************
    Joe Stein
    Founder, Principal Consultant
    Big Data Open Source Security LLC
    http://www.stealth.ly
    Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop>
    ********************************************/
  • Jay Kreps at Jun 5, 2014 at 9:19 pm
    Hey Todd,

    Can you elaborate on this? Certainly restricting access to and modification
    of data is important. But this doesn't imply storing the data encrypted.
    Are we assuming the attacker can (1) get on the network, (2) get on the
    kafka server as a non-root and non-kafka user or (3) get root on the Kafka
    server? If we assume (3) then it seems we are in a pretty bad state as
    almost any facility Kafka provides can be subverted by the root user just
    changing the Kafka code to not enforce that facility. Which of these levels
    of access are we assuming?

    Also which things actually need to be done inside Kafka and which can be
    done externally? Nothing prevents users from encrypting data they put into
    Kafka today, it is just that Kafka doesn't do this for you. But is there a
    reason you want Kafka to do this?

    The reason I am pushing on these things a bit is because I want to make
    sure we don't end up with a set of requirements so broad we can never
    really get them implemented...

    -Jay



    On Thu, Jun 5, 2014 at 2:05 PM, Todd Palino wrote:

    No, at-rest encryption is definitely important. When you start talking
    about data that is used for financial reporting, restricting access to it
    (both modification and visibility) is a critical component.

    -Todd

    On 6/5/14, 2:01 PM, "Jay Kreps" wrote:

    Hey Joe,

    I don't really understand the sections you added to the wiki. Can you
    clarify them?

    Is non-repudiation what SASL would call integrity checks? If so don't SSL
    and and many of the SASL schemes already support this as well as
    on-the-wire encryption?

    Or are you proposing an on-disk encryption scheme? Is this actually
    needed?
    Isn't a on-the-wire encryption when combined with mutual authentication
    and
    permissions sufficient for most uses?

    On-disk encryption seems unnecessary because if an attacker can get root
    on
    the kafka boxes it can potentially modify Kafka to do anything he or she
    wants with data. So this seems to break any security model.

    I understand the problem of a large organization not really having a
    trusted network and wanting to secure data transfer and limit and audit
    data access. The uses for these other things I don't totally understand.

    Also it would be worth understanding the state of other messaging and
    storage systems (Hadoop, dbs, etc). What features do they support. I think
    there is a sense in which you don't have to run faster than the bear, but
    only faster then your friends. :-)

    -Jay

    On Wed, Jun 4, 2014 at 5:57 PM, Joe Stein wrote:

    I like the idea of working on the spec and prioritizing. I will update
    the
    wiki.

    - Joestein

    On Wed, Jun 4, 2014 at 1:11 PM, Jay Kreps wrote:

    Hey Joe,

    Thanks for kicking this discussion off! I totally agree that for something
    that acts as a central message broker security is critical feature. I think
    a number of people have been interested in this topic and several people
    have put effort into special purpose security efforts.

    Since most the LinkedIn folks are working on the consumer right now I think
    this would be a great project for any other interested people to take on.
    There are some challenges in doing these things distributed but it can also
    be a lot of fun.

    I think a good first step would be to get a written plan we can all agree
    on for how things should work. Then we can break things down into chunks
    that can be done independently while still aiming at a good end state.

    I had tried to write up some notes that summarized at least the
    thoughts
    I
    had had on security:
    https://cwiki.apache.org/confluence/display/KAFKA/Security

    What do you think of that?

    One assumption I had (which may be incorrect) is that although we want all
    the things in your list, the two most pressing would be authentication and
    authorization, and that was all that write up covered. You have more
    experience in this domain, so I wonder how you would prioritize?

    Those notes are really sketchy, so I think the first goal I would have
    would be to get to a real spec we can all agree on and discuss. A lot of
    the security stuff has a high human interaction element and needs to work
    in pretty different domains and different companies so getting this
    kind
    of
    review is important.

    -Jay


    On Tue, Jun 3, 2014 at 12:57 PM, Joe Stein <joe.stein@stealth.ly>
    wrote:
    Hi,I wanted to re-ignite the discussion around Apache Kafka
    Security.
    This
    is a huge bottleneck (non-starter in some cases) for a lot of
    organizations
    (due to regulatory, compliance and other requirements). Below are my
    suggestions for specific changes in Kafka to accommodate security
    requirements. This comes from what folks are doing "in the wild" to
    workaround and implement security with Kafka as it is today and also what I
    have discovered from organizations about their blockers. It also
    picks
    up
    from the wiki (which I should have time to update later in the week
    based
    on the below and feedback from the thread).

    1) Transport Layer Security (i.e. SSL)

    This also includes client authentication in addition to in-transit security
    layer. This work has been picked up here
    https://issues.apache.org/jira/browse/KAFKA-1477 and do appreciate
    any
    thoughts, comments, feedback, tomatoes, whatever for this patch. It
    is a
    pickup from the fork of the work first done here
    https://github.com/relango/kafka/tree/kafka_security.

    2) Data encryption at rest.

    This is very important and something that can be facilitated within
    the
    wire protocol. It requires an additional map data structure for the
    "encrypted [data encryption key]". With this map (either in your
    object
    or
    in the wire protocol) you can store the dynamically generated
    symmetric
    key
    (for each message) and then encrypt the data using that dynamically
    generated key. You then encrypt the encryption key using each
    public
    key
    for whom is expected to be able to decrypt the encryption key to
    then
    decrypt the message. For each public key encrypted symmetric key
    (which
    is
    now the "encrypted [data encryption key]" along with which public
    key
    it
    was encrypted with for (so a map of [publicKey] =
    encryptedDataEncryptionKey) as a chain. Other patterns can be
    implemented
    but this is a pretty standard digital enveloping [0] pattern with
    only
    1
    field added. Other patterns should be able to use that field to-do
    their
    implementation too.

    3) Non-repudiation and long term non-repudiation.

    Non-repudiation is proving data hasn't changed. This is often (if
    not
    always) done with x509 public certificates (chained to a certificate
    authority).

    Long term non-repudiation is what happens when the certificates of
    the
    certificate authority are expired (or revoked) and everything ever
    signed
    (ever) with that certificate's public key then becomes "no longer provable
    as ever being authentic". That is where RFC3126 [1] and RFC3161 [2]
    come
    in (or worm drives [hardware], etc).

    For either (or both) of these it is an operation of the encryptor to
    sign/hash the data (with or without third party trusted timestap of
    the
    signing event) and encrypt that with their own private key and
    distribute
    the results (before and after encrypting if required) along with
    their
    public key. This structure is a bit more complex but feasible, it
    is a
    map
    of digital signature formats and the chain of dig sig attestations.
    The
    map's key being the method (i.e. CRC32, PKCS7 [3], XmlDigSig [4])
    and
    then
    a list of map where that key is "purpose" of signature (what your attesting
    too). As a sibling field to the list another field for "the
    attester"
    as
    bytes (e.g. their PKCS12 [5] for the map of PKCS7 signatures).

    4) Authorization

    We should have a policy of "404" for data, topics, partitions (etc)
    if
    authenticated connections do not have access. In "secure mode" any
    non
    authenticated connections should get a "404" type message on
    everything.
    Knowing "something is there" is a security risk in many uses cases.
    So
    if
    you don't have access you don't even see it. Baking "that" into
    Kafka
    along with some interface for entitlement (access management)
    systems
    (pretty standard) is all that I think needs to be done to the core project.
    I want to tackle item later in the year after summer after the
    other
    three
    are complete.

    I look forward to thoughts on this and anyone else interested in
    working
    with us on these items.

    [0]
    http://www.emc.com/emc-plus/rsa-labs/standards-initiatives/what-is-a-digi
    http://www.emc.com/emc-plus/rsa-labs/standards-initiatives/pkcs-7-cryptog
    raphic-message-syntax-standar.htm
    [4] http://en.wikipedia.org/wiki/XML_Signature
    [5] http://en.wikipedia.org/wiki/PKCS_12

    /*******************************************
    Joe Stein
    Founder, Principal Consultant
    Big Data Open Source Security LLC
    http://www.stealth.ly
    Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop>
    ********************************************/
  • Todd Palino at Jun 5, 2014 at 9:47 pm
    My concern is specifically around the rules for SOX compliance, or rules
    around PII, PCI, or HIPAA compliance. The audits get very complication,
    but my understanding is that the general rule is that sensitive data
    should be encrypted at rest and only decrypted when needed. And we don¹t
    just need to be concerned about a malicious user. Consider a ³typical²
    technology environment where many people have administrative access to
    systems. This is the one where you need to not have the data visible to
    anyone unless they have a specific use for it, which means having it
    encrypted. In almost any audit situation, you need to be able to show a
    trail of exactly who modified the data, and exactly who viewed the data.

    Now, I do agree that not everything has to be done within Kafka, and the
    producers and consumers can coordinate their own encryption. But I think
    it¹s useful to have the concept of an envelope for a message within Kafka.
    This can be used to hold all sorts of useful information, such as hashes
    of the encryption keys that were used to encrypt a message, or the
    signature of the message itself (so that you can have both confidentiality
    and integrity). It can also be used to hold things like the time a message
    was received into your infrastructure, or the specific Kafka cluster it
    was stored in. A special consumer and producer, such as the mirror maker,
    would be able to preserve this envelope across clusters.

    -Todd

    On 6/5/14, 2:18 PM, "Jay Kreps" wrote:

    Hey Todd,

    Can you elaborate on this? Certainly restricting access to and
    modification
    of data is important. But this doesn't imply storing the data encrypted.
    Are we assuming the attacker can (1) get on the network, (2) get on the
    kafka server as a non-root and non-kafka user or (3) get root on the Kafka
    server? If we assume (3) then it seems we are in a pretty bad state as
    almost any facility Kafka provides can be subverted by the root user just
    changing the Kafka code to not enforce that facility. Which of these
    levels
    of access are we assuming?

    Also which things actually need to be done inside Kafka and which can be
    done externally? Nothing prevents users from encrypting data they put into
    Kafka today, it is just that Kafka doesn't do this for you. But is there a
    reason you want Kafka to do this?

    The reason I am pushing on these things a bit is because I want to make
    sure we don't end up with a set of requirements so broad we can never
    really get them implemented...

    -Jay



    On Thu, Jun 5, 2014 at 2:05 PM, Todd Palino wrote:

    No, at-rest encryption is definitely important. When you start talking
    about data that is used for financial reporting, restricting access to
    it
    (both modification and visibility) is a critical component.

    -Todd

    On 6/5/14, 2:01 PM, "Jay Kreps" wrote:

    Hey Joe,

    I don't really understand the sections you added to the wiki. Can you
    clarify them?

    Is non-repudiation what SASL would call integrity checks? If so don't SSL
    and and many of the SASL schemes already support this as well as
    on-the-wire encryption?

    Or are you proposing an on-disk encryption scheme? Is this actually
    needed?
    Isn't a on-the-wire encryption when combined with mutual authentication
    and
    permissions sufficient for most uses?

    On-disk encryption seems unnecessary because if an attacker can get root
    on
    the kafka boxes it can potentially modify Kafka to do anything he or she
    wants with data. So this seems to break any security model.

    I understand the problem of a large organization not really having a
    trusted network and wanting to secure data transfer and limit and audit
    data access. The uses for these other things I don't totally
    understand.
    Also it would be worth understanding the state of other messaging and
    storage systems (Hadoop, dbs, etc). What features do they support. I think
    there is a sense in which you don't have to run faster than the bear, but
    only faster then your friends. :-)

    -Jay

    On Wed, Jun 4, 2014 at 5:57 PM, Joe Stein wrote:

    I like the idea of working on the spec and prioritizing. I will
    update
    the
    wiki.

    - Joestein


    On Wed, Jun 4, 2014 at 1:11 PM, Jay Kreps <jay.kreps@gmail.com>
    wrote:
    Hey Joe,

    Thanks for kicking this discussion off! I totally agree that for something
    that acts as a central message broker security is critical
    feature. I
    think
    a number of people have been interested in this topic and several people
    have put effort into special purpose security efforts.

    Since most the LinkedIn folks are working on the consumer right
    now I
    think
    this would be a great project for any other interested people to
    take
    on.
    There are some challenges in doing these things distributed but it
    can
    also
    be a lot of fun.

    I think a good first step would be to get a written plan we can all agree
    on for how things should work. Then we can break things down into chunks
    that can be done independently while still aiming at a good end
    state.
    I had tried to write up some notes that summarized at least the
    thoughts
    I
    had had on security:
    https://cwiki.apache.org/confluence/display/KAFKA/Security

    What do you think of that?

    One assumption I had (which may be incorrect) is that although we
    want
    all
    the things in your list, the two most pressing would be
    authentication
    and
    authorization, and that was all that write up covered. You have
    more
    experience in this domain, so I wonder how you would prioritize?

    Those notes are really sketchy, so I think the first goal I would
    have
    would be to get to a real spec we can all agree on and discuss. A
    lot
    of
    the security stuff has a high human interaction element and needs
    to
    work
    in pretty different domains and different companies so getting this
    kind
    of
    review is important.

    -Jay


    On Tue, Jun 3, 2014 at 12:57 PM, Joe Stein <joe.stein@stealth.ly>
    wrote:
    Hi,I wanted to re-ignite the discussion around Apache Kafka
    Security.
    This
    is a huge bottleneck (non-starter in some cases) for a lot of
    organizations
    (due to regulatory, compliance and other requirements). Below
    are my
    suggestions for specific changes in Kafka to accommodate security
    requirements. This comes from what folks are doing "in the
    wild" to
    workaround and implement security with Kafka as it is today and
    also
    what I
    have discovered from organizations about their blockers. It also
    picks
    up
    from the wiki (which I should have time to update later in the
    week
    based
    on the below and feedback from the thread).

    1) Transport Layer Security (i.e. SSL)

    This also includes client authentication in addition to
    in-transit
    security
    layer. This work has been picked up here
    https://issues.apache.org/jira/browse/KAFKA-1477 and do
    appreciate
    any
    thoughts, comments, feedback, tomatoes, whatever for this patch.
    It
    is a
    pickup from the fork of the work first done here
    https://github.com/relango/kafka/tree/kafka_security.

    2) Data encryption at rest.

    This is very important and something that can be facilitated
    within
    the
    wire protocol. It requires an additional map data structure for
    the
    "encrypted [data encryption key]". With this map (either in your
    object
    or
    in the wire protocol) you can store the dynamically generated
    symmetric
    key
    (for each message) and then encrypt the data using that
    dynamically
    generated key. You then encrypt the encryption key using each
    public
    key
    for whom is expected to be able to decrypt the encryption key to
    then
    decrypt the message. For each public key encrypted symmetric key
    (which
    is
    now the "encrypted [data encryption key]" along with which public
    key
    it
    was encrypted with for (so a map of [publicKey] =
    encryptedDataEncryptionKey) as a chain. Other patterns can be
    implemented
    but this is a pretty standard digital enveloping [0] pattern with
    only
    1
    field added. Other patterns should be able to use that field
    to-do
    their
    implementation too.

    3) Non-repudiation and long term non-repudiation.

    Non-repudiation is proving data hasn't changed. This is often
    (if
    not
    always) done with x509 public certificates (chained to a
    certificate
    authority).

    Long term non-repudiation is what happens when the certificates
    of
    the
    certificate authority are expired (or revoked) and everything
    ever
    signed
    (ever) with that certificate's public key then becomes "no longer provable
    as ever being authentic". That is where RFC3126 [1] and RFC3161
    [2]
    come
    in (or worm drives [hardware], etc).

    For either (or both) of these it is an operation of the
    encryptor to
    sign/hash the data (with or without third party trusted timestap
    of
    the
    signing event) and encrypt that with their own private key and
    distribute
    the results (before and after encrypting if required) along with
    their
    public key. This structure is a bit more complex but feasible, it
    is a
    map
    of digital signature formats and the chain of dig sig
    attestations.
    The
    map's key being the method (i.e. CRC32, PKCS7 [3], XmlDigSig [4])
    and
    then
    a list of map where that key is "purpose" of signature (what your attesting
    too). As a sibling field to the list another field for "the
    attester"
    as
    bytes (e.g. their PKCS12 [5] for the map of PKCS7 signatures).

    4) Authorization

    We should have a policy of "404" for data, topics, partitions
    (etc)
    if
    authenticated connections do not have access. In "secure mode"
    any
    non
    authenticated connections should get a "404" type message on
    everything.
    Knowing "something is there" is a security risk in many uses
    cases.
    So
    if
    you don't have access you don't even see it. Baking "that" into
    Kafka
    along with some interface for entitlement (access management)
    systems
    (pretty standard) is all that I think needs to be done to the
    core
    project.
    I want to tackle item later in the year after summer after the
    other
    three
    are complete.

    I look forward to thoughts on this and anyone else interested in
    working
    with us on these items.

    [0]
    http://www.emc.com/emc-plus/rsa-labs/standards-initiatives/what-is-a-digi
    http://www.emc.com/emc-plus/rsa-labs/standards-initiatives/pkcs-7-cryptog
    raphic-message-syntax-standar.htm
    [4] http://en.wikipedia.org/wiki/XML_Signature
    [5] http://en.wikipedia.org/wiki/PKCS_12

    /*******************************************
    Joe Stein
    Founder, Principal Consultant
    Big Data Open Source Security LLC
    http://www.stealth.ly
    Twitter: @allthingshadoop
    <http://www.twitter.com/allthingshadoop>
    ********************************************/
  • Rob Withers at Jun 6, 2014 at 5:44 pm
    This is quite interesting to me and it is an excelent opportunity to
    promote a slightly different security scheme. Object-capabilities are
    perfect for online security and would use ACL style authentication to
    gain capabilities filtered to those allowed resources for allow
    actions (READ/WRITE/DELETE/LIST/SCAN). Erights.org has the
    quitenscential (??) object capabilities model and capnproto is
    impleemting this for C++. I have a java implementation at http://github.com/pauwau/pauwau
       but the master is broken. 0.2 works, basically. B asically a TLS
    connection with no certificate server, it is peer to peer. It has
    some advanced features, but the lining of capabilities with
    authorization so that you can only invoke correct services is extended
    to the secure user.

    Regarding non-repudiation, on disk, why not prepend a CRC?

    Regarding on-disk encryption, multiple users/groups may need to
    access, with different capabilities. Sounds like zookeeper needs to
    store a cert for each class of access so that a group member can
    access the decrypted data from disk. Use cert-based async
    decryption. The only isue is storing the private key in zookeeper.
    Perhaps some hash magic could be used.

    Thanks for kafka,
    Rob
    On Jun 5, 2014, at 3:01 PM, Jay Kreps wrote:

    Hey Joe,

    I don't really understand the sections you added to the wiki. Can you
    clarify them?

    Is non-repudiation what SASL would call integrity checks? If so
    don't SSL
    and and many of the SASL schemes already support this as well as
    on-the-wire encryption?

    Or are you proposing an on-disk encryption scheme? Is this actually
    needed?
    Isn't a on-the-wire encryption when combined with mutual
    authentication and
    permissions sufficient for most uses?

    On-disk encryption seems unnecessary because if an attacker can get
    root on
    the kafka boxes it can potentially modify Kafka to do anything he or
    she
    wants with data. So this seems to break any security model.

    I understand the problem of a large organization not really having a
    trusted network and wanting to secure data transfer and limit and
    audit
    data access. The uses for these other things I don't totally
    understand.

    Also it would be worth understanding the state of other messaging and
    storage systems (Hadoop, dbs, etc). What features do they support. I
    think
    there is a sense in which you don't have to run faster than the
    bear, but
    only faster then your friends. :-)

    -Jay

    On Wed, Jun 4, 2014 at 5:57 PM, Joe Stein wrote:

    I like the idea of working on the spec and prioritizing. I will
    update the
    wiki.

    - Joestein


    On Wed, Jun 4, 2014 at 1:11 PM, Jay Kreps <jay.kreps@gmail.com>
    wrote:
    Hey Joe,

    Thanks for kicking this discussion off! I totally agree that for something
    that acts as a central message broker security is critical
    feature. I think
    a number of people have been interested in this topic and several
    people
    have put effort into special purpose security efforts.

    Since most the LinkedIn folks are working on the consumer right
    now I think
    this would be a great project for any other interested people to
    take on.
    There are some challenges in doing these things distributed but it
    can also
    be a lot of fun.

    I think a good first step would be to get a written plan we can
    all agree
    on for how things should work. Then we can break things down into
    chunks
    that can be done independently while still aiming at a good end
    state.

    I had tried to write up some notes that summarized at least the
    thoughts I
    had had on security:
    https://cwiki.apache.org/confluence/display/KAFKA/Security

    What do you think of that?

    One assumption I had (which may be incorrect) is that although we
    want all
    the things in your list, the two most pressing would be
    authentication and
    authorization, and that was all that write up covered. You have more
    experience in this domain, so I wonder how you would prioritize?

    Those notes are really sketchy, so I think the first goal I would
    have
    would be to get to a real spec we can all agree on and discuss. A
    lot of
    the security stuff has a high human interaction element and needs
    to work
    in pretty different domains and different companies so getting
    this kind of
    review is important.

    -Jay


    On Tue, Jun 3, 2014 at 12:57 PM, Joe Stein <joe.stein@stealth.ly>
    wrote:
    Hi,I wanted to re-ignite the discussion around Apache Kafka
    Security. This
    is a huge bottleneck (non-starter in some cases) for a lot of
    organizations
    (due to regulatory, compliance and other requirements). Below are
    my
    suggestions for specific changes in Kafka to accommodate security
    requirements. This comes from what folks are doing "in the wild"
    to
    workaround and implement security with Kafka as it is today and
    also what I
    have discovered from organizations about their blockers. It also
    picks
    up
    from the wiki (which I should have time to update later in the week
    based
    on the below and feedback from the thread).

    1) Transport Layer Security (i.e. SSL)

    This also includes client authentication in addition to in-transit security
    layer. This work has been picked up here
    https://issues.apache.org/jira/browse/KAFKA-1477 and do
    appreciate any
    thoughts, comments, feedback, tomatoes, whatever for this patch.
    It
    is a
    pickup from the fork of the work first done here
    https://github.com/relango/kafka/tree/kafka_security.

    2) Data encryption at rest.

    This is very important and something that can be facilitated
    within the
    wire protocol. It requires an additional map data structure for the
    "encrypted [data encryption key]". With this map (either in your
    object or
    in the wire protocol) you can store the dynamically generated
    symmetric key
    (for each message) and then encrypt the data using that dynamically
    generated key. You then encrypt the encryption key using each
    public
    key
    for whom is expected to be able to decrypt the encryption key to
    then
    decrypt the message. For each public key encrypted symmetric key
    (which
    is
    now the "encrypted [data encryption key]" along with which public
    key
    it
    was encrypted with for (so a map of [publicKey] =
    encryptedDataEncryptionKey) as a chain. Other patterns can be
    implemented
    but this is a pretty standard digital enveloping [0] pattern with
    only
    1
    field added. Other patterns should be able to use that field to-do
    their
    implementation too.

    3) Non-repudiation and long term non-repudiation.

    Non-repudiation is proving data hasn't changed. This is often
    (if not
    always) done with x509 public certificates (chained to a
    certificate
    authority).

    Long term non-repudiation is what happens when the certificates
    of the
    certificate authority are expired (or revoked) and everything ever
    signed
    (ever) with that certificate's public key then becomes "no longer provable
    as ever being authentic". That is where RFC3126 [1] and RFC3161
    [2]
    come
    in (or worm drives [hardware], etc).

    For either (or both) of these it is an operation of the encryptor
    to
    sign/hash the data (with or without third party trusted timestap
    of the
    signing event) and encrypt that with their own private key and
    distribute
    the results (before and after encrypting if required) along with
    their
    public key. This structure is a bit more complex but feasible, it
    is a map
    of digital signature formats and the chain of dig sig attestations.
    The
    map's key being the method (i.e. CRC32, PKCS7 [3], XmlDigSig [4])
    and then
    a list of map where that key is "purpose" of signature (what your attesting
    too). As a sibling field to the list another field for "the
    attester"
    as
    bytes (e.g. their PKCS12 [5] for the map of PKCS7 signatures).

    4) Authorization

    We should have a policy of "404" for data, topics, partitions
    (etc) if
    authenticated connections do not have access. In "secure mode"
    any non
    authenticated connections should get a "404" type message on
    everything.
    Knowing "something is there" is a security risk in many uses
    cases. So if
    you don't have access you don't even see it. Baking "that" into
    Kafka
    along with some interface for entitlement (access management)
    systems
    (pretty standard) is all that I think needs to be done to the core project.
    I want to tackle item later in the year after summer after the
    other three
    are complete.

    I look forward to thoughts on this and anyone else interested in
    working
    with us on these items.

    [0]
    http://www.emc.com/emc-plus/rsa-labs/standards-initiatives/what-is-a-digital-envelope.htm
    http://www.emc.com/emc-plus/rsa-labs/standards-initiatives/pkcs-7-cryptographic-message-syntax-standar.htm
    [4] http://en.wikipedia.org/wiki/XML_Signature
    [5] http://en.wikipedia.org/wiki/PKCS_12

    /*******************************************
    Joe Stein
    Founder, Principal Consultant
    Big Data Open Source Security LLC
    http://www.stealth.ly
    Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop>
    ********************************************/
  • Rob Withers at Jun 6, 2014 at 5:49 pm
    On consideration, if we have 3 different access groups (1 for
    production WRITE and 2 consumers) they all need to decode the same
    encryption and so all need the same public/private key....certs won't
    work, unless you write a CertAuthority to build multiple certs with
    the same keys. Better seems to not use certs and wrap the encryption
    specification with an ACL capabilities for each group of access.
    On Jun 6, 2014, at 11:43 AM, Rob Withers wrote:

    This is quite interesting to me and it is an excelent opportunity to
    promote a slightly different security scheme. Object-capabilities
    are perfect for online security and would use ACL style
    authentication to gain capabilities filtered to those allowed
    resources for allow actions (READ/WRITE/DELETE/LIST/SCAN).
    Erights.org has the quitenscential (??) object capabilities model
    and capnproto is impleemting this for C++. I have a java
    implementation at http://github.com/pauwau/pauwau but the master is
    broken. 0.2 works, basically. B asically a TLS connection with no
    certificate server, it is peer to peer. It has some advanced
    features, but the lining of capabilities with authorization so that
    you can only invoke correct services is extended to the secure user.

    Regarding non-repudiation, on disk, why not prepend a CRC?

    Regarding on-disk encryption, multiple users/groups may need to
    access, with different capabilities. Sounds like zookeeper needs to
    store a cert for each class of access so that a group member can
    access the decrypted data from disk. Use cert-based async
    decryption. The only isue is storing the private key in zookeeper.
    Perhaps some hash magic could be used.

    Thanks for kafka,
    Rob
    On Jun 5, 2014, at 3:01 PM, Jay Kreps wrote:

    Hey Joe,

    I don't really understand the sections you added to the wiki. Can you
    clarify them?

    Is non-repudiation what SASL would call integrity checks? If so
    don't SSL
    and and many of the SASL schemes already support this as well as
    on-the-wire encryption?

    Or are you proposing an on-disk encryption scheme? Is this actually
    needed?
    Isn't a on-the-wire encryption when combined with mutual
    authentication and
    permissions sufficient for most uses?

    On-disk encryption seems unnecessary because if an attacker can get
    root on
    the kafka boxes it can potentially modify Kafka to do anything he
    or she
    wants with data. So this seems to break any security model.

    I understand the problem of a large organization not really having a
    trusted network and wanting to secure data transfer and limit and
    audit
    data access. The uses for these other things I don't totally
    understand.

    Also it would be worth understanding the state of other messaging and
    storage systems (Hadoop, dbs, etc). What features do they support.
    I think
    there is a sense in which you don't have to run faster than the
    bear, but
    only faster then your friends. :-)

    -Jay


    On Wed, Jun 4, 2014 at 5:57 PM, Joe Stein <joe.stein@stealth.ly>
    wrote:
    I like the idea of working on the spec and prioritizing. I will
    update the
    wiki.

    - Joestein


    On Wed, Jun 4, 2014 at 1:11 PM, Jay Kreps <jay.kreps@gmail.com>
    wrote:
    Hey Joe,

    Thanks for kicking this discussion off! I totally agree that for something
    that acts as a central message broker security is critical
    feature. I think
    a number of people have been interested in this topic and several
    people
    have put effort into special purpose security efforts.

    Since most the LinkedIn folks are working on the consumer right
    now I think
    this would be a great project for any other interested people to
    take on.
    There are some challenges in doing these things distributed but
    it can also
    be a lot of fun.

    I think a good first step would be to get a written plan we can
    all agree
    on for how things should work. Then we can break things down into
    chunks
    that can be done independently while still aiming at a good end
    state.

    I had tried to write up some notes that summarized at least the
    thoughts I
    had had on security:
    https://cwiki.apache.org/confluence/display/KAFKA/Security

    What do you think of that?

    One assumption I had (which may be incorrect) is that although we
    want all
    the things in your list, the two most pressing would be
    authentication and
    authorization, and that was all that write up covered. You have
    more
    experience in this domain, so I wonder how you would prioritize?

    Those notes are really sketchy, so I think the first goal I would
    have
    would be to get to a real spec we can all agree on and discuss. A
    lot of
    the security stuff has a high human interaction element and needs
    to work
    in pretty different domains and different companies so getting
    this kind of
    review is important.

    -Jay


    On Tue, Jun 3, 2014 at 12:57 PM, Joe Stein <joe.stein@stealth.ly>
    wrote:
    Hi,I wanted to re-ignite the discussion around Apache Kafka
    Security. This
    is a huge bottleneck (non-starter in some cases) for a lot of
    organizations
    (due to regulatory, compliance and other requirements). Below
    are my
    suggestions for specific changes in Kafka to accommodate security
    requirements. This comes from what folks are doing "in the
    wild" to
    workaround and implement security with Kafka as it is today and
    also what I
    have discovered from organizations about their blockers. It also
    picks
    up
    from the wiki (which I should have time to update later in the
    week
    based
    on the below and feedback from the thread).

    1) Transport Layer Security (i.e. SSL)

    This also includes client authentication in addition to in-transit security
    layer. This work has been picked up here
    https://issues.apache.org/jira/browse/KAFKA-1477 and do
    appreciate any
    thoughts, comments, feedback, tomatoes, whatever for this
    patch. It
    is a
    pickup from the fork of the work first done here
    https://github.com/relango/kafka/tree/kafka_security.

    2) Data encryption at rest.

    This is very important and something that can be facilitated
    within the
    wire protocol. It requires an additional map data structure for
    the
    "encrypted [data encryption key]". With this map (either in your
    object or
    in the wire protocol) you can store the dynamically generated
    symmetric key
    (for each message) and then encrypt the data using that
    dynamically
    generated key. You then encrypt the encryption key using each
    public
    key
    for whom is expected to be able to decrypt the encryption key to
    then
    decrypt the message. For each public key encrypted symmetric key
    (which
    is
    now the "encrypted [data encryption key]" along with which
    public key
    it
    was encrypted with for (so a map of [publicKey] =
    encryptedDataEncryptionKey) as a chain. Other patterns can be
    implemented
    but this is a pretty standard digital enveloping [0] pattern
    with only
    1
    field added. Other patterns should be able to use that field to-do
    their
    implementation too.

    3) Non-repudiation and long term non-repudiation.

    Non-repudiation is proving data hasn't changed. This is often
    (if not
    always) done with x509 public certificates (chained to a
    certificate
    authority).

    Long term non-repudiation is what happens when the certificates
    of the
    certificate authority are expired (or revoked) and everything ever
    signed
    (ever) with that certificate's public key then becomes "no longer provable
    as ever being authentic". That is where RFC3126 [1] and RFC3161
    [2]
    come
    in (or worm drives [hardware], etc).

    For either (or both) of these it is an operation of the
    encryptor to
    sign/hash the data (with or without third party trusted timestap
    of the
    signing event) and encrypt that with their own private key and
    distribute
    the results (before and after encrypting if required) along with
    their
    public key. This structure is a bit more complex but feasible,
    it is a map
    of digital signature formats and the chain of dig sig
    attestations.
    The
    map's key being the method (i.e. CRC32, PKCS7 [3], XmlDigSig
    [4]) and then
    a list of map where that key is "purpose" of signature (what your attesting
    too). As a sibling field to the list another field for "the
    attester"
    as
    bytes (e.g. their PKCS12 [5] for the map of PKCS7 signatures).

    4) Authorization

    We should have a policy of "404" for data, topics, partitions
    (etc) if
    authenticated connections do not have access. In "secure mode"
    any non
    authenticated connections should get a "404" type message on
    everything.
    Knowing "something is there" is a security risk in many uses
    cases. So if
    you don't have access you don't even see it. Baking "that" into
    Kafka
    along with some interface for entitlement (access management)
    systems
    (pretty standard) is all that I think needs to be done to the core project.
    I want to tackle item later in the year after summer after the
    other three
    are complete.

    I look forward to thoughts on this and anyone else interested in
    working
    with us on these items.

    [0]
    http://www.emc.com/emc-plus/rsa-labs/standards-initiatives/what-is-a-digital-envelope.htm
    http://www.emc.com/emc-plus/rsa-labs/standards-initiatives/pkcs-7-cryptographic-message-syntax-standar.htm
    [4] http://en.wikipedia.org/wiki/XML_Signature
    [5] http://en.wikipedia.org/wiki/PKCS_12

    /*******************************************
    Joe Stein
    Founder, Principal Consultant
    Big Data Open Source Security LLC
    http://www.stealth.ly
    Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop>
    ********************************************/
  • Pradeep Gollakota at Jun 6, 2014 at 5:52 pm
    I'm actually not convinced that encryption needs to be handled server side
    in Kafka. I think the best solution for encryption is to handle it
    producer/consumer side just like compression. This will offload key
    management to the users and we'll still be able to leverage the sendfile
    optimization for better performance.

    On Fri, Jun 6, 2014 at 10:48 AM, Rob Withers wrote:

    On consideration, if we have 3 different access groups (1 for production
    WRITE and 2 consumers) they all need to decode the same encryption and so
    all need the same public/private key....certs won't work, unless you write
    a CertAuthority to build multiple certs with the same keys. Better seems
    to not use certs and wrap the encryption specification with an ACL
    capabilities for each group of access.


    On Jun 6, 2014, at 11:43 AM, Rob Withers wrote:

    This is quite interesting to me and it is an excelent opportunity to
    promote a slightly different security scheme. Object-capabilities are
    perfect for online security and would use ACL style authentication to gain
    capabilities filtered to those allowed resources for allow actions
    (READ/WRITE/DELETE/LIST/SCAN). Erights.org has the quitenscential (??)
    object capabilities model and capnproto is impleemting this for C++. I
    have a java implementation at http://github.com/pauwau/pauwau but the
    master is broken. 0.2 works, basically. B asically a TLS connection with
    no certificate server, it is peer to peer. It has some advanced features,
    but the lining of capabilities with authorization so that you can only
    invoke correct services is extended to the secure user.

    Regarding non-repudiation, on disk, why not prepend a CRC?

    Regarding on-disk encryption, multiple users/groups may need to access,
    with different capabilities. Sounds like zookeeper needs to store a cert
    for each class of access so that a group member can access the decrypted
    data from disk. Use cert-based async decryption. The only isue is storing
    the private key in zookeeper. Perhaps some hash magic could be used.

    Thanks for kafka,
    Rob

    On Jun 5, 2014, at 3:01 PM, Jay Kreps wrote:

    Hey Joe,
    I don't really understand the sections you added to the wiki. Can you
    clarify them?

    Is non-repudiation what SASL would call integrity checks? If so don't SSL
    and and many of the SASL schemes already support this as well as
    on-the-wire encryption?

    Or are you proposing an on-disk encryption scheme? Is this actually
    needed?
    Isn't a on-the-wire encryption when combined with mutual authentication
    and
    permissions sufficient for most uses?

    On-disk encryption seems unnecessary because if an attacker can get root
    on
    the kafka boxes it can potentially modify Kafka to do anything he or she
    wants with data. So this seems to break any security model.

    I understand the problem of a large organization not really having a
    trusted network and wanting to secure data transfer and limit and audit
    data access. The uses for these other things I don't totally understand.

    Also it would be worth understanding the state of other messaging and
    storage systems (Hadoop, dbs, etc). What features do they support. I
    think
    there is a sense in which you don't have to run faster than the bear, but
    only faster then your friends. :-)

    -Jay


    On Wed, Jun 4, 2014 at 5:57 PM, Joe Stein wrote:

    I like the idea of working on the spec and prioritizing. I will update
    the
    wiki.

    - Joestein


    On Wed, Jun 4, 2014 at 1:11 PM, Jay Kreps wrote:

    Hey Joe,
    Thanks for kicking this discussion off! I totally agree that for something
    that acts as a central message broker security is critical feature. I think
    a number of people have been interested in this topic and several
    people
    have put effort into special purpose security efforts.

    Since most the LinkedIn folks are working on the consumer right now I think
    this would be a great project for any other interested people to take
    on.
    There are some challenges in doing these things distributed but it can also
    be a lot of fun.

    I think a good first step would be to get a written plan we can all
    agree
    on for how things should work. Then we can break things down into
    chunks
    that can be done independently while still aiming at a good end state.

    I had tried to write up some notes that summarized at least the
    thoughts I
    had had on security:
    https://cwiki.apache.org/confluence/display/KAFKA/Security

    What do you think of that?

    One assumption I had (which may be incorrect) is that although we want all
    the things in your list, the two most pressing would be authentication and
    authorization, and that was all that write up covered. You have more
    experience in this domain, so I wonder how you would prioritize?

    Those notes are really sketchy, so I think the first goal I would have
    would be to get to a real spec we can all agree on and discuss. A lot
    of
    the security stuff has a high human interaction element and needs to
    work
    in pretty different domains and different companies so getting this
    kind of
    review is important.

    -Jay


    On Tue, Jun 3, 2014 at 12:57 PM, Joe Stein <joe.stein@stealth.ly>
    wrote:

    Hi,I wanted to re-ignite the discussion around Apache Kafka Security.
      This
    is a huge bottleneck (non-starter in some cases) for a lot of
    organizations
    (due to regulatory, compliance and other requirements). Below are my
    suggestions for specific changes in Kafka to accommodate security
    requirements. This comes from what folks are doing "in the wild" to
    workaround and implement security with Kafka as it is today and also what I
    have discovered from organizations about their blockers. It also picks
    up
    from the wiki (which I should have time to update later in the week
    based
    on the below and feedback from the thread).
    1) Transport Layer Security (i.e. SSL)

    This also includes client authentication in addition to in-transit security
    layer. This work has been picked up here
    https://issues.apache.org/jira/browse/KAFKA-1477 and do appreciate
    any
    thoughts, comments, feedback, tomatoes, whatever for this patch. It
    is a
    pickup from the fork of the work first done here
    https://github.com/relango/kafka/tree/kafka_security.

    2) Data encryption at rest.

    This is very important and something that can be facilitated within
    the
    wire protocol. It requires an additional map data structure for the
    "encrypted [data encryption key]". With this map (either in your
    object or
    in the wire protocol) you can store the dynamically generated
    symmetric key
    (for each message) and then encrypt the data using that dynamically
    generated key. You then encrypt the encryption key using each public
    key
    for whom is expected to be able to decrypt the encryption key to then
    decrypt the message. For each public key encrypted symmetric key
    (which
    is
    now the "encrypted [data encryption key]" along with which public key
    it
    was encrypted with for (so a map of [publicKey] =
    encryptedDataEncryptionKey) as a chain. Other patterns can be
    implemented
    but this is a pretty standard digital enveloping [0] pattern with only
    1
    field added. Other patterns should be able to use that field to-do
    their
    implementation too.
    3) Non-repudiation and long term non-repudiation.

    Non-repudiation is proving data hasn't changed. This is often (if not
    always) done with x509 public certificates (chained to a certificate
    authority).

    Long term non-repudiation is what happens when the certificates of the
    certificate authority are expired (or revoked) and everything ever
    signed
    (ever) with that certificate's public key then becomes "no longer
      provable
    as ever being authentic". That is where RFC3126 [1] and RFC3161 [2]
    come
    in (or worm drives [hardware], etc).
    For either (or both) of these it is an operation of the encryptor to
    sign/hash the data (with or without third party trusted timestap of
    the
    signing event) and encrypt that with their own private key and
    distribute
    the results (before and after encrypting if required) along with their
    public key. This structure is a bit more complex but feasible, it is a map
    of digital signature formats and the chain of dig sig attestations.
    The
    map's key being the method (i.e. CRC32, PKCS7 [3], XmlDigSig [4]) and
      then
    a list of map where that key is "purpose" of signature (what your attesting
    too). As a sibling field to the list another field for "the attester"
    as
    bytes (e.g. their PKCS12 [5] for the map of PKCS7 signatures).
    4) Authorization

    We should have a policy of "404" for data, topics, partitions (etc) if
    authenticated connections do not have access. In "secure mode" any
    non
    authenticated connections should get a "404" type message on
    everything.
    Knowing "something is there" is a security risk in many uses cases. So
      if
    you don't have access you don't even see it. Baking "that" into Kafka
    along with some interface for entitlement (access management) systems
    (pretty standard) is all that I think needs to be done to the core project.
    I want to tackle item later in the year after summer after the other three
    are complete.

    I look forward to thoughts on this and anyone else interested in
    working
    with us on these items.
    [0]

    http://www.emc.com/emc-plus/rsa-labs/standards-
    initiatives/what-is-a-digital-envelope.htm
    [1] http://tools.ietf.org/html/rfc3126
    http://www.emc.com/emc-plus/rsa-labs/standards-initiatives/pkcs-7-
    cryptographic-message-syntax-standar.htm
    [4] http://en.wikipedia.org/wiki/XML_Signature
    [5] http://en.wikipedia.org/wiki/PKCS_12

    /*******************************************
    Joe Stein
    Founder, Principal Consultant
    Big Data Open Source Security LLC
    http://www.stealth.ly
    Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop>
    ********************************************/
  • Robert Withers at Jun 6, 2014 at 5:55 pm
    It would be nice to have Alcatraz on-disk security for the discriminating client.

    Thanks,
    Rob
    On Jun 6, 2014, at 11:51 AM, Pradeep Gollakota wrote:

    I'm actually not convinced that encryption needs to be handled server side
    in Kafka. I think the best solution for encryption is to handle it
    producer/consumer side just like compression. This will offload key
    management to the users and we'll still be able to leverage the sendfile
    optimization for better performance.

    On Fri, Jun 6, 2014 at 10:48 AM, Rob Withers wrote:

    On consideration, if we have 3 different access groups (1 for production
    WRITE and 2 consumers) they all need to decode the same encryption and so
    all need the same public/private key....certs won't work, unless you write
    a CertAuthority to build multiple certs with the same keys. Better seems
    to not use certs and wrap the encryption specification with an ACL
    capabilities for each group of access.


    On Jun 6, 2014, at 11:43 AM, Rob Withers wrote:

    This is quite interesting to me and it is an excelent opportunity to
    promote a slightly different security scheme. Object-capabilities are
    perfect for online security and would use ACL style authentication to gain
    capabilities filtered to those allowed resources for allow actions
    (READ/WRITE/DELETE/LIST/SCAN). Erights.org has the quitenscential (??)
    object capabilities model and capnproto is impleemting this for C++. I
    have a java implementation at http://github.com/pauwau/pauwau but the
    master is broken. 0.2 works, basically. B asically a TLS connection with
    no certificate server, it is peer to peer. It has some advanced features,
    but the lining of capabilities with authorization so that you can only
    invoke correct services is extended to the secure user.

    Regarding non-repudiation, on disk, why not prepend a CRC?

    Regarding on-disk encryption, multiple users/groups may need to access,
    with different capabilities. Sounds like zookeeper needs to store a cert
    for each class of access so that a group member can access the decrypted
    data from disk. Use cert-based async decryption. The only isue is storing
    the private key in zookeeper. Perhaps some hash magic could be used.

    Thanks for kafka,
    Rob

    On Jun 5, 2014, at 3:01 PM, Jay Kreps wrote:

    Hey Joe,
    I don't really understand the sections you added to the wiki. Can you
    clarify them?

    Is non-repudiation what SASL would call integrity checks? If so don't SSL
    and and many of the SASL schemes already support this as well as
    on-the-wire encryption?

    Or are you proposing an on-disk encryption scheme? Is this actually
    needed?
    Isn't a on-the-wire encryption when combined with mutual authentication
    and
    permissions sufficient for most uses?

    On-disk encryption seems unnecessary because if an attacker can get root
    on
    the kafka boxes it can potentially modify Kafka to do anything he or she
    wants with data. So this seems to break any security model.

    I understand the problem of a large organization not really having a
    trusted network and wanting to secure data transfer and limit and audit
    data access. The uses for these other things I don't totally understand.

    Also it would be worth understanding the state of other messaging and
    storage systems (Hadoop, dbs, etc). What features do they support. I
    think
    there is a sense in which you don't have to run faster than the bear, but
    only faster then your friends. :-)

    -Jay


    On Wed, Jun 4, 2014 at 5:57 PM, Joe Stein wrote:

    I like the idea of working on the spec and prioritizing. I will update
    the
    wiki.

    - Joestein


    On Wed, Jun 4, 2014 at 1:11 PM, Jay Kreps wrote:

    Hey Joe,
    Thanks for kicking this discussion off! I totally agree that for something
    that acts as a central message broker security is critical feature. I think
    a number of people have been interested in this topic and several
    people
    have put effort into special purpose security efforts.

    Since most the LinkedIn folks are working on the consumer right now I think
    this would be a great project for any other interested people to take
    on.
    There are some challenges in doing these things distributed but it can also
    be a lot of fun.

    I think a good first step would be to get a written plan we can all
    agree
    on for how things should work. Then we can break things down into
    chunks
    that can be done independently while still aiming at a good end state.

    I had tried to write up some notes that summarized at least the
    thoughts I
    had had on security:
    https://cwiki.apache.org/confluence/display/KAFKA/Security

    What do you think of that?

    One assumption I had (which may be incorrect) is that although we want all
    the things in your list, the two most pressing would be authentication and
    authorization, and that was all that write up covered. You have more
    experience in this domain, so I wonder how you would prioritize?

    Those notes are really sketchy, so I think the first goal I would have
    would be to get to a real spec we can all agree on and discuss. A lot
    of
    the security stuff has a high human interaction element and needs to
    work
    in pretty different domains and different companies so getting this
    kind of
    review is important.

    -Jay


    On Tue, Jun 3, 2014 at 12:57 PM, Joe Stein <joe.stein@stealth.ly>
    wrote:

    Hi,I wanted to re-ignite the discussion around Apache Kafka Security.
    This
    is a huge bottleneck (non-starter in some cases) for a lot of
    organizations
    (due to regulatory, compliance and other requirements). Below are my
    suggestions for specific changes in Kafka to accommodate security
    requirements. This comes from what folks are doing "in the wild" to
    workaround and implement security with Kafka as it is today and also what I
    have discovered from organizations about their blockers. It also picks
    up
    from the wiki (which I should have time to update later in the week
    based
    on the below and feedback from the thread).
    1) Transport Layer Security (i.e. SSL)

    This also includes client authentication in addition to in-transit security
    layer. This work has been picked up here
    https://issues.apache.org/jira/browse/KAFKA-1477 and do appreciate
    any
    thoughts, comments, feedback, tomatoes, whatever for this patch. It
    is a
    pickup from the fork of the work first done here
    https://github.com/relango/kafka/tree/kafka_security.

    2) Data encryption at rest.

    This is very important and something that can be facilitated within
    the
    wire protocol. It requires an additional map data structure for the
    "encrypted [data encryption key]". With this map (either in your
    object or
    in the wire protocol) you can store the dynamically generated
    symmetric key
    (for each message) and then encrypt the data using that dynamically
    generated key. You then encrypt the encryption key using each public
    key
    for whom is expected to be able to decrypt the encryption key to then
    decrypt the message. For each public key encrypted symmetric key
    (which
    is
    now the "encrypted [data encryption key]" along with which public key
    it
    was encrypted with for (so a map of [publicKey] =
    encryptedDataEncryptionKey) as a chain. Other patterns can be
    implemented
    but this is a pretty standard digital enveloping [0] pattern with only
    1
    field added. Other patterns should be able to use that field to-do
    their
    implementation too.
    3) Non-repudiation and long term non-repudiation.

    Non-repudiation is proving data hasn't changed. This is often (if not
    always) done with x509 public certificates (chained to a certificate
    authority).

    Long term non-repudiation is what happens when the certificates of the
    certificate authority are expired (or revoked) and everything ever
    signed
    (ever) with that certificate's public key then becomes "no longer
    provable
    as ever being authentic". That is where RFC3126 [1] and RFC3161 [2]
    come
    in (or worm drives [hardware], etc).
    For either (or both) of these it is an operation of the encryptor to
    sign/hash the data (with or without third party trusted timestap of
    the
    signing event) and encrypt that with their own private key and
    distribute
    the results (before and after encrypting if required) along with their
    public key. This structure is a bit more complex but feasible, it is a map
    of digital signature formats and the chain of dig sig attestations.
    The
    map's key being the method (i.e. CRC32, PKCS7 [3], XmlDigSig [4]) and
    then
    a list of map where that key is "purpose" of signature (what your attesting
    too). As a sibling field to the list another field for "the attester"
    as
    bytes (e.g. their PKCS12 [5] for the map of PKCS7 signatures).
    4) Authorization

    We should have a policy of "404" for data, topics, partitions (etc) if
    authenticated connections do not have access. In "secure mode" any
    non
    authenticated connections should get a "404" type message on
    everything.
    Knowing "something is there" is a security risk in many uses cases. So
    if
    you don't have access you don't even see it. Baking "that" into Kafka
    along with some interface for entitlement (access management) systems
    (pretty standard) is all that I think needs to be done to the core project.
    I want to tackle item later in the year after summer after the other three
    are complete.

    I look forward to thoughts on this and anyone else interested in
    working
    with us on these items.
    [0]
    http://www.emc.com/emc-plus/rsa-labs/standards-
    initiatives/what-is-a-digital-envelope.htm
    [1] http://tools.ietf.org/html/rfc3126
    http://www.emc.com/emc-plus/rsa-labs/standards-initiatives/pkcs-7-
    cryptographic-message-syntax-standar.htm
    [4] http://en.wikipedia.org/wiki/XML_Signature
    [5] http://en.wikipedia.org/wiki/PKCS_12

    /*******************************************
    Joe Stein
    Founder, Principal Consultant
    Big Data Open Source Security LLC
    http://www.stealth.ly
    Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop>
    ********************************************/
  • Todd Palino at Jun 6, 2014 at 6:15 pm
    Yes, I realized last night that I needed to be clearer in what I was
    saying. Encryption should ABSOLUTELY not be handled server-side. I think
    it¹s a good idea to enable use of it in the consumer/producer, but doing
    it server side will not solve many use cases for needing encryption
    because the server then has access to all the keys. You could say that
    this eliminates the need for TLS, but TLS is pretty low-hanging fruit, and
    there¹s definitely a need for encryption of the traffic across the network
    even if you don¹t need at-rest encryption as well.

    And as you mentioned, something needs to be done about key management.
    Storing information with the message about which key(s) was used is a good
    idea, because it allows you to know when a producer has switched keys.
    There are definitely some alternative solutions to that as well. But
    storing the keys in the broker, Zookeeper, or other systems like that are
    not. There needs to be a system used where the keys are only available to
    the producers and consumers that need them, and they only get access to
    the appropriate part of the key pair. Even as the guy running Kafka and
    Zookeeper, I should not have access to the keys being used, and if data is
    encrypted I should not be able to see the cleartext.

    And even if we decide not to put anything about at-rest encryption in the
    consumer/producer clients directly, and leave it for an exercise above
    that level (you have to pass the ciphertext as the message to the client),
    I still think there is a good case for implementing a message envelope
    that can store the information about which key was used, and other
    pertinent metadata, and have the ability for special applications like
    mirror maker to be able to preserve it across clusters. This still helps
    to enable the use of encryption and other features (like auditing) even if
    we decide it¹s too large a scope to fully implement.

    -Todd
    On 6/6/14, 10:51 AM, "Pradeep Gollakota" wrote:

    I'm actually not convinced that encryption needs to be handled server side
    in Kafka. I think the best solution for encryption is to handle it
    producer/consumer side just like compression. This will offload key
    management to the users and we'll still be able to leverage the sendfile
    optimization for better performance.

    On Fri, Jun 6, 2014 at 10:48 AM, Rob Withers wrote:

    On consideration, if we have 3 different access groups (1 for production
    WRITE and 2 consumers) they all need to decode the same encryption and
    so
    all need the same public/private key....certs won't work, unless you
    write
    a CertAuthority to build multiple certs with the same keys. Better
    seems
    to not use certs and wrap the encryption specification with an ACL
    capabilities for each group of access.


    On Jun 6, 2014, at 11:43 AM, Rob Withers wrote:

    This is quite interesting to me and it is an excelent opportunity to
    promote a slightly different security scheme. Object-capabilities are
    perfect for online security and would use ACL style authentication to
    gain
    capabilities filtered to those allowed resources for allow actions
    (READ/WRITE/DELETE/LIST/SCAN). Erights.org has the quitenscential (??)
    object capabilities model and capnproto is impleemting this for C++. I
    have a java implementation at http://github.com/pauwau/pauwau but the
    master is broken. 0.2 works, basically. B asically a TLS connection
    with
    no certificate server, it is peer to peer. It has some advanced
    features,
    but the lining of capabilities with authorization so that you can only
    invoke correct services is extended to the secure user.

    Regarding non-repudiation, on disk, why not prepend a CRC?

    Regarding on-disk encryption, multiple users/groups may need to access,
    with different capabilities. Sounds like zookeeper needs to store a
    cert
    for each class of access so that a group member can access the
    decrypted
    data from disk. Use cert-based async decryption. The only isue is
    storing
    the private key in zookeeper. Perhaps some hash magic could be used.

    Thanks for kafka,
    Rob

    On Jun 5, 2014, at 3:01 PM, Jay Kreps wrote:

    Hey Joe,
    I don't really understand the sections you added to the wiki. Can you
    clarify them?

    Is non-repudiation what SASL would call integrity checks? If so don't
    SSL
    and and many of the SASL schemes already support this as well as
    on-the-wire encryption?

    Or are you proposing an on-disk encryption scheme? Is this actually
    needed?
    Isn't a on-the-wire encryption when combined with mutual
    authentication
    and
    permissions sufficient for most uses?

    On-disk encryption seems unnecessary because if an attacker can get
    root
    on
    the kafka boxes it can potentially modify Kafka to do anything he or
    she
    wants with data. So this seems to break any security model.

    I understand the problem of a large organization not really having a
    trusted network and wanting to secure data transfer and limit and
    audit
    data access. The uses for these other things I don't totally
    understand.

    Also it would be worth understanding the state of other messaging and
    storage systems (Hadoop, dbs, etc). What features do they support. I
    think
    there is a sense in which you don't have to run faster than the bear,
    but
    only faster then your friends. :-)

    -Jay


    On Wed, Jun 4, 2014 at 5:57 PM, Joe Stein <joe.stein@stealth.ly>
    wrote:

    I like the idea of working on the spec and prioritizing. I will
    update
    the
    wiki.

    - Joestein


    On Wed, Jun 4, 2014 at 1:11 PM, Jay Kreps <jay.kreps@gmail.com>
    wrote:

    Hey Joe,
    Thanks for kicking this discussion off! I totally agree that for something
    that acts as a central message broker security is critical feature.
    I think
    a number of people have been interested in this topic and several
    people
    have put effort into special purpose security efforts.

    Since most the LinkedIn folks are working on the consumer right now
    I think
    this would be a great project for any other interested people to
    take
    on.
    There are some challenges in doing these things distributed but it
    can also
    be a lot of fun.

    I think a good first step would be to get a written plan we can all
    agree
    on for how things should work. Then we can break things down into
    chunks
    that can be done independently while still aiming at a good end
    state.

    I had tried to write up some notes that summarized at least the
    thoughts I
    had had on security:
    https://cwiki.apache.org/confluence/display/KAFKA/Security

    What do you think of that?

    One assumption I had (which may be incorrect) is that although we
    want all
    the things in your list, the two most pressing would be
    authentication and
    authorization, and that was all that write up covered. You have more
    experience in this domain, so I wonder how you would prioritize?

    Those notes are really sketchy, so I think the first goal I would
    have
    would be to get to a real spec we can all agree on and discuss. A
    lot
    of
    the security stuff has a high human interaction element and needs to
    work
    in pretty different domains and different companies so getting this
    kind of
    review is important.

    -Jay


    On Tue, Jun 3, 2014 at 12:57 PM, Joe Stein <joe.stein@stealth.ly>
    wrote:

    Hi,I wanted to re-ignite the discussion around Apache Kafka
    Security.
      This
    is a huge bottleneck (non-starter in some cases) for a lot of
    organizations
    (due to regulatory, compliance and other requirements). Below are
    my
    suggestions for specific changes in Kafka to accommodate security
    requirements. This comes from what folks are doing "in the wild"
    to
    workaround and implement security with Kafka as it is today and
    also what I
    have discovered from organizations about their blockers. It also
    picks
    up
    from the wiki (which I should have time to update later in the week
    based
    on the below and feedback from the thread).
    1) Transport Layer Security (i.e. SSL)

    This also includes client authentication in addition to in-transit security
    layer. This work has been picked up here
    https://issues.apache.org/jira/browse/KAFKA-1477 and do appreciate
    any
    thoughts, comments, feedback, tomatoes, whatever for this patch.
    It
    is a
    pickup from the fork of the work first done here
    https://github.com/relango/kafka/tree/kafka_security.

    2) Data encryption at rest.

    This is very important and something that can be facilitated within
    the
    wire protocol. It requires an additional map data structure for the
    "encrypted [data encryption key]". With this map (either in your
    object or
    in the wire protocol) you can store the dynamically generated
    symmetric key
    (for each message) and then encrypt the data using that dynamically
    generated key. You then encrypt the encryption key using each
    public
    key
    for whom is expected to be able to decrypt the encryption key to
    then
    decrypt the message. For each public key encrypted symmetric key
    (which
    is
    now the "encrypted [data encryption key]" along with which public
    key
    it
    was encrypted with for (so a map of [publicKey] =
    encryptedDataEncryptionKey) as a chain. Other patterns can be
    implemented
    but this is a pretty standard digital enveloping [0] pattern with
    only
    1
    field added. Other patterns should be able to use that field to-do
    their
    implementation too.
    3) Non-repudiation and long term non-repudiation.

    Non-repudiation is proving data hasn't changed. This is often (if
    not
    always) done with x509 public certificates (chained to a
    certificate
    authority).

    Long term non-repudiation is what happens when the certificates of
    the
    certificate authority are expired (or revoked) and everything ever
    signed
    (ever) with that certificate's public key then becomes "no longer
      provable
    as ever being authentic". That is where RFC3126 [1] and RFC3161
    [2]
    come
    in (or worm drives [hardware], etc).
    For either (or both) of these it is an operation of the encryptor
    to
    sign/hash the data (with or without third party trusted timestap of
    the
    signing event) and encrypt that with their own private key and
    distribute
    the results (before and after encrypting if required) along with
    their
    public key. This structure is a bit more complex but feasible, it
    is a map
    of digital signature formats and the chain of dig sig attestations.
    The
    map's key being the method (i.e. CRC32, PKCS7 [3], XmlDigSig [4])
    and
      then
    a list of map where that key is "purpose" of signature (what your attesting
    too). As a sibling field to the list another field for "the
    attester"
    as
    bytes (e.g. their PKCS12 [5] for the map of PKCS7 signatures).
    4) Authorization

    We should have a policy of "404" for data, topics, partitions
    (etc) if
    authenticated connections do not have access. In "secure mode" any
    non
    authenticated connections should get a "404" type message on
    everything.
    Knowing "something is there" is a security risk in many uses cases.
    So
      if
    you don't have access you don't even see it. Baking "that" into
    Kafka
    along with some interface for entitlement (access management)
    systems
    (pretty standard) is all that I think needs to be done to the core project.
    I want to tackle item later in the year after summer after the
    other three
    are complete.

    I look forward to thoughts on this and anyone else interested in
    working
    with us on these items.
    [0]

    http://www.emc.com/emc-plus/rsa-labs/standards-
    initiatives/what-is-a-digital-envelope.htm
    [1] http://tools.ietf.org/html/rfc3126
    http://www.emc.com/emc-plus/rsa-labs/standards-initiatives/pkcs-7-
    cryptographic-message-syntax-standar.htm
    [4] http://en.wikipedia.org/wiki/XML_Signature
    [5] http://en.wikipedia.org/wiki/PKCS_12

    /*******************************************
    Joe Stein
    Founder, Principal Consultant
    Big Data Open Source Security LLC
    http://www.stealth.ly
    Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop>
    ********************************************/
  • Steven Schlansker at Jun 6, 2014 at 9:20 pm
    Hi,
    I’m glad there’s so much thought into getting security right!
    But as a user of Kafka who doesn’t need Enterprise Security sort of features,
    I would ask whether doing such a large project built into Kafka is the appropriate
    use of developer time at this point in its lifecycle.

    For example, even basic functionality like deleting topics or balancing partitions
    does not work reliably / correctly, or is harder than it needs to be.

    Kafka is a message broker and IMHO it would be nice if it fulfills that role well
    over adding such complicated and time-consuming features, which inevitably will
    introduce new bugs, new maintenance concerns, and split developer focus.

    Not trying to rain on the parade, just voicing my 2¢,
    Steven
    On Jun 6, 2014, at 11:15 AM, Todd Palino wrote:

    Yes, I realized last night that I needed to be clearer in what I was
    saying. Encryption should ABSOLUTELY not be handled server-side. I think
    it¹s a good idea to enable use of it in the consumer/producer, but doing
    it server side will not solve many use cases for needing encryption
    because the server then has access to all the keys. You could say that
    this eliminates the need for TLS, but TLS is pretty low-hanging fruit, and
    there¹s definitely a need for encryption of the traffic across the network
    even if you don¹t need at-rest encryption as well.

    And as you mentioned, something needs to be done about key management.
    Storing information with the message about which key(s) was used is a good
    idea, because it allows you to know when a producer has switched keys.
    There are definitely some alternative solutions to that as well. But
    storing the keys in the broker, Zookeeper, or other systems like that are
    not. There needs to be a system used where the keys are only available to
    the producers and consumers that need them, and they only get access to
    the appropriate part of the key pair. Even as the guy running Kafka and
    Zookeeper, I should not have access to the keys being used, and if data is
    encrypted I should not be able to see the cleartext.

    And even if we decide not to put anything about at-rest encryption in the
    consumer/producer clients directly, and leave it for an exercise above
    that level (you have to pass the ciphertext as the message to the client),
    I still think there is a good case for implementing a message envelope
    that can store the information about which key was used, and other
    pertinent metadata, and have the ability for special applications like
    mirror maker to be able to preserve it across clusters. This still helps
    to enable the use of encryption and other features (like auditing) even if
    we decide it¹s too large a scope to fully implement.

    -Todd
    On 6/6/14, 10:51 AM, "Pradeep Gollakota" wrote:

    I'm actually not convinced that encryption needs to be handled server side
    in Kafka. I think the best solution for encryption is to handle it
    producer/consumer side just like compression. This will offload key
    management to the users and we'll still be able to leverage the sendfile
    optimization for better performance.


    On Fri, Jun 6, 2014 at 10:48 AM, Rob Withers <robert.w.withers@gmail.com>
    wrote:
    On consideration, if we have 3 different access groups (1 for production
    WRITE and 2 consumers) they all need to decode the same encryption and
    so
    all need the same public/private key....certs won't work, unless you
    write
    a CertAuthority to build multiple certs with the same keys. Better
    seems
    to not use certs and wrap the encryption specification with an ACL
    capabilities for each group of access.


    On Jun 6, 2014, at 11:43 AM, Rob Withers wrote:

    This is quite interesting to me and it is an excelent opportunity to
    promote a slightly different security scheme. Object-capabilities are
    perfect for online security and would use ACL style authentication to
    gain
    capabilities filtered to those allowed resources for allow actions
    (READ/WRITE/DELETE/LIST/SCAN). Erights.org has the quitenscential (??)
    object capabilities model and capnproto is impleemting this for C++. I
    have a java implementation at http://github.com/pauwau/pauwau but the
    master is broken. 0.2 works, basically. B asically a TLS connection
    with
    no certificate server, it is peer to peer. It has some advanced
    features,
    but the lining of capabilities with authorization so that you can only
    invoke correct services is extended to the secure user.

    Regarding non-repudiation, on disk, why not prepend a CRC?

    Regarding on-disk encryption, multiple users/groups may need to access,
    with different capabilities. Sounds like zookeeper needs to store a
    cert
    for each class of access so that a group member can access the
    decrypted
    data from disk. Use cert-based async decryption. The only isue is
    storing
    the private key in zookeeper. Perhaps some hash magic could be used.

    Thanks for kafka,
    Rob

    On Jun 5, 2014, at 3:01 PM, Jay Kreps wrote:

    Hey Joe,
    I don't really understand the sections you added to the wiki. Can you
    clarify them?

    Is non-repudiation what SASL would call integrity checks? If so don't
    SSL
    and and many of the SASL schemes already support this as well as
    on-the-wire encryption?

    Or are you proposing an on-disk encryption scheme? Is this actually
    needed?
    Isn't a on-the-wire encryption when combined with mutual
    authentication
    and
    permissions sufficient for most uses?

    On-disk encryption seems unnecessary because if an attacker can get
    root
    on
    the kafka boxes it can potentially modify Kafka to do anything he or
    she
    wants with data. So this seems to break any security model.

    I understand the problem of a large organization not really having a
    trusted network and wanting to secure data transfer and limit and
    audit
    data access. The uses for these other things I don't totally
    understand.

    Also it would be worth understanding the state of other messaging and
    storage systems (Hadoop, dbs, etc). What features do they support. I
    think
    there is a sense in which you don't have to run faster than the bear,
    but
    only faster then your friends. :-)

    -Jay


    On Wed, Jun 4, 2014 at 5:57 PM, Joe Stein <joe.stein@stealth.ly>
    wrote:

    I like the idea of working on the spec and prioritizing. I will
    update
    the
    wiki.

    - Joestein


    On Wed, Jun 4, 2014 at 1:11 PM, Jay Kreps <jay.kreps@gmail.com>
    wrote:

    Hey Joe,
    Thanks for kicking this discussion off! I totally agree that for something
    that acts as a central message broker security is critical feature.
    I think
    a number of people have been interested in this topic and several
    people
    have put effort into special purpose security efforts.

    Since most the LinkedIn folks are working on the consumer right now
    I think
    this would be a great project for any other interested people to
    take
    on.
    There are some challenges in doing these things distributed but it
    can also
    be a lot of fun.

    I think a good first step would be to get a written plan we can all
    agree
    on for how things should work. Then we can break things down into
    chunks
    that can be done independently while still aiming at a good end
    state.

    I had tried to write up some notes that summarized at least the
    thoughts I
    had had on security:
    https://cwiki.apache.org/confluence/display/KAFKA/Security

    What do you think of that?

    One assumption I had (which may be incorrect) is that although we
    want all
    the things in your list, the two most pressing would be
    authentication and
    authorization, and that was all that write up covered. You have more
    experience in this domain, so I wonder how you would prioritize?

    Those notes are really sketchy, so I think the first goal I would
    have
    would be to get to a real spec we can all agree on and discuss. A
    lot
    of
    the security stuff has a high human interaction element and needs to
    work
    in pretty different domains and different companies so getting this
    kind of
    review is important.

    -Jay


    On Tue, Jun 3, 2014 at 12:57 PM, Joe Stein <joe.stein@stealth.ly>
    wrote:

    Hi,I wanted to re-ignite the discussion around Apache Kafka
    Security.
      This
    is a huge bottleneck (non-starter in some cases) for a lot of
    organizations
    (due to regulatory, compliance and other requirements). Below are
    my
    suggestions for specific changes in Kafka to accommodate security
    requirements. This comes from what folks are doing "in the wild"
    to
    workaround and implement security with Kafka as it is today and
    also what I
    have discovered from organizations about their blockers. It also
    picks
    up
    from the wiki (which I should have time to update later in the week
    based
    on the below and feedback from the thread).
    1) Transport Layer Security (i.e. SSL)

    This also includes client authentication in addition to in-transit security
    layer. This work has been picked up here
    https://issues.apache.org/jira/browse/KAFKA-1477 and do appreciate
    any
    thoughts, comments, feedback, tomatoes, whatever for this patch.
    It
    is a
    pickup from the fork of the work first done here
    https://github.com/relango/kafka/tree/kafka_security.

    2) Data encryption at rest.

    This is very important and something that can be facilitated within
    the
    wire protocol. It requires an additional map data structure for the
    "encrypted [data encryption key]". With this map (either in your
    object or
    in the wire protocol) you can store the dynamically generated
    symmetric key
    (for each message) and then encrypt the data using that dynamically
    generated key. You then encrypt the encryption key using each
    public
    key
    for whom is expected to be able to decrypt the encryption key to
    then
    decrypt the message. For each public key encrypted symmetric key
    (which
    is
    now the "encrypted [data encryption key]" along with which public
    key
    it
    was encrypted with for (so a map of [publicKey] =
    encryptedDataEncryptionKey) as a chain. Other patterns can be
    implemented
    but this is a pretty standard digital enveloping [0] pattern with
    only
    1
    field added. Other patterns should be able to use that field to-do
    their
    implementation too.
    3) Non-repudiation and long term non-repudiation.

    Non-repudiation is proving data hasn't changed. This is often (if
    not
    always) done with x509 public certificates (chained to a
    certificate
    authority).

    Long term non-repudiation is what happens when the certificates of
    the
    certificate authority are expired (or revoked) and everything ever
    signed
    (ever) with that certificate's public key then becomes "no longer
      provable
    as ever being authentic". That is where RFC3126 [1] and RFC3161
    [2]
    come
    in (or worm drives [hardware], etc).
    For either (or both) of these it is an operation of the encryptor
    to
    sign/hash the data (with or without third party trusted timestap of
    the
    signing event) and encrypt that with their own private key and
    distribute
    the results (before and after encrypting if required) along with
    their
    public key. This structure is a bit more complex but feasible, it
    is a map
    of digital signature formats and the chain of dig sig attestations.
    The
    map's key being the method (i.e. CRC32, PKCS7 [3], XmlDigSig [4])
    and
      then
    a list of map where that key is "purpose" of signature (what your attesting
    too). As a sibling field to the list another field for "the
    attester"
    as
    bytes (e.g. their PKCS12 [5] for the map of PKCS7 signatures).
    4) Authorization

    We should have a policy of "404" for data, topics, partitions
    (etc) if
    authenticated connections do not have access. In "secure mode" any
    non
    authenticated connections should get a "404" type message on
    everything.
    Knowing "something is there" is a security risk in many uses cases.
    So
      if
    you don't have access you don't even see it. Baking "that" into
    Kafka
    along with some interface for entitlement (access management)
    systems
    (pretty standard) is all that I think needs to be done to the core project.
    I want to tackle item later in the year after summer after the
    other three
    are complete.

    I look forward to thoughts on this and anyone else interested in
    working
    with us on these items.
    [0]

    http://www.emc.com/emc-plus/rsa-labs/standards-
    initiatives/what-is-a-digital-envelope.htm
    [1] http://tools.ietf.org/html/rfc3126
    http://www.emc.com/emc-plus/rsa-labs/standards-initiatives/pkcs-7-
    cryptographic-message-syntax-standar.htm
    [4] http://en.wikipedia.org/wiki/XML_Signature
    [5] http://en.wikipedia.org/wiki/PKCS_12

    /*******************************************
    Joe Stein
    Founder, Principal Consultant
    Big Data Open Source Security LLC
    http://www.stealth.ly
    Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop>
    ********************************************/
  • Rob Withers at Jun 7, 2014 at 7:26 pm
    At one level this makes sense to me to externalize the security issue
    to producers and consumers. On consideration I realized that this
    adds a lot of coordination requirements to the app layer across teams
    or even companies. Another issue I feel is that you want a specific
    unchanging encryption for the data and the clients (producers/
    consumers) will need to be able to decode frozen data. If certs are
    used they cannot expire. Also, different clients would need to use
    the same cert.

    So, you statement that it should ABSOLUTELY not include internal
    encryption rings seems misplaced. There are some customers of kafka
    that would opt to encrypt the on-disk data and key management is a
    significant issue. This is best handled internally, with key
    management stored in either ZK or in a topic. Truly, perhaps
    annealing Hadoop/HBASE as a metadata store seems applicable.

    Thanks, another 2 cents,
    Rob
    On Jun 6, 2014, at 12:15 PM, Todd Palino wrote:

    Yes, I realized last night that I needed to be clearer in what I was
    saying. Encryption should ABSOLUTELY not be handled server-side. I
    think
    it¹s a good idea to enable use of it in the consumer/producer, but
    doing
    it server side will not solve many use cases for needing encryption
    because the server then has access to all the keys. You could say that
    this eliminates the need for TLS, but TLS is pretty low-hanging
    fruit, and
    there¹s definitely a need for encryption of the traffic across the
    network
    even if you don¹t need at-rest encryption as well.

    And as you mentioned, something needs to be done about key management.
    Storing information with the message about which key(s) was used is
    a good
    idea, because it allows you to know when a producer has switched keys.
    There are definitely some alternative solutions to that as well. But
    storing the keys in the broker, Zookeeper, or other systems like
    that are
    not. There needs to be a system used where the keys are only
    available to
    the producers and consumers that need them, and they only get access
    to
    the appropriate part of the key pair. Even as the guy running Kafka
    and
    Zookeeper, I should not have access to the keys being used, and if
    data is
    encrypted I should not be able to see the cleartext.

    And even if we decide not to put anything about at-rest encryption
    in the
    consumer/producer clients directly, and leave it for an exercise above
    that level (you have to pass the ciphertext as the message to the
    client),
    I still think there is a good case for implementing a message envelope
    that can store the information about which key was used, and other
    pertinent metadata, and have the ability for special applications like
    mirror maker to be able to preserve it across clusters. This still
    helps
    to enable the use of encryption and other features (like auditing)
    even if
    we decide it¹s too large a scope to fully implement.

    -Todd
    On 6/6/14, 10:51 AM, "Pradeep Gollakota" wrote:

    I'm actually not convinced that encryption needs to be handled
    server side
    in Kafka. I think the best solution for encryption is to handle it
    producer/consumer side just like compression. This will offload key
    management to the users and we'll still be able to leverage the
    sendfile
    optimization for better performance.


    On Fri, Jun 6, 2014 at 10:48 AM, Rob Withers <robert.w.withers@gmail.com
      wrote:
    On consideration, if we have 3 different access groups (1 for
    production
    WRITE and 2 consumers) they all need to decode the same encryption
    and
    so
    all need the same public/private key....certs won't work, unless you
    write
    a CertAuthority to build multiple certs with the same keys. Better
    seems
    to not use certs and wrap the encryption specification with an ACL
    capabilities for each group of access.


    On Jun 6, 2014, at 11:43 AM, Rob Withers wrote:

    This is quite interesting to me and it is an excelent opportunity to
    promote a slightly different security scheme. Object-
    capabilities are
    perfect for online security and would use ACL style
    authentication to
    gain
    capabilities filtered to those allowed resources for allow actions
    (READ/WRITE/DELETE/LIST/SCAN). Erights.org has the
    quitenscential (??)
    object capabilities model and capnproto is impleemting this for C+
    +. I
    have a java implementation at http://github.com/pauwau/pauwau but
    the
    master is broken. 0.2 works, basically. B asically a TLS
    connection
    with
    no certificate server, it is peer to peer. It has some advanced
    features,
    but the lining of capabilities with authorization so that you can
    only
    invoke correct services is extended to the secure user.

    Regarding non-repudiation, on disk, why not prepend a CRC?

    Regarding on-disk encryption, multiple users/groups may need to
    access,
    with different capabilities. Sounds like zookeeper needs to
    store a
    cert
    for each class of access so that a group member can access the
    decrypted
    data from disk. Use cert-based async decryption. The only isue is
    storing
    the private key in zookeeper. Perhaps some hash magic could be
    used.

    Thanks for kafka,
    Rob

    On Jun 5, 2014, at 3:01 PM, Jay Kreps wrote:

    Hey Joe,
    I don't really understand the sections you added to the wiki.
    Can you
    clarify them?

    Is non-repudiation what SASL would call integrity checks? If so
    don't
    SSL
    and and many of the SASL schemes already support this as well as
    on-the-wire encryption?

    Or are you proposing an on-disk encryption scheme? Is this
    actually
    needed?
    Isn't a on-the-wire encryption when combined with mutual
    authentication
    and
    permissions sufficient for most uses?

    On-disk encryption seems unnecessary because if an attacker can
    get
    root
    on
    the kafka boxes it can potentially modify Kafka to do anything
    he or
    she
    wants with data. So this seems to break any security model.

    I understand the problem of a large organization not really
    having a
    trusted network and wanting to secure data transfer and limit and
    audit
    data access. The uses for these other things I don't totally
    understand.

    Also it would be worth understanding the state of other
    messaging and
    storage systems (Hadoop, dbs, etc). What features do they
    support. I
    think
    there is a sense in which you don't have to run faster than the
    bear,
    but
    only faster then your friends. :-)

    -Jay


    On Wed, Jun 4, 2014 at 5:57 PM, Joe Stein <joe.stein@stealth.ly>
    wrote:

    I like the idea of working on the spec and prioritizing. I will
    update
    the
    wiki.

    - Joestein


    On Wed, Jun 4, 2014 at 1:11 PM, Jay Kreps <jay.kreps@gmail.com>
    wrote:

    Hey Joe,
    Thanks for kicking this discussion off! I totally agree that for something
    that acts as a central message broker security is critical
    feature.
    I think
    a number of people have been interested in this topic and
    several
    people
    have put effort into special purpose security efforts.

    Since most the LinkedIn folks are working on the consumer
    right now
    I think
    this would be a great project for any other interested people to
    take
    on.
    There are some challenges in doing these things distributed
    but it
    can also
    be a lot of fun.

    I think a good first step would be to get a written plan we
    can all
    agree
    on for how things should work. Then we can break things down
    into
    chunks
    that can be done independently while still aiming at a good end
    state.

    I had tried to write up some notes that summarized at least the
    thoughts I
    had had on security:
    https://cwiki.apache.org/confluence/display/KAFKA/Security

    What do you think of that?

    One assumption I had (which may be incorrect) is that although
    we
    want all
    the things in your list, the two most pressing would be
    authentication and
    authorization, and that was all that write up covered. You
    have more
    experience in this domain, so I wonder how you would prioritize?

    Those notes are really sketchy, so I think the first goal I
    would
    have
    would be to get to a real spec we can all agree on and
    discuss. A
    lot
    of
    the security stuff has a high human interaction element and
    needs to
    work
    in pretty different domains and different companies so getting
    this
    kind of
    review is important.

    -Jay


    On Tue, Jun 3, 2014 at 12:57 PM, Joe Stein
    <joe.stein@stealth.ly>
    wrote:

    Hi,I wanted to re-ignite the discussion around Apache Kafka
    Security.
      This
    is a huge bottleneck (non-starter in some cases) for a lot of
    organizations
    (due to regulatory, compliance and other requirements). Below
    are
    my
    suggestions for specific changes in Kafka to accommodate
    security
    requirements. This comes from what folks are doing "in the
    wild"
    to
    workaround and implement security with Kafka as it is today and
    also what I
    have discovered from organizations about their blockers. It
    also
    picks
    up
    from the wiki (which I should have time to update later in the
    week
    based
    on the below and feedback from the thread).
    1) Transport Layer Security (i.e. SSL)

    This also includes client authentication in addition to in-
    transit security
    layer. This work has been picked up here
    https://issues.apache.org/jira/browse/KAFKA-1477 and do
    appreciate
    any
    thoughts, comments, feedback, tomatoes, whatever for this
    patch.
    It
    is a
    pickup from the fork of the work first done here
    https://github.com/relango/kafka/tree/kafka_security.

    2) Data encryption at rest.

    This is very important and something that can be facilitated
    within
    the
    wire protocol. It requires an additional map data structure
    for the
    "encrypted [data encryption key]". With this map (either in
    your
    object or
    in the wire protocol) you can store the dynamically generated
    symmetric key
    (for each message) and then encrypt the data using that
    dynamically
    generated key. You then encrypt the encryption key using each
    public
    key
    for whom is expected to be able to decrypt the encryption key to
    then
    decrypt the message. For each public key encrypted symmetric
    key
    (which
    is
    now the "encrypted [data encryption key]" along with which
    public
    key
    it
    was encrypted with for (so a map of [publicKey] =
    encryptedDataEncryptionKey) as a chain. Other patterns can be
    implemented
    but this is a pretty standard digital enveloping [0] pattern
    with
    only
    1
    field added. Other patterns should be able to use that field
    to-do
    their
    implementation too.
    3) Non-repudiation and long term non-repudiation.

    Non-repudiation is proving data hasn't changed. This is
    often (if
    not
    always) done with x509 public certificates (chained to a
    certificate
    authority).

    Long term non-repudiation is what happens when the
    certificates of
    the
    certificate authority are expired (or revoked) and everything
    ever
    signed
    (ever) with that certificate's public key then becomes "no
    longer
      provable
    as ever being authentic". That is where RFC3126 [1] and
    RFC3161
    [2]
    come
    in (or worm drives [hardware], etc).
    For either (or both) of these it is an operation of the
    encryptor
    to
    sign/hash the data (with or without third party trusted
    timestap of
    the
    signing event) and encrypt that with their own private key and
    distribute
    the results (before and after encrypting if required) along with
    their
    public key. This structure is a bit more complex but
    feasible, it
    is a map
    of digital signature formats and the chain of dig sig
    attestations.
    The
    map's key being the method (i.e. CRC32, PKCS7 [3], XmlDigSig
    [4])
    and
      then
    a list of map where that key is "purpose" of signature (what
    your attesting
    too). As a sibling field to the list another field for "the
    attester"
    as
    bytes (e.g. their PKCS12 [5] for the map of PKCS7 signatures).
    4) Authorization

    We should have a policy of "404" for data, topics, partitions
    (etc) if
    authenticated connections do not have access. In "secure
    mode" any
    non
    authenticated connections should get a "404" type message on
    everything.
    Knowing "something is there" is a security risk in many uses
    cases.
    So
      if
    you don't have access you don't even see it. Baking "that"
    into
    Kafka
    along with some interface for entitlement (access management)
    systems
    (pretty standard) is all that I think needs to be done to the
    core project.
    I want to tackle item later in the year after summer after the
    other three
    are complete.

    I look forward to thoughts on this and anyone else interested
    in
    working
    with us on these items.
    [0]

    http://www.emc.com/emc-plus/rsa-labs/standards-
    initiatives/what-is-a-digital-envelope.htm
    [1] http://tools.ietf.org/html/rfc3126
    http://www.emc.com/emc-plus/rsa-labs/standards-initiatives/pkcs-7-
    cryptographic-message-syntax-standar.htm
    [4] http://en.wikipedia.org/wiki/XML_Signature
    [5] http://en.wikipedia.org/wiki/PKCS_12

    /*******************************************
    Joe Stein
    Founder, Principal Consultant
    Big Data Open Source Security LLC
    http://www.stealth.ly
    Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop
    ********************************************/
  • Todd Palino at Jun 8, 2014 at 9:07 pm
    I’ll agree that perhaps the “absolutely not” is not quite right. There are
    certainly some uses for a simpler solution, but I would still say it
    cannot only be encryption at the broker. This would leave many use cases
    for at-rest encryption out of the loop (most auditing cases for SOX, PCI,
    HIPAA, and other PII standards). Yes, it does add external overhead that
    must be managed, but it’s just the nature of the beast. We can’t solve all
    of the external infrastructure needed for this, but we can make it easier
    to use for consumers and producers by adding metadata.

    There’s no need for unchanging encryption, and that’s specifically why I
    want to see a message envelope that will help consumers determine the
    encryption uses for a particular message. You can definitely still expire
    keys, you just have to keep the expired keys around as long as the
    encrypted data stays around, and your endpoints need to know when they are
    decrypting data with an expired key (you might want to throw up a warning,
    or do something else to let the users know that it’s happening). And as
    someone else mentioned, there are solutions for encrypting data for
    multiple consumers. You can encrypt the data with an OTP, and then
    multiply encrypt the OTP once for each consumer and store those encrypted
    strings in the envelope.

    -Todd
    On 6/7/14, 12:25 PM, "Rob Withers" wrote:

    At one level this makes sense to me to externalize the security issue
    to producers and consumers. On consideration I realized that this
    adds a lot of coordination requirements to the app layer across teams
    or even companies. Another issue I feel is that you want a specific
    unchanging encryption for the data and the clients (producers/
    consumers) will need to be able to decode frozen data. If certs are
    used they cannot expire. Also, different clients would need to use
    the same cert.

    So, you statement that it should ABSOLUTELY not include internal
    encryption rings seems misplaced. There are some customers of kafka
    that would opt to encrypt the on-disk data and key management is a
    significant issue. This is best handled internally, with key
    management stored in either ZK or in a topic. Truly, perhaps
    annealing Hadoop/HBASE as a metadata store seems applicable.

    Thanks, another 2 cents,
    Rob
    On Jun 6, 2014, at 12:15 PM, Todd Palino wrote:

    Yes, I realized last night that I needed to be clearer in what I was
    saying. Encryption should ABSOLUTELY not be handled server-side. I
    think
    it¹s a good idea to enable use of it in the consumer/producer, but
    doing
    it server side will not solve many use cases for needing encryption
    because the server then has access to all the keys. You could say that
    this eliminates the need for TLS, but TLS is pretty low-hanging
    fruit, and
    there¹s definitely a need for encryption of the traffic across the
    network
    even if you don¹t need at-rest encryption as well.

    And as you mentioned, something needs to be done about key management.
    Storing information with the message about which key(s) was used is
    a good
    idea, because it allows you to know when a producer has switched keys.
    There are definitely some alternative solutions to that as well. But
    storing the keys in the broker, Zookeeper, or other systems like
    that are
    not. There needs to be a system used where the keys are only
    available to
    the producers and consumers that need them, and they only get access
    to
    the appropriate part of the key pair. Even as the guy running Kafka
    and
    Zookeeper, I should not have access to the keys being used, and if
    data is
    encrypted I should not be able to see the cleartext.

    And even if we decide not to put anything about at-rest encryption
    in the
    consumer/producer clients directly, and leave it for an exercise above
    that level (you have to pass the ciphertext as the message to the
    client),
    I still think there is a good case for implementing a message envelope
    that can store the information about which key was used, and other
    pertinent metadata, and have the ability for special applications like
    mirror maker to be able to preserve it across clusters. This still
    helps
    to enable the use of encryption and other features (like auditing)
    even if
    we decide it¹s too large a scope to fully implement.

    -Todd
    On 6/6/14, 10:51 AM, "Pradeep Gollakota" wrote:

    I'm actually not convinced that encryption needs to be handled
    server side
    in Kafka. I think the best solution for encryption is to handle it
    producer/consumer side just like compression. This will offload key
    management to the users and we'll still be able to leverage the
    sendfile
    optimization for better performance.


    On Fri, Jun 6, 2014 at 10:48 AM, Rob Withers
    <robert.w.withers@gmail.com
      wrote:
    On consideration, if we have 3 different access groups (1 for
    production
    WRITE and 2 consumers) they all need to decode the same encryption
    and
    so
    all need the same public/private key....certs won't work, unless you
    write
    a CertAuthority to build multiple certs with the same keys. Better
    seems
    to not use certs and wrap the encryption specification with an ACL
    capabilities for each group of access.


    On Jun 6, 2014, at 11:43 AM, Rob Withers wrote:

    This is quite interesting to me and it is an excelent opportunity to
    promote a slightly different security scheme. Object-
    capabilities are
    perfect for online security and would use ACL style
    authentication to
    gain
    capabilities filtered to those allowed resources for allow actions
    (READ/WRITE/DELETE/LIST/SCAN). Erights.org has the
    quitenscential (??)
    object capabilities model and capnproto is impleemting this for C+
    +. I
    have a java implementation at http://github.com/pauwau/pauwau but
    the
    master is broken. 0.2 works, basically. B asically a TLS
    connection
    with
    no certificate server, it is peer to peer. It has some advanced
    features,
    but the lining of capabilities with authorization so that you can
    only
    invoke correct services is extended to the secure user.

    Regarding non-repudiation, on disk, why not prepend a CRC?

    Regarding on-disk encryption, multiple users/groups may need to
    access,
    with different capabilities. Sounds like zookeeper needs to
    store a
    cert
    for each class of access so that a group member can access the
    decrypted
    data from disk. Use cert-based async decryption. The only isue is
    storing
    the private key in zookeeper. Perhaps some hash magic could be
    used.

    Thanks for kafka,
    Rob

    On Jun 5, 2014, at 3:01 PM, Jay Kreps wrote:

    Hey Joe,
    I don't really understand the sections you added to the wiki.
    Can you
    clarify them?

    Is non-repudiation what SASL would call integrity checks? If so
    don't
    SSL
    and and many of the SASL schemes already support this as well as
    on-the-wire encryption?

    Or are you proposing an on-disk encryption scheme? Is this
    actually
    needed?
    Isn't a on-the-wire encryption when combined with mutual
    authentication
    and
    permissions sufficient for most uses?

    On-disk encryption seems unnecessary because if an attacker can
    get
    root
    on
    the kafka boxes it can potentially modify Kafka to do anything
    he or
    she
    wants with data. So this seems to break any security model.

    I understand the problem of a large organization not really
    having a
    trusted network and wanting to secure data transfer and limit and
    audit
    data access. The uses for these other things I don't totally
    understand.

    Also it would be worth understanding the state of other
    messaging and
    storage systems (Hadoop, dbs, etc). What features do they
    support. I
    think
    there is a sense in which you don't have to run faster than the
    bear,
    but
    only faster then your friends. :-)

    -Jay


    On Wed, Jun 4, 2014 at 5:57 PM, Joe Stein <joe.stein@stealth.ly>
    wrote:

    I like the idea of working on the spec and prioritizing. I will
    update
    the
    wiki.

    - Joestein


    On Wed, Jun 4, 2014 at 1:11 PM, Jay Kreps <jay.kreps@gmail.com>
    wrote:

    Hey Joe,
    Thanks for kicking this discussion off! I totally agree that for something
    that acts as a central message broker security is critical
    feature.
    I think
    a number of people have been interested in this topic and
    several
    people
    have put effort into special purpose security efforts.

    Since most the LinkedIn folks are working on the consumer
    right now
    I think
    this would be a great project for any other interested people to
    take
    on.
    There are some challenges in doing these things distributed
    but it
    can also
    be a lot of fun.

    I think a good first step would be to get a written plan we
    can all
    agree
    on for how things should work. Then we can break things down
    into
    chunks
    that can be done independently while still aiming at a good end
    state.

    I had tried to write up some notes that summarized at least the
    thoughts I
    had had on security:
    https://cwiki.apache.org/confluence/display/KAFKA/Security

    What do you think of that?

    One assumption I had (which may be incorrect) is that although
    we
    want all
    the things in your list, the two most pressing would be
    authentication and
    authorization, and that was all that write up covered. You
    have more
    experience in this domain, so I wonder how you would prioritize?

    Those notes are really sketchy, so I think the first goal I
    would
    have
    would be to get to a real spec we can all agree on and
    discuss. A
    lot
    of
    the security stuff has a high human interaction element and
    needs to
    work
    in pretty different domains and different companies so getting
    this
    kind of
    review is important.

    -Jay


    On Tue, Jun 3, 2014 at 12:57 PM, Joe Stein
    <joe.stein@stealth.ly>
    wrote:

    Hi,I wanted to re-ignite the discussion around Apache Kafka
    Security.
      This
    is a huge bottleneck (non-starter in some cases) for a lot of
    organizations
    (due to regulatory, compliance and other requirements). Below
    are
    my
    suggestions for specific changes in Kafka to accommodate
    security
    requirements. This comes from what folks are doing "in the
    wild"
    to
    workaround and implement security with Kafka as it is today and
    also what I
    have discovered from organizations about their blockers. It
    also
    picks
    up
    from the wiki (which I should have time to update later in the
    week
    based
    on the below and feedback from the thread).
    1) Transport Layer Security (i.e. SSL)

    This also includes client authentication in addition to in-
    transit security
    layer. This work has been picked up here
    https://issues.apache.org/jira/browse/KAFKA-1477 and do
    appreciate
    any
    thoughts, comments, feedback, tomatoes, whatever for this
    patch.
    It
    is a
    pickup from the fork of the work first done here
    https://github.com/relango/kafka/tree/kafka_security.

    2) Data encryption at rest.

    This is very important and something that can be facilitated
    within
    the
    wire protocol. It requires an additional map data structure
    for the
    "encrypted [data encryption key]". With this map (either in
    your
    object or
    in the wire protocol) you can store the dynamically generated
    symmetric key
    (for each message) and then encrypt the data using that
    dynamically
    generated key. You then encrypt the encryption key using each
    public
    key
    for whom is expected to be able to decrypt the encryption key to
    then
    decrypt the message. For each public key encrypted symmetric
    key
    (which
    is
    now the "encrypted [data encryption key]" along with which
    public
    key
    it
    was encrypted with for (so a map of [publicKey] =
    encryptedDataEncryptionKey) as a chain. Other patterns can be
    implemented
    but this is a pretty standard digital enveloping [0] pattern
    with
    only
    1
    field added. Other patterns should be able to use that field
    to-do
    their
    implementation too.
    3) Non-repudiation and long term non-repudiation.

    Non-repudiation is proving data hasn't changed. This is
    often (if
    not
    always) done with x509 public certificates (chained to a
    certificate
    authority).

    Long term non-repudiation is what happens when the
    certificates of
    the
    certificate authority are expired (or revoked) and everything
    ever
    signed
    (ever) with that certificate's public key then becomes "no
    longer
      provable
    as ever being authentic". That is where RFC3126 [1] and
    RFC3161
    [2]
    come
    in (or worm drives [hardware], etc).
    For either (or both) of these it is an operation of the
    encryptor
    to
    sign/hash the data (with or without third party trusted
    timestap of
    the
    signing event) and encrypt that with their own private key and
    distribute
    the results (before and after encrypting if required) along with
    their
    public key. This structure is a bit more complex but
    feasible, it
    is a map
    of digital signature formats and the chain of dig sig
    attestations.
    The
    map's key being the method (i.e. CRC32, PKCS7 [3], XmlDigSig
    [4])
    and
      then
    a list of map where that key is "purpose" of signature (what
    your attesting
    too). As a sibling field to the list another field for "the
    attester"
    as
    bytes (e.g. their PKCS12 [5] for the map of PKCS7 signatures).
    4) Authorization

    We should have a policy of "404" for data, topics, partitions
    (etc) if
    authenticated connections do not have access. In "secure
    mode" any
    non
    authenticated connections should get a "404" type message on
    everything.
    Knowing "something is there" is a security risk in many uses
    cases.
    So
      if
    you don't have access you don't even see it. Baking "that"
    into
    Kafka
    along with some interface for entitlement (access management)
    systems
    (pretty standard) is all that I think needs to be done to the
    core project.
    I want to tackle item later in the year after summer after the
    other three
    are complete.

    I look forward to thoughts on this and anyone else interested
    in
    working
    with us on these items.
    [0]

    http://www.emc.com/emc-plus/rsa-labs/standards-
    initiatives/what-is-a-digital-envelope.htm
    [1] http://tools.ietf.org/html/rfc3126
    http://www.emc.com/emc-plus/rsa-labs/standards-initiatives/pkcs-7-
    cryptographic-message-syntax-standar.htm
    [4] http://en.wikipedia.org/wiki/XML_Signature
    [5] http://en.wikipedia.org/wiki/PKCS_12

    /*******************************************
    Joe Stein
    Founder, Principal Consultant
    Big Data Open Source Security LLC
    http://www.stealth.ly
    Twitter: @allthingshadoop
    <http://www.twitter.com/allthingshadoop
    ********************************************/
  • Rob Withers at Jun 8, 2014 at 9:34 pm
    I like the use of meta envelopes. We did this recently, on the job,
    as we have an envelope that specifies the type for decoding. We
    discussed adding the encodinType and you are suggesting adding
    encryption metadata for that msg. All good.

    I don't see your OTP example. Could you delve deeper for me, please?
    The model I envision is internal OTP, with access to decryption
    accessed by cert. A double layer of security, with the internal at-
    rest encryption being an unchanging OTP with ACL access to it as the
    upper layer. Are you saying it is possible to re-encrypt with new
    keys or that there is a chain of keys over time?

    Thanks,
    Rob
    On Jun 8, 2014, at 3:06 PM, Todd Palino wrote:

    I’ll agree that perhaps the “absolutely not” is not quite right.
    There are
    certainly some uses for a simpler solution, but I would still say it
    cannot only be encryption at the broker. This would leave many use
    cases
    for at-rest encryption out of the loop (most auditing cases for SOX,
    PCI,
    HIPAA, and other PII standards). Yes, it does add external overhead
    that
    must be managed, but it’s just the nature of the beast. We can’t
    solve all
    of the external infrastructure needed for this, but we can make it
    easier
    to use for consumers and producers by adding metadata.

    There’s no need for unchanging encryption, and that’s specifically
    why I
    want to see a message envelope that will help consumers determine the
    encryption uses for a particular message. You can definitely still
    expire
    keys, you just have to keep the expired keys around as long as the
    encrypted data stays around, and your endpoints need to know when
    they are
    decrypting data with an expired key (you might want to throw up a
    warning,
    or do something else to let the users know that it’s happening). And
    as
    someone else mentioned, there are solutions for encrypting data for
    multiple consumers. You can encrypt the data with an OTP, and then
    multiply encrypt the OTP once for each consumer and store those
    encrypted
    strings in the envelope.

    -Todd
    On 6/7/14, 12:25 PM, "Rob Withers" wrote:

    At one level this makes sense to me to externalize the security issue
    to producers and consumers. On consideration I realized that this
    adds a lot of coordination requirements to the app layer across teams
    or even companies. Another issue I feel is that you want a specific
    unchanging encryption for the data and the clients (producers/
    consumers) will need to be able to decode frozen data. If certs are
    used they cannot expire. Also, different clients would need to use
    the same cert.

    So, you statement that it should ABSOLUTELY not include internal
    encryption rings seems misplaced. There are some customers of kafka
    that would opt to encrypt the on-disk data and key management is a
    significant issue. This is best handled internally, with key
    management stored in either ZK or in a topic. Truly, perhaps
    annealing Hadoop/HBASE as a metadata store seems applicable.

    Thanks, another 2 cents,
    Rob
    On Jun 6, 2014, at 12:15 PM, Todd Palino wrote:

    Yes, I realized last night that I needed to be clearer in what I was
    saying. Encryption should ABSOLUTELY not be handled server-side. I
    think
    it¹s a good idea to enable use of it in the consumer/producer, but
    doing
    it server side will not solve many use cases for needing encryption
    because the server then has access to all the keys. You could say
    that
    this eliminates the need for TLS, but TLS is pretty low-hanging
    fruit, and
    there¹s definitely a need for encryption of the traffic across the
    network
    even if you don¹t need at-rest encryption as well.

    And as you mentioned, something needs to be done about key
    management.
    Storing information with the message about which key(s) was used is
    a good
    idea, because it allows you to know when a producer has switched
    keys.
    There are definitely some alternative solutions to that as well. But
    storing the keys in the broker, Zookeeper, or other systems like
    that are
    not. There needs to be a system used where the keys are only
    available to
    the producers and consumers that need them, and they only get access
    to
    the appropriate part of the key pair. Even as the guy running Kafka
    and
    Zookeeper, I should not have access to the keys being used, and if
    data is
    encrypted I should not be able to see the cleartext.

    And even if we decide not to put anything about at-rest encryption
    in the
    consumer/producer clients directly, and leave it for an exercise
    above
    that level (you have to pass the ciphertext as the message to the
    client),
    I still think there is a good case for implementing a message
    envelope
    that can store the information about which key was used, and other
    pertinent metadata, and have the ability for special applications
    like
    mirror maker to be able to preserve it across clusters. This still
    helps
    to enable the use of encryption and other features (like auditing)
    even if
    we decide it¹s too large a scope to fully implement.

    -Todd

    On 6/6/14, 10:51 AM, "Pradeep Gollakota" <pradeepg26@gmail.com>
    wrote:
    I'm actually not convinced that encryption needs to be handled
    server side
    in Kafka. I think the best solution for encryption is to handle it
    producer/consumer side just like compression. This will offload key
    management to the users and we'll still be able to leverage the
    sendfile
    optimization for better performance.


    On Fri, Jun 6, 2014 at 10:48 AM, Rob Withers
    <robert.w.withers@gmail.com
      wrote:
    On consideration, if we have 3 different access groups (1 for
    production
    WRITE and 2 consumers) they all need to decode the same encryption
    and
    so
    all need the same public/private key....certs won't work, unless
    you
    write
    a CertAuthority to build multiple certs with the same keys.
    Better
    seems
    to not use certs and wrap the encryption specification with an ACL
    capabilities for each group of access.


    On Jun 6, 2014, at 11:43 AM, Rob Withers wrote:

    This is quite interesting to me and it is an excelent
    opportunity to
    promote a slightly different security scheme. Object-
    capabilities are
    perfect for online security and would use ACL style
    authentication to
    gain
    capabilities filtered to those allowed resources for allow
    actions
    (READ/WRITE/DELETE/LIST/SCAN). Erights.org has the
    quitenscential (??)
    object capabilities model and capnproto is impleemting this for
    C+
    +. I
    have a java implementation at http://github.com/pauwau/pauwau but
    the
    master is broken. 0.2 works, basically. B asically a TLS
    connection
    with
    no certificate server, it is peer to peer. It has some advanced
    features,
    but the lining of capabilities with authorization so that you can
    only
    invoke correct services is extended to the secure user.

    Regarding non-repudiation, on disk, why not prepend a CRC?

    Regarding on-disk encryption, multiple users/groups may need to
    access,
    with different capabilities. Sounds like zookeeper needs to
    store a
    cert
    for each class of access so that a group member can access the
    decrypted
    data from disk. Use cert-based async decryption. The only
    isue is
    storing
    the private key in zookeeper. Perhaps some hash magic could be
    used.

    Thanks for kafka,
    Rob

    On Jun 5, 2014, at 3:01 PM, Jay Kreps wrote:

    Hey Joe,
    I don't really understand the sections you added to the wiki.
    Can you
    clarify them?

    Is non-repudiation what SASL would call integrity checks? If so
    don't
    SSL
    and and many of the SASL schemes already support this as well as
    on-the-wire encryption?

    Or are you proposing an on-disk encryption scheme? Is this
    actually
    needed?
    Isn't a on-the-wire encryption when combined with mutual
    authentication
    and
    permissions sufficient for most uses?

    On-disk encryption seems unnecessary because if an attacker can
    get
    root
    on
    the kafka boxes it can potentially modify Kafka to do anything
    he or
    she
    wants with data. So this seems to break any security model.

    I understand the problem of a large organization not really
    having a
    trusted network and wanting to secure data transfer and limit
    and
    audit
    data access. The uses for these other things I don't totally
    understand.

    Also it would be worth understanding the state of other
    messaging and
    storage systems (Hadoop, dbs, etc). What features do they
    support. I
    think
    there is a sense in which you don't have to run faster than the
    bear,
    but
    only faster then your friends. :-)

    -Jay


    On Wed, Jun 4, 2014 at 5:57 PM, Joe Stein <joe.stein@stealth.ly>
    wrote:

    I like the idea of working on the spec and prioritizing. I will
    update
    the
    wiki.

    - Joestein


    On Wed, Jun 4, 2014 at 1:11 PM, Jay Kreps <jay.kreps@gmail.com>
    wrote:

    Hey Joe,
    Thanks for kicking this discussion off! I totally agree that
    for something
    that acts as a central message broker security is critical
    feature.
    I think
    a number of people have been interested in this topic and
    several
    people
    have put effort into special purpose security efforts.

    Since most the LinkedIn folks are working on the consumer
    right now
    I think
    this would be a great project for any other interested
    people to
    take
    on.
    There are some challenges in doing these things distributed
    but it
    can also
    be a lot of fun.

    I think a good first step would be to get a written plan we
    can all
    agree
    on for how things should work. Then we can break things down
    into
    chunks
    that can be done independently while still aiming at a good
    end
    state.

    I had tried to write up some notes that summarized at least
    the
    thoughts I
    had had on security:
    https://cwiki.apache.org/confluence/display/KAFKA/Security

    What do you think of that?

    One assumption I had (which may be incorrect) is that although
    we
    want all
    the things in your list, the two most pressing would be
    authentication and
    authorization, and that was all that write up covered. You
    have more
    experience in this domain, so I wonder how you would
    prioritize?

    Those notes are really sketchy, so I think the first goal I
    would
    have
    would be to get to a real spec we can all agree on and
    discuss. A
    lot
    of
    the security stuff has a high human interaction element and
    needs to
    work
    in pretty different domains and different companies so getting
    this
    kind of
    review is important.

    -Jay


    On Tue, Jun 3, 2014 at 12:57 PM, Joe Stein
    <joe.stein@stealth.ly>
    wrote:

    Hi,I wanted to re-ignite the discussion around Apache Kafka
    Security.
      This
    is a huge bottleneck (non-starter in some cases) for a lot of
    organizations
    (due to regulatory, compliance and other requirements). Below
    are
    my
    suggestions for specific changes in Kafka to accommodate
    security
    requirements. This comes from what folks are doing "in the
    wild"
    to
    workaround and implement security with Kafka as it is today
    and
    also what I
    have discovered from organizations about their blockers. It
    also
    picks
    up
    from the wiki (which I should have time to update later in the
    week
    based
    on the below and feedback from the thread).
    1) Transport Layer Security (i.e. SSL)

    This also includes client authentication in addition to in-
    transit security
    layer. This work has been picked up here
    https://issues.apache.org/jira/browse/KAFKA-1477 and do
    appreciate
    any
    thoughts, comments, feedback, tomatoes, whatever for this
    patch.
    It
    is a
    pickup from the fork of the work first done here
    https://github.com/relango/kafka/tree/kafka_security.

    2) Data encryption at rest.

    This is very important and something that can be facilitated
    within
    the
    wire protocol. It requires an additional map data structure
    for the
    "encrypted [data encryption key]". With this map (either in
    your
    object or
    in the wire protocol) you can store the dynamically generated
    symmetric key
    (for each message) and then encrypt the data using that
    dynamically
    generated key. You then encrypt the encryption key using
    each
    public
    key
    for whom is expected to be able to decrypt the encryption
    key to
    then
    decrypt the message. For each public key encrypted symmetric
    key
    (which
    is
    now the "encrypted [data encryption key]" along with which
    public
    key
    it
    was encrypted with for (so a map of [publicKey] =
    encryptedDataEncryptionKey) as a chain. Other patterns
    can be
    implemented
    but this is a pretty standard digital enveloping [0] pattern
    with
    only
    1
    field added. Other patterns should be able to use that field
    to-do
    their
    implementation too.
    3) Non-repudiation and long term non-repudiation.

    Non-repudiation is proving data hasn't changed. This is
    often (if
    not
    always) done with x509 public certificates (chained to a
    certificate
    authority).

    Long term non-repudiation is what happens when the
    certificates of
    the
    certificate authority are expired (or revoked) and everything
    ever
    signed
    (ever) with that certificate's public key then becomes "no
    longer
      provable
    as ever being authentic". That is where RFC3126 [1] and
    RFC3161
    [2]
    come
    in (or worm drives [hardware], etc).
    For either (or both) of these it is an operation of the
    encryptor
    to
    sign/hash the data (with or without third party trusted
    timestap of
    the
    signing event) and encrypt that with their own private key
    and
    distribute
    the results (before and after encrypting if required) along
    with
    their
    public key. This structure is a bit more complex but
    feasible, it
    is a map
    of digital signature formats and the chain of dig sig
    attestations.
    The
    map's key being the method (i.e. CRC32, PKCS7 [3], XmlDigSig
    [4])
    and
      then
    a list of map where that key is "purpose" of signature (what
    your attesting
    too). As a sibling field to the list another field for "the
    attester"
    as
    bytes (e.g. their PKCS12 [5] for the map of PKCS7 signatures).
    4) Authorization

    We should have a policy of "404" for data, topics, partitions
    (etc) if
    authenticated connections do not have access. In "secure
    mode" any
    non
    authenticated connections should get a "404" type message on
    everything.
    Knowing "something is there" is a security risk in many uses
    cases.
    So
      if
    you don't have access you don't even see it. Baking "that"
    into
    Kafka
    along with some interface for entitlement (access management)
    systems
    (pretty standard) is all that I think needs to be done to the
    core project.
    I want to tackle item later in the year after summer after
    the
    other three
    are complete.

    I look forward to thoughts on this and anyone else interested
    in
    working
    with us on these items.
    [0]

    http://www.emc.com/emc-plus/rsa-labs/standards-
    initiatives/what-is-a-digital-envelope.htm
    [1] http://tools.ietf.org/html/rfc3126
    http://www.emc.com/emc-plus/rsa-labs/standards-initiatives/pkcs-7-
    cryptographic-message-syntax-standar.htm
    [4] http://en.wikipedia.org/wiki/XML_Signature
    [5] http://en.wikipedia.org/wiki/PKCS_12

    /*******************************************
    Joe Stein
    Founder, Principal Consultant
    Big Data Open Source Security LLC
    http://www.stealth.ly
    Twitter: @allthingshadoop
    <http://www.twitter.com/allthingshadoop
    ********************************************/
  • Todd Palino at Jun 9, 2014 at 1:57 pm
    It’s the same method used by S/MIME and many other encryption
    specifications with the potential for multiple recipients. The sender
    generates a session key, and uses that key to encrypt the message. The
    session key is then encrypted once for each recipient with that
    recipient’s public key. All of the encrypted copies of the session key are
    then included with the encrypted message. This way, you avoid having to
    encrypt the message multiple times (this assumes, of course, that the
    message itself is larger than the key).

    In our case, we have some options available to us. We could do that, and
    put all the encrypted keys in the message metadata. Or we could treat it
    more like a session and have the encrypted session keys in a special topic
    (e.g. __session_keys), much like offsets are now. When the producer starts
    up, they create a session key and encrypt it for each consumer with the
    current consumer key. The producer publishes the bundle of encrypted keys
    into __session_keys as a single message. The producer then publishes
    messages to the normal topic encrypted with the session key. The metadata
    for each of those messages would contain something the offset into
    __session_keys to identify the bundle. This has the added benefit of not
    increasing the per-message data size too much.

    Whenever a consumer key is invalidated, or however often the session key
    should be rotated, the producer would publish a new bundle. This maintains
    a history of session keys that can be used to decrypt any messages, so the
    retention on __session_keys must be at least as long as any topic which
    may potentially contain encrypted data. Past that point, it’s up to the
    consumer what they want to do with the data. A consumer like Hadoop might
    re-encrypt it for local storage, or store it in plaintext (depending on
    the security and requirements of that system).

    -Todd
    On 6/8/14, 2:33 PM, "Rob Withers" wrote:

    I like the use of meta envelopes. We did this recently, on the job,
    as we have an envelope that specifies the type for decoding. We
    discussed adding the encodinType and you are suggesting adding
    encryption metadata for that msg. All good.

    I don't see your OTP example. Could you delve deeper for me, please?
    The model I envision is internal OTP, with access to decryption
    accessed by cert. A double layer of security, with the internal at-
    rest encryption being an unchanging OTP with ACL access to it as the
    upper layer. Are you saying it is possible to re-encrypt with new
    keys or that there is a chain of keys over time?

    Thanks,
    Rob
    On Jun 8, 2014, at 3:06 PM, Todd Palino wrote:

    I’ll agree that perhaps the “absolutely not” is not quite right.
    There are
    certainly some uses for a simpler solution, but I would still say it
    cannot only be encryption at the broker. This would leave many use
    cases
    for at-rest encryption out of the loop (most auditing cases for SOX,
    PCI,
    HIPAA, and other PII standards). Yes, it does add external overhead
    that
    must be managed, but it’s just the nature of the beast. We can’t
    solve all
    of the external infrastructure needed for this, but we can make it
    easier
    to use for consumers and producers by adding metadata.

    There’s no need for unchanging encryption, and that’s specifically
    why I
    want to see a message envelope that will help consumers determine the
    encryption uses for a particular message. You can definitely still
    expire
    keys, you just have to keep the expired keys around as long as the
    encrypted data stays around, and your endpoints need to know when
    they are
    decrypting data with an expired key (you might want to throw up a
    warning,
    or do something else to let the users know that it’s happening). And
    as
    someone else mentioned, there are solutions for encrypting data for
    multiple consumers. You can encrypt the data with an OTP, and then
    multiply encrypt the OTP once for each consumer and store those
    encrypted
    strings in the envelope.

    -Todd
    On 6/7/14, 12:25 PM, "Rob Withers" wrote:

    At one level this makes sense to me to externalize the security issue
    to producers and consumers. On consideration I realized that this
    adds a lot of coordination requirements to the app layer across teams
    or even companies. Another issue I feel is that you want a specific
    unchanging encryption for the data and the clients (producers/
    consumers) will need to be able to decode frozen data. If certs are
    used they cannot expire. Also, different clients would need to use
    the same cert.

    So, you statement that it should ABSOLUTELY not include internal
    encryption rings seems misplaced. There are some customers of kafka
    that would opt to encrypt the on-disk data and key management is a
    significant issue. This is best handled internally, with key
    management stored in either ZK or in a topic. Truly, perhaps
    annealing Hadoop/HBASE as a metadata store seems applicable.

    Thanks, another 2 cents,
    Rob
    On Jun 6, 2014, at 12:15 PM, Todd Palino wrote:

    Yes, I realized last night that I needed to be clearer in what I was
    saying. Encryption should ABSOLUTELY not be handled server-side. I
    think
    it¹s a good idea to enable use of it in the consumer/producer, but
    doing
    it server side will not solve many use cases for needing encryption
    because the server then has access to all the keys. You could say
    that
    this eliminates the need for TLS, but TLS is pretty low-hanging
    fruit, and
    there¹s definitely a need for encryption of the traffic across the
    network
    even if you don¹t need at-rest encryption as well.

    And as you mentioned, something needs to be done about key
    management.
    Storing information with the message about which key(s) was used is
    a good
    idea, because it allows you to know when a producer has switched
    keys.
    There are definitely some alternative solutions to that as well. But
    storing the keys in the broker, Zookeeper, or other systems like
    that are
    not. There needs to be a system used where the keys are only
    available to
    the producers and consumers that need them, and they only get access
    to
    the appropriate part of the key pair. Even as the guy running Kafka
    and
    Zookeeper, I should not have access to the keys being used, and if
    data is
    encrypted I should not be able to see the cleartext.

    And even if we decide not to put anything about at-rest encryption
    in the
    consumer/producer clients directly, and leave it for an exercise
    above
    that level (you have to pass the ciphertext as the message to the
    client),
    I still think there is a good case for implementing a message
    envelope
    that can store the information about which key was used, and other
    pertinent metadata, and have the ability for special applications
    like
    mirror maker to be able to preserve it across clusters. This still
    helps
    to enable the use of encryption and other features (like auditing)
    even if
    we decide it¹s too large a scope to fully implement.

    -Todd

    On 6/6/14, 10:51 AM, "Pradeep Gollakota" <pradeepg26@gmail.com>
    wrote:
    I'm actually not convinced that encryption needs to be handled
    server side
    in Kafka. I think the best solution for encryption is to handle it
    producer/consumer side just like compression. This will offload key
    management to the users and we'll still be able to leverage the
    sendfile
    optimization for better performance.


    On Fri, Jun 6, 2014 at 10:48 AM, Rob Withers
    <robert.w.withers@gmail.com
      wrote:
    On consideration, if we have 3 different access groups (1 for
    production
    WRITE and 2 consumers) they all need to decode the same encryption
    and
    so
    all need the same public/private key....certs won't work, unless
    you
    write
    a CertAuthority to build multiple certs with the same keys.
    Better
    seems
    to not use certs and wrap the encryption specification with an ACL
    capabilities for each group of access.


    On Jun 6, 2014, at 11:43 AM, Rob Withers wrote:

    This is quite interesting to me and it is an excelent
    opportunity to
    promote a slightly different security scheme. Object-
    capabilities are
    perfect for online security and would use ACL style
    authentication to
    gain
    capabilities filtered to those allowed resources for allow
    actions
    (READ/WRITE/DELETE/LIST/SCAN). Erights.org has the
    quitenscential (??)
    object capabilities model and capnproto is impleemting this for
    C+
    +. I
    have a java implementation at http://github.com/pauwau/pauwau but
    the
    master is broken. 0.2 works, basically. B asically a TLS
    connection
    with
    no certificate server, it is peer to peer. It has some advanced
    features,
    but the lining of capabilities with authorization so that you can
    only
    invoke correct services is extended to the secure user.

    Regarding non-repudiation, on disk, why not prepend a CRC?

    Regarding on-disk encryption, multiple users/groups may need to
    access,
    with different capabilities. Sounds like zookeeper needs to
    store a
    cert
    for each class of access so that a group member can access the
    decrypted
    data from disk. Use cert-based async decryption. The only
    isue is
    storing
    the private key in zookeeper. Perhaps some hash magic could be
    used.

    Thanks for kafka,
    Rob

    On Jun 5, 2014, at 3:01 PM, Jay Kreps wrote:

    Hey Joe,
    I don't really understand the sections you added to the wiki.
    Can you
    clarify them?

    Is non-repudiation what SASL would call integrity checks? If so
    don't
    SSL
    and and many of the SASL schemes already support this as well as
    on-the-wire encryption?

    Or are you proposing an on-disk encryption scheme? Is this
    actually
    needed?
    Isn't a on-the-wire encryption when combined with mutual
    authentication
    and
    permissions sufficient for most uses?

    On-disk encryption seems unnecessary because if an attacker can
    get
    root
    on
    the kafka boxes it can potentially modify Kafka to do anything
    he or
    she
    wants with data. So this seems to break any security model.

    I understand the problem of a large organization not really
    having a
    trusted network and wanting to secure data transfer and limit
    and
    audit
    data access. The uses for these other things I don't totally
    understand.

    Also it would be worth understanding the state of other
    messaging and
    storage systems (Hadoop, dbs, etc). What features do they
    support. I
    think
    there is a sense in which you don't have to run faster than the
    bear,
    but
    only faster then your friends. :-)

    -Jay


    On Wed, Jun 4, 2014 at 5:57 PM, Joe Stein <joe.stein@stealth.ly>
    wrote:

    I like the idea of working on the spec and prioritizing. I will
    update
    the
    wiki.

    - Joestein


    On Wed, Jun 4, 2014 at 1:11 PM, Jay Kreps <jay.kreps@gmail.com>
    wrote:

    Hey Joe,
    Thanks for kicking this discussion off! I totally agree that
    for something
    that acts as a central message broker security is critical
    feature.
    I think
    a number of people have been interested in this topic and
    several
    people
    have put effort into special purpose security efforts.

    Since most the LinkedIn folks are working on the consumer
    right now
    I think
    this would be a great project for any other interested
    people to
    take
    on.
    There are some challenges in doing these things distributed
    but it
    can also
    be a lot of fun.

    I think a good first step would be to get a written plan we
    can all
    agree
    on for how things should work. Then we can break things down
    into
    chunks
    that can be done independently while still aiming at a good
    end
    state.

    I had tried to write up some notes that summarized at least
    the
    thoughts I
    had had on security:
    https://cwiki.apache.org/confluence/display/KAFKA/Security

    What do you think of that?

    One assumption I had (which may be incorrect) is that although
    we
    want all
    the things in your list, the two most pressing would be
    authentication and
    authorization, and that was all that write up covered. You
    have more
    experience in this domain, so I wonder how you would
    prioritize?

    Those notes are really sketchy, so I think the first goal I
    would
    have
    would be to get to a real spec we can all agree on and
    discuss. A
    lot
    of
    the security stuff has a high human interaction element and
    needs to
    work
    in pretty different domains and different companies so getting
    this
    kind of
    review is important.

    -Jay


    On Tue, Jun 3, 2014 at 12:57 PM, Joe Stein
    <joe.stein@stealth.ly>
    wrote:

    Hi,I wanted to re-ignite the discussion around Apache Kafka
    Security.
      This
    is a huge bottleneck (non-starter in some cases) for a lot of
    organizations
    (due to regulatory, compliance and other requirements). Below
    are
    my
    suggestions for specific changes in Kafka to accommodate
    security
    requirements. This comes from what folks are doing "in the
    wild"
    to
    workaround and implement security with Kafka as it is today
    and
    also what I
    have discovered from organizations about their blockers. It
    also
    picks
    up
    from the wiki (which I should have time to update later in the
    week
    based
    on the below and feedback from the thread).
    1) Transport Layer Security (i.e. SSL)

    This also includes client authentication in addition to in-
    transit security
    layer. This work has been picked up here
    https://issues.apache.org/jira/browse/KAFKA-1477 and do
    appreciate
    any
    thoughts, comments, feedback, tomatoes, whatever for this
    patch.
    It
    is a
    pickup from the fork of the work first done here
    https://github.com/relango/kafka/tree/kafka_security.

    2) Data encryption at rest.

    This is very important and something that can be facilitated
    within
    the
    wire protocol. It requires an additional map data structure
    for the
    "encrypted [data encryption key]". With this map (either in
    your
    object or
    in the wire protocol) you can store the dynamically generated
    symmetric key
    (for each message) and then encrypt the data using that
    dynamically
    generated key. You then encrypt the encryption key using
    each
    public
    key
    for whom is expected to be able to decrypt the encryption
    key to
    then
    decrypt the message. For each public key encrypted symmetric
    key
    (which
    is
    now the "encrypted [data encryption key]" along with which
    public
    key
    it
    was encrypted with for (so a map of [publicKey] =
    encryptedDataEncryptionKey) as a chain. Other patterns
    can be
    implemented
    but this is a pretty standard digital enveloping [0] pattern
    with
    only
    1
    field added. Other patterns should be able to use that field
    to-do
    their
    implementation too.
    3) Non-repudiation and long term non-repudiation.

    Non-repudiation is proving data hasn't changed. This is
    often (if
    not
    always) done with x509 public certificates (chained to a
    certificate
    authority).

    Long term non-repudiation is what happens when the
    certificates of
    the
    certificate authority are expired (or revoked) and everything
    ever
    signed
    (ever) with that certificate's public key then becomes "no
    longer
      provable
    as ever being authentic". That is where RFC3126 [1] and
    RFC3161
    [2]
    come
    in (or worm drives [hardware], etc).
    For either (or both) of these it is an operation of the
    encryptor
    to
    sign/hash the data (with or without third party trusted
    timestap of
    the
    signing event) and encrypt that with their own private key
    and
    distribute
    the results (before and after encrypting if required) along
    with
    their
    public key. This structure is a bit more complex but
    feasible, it
    is a map
    of digital signature formats and the chain of dig sig
    attestations.
    The
    map's key being the method (i.e. CRC32, PKCS7 [3], XmlDigSig
    [4])
    and
      then
    a list of map where that key is "purpose" of signature (what
    your attesting
    too). As a sibling field to the list another field for "the
    attester"
    as
    bytes (e.g. their PKCS12 [5] for the map of PKCS7 signatures).
    4) Authorization

    We should have a policy of "404" for data, topics, partitions
    (etc) if
    authenticated connections do not have access. In "secure
    mode" any
    non
    authenticated connections should get a "404" type message on
    everything.
    Knowing "something is there" is a security risk in many uses
    cases.
    So
      if
    you don't have access you don't even see it. Baking "that"
    into
    Kafka
    along with some interface for entitlement (access management)
    systems
    (pretty standard) is all that I think needs to be done to the
    core project.
    I want to tackle item later in the year after summer after
    the
    other three
    are complete.

    I look forward to thoughts on this and anyone else interested
    in
    working
    with us on these items.
    [0]

    http://www.emc.com/emc-plus/rsa-labs/standards-
    initiatives/what-is-a-digital-envelope.htm
    [1] http://tools.ietf.org/html/rfc3126
    http://www.emc.com/emc-plus/rsa-labs/standards-initiatives/pkcs-7
    -
    cryptographic-message-syntax-standar.htm
    [4] http://en.wikipedia.org/wiki/XML_Signature
    [5] http://en.wikipedia.org/wiki/PKCS_12

    /*******************************************
    Joe Stein
    Founder, Principal Consultant
    Big Data Open Source Security LLC
    http://www.stealth.ly
    Twitter: @allthingshadoop
    <http://www.twitter.com/allthingshadoop
    ********************************************/
  • Robert Withers at Jun 9, 2014 at 3:17 pm
    Yes, that sounds familiar as I helped write (minimally) S/MIME in squeak (open source Smalltalk environment). This what I was thinking in my alternative here, though I have a concern...

    Production may occur before the consumer is coded and executed. In the analogy of mail, the mail is sent before the complete recipient list is known.

    This seems to mean that the private key (cert or OTP) must be stored and interacted with. My feeling is that key metadata are in a system encrypted Hbase store (session key store), for low latency reads, rather than a topic requiring scanning. Store the private keys and then give client access (producers/consumers) with the hash of the OTP. A new consumer comes along, create a new cert encoding the OTP hash.

    On write, use the producer cert to send a topic hash with the msg which would allow the broker to reuse or generate an OTP, stored in the session key store.

    On read (consumer), if we have a previously run reader, use the encrypted hash. If new, create consumer cert and encrypt the hash for that session.

    The reader/writer will pass a cert encrypted session hash. The trick seems to be converting hash to PK to encrypt/decrypt. Given Kafka resource distribution, we need system encryption for metadata and cert-based key exchange. This seems to mean triple encryption:
    1) client to/from broker
    2) system key/hash mgmt/translation
    3) at-rest encryption

    Thanks,
    Rob
    On Jun 9, 2014, at 7:57 AM, Todd Palino wrote:

    It’s the same method used by S/MIME and many other encryption
    specifications with the potential for multiple recipients. The sender
    generates a session key, and uses that key to encrypt the message. The
    session key is then encrypted once for each recipient with that
    recipient’s public key. All of the encrypted copies of the session key are
    then included with the encrypted message. This way, you avoid having to
    encrypt the message multiple times (this assumes, of course, that the
    message itself is larger than the key).

    In our case, we have some options available to us. We could do that, and
    put all the encrypted keys in the message metadata. Or we could treat it
    more like a session and have the encrypted session keys in a special topic
    (e.g. __session_keys), much like offsets are now. When the producer starts
    up, they create a session key and encrypt it for each consumer with the
    current consumer key. The producer publishes the bundle of encrypted keys
    into __session_keys as a single message. The producer then publishes
    messages to the normal topic encrypted with the session key. The metadata
    for each of those messages would contain something the offset into
    __session_keys to identify the bundle. This has the added benefit of not
    increasing the per-message data size too much.

    Whenever a consumer key is invalidated, or however often the session key
    should be rotated, the producer would publish a new bundle. This maintains
    a history of session keys that can be used to decrypt any messages, so the
    retention on __session_keys must be at least as long as any topic which
    may potentially contain encrypted data. Past that point, it’s up to the
    consumer what they want to do with the data. A consumer like Hadoop might
    re-encrypt it for local storage, or store it in plaintext (depending on
    the security and requirements of that system).

    -Todd
    On 6/8/14, 2:33 PM, "Rob Withers" wrote:

    I like the use of meta envelopes. We did this recently, on the job,
    as we have an envelope that specifies the type for decoding. We
    discussed adding the encodinType and you are suggesting adding
    encryption metadata for that msg. All good.

    I don't see your OTP example. Could you delve deeper for me, please?
    The model I envision is internal OTP, with access to decryption
    accessed by cert. A double layer of security, with the internal at-
    rest encryption being an unchanging OTP with ACL access to it as the
    upper layer. Are you saying it is possible to re-encrypt with new
    keys or that there is a chain of keys over time?

    Thanks,
    Rob
    On Jun 8, 2014, at 3:06 PM, Todd Palino wrote:

    I’ll agree that perhaps the “absolutely not” is not quite right.
    There are
    certainly some uses for a simpler solution, but I would still say it
    cannot only be encryption at the broker. This would leave many use
    cases
    for at-rest encryption out of the loop (most auditing cases for SOX,
    PCI,
    HIPAA, and other PII standards). Yes, it does add external overhead
    that
    must be managed, but it’s just the nature of the beast. We can’t
    solve all
    of the external infrastructure needed for this, but we can make it
    easier
    to use for consumers and producers by adding metadata.

    There’s no need for unchanging encryption, and that’s specifically
    why I
    want to see a message envelope that will help consumers determine the
    encryption uses for a particular message. You can definitely still
    expire
    keys, you just have to keep the expired keys around as long as the
    encrypted data stays around, and your endpoints need to know when
    they are
    decrypting data with an expired key (you might want to throw up a
    warning,
    or do something else to let the users know that it’s happening). And
    as
    someone else mentioned, there are solutions for encrypting data for
    multiple consumers. You can encrypt the data with an OTP, and then
    multiply encrypt the OTP once for each consumer and store those
    encrypted
    strings in the envelope.

    -Todd
    On 6/7/14, 12:25 PM, "Rob Withers" wrote:

    At one level this makes sense to me to externalize the security issue
    to producers and consumers. On consideration I realized that this
    adds a lot of coordination requirements to the app layer across teams
    or even companies. Another issue I feel is that you want a specific
    unchanging encryption for the data and the clients (producers/
    consumers) will need to be able to decode frozen data. If certs are
    used they cannot expire. Also, different clients would need to use
    the same cert.

    So, you statement that it should ABSOLUTELY not include internal
    encryption rings seems misplaced. There are some customers of kafka
    that would opt to encrypt the on-disk data and key management is a
    significant issue. This is best handled internally, with key
    management stored in either ZK or in a topic. Truly, perhaps
    annealing Hadoop/HBASE as a metadata store seems applicable.

    Thanks, another 2 cents,
    Rob
    On Jun 6, 2014, at 12:15 PM, Todd Palino wrote:

    Yes, I realized last night that I needed to be clearer in what I was
    saying. Encryption should ABSOLUTELY not be handled server-side. I
    think
    it¹s a good idea to enable use of it in the consumer/producer, but
    doing
    it server side will not solve many use cases for needing encryption
    because the server then has access to all the keys. You could say
    that
    this eliminates the need for TLS, but TLS is pretty low-hanging
    fruit, and
    there¹s definitely a need for encryption of the traffic across the
    network
    even if you don¹t need at-rest encryption as well.

    And as you mentioned, something needs to be done about key
    management.
    Storing information with the message about which key(s) was used is
    a good
    idea, because it allows you to know when a producer has switched
    keys.
    There are definitely some alternative solutions to that as well. But
    storing the keys in the broker, Zookeeper, or other systems like
    that are
    not. There needs to be a system used where the keys are only
    available to
    the producers and consumers that need them, and they only get access
    to
    the appropriate part of the key pair. Even as the guy running Kafka
    and
    Zookeeper, I should not have access to the keys being used, and if
    data is
    encrypted I should not be able to see the cleartext.

    And even if we decide not to put anything about at-rest encryption
    in the
    consumer/producer clients directly, and leave it for an exercise
    above
    that level (you have to pass the ciphertext as the message to the
    client),
    I still think there is a good case for implementing a message
    envelope
    that can store the information about which key was used, and other
    pertinent metadata, and have the ability for special applications
    like
    mirror maker to be able to preserve it across clusters. This still
    helps
    to enable the use of encryption and other features (like auditing)
    even if
    we decide it¹s too large a scope to fully implement.

    -Todd

    On 6/6/14, 10:51 AM, "Pradeep Gollakota" <pradeepg26@gmail.com>
    wrote:
    I'm actually not convinced that encryption needs to be handled
    server side
    in Kafka. I think the best solution for encryption is to handle it
    producer/consumer side just like compression. This will offload key
    management to the users and we'll still be able to leverage the
    sendfile
    optimization for better performance.


    On Fri, Jun 6, 2014 at 10:48 AM, Rob Withers
    <robert.w.withers@gmail.com
    wrote:
    On consideration, if we have 3 different access groups (1 for
    production
    WRITE and 2 consumers) they all need to decode the same encryption
    and
    so
    all need the same public/private key....certs won't work, unless
    you
    write
    a CertAuthority to build multiple certs with the same keys.
    Better
    seems
    to not use certs and wrap the encryption specification with an ACL
    capabilities for each group of access.


    On Jun 6, 2014, at 11:43 AM, Rob Withers wrote:

    This is quite interesting to me and it is an excelent
    opportunity to
    promote a slightly different security scheme. Object-
    capabilities are
    perfect for online security and would use ACL style
    authentication to
    gain
    capabilities filtered to those allowed resources for allow
    actions
    (READ/WRITE/DELETE/LIST/SCAN). Erights.org has the
    quitenscential (??)
    object capabilities model and capnproto is impleemting this for
    C+
    +. I
    have a java implementation at http://github.com/pauwau/pauwau but
    the
    master is broken. 0.2 works, basically. B asically a TLS
    connection
    with
    no certificate server, it is peer to peer. It has some advanced
    features,
    but the lining of capabilities with authorization so that you can
    only
    invoke correct services is extended to the secure user.

    Regarding non-repudiation, on disk, why not prepend a CRC?

    Regarding on-disk encryption, multiple users/groups may need to
    access,
    with different capabilities. Sounds like zookeeper needs to
    store a
    cert
    for each class of access so that a group member can access the
    decrypted
    data from disk. Use cert-based async decryption. The only
    isue is
    storing
    the private key in zookeeper. Perhaps some hash magic could be
    used.

    Thanks for kafka,
    Rob

    On Jun 5, 2014, at 3:01 PM, Jay Kreps wrote:

    Hey Joe,
    I don't really understand the sections you added to the wiki.
    Can you
    clarify them?

    Is non-repudiation what SASL would call integrity checks? If so
    don't
    SSL
    and and many of the SASL schemes already support this as well as
    on-the-wire encryption?

    Or are you proposing an on-disk encryption scheme? Is this
    actually
    needed?
    Isn't a on-the-wire encryption when combined with mutual
    authentication
    and
    permissions sufficient for most uses?

    On-disk encryption seems unnecessary because if an attacker can
    get
    root
    on
    the kafka boxes it can potentially modify Kafka to do anything
    he or
    she
    wants with data. So this seems to break any security model.

    I understand the problem of a large organization not really
    having a
    trusted network and wanting to secure data transfer and limit
    and
    audit
    data access. The uses for these other things I don't totally
    understand.

    Also it would be worth understanding the state of other
    messaging and
    storage systems (Hadoop, dbs, etc). What features do they
    support. I
    think
    there is a sense in which you don't have to run faster than the
    bear,
    but
    only faster then your friends. :-)

    -Jay


    On Wed, Jun 4, 2014 at 5:57 PM, Joe Stein <joe.stein@stealth.ly>
    wrote:

    I like the idea of working on the spec and prioritizing. I will
    update
    the
    wiki.

    - Joestein


    On Wed, Jun 4, 2014 at 1:11 PM, Jay Kreps <jay.kreps@gmail.com>
    wrote:

    Hey Joe,
    Thanks for kicking this discussion off! I totally agree that
    for something
    that acts as a central message broker security is critical
    feature.
    I think
    a number of people have been interested in this topic and
    several
    people
    have put effort into special purpose security efforts.

    Since most the LinkedIn folks are working on the consumer
    right now
    I think
    this would be a great project for any other interested
    people to
    take
    on.
    There are some challenges in doing these things distributed
    but it
    can also
    be a lot of fun.

    I think a good first step would be to get a written plan we
    can all
    agree
    on for how things should work. Then we can break things down
    into
    chunks
    that can be done independently while still aiming at a good
    end
    state.

    I had tried to write up some notes that summarized at least
    the
    thoughts I
    had had on security:
    https://cwiki.apache.org/confluence/display/KAFKA/Security

    What do you think of that?

    One assumption I had (which may be incorrect) is that although
    we
    want all
    the things in your list, the two most pressing would be
    authentication and
    authorization, and that was all that write up covered. You
    have more
    experience in this domain, so I wonder how you would
    prioritize?

    Those notes are really sketchy, so I think the first goal I
    would
    have
    would be to get to a real spec we can all agree on and
    discuss. A
    lot
    of
    the security stuff has a high human interaction element and
    needs to
    work
    in pretty different domains and different companies so getting
    this
    kind of
    review is important.

    -Jay


    On Tue, Jun 3, 2014 at 12:57 PM, Joe Stein
    <joe.stein@stealth.ly>
    wrote:

    Hi,I wanted to re-ignite the discussion around Apache Kafka
    Security.
    This
    is a huge bottleneck (non-starter in some cases) for a lot of
    organizations
    (due to regulatory, compliance and other requirements). Below
    are
    my
    suggestions for specific changes in Kafka to accommodate
    security
    requirements. This comes from what folks are doing "in the
    wild"
    to
    workaround and implement security with Kafka as it is today
    and
    also what I
    have discovered from organizations about their blockers. It
    also
    picks
    up
    from the wiki (which I should have time to update later in the
    week
    based
    on the below and feedback from the thread).
    1) Transport Layer Security (i.e. SSL)

    This also includes client authentication in addition to in-
    transit security
    layer. This work has been picked up here
    https://issues.apache.org/jira/browse/KAFKA-1477 and do
    appreciate
    any
    thoughts, comments, feedback, tomatoes, whatever for this
    patch.
    It
    is a
    pickup from the fork of the work first done here
    https://github.com/relango/kafka/tree/kafka_security.

    2) Data encryption at rest.

    This is very important and something that can be facilitated
    within
    the
    wire protocol. It requires an additional map data structure
    for the
    "encrypted [data encryption key]". With this map (either in
    your
    object or
    in the wire protocol) you can store the dynamically generated
    symmetric key
    (for each message) and then encrypt the data using that
    dynamically
    generated key. You then encrypt the encryption key using
    each
    public
    key
    for whom is expected to be able to decrypt the encryption
    key to
    then
    decrypt the message. For each public key encrypted symmetric
    key
    (which
    is
    now the "encrypted [data encryption key]" along with which
    public
    key
    it
    was encrypted with for (so a map of [publicKey] =
    encryptedDataEncryptionKey) as a chain. Other patterns
    can be
    implemented
    but this is a pretty standard digital enveloping [0] pattern
    with
    only
    1
    field added. Other patterns should be able to use that field
    to-do
    their
    implementation too.
    3) Non-repudiation and long term non-repudiation.

    Non-repudiation is proving data hasn't changed. This is
    often (if
    not
    always) done with x509 public certificates (chained to a
    certificate
    authority).

    Long term non-repudiation is what happens when the
    certificates of
    the
    certificate authority are expired (or revoked) and everything
    ever
    signed
    (ever) with that certificate's public key then becomes "no
    longer
    provable
    as ever being authentic". That is where RFC3126 [1] and
    RFC3161
    [2]
    come
    in (or worm drives [hardware], etc).
    For either (or both) of these it is an operation of the
    encryptor
    to
    sign/hash the data (with or without third party trusted
    timestap of
    the
    signing event) and encrypt that with their own private key
    and
    distribute
    the results (before and after encrypting if required) along
    with
    their
    public key. This structure is a bit more complex but
    feasible, it
    is a map
    of digital signature formats and the chain of dig sig
    attestations.
    The
    map's key being the method (i.e. CRC32, PKCS7 [3], XmlDigSig
    [4])
    and
    then
    a list of map where that key is "purpose" of signature (what
    your attesting
    too). As a sibling field to the list another field for "the
    attester"
    as
    bytes (e.g. their PKCS12 [5] for the map of PKCS7 signatures).
    4) Authorization

    We should have a policy of "404" for data, topics, partitions
    (etc) if
    authenticated connections do not have access. In "secure
    mode" any
    non
    authenticated connections should get a "404" type message on
    everything.
    Knowing "something is there" is a security risk in many uses
    cases.
    So
    if
    you don't have access you don't even see it. Baking "that"
    into
    Kafka
    along with some interface for entitlement (access management)
    systems
    (pretty standard) is all that I think needs to be done to the
    core project.
    I want to tackle item later in the year after summer after
    the
    other three
    are complete.

    I look forward to thoughts on this and anyone else interested
    in
    working
    with us on these items.
    [0]
    http://www.emc.com/emc-plus/rsa-labs/standards-
    initiatives/what-is-a-digital-envelope.htm
    [1] http://tools.ietf.org/html/rfc3126
    http://www.emc.com/emc-plus/rsa-labs/standards-initiatives/pkcs-7
    -
    cryptographic-message-syntax-standar.htm
    [4] http://en.wikipedia.org/wiki/XML_Signature
    [5] http://en.wikipedia.org/wiki/PKCS_12

    /*******************************************
    Joe Stein
    Founder, Principal Consultant
    Big Data Open Source Security LLC
    http://www.stealth.ly
    Twitter: @allthingshadoop
    <http://www.twitter.com/allthingshadoop
    ********************************************/
  • Todd Palino at Jun 10, 2014 at 10:02 pm
    The situation of production before having the consumer is definitely a
    good one. That’s why I wanted to take a little time before responding. Had
    to think about it.

    I think that while we may certainly produce data before the consumer is
    ready, that doesn’t mean that the consumer can’t have a key pair generated
    for it already, so the producer could start encrypting for that consumer
    before it exists. This would probably work fine for lower retention
    periods (a week or two), but could be a little more difficult to manage if
    you are keeping data in Kafka longer than that. My gut reaction is that
    it’s better to handle it that way and keep the key pair and session key
    handling simple. The more we can do that, the more we can leave key
    management as a separate component that can be swapped out so the user can
    decide how it should be done.

    -Todd

    On 6/9/14, 8:16 AM, "Robert Withers" wrote:

    Yes, that sounds familiar as I helped write (minimally) S/MIME in squeak
    (open source Smalltalk environment). This what I was thinking in my
    alternative here, though I have a concern...

    Production may occur before the consumer is coded and executed. In the
    analogy of mail, the mail is sent before the complete recipient list is
    known.

    This seems to mean that the private key (cert or OTP) must be stored and
    interacted with. My feeling is that key metadata are in a system
    encrypted Hbase store (session key store), for low latency reads, rather
    than a topic requiring scanning. Store the private keys and then give
    client access (producers/consumers) with the hash of the OTP. A new
    consumer comes along, create a new cert encoding the OTP hash.

    On write, use the producer cert to send a topic hash with the msg which
    would allow the broker to reuse or generate an OTP, stored in the session
    key store.

    On read (consumer), if we have a previously run reader, use the encrypted
    hash. If new, create consumer cert and encrypt the hash for that session.

    The reader/writer will pass a cert encrypted session hash. The trick
    seems to be converting hash to PK to encrypt/decrypt. Given Kafka
    resource distribution, we need system encryption for metadata and
    cert-based key exchange. This seems to mean triple encryption:
    1) client to/from broker
    2) system key/hash mgmt/translation
    3) at-rest encryption

    Thanks,
    Rob
    On Jun 9, 2014, at 7:57 AM, Todd Palino <tpalino@linkedin.com.invalid>
    wrote:

    It’s the same method used by S/MIME and many other encryption
    specifications with the potential for multiple recipients. The sender
    generates a session key, and uses that key to encrypt the message. The
    session key is then encrypted once for each recipient with that
    recipient’s public key. All of the encrypted copies of the session key
    are
    then included with the encrypted message. This way, you avoid having to
    encrypt the message multiple times (this assumes, of course, that the
    message itself is larger than the key).

    In our case, we have some options available to us. We could do that, and
    put all the encrypted keys in the message metadata. Or we could treat it
    more like a session and have the encrypted session keys in a special
    topic
    (e.g. __session_keys), much like offsets are now. When the producer
    starts
    up, they create a session key and encrypt it for each consumer with the
    current consumer key. The producer publishes the bundle of encrypted
    keys
    into __session_keys as a single message. The producer then publishes
    messages to the normal topic encrypted with the session key. The
    metadata
    for each of those messages would contain something the offset into
    __session_keys to identify the bundle. This has the added benefit of not
    increasing the per-message data size too much.

    Whenever a consumer key is invalidated, or however often the session key
    should be rotated, the producer would publish a new bundle. This
    maintains
    a history of session keys that can be used to decrypt any messages, so
    the
    retention on __session_keys must be at least as long as any topic which
    may potentially contain encrypted data. Past that point, it’s up to the
    consumer what they want to do with the data. A consumer like Hadoop
    might
    re-encrypt it for local storage, or store it in plaintext (depending on
    the security and requirements of that system).

    -Todd
    On 6/8/14, 2:33 PM, "Rob Withers" wrote:

    I like the use of meta envelopes. We did this recently, on the job,
    as we have an envelope that specifies the type for decoding. We
    discussed adding the encodinType and you are suggesting adding
    encryption metadata for that msg. All good.

    I don't see your OTP example. Could you delve deeper for me, please?
    The model I envision is internal OTP, with access to decryption
    accessed by cert. A double layer of security, with the internal at-
    rest encryption being an unchanging OTP with ACL access to it as the
    upper layer. Are you saying it is possible to re-encrypt with new
    keys or that there is a chain of keys over time?

    Thanks,
    Rob
    On Jun 8, 2014, at 3:06 PM, Todd Palino wrote:

    I’ll agree that perhaps the “absolutely not” is not quite right.
    There are
    certainly some uses for a simpler solution, but I would still say it
    cannot only be encryption at the broker. This would leave many use
    cases
    for at-rest encryption out of the loop (most auditing cases for SOX,
    PCI,
    HIPAA, and other PII standards). Yes, it does add external overhead
    that
    must be managed, but it’s just the nature of the beast. We can’t
    solve all
    of the external infrastructure needed for this, but we can make it
    easier
    to use for consumers and producers by adding metadata.

    There’s no need for unchanging encryption, and that’s specifically
    why I
    want to see a message envelope that will help consumers determine the
    encryption uses for a particular message. You can definitely still
    expire
    keys, you just have to keep the expired keys around as long as the
    encrypted data stays around, and your endpoints need to know when
    they are
    decrypting data with an expired key (you might want to throw up a
    warning,
    or do something else to let the users know that it’s happening). And
    as
    someone else mentioned, there are solutions for encrypting data for
    multiple consumers. You can encrypt the data with an OTP, and then
    multiply encrypt the OTP once for each consumer and store those
    encrypted
    strings in the envelope.

    -Todd
    On 6/7/14, 12:25 PM, "Rob Withers" <robert.w.withers@gmail.com>
    wrote:

    At one level this makes sense to me to externalize the security issue
    to producers and consumers. On consideration I realized that this
    adds a lot of coordination requirements to the app layer across teams
    or even companies. Another issue I feel is that you want a specific
    unchanging encryption for the data and the clients (producers/
    consumers) will need to be able to decode frozen data. If certs are
    used they cannot expire. Also, different clients would need to use
    the same cert.

    So, you statement that it should ABSOLUTELY not include internal
    encryption rings seems misplaced. There are some customers of kafka
    that would opt to encrypt the on-disk data and key management is a
    significant issue. This is best handled internally, with key
    management stored in either ZK or in a topic. Truly, perhaps
    annealing Hadoop/HBASE as a metadata store seems applicable.

    Thanks, another 2 cents,
    Rob
    On Jun 6, 2014, at 12:15 PM, Todd Palino wrote:

    Yes, I realized last night that I needed to be clearer in what I was
    saying. Encryption should ABSOLUTELY not be handled server-side. I
    think
    it¹s a good idea to enable use of it in the consumer/producer, but
    doing
    it server side will not solve many use cases for needing encryption
    because the server then has access to all the keys. You could say
    that
    this eliminates the need for TLS, but TLS is pretty low-hanging
    fruit, and
    there¹s definitely a need for encryption of the traffic across the
    network
    even if you don¹t need at-rest encryption as well.

    And as you mentioned, something needs to be done about key
    management.
    Storing information with the message about which key(s) was used is
    a good
    idea, because it allows you to know when a producer has switched
    keys.
    There are definitely some alternative solutions to that as well. But
    storing the keys in the broker, Zookeeper, or other systems like
    that are
    not. There needs to be a system used where the keys are only
    available to
    the producers and consumers that need them, and they only get access
    to
    the appropriate part of the key pair. Even as the guy running Kafka
    and
    Zookeeper, I should not have access to the keys being used, and if
    data is
    encrypted I should not be able to see the cleartext.

    And even if we decide not to put anything about at-rest encryption
    in the
    consumer/producer clients directly, and leave it for an exercise
    above
    that level (you have to pass the ciphertext as the message to the
    client),
    I still think there is a good case for implementing a message
    envelope
    that can store the information about which key was used, and other
    pertinent metadata, and have the ability for special applications
    like
    mirror maker to be able to preserve it across clusters. This still
    helps
    to enable the use of encryption and other features (like auditing)
    even if
    we decide it¹s too large a scope to fully implement.

    -Todd

    On 6/6/14, 10:51 AM, "Pradeep Gollakota" <pradeepg26@gmail.com>
    wrote:
    I'm actually not convinced that encryption needs to be handled
    server side
    in Kafka. I think the best solution for encryption is to handle it
    producer/consumer side just like compression. This will offload key
    management to the users and we'll still be able to leverage the
    sendfile
    optimization for better performance.


    On Fri, Jun 6, 2014 at 10:48 AM, Rob Withers
    <robert.w.withers@gmail.com
    wrote:
    On consideration, if we have 3 different access groups (1 for
    production
    WRITE and 2 consumers) they all need to decode the same encryption
    and
    so
    all need the same public/private key....certs won't work, unless
    you
    write
    a CertAuthority to build multiple certs with the same keys.
    Better
    seems
    to not use certs and wrap the encryption specification with an ACL
    capabilities for each group of access.


    On Jun 6, 2014, at 11:43 AM, Rob Withers wrote:

    This is quite interesting to me and it is an excelent
    opportunity to
    promote a slightly different security scheme. Object-
    capabilities are
    perfect for online security and would use ACL style
    authentication to
    gain
    capabilities filtered to those allowed resources for allow
    actions
    (READ/WRITE/DELETE/LIST/SCAN). Erights.org has the
    quitenscential (??)
    object capabilities model and capnproto is impleemting this for
    C+
    +. I
    have a java implementation at http://github.com/pauwau/pauwau but
    the
    master is broken. 0.2 works, basically. B asically a TLS
    connection
    with
    no certificate server, it is peer to peer. It has some advanced
    features,
    but the lining of capabilities with authorization so that you can
    only
    invoke correct services is extended to the secure user.

    Regarding non-repudiation, on disk, why not prepend a CRC?

    Regarding on-disk encryption, multiple users/groups may need to
    access,
    with different capabilities. Sounds like zookeeper needs to
    store a
    cert
    for each class of access so that a group member can access the
    decrypted
    data from disk. Use cert-based async decryption. The only
    isue is
    storing
    the private key in zookeeper. Perhaps some hash magic could be
    used.

    Thanks for kafka,
    Rob

    On Jun 5, 2014, at 3:01 PM, Jay Kreps wrote:

    Hey Joe,
    I don't really understand the sections you added to the wiki.
    Can you
    clarify them?

    Is non-repudiation what SASL would call integrity checks? If so
    don't
    SSL
    and and many of the SASL schemes already support this as well as
    on-the-wire encryption?

    Or are you proposing an on-disk encryption scheme? Is this
    actually
    needed?
    Isn't a on-the-wire encryption when combined with mutual
    authentication
    and
    permissions sufficient for most uses?

    On-disk encryption seems unnecessary because if an attacker can
    get
    root
    on
    the kafka boxes it can potentially modify Kafka to do anything
    he or
    she
    wants with data. So this seems to break any security model.

    I understand the problem of a large organization not really
    having a
    trusted network and wanting to secure data transfer and limit
    and
    audit
    data access. The uses for these other things I don't totally
    understand.

    Also it would be worth understanding the state of other
    messaging and
    storage systems (Hadoop, dbs, etc). What features do they
    support. I
    think
    there is a sense in which you don't have to run faster than the
    bear,
    but
    only faster then your friends. :-)

    -Jay


    On Wed, Jun 4, 2014 at 5:57 PM, Joe Stein <joe.stein@stealth.ly>
    wrote:

    I like the idea of working on the spec and prioritizing. I will
    update
    the
    wiki.

    - Joestein


    On Wed, Jun 4, 2014 at 1:11 PM, Jay Kreps <jay.kreps@gmail.com>
    wrote:

    Hey Joe,
    Thanks for kicking this discussion off! I totally agree that
    for something
    that acts as a central message broker security is critical
    feature.
    I think
    a number of people have been interested in this topic and
    several
    people
    have put effort into special purpose security efforts.

    Since most the LinkedIn folks are working on the consumer
    right now
    I think
    this would be a great project for any other interested
    people to
    take
    on.
    There are some challenges in doing these things distributed
    but it
    can also
    be a lot of fun.

    I think a good first step would be to get a written plan we
    can all
    agree
    on for how things should work. Then we can break things down
    into
    chunks
    that can be done independently while still aiming at a good
    end
    state.

    I had tried to write up some notes that summarized at least
    the
    thoughts I
    had had on security:
    https://cwiki.apache.org/confluence/display/KAFKA/Security

    What do you think of that?

    One assumption I had (which may be incorrect) is that although
    we
    want all
    the things in your list, the two most pressing would be
    authentication and
    authorization, and that was all that write up covered. You
    have more
    experience in this domain, so I wonder how you would
    prioritize?

    Those notes are really sketchy, so I think the first goal I
    would
    have
    would be to get to a real spec we can all agree on and
    discuss. A
    lot
    of
    the security stuff has a high human interaction element and
    needs to
    work
    in pretty different domains and different companies so getting
    this
    kind of
    review is important.

    -Jay


    On Tue, Jun 3, 2014 at 12:57 PM, Joe Stein
    <joe.stein@stealth.ly>
    wrote:

    Hi,I wanted to re-ignite the discussion around Apache Kafka
    Security.
    This
    is a huge bottleneck (non-starter in some cases) for a lot of
    organizations
    (due to regulatory, compliance and other requirements). Below
    are
    my
    suggestions for specific changes in Kafka to accommodate
    security
    requirements. This comes from what folks are doing "in the
    wild"
    to
    workaround and implement security with Kafka as it is today
    and
    also what I
    have discovered from organizations about their blockers. It
    also
    picks
    up
    from the wiki (which I should have time to update later in the
    week
    based
    on the below and feedback from the thread).
    1) Transport Layer Security (i.e. SSL)

    This also includes client authentication in addition to in-
    transit security
    layer. This work has been picked up here
    https://issues.apache.org/jira/browse/KAFKA-1477 and do
    appreciate
    any
    thoughts, comments, feedback, tomatoes, whatever for this
    patch.
    It
    is a
    pickup from the fork of the work first done here
    https://github.com/relango/kafka/tree/kafka_security.

    2) Data encryption at rest.

    This is very important and something that can be facilitated
    within
    the
    wire protocol. It requires an additional map data structure
    for the
    "encrypted [data encryption key]". With this map (either in
    your
    object or
    in the wire protocol) you can store the dynamically generated
    symmetric key
    (for each message) and then encrypt the data using that
    dynamically
    generated key. You then encrypt the encryption key using
    each
    public
    key
    for whom is expected to be able to decrypt the encryption
    key to
    then
    decrypt the message. For each public key encrypted symmetric
    key
    (which
    is
    now the "encrypted [data encryption key]" along with which
    public
    key
    it
    was encrypted with for (so a map of [publicKey] =
    encryptedDataEncryptionKey) as a chain. Other patterns
    can be
    implemented
    but this is a pretty standard digital enveloping [0] pattern
    with
    only
    1
    field added. Other patterns should be able to use that field
    to-do
    their
    implementation too.
    3) Non-repudiation and long term non-repudiation.

    Non-repudiation is proving data hasn't changed. This is
    often (if
    not
    always) done with x509 public certificates (chained to a
    certificate
    authority).

    Long term non-repudiation is what happens when the
    certificates of
    the
    certificate authority are expired (or revoked) and everything
    ever
    signed
    (ever) with that certificate's public key then becomes "no
    longer
    provable
    as ever being authentic". That is where RFC3126 [1] and
    RFC3161
    [2]
    come
    in (or worm drives [hardware], etc).
    For either (or both) of these it is an operation of the
    encryptor
    to
    sign/hash the data (with or without third party trusted
    timestap of
    the
    signing event) and encrypt that with their own private key
    and
    distribute
    the results (before and after encrypting if required) along
    with
    their
    public key. This structure is a bit more complex but
    feasible, it
    is a map
    of digital signature formats and the chain of dig sig
    attestations.
    The
    map's key being the method (i.e. CRC32, PKCS7 [3], XmlDigSig
    [4])
    and
    then
    a list of map where that key is "purpose" of signature (what
    your attesting
    too). As a sibling field to the list another field for "the
    attester"
    as
    bytes (e.g. their PKCS12 [5] for the map of PKCS7 signatures).
    4) Authorization

    We should have a policy of "404" for data, topics, partitions
    (etc) if
    authenticated connections do not have access. In "secure
    mode" any
    non
    authenticated connections should get a "404" type message on
    everything.
    Knowing "something is there" is a security risk in many uses
    cases.
    So
    if
    you don't have access you don't even see it. Baking "that"
    into
    Kafka
    along with some interface for entitlement (access management)
    systems
    (pretty standard) is all that I think needs to be done to the
    core project.
    I want to tackle item later in the year after summer after
    the
    other three
    are complete.

    I look forward to thoughts on this and anyone else interested
    in
    working
    with us on these items.
    [0]
    http://www.emc.com/emc-plus/rsa-labs/standards-
    initiatives/what-is-a-digital-envelope.htm
    [1] http://tools.ietf.org/html/rfc3126

    http://www.emc.com/emc-plus/rsa-labs/standards-initiatives/pkcs
    -7
    -
    cryptographic-message-syntax-standar.htm
    [4] http://en.wikipedia.org/wiki/XML_Signature
    [5] http://en.wikipedia.org/wiki/PKCS_12

    /*******************************************
    Joe Stein
    Founder, Principal Consultant
    Big Data Open Source Security LLC
    http://www.stealth.ly
    Twitter: @allthingshadoop
    <http://www.twitter.com/allthingshadoop
    ********************************************/
  • Robert Withers at Jun 10, 2014 at 10:27 pm
    What strikes me as an opportunity is to define a plug gable at-rest encryption module interface, that supports each/both of our security needs.

    Thanks,
    Rob
    On Jun 10, 2014, at 4:01 PM, Todd Palino wrote:

    The situation of production before having the consumer is definitely a
    good one. That’s why I wanted to take a little time before responding. Had
    to think about it.

    I think that while we may certainly produce data before the consumer is
    ready, that doesn’t mean that the consumer can’t have a key pair generated
    for it already, so the producer could start encrypting for that consumer
    before it exists. This would probably work fine for lower retention
    periods (a week or two), but could be a little more difficult to manage if
    you are keeping data in Kafka longer than that. My gut reaction is that
    it’s better to handle it that way and keep the key pair and session key
    handling simple. The more we can do that, the more we can leave key
    management as a separate component that can be swapped out so the user can
    decide how it should be done.

    -Todd

    On 6/9/14, 8:16 AM, "Robert Withers" wrote:

    Yes, that sounds familiar as I helped write (minimally) S/MIME in squeak
    (open source Smalltalk environment). This what I was thinking in my
    alternative here, though I have a concern...

    Production may occur before the consumer is coded and executed. In the
    analogy of mail, the mail is sent before the complete recipient list is
    known.

    This seems to mean that the private key (cert or OTP) must be stored and
    interacted with. My feeling is that key metadata are in a system
    encrypted Hbase store (session key store), for low latency reads, rather
    than a topic requiring scanning. Store the private keys and then give
    client access (producers/consumers) with the hash of the OTP. A new
    consumer comes along, create a new cert encoding the OTP hash.

    On write, use the producer cert to send a topic hash with the msg which
    would allow the broker to reuse or generate an OTP, stored in the session
    key store.

    On read (consumer), if we have a previously run reader, use the encrypted
    hash. If new, create consumer cert and encrypt the hash for that session.

    The reader/writer will pass a cert encrypted session hash. The trick
    seems to be converting hash to PK to encrypt/decrypt. Given Kafka
    resource distribution, we need system encryption for metadata and
    cert-based key exchange. This seems to mean triple encryption:
    1) client to/from broker
    2) system key/hash mgmt/translation
    3) at-rest encryption

    Thanks,
    Rob
    On Jun 9, 2014, at 7:57 AM, Todd Palino <tpalino@linkedin.com.invalid>
    wrote:

    It’s the same method used by S/MIME and many other encryption
    specifications with the potential for multiple recipients. The sender
    generates a session key, and uses that key to encrypt the message. The
    session key is then encrypted once for each recipient with that
    recipient’s public key. All of the encrypted copies of the session key
    are
    then included with the encrypted message. This way, you avoid having to
    encrypt the message multiple times (this assumes, of course, that the
    message itself is larger than the key).

    In our case, we have some options available to us. We could do that, and
    put all the encrypted keys in the message metadata. Or we could treat it
    more like a session and have the encrypted session keys in a special
    topic
    (e.g. __session_keys), much like offsets are now. When the producer
    starts
    up, they create a session key and encrypt it for each consumer with the
    current consumer key. The producer publishes the bundle of encrypted
    keys
    into __session_keys as a single message. The producer then publishes
    messages to the normal topic encrypted with the session key. The
    metadata
    for each of those messages would contain something the offset into
    __session_keys to identify the bundle. This has the added benefit of not
    increasing the per-message data size too much.

    Whenever a consumer key is invalidated, or however often the session key
    should be rotated, the producer would publish a new bundle. This
    maintains
    a history of session keys that can be used to decrypt any messages, so
    the
    retention on __session_keys must be at least as long as any topic which
    may potentially contain encrypted data. Past that point, it’s up to the
    consumer what they want to do with the data. A consumer like Hadoop
    might
    re-encrypt it for local storage, or store it in plaintext (depending on
    the security and requirements of that system).

    -Todd
    On 6/8/14, 2:33 PM, "Rob Withers" wrote:

    I like the use of meta envelopes. We did this recently, on the job,
    as we have an envelope that specifies the type for decoding. We
    discussed adding the encodinType and you are suggesting adding
    encryption metadata for that msg. All good.

    I don't see your OTP example. Could you delve deeper for me, please?
    The model I envision is internal OTP, with access to decryption
    accessed by cert. A double layer of security, with the internal at-
    rest encryption being an unchanging OTP with ACL access to it as the
    upper layer. Are you saying it is possible to re-encrypt with new
    keys or that there is a chain of keys over time?

    Thanks,
    Rob
    On Jun 8, 2014, at 3:06 PM, Todd Palino wrote:

    I’ll agree that perhaps the “absolutely not” is not quite right.
    There are
    certainly some uses for a simpler solution, but I would still say it
    cannot only be encryption at the broker. This would leave many use
    cases
    for at-rest encryption out of the loop (most auditing cases for SOX,
    PCI,
    HIPAA, and other PII standards). Yes, it does add external overhead
    that
    must be managed, but it’s just the nature of the beast. We can’t
    solve all
    of the external infrastructure needed for this, but we can make it
    easier
    to use for consumers and producers by adding metadata.

    There’s no need for unchanging encryption, and that’s specifically
    why I
    want to see a message envelope that will help consumers determine the
    encryption uses for a particular message. You can definitely still
    expire
    keys, you just have to keep the expired keys around as long as the
    encrypted data stays around, and your endpoints need to know when
    they are
    decrypting data with an expired key (you might want to throw up a
    warning,
    or do something else to let the users know that it’s happening). And
    as
    someone else mentioned, there are solutions for encrypting data for
    multiple consumers. You can encrypt the data with an OTP, and then
    multiply encrypt the OTP once for each consumer and store those
    encrypted
    strings in the envelope.

    -Todd
    On 6/7/14, 12:25 PM, "Rob Withers" <robert.w.withers@gmail.com>
    wrote:

    At one level this makes sense to me to externalize the security issue
    to producers and consumers. On consideration I realized that this
    adds a lot of coordination requirements to the app layer across teams
    or even companies. Another issue I feel is that you want a specific
    unchanging encryption for the data and the clients (producers/
    consumers) will need to be able to decode frozen data. If certs are
    used they cannot expire. Also, different clients would need to use
    the same cert.

    So, you statement that it should ABSOLUTELY not include internal
    encryption rings seems misplaced. There are some customers of kafka
    that would opt to encrypt the on-disk data and key management is a
    significant issue. This is best handled internally, with key
    management stored in either ZK or in a topic. Truly, perhaps
    annealing Hadoop/HBASE as a metadata store seems applicable.

    Thanks, another 2 cents,
    Rob
    On Jun 6, 2014, at 12:15 PM, Todd Palino wrote:

    Yes, I realized last night that I needed to be clearer in what I was
    saying. Encryption should ABSOLUTELY not be handled server-side. I
    think
    it¹s a good idea to enable use of it in the consumer/producer, but
    doing
    it server side will not solve many use cases for needing encryption
    because the server then has access to all the keys. You could say
    that
    this eliminates the need for TLS, but TLS is pretty low-hanging
    fruit, and
    there¹s definitely a need for encryption of the traffic across the
    network
    even if you don¹t need at-rest encryption as well.

    And as you mentioned, something needs to be done about key
    management.
    Storing information with the message about which key(s) was used is
    a good
    idea, because it allows you to know when a producer has switched
    keys.
    There are definitely some alternative solutions to that as well. But
    storing the keys in the broker, Zookeeper, or other systems like
    that are
    not. There needs to be a system used where the keys are only
    available to
    the producers and consumers that need them, and they only get access
    to
    the appropriate part of the key pair. Even as the guy running Kafka
    and
    Zookeeper, I should not have access to the keys being used, and if
    data is
    encrypted I should not be able to see the cleartext.

    And even if we decide not to put anything about at-rest encryption
    in the
    consumer/producer clients directly, and leave it for an exercise
    above
    that level (you have to pass the ciphertext as the message to the
    client),
    I still think there is a good case for implementing a message
    envelope
    that can store the information about which key was used, and other
    pertinent metadata, and have the ability for special applications
    like
    mirror maker to be able to preserve it across clusters. This still
    helps
    to enable the use of encryption and other features (like auditing)
    even if
    we decide it¹s too large a scope to fully implement.

    -Todd

    On 6/6/14, 10:51 AM, "Pradeep Gollakota" <pradeepg26@gmail.com>
    wrote:
    I'm actually not convinced that encryption needs to be handled
    server side
    in Kafka. I think the best solution for encryption is to handle it
    producer/consumer side just like compression. This will offload key
    management to the users and we'll still be able to leverage the
    sendfile
    optimization for better performance.


    On Fri, Jun 6, 2014 at 10:48 AM, Rob Withers
    <robert.w.withers@gmail.com
    wrote:
    On consideration, if we have 3 different access groups (1 for
    production
    WRITE and 2 consumers) they all need to decode the same encryption
    and
    so
    all need the same public/private key....certs won't work, unless
    you
    write
    a CertAuthority to build multiple certs with the same keys.
    Better
    seems
    to not use certs and wrap the encryption specification with an ACL
    capabilities for each group of access.


    On Jun 6, 2014, at 11:43 AM, Rob Withers wrote:

    This is quite interesting to me and it is an excelent
    opportunity to
    promote a slightly different security scheme. Object-
    capabilities are
    perfect for online security and would use ACL style
    authentication to
    gain
    capabilities filtered to those allowed resources for allow
    actions
    (READ/WRITE/DELETE/LIST/SCAN). Erights.org has the
    quitenscential (??)
    object capabilities model and capnproto is impleemting this for
    C+
    +. I
    have a java implementation at http://github.com/pauwau/pauwau but
    the
    master is broken. 0.2 works, basically. B asically a TLS
    connection
    with
    no certificate server, it is peer to peer. It has some advanced
    features,
    but the lining of capabilities with authorization so that you can
    only
    invoke correct services is extended to the secure user.

    Regarding non-repudiation, on disk, why not prepend a CRC?

    Regarding on-disk encryption, multiple users/groups may need to
    access,
    with different capabilities. Sounds like zookeeper needs to
    store a
    cert
    for each class of access so that a group member can access the
    decrypted
    data from disk. Use cert-based async decryption. The only
    isue is
    storing
    the private key in zookeeper. Perhaps some hash magic could be
    used.

    Thanks for kafka,
    Rob

    On Jun 5, 2014, at 3:01 PM, Jay Kreps wrote:

    Hey Joe,
    I don't really understand the sections you added to the wiki.
    Can you
    clarify them?

    Is non-repudiation what SASL would call integrity checks? If so
    don't
    SSL
    and and many of the SASL schemes already support this as well as
    on-the-wire encryption?

    Or are you proposing an on-disk encryption scheme? Is this
    actually
    needed?
    Isn't a on-the-wire encryption when combined with mutual
    authentication
    and
    permissions sufficient for most uses?

    On-disk encryption seems unnecessary because if an attacker can
    get
    root
    on
    the kafka boxes it can potentially modify Kafka to do anything
    he or
    she
    wants with data. So this seems to break any security model.

    I understand the problem of a large organization not really
    having a
    trusted network and wanting to secure data transfer and limit
    and
    audit
    data access. The uses for these other things I don't totally
    understand.

    Also it would be worth understanding the state of other
    messaging and
    storage systems (Hadoop, dbs, etc). What features do they
    support. I
    think
    there is a sense in which you don't have to run faster than the
    bear,
    but
    only faster then your friends. :-)

    -Jay


    On Wed, Jun 4, 2014 at 5:57 PM, Joe Stein <joe.stein@stealth.ly>
    wrote:

    I like the idea of working on the spec and prioritizing. I will
    update
    the
    wiki.

    - Joestein


    On Wed, Jun 4, 2014 at 1:11 PM, Jay Kreps <jay.kreps@gmail.com>
    wrote:

    Hey Joe,
    Thanks for kicking this discussion off! I totally agree that
    for something
    that acts as a central message broker security is critical
    feature.
    I think
    a number of people have been interested in this topic and
    several
    people
    have put effort into special purpose security efforts.

    Since most the LinkedIn folks are working on the consumer
    right now
    I think
    this would be a great project for any other interested
    people to
    take
    on.
    There are some challenges in doing these things distributed
    but it
    can also
    be a lot of fun.

    I think a good first step would be to get a written plan we
    can all
    agree
    on for how things should work. Then we can break things down
    into
    chunks
    that can be done independently while still aiming at a good
    end
    state.

    I had tried to write up some notes that summarized at least
    the
    thoughts I
    had had on security:
    https://cwiki.apache.org/confluence/display/KAFKA/Security

    What do you think of that?

    One assumption I had (which may be incorrect) is that although
    we
    want all
    the things in your list, the two most pressing would be
    authentication and
    authorization, and that was all that write up covered. You
    have more
    experience in this domain, so I wonder how you would
    prioritize?

    Those notes are really sketchy, so I think the first goal I
    would
    have
    would be to get to a real spec we can all agree on and
    discuss. A
    lot
    of
    the security stuff has a high human interaction element and
    needs to
    work
    in pretty different domains and different companies so getting
    this
    kind of
    review is important.

    -Jay


    On Tue, Jun 3, 2014 at 12:57 PM, Joe Stein
    <joe.stein@stealth.ly>
    wrote:

    Hi,I wanted to re-ignite the discussion around Apache Kafka
    Security.
    This
    is a huge bottleneck (non-starter in some cases) for a lot of
    organizations
    (due to regulatory, compliance and other requirements). Below
    are
    my
    suggestions for specific changes in Kafka to accommodate
    security
    requirements. This comes from what folks are doing "in the
    wild"
    to
    workaround and implement security with Kafka as it is today
    and
    also what I
    have discovered from organizations about their blockers. It
    also
    picks
    up
    from the wiki (which I should have time to update later in the
    week
    based
    on the below and feedback from the thread).
    1) Transport Layer Security (i.e. SSL)

    This also includes client authentication in addition to in-
    transit security
    layer. This work has been picked up here
    https://issues.apache.org/jira/browse/KAFKA-1477 and do
    appreciate
    any
    thoughts, comments, feedback, tomatoes, whatever for this
    patch.
    It
    is a
    pickup from the fork of the work first done here
    https://github.com/relango/kafka/tree/kafka_security.

    2) Data encryption at rest.

    This is very important and something that can be facilitated
    within
    the
    wire protocol. It requires an additional map data structure
    for the
    "encrypted [data encryption key]". With this map (either in
    your
    object or
    in the wire protocol) you can store the dynamically generated
    symmetric key
    (for each message) and then encrypt the data using that
    dynamically
    generated key. You then encrypt the encryption key using
    each
    public
    key
    for whom is expected to be able to decrypt the encryption
    key to
    then
    decrypt the message. For each public key encrypted symmetric
    key
    (which
    is
    now the "encrypted [data encryption key]" along with which
    public
    key
    it
    was encrypted with for (so a map of [publicKey] =
    encryptedDataEncryptionKey) as a chain. Other patterns
    can be
    implemented
    but this is a pretty standard digital enveloping [0] pattern
    with
    only
    1
    field added. Other patterns should be able to use that field
    to-do
    their
    implementation too.
    3) Non-repudiation and long term non-repudiation.

    Non-repudiation is proving data hasn't changed. This is
    often (if
    not
    always) done with x509 public certificates (chained to a
    certificate
    authority).

    Long term non-repudiation is what happens when the
    certificates of
    the
    certificate authority are expired (or revoked) and everything
    ever
    signed
    (ever) with that certificate's public key then becomes "no
    longer
    provable
    as ever being authentic". That is where RFC3126 [1] and
    RFC3161
    [2]
    come
    in (or worm drives [hardware], etc).
    For either (or both) of these it is an operation of the
    encryptor
    to
    sign/hash the data (with or without third party trusted
    timestap of
    the
    signing event) and encrypt that with their own private key
    and
    distribute
    the results (before and after encrypting if required) along
    with
    their
    public key. This structure is a bit more complex but
    feasible, it
    is a map
    of digital signature formats and the chain of dig sig
    attestations.
    The
    map's key being the method (i.e. CRC32, PKCS7 [3], XmlDigSig
    [4])
    and
    then
    a list of map where that key is "purpose" of signature (what
    your attesting
    too). As a sibling field to the list another field for "the
    attester"
    as
    bytes (e.g. their PKCS12 [5] for the map of PKCS7 signatures).
    4) Authorization

    We should have a policy of "404" for data, topics, partitions
    (etc) if
    authenticated connections do not have access. In "secure
    mode" any
    non
    authenticated connections should get a "404" type message on
    everything.
    Knowing "something is there" is a security risk in many uses
    cases.
    So
    if
    you don't have access you don't even see it. Baking "that"
    into
    Kafka
    along with some interface for entitlement (access management)
    systems
    (pretty standard) is all that I think needs to be done to the
    core project.
    I want to tackle item later in the year after summer after
    the
    other three
    are complete.

    I look forward to thoughts on this and anyone else interested
    in
    working
    with us on these items.
    [0]
    http://www.emc.com/emc-plus/rsa-labs/standards-
    initiatives/what-is-a-digital-envelope.htm
    [1] http://tools.ietf.org/html/rfc3126

    http://www.emc.com/emc-plus/rsa-labs/standards-initiatives/pkcs
    -7
    -
    cryptographic-message-syntax-standar.htm
    [4] http://en.wikipedia.org/wiki/XML_Signature
    [5] http://en.wikipedia.org/wiki/PKCS_12

    /*******************************************
    Joe Stein
    Founder, Principal Consultant
    Big Data Open Source Security LLC
    http://www.stealth.ly
    Twitter: @allthingshadoop
    <http://www.twitter.com/allthingshadoop
    ********************************************/
  • Todd Palino at Jun 10, 2014 at 10:38 pm
    Yes, I agree. There are definitely a variety of use cases that demand
    differing levels of complexity here. It comes back to enabling the
    development of at-rest encryption and making it as easy as possible to
    implement within the Kafka system. I think that this can be done with the
    concept of message metadata that can be preserved across clusters, which
    is separate from the message itself (so it’s not an overlay of a schema on
    top of the message, but rather a separate structure entirely that is
    stored with the message).

    -Todd
    On 6/10/14, 3:26 PM, "Robert Withers" wrote:

    What strikes me as an opportunity is to define a plug gable at-rest
    encryption module interface, that supports each/both of our security
    needs.

    Thanks,
    Rob
    On Jun 10, 2014, at 4:01 PM, Todd Palino <tpalino@linkedin.com.invalid>
    wrote:

    The situation of production before having the consumer is definitely a
    good one. That’s why I wanted to take a little time before responding.
    Had
    to think about it.

    I think that while we may certainly produce data before the consumer is
    ready, that doesn’t mean that the consumer can’t have a key pair
    generated
    for it already, so the producer could start encrypting for that consumer
    before it exists. This would probably work fine for lower retention
    periods (a week or two), but could be a little more difficult to manage
    if
    you are keeping data in Kafka longer than that. My gut reaction is that
    it’s better to handle it that way and keep the key pair and session key
    handling simple. The more we can do that, the more we can leave key
    management as a separate component that can be swapped out so the user
    can
    decide how it should be done.

    -Todd

    On 6/9/14, 8:16 AM, "Robert Withers" <robert.w.withers@gmail.com>
    wrote:

    Yes, that sounds familiar as I helped write (minimally) S/MIME in
    squeak
    (open source Smalltalk environment). This what I was thinking in my
    alternative here, though I have a concern...

    Production may occur before the consumer is coded and executed. In the
    analogy of mail, the mail is sent before the complete recipient list is
    known.

    This seems to mean that the private key (cert or OTP) must be stored
    and
    interacted with. My feeling is that key metadata are in a system
    encrypted Hbase store (session key store), for low latency reads,
    rather
    than a topic requiring scanning. Store the private keys and then give
    client access (producers/consumers) with the hash of the OTP. A new
    consumer comes along, create a new cert encoding the OTP hash.

    On write, use the producer cert to send a topic hash with the msg which
    would allow the broker to reuse or generate an OTP, stored in the
    session
    key store.

    On read (consumer), if we have a previously run reader, use the
    encrypted
    hash. If new, create consumer cert and encrypt the hash for that
    session.

    The reader/writer will pass a cert encrypted session hash. The trick
    seems to be converting hash to PK to encrypt/decrypt. Given Kafka
    resource distribution, we need system encryption for metadata and
    cert-based key exchange. This seems to mean triple encryption:
    1) client to/from broker
    2) system key/hash mgmt/translation
    3) at-rest encryption

    Thanks,
    Rob
    On Jun 9, 2014, at 7:57 AM, Todd Palino <tpalino@linkedin.com.invalid>
    wrote:

    It’s the same method used by S/MIME and many other encryption
    specifications with the potential for multiple recipients. The sender
    generates a session key, and uses that key to encrypt the message. The
    session key is then encrypted once for each recipient with that
    recipient’s public key. All of the encrypted copies of the session key
    are
    then included with the encrypted message. This way, you avoid having
    to
    encrypt the message multiple times (this assumes, of course, that the
    message itself is larger than the key).

    In our case, we have some options available to us. We could do that,
    and
    put all the encrypted keys in the message metadata. Or we could treat
    it
    more like a session and have the encrypted session keys in a special
    topic
    (e.g. __session_keys), much like offsets are now. When the producer
    starts
    up, they create a session key and encrypt it for each consumer with
    the
    current consumer key. The producer publishes the bundle of encrypted
    keys
    into __session_keys as a single message. The producer then publishes
    messages to the normal topic encrypted with the session key. The
    metadata
    for each of those messages would contain something the offset into
    __session_keys to identify the bundle. This has the added benefit of
    not
    increasing the per-message data size too much.

    Whenever a consumer key is invalidated, or however often the session
    key
    should be rotated, the producer would publish a new bundle. This
    maintains
    a history of session keys that can be used to decrypt any messages, so
    the
    retention on __session_keys must be at least as long as any topic
    which
    may potentially contain encrypted data. Past that point, it’s up to
    the
    consumer what they want to do with the data. A consumer like Hadoop
    might
    re-encrypt it for local storage, or store it in plaintext (depending
    on
    the security and requirements of that system).

    -Todd
    On 6/8/14, 2:33 PM, "Rob Withers" wrote:

    I like the use of meta envelopes. We did this recently, on the job,
    as we have an envelope that specifies the type for decoding. We
    discussed adding the encodinType and you are suggesting adding
    encryption metadata for that msg. All good.

    I don't see your OTP example. Could you delve deeper for me, please?
    The model I envision is internal OTP, with access to decryption
    accessed by cert. A double layer of security, with the internal at-
    rest encryption being an unchanging OTP with ACL access to it as the
    upper layer. Are you saying it is possible to re-encrypt with new
    keys or that there is a chain of keys over time?

    Thanks,
    Rob
    On Jun 8, 2014, at 3:06 PM, Todd Palino wrote:

    I’ll agree that perhaps the “absolutely not” is not quite right.
    There are
    certainly some uses for a simpler solution, but I would still say it
    cannot only be encryption at the broker. This would leave many use
    cases
    for at-rest encryption out of the loop (most auditing cases for SOX,
    PCI,
    HIPAA, and other PII standards). Yes, it does add external overhead
    that
    must be managed, but it’s just the nature of the beast. We can’t
    solve all
    of the external infrastructure needed for this, but we can make it
    easier
    to use for consumers and producers by adding metadata.

    There’s no need for unchanging encryption, and that’s specifically
    why I
    want to see a message envelope that will help consumers determine
    the
    encryption uses for a particular message. You can definitely still
    expire
    keys, you just have to keep the expired keys around as long as the
    encrypted data stays around, and your endpoints need to know when
    they are
    decrypting data with an expired key (you might want to throw up a
    warning,
    or do something else to let the users know that it’s happening). And
    as
    someone else mentioned, there are solutions for encrypting data for
    multiple consumers. You can encrypt the data with an OTP, and then
    multiply encrypt the OTP once for each consumer and store those
    encrypted
    strings in the envelope.

    -Todd
    On 6/7/14, 12:25 PM, "Rob Withers" <robert.w.withers@gmail.com>
    wrote:

    At one level this makes sense to me to externalize the security
    issue
    to producers and consumers. On consideration I realized that this
    adds a lot of coordination requirements to the app layer across
    teams
    or even companies. Another issue I feel is that you want a
    specific
    unchanging encryption for the data and the clients (producers/
    consumers) will need to be able to decode frozen data. If certs
    are
    used they cannot expire. Also, different clients would need to use
    the same cert.

    So, you statement that it should ABSOLUTELY not include internal
    encryption rings seems misplaced. There are some customers of
    kafka
    that would opt to encrypt the on-disk data and key management is a
    significant issue. This is best handled internally, with key
    management stored in either ZK or in a topic. Truly, perhaps
    annealing Hadoop/HBASE as a metadata store seems applicable.

    Thanks, another 2 cents,
    Rob
    On Jun 6, 2014, at 12:15 PM, Todd Palino wrote:

    Yes, I realized last night that I needed to be clearer in what I
    was
    saying. Encryption should ABSOLUTELY not be handled server-side. I
    think
    it¹s a good idea to enable use of it in the consumer/producer, but
    doing
    it server side will not solve many use cases for needing
    encryption
    because the server then has access to all the keys. You could say
    that
    this eliminates the need for TLS, but TLS is pretty low-hanging
    fruit, and
    there¹s definitely a need for encryption of the traffic across the
    network
    even if you don¹t need at-rest encryption as well.

    And as you mentioned, something needs to be done about key
    management.
    Storing information with the message about which key(s) was used
    is
    a good
    idea, because it allows you to know when a producer has switched
    keys.
    There are definitely some alternative solutions to that as well.
    But
    storing the keys in the broker, Zookeeper, or other systems like
    that are
    not. There needs to be a system used where the keys are only
    available to
    the producers and consumers that need them, and they only get
    access
    to
    the appropriate part of the key pair. Even as the guy running
    Kafka
    and
    Zookeeper, I should not have access to the keys being used, and if
    data is
    encrypted I should not be able to see the cleartext.

    And even if we decide not to put anything about at-rest encryption
    in the
    consumer/producer clients directly, and leave it for an exercise
    above
    that level (you have to pass the ciphertext as the message to the
    client),
    I still think there is a good case for implementing a message
    envelope
    that can store the information about which key was used, and other
    pertinent metadata, and have the ability for special applications
    like
    mirror maker to be able to preserve it across clusters. This still
    helps
    to enable the use of encryption and other features (like auditing)
    even if
    we decide it¹s too large a scope to fully implement.

    -Todd

    On 6/6/14, 10:51 AM, "Pradeep Gollakota" <pradeepg26@gmail.com>
    wrote:
    I'm actually not convinced that encryption needs to be handled
    server side
    in Kafka. I think the best solution for encryption is to handle
    it
    producer/consumer side just like compression. This will offload
    key
    management to the users and we'll still be able to leverage the
    sendfile
    optimization for better performance.


    On Fri, Jun 6, 2014 at 10:48 AM, Rob Withers
    <robert.w.withers@gmail.com
    wrote:
    On consideration, if we have 3 different access groups (1 for
    production
    WRITE and 2 consumers) they all need to decode the same
    encryption
    and
    so
    all need the same public/private key....certs won't work, unless
    you
    write
    a CertAuthority to build multiple certs with the same keys.
    Better
    seems
    to not use certs and wrap the encryption specification with an
    ACL
    capabilities for each group of access.


    On Jun 6, 2014, at 11:43 AM, Rob Withers wrote:

    This is quite interesting to me and it is an excelent
    opportunity to
    promote a slightly different security scheme. Object-
    capabilities are
    perfect for online security and would use ACL style
    authentication to
    gain
    capabilities filtered to those allowed resources for allow
    actions
    (READ/WRITE/DELETE/LIST/SCAN). Erights.org has the
    quitenscential (??)
    object capabilities model and capnproto is impleemting this for
    C+
    +. I
    have a java implementation at http://github.com/pauwau/pauwau
    but
    the
    master is broken. 0.2 works, basically. B asically a TLS
    connection
    with
    no certificate server, it is peer to peer. It has some
    advanced
    features,
    but the lining of capabilities with authorization so that you
    can
    only
    invoke correct services is extended to the secure user.

    Regarding non-repudiation, on disk, why not prepend a CRC?

    Regarding on-disk encryption, multiple users/groups may need to
    access,
    with different capabilities. Sounds like zookeeper needs to
    store a
    cert
    for each class of access so that a group member can access the
    decrypted
    data from disk. Use cert-based async decryption. The only
    isue is
    storing
    the private key in zookeeper. Perhaps some hash magic could be
    used.

    Thanks for kafka,
    Rob

    On Jun 5, 2014, at 3:01 PM, Jay Kreps wrote:

    Hey Joe,
    I don't really understand the sections you added to the wiki.
    Can you
    clarify them?

    Is non-repudiation what SASL would call integrity checks? If
    so
    don't
    SSL
    and and many of the SASL schemes already support this as well
    as
    on-the-wire encryption?

    Or are you proposing an on-disk encryption scheme? Is this
    actually
    needed?
    Isn't a on-the-wire encryption when combined with mutual
    authentication
    and
    permissions sufficient for most uses?

    On-disk encryption seems unnecessary because if an attacker
    can
    get
    root
    on
    the kafka boxes it can potentially modify Kafka to do anything
    he or
    she
    wants with data. So this seems to break any security model.

    I understand the problem of a large organization not really
    having a
    trusted network and wanting to secure data transfer and limit
    and
    audit
    data access. The uses for these other things I don't totally
    understand.

    Also it would be worth understanding the state of other
    messaging and
    storage systems (Hadoop, dbs, etc). What features do they
    support. I
    think
    there is a sense in which you don't have to run faster than
    the
    bear,
    but
    only faster then your friends. :-)

    -Jay


    On Wed, Jun 4, 2014 at 5:57 PM, Joe Stein
    <joe.stein@stealth.ly>
    wrote:

    I like the idea of working on the spec and prioritizing. I
    will
    update
    the
    wiki.

    - Joestein


    On Wed, Jun 4, 2014 at 1:11 PM, Jay Kreps
    <jay.kreps@gmail.com>
    wrote:

    Hey Joe,
    Thanks for kicking this discussion off! I totally agree that
    for something
    that acts as a central message broker security is critical
    feature.
    I think
    a number of people have been interested in this topic and
    several
    people
    have put effort into special purpose security efforts.

    Since most the LinkedIn folks are working on the consumer
    right now
    I think
    this would be a great project for any other interested
    people to
    take
    on.
    There are some challenges in doing these things distributed
    but it
    can also
    be a lot of fun.

    I think a good first step would be to get a written plan we
    can all
    agree
    on for how things should work. Then we can break things down
    into
    chunks
    that can be done independently while still aiming at a good
    end
    state.

    I had tried to write up some notes that summarized at least
    the
    thoughts I
    had had on security:
    https://cwiki.apache.org/confluence/display/KAFKA/Security

    What do you think of that?

    One assumption I had (which may be incorrect) is that
    although
    we
    want all
    the things in your list, the two most pressing would be
    authentication and
    authorization, and that was all that write up covered. You
    have more
    experience in this domain, so I wonder how you would
    prioritize?

    Those notes are really sketchy, so I think the first goal I
    would
    have
    would be to get to a real spec we can all agree on and
    discuss. A
    lot
    of
    the security stuff has a high human interaction element and
    needs to
    work
    in pretty different domains and different companies so
    getting
    this
    kind of
    review is important.

    -Jay


    On Tue, Jun 3, 2014 at 12:57 PM, Joe Stein
    <joe.stein@stealth.ly>
    wrote:

    Hi,I wanted to re-ignite the discussion around Apache Kafka
    Security.
    This
    is a huge bottleneck (non-starter in some cases) for a lot
    of
    organizations
    (due to regulatory, compliance and other requirements).
    Below
    are
    my
    suggestions for specific changes in Kafka to accommodate
    security
    requirements. This comes from what folks are doing "in the
    wild"
    to
    workaround and implement security with Kafka as it is today
    and
    also what I
    have discovered from organizations about their blockers. It
    also
    picks
    up
    from the wiki (which I should have time to update later in
    the
    week
    based
    on the below and feedback from the thread).
    1) Transport Layer Security (i.e. SSL)

    This also includes client authentication in addition to in-
    transit security
    layer. This work has been picked up here
    https://issues.apache.org/jira/browse/KAFKA-1477 and do
    appreciate
    any
    thoughts, comments, feedback, tomatoes, whatever for this
    patch.
    It
    is a
    pickup from the fork of the work first done here
    https://github.com/relango/kafka/tree/kafka_security.

    2) Data encryption at rest.

    This is very important and something that can be
    facilitated
    within
    the
    wire protocol. It requires an additional map data structure
    for the
    "encrypted [data encryption key]". With this map (either in
    your
    object or
    in the wire protocol) you can store the dynamically
    generated
    symmetric key
    (for each message) and then encrypt the data using that
    dynamically
    generated key. You then encrypt the encryption key using
    each
    public
    key
    for whom is expected to be able to decrypt the encryption
    key to
    then
    decrypt the message. For each public key encrypted
    symmetric
    key
    (which
    is
    now the "encrypted [data encryption key]" along with which
    public
    key
    it
    was encrypted with for (so a map of [publicKey] =
    encryptedDataEncryptionKey) as a chain. Other patterns
    can be
    implemented
    but this is a pretty standard digital enveloping [0]
    pattern
    with
    only
    1
    field added. Other patterns should be able to use that field
    to-do
    their
    implementation too.
    3) Non-repudiation and long term non-repudiation.

    Non-repudiation is proving data hasn't changed. This is
    often (if
    not
    always) done with x509 public certificates (chained to a
    certificate
    authority).

    Long term non-repudiation is what happens when the
    certificates of
    the
    certificate authority are expired (or revoked) and
    everything
    ever
    signed
    (ever) with that certificate's public key then becomes "no
    longer
    provable
    as ever being authentic". That is where RFC3126 [1] and
    RFC3161
    [2]
    come
    in (or worm drives [hardware], etc).
    For either (or both) of these it is an operation of the
    encryptor
    to
    sign/hash the data (with or without third party trusted
    timestap of
    the
    signing event) and encrypt that with their own private key
    and
    distribute
    the results (before and after encrypting if required) along
    with
    their
    public key. This structure is a bit more complex but
    feasible, it
    is a map
    of digital signature formats and the chain of dig sig
    attestations.
    The
    map's key being the method (i.e. CRC32, PKCS7 [3], XmlDigSig
    [4])
    and
    then
    a list of map where that key is "purpose" of signature
    (what
    your attesting
    too). As a sibling field to the list another field for
    "the
    attester"
    as
    bytes (e.g. their PKCS12 [5] for the map of PKCS7
    signatures).
    4) Authorization

    We should have a policy of "404" for data, topics,
    partitions
    (etc) if
    authenticated connections do not have access. In "secure
    mode" any
    non
    authenticated connections should get a "404" type message
    on
    everything.
    Knowing "something is there" is a security risk in many uses
    cases.
    So
    if
    you don't have access you don't even see it. Baking "that"
    into
    Kafka
    along with some interface for entitlement (access
    management)
    systems
    (pretty standard) is all that I think needs to be done to
    the
    core project.
    I want to tackle item later in the year after summer after
    the
    other three
    are complete.

    I look forward to thoughts on this and anyone else
    interested
    in
    working
    with us on these items.
    [0]
    http://www.emc.com/emc-plus/rsa-labs/standards-
    initiatives/what-is-a-digital-envelope.htm
    [1] http://tools.ietf.org/html/rfc3126


    http://www.emc.com/emc-plus/rsa-labs/standards-initiatives/pk
    cs
    -7
    -
    cryptographic-message-syntax-standar.htm
    [4] http://en.wikipedia.org/wiki/XML_Signature
    [5] http://en.wikipedia.org/wiki/PKCS_12

    /*******************************************
    Joe Stein
    Founder, Principal Consultant
    Big Data Open Source Security LLC
    http://www.stealth.ly
    Twitter: @allthingshadoop
    <http://www.twitter.com/allthingshadoop
    ********************************************/
  • Robert Withers at Jun 10, 2014 at 11:51 pm
    We could extend the existing metadata to include a Kerberos-style token, whichever scheme is used. This would mean creating a producer or consumer with a security context and session negotiation would result in a token. It may be a lease. Both of our modules would authenticate and authorize the token, then do de/encryption, each our own way.

    Thanks,
    Rob
    On Jun 10, 2014, at 4:38 PM, Todd Palino wrote:

    Yes, I agree. There are definitely a variety of use cases that demand
    differing levels of complexity here. It comes back to enabling the
    development of at-rest encryption and making it as easy as possible to
    implement within the Kafka system. I think that this can be done with the
    concept of message metadata that can be preserved across clusters, which
    is separate from the message itself (so it’s not an overlay of a schema on
    top of the message, but rather a separate structure entirely that is
    stored with the message).

    -Todd
    On 6/10/14, 3:26 PM, "Robert Withers" wrote:

    What strikes me as an opportunity is to define a plug gable at-rest
    encryption module interface, that supports each/both of our security
    needs.

    Thanks,
    Rob
    On Jun 10, 2014, at 4:01 PM, Todd Palino <tpalino@linkedin.com.invalid>
    wrote:

    The situation of production before having the consumer is definitely a
    good one. That’s why I wanted to take a little time before responding.
    Had
    to think about it.

    I think that while we may certainly produce data before the consumer is
    ready, that doesn’t mean that the consumer can’t have a key pair
    generated
    for it already, so the producer could start encrypting for that consumer
    before it exists. This would probably work fine for lower retention
    periods (a week or two), but could be a little more difficult to manage
    if
    you are keeping data in Kafka longer than that. My gut reaction is that
    it’s better to handle it that way and keep the key pair and session key
    handling simple. The more we can do that, the more we can leave key
    management as a separate component that can be swapped out so the user
    can
    decide how it should be done.

    -Todd

    On 6/9/14, 8:16 AM, "Robert Withers" <robert.w.withers@gmail.com>
    wrote:

    Yes, that sounds familiar as I helped write (minimally) S/MIME in
    squeak
    (open source Smalltalk environment). This what I was thinking in my
    alternative here, though I have a concern...

    Production may occur before the consumer is coded and executed. In the
    analogy of mail, the mail is sent before the complete recipient list is
    known.

    This seems to mean that the private key (cert or OTP) must be stored
    and
    interacted with. My feeling is that key metadata are in a system
    encrypted Hbase store (session key store), for low latency reads,
    rather
    than a topic requiring scanning. Store the private keys and then give
    client access (producers/consumers) with the hash of the OTP. A new
    consumer comes along, create a new cert encoding the OTP hash.

    On write, use the producer cert to send a topic hash with the msg which
    would allow the broker to reuse or generate an OTP, stored in the
    session
    key store.

    On read (consumer), if we have a previously run reader, use the
    encrypted
    hash. If new, create consumer cert and encrypt the hash for that
    session.

    The reader/writer will pass a cert encrypted session hash. The trick
    seems to be converting hash to PK to encrypt/decrypt. Given Kafka
    resource distribution, we need system encryption for metadata and
    cert-based key exchange. This seems to mean triple encryption:
    1) client to/from broker
    2) system key/hash mgmt/translation
    3) at-rest encryption

    Thanks,
    Rob
    On Jun 9, 2014, at 7:57 AM, Todd Palino <tpalino@linkedin.com.invalid>
    wrote:

    It’s the same method used by S/MIME and many other encryption
    specifications with the potential for multiple recipients. The sender
    generates a session key, and uses that key to encrypt the message. The
    session key is then encrypted once for each recipient with that
    recipient’s public key. All of the encrypted copies of the session key
    are
    then included with the encrypted message. This way, you avoid having
    to
    encrypt the message multiple times (this assumes, of course, that the
    message itself is larger than the key).

    In our case, we have some options available to us. We could do that,
    and
    put all the encrypted keys in the message metadata. Or we could treat
    it
    more like a session and have the encrypted session keys in a special
    topic
    (e.g. __session_keys), much like offsets are now. When the producer
    starts
    up, they create a session key and encrypt it for each consumer with
    the
    current consumer key. The producer publishes the bundle of encrypted
    keys
    into __session_keys as a single message. The producer then publishes
    messages to the normal topic encrypted with the session key. The
    metadata
    for each of those messages would contain something the offset into
    __session_keys to identify the bundle. This has the added benefit of
    not
    increasing the per-message data size too much.

    Whenever a consumer key is invalidated, or however often the session
    key
    should be rotated, the producer would publish a new bundle. This
    maintains
    a history of session keys that can be used to decrypt any messages, so
    the
    retention on __session_keys must be at least as long as any topic
    which
    may potentially contain encrypted data. Past that point, it’s up to
    the
    consumer what they want to do with the data. A consumer like Hadoop
    might
    re-encrypt it for local storage, or store it in plaintext (depending
    on
    the security and requirements of that system).

    -Todd
    On 6/8/14, 2:33 PM, "Rob Withers" wrote:

    I like the use of meta envelopes. We did this recently, on the job,
    as we have an envelope that specifies the type for decoding. We
    discussed adding the encodinType and you are suggesting adding
    encryption metadata for that msg. All good.

    I don't see your OTP example. Could you delve deeper for me, please?
    The model I envision is internal OTP, with access to decryption
    accessed by cert. A double layer of security, with the internal at-
    rest encryption being an unchanging OTP with ACL access to it as the
    upper layer. Are you saying it is possible to re-encrypt with new
    keys or that there is a chain of keys over time?

    Thanks,
    Rob
    On Jun 8, 2014, at 3:06 PM, Todd Palino wrote:

    I’ll agree that perhaps the “absolutely not” is not quite right.
    There are
    certainly some uses for a simpler solution, but I would still say it
    cannot only be encryption at the broker. This would leave many use
    cases
    for at-rest encryption out of the loop (most auditing cases for SOX,
    PCI,
    HIPAA, and other PII standards). Yes, it does add external overhead
    that
    must be managed, but it’s just the nature of the beast. We can’t
    solve all
    of the external infrastructure needed for this, but we can make it
    easier
    to use for consumers and producers by adding metadata.

    There’s no need for unchanging encryption, and that’s specifically
    why I
    want to see a message envelope that will help consumers determine
    the
    encryption uses for a particular message. You can definitely still
    expire
    keys, you just have to keep the expired keys around as long as the
    encrypted data stays around, and your endpoints need to know when
    they are
    decrypting data with an expired key (you might want to throw up a
    warning,
    or do something else to let the users know that it’s happening). And
    as
    someone else mentioned, there are solutions for encrypting data for
    multiple consumers. You can encrypt the data with an OTP, and then
    multiply encrypt the OTP once for each consumer and store those
    encrypted
    strings in the envelope.

    -Todd
    On 6/7/14, 12:25 PM, "Rob Withers" <robert.w.withers@gmail.com>
    wrote:

    At one level this makes sense to me to externalize the security
    issue
    to producers and consumers. On consideration I realized that this
    adds a lot of coordination requirements to the app layer across
    teams
    or even companies. Another issue I feel is that you want a
    specific
    unchanging encryption for the data and the clients (producers/
    consumers) will need to be able to decode frozen data. If certs
    are
    used they cannot expire. Also, different clients would need to use
    the same cert.

    So, you statement that it should ABSOLUTELY not include internal
    encryption rings seems misplaced. There are some customers of
    kafka
    that would opt to encrypt the on-disk data and key management is a
    significant issue. This is best handled internally, with key
    management stored in either ZK or in a topic. Truly, perhaps
    annealing Hadoop/HBASE as a metadata store seems applicable.

    Thanks, another 2 cents,
    Rob
    On Jun 6, 2014, at 12:15 PM, Todd Palino wrote:

    Yes, I realized last night that I needed to be clearer in what I
    was
    saying. Encryption should ABSOLUTELY not be handled server-side. I
    think
    it¹s a good idea to enable use of it in the consumer/producer, but
    doing
    it server side will not solve many use cases for needing
    encryption
    because the server then has access to all the keys. You could say
    that
    this eliminates the need for TLS, but TLS is pretty low-hanging
    fruit, and
    there¹s definitely a need for encryption of the traffic across the
    network
    even if you don¹t need at-rest encryption as well.

    And as you mentioned, something needs to be done about key
    management.
    Storing information with the message about which key(s) was used
    is
    a good
    idea, because it allows you to know when a producer has switched
    keys.
    There are definitely some alternative solutions to that as well.
    But
    storing the keys in the broker, Zookeeper, or other systems like
    that are
    not. There needs to be a system used where the keys are only
    available to
    the producers and consumers that need them, and they only get
    access
    to
    the appropriate part of the key pair. Even as the guy running
    Kafka
    and
    Zookeeper, I should not have access to the keys being used, and if
    data is
    encrypted I should not be able to see the cleartext.

    And even if we decide not to put anything about at-rest encryption
    in the
    consumer/producer clients directly, and leave it for an exercise
    above
    that level (you have to pass the ciphertext as the message to the
    client),
    I still think there is a good case for implementing a message
    envelope
    that can store the information about which key was used, and other
    pertinent metadata, and have the ability for special applications
    like
    mirror maker to be able to preserve it across clusters. This still
    helps
    to enable the use of encryption and other features (like auditing)
    even if
    we decide it¹s too large a scope to fully implement.

    -Todd

    On 6/6/14, 10:51 AM, "Pradeep Gollakota" <pradeepg26@gmail.com>
    wrote:
    I'm actually not convinced that encryption needs to be handled
    server side
    in Kafka. I think the best solution for encryption is to handle
    it
    producer/consumer side just like compression. This will offload
    key
    management to the users and we'll still be able to leverage the
    sendfile
    optimization for better performance.


    On Fri, Jun 6, 2014 at 10:48 AM, Rob Withers
    <robert.w.withers@gmail.com
    wrote:
    On consideration, if we have 3 different access groups (1 for
    production
    WRITE and 2 consumers) they all need to decode the same
    encryption
    and
    so
    all need the same public/private key....certs won't work, unless
    you
    write
    a CertAuthority to build multiple certs with the same keys.
    Better
    seems
    to not use certs and wrap the encryption specification with an
    ACL
    capabilities for each group of access.


    On Jun 6, 2014, at 11:43 AM, Rob Withers wrote:

    This is quite interesting to me and it is an excelent
    opportunity to
    promote a slightly different security scheme. Object-
    capabilities are
    perfect for online security and would use ACL style
    authentication to
    gain
    capabilities filtered to those allowed resources for allow
    actions
    (READ/WRITE/DELETE/LIST/SCAN). Erights.org has the
    quitenscential (??)
    object capabilities model and capnproto is impleemting this for
    C+
    +. I
    have a java implementation at http://github.com/pauwau/pauwau
    but
    the
    master is broken. 0.2 works, basically. B asically a TLS
    connection
    with
    no certificate server, it is peer to peer. It has some
    advanced
    features,
    but the lining of capabilities with authorization so that you
    can
    only
    invoke correct services is extended to the secure user.

    Regarding non-repudiation, on disk, why not prepend a CRC?

    Regarding on-disk encryption, multiple users/groups may need to
    access,
    with different capabilities. Sounds like zookeeper needs to
    store a
    cert
    for each class of access so that a group member can access the
    decrypted
    data from disk. Use cert-based async decryption. The only
    isue is
    storing
    the private key in zookeeper. Perhaps some hash magic could be
    used.

    Thanks for kafka,
    Rob

    On Jun 5, 2014, at 3:01 PM, Jay Kreps wrote:

    Hey Joe,
    I don't really understand the sections you added to the wiki.
    Can you
    clarify them?

    Is non-repudiation what SASL would call integrity checks? If
    so
    don't
    SSL
    and and many of the SASL schemes already support this as well
    as
    on-the-wire encryption?

    Or are you proposing an on-disk encryption scheme? Is this
    actually
    needed?
    Isn't a on-the-wire encryption when combined with mutual
    authentication
    and
    permissions sufficient for most uses?

    On-disk encryption seems unnecessary because if an attacker
    can
    get
    root
    on
    the kafka boxes it can potentially modify Kafka to do anything
    he or
    she
    wants with data. So this seems to break any security model.

    I understand the problem of a large organization not really
    having a
    trusted network and wanting to secure data transfer and limit
    and
    audit
    data access. The uses for these other things I don't totally
    understand.

    Also it would be worth understanding the state of other
    messaging and
    storage systems (Hadoop, dbs, etc). What features do they
    support. I
    think
    there is a sense in which you don't have to run faster than
    the
    bear,
    but
    only faster then your friends. :-)

    -Jay


    On Wed, Jun 4, 2014 at 5:57 PM, Joe Stein
    <joe.stein@stealth.ly>
    wrote:

    I like the idea of working on the spec and prioritizing. I
    will
    update
    the
    wiki.

    - Joestein


    On Wed, Jun 4, 2014 at 1:11 PM, Jay Kreps
    <jay.kreps@gmail.com>
    wrote:

    Hey Joe,
    Thanks for kicking this discussion off! I totally agree that
    for something
    that acts as a central message broker security is critical
    feature.
    I think
    a number of people have been interested in this topic and
    several
    people
    have put effort into special purpose security efforts.

    Since most the LinkedIn folks are working on the consumer
    right now
    I think
    this would be a great project for any other interested
    people to
    take
    on.
    There are some challenges in doing these things distributed
    but it
    can also
    be a lot of fun.

    I think a good first step would be to get a written plan we
    can all
    agree
    on for how things should work. Then we can break things down
    into
    chunks
    that can be done independently while still aiming at a good
    end
    state.

    I had tried to write up some notes that summarized at least
    the
    thoughts I
    had had on security:
    https://cwiki.apache.org/confluence/display/KAFKA/Security

    What do you think of that?

    One assumption I had (which may be incorrect) is that
    although
    we
    want all
    the things in your list, the two most pressing would be
    authentication and
    authorization, and that was all that write up covered. You
    have more
    experience in this domain, so I wonder how you would
    prioritize?

    Those notes are really sketchy, so I think the first goal I
    would
    have
    would be to get to a real spec we can all agree on and
    discuss. A
    lot
    of
    the security stuff has a high human interaction element and
    needs to
    work
    in pretty different domains and different companies so
    getting
    this
    kind of
    review is important.

    -Jay


    On Tue, Jun 3, 2014 at 12:57 PM, Joe Stein
    <joe.stein@stealth.ly>
    wrote:

    Hi,I wanted to re-ignite the discussion around Apache Kafka
    Security.
    This
    is a huge bottleneck (non-starter in some cases) for a lot
    of
    organizations
    (due to regulatory, compliance and other requirements).
    Below
    are
    my
    suggestions for specific changes in Kafka to accommodate
    security
    requirements. This comes from what folks are doing "in the
    wild"
    to
    workaround and implement security with Kafka as it is today
    and
    also what I
    have discovered from organizations about their blockers. It
    also
    picks
    up
    from the wiki (which I should have time to update later in
    the
    week
    based
    on the below and feedback from the thread).
    1) Transport Layer Security (i.e. SSL)

    This also includes client authentication in addition to in-
    transit security
    layer. This work has been picked up here
    https://issues.apache.org/jira/browse/KAFKA-1477 and do
    appreciate
    any
    thoughts, comments, feedback, tomatoes, whatever for this
    patch.
    It
    is a
    pickup from the fork of the work first done here
    https://github.com/relango/kafka/tree/kafka_security.

    2) Data encryption at rest.

    This is very important and something that can be
    facilitated
    within
    the
    wire protocol. It requires an additional map data structure
    for the
    "encrypted [data encryption key]". With this map (either in
    your
    object or
    in the wire protocol) you can store the dynamically
    generated
    symmetric key
    (for each message) and then encrypt the data using that
    dynamically
    generated key. You then encrypt the encryption key using
    each
    public
    key
    for whom is expected to be able to decrypt the encryption
    key to
    then
    decrypt the message. For each public key encrypted
    symmetric
    key
    (which
    is
    now the "encrypted [data encryption key]" along with which
    public
    key
    it
    was encrypted with for (so a map of [publicKey] =
    encryptedDataEncryptionKey) as a chain. Other patterns
    can be
    implemented
    but this is a pretty standard digital enveloping [0]
    pattern
    with
    only
    1
    field added. Other patterns should be able to use that field
    to-do
    their
    implementation too.
    3) Non-repudiation and long term non-repudiation.

    Non-repudiation is proving data hasn't changed. This is
    often (if
    not
    always) done with x509 public certificates (chained to a
    certificate
    authority).

    Long term non-repudiation is what happens when the
    certificates of
    the
    certificate authority are expired (or revoked) and
    everything
    ever
    signed
    (ever) with that certificate's public key then becomes "no
    longer
    provable
    as ever being authentic". That is where RFC3126 [1] and
    RFC3161
    [2]
    come
    in (or worm drives [hardware], etc).
    For either (or both) of these it is an operation of the
    encryptor
    to
    sign/hash the data (with or without third party trusted
    timestap of
    the
    signing event) and encrypt that with their own private key
    and
    distribute
    the results (before and after encrypting if required) along
    with
    their
    public key. This structure is a bit more complex but
    feasible, it
    is a map
    of digital signature formats and the chain of dig sig
    attestations.
    The
    map's key being the method (i.e. CRC32, PKCS7 [3], XmlDigSig
    [4])
    and
    then
    a list of map where that key is "purpose" of signature
    (what
    your attesting
    too). As a sibling field to the list another field for
    "the
    attester"
    as
    bytes (e.g. their PKCS12 [5] for the map of PKCS7
    signatures).
    4) Authorization

    We should have a policy of "404" for data, topics,
    partitions
    (etc) if
    authenticated connections do not have access. In "secure
    mode" any
    non
    authenticated connections should get a "404" type message
    on
    everything.
    Knowing "something is there" is a security risk in many uses
    cases.
    So
    if
    you don't have access you don't even see it. Baking "that"
    into
    Kafka
    along with some interface for entitlement (access
    management)
    systems
    (pretty standard) is all that I think needs to be done to
    the
    core project.
    I want to tackle item later in the year after summer after
    the
    other three
    are complete.

    I look forward to thoughts on this and anyone else
    interested
    in
    working
    with us on these items.
    [0]
    http://www.emc.com/emc-plus/rsa-labs/standards-
    initiatives/what-is-a-digital-envelope.htm
    [1] http://tools.ietf.org/html/rfc3126


    http://www.emc.com/emc-plus/rsa-labs/standards-initiatives/pk
    cs
    -7
    -
    cryptographic-message-syntax-standar.htm
    [4] http://en.wikipedia.org/wiki/XML_Signature
    [5] http://en.wikipedia.org/wiki/PKCS_12

    /*******************************************
    Joe Stein
    Founder, Principal Consultant
    Big Data Open Source Security LLC
    http://www.stealth.ly
    Twitter: @allthingshadoop
    <http://www.twitter.com/allthingshadoop
    ********************************************/

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupusers @
categorieskafka
postedJun 3, '14 at 7:57p
activeJun 10, '14 at 11:51p
posts27
users8
websitekafka.apache.org
irc#kafka

People

Translate

site design / logo © 2019 Grokbase