Hi,I wanted to re-ignite the discussion around Apache Kafka Security. This
is a huge bottleneck (non-starter in some cases) for a lot of organizations
(due to regulatory, compliance and other requirements). Below are my
suggestions for specific changes in Kafka to accommodate security
requirements. This comes from what folks are doing "in the wild" to
workaround and implement security with Kafka as it is today and also what I
have discovered from organizations about their blockers. It also picks up
from the wiki (which I should have time to update later in the week based
on the below and feedback from the thread).
1) Transport Layer Security (i.e. SSL)
This also includes client authentication in addition to in-transit security
layer. This work has been picked up here
https://issues.apache.org/jira/browse/KAFKA-1477 and do appreciate any
thoughts, comments, feedback, tomatoes, whatever for this patch. It is a
pickup from the fork of the work first done here
2) Data encryption at rest.
This is very important and something that can be facilitated within the
wire protocol. It requires an additional map data structure for the
"encrypted [data encryption key]". With this map (either in your object or
in the wire protocol) you can store the dynamically generated symmetric key
(for each message) and then encrypt the data using that dynamically
generated key. You then encrypt the encryption key using each public key
for whom is expected to be able to decrypt the encryption key to then
decrypt the message. For each public key encrypted symmetric key (which is
now the "encrypted [data encryption key]" along with which public key it
was encrypted with for (so a map of [publicKey] =
encryptedDataEncryptionKey) as a chain. Other patterns can be implemented
but this is a pretty standard digital enveloping  pattern with only 1
field added. Other patterns should be able to use that field to-do their
3) Non-repudiation and long term non-repudiation.
Non-repudiation is proving data hasn't changed. This is often (if not
always) done with x509 public certificates (chained to a certificate
Long term non-repudiation is what happens when the certificates of the
certificate authority are expired (or revoked) and everything ever signed
(ever) with that certificate's public key then becomes "no longer provable
as ever being authentic". That is where RFC3126  and RFC3161  come
in (or worm drives [hardware], etc).
For either (or both) of these it is an operation of the encryptor to
sign/hash the data (with or without third party trusted timestap of the
signing event) and encrypt that with their own private key and distribute
the results (before and after encrypting if required) along with their
public key. This structure is a bit more complex but feasible, it is a map
of digital signature formats and the chain of dig sig attestations. The
map's key being the method (i.e. CRC32, PKCS7 , XmlDigSig ) and then
a list of map where that key is "purpose" of signature (what your attesting
too). As a sibling field to the list another field for "the attester" as
bytes (e.g. their PKCS12  for the map of PKCS7 signatures).
We should have a policy of "404" for data, topics, partitions (etc) if
authenticated connections do not have access. In "secure mode" any non
authenticated connections should get a "404" type message on everything.
Knowing "something is there" is a security risk in many uses cases. So if
you don't have access you don't even see it. Baking "that" into Kafka
along with some interface for entitlement (access management) systems
(pretty standard) is all that I think needs to be done to the core project.
I want to tackle item later in the year after summer after the other three
I look forward to thoughts on this and anyone else interested in working
with us on these items.
Founder, Principal Consultant
Big Data Open Source Security LLC
Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop>