Grokbase Groups HBase user June 2016
FAQ
Hi, all,

We are trying to deploy the 'transparent encryption' feature of HBase , described in HBase reference guide: https://hbase.apache.org/book.html#hbase.encryption.server , in our product.
We heard from various sources that it is not production ready before.

During our tests, we do find out it works not very stable, but probably due to our lack of experience of this feature. It works sometime, sometimes not work, and retry the same configuration, it work again. We were using HBase 1.0.

Could anyone give us some information that this feature is already stable and can be used in a production environment?

And, we now save the encryption key in the disk, so we were wondering, this is something not secure. Since the key is at same place with data, anyone can decode the data because if he/she can access the data, he/she can access the key as well. Is there any best practice about how to manage the key?

Thanks,
Ming

Search Discussions

  • Andrew Purtell at Jun 3, 2016 at 1:00 am
    We heard from various sources that it is not production ready before.
    ​Said by whom, specifically? ​

    ​> During our tests, we do find out it works not very stable, but probably
    due to our lack of experience of this feature.

    If you have something repeatable, please consider filing a JIRA to report
    the problem.
    And, we now save the encryption key in the disk, so we were wondering,
    this is something not secure.

    Data keys are encrypted with a master key which must be protected. The out
    of the box key provider stores the master key in a local keystore. That's
    not sufficient protection. In a production environment you will want to use
    a HSM. Most (all?) HSMs support the keystore API. If that is not
    sufficient, our KeyProvider API is extensible for the solution you choose
    to employ in production.

    ​Have you looked at HDFS transparent encryption?
    https://hadoop.apache.org/docs/r2.7.2/hadoop-project-dist/hadoop-hdfs/TransparentEncryption.html
    Because it works at the HDFS layer it's a more general solution. Be careful
    what version of Hadoop you use if opting for HDFS TDE, though. Pick the
    most recent release. Slightly older versions (like 2.6.0) had fatal bugs if
    used in conjunction with HBase.


    On Thu, Jun 2, 2016 at 5:52 PM, Liu, Ming (Ming) wrote:

    Hi, all,

    We are trying to deploy the 'transparent encryption' feature of HBase ,
    described in HBase reference guide:
    https://hbase.apache.org/book.html#hbase.encryption.server , in our
    product.
    We heard from various sources that it is not production ready before.

    During our tests, we do find out it works not very stable, but probably
    due to our lack of experience of this feature. It works sometime, sometimes
    not work, and retry the same configuration, it work again. We were using
    HBase 1.0.

    Could anyone give us some information that this feature is already stable
    and can be used in a production environment?

    And, we now save the encryption key in the disk, so we were wondering,
    this is something not secure. Since the key is at same place with data,
    anyone can decode the data because if he/she can access the data, he/she
    can access the key as well. Is there any best practice about how to manage
    the key?

    Thanks,
    Ming

    --
    Best regards,

        - Andy

    Problems worthy of attack prove their worth by hitting back. - Piet Hein
    (via Tom White)
  • Liu, Ming (Ming) at Jun 3, 2016 at 4:16 am
    Thank you Andrew!

    What we hear must be rumor :-) We are now confident to use this feature.

    HSM is a good option, I am new to it. But will look at it.

    Thanks,
    Ming
    -----邮件原件-----
    发件人: Andrew Purtell
    发送时间: 2016年6月3日 8:59
    收件人: user@hbase.apache.org
    抄送: Zhang, Yi (Eason) <yi.zhang@esgyn.cn>
    主题: Re: hbase 'transparent encryption' feature is production ready or not?
    We heard from various sources that it is not production ready before.
    ​Said by whom, specifically? ​

    ​> During our tests, we do find out it works not very stable, but probably due to our lack of experience of this feature.

    If you have something repeatable, please consider filing a JIRA to report the problem.
    And, we now save the encryption key in the disk, so we were wondering,
    this is something not secure.

    Data keys are encrypted with a master key which must be protected. The out of the box key provider stores the master key in a local keystore. That's not sufficient protection. In a production environment you will want to use a HSM. Most (all?) HSMs support the keystore API. If that is not sufficient, our KeyProvider API is extensible for the solution you choose to employ in production.

    ​Have you looked at HDFS transparent encryption?
    https://hadoop.apache.org/docs/r2.7.2/hadoop-project-dist/hadoop-hdfs/TransparentEncryption.html
    Because it works at the HDFS layer it's a more general solution. Be careful what version of Hadoop you use if opting for HDFS TDE, though. Pick the most recent release. Slightly older versions (like 2.6.0) had fatal bugs if used in conjunction with HBase.


    On Thu, Jun 2, 2016 at 5:52 PM, Liu, Ming (Ming) wrote:

    Hi, all,

    We are trying to deploy the 'transparent encryption' feature of HBase
    , described in HBase reference guide:
    https://hbase.apache.org/book.html#hbase.encryption.server , in our
    product.
    We heard from various sources that it is not production ready before.

    During our tests, we do find out it works not very stable, but
    probably due to our lack of experience of this feature. It works
    sometime, sometimes not work, and retry the same configuration, it
    work again. We were using HBase 1.0.

    Could anyone give us some information that this feature is already
    stable and can be used in a production environment?

    And, we now save the encryption key in the disk, so we were wondering,
    this is something not secure. Since the key is at same place with
    data, anyone can decode the data because if he/she can access the
    data, he/she can access the key as well. Is there any best practice
    about how to manage the key?

    Thanks,
    Ming

    --
    Best regards,

        - Andy

    Problems worthy of attack prove their worth by hitting back. - Piet Hein (via Tom White)
  • Andrew Purtell at Jun 3, 2016 at 4:27 am

    We are now confident to use this feature.
    You should test carefully for your use case in any case.
    HSM is a good option, I am new to it. But will look at it.
    I recommend using HDFS's transparent encryption feature instead of HBase
    transparent encryption if you're only just now thinking about HSMs and key
    protection in general. Storing the master key on the same nodes as the
    encrypted data will defeat protection. This should be offloaded to a
    protected domain. Hadoop ships with a software KMS that, while it has
    limitations, can be set up on a specially secured server and HDFS TDE can
    take advantage of it. (HBase TDE doesn't support the Hadoop KMS.)

    Advice offered for what it's worth (smile)

    On Thu, Jun 2, 2016 at 9:16 PM, Liu, Ming (Ming) wrote:

    Thank you Andrew!

    What we hear must be rumor :-) We are now confident to use this feature.

    HSM is a good option, I am new to it. But will look at it.

    Thanks,
    Ming
    -----邮件原件-----
    发件人: Andrew Purtell
    发送时间: 2016年6月3日 8:59
    收件人: user@hbase.apache.org
    抄送: Zhang, Yi (Eason) <yi.zhang@esgyn.cn>
    主题: Re: hbase 'transparent encryption' feature is production ready or not?
    We heard from various sources that it is not production ready before.
    ​Said by whom, specifically? ​

    ​> During our tests, we do find out it works not very stable, but probably
    due to our lack of experience of this feature.

    If you have something repeatable, please consider filing a JIRA to report
    the problem.
    And, we now save the encryption key in the disk, so we were wondering,
    this is something not secure.

    Data keys are encrypted with a master key which must be protected. The out
    of the box key provider stores the master key in a local keystore. That's
    not sufficient protection. In a production environment you will want to use
    a HSM. Most (all?) HSMs support the keystore API. If that is not
    sufficient, our KeyProvider API is extensible for the solution you choose
    to employ in production.

    ​Have you looked at HDFS transparent encryption?

    https://hadoop.apache.org/docs/r2.7.2/hadoop-project-dist/hadoop-hdfs/TransparentEncryption.html
    Because it works at the HDFS layer it's a more general solution. Be
    careful what version of Hadoop you use if opting for HDFS TDE, though. Pick
    the most recent release. Slightly older versions (like 2.6.0) had fatal
    bugs if used in conjunction with HBase.


    On Thu, Jun 2, 2016 at 5:52 PM, Liu, Ming (Ming) wrote:

    Hi, all,

    We are trying to deploy the 'transparent encryption' feature of HBase
    , described in HBase reference guide:
    https://hbase.apache.org/book.html#hbase.encryption.server , in our
    product.
    We heard from various sources that it is not production ready before.

    During our tests, we do find out it works not very stable, but
    probably due to our lack of experience of this feature. It works
    sometime, sometimes not work, and retry the same configuration, it
    work again. We were using HBase 1.0.

    Could anyone give us some information that this feature is already
    stable and can be used in a production environment?

    And, we now save the encryption key in the disk, so we were wondering,
    this is something not secure. Since the key is at same place with
    data, anyone can decode the data because if he/she can access the
    data, he/she can access the key as well. Is there any best practice
    about how to manage the key?

    Thanks,
    Ming

    --
    Best regards,

    - Andy

    Problems worthy of attack prove their worth by hitting back. - Piet Hein
    (via Tom White)


    --
    Best regards,

        - Andy

    Problems worthy of attack prove their worth by hitting back. - Piet Hein
    (via Tom White)

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupuser @
categorieshbase, hadoop
postedJun 3, '16 at 12:52a
activeJun 3, '16 at 4:27a
posts4
users2
websitehbase.apache.org

People

Translate

site design / logo © 2018 Grokbase