FAQ
Hello,

I want to invoke metadata refresh manually from a Java client. Currently,
the JDBC driver prohibits me from doing that.
I can look into impala-shell sources and do a refresh via Thrift/RPC in the
same fashion - however, is there any recommended way for refreshing?

Thanks,
- Alex.

Search Discussions

  • Ishaan Joshi at May 20, 2013 at 5:19 pm
    Serega, Alex,

       At the moment, you have the explicitly refresh the impalad from the
    impala-shell. Note that the refresh roc will not automatically refresh each
    running impalad, just the one you're connected to.

    Thanks,

    -- Ishaan


    On Mon, May 20, 2013 at 12:17 AM, Serega Sheypak
    wrote:
    The same problem! Is there any way to do it?
    I have a table which partitions are updated each hour. So I have to submit
    "refresh" in some way bwfore running query...


    понедельник, 1 апреля 2013 г., 19:55:10 UTC+4 пользователь Alex I написал:
    Hello,

    I want to invoke metadata refresh manually from a Java client. Currently,
    the JDBC driver prohibits me from doing that.
    I can look into impala-shell sources and do a refresh via Thrift/RPC in
    the same fashion - however, is there any recommended way for refreshing?

    Thanks,
    - Alex.
  • Serega Sheypak at May 20, 2013 at 5:35 pm
    Yes, I understand it. Just wanted to hear it from Cloudera before starting
    to invent the bicycle.
    I've created brand-new bicycle. It rides to random impalad taken from
    application configuration and refreshes metadata before giving JDBC
    connection to the same impalad.
    It's rather ugly, but works.

    public class ImpaladMetadataRefresher {

         private final Logger LOG =
    LoggerFactory.getLogger(ImpaladMetadataRefresher.class);

         public void refresh(String user, String password, String host, int
    port) throws ImpalaRefresherException{
             LOG.debug("Refreshing with params: user={}, password={}, host={},
    port={}", user, password, host, port);
             Session session = null;
             Channel channel = null;

             try{
                 JSch jsch=new JSch();
                 session = jsch.getSession(user, host, port);
                 session.setConfig("StrictHostKeyChecking", "no");
                 session.setTimeout(1500);
                 session.setPassword(password);
                 session.connect();

                 channel = session.openChannel("exec");
                 InputStream stdout = channel.getInputStream();
                 InputStream stderr = ((ChannelExec)channel).getErrStream();

                 ((ChannelExec)channel).setCommand("impala-shell -i
    'localhost:21000' -q 'refresh web_resource_rating'");
                 channel.connect();
                 channel.run();

                 String stdoutString = IOUtils.toString(stdout);
                 LOG.info("stdout:{}", stdoutString);

                 String stdErrString = IOUtils.toString(stderr);
                 LOG.info("stderr:{}", stdErrString);

                 if (stdoutString == null || stdoutString.isEmpty() ||
    !stdoutString.contains("Successfully refreshed table")){
                     throw new ImpalaRefresherException(String.format("Looks
    like ssh exec failed: %s", stdErrString));
                 }


             }catch (Exception e){
                 LOG.error("Error while refreshing metadata on impalad daemon",
    e);
                 throw new ImpalaRefresherException("Rethrown original
    exception", e);
             }finally {
                 closeSilently(channel);
                 closeSilently(session);
             }
         }

         public void closeSilently(Session session){
             try{
                 if(session!=null && session.isConnected()){
                     session.disconnect();
                 }
             }catch (Exception e){
                 LOG.error("Error while closing ssh session", e);
             }
         }

         public void closeSilently(Channel channel){
             try{
                 if(channel!=null && channel.isConnected()){
                     channel.disconnect();
                 }
             }catch (Exception e){
                 LOG.error("Error while closing ssh channel", e);
             }
         }
    }


    2013/5/20 Ishaan Joshi <ishaan@cloudera.com>
    Serega, Alex,

    At the moment, you have the explicitly refresh the impalad from the
    impala-shell. Note that the refresh roc will not automatically refresh each
    running impalad, just the one you're connected to.

    Thanks,

    -- Ishaan


    On Mon, May 20, 2013 at 12:17 AM, Serega Sheypak <serega.sheypak@gmail.com
    wrote:
    The same problem! Is there any way to do it?
    I have a table which partitions are updated each hour. So I have to
    submit "refresh" in some way bwfore running query...


    понедельник, 1 апреля 2013 г., 19:55:10 UTC+4 пользователь Alex I написал:
    Hello,

    I want to invoke metadata refresh manually from a Java client.
    Currently, the JDBC driver prohibits me from doing that.
    I can look into impala-shell sources and do a refresh via Thrift/RPC in
    the same fashion - however, is there any recommended way for refreshing?

    Thanks,
    - Alex.
  • Ricky Saltzer at May 20, 2013 at 6:09 pm
    Hey guys -

    I've recently wrote a program to do exactly this, it will refresh the
    metadata on a specific table across all
    the daemons concurrently. I'm in the process of cleaning the code up, I'll
    try and put it on github if
    any of you are interested.

    Ricky

    On Mon, May 20, 2013 at 1:35 PM, Serega Sheypak wrote:

    Yes, I understand it. Just wanted to hear it from Cloudera before starting
    to invent the bicycle.
    I've created brand-new bicycle. It rides to random impalad taken from
    application configuration and refreshes metadata before giving JDBC
    connection to the same impalad.
    It's rather ugly, but works.

    public class ImpaladMetadataRefresher {

    private final Logger LOG =
    LoggerFactory.getLogger(ImpaladMetadataRefresher.class);

    public void refresh(String user, String password, String host, int
    port) throws ImpalaRefresherException{
    LOG.debug("Refreshing with params: user={}, password={}, host={},
    port={}", user, password, host, port);
    Session session = null;
    Channel channel = null;

    try{
    JSch jsch=new JSch();
    session = jsch.getSession(user, host, port);
    session.setConfig("StrictHostKeyChecking", "no");
    session.setTimeout(1500);
    session.setPassword(password);
    session.connect();

    channel = session.openChannel("exec");
    InputStream stdout = channel.getInputStream();
    InputStream stderr = ((ChannelExec)channel).getErrStream();

    ((ChannelExec)channel).setCommand("impala-shell -i
    'localhost:21000' -q 'refresh web_resource_rating'");
    channel.connect();
    channel.run();

    String stdoutString = IOUtils.toString(stdout);
    LOG.info("stdout:{}", stdoutString);

    String stdErrString = IOUtils.toString(stderr);
    LOG.info("stderr:{}", stdErrString);

    if (stdoutString == null || stdoutString.isEmpty() ||
    !stdoutString.contains("Successfully refreshed table")){
    throw new ImpalaRefresherException(String.format("Looks
    like ssh exec failed: %s", stdErrString));
    }


    }catch (Exception e){
    LOG.error("Error while refreshing metadata on impalad daemon",
    e);
    throw new ImpalaRefresherException("Rethrown original
    exception", e);
    }finally {
    closeSilently(channel);
    closeSilently(session);
    }
    }

    public void closeSilently(Session session){
    try{
    if(session!=null && session.isConnected()){
    session.disconnect();
    }
    }catch (Exception e){
    LOG.error("Error while closing ssh session", e);
    }
    }

    public void closeSilently(Channel channel){
    try{
    if(channel!=null && channel.isConnected()){
    channel.disconnect();
    }
    }catch (Exception e){
    LOG.error("Error while closing ssh channel", e);
    }
    }
    }


    2013/5/20 Ishaan Joshi <ishaan@cloudera.com>
    Serega, Alex,

    At the moment, you have the explicitly refresh the impalad from the
    impala-shell. Note that the refresh roc will not automatically refresh each
    running impalad, just the one you're connected to.

    Thanks,

    -- Ishaan


    On Mon, May 20, 2013 at 12:17 AM, Serega Sheypak <
    serega.sheypak@gmail.com> wrote:
    The same problem! Is there any way to do it?
    I have a table which partitions are updated each hour. So I have to
    submit "refresh" in some way bwfore running query...


    понедельник, 1 апреля 2013 г., 19:55:10 UTC+4 пользователь Alex I
    написал:
    Hello,

    I want to invoke metadata refresh manually from a Java client.
    Currently, the JDBC driver prohibits me from doing that.
    I can look into impala-shell sources and do a refresh via Thrift/RPC in
    the same fashion - however, is there any recommended way for refreshing?

    Thanks,
    - Alex.

    --
    Ricky Saltzer
    Tools Developer
    http://www.cloudera.com
  • Serega Sheypak at May 20, 2013 at 6:23 pm
    Thanks. I'll be great. We have 19 impalad instances. Don't you think it can
    bring significant pressure on hive metastore?
      20.05.2013 20:09 пользователь "Ricky Saltzer" <ricky@cloudera.com> написал:
    Hey guys -

    I've recently wrote a program to do exactly this, it will refresh the
    metadata on a specific table across all
    the daemons concurrently. I'm in the process of cleaning the code up, I'll
    try and put it on github if
    any of you are interested.

    Ricky

    On Mon, May 20, 2013 at 1:35 PM, Serega Sheypak wrote:

    Yes, I understand it. Just wanted to hear it from Cloudera before
    starting to invent the bicycle.
    I've created brand-new bicycle. It rides to random impalad taken from
    application configuration and refreshes metadata before giving JDBC
    connection to the same impalad.
    It's rather ugly, but works.

    public class ImpaladMetadataRefresher {

    private final Logger LOG =
    LoggerFactory.getLogger(ImpaladMetadataRefresher.class);

    public void refresh(String user, String password, String host, int
    port) throws ImpalaRefresherException{
    LOG.debug("Refreshing with params: user={}, password={}, host={},
    port={}", user, password, host, port);
    Session session = null;
    Channel channel = null;

    try{
    JSch jsch=new JSch();
    session = jsch.getSession(user, host, port);
    session.setConfig("StrictHostKeyChecking", "no");
    session.setTimeout(1500);
    session.setPassword(password);
    session.connect();

    channel = session.openChannel("exec");
    InputStream stdout = channel.getInputStream();
    InputStream stderr = ((ChannelExec)channel).getErrStream();

    ((ChannelExec)channel).setCommand("impala-shell -i
    'localhost:21000' -q 'refresh web_resource_rating'");
    channel.connect();
    channel.run();

    String stdoutString = IOUtils.toString(stdout);
    LOG.info("stdout:{}", stdoutString);

    String stdErrString = IOUtils.toString(stderr);
    LOG.info("stderr:{}", stdErrString);

    if (stdoutString == null || stdoutString.isEmpty() ||
    !stdoutString.contains("Successfully refreshed table")){
    throw new ImpalaRefresherException(String.format("Looks
    like ssh exec failed: %s", stdErrString));
    }


    }catch (Exception e){
    LOG.error("Error while refreshing metadata on impalad
    daemon", e);
    throw new ImpalaRefresherException("Rethrown original
    exception", e);
    }finally {
    closeSilently(channel);
    closeSilently(session);
    }
    }

    public void closeSilently(Session session){
    try{
    if(session!=null && session.isConnected()){
    session.disconnect();
    }
    }catch (Exception e){
    LOG.error("Error while closing ssh session", e);
    }
    }

    public void closeSilently(Channel channel){
    try{
    if(channel!=null && channel.isConnected()){
    channel.disconnect();
    }
    }catch (Exception e){
    LOG.error("Error while closing ssh channel", e);
    }
    }
    }


    2013/5/20 Ishaan Joshi <ishaan@cloudera.com>
    Serega, Alex,

    At the moment, you have the explicitly refresh the impalad from the
    impala-shell. Note that the refresh roc will not automatically refresh each
    running impalad, just the one you're connected to.

    Thanks,

    -- Ishaan


    On Mon, May 20, 2013 at 12:17 AM, Serega Sheypak <
    serega.sheypak@gmail.com> wrote:
    The same problem! Is there any way to do it?
    I have a table which partitions are updated each hour. So I have to
    submit "refresh" in some way bwfore running query...


    понедельник, 1 апреля 2013 г., 19:55:10 UTC+4 пользователь Alex I
    написал:
    Hello,

    I want to invoke metadata refresh manually from a Java client.
    Currently, the JDBC driver prohibits me from doing that.
    I can look into impala-shell sources and do a refresh via Thrift/RPC
    in the same fashion - however, is there any recommended way for refreshing?

    Thanks,
    - Alex.

    --
    Ricky Saltzer
    Tools Developer
    http://www.cloudera.com

  • Ricky Saltzer at May 20, 2013 at 6:28 pm
    I have a 14 node cluster and running a refresh all at the same time doesn't
    seem to run any slower than doing one.

    On Mon, May 20, 2013 at 2:23 PM, Serega Sheypak wrote:

    Thanks. I'll be great. We have 19 impalad instances. Don't you think it
    can bring significant pressure on hive metastore?
    20.05.2013 20:09 пользователь "Ricky Saltzer" <ricky@cloudera.com>
    написал:

    Hey guys -
    I've recently wrote a program to do exactly this, it will refresh the
    metadata on a specific table across all
    the daemons concurrently. I'm in the process of cleaning the code up,
    I'll try and put it on github if
    any of you are interested.

    Ricky


    On Mon, May 20, 2013 at 1:35 PM, Serega Sheypak <serega.sheypak@gmail.com
    wrote:
    Yes, I understand it. Just wanted to hear it from Cloudera before
    starting to invent the bicycle.
    I've created brand-new bicycle. It rides to random impalad taken from
    application configuration and refreshes metadata before giving JDBC
    connection to the same impalad.
    It's rather ugly, but works.

    public class ImpaladMetadataRefresher {

    private final Logger LOG =
    LoggerFactory.getLogger(ImpaladMetadataRefresher.class);

    public void refresh(String user, String password, String host, int
    port) throws ImpalaRefresherException{
    LOG.debug("Refreshing with params: user={}, password={},
    host={}, port={}", user, password, host, port);
    Session session = null;
    Channel channel = null;

    try{
    JSch jsch=new JSch();
    session = jsch.getSession(user, host, port);
    session.setConfig("StrictHostKeyChecking", "no");
    session.setTimeout(1500);
    session.setPassword(password);
    session.connect();

    channel = session.openChannel("exec");
    InputStream stdout = channel.getInputStream();
    InputStream stderr = ((ChannelExec)channel).getErrStream();

    ((ChannelExec)channel).setCommand("impala-shell -i
    'localhost:21000' -q 'refresh web_resource_rating'");
    channel.connect();
    channel.run();

    String stdoutString = IOUtils.toString(stdout);
    LOG.info("stdout:{}", stdoutString);

    String stdErrString = IOUtils.toString(stderr);
    LOG.info("stderr:{}", stdErrString);

    if (stdoutString == null || stdoutString.isEmpty() ||
    !stdoutString.contains("Successfully refreshed table")){
    throw new ImpalaRefresherException(String.format("Looks
    like ssh exec failed: %s", stdErrString));
    }


    }catch (Exception e){
    LOG.error("Error while refreshing metadata on impalad
    daemon", e);
    throw new ImpalaRefresherException("Rethrown original
    exception", e);
    }finally {
    closeSilently(channel);
    closeSilently(session);
    }
    }

    public void closeSilently(Session session){
    try{
    if(session!=null && session.isConnected()){
    session.disconnect();
    }
    }catch (Exception e){
    LOG.error("Error while closing ssh session", e);
    }
    }

    public void closeSilently(Channel channel){
    try{
    if(channel!=null && channel.isConnected()){
    channel.disconnect();
    }
    }catch (Exception e){
    LOG.error("Error while closing ssh channel", e);
    }
    }
    }


    2013/5/20 Ishaan Joshi <ishaan@cloudera.com>
    Serega, Alex,

    At the moment, you have the explicitly refresh the impalad from the
    impala-shell. Note that the refresh roc will not automatically refresh each
    running impalad, just the one you're connected to.

    Thanks,

    -- Ishaan


    On Mon, May 20, 2013 at 12:17 AM, Serega Sheypak <
    serega.sheypak@gmail.com> wrote:
    The same problem! Is there any way to do it?
    I have a table which partitions are updated each hour. So I have to
    submit "refresh" in some way bwfore running query...


    понедельник, 1 апреля 2013 г., 19:55:10 UTC+4 пользователь Alex I
    написал:
    Hello,

    I want to invoke metadata refresh manually from a Java client.
    Currently, the JDBC driver prohibits me from doing that.
    I can look into impala-shell sources and do a refresh via Thrift/RPC
    in the same fashion - however, is there any recommended way for refreshing?

    Thanks,
    - Alex.

    --
    Ricky Saltzer
    Tools Developer
    http://www.cloudera.com


    --
    Ricky Saltzer
    Tools Developer
    http://www.cloudera.com
  • Ricky Saltzer at May 20, 2013 at 11:28 pm
    Hey -

    The second link won't work for you, here's the correct instructions :)

    $ git clone git://github.com/rickysaltzer/impala-refresher.git
    $ cd impala-refresher
    $ go build

    On Mon, May 20, 2013 at 5:37 PM, Ricky Saltzer wrote:

    Hey Serega -

    Here's a link to the repo, feel free to fork and contribute.

    https://github.com/rickysaltzer/impala-refresher

    You'll need the go compiler <http://golang.org/> to build it, you can
    re-distribute the binary once it's built though.

    $ git clone git://github.mtv.cloudera.com/ricky/impala-refresher.git
    $ cd impala-refresher
    $ go build

    Ricky



    On Mon, May 20, 2013 at 2:28 PM, Ricky Saltzer wrote:

    I have a 14 node cluster and running a refresh all at the same time
    doesn't seem to run any slower than doing one.


    On Mon, May 20, 2013 at 2:23 PM, Serega Sheypak <serega.sheypak@gmail.com
    wrote:
    Thanks. I'll be great. We have 19 impalad instances. Don't you think it
    can bring significant pressure on hive metastore?
    20.05.2013 20:09 пользователь "Ricky Saltzer" <ricky@cloudera.com>
    написал:

    Hey guys -
    I've recently wrote a program to do exactly this, it will refresh the
    metadata on a specific table across all
    the daemons concurrently. I'm in the process of cleaning the code up,
    I'll try and put it on github if
    any of you are interested.

    Ricky


    On Mon, May 20, 2013 at 1:35 PM, Serega Sheypak <
    serega.sheypak@gmail.com> wrote:
    Yes, I understand it. Just wanted to hear it from Cloudera before
    starting to invent the bicycle.
    I've created brand-new bicycle. It rides to random impalad taken from
    application configuration and refreshes metadata before giving JDBC
    connection to the same impalad.
    It's rather ugly, but works.

    public class ImpaladMetadataRefresher {

    private final Logger LOG =
    LoggerFactory.getLogger(ImpaladMetadataRefresher.class);

    public void refresh(String user, String password, String host, int
    port) throws ImpalaRefresherException{
    LOG.debug("Refreshing with params: user={}, password={},
    host={}, port={}", user, password, host, port);
    Session session = null;
    Channel channel = null;

    try{
    JSch jsch=new JSch();
    session = jsch.getSession(user, host, port);
    session.setConfig("StrictHostKeyChecking", "no");
    session.setTimeout(1500);
    session.setPassword(password);
    session.connect();

    channel = session.openChannel("exec");
    InputStream stdout = channel.getInputStream();
    InputStream stderr = ((ChannelExec)channel).getErrStream();

    ((ChannelExec)channel).setCommand("impala-shell -i
    'localhost:21000' -q 'refresh web_resource_rating'");
    channel.connect();
    channel.run();

    String stdoutString = IOUtils.toString(stdout);
    LOG.info("stdout:{}", stdoutString);

    String stdErrString = IOUtils.toString(stderr);
    LOG.info("stderr:{}", stdErrString);

    if (stdoutString == null || stdoutString.isEmpty() ||
    !stdoutString.contains("Successfully refreshed table")){
    throw new
    ImpalaRefresherException(String.format("Looks like ssh exec failed: %s",
    stdErrString));
    }


    }catch (Exception e){
    LOG.error("Error while refreshing metadata on impalad
    daemon", e);
    throw new ImpalaRefresherException("Rethrown original
    exception", e);
    }finally {
    closeSilently(channel);
    closeSilently(session);
    }
    }

    public void closeSilently(Session session){
    try{
    if(session!=null && session.isConnected()){
    session.disconnect();
    }
    }catch (Exception e){
    LOG.error("Error while closing ssh session", e);
    }
    }

    public void closeSilently(Channel channel){
    try{
    if(channel!=null && channel.isConnected()){
    channel.disconnect();
    }
    }catch (Exception e){
    LOG.error("Error while closing ssh channel", e);
    }
    }
    }


    2013/5/20 Ishaan Joshi <ishaan@cloudera.com>
    Serega, Alex,

    At the moment, you have the explicitly refresh the impalad from the
    impala-shell. Note that the refresh roc will not automatically refresh each
    running impalad, just the one you're connected to.

    Thanks,

    -- Ishaan


    On Mon, May 20, 2013 at 12:17 AM, Serega Sheypak <
    serega.sheypak@gmail.com> wrote:
    The same problem! Is there any way to do it?
    I have a table which partitions are updated each hour. So I have to
    submit "refresh" in some way bwfore running query...


    понедельник, 1 апреля 2013 г., 19:55:10 UTC+4 пользователь Alex I
    написал:
    Hello,

    I want to invoke metadata refresh manually from a Java client.
    Currently, the JDBC driver prohibits me from doing that.
    I can look into impala-shell sources and do a refresh via
    Thrift/RPC in the same fashion - however, is there any recommended way for
    refreshing?

    Thanks,
    - Alex.

    --
    Ricky Saltzer
    Tools Developer
    http://www.cloudera.com


    --
    Ricky Saltzer
    Tools Developer
    http://www.cloudera.com


    --
    Ricky Saltzer
    Tools Developer
    http://www.cloudera.com


    --
    Ricky Saltzer
    Tools Developer
    http://www.cloudera.com

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupimpala-user @
categorieshadoop
postedApr 1, '13 at 3:55p
activeMay 20, '13 at 11:28p
posts7
users4
websitecloudera.com
irc#hadoop

People

Translate

site design / logo © 2022 Grokbase