How to invoke refresh metadata on the client side

Serega, Alex,

   At the moment, you have the explicitly refresh the impalad from the
impala-shell. Note that the refresh roc will not automatically refresh each
running impalad, just the one you're connected to.

Thanks,

-- Ishaan


On Mon, May 20, 2013 at 12:17 AM, Serega Sheypak
wrote:

Yes, I understand it. Just wanted to hear it from Cloudera before starting
to invent the bicycle.
I've created brand-new bicycle. It rides to random impalad taken from
application configuration and refreshes metadata before giving JDBC
connection to the same impalad.
It's rather ugly, but works.

public class ImpaladMetadataRefresher {

     private final Logger LOG =
LoggerFactory.getLogger(ImpaladMetadataRefresher.class);

     public void refresh(String user, String password, String host, int
port) throws ImpalaRefresherException{
         LOG.debug("Refreshing with params: user={}, password={}, host={},
port={}", user, password, host, port);
         Session session = null;
         Channel channel = null;

         try{
             JSch jsch=new JSch();
             session = jsch.getSession(user, host, port);
             session.setConfig("StrictHostKeyChecking", "no");
             session.setTimeout(1500);
             session.setPassword(password);
             session.connect();

             channel = session.openChannel("exec");
             InputStream stdout = channel.getInputStream();
             InputStream stderr = ((ChannelExec)channel).getErrStream();

             ((ChannelExec)channel).setCommand("impala-shell -i
'localhost:21000' -q 'refresh web_resource_rating'");
             channel.connect();
             channel.run();

             String stdoutString = IOUtils.toString(stdout);
             LOG.info("stdout:{}", stdoutString);

             String stdErrString = IOUtils.toString(stderr);
             LOG.info("stderr:{}", stdErrString);

             if (stdoutString == null || stdoutString.isEmpty() ||
!stdoutString.contains("Successfully refreshed table")){
                 throw new ImpalaRefresherException(String.format("Looks
like ssh exec failed: %s", stdErrString));
             }


         }catch (Exception e){
             LOG.error("Error while refreshing metadata on impalad daemon",
e);
             throw new ImpalaRefresherException("Rethrown original
exception", e);
         }finally {
             closeSilently(channel);
             closeSilently(session);
         }
     }

     public void closeSilently(Session session){
         try{
             if(session!=null && session.isConnected()){
                 session.disconnect();
             }
         }catch (Exception e){
             LOG.error("Error while closing ssh session", e);
         }
     }

     public void closeSilently(Channel channel){
         try{
             if(channel!=null && channel.isConnected()){
                 channel.disconnect();
             }
         }catch (Exception e){
             LOG.error("Error while closing ssh channel", e);
         }
     }
}


2013/5/20 Ishaan Joshi <ishaan@cloudera.com>

Hey guys -

I've recently wrote a program to do exactly this, it will refresh the
metadata on a specific table across all
the daemons concurrently. I'm in the process of cleaning the code up, I'll
try and put it on github if
any of you are interested.

Ricky

Thanks. I'll be great. We have 19 impalad instances. Don't you think it can
bring significant pressure on hive metastore?
  20.05.2013 20:09 пользователь "Ricky Saltzer" <ricky@cloudera.com> написал:

I have a 14 node cluster and running a refresh all at the same time doesn't
seem to run any slower than doing one.

Hey -

The second link won't work for you, here's the correct instructions :)

$ git clone git://github.com/rickysaltzer/impala-refresher.git
$ cd impala-refresher
$ go build