[HBase-dev] write performance degradation

Can you come up on release 0.90.0 Ted?

17MB client buffer might be a little large.

What do you see in your ganglia? Anything else running on cluster at
the time? What are the machines doing? CPU, iowait? Swapping?

Thanks,
St.Ack

RC3 was promoted to 0.90 release, so I didn't expect much difference.
I also tried 0.90 build which also included newer patches such as HBASE-3455
and observed same result.

The cluster only runs one job at a time.

Client buffer size for our 0.20.6 cluster was set to 20MB and I didn't see
similar issue.

Here is output from top on one node:

top - 22:42:26 up 104 days, 20:38, 1 user, load average: 1.04, 1.34, 2.13
Tasks: 152 total, 1 running, 151 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.5%us, 0.2%sy, 0.0%ni, 98.8%id, 0.0%wa, 0.0%hi, 0.4%si,
0.0%st
Mem: 16440036k total, 15532616k used, 907420k free, 42472k buffers
Swap: 16777208k total, 5719488k used, 11057720k free, 4530368k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+
COMMAND

15261 hadoop 25 0 4776m 1.0g 9312 S 2.0 6.7 133:27.04 java
16442 hadoop 21 0 1625m 760m 14m S 1.3 4.7 0:39.28
java

16453 hadoop 18 0 4690m 3.4g 9712 S 1.3 21.4 311:49.25 java

15261 DataNode
19442 map task
16453 HRegionServer

---------
[sjc1-hadoop3.sjc1:hadoop 1298]iostat
Linux 2.6.18-53.el5 (sjc1-hadoop3.sjc1.carrieriq.com) 01/30/2011

avg-cpu: %user %nice %system %iowait %steal %idle
4.56 0.00 0.48 0.60 0.00 94.35

Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
sda 6.15 403.62 411.58 3656841492 3728977064
sdb 7.64 131.10 303.53 1187817806 2750029496
sdc 7.38 396.23 387.12 3589886886 3507388096
sdd 6.35 403.99 399.77 3660260454 3621972144
dm-0 22.51 37.78 173.52 342296914 1572156872
dm-1 27.92 93.32 130.01 845519592 1177872624

Load is low.

Is this swap active? Are you swapping?

Regards the rest, its hard to offer advice based off a single
snapshot. Do you have ganglia up and running on this cluster?

St.Ack

I will setup Ganglia on the dev cluster.

There wasn't sign of GC pause from the region server logs BTW

It could be useful to see the output of "vmstat 1" (let it run for at
least 30 seconds while the problem is occurring and pastebin the
result). If the problem seems i/o related, the output of "iostat -xkd
1" can also help.

Right now my bigger concern is the following:

java.io.IOException: All datanodes 10.202.50.79:50010 are bad. Aborting...
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2603)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$1600(DFSClient.java:2139)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2306)
2011-01-31 05:42:36,927 WARN org.apache.hadoop.mapred.TaskTracker:
Error running child
java.lang.RuntimeException: java.io.IOException: All datanodes
10.202.50.79:50010 are bad. Aborting...

hadoop-core-0.20-append-r1056497.jar is newer than the jar from cdh3b2.
Some user is using it but I haven't got it working on cdh3b2 installation.

Here is output from vmstat:
http://pastebin.com/VeGC5KQr

Here is output from iostat:
http://pastebin.com/iCMNSZcm

This indicates that you're actively bringing in things from swap,
which kills performance. The columns "si" (swap in) and "so" (swap
out) should ideally always be 0.

What's printed by /sbin/sysctl vm.swappiness ?

Thanks for the reminder.

After adjusting swappiness to 2 on each node, I see:
http://pastebin.com/tFcAj7rw

Client buffer is now 10MB.
But map tasks seem to be slower than before.

The output of vmstat doesn't show anything useful here, the machine is
100% idle up until the 3rd section where it's reading a bit from disk,
but it's still mostly idle. Something else that we don't see from
this trace is amiss.