Paul Baker (b...@google.com)

The states output is from only one perlbal instance, so the real numbers are
basically double that. Here is my basic config:

CREATE SERVICE feeds
  SET role                  = reverse_proxy
  SET pool                  = feedspool
  SET persist_client        = on
  SET persist_backend       = on
  SET verify_backend        = on
  SET connect_ahead         = 50
  SET enable_error_retries  = on
  SET backend_persist_cache = 20
  SET balance_method        = random
ENABLE feeds

I have my backends configured to handle 275 threads each (which if they
actually did serve that many at once would explode) because I haven't really
optimized that number for "the perlbal way". Perlbal has yet to actually
connect that many to a backend by a long shot. Does it make sense for me to
go with a large connect_ahead and/or backend_persist_cache? I'm still a bit
unclear on what specifically each does? Or should I stick with the numbers I
have? Should they be the same number? I figured that setting it high would
just waste more cpu cycles if Perlbal is constantly iterating through a
bunch of bored connections deciding whether to keep them connected or not.

So I'm giving Perlbal a taste of some production traffic, and I've basically
topped out at 40Mbit/s and 700req/s. At least I think it's topped out at
that since the CPU is pegged at 100%. I'm running on a Xeon 2.8Ghz with
800Mhz FSB and 2MB L2 cache. 3GB of RAM although Perlbal is only using
130MB. GigE nic. I'm running on Debian Etch with the 2.6.18-5-amd64 kernel
(pseudo 64-bit baby!) I have 14 backend nodes. Since the AMD64 kernel sees
two cpu's (because of hyperthreading) I'm actually running two perlbal
instances configured exactly the same except for different listening ports.
(is this good?) Traffic is sent to the two perlbal instances by a pair of
NetScaler 9000s which are also in "reverse_proxy" mode.

Here is a 10-second sample of prof data:
Perlbal::BackendHTTP-read 2.32415 0.40803 3927 0.0005918 0.0001039
Perlbal::BackendHTTP-write 0.47603 0.08401 723 0.0006584 0.0001162
Perlbal::ClientHTTPBase-err 0.00400 0.00000 20 0.0002000 0.0000000
Perlbal::ClientHTTPBase-read 2.98418 0.59204 3613 0.0008260 0.0001639
Perlbal::ClientManage-read 0.00000 0.00000 1 0.0000000 0.0000000
Perlbal::ClientProxy-read 0.00800 0.00400 14 0.0005714 0.0002858
Perlbal::ClientProxy-write 0.02400 0.01600 97 0.0002475 0.0001650
Perlbal::TCPListener-read 0.14801 0.03200 50 0.0029602 0.0006400

States ouput:
Perlbal::BackendHTTP bored 41
Perlbal::BackendHTTP closed 1
Perlbal::BackendHTTP connecting 7
Perlbal::BackendHTTP wait_res 133
Perlbal::BackendHTTP xfer_res 16
Perlbal::ClientHTTPBase persist_wait 546
Perlbal::ClientHTTPBase reading_headers 13
Perlbal::ClientProxy closed 1
Perlbal::ClientProxy draining_res 13
Perlbal::ClientProxy wait_res 133
Perlbal::ClientProxy xfer_res 16

I am using the XS Headers module, xs output:
XS module status:
   headers: installed, enabled

Traffic graph: http://givetheworld.net/graph_image-1.png
CPU graph: http://givetheworld.net/graph_image-2.png

Is this the performance level I should expect? Or am I doing something
wrong?

-Paul

I'm letting perlbal reverse_proxying some production traffic and out
of the about 200,000 requests in 40 minutes it has served, perlbal has
spit out 28 of these warnings:

Undef client_ip (Perlbal::ClientProxy=ARRAY(0x1943fb0)) in
assign_client.  Closing. at
/usr/local/share/perl/5.8.8/Perlbal/BackendHTTP.pm line 247.

Is this something I should be worried about?

I couldn't find this search the list archive and I tired reading the code to
see what happens but I haven't been able to put it together in my head yet.
Basically I want to know exactly what happens to the connections a node has
when it is removed from a pool, either through the command interface or from
a nodelist.dat file sync. If there are connections in wait_res or xfer_res
for instance, are they allowed to gracefully complete, or are they severed
immediately (and an error sent to the client?)