[vcap-dev] All the gateway of services can not work suddenly when I re-configured the mbus.

On Jun 18, 5:36 pm, Chunjie Zhu wrote:
"update_varz" is a periodical operation, and, the timer may alarm at the Kernel.exit (https://github.com/cloudfoundry/vcap-services-base/blob/master/lib/ba...) time. Kernel.exit is just like POSIX atexit. So, you see, at the application termination time, both "send deactivation" action & "update_varz" action happen ed (I infer this from your log), then something unexpected, I mean, the "Nil Class" exception comes out.

BTW, "register" & "fetching handles" communication between CC and Gateway is not piggybacked on NATS but through HTTP, so, one suggestion is, please make sure that the HTTP channel between CC and Gateway is working properly.

Regards,
Chunjie







----- Original Message -----

From: "张 磊" <[email protected]>
To: "vcap-dev" <[email protected]>
Sent: Monday, June 18, 2012 4:47:41 PM
Subject: [vcap-dev] All the gateway of services can not work suddenly when I re-configured the mbus.

As I were doing some experiments about muti-node CF, I needed to
change the binded IP of NATS in the config files of
mysql,redis,rabbitmq ... gateway. For example, in order to connect the
gateways to my new NATS node, I change the original mbus:xxx.xxx.xxx.
219 to xxx.xxx.xxx.229. This works.

But when I restore the IP back from 229 to 219, I found all the
gateways can not work at all. And in their logs, there is the same
error:

[2012-06-18 15:55:41.756737] mysql_gateway - pid=28002 tid=64a0
fid=25fd INFO -- Sending info to cloud controller:http://api.vcap.me/services/v1/offerings
[2012-06-18 15:55:46.763478] mysql_gateway - pid=28002 tid=64a0
fid=25fd INFO -- Fetching handles from cloud controller @http://api.vcap.me/services/v1/offerings/mysql-5.1/handles
[2012-06-18 15:55:51.765531] mysql_gateway - pid=28002 tid=64a0
fid=25fd ERROR -- Failed registering with cloud controller:
[2012-06-18 15:55:51.765829] mysql_gateway - pid=28002 tid=64a0
fid=25fd ERROR -- Failed fetching handles:
[2012-06-18 15:55:51.766127] mysql_gateway - pid=28002 tid=64a0
fid=25fd INFO -- Sending deactivation notice to cloud controller:http://api.vcap.me/services/v1/offerings
[2012-06-18 15:55:56.800292] mysql_gateway - pid=28002 tid=64a0
fid=25fd INFO -- Successfully deactivated with cloud controller
/root/cloudfoundry/.deployments/devbox/deploy/rubies/ruby-1.9.2-p180/
lib/ruby/gems/1.9.1/gems/vcap_services_base-0.1.9/lib/base/base.rb:
84:in `block in update_varz': undefined method `[]=' for nil:NilClass
(NoMethodError)
from /root/cloudfoundry/.deployments/devbox/deploy/rubies/
ruby-1.9.2-p180/lib/ruby/gems/1.9.1/gems/vcap_services_base-0.1.9/lib/
base/base.rb:83:in `each'
from /root/cloudfoundry/.deployments/devbox/deploy/rubies/
ruby-1.9.2-p180/lib/ruby/gems/1.9.1/gems/vcap_services_base-0.1.9/lib/
base/base.rb:83:in `update_varz'
from /root/cloudfoundry/.deployments/devbox/deploy/rubies/
ruby-1.9.2-p180/lib/ruby/gems/1.9.1/gems/vcap_services_base-0.1.9/lib/
base/provisioner.rb:43:in `block in initialize'
from /root/cloudfoundry/.deployments/devbox/deploy/rubies/
ruby-1.9.2-p180/lib/ruby/gems/1.9.1/gems/
eventmachine-0.12.11.cloudfoundry.3/lib/eventmachine.rb:256:in `call'
from /root/cloudfoundry/.deployments/devbox/deploy/rubies/
ruby-1.9.2-p180/lib/ruby/gems/1.9.1/gems/
eventmachine-0.12.11.cloudfoundry.3/lib/eventmachine.rb:256:in
`run_machine'
from /root/cloudfoundry/.deployments/devbox/deploy/rubies/
ruby-1.9.2-p180/lib/ruby/gems/1.9.1/gems/
eventmachine-0.12.11.cloudfoundry.3/lib/eventmachine.rb:256:in `run'
from /root/cloudfoundry/.deployments/devbox/deploy/rubies/
ruby-1.9.2-p180/lib/ruby/gems/1.9.1/gems/vcap_services_base-0.1.9/lib/
base/gateway.rb:94:in `start'
from /root/cloudfoundry/vcap/services/mysql/bin/mysql_gateway:
32:in `<main>'

I have tried to install CF again,but that does not works. And I think
this is caused by my re-configuring because for those gateways which
I do not changed mbus IP, there're no this error.

P.S. Maybe gateways and CC or NATS can not be running on different
nodes?

On Tue, Jun 19, 2012 at 9:54 AM, 张 磊 wrote:

Thank you for your reply. But how can I find out the status of HTTP
channel between CC and Gateway? I'm not very clear about that.
Kernel.exit (
https://github.com/cloudfoundry/vcap-services-base/blob/master/lib/ba...)
time. Kernel.exit is just like POSIX atexit. So, you see, at the
application termination time, both "send deactivation" action &
"update_varz" action happen ed (I infer this from your log), then something
unexpected, I mean, the "Nil Class" exception comes out.
Gateway is not piggybacked on NATS but through HTTP, so, one suggestion is,
please make sure that the HTTP channel between CC and Gateway is working
properly.

On Jun 19, 12:56 pm, Chunjie Zhu wrote:
The intuitive way is to watch cloud controller's log, see whether it can
receive service gateway's request.

Another suggestion, have a try to modify source code, such as "print more
debug log" during service gateway initialization time, then you may get
some hint to fix the issue.

Regards,
Chunjie








--
Regards,
Chunjie

On Jun 19, 12:56 pm, Chunjie Zhu wrote:
The intuitive way is to watch cloud controller's log, see whether it can
receive service gateway's request.

Another suggestion, have a try to modify source code, such as "print more
debug log" during service gateway initialization time, then you may get
some hint to fix the issue.

Regards,
Chunjie








--
Regards,
Chunjie

On Jun 20, 9:45 am, Chunjie Zhu wrote:
There are no "service gateway" requests in the cloud_controller log, seems it totally does not work.

Here are some general trouble shooting steps, for your reference.
1. see whether cloud controller is ready to accept the requests
telnet <your cc host> 9022 # 9022 is the default public port for CC

2. see whether router is really working, each request will first arrive at router then be forwarded to its destine
grep "Registering api.vcap.me at <your cc host>:9022" in router.log # not present then something wrong

3. see whether your firewall policy is really correct
#depends on your network settings

4. api.vcap.me is mapped to "127.0.0.1", VMware takes acquisition of this domain name, so if your cc and service gateway are not on the same machine, then there may be something wrong

Regards,
Chunjie







----- Original Message -----

From: "张 磊" <[email protected]>
To: "vcap-dev" <[email protected]>
Sent: Tuesday, June 19, 2012 4:07:12 PM
Subject: [vcap-dev] Re: All the gateway of services can not work suddenly when I re-configured the mbus.

WARNING: 'require 'rake/rdoctask'' is deprecated. Please use 'require
'rdoc/task' (in RDoc 2.4.2+)' instead.
at /root/cloudfoundry/.deployments/devbox/deploy/rubies/ruby-1.9.2-
p180/lib/ruby/gems/1.9.1/gems/rake-0.9.2.2/lib/rake/rdoctask.rb
WARNING: Global access to Rake DSL methods is deprecated. Please
include
... Rake::DSL into classes and modules which use the Rake DSL
methods.
WARNING: DSL method CloudController::Application#task called at /root/
cloudfoundry/.deployments/devbox/deploy/rubies/ruby-1.9.2-p180/lib/
ruby/gems/1.9.1/gems/railties-3.0.5/lib/rails/application.rb:214:in
`initialize_tasks'
Rails Error: Unable to access log file. Please ensure that exists and
is chmod 0666. The log level has been raised to WARN and the output
directed to STDERR until the problem is fixed.
Rails Error: Unable to access log file. Please ensure that exists and
is chmod 0666. The log level has been raised to WARN and the output
directed to STDERR until the problem is fixed.
[2012-06-19 13:19:47.479294] cc_events - pid=12661 tid=3a38 fid=a23e
INFO -- [2012-06-19 13:19:47 +0800, :SYSTEM, "Starting VCAP
CloudController (0.999)"]
[2012-06-19 13:19:47.483766] cc_events - pid=12661 tid=3a38 fid=a23e
INFO -- [2012-06-19 13:19:47 +0800, :SYSTEM, "Socket Limit:32768"]

...

other service node msg except gateways

...

I have looked into the asynchronous_service_gateway.rb and try to
print more info about the object "http" to the log file. But I got
nothing. So I can not figure out where is wrong. It is true that HTTP
request between CC and gateways is not working.

How can I make sure whether the request create by Gateway is valid and
send to CC? And then I can track that request.