FAQ
As I were doing some experiments about muti-node CF, I needed to
change the binded IP of NATS in the config files of
mysql,redis,rabbitmq ... gateway. For example, in order to connect the
gateways to my new NATS node, I change the original mbus:xxx.xxx.xxx.
219 to xxx.xxx.xxx.229. This works.

But when I restore the IP back from 229 to 219, I found all the
gateways can not work at all. And in their logs, there is the same
error:

[2012-06-18 15:55:41.756737] mysql_gateway - pid=28002 tid=64a0
fid=25fd INFO -- Sending info to cloud controller:
http://api.vcap.me/services/v1/offerings
[2012-06-18 15:55:46.763478] mysql_gateway - pid=28002 tid=64a0
fid=25fd INFO -- Fetching handles from cloud controller @
http://api.vcap.me/services/v1/offerings/mysql-5.1/handles
[2012-06-18 15:55:51.765531] mysql_gateway - pid=28002 tid=64a0
fid=25fd ERROR -- Failed registering with cloud controller:
[2012-06-18 15:55:51.765829] mysql_gateway - pid=28002 tid=64a0
fid=25fd ERROR -- Failed fetching handles:
[2012-06-18 15:55:51.766127] mysql_gateway - pid=28002 tid=64a0
fid=25fd INFO -- Sending deactivation notice to cloud controller:
http://api.vcap.me/services/v1/offerings
[2012-06-18 15:55:56.800292] mysql_gateway - pid=28002 tid=64a0
fid=25fd INFO -- Successfully deactivated with cloud controller
/root/cloudfoundry/.deployments/devbox/deploy/rubies/ruby-1.9.2-p180/
lib/ruby/gems/1.9.1/gems/vcap_services_base-0.1.9/lib/base/base.rb:
84:in `block in update_varz': undefined method `[]=' for nil:NilClass
(NoMethodError)
from /root/cloudfoundry/.deployments/devbox/deploy/rubies/
ruby-1.9.2-p180/lib/ruby/gems/1.9.1/gems/vcap_services_base-0.1.9/lib/
base/base.rb:83:in `each'
from /root/cloudfoundry/.deployments/devbox/deploy/rubies/
ruby-1.9.2-p180/lib/ruby/gems/1.9.1/gems/vcap_services_base-0.1.9/lib/
base/base.rb:83:in `update_varz'
from /root/cloudfoundry/.deployments/devbox/deploy/rubies/
ruby-1.9.2-p180/lib/ruby/gems/1.9.1/gems/vcap_services_base-0.1.9/lib/
base/provisioner.rb:43:in `block in initialize'
from /root/cloudfoundry/.deployments/devbox/deploy/rubies/
ruby-1.9.2-p180/lib/ruby/gems/1.9.1/gems/
eventmachine-0.12.11.cloudfoundry.3/lib/eventmachine.rb:256:in `call'
from /root/cloudfoundry/.deployments/devbox/deploy/rubies/
ruby-1.9.2-p180/lib/ruby/gems/1.9.1/gems/
eventmachine-0.12.11.cloudfoundry.3/lib/eventmachine.rb:256:in
`run_machine'
from /root/cloudfoundry/.deployments/devbox/deploy/rubies/
ruby-1.9.2-p180/lib/ruby/gems/1.9.1/gems/
eventmachine-0.12.11.cloudfoundry.3/lib/eventmachine.rb:256:in `run'
from /root/cloudfoundry/.deployments/devbox/deploy/rubies/
ruby-1.9.2-p180/lib/ruby/gems/1.9.1/gems/vcap_services_base-0.1.9/lib/
base/gateway.rb:94:in `start'
from /root/cloudfoundry/vcap/services/mysql/bin/mysql_gateway:
32:in `<main>'

I have tried to install CF again,but that does not works. And I think
this is caused by my re-configuring because for those gateways which
I do not changed mbus IP, there're no this error.

P.S. Maybe gateways and CC or NATS can not be running on different
nodes?

Search Discussions

  • Chunjie Zhu at Jun 18, 2012 at 9:36 am
    "update_varz" is a periodical operation, and, the timer may alarm at the Kernel.exit (https://github.com/cloudfoundry/vcap-services-base/blob/master/lib/base/asynchronous_service_gateway.rb#L89) time. Kernel.exit is just like POSIX atexit. So, you see, at the application termination time, both "send deactivation" action & "update_varz" action happen ed (I infer this from your log), then something unexpected, I mean, the "Nil Class" exception comes out.

    BTW, "register" & "fetching handles" communication between CC and Gateway is not piggybacked on NATS but through HTTP, so, one suggestion is, please make sure that the HTTP channel between CC and Gateway is working properly.

    Regards,
    Chunjie

    ----- Original Message -----

    From: "张 磊" <resouer@163.com>
    To: "vcap-dev" <vcap-dev@cloudfoundry.org>
    Sent: Monday, June 18, 2012 4:47:41 PM
    Subject: [vcap-dev] All the gateway of services can not work suddenly when I re-configured the mbus.

    As I were doing some experiments about muti-node CF, I needed to
    change the binded IP of NATS in the config files of
    mysql,redis,rabbitmq ... gateway. For example, in order to connect the
    gateways to my new NATS node, I change the original mbus:xxx.xxx.xxx.
    219 to xxx.xxx.xxx.229. This works.

    But when I restore the IP back from 229 to 219, I found all the
    gateways can not work at all. And in their logs, there is the same
    error:

    [2012-06-18 15:55:41.756737] mysql_gateway - pid=28002 tid=64a0
    fid=25fd INFO -- Sending info to cloud controller:
    http://api.vcap.me/services/v1/offerings
    [2012-06-18 15:55:46.763478] mysql_gateway - pid=28002 tid=64a0
    fid=25fd INFO -- Fetching handles from cloud controller @
    http://api.vcap.me/services/v1/offerings/mysql-5.1/handles
    [2012-06-18 15:55:51.765531] mysql_gateway - pid=28002 tid=64a0
    fid=25fd ERROR -- Failed registering with cloud controller:
    [2012-06-18 15:55:51.765829] mysql_gateway - pid=28002 tid=64a0
    fid=25fd ERROR -- Failed fetching handles:
    [2012-06-18 15:55:51.766127] mysql_gateway - pid=28002 tid=64a0
    fid=25fd INFO -- Sending deactivation notice to cloud controller:
    http://api.vcap.me/services/v1/offerings
    [2012-06-18 15:55:56.800292] mysql_gateway - pid=28002 tid=64a0
    fid=25fd INFO -- Successfully deactivated with cloud controller
    /root/cloudfoundry/.deployments/devbox/deploy/rubies/ruby-1.9.2-p180/
    lib/ruby/gems/1.9.1/gems/vcap_services_base-0.1.9/lib/base/base.rb:
    84:in `block in update_varz': undefined method `[]=' for nil:NilClass
    (NoMethodError)
    from /root/cloudfoundry/.deployments/devbox/deploy/rubies/
    ruby-1.9.2-p180/lib/ruby/gems/1.9.1/gems/vcap_services_base-0.1.9/lib/
    base/base.rb:83:in `each'
    from /root/cloudfoundry/.deployments/devbox/deploy/rubies/
    ruby-1.9.2-p180/lib/ruby/gems/1.9.1/gems/vcap_services_base-0.1.9/lib/
    base/base.rb:83:in `update_varz'
    from /root/cloudfoundry/.deployments/devbox/deploy/rubies/
    ruby-1.9.2-p180/lib/ruby/gems/1.9.1/gems/vcap_services_base-0.1.9/lib/
    base/provisioner.rb:43:in `block in initialize'
    from /root/cloudfoundry/.deployments/devbox/deploy/rubies/
    ruby-1.9.2-p180/lib/ruby/gems/1.9.1/gems/
    eventmachine-0.12.11.cloudfoundry.3/lib/eventmachine.rb:256:in `call'
    from /root/cloudfoundry/.deployments/devbox/deploy/rubies/
    ruby-1.9.2-p180/lib/ruby/gems/1.9.1/gems/
    eventmachine-0.12.11.cloudfoundry.3/lib/eventmachine.rb:256:in
    `run_machine'
    from /root/cloudfoundry/.deployments/devbox/deploy/rubies/
    ruby-1.9.2-p180/lib/ruby/gems/1.9.1/gems/
    eventmachine-0.12.11.cloudfoundry.3/lib/eventmachine.rb:256:in `run'
    from /root/cloudfoundry/.deployments/devbox/deploy/rubies/
    ruby-1.9.2-p180/lib/ruby/gems/1.9.1/gems/vcap_services_base-0.1.9/lib/
    base/gateway.rb:94:in `start'
    from /root/cloudfoundry/vcap/services/mysql/bin/mysql_gateway:
    32:in `<main>'

    I have tried to install CF again,but that does not works. And I think
    this is caused by my re-configuring because for those gateways which
    I do not changed mbus IP, there're no this error.

    P.S. Maybe gateways and CC or NATS can not be running on different
    nodes?
  • 张 磊 at Jun 19, 2012 at 1:55 am
    Thank you for your reply. But how can I find out the status of HTTP
    channel between CC and Gateway? I'm not very clear about that.
    On Jun 18, 5:36 pm, Chunjie Zhu wrote:
    "update_varz" is a periodical operation, and, the timer may alarm at the Kernel.exit (https://github.com/cloudfoundry/vcap-services-base/blob/master/lib/ba...) time. Kernel.exit is just like POSIX atexit. So, you see, at the application termination time, both "send deactivation" action & "update_varz" action happen ed (I infer this from your log), then something unexpected, I mean, the "Nil Class" exception comes out.

    BTW, "register" & "fetching handles" communication between CC and Gateway is not piggybacked on NATS but through HTTP, so, one suggestion is, please make sure that the HTTP channel between CC and Gateway is working properly.

    Regards,
    Chunjie







    ----- Original Message -----

    From: "张 磊" <reso...@163.com>
    To: "vcap-dev" <vcap-...@cloudfoundry.org>
    Sent: Monday, June 18, 2012 4:47:41 PM
    Subject: [vcap-dev] All the gateway of services can not work suddenly when I re-configured the mbus.

    As I were doing some experiments about muti-node CF, I needed to
    change the binded IP of NATS in the config files of
    mysql,redis,rabbitmq ... gateway. For example, in order to connect the
    gateways to my new NATS node, I change the original mbus:xxx.xxx.xxx.
    219 to xxx.xxx.xxx.229. This works.

    But when I restore the IP back from 229 to 219, I found all the
    gateways can not work at all. And in their logs, there is the same
    error:

    [2012-06-18 15:55:41.756737] mysql_gateway - pid=28002 tid=64a0
    fid=25fd INFO -- Sending info to cloud controller:http://api.vcap.me/services/v1/offerings
    [2012-06-18 15:55:46.763478] mysql_gateway - pid=28002 tid=64a0
    fid=25fd INFO -- Fetching handles from cloud controller @http://api.vcap.me/services/v1/offerings/mysql-5.1/handles
    [2012-06-18 15:55:51.765531] mysql_gateway - pid=28002 tid=64a0
    fid=25fd ERROR -- Failed registering with cloud controller:
    [2012-06-18 15:55:51.765829] mysql_gateway - pid=28002 tid=64a0
    fid=25fd ERROR -- Failed fetching handles:
    [2012-06-18 15:55:51.766127] mysql_gateway - pid=28002 tid=64a0
    fid=25fd INFO -- Sending deactivation notice to cloud controller:http://api.vcap.me/services/v1/offerings
    [2012-06-18 15:55:56.800292] mysql_gateway - pid=28002 tid=64a0
    fid=25fd INFO -- Successfully deactivated with cloud controller
    /root/cloudfoundry/.deployments/devbox/deploy/rubies/ruby-1.9.2-p180/
    lib/ruby/gems/1.9.1/gems/vcap_services_base-0.1.9/lib/base/base.rb:
    84:in `block in update_varz': undefined method `[]=' for nil:NilClass
    (NoMethodError)
    from /root/cloudfoundry/.deployments/devbox/deploy/rubies/
    ruby-1.9.2-p180/lib/ruby/gems/1.9.1/gems/vcap_services_base-0.1.9/lib/
    base/base.rb:83:in `each'
    from /root/cloudfoundry/.deployments/devbox/deploy/rubies/
    ruby-1.9.2-p180/lib/ruby/gems/1.9.1/gems/vcap_services_base-0.1.9/lib/
    base/base.rb:83:in `update_varz'
    from /root/cloudfoundry/.deployments/devbox/deploy/rubies/
    ruby-1.9.2-p180/lib/ruby/gems/1.9.1/gems/vcap_services_base-0.1.9/lib/
    base/provisioner.rb:43:in `block in initialize'
    from /root/cloudfoundry/.deployments/devbox/deploy/rubies/
    ruby-1.9.2-p180/lib/ruby/gems/1.9.1/gems/
    eventmachine-0.12.11.cloudfoundry.3/lib/eventmachine.rb:256:in `call'
    from /root/cloudfoundry/.deployments/devbox/deploy/rubies/
    ruby-1.9.2-p180/lib/ruby/gems/1.9.1/gems/
    eventmachine-0.12.11.cloudfoundry.3/lib/eventmachine.rb:256:in
    `run_machine'
    from /root/cloudfoundry/.deployments/devbox/deploy/rubies/
    ruby-1.9.2-p180/lib/ruby/gems/1.9.1/gems/
    eventmachine-0.12.11.cloudfoundry.3/lib/eventmachine.rb:256:in `run'
    from /root/cloudfoundry/.deployments/devbox/deploy/rubies/
    ruby-1.9.2-p180/lib/ruby/gems/1.9.1/gems/vcap_services_base-0.1.9/lib/
    base/gateway.rb:94:in `start'
    from /root/cloudfoundry/vcap/services/mysql/bin/mysql_gateway:
    32:in `<main>'

    I have tried to install CF again,but that does not works. And I think
    this is caused by my re-configuring because for those gateways which
    I do not changed mbus IP, there're no this error.

    P.S. Maybe gateways and CC or NATS can not be running on different
    nodes?
  • Chunjie Zhu at Jun 19, 2012 at 4:56 am
    The intuitive way is to watch cloud controller's log, see whether it can
    receive service gateway's request.

    Another suggestion, have a try to modify source code, such as "print more
    debug log" during service gateway initialization time, then you may get
    some hint to fix the issue.

    Regards,
    Chunjie
    On Tue, Jun 19, 2012 at 9:54 AM, 张 磊 wrote:

    Thank you for your reply. But how can I find out the status of HTTP
    channel between CC and Gateway? I'm not very clear about that.
    On Jun 18, 5:36 pm, Chunjie Zhu wrote:
    "update_varz" is a periodical operation, and, the timer may alarm at the
    Kernel.exit (
    https://github.com/cloudfoundry/vcap-services-base/blob/master/lib/ba...)
    time. Kernel.exit is just like POSIX atexit. So, you see, at the
    application termination time, both "send deactivation" action &
    "update_varz" action happen ed (I infer this from your log), then something
    unexpected, I mean, the "Nil Class" exception comes out.
    BTW, "register" & "fetching handles" communication between CC and
    Gateway is not piggybacked on NATS but through HTTP, so, one suggestion is,
    please make sure that the HTTP channel between CC and Gateway is working
    properly.
    Regards,
    Chunjie







    ----- Original Message -----

    From: "张 磊" <reso...@163.com>
    To: "vcap-dev" <vcap-...@cloudfoundry.org>
    Sent: Monday, June 18, 2012 4:47:41 PM
    Subject: [vcap-dev] All the gateway of services can not work suddenly
    when I re-configured the mbus.
    As I were doing some experiments about muti-node CF, I needed to
    change the binded IP of NATS in the config files of
    mysql,redis,rabbitmq ... gateway. For example, in order to connect the
    gateways to my new NATS node, I change the original mbus:xxx.xxx.xxx.
    219 to xxx.xxx.xxx.229. This works.

    But when I restore the IP back from 229 to 219, I found all the
    gateways can not work at all. And in their logs, there is the same
    error:

    [2012-06-18 15:55:41.756737] mysql_gateway - pid=28002 tid=64a0
    fid=25fd INFO -- Sending info to cloud controller:
    http://api.vcap.me/services/v1/offerings
    [2012-06-18 15:55:46.763478] mysql_gateway - pid=28002 tid=64a0
    fid=25fd INFO -- Fetching handles from cloud controller @
    http://api.vcap.me/services/v1/offerings/mysql-5.1/handles
    [2012-06-18 15:55:51.765531] mysql_gateway - pid=28002 tid=64a0
    fid=25fd ERROR -- Failed registering with cloud controller:
    [2012-06-18 15:55:51.765829] mysql_gateway - pid=28002 tid=64a0
    fid=25fd ERROR -- Failed fetching handles:
    [2012-06-18 15:55:51.766127] mysql_gateway - pid=28002 tid=64a0
    fid=25fd INFO -- Sending deactivation notice to cloud controller:
    http://api.vcap.me/services/v1/offerings
    [2012-06-18 15:55:56.800292] mysql_gateway - pid=28002 tid=64a0
    fid=25fd INFO -- Successfully deactivated with cloud controller
    /root/cloudfoundry/.deployments/devbox/deploy/rubies/ruby-1.9.2-p180/
    lib/ruby/gems/1.9.1/gems/vcap_services_base-0.1.9/lib/base/base.rb:
    84:in `block in update_varz': undefined method `[]=' for nil:NilClass
    (NoMethodError)
    from /root/cloudfoundry/.deployments/devbox/deploy/rubies/
    ruby-1.9.2-p180/lib/ruby/gems/1.9.1/gems/vcap_services_base-0.1.9/lib/
    base/base.rb:83:in `each'
    from /root/cloudfoundry/.deployments/devbox/deploy/rubies/
    ruby-1.9.2-p180/lib/ruby/gems/1.9.1/gems/vcap_services_base-0.1.9/lib/
    base/base.rb:83:in `update_varz'
    from /root/cloudfoundry/.deployments/devbox/deploy/rubies/
    ruby-1.9.2-p180/lib/ruby/gems/1.9.1/gems/vcap_services_base-0.1.9/lib/
    base/provisioner.rb:43:in `block in initialize'
    from /root/cloudfoundry/.deployments/devbox/deploy/rubies/
    ruby-1.9.2-p180/lib/ruby/gems/1.9.1/gems/
    eventmachine-0.12.11.cloudfoundry.3/lib/eventmachine.rb:256:in `call'
    from /root/cloudfoundry/.deployments/devbox/deploy/rubies/
    ruby-1.9.2-p180/lib/ruby/gems/1.9.1/gems/
    eventmachine-0.12.11.cloudfoundry.3/lib/eventmachine.rb:256:in
    `run_machine'
    from /root/cloudfoundry/.deployments/devbox/deploy/rubies/
    ruby-1.9.2-p180/lib/ruby/gems/1.9.1/gems/
    eventmachine-0.12.11.cloudfoundry.3/lib/eventmachine.rb:256:in `run'
    from /root/cloudfoundry/.deployments/devbox/deploy/rubies/
    ruby-1.9.2-p180/lib/ruby/gems/1.9.1/gems/vcap_services_base-0.1.9/lib/
    base/gateway.rb:94:in `start'
    from /root/cloudfoundry/vcap/services/mysql/bin/mysql_gateway:
    32:in `<main>'

    I have tried to install CF again,but that does not works. And I think
    this is caused by my re-configuring because for those gateways which
    I do not changed mbus IP, there're no this error.

    P.S. Maybe gateways and CC or NATS can not be running on different
    nodes?


    --
    Regards,
    Chunjie
  • 张 磊 at Jun 19, 2012 at 8:07 am
    WARNING: 'require 'rake/rdoctask'' is deprecated. Please use 'require
    'rdoc/task' (in RDoc 2.4.2+)' instead.
    at /root/cloudfoundry/.deployments/devbox/deploy/rubies/ruby-1.9.2-
    p180/lib/ruby/gems/1.9.1/gems/rake-0.9.2.2/lib/rake/rdoctask.rb
    WARNING: Global access to Rake DSL methods is deprecated. Please
    include
    ... Rake::DSL into classes and modules which use the Rake DSL
    methods.
    WARNING: DSL method CloudController::Application#task called at /root/
    cloudfoundry/.deployments/devbox/deploy/rubies/ruby-1.9.2-p180/lib/
    ruby/gems/1.9.1/gems/railties-3.0.5/lib/rails/application.rb:214:in
    `initialize_tasks'
    Rails Error: Unable to access log file. Please ensure that exists and
    is chmod 0666. The log level has been raised to WARN and the output
    directed to STDERR until the problem is fixed.
    Rails Error: Unable to access log file. Please ensure that exists and
    is chmod 0666. The log level has been raised to WARN and the output
    directed to STDERR until the problem is fixed.
    Thin web server (v1.3.1 codename Triple Espresso)
    Maximum connections set to 1024
    Listening on 172.17.4.226:9022, CTRL+C to stop
    [2012-06-19 13:19:47.479294] cc_events - pid=12661 tid=3a38 fid=a23e
    INFO -- [2012-06-19 13:19:47 +0800, :SYSTEM, "Starting VCAP
    CloudController (0.999)"]
    [2012-06-19 13:19:47.483766] cc_events - pid=12661 tid=3a38 fid=a23e
    INFO -- [2012-06-19 13:19:47 +0800, :SYSTEM, "Socket Limit:32768"]

    ...

    other service node msg except gateways

    ...

    I have looked into the asynchronous_service_gateway.rb and try to
    print more info about the object "http" to the log file. But I got
    nothing. So I can not figure out where is wrong. It is true that HTTP
    request between CC and gateways is not working.

    How can I make sure whether the request create by Gateway is valid and
    send to CC? And then I can track that request.
    On Jun 19, 12:56 pm, Chunjie Zhu wrote:
    The intuitive way is to watch cloud controller's log, see whether it can
    receive service gateway's request.

    Another suggestion, have a try to modify source code, such as "print more
    debug log" during service gateway initialization time, then you may get
    some hint to fix the issue.

    Regards,
    Chunjie








    On Tue, Jun 19, 2012 at 9:54 AM, 张 磊 wrote:
    Thank you for your reply. But how can I find out the status of HTTP
    channel between CC and Gateway? I'm not very clear about that.
    On Jun 18, 5:36 pm, Chunjie Zhu wrote:
    "update_varz" is a periodical operation, and, the timer may alarm at the
    Kernel.exit (
    https://github.com/cloudfoundry/vcap-services-base/blob/master/lib/ba...)
    time. Kernel.exit is just like POSIX atexit. So, you see, at the
    application termination time, both "send deactivation" action &
    "update_varz" action happen ed (I infer this from your log), then something
    unexpected, I mean, the "Nil Class" exception comes out.
    BTW, "register" & "fetching handles" communication between CC and
    Gateway is not piggybacked on NATS but through HTTP, so, one suggestion is,
    please make sure that the HTTP channel between CC and Gateway is working
    properly.
    Regards,
    Chunjie
    ----- Original Message -----
    From: "张 磊" <reso...@163.com>
    To: "vcap-dev" <vcap-...@cloudfoundry.org>
    Sent: Monday, June 18, 2012 4:47:41 PM
    Subject: [vcap-dev] All the gateway of services can not work suddenly
    when I re-configured the mbus.
    As I were doing some experiments about muti-node CF, I needed to
    change the binded IP of NATS in the config files of
    mysql,redis,rabbitmq ... gateway. For example, in order to connect the
    gateways to my new NATS node, I change the original mbus:xxx.xxx.xxx.
    219 to xxx.xxx.xxx.229. This works.
    But when I restore the IP back from 229 to 219, I found all the
    gateways can not work at all. And in their logs, there is the same
    error:
    [2012-06-18 15:55:41.756737] mysql_gateway - pid=28002 tid=64a0
    fid=25fd INFO -- Sending info to cloud controller:
    http://api.vcap.me/services/v1/offerings
    [2012-06-18 15:55:46.763478] mysql_gateway - pid=28002 tid=64a0
    fid=25fd INFO -- Fetching handles from cloud controller @
    http://api.vcap.me/services/v1/offerings/mysql-5.1/handles
    [2012-06-18 15:55:51.765531] mysql_gateway - pid=28002 tid=64a0
    fid=25fd ERROR -- Failed registering with cloud controller:
    [2012-06-18 15:55:51.765829] mysql_gateway - pid=28002 tid=64a0
    fid=25fd ERROR -- Failed fetching handles:
    [2012-06-18 15:55:51.766127] mysql_gateway - pid=28002 tid=64a0
    fid=25fd INFO -- Sending deactivation notice to cloud controller:
    http://api.vcap.me/services/v1/offerings
    [2012-06-18 15:55:56.800292] mysql_gateway - pid=28002 tid=64a0
    fid=25fd INFO -- Successfully deactivated with cloud controller
    /root/cloudfoundry/.deployments/devbox/deploy/rubies/ruby-1.9.2-p180/
    lib/ruby/gems/1.9.1/gems/vcap_services_base-0.1.9/lib/base/base.rb:
    84:in `block in update_varz': undefined method `[]=' for nil:NilClass
    (NoMethodError)
    from /root/cloudfoundry/.deployments/devbox/deploy/rubies/
    ruby-1.9.2-p180/lib/ruby/gems/1.9.1/gems/vcap_services_base-0.1.9/lib/
    base/base.rb:83:in `each'
    from /root/cloudfoundry/.deployments/devbox/deploy/rubies/
    ruby-1.9.2-p180/lib/ruby/gems/1.9.1/gems/vcap_services_base-0.1.9/lib/
    base/base.rb:83:in `update_varz'
    from /root/cloudfoundry/.deployments/devbox/deploy/rubies/
    ruby-1.9.2-p180/lib/ruby/gems/1.9.1/gems/vcap_services_base-0.1.9/lib/
    base/provisioner.rb:43:in `block in initialize'
    from /root/cloudfoundry/.deployments/devbox/deploy/rubies/
    ruby-1.9.2-p180/lib/ruby/gems/1.9.1/gems/
    eventmachine-0.12.11.cloudfoundry.3/lib/eventmachine.rb:256:in `call'
    from /root/cloudfoundry/.deployments/devbox/deploy/rubies/
    ruby-1.9.2-p180/lib/ruby/gems/1.9.1/gems/
    eventmachine-0.12.11.cloudfoundry.3/lib/eventmachine.rb:256:in
    `run_machine'
    from /root/cloudfoundry/.deployments/devbox/deploy/rubies/
    ruby-1.9.2-p180/lib/ruby/gems/1.9.1/gems/
    eventmachine-0.12.11.cloudfoundry.3/lib/eventmachine.rb:256:in `run'
    from /root/cloudfoundry/.deployments/devbox/deploy/rubies/
    ruby-1.9.2-p180/lib/ruby/gems/1.9.1/gems/vcap_services_base-0.1.9/lib/
    base/gateway.rb:94:in `start'
    from /root/cloudfoundry/vcap/services/mysql/bin/mysql_gateway:
    32:in `<main>'
    I have tried to install CF again,but that does not works. And I think
    this is caused by my re-configuring because for those gateways which
    I do not changed mbus IP, there're no this error.
    P.S. Maybe gateways and CC or NATS can not be running on different
    nodes?
    --
    Regards,
    Chunjie
  • Chunjie Zhu at Jun 20, 2012 at 1:46 am
    There are no "service gateway" requests in the cloud_controller log, seems it totally does not work.

    Here are some general trouble shooting steps, for your reference.
    1. see whether cloud controller is ready to accept the requests
    telnet <your cc host> 9022 # 9022 is the default public port for CC

    2. see whether router is really working, each request will first arrive at router then be forwarded to its destine
    grep "Registering api.vcap.me at <your cc host>:9022" in router.log # not present then something wrong

    3. see whether your firewall policy is really correct
    #depends on your network settings

    4. api.vcap.me is mapped to "127.0.0.1", VMware takes acquisition of this domain name, so if your cc and service gateway are not on the same machine, then there may be something wrong

    Regards,
    Chunjie

    ----- Original Message -----

    From: "张 磊" <resouer@163.com>
    To: "vcap-dev" <vcap-dev@cloudfoundry.org>
    Sent: Tuesday, June 19, 2012 4:07:12 PM
    Subject: [vcap-dev] Re: All the gateway of services can not work suddenly when I re-configured the mbus.

    WARNING: 'require 'rake/rdoctask'' is deprecated. Please use 'require
    'rdoc/task' (in RDoc 2.4.2+)' instead.
    at /root/cloudfoundry/.deployments/devbox/deploy/rubies/ruby-1.9.2-
    p180/lib/ruby/gems/1.9.1/gems/rake-0.9.2.2/lib/rake/rdoctask.rb
    WARNING: Global access to Rake DSL methods is deprecated. Please
    include
    ... Rake::DSL into classes and modules which use the Rake DSL
    methods.
    WARNING: DSL method CloudController::Application#task called at /root/
    cloudfoundry/.deployments/devbox/deploy/rubies/ruby-1.9.2-p180/lib/
    ruby/gems/1.9.1/gems/railties-3.0.5/lib/rails/application.rb:214:in
    `initialize_tasks'
    Rails Error: Unable to access log file. Please ensure that exists and
    is chmod 0666. The log level has been raised to WARN and the output
    directed to STDERR until the problem is fixed.
    Rails Error: Unable to access log file. Please ensure that exists and
    is chmod 0666. The log level has been raised to WARN and the output
    directed to STDERR until the problem is fixed.
    Thin web server (v1.3.1 codename Triple Espresso)
    Maximum connections set to 1024
    Listening on 172.17.4.226:9022, CTRL+C to stop
    [2012-06-19 13:19:47.479294] cc_events - pid=12661 tid=3a38 fid=a23e
    INFO -- [2012-06-19 13:19:47 +0800, :SYSTEM, "Starting VCAP
    CloudController (0.999)"]
    [2012-06-19 13:19:47.483766] cc_events - pid=12661 tid=3a38 fid=a23e
    INFO -- [2012-06-19 13:19:47 +0800, :SYSTEM, "Socket Limit:32768"]

    ...

    other service node msg except gateways

    ...

    I have looked into the asynchronous_service_gateway.rb and try to
    print more info about the object "http" to the log file. But I got
    nothing. So I can not figure out where is wrong. It is true that HTTP
    request between CC and gateways is not working.

    How can I make sure whether the request create by Gateway is valid and
    send to CC? And then I can track that request.
    On Jun 19, 12:56 pm, Chunjie Zhu wrote:
    The intuitive way is to watch cloud controller's log, see whether it can
    receive service gateway's request.

    Another suggestion, have a try to modify source code, such as "print more
    debug log" during service gateway initialization time, then you may get
    some hint to fix the issue.

    Regards,
    Chunjie








    On Tue, Jun 19, 2012 at 9:54 AM, 张 磊 wrote:
    Thank you for your reply. But how can I find out the status of HTTP
    channel between CC and Gateway? I'm not very clear about that.
    On Jun 18, 5:36 pm, Chunjie Zhu wrote:
    "update_varz" is a periodical operation, and, the timer may alarm at the
    Kernel.exit (
    https://github.com/cloudfoundry/vcap-services-base/blob/master/lib/ba...)
    time. Kernel.exit is just like POSIX atexit. So, you see, at the
    application termination time, both "send deactivation" action &
    "update_varz" action happen ed (I infer this from your log), then something
    unexpected, I mean, the "Nil Class" exception comes out.
    BTW, "register" & "fetching handles" communication between CC and
    Gateway is not piggybacked on NATS but through HTTP, so, one suggestion is,
    please make sure that the HTTP channel between CC and Gateway is working
    properly.
    Regards,
    Chunjie
    ----- Original Message -----
    From: "张 磊" <reso...@163.com>
    To: "vcap-dev" <vcap-...@cloudfoundry.org>
    Sent: Monday, June 18, 2012 4:47:41 PM
    Subject: [vcap-dev] All the gateway of services can not work suddenly
    when I re-configured the mbus.
    As I were doing some experiments about muti-node CF, I needed to
    change the binded IP of NATS in the config files of
    mysql,redis,rabbitmq ... gateway. For example, in order to connect the
    gateways to my new NATS node, I change the original mbus:xxx.xxx.xxx.
    219 to xxx.xxx.xxx.229. This works.
    But when I restore the IP back from 229 to 219, I found all the
    gateways can not work at all. And in their logs, there is the same
    error:
    [2012-06-18 15:55:41.756737] mysql_gateway - pid=28002 tid=64a0
    fid=25fd INFO -- Sending info to cloud controller:
    http://api.vcap.me/services/v1/offerings
    [2012-06-18 15:55:46.763478] mysql_gateway - pid=28002 tid=64a0
    fid=25fd INFO -- Fetching handles from cloud controller @
    http://api.vcap.me/services/v1/offerings/mysql-5.1/handles
    [2012-06-18 15:55:51.765531] mysql_gateway - pid=28002 tid=64a0
    fid=25fd ERROR -- Failed registering with cloud controller:
    [2012-06-18 15:55:51.765829] mysql_gateway - pid=28002 tid=64a0
    fid=25fd ERROR -- Failed fetching handles:
    [2012-06-18 15:55:51.766127] mysql_gateway - pid=28002 tid=64a0
    fid=25fd INFO -- Sending deactivation notice to cloud controller:
    http://api.vcap.me/services/v1/offerings
    [2012-06-18 15:55:56.800292] mysql_gateway - pid=28002 tid=64a0
    fid=25fd INFO -- Successfully deactivated with cloud controller
    /root/cloudfoundry/.deployments/devbox/deploy/rubies/ruby-1.9.2-p180/
    lib/ruby/gems/1.9.1/gems/vcap_services_base-0.1.9/lib/base/base.rb:
    84:in `block in update_varz': undefined method `[]=' for nil:NilClass
    (NoMethodError)
    from /root/cloudfoundry/.deployments/devbox/deploy/rubies/
    ruby-1.9.2-p180/lib/ruby/gems/1.9.1/gems/vcap_services_base-0.1.9/lib/
    base/base.rb:83:in `each'
    from /root/cloudfoundry/.deployments/devbox/deploy/rubies/
    ruby-1.9.2-p180/lib/ruby/gems/1.9.1/gems/vcap_services_base-0.1.9/lib/
    base/base.rb:83:in `update_varz'
    from /root/cloudfoundry/.deployments/devbox/deploy/rubies/
    ruby-1.9.2-p180/lib/ruby/gems/1.9.1/gems/vcap_services_base-0.1.9/lib/
    base/provisioner.rb:43:in `block in initialize'
    from /root/cloudfoundry/.deployments/devbox/deploy/rubies/
    ruby-1.9.2-p180/lib/ruby/gems/1.9.1/gems/
    eventmachine-0.12.11.cloudfoundry.3/lib/eventmachine.rb:256:in `call'
    from /root/cloudfoundry/.deployments/devbox/deploy/rubies/
    ruby-1.9.2-p180/lib/ruby/gems/1.9.1/gems/
    eventmachine-0.12.11.cloudfoundry.3/lib/eventmachine.rb:256:in
    `run_machine'
    from /root/cloudfoundry/.deployments/devbox/deploy/rubies/
    ruby-1.9.2-p180/lib/ruby/gems/1.9.1/gems/
    eventmachine-0.12.11.cloudfoundry.3/lib/eventmachine.rb:256:in `run'
    from /root/cloudfoundry/.deployments/devbox/deploy/rubies/
    ruby-1.9.2-p180/lib/ruby/gems/1.9.1/gems/vcap_services_base-0.1.9/lib/
    base/gateway.rb:94:in `start'
    from /root/cloudfoundry/vcap/services/mysql/bin/mysql_gateway:
    32:in `<main>'
    I have tried to install CF again,but that does not works. And I think
    this is caused by my re-configuring because for those gateways which
    I do not changed mbus IP, there're no this error.
    P.S. Maybe gateways and CC or NATS can not be running on different
    nodes?
    --
    Regards,
    Chunjie
  • 张 磊 at Jun 25, 2012 at 8:21 am
    Thanks a lot. I have found the problem. It is due to my CF is running
    on CloudStack, and the Router VM in my IaaS of CloudStack corrupted
    due to some unknown error, which leads to the failure of requests from
    gateways.I found this by using nslookup api.vcap.me.
    On Jun 20, 9:45 am, Chunjie Zhu wrote:
    There are no "service gateway" requests in the cloud_controller log, seems it totally does not work.

    Here are some general trouble shooting steps, for your reference.
    1. see whether cloud controller is ready to accept the requests
    telnet <your cc host> 9022 # 9022 is the default public port for CC

    2. see whether router is really working, each request will first arrive at router then be forwarded to its destine
    grep "Registering api.vcap.me at <your cc host>:9022" in router.log # not present then something wrong

    3. see whether your firewall policy is really correct
    #depends on your network settings

    4. api.vcap.me is mapped to "127.0.0.1", VMware takes acquisition of this domain name, so if your cc and service gateway are not on the same machine, then there may be something wrong

    Regards,
    Chunjie







    ----- Original Message -----

    From: "张 磊" <reso...@163.com>
    To: "vcap-dev" <vcap-...@cloudfoundry.org>
    Sent: Tuesday, June 19, 2012 4:07:12 PM
    Subject: [vcap-dev] Re: All the gateway of services can not work suddenly when I re-configured the mbus.

    WARNING: 'require 'rake/rdoctask'' is deprecated. Please use 'require
    'rdoc/task' (in RDoc 2.4.2+)' instead.
    at /root/cloudfoundry/.deployments/devbox/deploy/rubies/ruby-1.9.2-
    p180/lib/ruby/gems/1.9.1/gems/rake-0.9.2.2/lib/rake/rdoctask.rb
    WARNING: Global access to Rake DSL methods is deprecated. Please
    include
    ... Rake::DSL into classes and modules which use the Rake DSL
    methods.
    WARNING: DSL method CloudController::Application#task called at /root/
    cloudfoundry/.deployments/devbox/deploy/rubies/ruby-1.9.2-p180/lib/
    ruby/gems/1.9.1/gems/railties-3.0.5/lib/rails/application.rb:214:in
    `initialize_tasks'
    Rails Error: Unable to access log file. Please ensure that exists and
    is chmod 0666. The log level has been raised to WARN and the output
    directed to STDERR until the problem is fixed.
    Rails Error: Unable to access log file. Please ensure that exists and
    is chmod 0666. The log level has been raised to WARN and the output
    directed to STDERR until the problem is fixed.
    Thin web server (v1.3.1 codename Triple Espresso)
    Maximum connections set to 1024
    Listening on 172.17.4.226:9022, CTRL+C to stop
    [2012-06-19 13:19:47.479294] cc_events - pid=12661 tid=3a38 fid=a23e
    INFO -- [2012-06-19 13:19:47 +0800, :SYSTEM, "Starting VCAP
    CloudController (0.999)"]
    [2012-06-19 13:19:47.483766] cc_events - pid=12661 tid=3a38 fid=a23e
    INFO -- [2012-06-19 13:19:47 +0800, :SYSTEM, "Socket Limit:32768"]

    ...

    other service node msg except gateways

    ...

    I have looked into the asynchronous_service_gateway.rb and try to
    print more info about the object "http" to the log file. But I got
    nothing. So I can not figure out where is wrong. It is true that HTTP
    request between CC and gateways is not working.

    How can I make sure whether the request create by Gateway is valid and
    send to CC? And then I can track that request.
    On Jun 19, 12:56 pm, Chunjie Zhu wrote:
    The intuitive way is to watch cloud controller's log, see whether it can
    receive service gateway's request.
    Another suggestion, have a try to modify source code, such as "print more
    debug log" during service gateway initialization time, then you may get
    some hint to fix the issue.
    Regards,
    Chunjie
    On Tue, Jun 19, 2012 at 9:54 AM, 张 磊 wrote:
    Thank you for your reply. But how can I find out the status of HTTP
    channel between CC and Gateway? I'm not very clear about that.
    On Jun 18, 5:36 pm, Chunjie Zhu wrote:
    "update_varz" is a periodical operation, and, the timer may alarm at the
    Kernel.exit (
    https://github.com/cloudfoundry/vcap-services-base/blob/master/lib/ba...)
    time. Kernel.exit is just like POSIX atexit. So, you see, at the
    application termination time, both "send deactivation" action &
    "update_varz" action happen ed (I infer this from your log), then something
    unexpected, I mean, the "Nil Class" exception comes out.
    BTW, "register" & "fetching handles" communication between CC and
    Gateway is not piggybacked on NATS but through HTTP, so, one suggestion is,
    please make sure that the HTTP channel between CC and Gateway is working
    properly.
    Regards,
    Chunjie
    ----- Original Message -----
    From: "张 磊" <reso...@163.com>
    To: "vcap-dev" <vcap-...@cloudfoundry.org>
    Sent: Monday, June 18, 2012 4:47:41 PM
    Subject: [vcap-dev] All the gateway of services can not work suddenly
    when I re-configured the mbus.
    As I were doing some experiments about muti-node CF, I needed to
    change the binded IP of NATS in the config files of
    mysql,redis,rabbitmq ... gateway. For example, in order to connect the
    gateways to my new NATS node, I change the original mbus:xxx.xxx.xxx.
    219 to xxx.xxx.xxx.229. This works.
    But when I restore the IP back from 229 to 219, I found all the
    gateways can not work at all. And in their logs, there is the same
    error:
    [2012-06-18 15:55:41.756737] mysql_gateway - pid=28002 tid=64a0
    fid=25fd INFO -- Sending info to cloud controller:
    http://api.vcap.me/services/v1/offerings
    [2012-06-18 15:55:46.763478] mysql_gateway - pid=28002 tid=64a0
    fid=25fd INFO -- Fetching handles from cloud controller @
    http://api.vcap.me/services/v1/offerings/mysql-5.1/handles
    [2012-06-18 15:55:51.765531] mysql_gateway - pid=28002 tid=64a0
    fid=25fd ERROR -- Failed registering with cloud controller:
    [2012-06-18 15:55:51.765829] mysql_gateway - pid=28002 tid=64a0
    fid=25fd ERROR -- Failed fetching handles:
    [2012-06-18 15:55:51.766127] mysql_gateway - pid=28002 tid=64a0
    fid=25fd INFO -- Sending deactivation notice to cloud controller:
    http://api.vcap.me/services/v1/offerings
    [2012-06-18 15:55:56.800292] mysql_gateway - pid=28002 tid=64a0
    fid=25fd INFO -- Successfully deactivated with cloud controller
    /root/cloudfoundry/.deployments/devbox/deploy/rubies/ruby-1.9.2-p180/
    lib/ruby/gems/1.9.1/gems/vcap_services_base-0.1.9/lib/base/base.rb:
    84:in `block in update_varz': undefined method `[]=' for nil:NilClass
    (NoMethodError)
    from /root/cloudfoundry/.deployments/devbox/deploy/rubies/
    ruby-1.9.2-p180/lib/ruby/gems/1.9.1/gems/vcap_services_base-0.1.9/lib/
    base/base.rb:83:in `each'
    from /root/cloudfoundry/.deployments/devbox/deploy/rubies/
    ruby-1.9.2-p180/lib/ruby/gems/1.9.1/gems/vcap_services_base-0.1.9/lib/
    base/base.rb:83:in `update_varz'
    from /root/cloudfoundry/.deployments/devbox/deploy/rubies/
    ruby-1.9.2-p180/lib/ruby/gems/1.9.1/gems/vcap_services_base-0.1.9/lib/
    base/provisioner.rb:43:in `block in initialize'
    from /root/cloudfoundry/.deployments/devbox/deploy/rubies/
    ruby-1.9.2-p180/lib/ruby/gems/1.9.1/gems/
    eventmachine-0.12.11.cloudfoundry.3/lib/eventmachine.rb:256:in `call'
    from /root/cloudfoundry/.deployments/devbox/deploy/rubies/
    ruby-1.9.2-p180/lib/ruby/gems/1.9.1/gems/
    eventmachine-0.12.11.cloudfoundry.3/lib/eventmachine.rb:256:in
    `run_machine'
    from /root/cloudfoundry/.deployments/devbox/deploy/rubies/
    ruby-1.9.2-p180/lib/ruby/gems/1.9.1/gems/
    eventmachine-0.12.11.cloudfoundry.3/lib/eventmachine.rb:256:in `run'
    from /root/cloudfoundry/.deployments/devbox/deploy/rubies/
    ruby-1.9.2-p180/lib/ruby/gems/1.9.1/gems/vcap_services_base-0.1.9/lib/
    base/gateway.rb:94:in `start'
    from /root/cloudfoundry/vcap/services/mysql/bin/mysql_gateway:
    32:in `<main>'
    I have tried to install CF again,but that does not works. And I think
    this is caused by my re-configuring because for those gateways which
    I do not changed mbus IP, there're no this error.
    P.S. Maybe gateways and CC or NATS can not be running on different
    nodes?
    --
    Regards,
    Chunjie

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupvcap-dev @
postedJun 18, '12 at 8:47a
activeJun 25, '12 at 8:21a
posts7
users2

2 users in discussion

张 磊: 4 posts Chunjie Zhu: 3 posts

People

Translate

site design / logo © 2022 Grokbase