FAQ
Hi...

I'm trying to add a new CPI f or Softlayer. Softlayer doesn't allow image
uploads to the cloud so the short story is that I'd like to use the
stemcell build process to build my new custom stemcell "into" the VM that
I'm running the build process in. So instead of creating a new
tarball/image file I'd like the build process to be able to install all of
its stuff into the OS of the running VM. Then I can capture this vm as a
new image in Softlayer and have it be my stemcell (one for micro and one
for full bosh). This is the plan anyway.

I've been playing with this for a few days and it seems to be possible but
I'd love to get the group's sense of whether this is reasonable. Basically
I've tried changing the work_dir to / and running the bundle exec rake
stemcell:micro[softlayer] command. There's more to it than that but
hopefully you get the idea of what I'm trying to do. Any pointers or
gotchas you guys can think of that would make this a fruitless approach?

Thanks!
Ethan

Search Discussions

  • Ferran Rodenas at Jul 15, 2013 at 1:22 pm
    Hi Ethan,

    Seems an interesting path to explore. I suppose you have had to modify the
    run_in_chroot helper? Any other modifications?

    Another interesting path if you want to take a look is @drnic
    chef-bosh_agent
    <https://github.com/cloudfoundry-community/chef-bosh_agent> tool,
    basically reproduces all the stemcell builder steps using chef. I believe
    it only supports Vagrant right now, but it doesn't seems very difficult to
    adapt it to other systems.

    I've been doing something similar with another CPI that doesn't support
    image uploads, but in my case, I booted a vanilla Ubuntu 10.04 and hardened
    the os manually. For the bosh_agent modifications, I develop it locally on
    my laptop, and sync the changes to the vm using rsync. Whenever I want to
    create a new stemcell version, I snapshot the image and create a fake
    stemcell with the image id of the snapshot.

    - Ferdy


    2013/7/15 <ekmerril@gmail.com>
    Hi...

    I'm trying to add a new CPI f or Softlayer. Softlayer doesn't allow image
    uploads to the cloud so the short story is that I'd like to use the
    stemcell build process to build my new custom stemcell "into" the VM that
    I'm running the build process in. So instead of creating a new
    tarball/image file I'd like the build process to be able to install all of
    its stuff into the OS of the running VM. Then I can capture this vm as a
    new image in Softlayer and have it be my stemcell (one for micro and one
    for full bosh). This is the plan anyway.

    I've been playing with this for a few days and it seems to be possible but
    I'd love to get the group's sense of whether this is reasonable. Basically
    I've tried changing the work_dir to / and running the bundle exec rake
    stemcell:micro[softlayer] command. There's more to it than that but
    hopefully you get the idea of what I'm trying to do. Any pointers or
    gotchas you guys can think of that would make this a fruitless approach?

    Thanks!
    Ethan

  • Ekmerril at Jul 15, 2013 at 1:33 pm
    Very cool...yeah, I was looking at the good doctor's tool but didn't yet
    look at extracting it from vagrant.

    If you get a chance I'd really appreciate some more details on the "develop
    locally and rsync". If it's not too complex perhaps you could share
    exactly what you're doing there. I suspect it would save me a ton of time
    as I've already got the bosh source locally and have become pretty familiar
    with it at this point.

    Thanks,
    Ethan
    On Monday, July 15, 2013 9:22:26 AM UTC-4, ferdy wrote:

    Hi Ethan,

    Seems an interesting path to explore. I suppose you have had to modify the
    run_in_chroot helper? Any other modifications?

    Another interesting path if you want to take a look is @drnic
    chef-bosh_agent<https://github.com/cloudfoundry-community/chef-bosh_agent> tool,
    basically reproduces all the stemcell builder steps using chef. I believe
    it only supports Vagrant right now, but it doesn't seems very difficult to
    adapt it to other systems.

    I've been doing something similar with another CPI that doesn't support
    image uploads, but in my case, I booted a vanilla Ubuntu 10.04 and hardened
    the os manually. For the bosh_agent modifications, I develop it locally on
    my laptop, and sync the changes to the vm using rsync. Whenever I want to
    create a new stemcell version, I snapshot the image and create a fake
    stemcell with the image id of the snapshot.

    - Ferdy


    2013/7/15 <ekme...@gmail.com <javascript:>>
    Hi...

    I'm trying to add a new CPI f or Softlayer. Softlayer doesn't allow
    image uploads to the cloud so the short story is that I'd like to use the
    stemcell build process to build my new custom stemcell "into" the VM that
    I'm running the build process in. So instead of creating a new
    tarball/image file I'd like the build process to be able to install all of
    its stuff into the OS of the running VM. Then I can capture this vm as a
    new image in Softlayer and have it be my stemcell (one for micro and one
    for full bosh). This is the plan anyway.

    I've been playing with this for a few days and it seems to be possible
    but I'd love to get the group's sense of whether this is reasonable.
    Basically I've tried changing the work_dir to / and running the bundle
    exec rake stemcell:micro[softlayer] command. There's more to it than that
    but hopefully you get the idea of what I'm trying to do. Any pointers or
    gotchas you guys can think of that would make this a fruitless approach?

    Thanks!
    Ethan

  • Ferran Rodenas at Jul 15, 2013 at 2:44 pm
    It's a bit 'pedestrian', and only works if you don't modify agent's gems.
    I'm using the watch program inside the bosh_agent/lib dir on my laptop:

    watch -n 30 rsync -av --delete . root@
    <ip-address>:/var/vcap/bosh/lib/ruby/gems/1.9.1/gems/bosh_agent-1.5.0.pre.3/lib

    Hope it helps!
    - Ferdy


    2013/7/15 <ekmerril@gmail.com>
    Very cool...yeah, I was looking at the good doctor's tool but didn't yet
    look at extracting it from vagrant.

    If you get a chance I'd really appreciate some more details on the
    "develop locally and rsync". If it's not too complex perhaps you could
    share exactly what you're doing there. I suspect it would save me a ton of
    time as I've already got the bosh source locally and have become pretty
    familiar with it at this point.

    Thanks,
    Ethan

    On Monday, July 15, 2013 9:22:26 AM UTC-4, ferdy wrote:

    Hi Ethan,

    Seems an interesting path to explore. I suppose you have had to modify
    the run_in_chroot helper? Any other modifications?

    Another interesting path if you want to take a look is @drnic
    chef-bosh_agent<https://github.com/cloudfoundry-community/chef-bosh_agent> tool,
    basically reproduces all the stemcell builder steps using chef. I believe
    it only supports Vagrant right now, but it doesn't seems very difficult to
    adapt it to other systems.

    I've been doing something similar with another CPI that doesn't support
    image uploads, but in my case, I booted a vanilla Ubuntu 10.04 and hardened
    the os manually. For the bosh_agent modifications, I develop it locally on
    my laptop, and sync the changes to the vm using rsync. Whenever I want to
    create a new stemcell version, I snapshot the image and create a fake
    stemcell with the image id of the snapshot.

    - Ferdy


    2013/7/15 <ekme...@gmail.com>

    Hi...
    I'm trying to add a new CPI f or Softlayer. Softlayer doesn't allow
    image uploads to the cloud so the short story is that I'd like to use the
    stemcell build process to build my new custom stemcell "into" the VM that
    I'm running the build process in. So instead of creating a new
    tarball/image file I'd like the build process to be able to install all of
    its stuff into the OS of the running VM. Then I can capture this vm as a
    new image in Softlayer and have it be my stemcell (one for micro and one
    for full bosh). This is the plan anyway.

    I've been playing with this for a few days and it seems to be possible
    but I'd love to get the group's sense of whether this is reasonable.
    Basically I've tried changing the work_dir to / and running the bundle
    exec rake stemcell:micro[softlayer] command. There's more to it than that
    but hopefully you get the idea of what I'm trying to do. Any pointers or
    gotchas you guys can think of that would make this a fruitless approach?

    Thanks!
    Ethan

  • Ekmerril at Jul 15, 2013 at 8:24 pm
    Hi Ferdy...

    I need to step back for a sec. and ask a few higher level questions I
    think. It appears like my stemcell creation process is working ok but when
    I try to deploy the new stemcell using the micro cli it can never connect
    to the agent. If I start the agent manually on the new stemcell vm and try
    to target from the cli vm then it can't connect to the director.

    I'm seeing that the stemcell builder process (for a micro stemcell) doesn't
    seem to add any auto-starting of the bosh agent before capturing the image.
      Does this sound right or does it mean my build is not executing properly?
      Also I don't see any other jobs like director configured to start. So I'm
    a little confused about what the desired state is supposed to be after a
    successful stemcell build. Should the director be auto-configured to
    start? Or is this supposed to be all setup when a stemcell is deployed
    (i.e. the cli talks to the agent and sets the director up as part of the
    deployment)? Should the bosh_agent be auto-configured to start after a
    stemcell build?

    Can you tell me or point me to some doc to help with this?

    Thanks,
    Ethan
  • Ferran Rodenas at Jul 15, 2013 at 9:03 pm
    Ethan,

    The bosh_agent stage<https://github.com/cloudfoundry/bosh/tree/master/stemcell_builder/stages/bosh_agent>in
    the stemcell builder should create
    a service
    <https://github.com/cloudfoundry/bosh/blob/master/stemcell_builder/stages/bosh_agent/apply.sh#L38>that
    will be monitored by runit, so the agent is started automatically when the
    vm boots up. Check if the stemcell you build has a /etc/sv/agent dir and it
    has runit package installed.

    The rest of bosh components will be deployed by the micro cli. The workflow
    is: micro cli boots the stemcell (which starts automatically the bosh
    agent), queries agent for a pre-backed apply
    spec<https://github.com/cloudfoundry/bosh/tree/master/release/micro>in
    the vm (located at /var/vcap/micro in the stemcell), modifies it
    according to the local micro_bosh.yml, and sends a command to the agent to
    apply the new spec. This new spec is who deploys all bosh components
    (director, registry, powerdns, ...), that btw, are also prebacked in the
    stemcell (bosh_micro
    stage<https://github.com/cloudfoundry/bosh/tree/master/stemcell_builder/stages/bosh_micro>will
    call
    package_compiler<https://github.com/cloudfoundry/bosh/tree/master/package_compiler>,
    who will store them in
    /var/tmp/bosh/microbosh/work/work/chroot/var/vcap/micro_bosh/data/cache in
    the stemcell), When the apply has been successfully deployed, then the
    micro cli can start talking to director.

    - Ferdy


    2013/7/15 <ekmerril@gmail.com>
    Hi Ferdy...

    I need to step back for a sec. and ask a few higher level questions I
    think. It appears like my stemcell creation process is working ok but when
    I try to deploy the new stemcell using the micro cli it can never connect
    to the agent. If I start the agent manually on the new stemcell vm and try
    to target from the cli vm then it can't connect to the director.

    I'm seeing that the stemcell builder process (for a micro stemcell)
    doesn't seem to add any auto-starting of the bosh agent before capturing
    the image. Does this sound right or does it mean my build is not executing
    properly? Also I don't see any other jobs like director configured to
    start. So I'm a little confused about what the desired state is supposed
    to be after a successful stemcell build. Should the director be
    auto-configured to start? Or is this supposed to be all setup when a
    stemcell is deployed (i.e. the cli talks to the agent and sets the director
    up as part of the deployment)? Should the bosh_agent be auto-configured to
    start after a stemcell build?

    Can you tell me or point me to some doc to help with this?

    Thanks,
    Ethan
  • Ekmerril at Jul 15, 2013 at 10:35 pm
    Very good info and thanks for the quick reply. This should help me get
    over a hump.

    Ethan
  • Ekmerril at Jul 16, 2013 at 6:23 pm
    Hi...I'm definitely making progress but still getting tripped up in the
    stemcell deployment. I notice that the agent is looking for various
    settings like "blobstore" properties and "mbus". However these are coming
    back nil for me. Is my cpi supposed to set these in the user_data.json?
      Should they be set in the micro_bosh.yml that the cli deployer uses? Some
    other mechanism?

    I also see that the cli is trying to connect to the agent using
    "https:vcap:b00tstrap@:6868/agent" (it's missing an ip address).

    Thanks!
    Ethan
  • Gabrielle Sweda at Jul 16, 2013 at 7:57 pm
    Yes, typically those values come from the apply_spec micro_bosh.yml. Mine
    looks like this (for aws w VPC):

    apply_spec:
       agent:
         blobstore:
           address: 10.10.0.6
         nats:
           address: 10.10.0.6
       properties:
         ntp:
         - 0.north-america.pool.ntp.org
         - 1.north-america.pool.ntp.org
         - 2.north-america.pool.ntp.org
         - 3.north-america.pool.ntp.org
         registry:
           address: 10.10.0.6
         dns:
           recursor: 208.67.220.220
         hm:
           resurrector_enabled: true
           director_account:
             user: hm
             password: made_up
         aws:
           access_key_id: keystuff
           secret_access_key: secretstuff
           default_key_name: bosh
           default_security_groups: ["bosh"]
           ec2_endpoint: ec2.us-east-1.amazonaws.com
           region: us-east-1


    I assume the mbus URI gets created from the nats address with a default
    from the port. this may happen in the agent code.



    On Tue, Jul 16, 2013 at 2:23 PM, wrote:


    Hi...I'm definitely making progress but still getting tripped up in the
    stemcell deployment. I notice that the agent is looking for various
    settings like "blobstore" properties and "mbus". However these are coming
    back nil for me. Is my cpi supposed to set these in the user_data.json?
    Should they be set in the micro_bosh.yml that the cli deployer uses? Some
    other mechanism?

    I also see that the cli is trying to connect to the agent using
    "https:vcap:b00tstrap@:6868/agent" (it's missing an ip address).

    Thanks!
    Ethan
  • Ekmerril at Jul 16, 2013 at 8:37 pm

    Hi...
    So in your above example is the 10.10.0.6 address the address of the newly
    deployed micro_bosh vm or the vm running the cli? I assume its the new
    micro vm in which case do you need to know its address ahead of time
    (before deployment)?
    Ethan
  • Gabrielle Sweda at Jul 16, 2013 at 10:57 pm
    Yes, the microbosh needs an IP assigned afaik. I have never tried to
    deploy one with DHCP.

    The network settings look like this


    network:
       type: manual
       vip: 54.208.???.??? <<-- this is the AWS EIP
       ip: 10.10.0.6 <<--- this is the VPC IP assigned
       dns:
       - 10.10.0.2
       cloud_properties:
         subnet: subnet-????????


    On Tue, Jul 16, 2013 at 4:37 PM, wrote:

    Hi...

    So in your above example is the 10.10.0.6 address the address of the
    newly deployed micro_bosh vm or the vm running the cli? I assume its the
    new micro vm in which case do you need to know its address ahead of time
    (before deployment)?
    Ethan

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupbosh-dev @
postedJul 15, '13 at 12:47p
activeJul 16, '13 at 10:57p
posts11
users3

People

Translate

site design / logo © 2021 Grokbase