Hi,

After having to mess around with using different VMs/laptops for different
parts of the create/deploy process, I had the idea that I could package a
release into a tarball and distribute that, just like you can distribute a
stemcell. That is, when I do $bosh upload release and I see
Copying packages
...
Building tarball
----------------
Generated /tmp/d20130110-9288-3jg7h0/d20130110-9288-193ciw3/release.tgz
Release size: 1.5G
...
Uploading release
release.tgz: 2% | | 32.3MB 825.8KB/s ETA:
00:31:53
...

Then I'd rather do
$bosh package release
Copying packages
...
Building tarball with every "FOUND LOCAL" package and job
----------------
Generated /output_directory/release.tgz
Release size: 1.5G

$bosh upload release /output_directory/release.tgz
Verifying release...
File exists and readable OK
Extract tarball OK
Manifest exists OK
Release name/version OK
...
Uploading release
release.tgz: 20% |oooo | 316.6MB 822.5KB/s ETA:
00:26:07


That may open up future things like $bosh download public releases

Just a thought

Search Discussions

  • Dr Nic Williams at Jan 10, 2013 at 2:35 am
    I like this idea. It wouldn't technically be a lot of dev effort - it's a
    large package to upload/download like a stemcell. CLI and director
    changes. Want
    to have a crack at it? I could help you along if you have questions etc.

    A bonus v2 version of it, much more effort I think, is to distribute
    releases that include precompiled packages (for a given stemcell). Imagine
    not having to compile 50+ cf-release packages when you're just getting
    started!

    On Wednesday, January 9, 2013, wrote:

    Hi,

    After having to mess around with using different VMs/laptops for different
    parts of the create/deploy process, I had the idea that I could package a
    release into a tarball and distribute that, just like you can distribute a
    stemcell. That is, when I do $bosh upload release and I see
    Copying packages
    ...
    Building tarball
    ----------------
    Generated /tmp/d20130110-9288-3jg7h0/d20130110-9288-193ciw3/release.tgz
    Release size: 1.5G
    ...
    Uploading release
    release.tgz: 2% | | 32.3MB 825.8KB/s ETA:
    00:31:53
    ...

    Then I'd rather do
    $bosh package release
    Copying packages
    ...
    Building tarball with every "FOUND LOCAL" package and job
    ----------------
    Generated /output_directory/release.tgz
    Release size: 1.5G

    $bosh upload release /output_directory/release.tgz
    Verifying release...
    File exists and readable OK
    Extract tarball OK
    Manifest exists OK
    Release name/version OK
    ...
    Uploading release
    release.tgz: 20% |oooo | 316.6MB 822.5KB/s ETA:
    00:26:07


    That may open up future things like $bosh download public releases

    Just a thought

    --
    Dr Nic Williams
    http://drnicwilliams.com
    cell +1 (415) 860-2185
  • Martin Englund at Jan 10, 2013 at 5:01 am
    Nic,

    the second part is something I've thought about a lot when testing micro bosh stemcells. The way it works now, every package gets recompiled whenever you build a new stemcell, even if the base OS hasn't changed. You also have excessive compiling when you have multiple bosh directors deploying the same release (using the same stemcell).

    If we make compiled packages (for final versions of releases) available for download (keyed on stemcell version) a deploy will become considerably faster!

    cheers,
    /Martin
    --
    Martin Englund, Staff Engineer, Cloud Foundry, VMware Inc.
    "The question is not if you are paranoid, it is if you are paranoid enough."
    On Jan 9, 2013, at 18:35, Dr Nic Williams wrote:

    I like this idea. It wouldn't technically be a lot of dev effort - it's a large package to upload/download like a stemcell. CLI and director changes. Want to have a crack at it? I could help you along if you have questions etc.

    A bonus v2 version of it, much more effort I think, is to distribute releases that include precompiled packages (for a given stemcell). Imagine not having to compile 50+ cf-release packages when you're just getting started!

    On Wednesday, January 9, 2013, wrote:
    Hi,

    After having to mess around with using different VMs/laptops for different parts of the create/deploy process, I had the idea that I could package a release into a tarball and distribute that, just like you can distribute a stemcell. That is, when I do $bosh upload release and I see
    Copying packages
    ...
    Building tarball
    ----------------
    Generated /tmp/d20130110-9288-3jg7h0/d20130110-9288-193ciw3/release.tgz
    Release size: 1.5G
    ...
    Uploading release
    release.tgz: 2% | | 32.3MB 825.8KB/s ETA: 00:31:53
    ...

    Then I'd rather do
    $bosh package release
    Copying packages
    ...
    Building tarball with every "FOUND LOCAL" package and job
    ----------------
    Generated /output_directory/release.tgz
    Release size: 1.5G

    $bosh upload release /output_directory/release.tgz
    Verifying release...
    File exists and readable OK
    Extract tarball OK
    Manifest exists OK
    Release name/version OK
    ...
    Uploading release
    release.tgz: 20% |oooo | 316.6MB 822.5KB/s ETA: 00:26:07


    That may open up future things like $bosh download public releases

    Just a thought

    --
    Dr Nic Williams
    http://drnicwilliams.com
    cell +1 (415) 860-2185
  • Matthew Kocher at Jan 10, 2013 at 5:53 am
    Agreed, sharing compiled packages somewhere central makes a lot of sense.
    I think they can only be use if the stemcells are exactly the same
    however, as we can't know if all the shared libraries are the same on an
    upgraded stemcell.

    I'd love if the package cache could just be a blobstore with no server in
    front of it with well defined file names (eg,
    stemcell-bosh-stemcell-aws-0.6.4-sha1ofstecell-postgres-9-1-sha1ofpostgres).
    I think at the moment bosh doesn't assume it can chose a filename in the
    blobstore, but I'm not sure where that restriction comes from.

    On Wed, Jan 9, 2013 at 8:55 PM, Martin Englund wrote:

    the second part is something I've thought about a lot when testing micro
    bosh stemcells. The way it works now, every package gets recompiled
    whenever you build a new stemcell, even if the base OS hasn't changed. You
    also have excessive compiling when you have multiple bosh directors
    deploying the same release (using the same stemcell).
  • James Bayer at Jan 10, 2013 at 7:09 am
    Off-topic. Speaking of blob stores, just for fun you can take a look at this next-gen blob store called camilstore under development by Brad Fitzpatrick that Adrian Crockcoft retweeted tonight that I thought was interesting.
    http://camlistore.org/talks/2011-05-07-Camlistore-Sao-Paolo/#1

    Thank you,

    James Bayer

    On Jan 9, 2013, at 9:52 PM, Matthew Kocher wrote:

    Agreed, sharing compiled packages somewhere central makes a lot of sense. I think they can only be use if the stemcells are exactly the same however, as we can't know if all the shared libraries are the same on an upgraded stemcell.

    I'd love if the package cache could just be a blobstore with no server in front of it with well defined file names (eg, stemcell-bosh-stemcell-aws-0.6.4-sha1ofstecell-postgres-9-1-sha1ofpostgres). I think at the moment bosh doesn't assume it can chose a filename in the blobstore, but I'm not sure where that restriction comes from.


    On Wed, Jan 9, 2013 at 8:55 PM, Martin Englund wrote:
    the second part is something I've thought about a lot when testing micro bosh stemcells. The way it works now, every package gets recompiled whenever you build a new stemcell, even if the base OS hasn't changed. You also have excessive compiling when you have multiple bosh directors deploying the same release (using the same stemcell).
  • Martin Englund at Jan 10, 2013 at 1:47 pm
    Looks like a very good architecture. Once the blobstore can run on something other than GAE it'll be an interesting option.

    cheers,
    /Martin
    --
    Martin Englund, Staff Engineer, Cloud Foundry, VMware Inc.
    "The question is not if you are paranoid, it is if you are paranoid enough."
    On Jan 9, 2013, at 23:09, James Bayer wrote:

    Off-topic. Speaking of blob stores, just for fun you can take a look at this next-gen blob store called camilstore under development by Brad Fitzpatrick that Adrian Crockcoft retweeted tonight that I thought was interesting.
    http://camlistore.org/talks/2011-05-07-Camlistore-Sao-Paolo/#1

    Thank you,

    James Bayer

    On Jan 9, 2013, at 9:52 PM, Matthew Kocher wrote:

    Agreed, sharing compiled packages somewhere central makes a lot of sense. I think they can only be use if the stemcells are exactly the same however, as we can't know if all the shared libraries are the same on an upgraded stemcell.

    I'd love if the package cache could just be a blobstore with no server in front of it with well defined file names (eg, stemcell-bosh-stemcell-aws-0.6.4-sha1ofstecell-postgres-9-1-sha1ofpostgres). I think at the moment bosh doesn't assume it can chose a filename in the blobstore, but I'm not sure where that restriction comes from.

    On Wed, Jan 9, 2013 at 8:55 PM, Martin Englund wrote:
    the second part is something I've thought about a lot when testing micro bosh stemcells. The way it works now, every package gets recompiled whenever you build a new stemcell, even if the base OS hasn't changed. You also have excessive compiling when you have multiple bosh directors deploying the same release (using the same stemcell).
  • Dr Nic Williams at Jan 11, 2013 at 1:47 pm
    James, that slide deck only has 3 slides in it for me. Anyone else? If you
    have a fuller set of slides can you email them?

    Nic

    On Wed, Jan 9, 2013 at 11:09 PM, James Bayer wrote:

    Off-topic. Speaking of blob stores, just for fun you can take a look at
    this next-gen blob store called camilstore under development by Brad
    Fitzpatrick that Adrian Crockcoft retweeted tonight that I thought was
    interesting.
    http://camlistore.org/talks/2011-05-07-Camlistore-Sao-Paolo/#1

    Thank you,

    James Bayer


    On Jan 9, 2013, at 9:52 PM, Matthew Kocher wrote:

    Agreed, sharing compiled packages somewhere central makes a lot of sense.
    I think they can only be use if the stemcells are exactly the same
    however, as we can't know if all the shared libraries are the same on an
    upgraded stemcell.

    I'd love if the package cache could just be a blobstore with no server in
    front of it with well defined file names (eg,
    stemcell-bosh-stemcell-aws-0.6.4-sha1ofstecell-postgres-9-1-sha1ofpostgres).
    I think at the moment bosh doesn't assume it can chose a filename in the
    blobstore, but I'm not sure where that restriction comes from.

    On Wed, Jan 9, 2013 at 8:55 PM, Martin Englund wrote:

    the second part is something I've thought about a lot when testing micro
    bosh stemcells. The way it works now, every package gets recompiled
    whenever you build a new stemcell, even if the base OS hasn't changed. You
    also have excessive compiling when you have multiple bosh directors
    deploying the same release (using the same stemcell).


    --
    Dr Nic Williams
    http://drnicwilliams.com
    cell +1 (415) 860-2185
  • Dr Nic Williams at Jan 11, 2013 at 1:49 pm
    From http://camlistore.org/

    There is an older presentation with all its slides:
    https://docs.google.com/presentation/d/1otsEBBmOYoPY6T7p4YtM0ysX1vUG3amZQAykr9jZZgk/present#slide=id.i0

    On Fri, Jan 11, 2013 at 5:47 AM, Dr Nic Williams wrote:

    James, that slide deck only has 3 slides in it for me. Anyone else? If you
    have a fuller set of slides can you email them?

    Nic

    On Wed, Jan 9, 2013 at 11:09 PM, James Bayer wrote:

    Off-topic. Speaking of blob stores, just for fun you can take a look at
    this next-gen blob store called camilstore under development by Brad
    Fitzpatrick that Adrian Crockcoft retweeted tonight that I thought was
    interesting.
    http://camlistore.org/talks/2011-05-07-Camlistore-Sao-Paolo/#1

    Thank you,

    James Bayer


    On Jan 9, 2013, at 9:52 PM, Matthew Kocher <mkocher@pivotallabs.com>
    wrote:

    Agreed, sharing compiled packages somewhere central makes a lot of sense.
    I think they can only be use if the stemcells are exactly the same
    however, as we can't know if all the shared libraries are the same on an
    upgraded stemcell.

    I'd love if the package cache could just be a blobstore with no server in
    front of it with well defined file names (eg,
    stemcell-bosh-stemcell-aws-0.6.4-sha1ofstecell-postgres-9-1-sha1ofpostgres).
    I think at the moment bosh doesn't assume it can chose a filename in the
    blobstore, but I'm not sure where that restriction comes from.

    On Wed, Jan 9, 2013 at 8:55 PM, Martin Englund wrote:

    the second part is something I've thought about a lot when testing micro
    bosh stemcells. The way it works now, every package gets recompiled
    whenever you build a new stemcell, even if the base OS hasn't changed. You
    also have excessive compiling when you have multiple bosh directors
    deploying the same release (using the same stemcell).


    --
    Dr Nic Williams
    http://drnicwilliams.com
    cell +1 (415) 860-2185


    --
    Dr Nic Williams
    http://drnicwilliams.com
    cell +1 (415) 860-2185
  • Martin Englund at Jan 10, 2013 at 1:32 pm
    If the stemcell is the same version, we know that the libraries are the same. We never upgrade the OS contents, other than upgrading the stemcell, and then we have a known set of libraries.

    The blobstore can be the actual service, we added the blobstore server component to make sure it lives as close to the consumers (VMs that bosh creates) as possible. If you are running on AWS, you can ditch the blobstore server and just configure bosh to use S3.

    cheers,
    /Martin
    --
    Martin Englund, Staff Engineer, Cloud Foundry, VMware Inc.
    "The question is not if you are paranoid, it is if you are paranoid enough."
    On Jan 9, 2013, at 21:52, Matthew Kocher wrote:

    Agreed, sharing compiled packages somewhere central makes a lot of sense. I think they can only be use if the stemcells are exactly the same however, as we can't know if all the shared libraries are the same on an upgraded stemcell.

    I'd love if the package cache could just be a blobstore with no server in front of it with well defined file names (eg, stemcell-bosh-stemcell-aws-0.6.4-sha1ofstecell-postgres-9-1-sha1ofpostgres). I think at the moment bosh doesn't assume it can chose a filename in the blobstore, but I'm not sure where that restriction comes from.

    On Wed, Jan 9, 2013 at 8:55 PM, Martin Englund wrote:
    the second part is something I've thought about a lot when testing micro bosh stemcells. The way it works now, every package gets recompiled whenever you build a new stemcell, even if the base OS hasn't changed. You also have excessive compiling when you have multiple bosh directors deploying the same release (using the same stemcell).
  • Martin Englund at Jan 10, 2013 at 4:50 am
    Daniel,

    we already have that - you get it by running:
    bosh create release --with-tarball

    cheers,
    /Martin
    --
    Martin Englund, Staff Engineer, Cloud Foundry, VMware Inc.
    "The question is not if you are paranoid, it is if you are paranoid enough."
    On Jan 9, 2013, at 17:11, daniel.k.winsor@gmail.com wrote:

    Hi,

    After having to mess around with using different VMs/laptops for different parts of the create/deploy process, I had the idea that I could package a release into a tarball and distribute that, just like you can distribute a stemcell. That is, when I do $bosh upload release and I see
    Copying packages
    ...
    Building tarball
    ----------------
    Generated /tmp/d20130110-9288-3jg7h0/d20130110-9288-193ciw3/release.tgz
    Release size: 1.5G
    ...
    Uploading release
    release.tgz: 2% | | 32.3MB 825.8KB/s ETA: 00:31:53
    ...

    Then I'd rather do
    $bosh package release
    Copying packages
    ...
    Building tarball with every "FOUND LOCAL" package and job
    ----------------
    Generated /output_directory/release.tgz
    Release size: 1.5G

    $bosh upload release /output_directory/release.tgz
    Verifying release...
    File exists and readable OK
    Extract tarball OK
    Manifest exists OK
    Release name/version OK
    ...
    Uploading release
    release.tgz: 20% |oooo | 316.6MB 822.5KB/s ETA: 00:26:07


    That may open up future things like $bosh download public releases

    Just a thought

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupbosh-users @
postedJan 10, '13 at 1:11a
activeJan 11, '13 at 1:49p
posts10
users5

People

Translate

site design / logo © 2022 Grokbase