We're trying to fork cf-release and migrate our blobs to S3. We have been
able to move all of the blobs in cf-release/blobs over. That part was easy.
When we try to create a release doing 'bosh create release' we get errors
saying that it can't find packages in the blobstore.

Specifically we're getting:

Building DEV release
---------------------------------

Building packages
-----------------
Building daylimit...
Final version: NOT FOUND
Dev version: FOUND LOCAL

Building mysql55...
Final version: Downloading `mysql55 (4)'...
Final version of `mysql55' not found in blobstore



Is there an easy way to migrate the package blobs to S3?

-Mike

Search Discussions

  • Dr Nic Williams at Jan 16, 2013 at 10:57 pm
    bosh create release --force - should skip the check

    or
    bosh upload blobs
    bosh create release

    On Wed, Jan 16, 2013 at 2:19 PM, Mike Heath wrote:

    We're trying to fork cf-release and migrate our blobs to S3. We have been
    able to move all of the blobs in cf-release/blobs over. That part was easy.
    When we try to create a release doing 'bosh create release' we get errors
    saying that it can't find packages in the blobstore.

    Specifically we're getting:

    Building DEV release
    ---------------------------------

    Building packages
    -----------------
    Building daylimit...
    Final version: NOT FOUND
    Dev version: FOUND LOCAL

    Building mysql55...
    Final version: Downloading `mysql55 (4)'...
    Final version of `mysql55' not found in blobstore



    Is there an easy way to migrate the package blobs to S3?

    -Mike


    --
    Dr Nic Williams
    http://drnicwilliams.com
    cell +1 (415) 860-2185
  • Mike Heath at Jan 16, 2013 at 11:24 pm
    Adding --force eliminates the warnings about my changes not being committed
    in git but doesn't prevent bosh from trying to download packages from the
    blobstore, unfortunately.
    On Wednesday, January 16, 2013 3:57:19 PM UTC-7, Dr Nic Williams wrote:

    bosh create release --force - should skip the check

    or
    bosh upload blobs
    bosh create release


    On Wed, Jan 16, 2013 at 2:19 PM, Mike Heath <elc...@gmail.com<javascript:>
    wrote:
    We're trying to fork cf-release and migrate our blobs to S3. We have been
    able to move all of the blobs in cf-release/blobs over. That part was easy.
    When we try to create a release doing 'bosh create release' we get errors
    saying that it can't find packages in the blobstore.

    Specifically we're getting:

    Building DEV release
    ---------------------------------

    Building packages
    -----------------
    Building daylimit...
    Final version: NOT FOUND
    Dev version: FOUND LOCAL

    Building mysql55...
    Final version: Downloading `mysql55 (4)'...
    Final version of `mysql55' not found in blobstore



    Is there an easy way to migrate the package blobs to S3?

    -Mike


    --
    Dr Nic Williams
    http://drnicwilliams.com
    cell +1 (415) 860-2185
  • Dr Nic Williams at Jan 17, 2013 at 12:07 am
    What does "bosh upload blobs" do?

    On Wed, Jan 16, 2013 at 3:24 PM, Mike Heath wrote:

    Adding --force eliminates the warnings about my changes not being
    committed in git but doesn't prevent bosh from trying to download packages
    from the blobstore, unfortunately.

    On Wednesday, January 16, 2013 3:57:19 PM UTC-7, Dr Nic Williams wrote:

    bosh create release --force - should skip the check

    or
    bosh upload blobs
    bosh create release

    On Wed, Jan 16, 2013 at 2:19 PM, Mike Heath wrote:

    We're trying to fork cf-release and migrate our blobs to S3. We have
    been able to move all of the blobs in cf-release/blobs over. That part was
    easy. When we try to create a release doing 'bosh create release' we get
    errors saying that it can't find packages in the blobstore.

    Specifically we're getting:

    Building DEV release
    ------------------------------**---

    Building packages
    -----------------
    Building daylimit...
    Final version: NOT FOUND
    Dev version: FOUND LOCAL

    Building mysql55...
    Final version: Downloading `mysql55 (4)'...
    Final version of `mysql55' not found in blobstore



    Is there an easy way to migrate the package blobs to S3?

    -Mike


    --
    Dr Nic Williams
    http://drnicwilliams.com
    cell +1 (415) 860-2185

    --
    Dr Nic Williams
    http://drnicwilliams.com
    cell +1 (415) 860-2185
  • Nicholas Terry at Jan 17, 2013 at 12:32 am
    There is no simple way to do what you want. We dealt with this issue a
    little while back.The best way to accomplish this is to modify the source
    to use two blobstores. (Its how we had to do it)

    There may be a hacky way to get the blobstore populated...
    First, make sure the final.yml and provate.yml files in the config dir are
    the vanilla copies and do a bosh sync blobs

    1. Recreate the non-submodule directories in the src/ folder to the blobs
    directory (not to be confused with the (dot)blobs directory)
    2. For each symlink in each folder in the src dir, copy the referenced blob
    to its corresponding folder in the blobs directory. For example:

    Given: src/debian_nfs_server/whatever.tgz ->
    /path/to/.blobs/a44b3f8319984bace61f21bb6d15cee1419e5bf3 (THIS IS AN
    EXAMPLE)

    Do the following:

    cp /path/to/.blobs/a44b3f8319984bace61f21bb6d15cee1419e5bf3
    blobs/debian_nfs_server/whatever.tgz

    3. Empty out blobs.yml in the config folder so that is only contains --- {}
    4. Then change your private and final yaml files to you appropriate setup
    for your s3 store
    5. Then do a bosh upload blobs. You should be prompted to upload each blob
    in turn and blobs.yml will be populated with your blobs and their ids and
    stuff.


    If you are unsure of how to format you final and provate yml files, go here:

    https://github.com/cloudfoundry/bosh/blob/master/cli/lib/cli/release.rb#L72-L86

    make sure you have provisioned your s3 bucket correctly you need to follow
    these directions:

    https://github.com/cloudfoundry/oss-docs/blob/master/bosh/documentation/bosh_releases.md#setting-up-a-new-release-repository-using-s3

    (The bucket policy referenced there is incorrect. Use the IAM builder to
    make it)

    Let me know if this works for you
  • Martin Englund at Jan 17, 2013 at 12:35 am
    Excellent write-up! Thanks for taking the time to do it

    /M

    Sent from my iPhone
    On Jan 16, 2013, at 16:25, Nicholas Terry wrote:

    There is no simple way to do what you want. We dealt with this issue a little while back.The best way to accomplish this is to modify the source to use two blobstores. (Its how we had to do it)

    There may be a hacky way to get the blobstore populated...
    First, make sure the final.yml and provate.yml files in the config dir are the vanilla copies and do a bosh sync blobs

    1. Recreate the non-submodule directories in the src/ folder to the blobs directory (not to be confused with the (dot)blobs directory)
    2. For each symlink in each folder in the src dir, copy the referenced blob to its corresponding folder in the blobs directory. For example:

    Given: src/debian_nfs_server/whatever.tgz -> /path/to/.blobs/a44b3f8319984bace61f21bb6d15cee1419e5bf3 (THIS IS AN EXAMPLE)

    Do the following:

    cp /path/to/.blobs/a44b3f8319984bace61f21bb6d15cee1419e5bf3 blobs/debian_nfs_server/whatever.tgz

    3. Empty out blobs.yml in the config folder so that is only contains --- {}
    4. Then change your private and final yaml files to you appropriate setup for your s3 store
    5. Then do a bosh upload blobs. You should be prompted to upload each blob in turn and blobs.yml will be populated with your blobs and their ids and stuff.


    If you are unsure of how to format you final and provate yml files, go here:

    https://github.com/cloudfoundry/bosh/blob/master/cli/lib/cli/release.rb#L72-L86

    make sure you have provisioned your s3 bucket correctly you need to follow these directions:

    https://github.com/cloudfoundry/oss-docs/blob/master/bosh/documentation/bosh_releases.md#setting-up-a-new-release-repository-using-s3

    (The bucket policy referenced there is incorrect. Use the IAM builder to make it)

    Let me know if this works for you
  • Mike Heath at Jan 17, 2013 at 12:54 am
    It uploads all the blobs in my blobs/* directory just fine. I have those
    uploaded to S3 and it works great. I have a script that keeps the
    cf-release blobs synced with our blobs in S3. The other devs on my team can
    checkout our cf-release fork and do a 'bosh sync blobs' and all the blobs
    in the in the blobs directory get downloaded. No problems there.

    The problem we're seeing (that I did not anticipate) is that it appears
    that jobs and packages get cached in the blobstore as well. The index.yml
    files in /.final_builds are full of blobstore_id fields that reference what
    appear to be tarballs in the http://blob.cfblob.com blobstore.

    These job and package blobs are not listed in the config/blobs.yml. Which
    is why I was surprised to find this problem. I like the idea of caching
    these, especially to ensure consistency for future deployments. I don't
    like that I can't do a "bosh add blob ..." to get them in S3. :)

    I've been chatting with Nick Terry about his patch. We'll likely go down
    that route. I like the idea of being able to chain blobstores.

    -Mike

    On Wed, Jan 16, 2013 at 5:07 PM, Dr Nic Williams wrote:

    What does "bosh upload blobs" do?

    On Wed, Jan 16, 2013 at 3:24 PM, Mike Heath wrote:

    Adding --force eliminates the warnings about my changes not being
    committed in git but doesn't prevent bosh from trying to download packages
    from the blobstore, unfortunately.

    On Wednesday, January 16, 2013 3:57:19 PM UTC-7, Dr Nic Williams wrote:

    bosh create release --force - should skip the check

    or
    bosh upload blobs
    bosh create release

    On Wed, Jan 16, 2013 at 2:19 PM, Mike Heath wrote:

    We're trying to fork cf-release and migrate our blobs to S3. We have
    been able to move all of the blobs in cf-release/blobs over. That part was
    easy. When we try to create a release doing 'bosh create release' we get
    errors saying that it can't find packages in the blobstore.

    Specifically we're getting:

    Building DEV release
    ------------------------------**---

    Building packages
    -----------------
    Building daylimit...
    Final version: NOT FOUND
    Dev version: FOUND LOCAL

    Building mysql55...
    Final version: Downloading `mysql55 (4)'...
    Final version of `mysql55' not found in blobstore



    Is there an easy way to migrate the package blobs to S3?

    -Mike


    --
    Dr Nic Williams
    http://drnicwilliams.com
    cell +1 (415) 860-2185

    --
    Dr Nic Williams
    http://drnicwilliams.com
    cell +1 (415) 860-2185
  • Dr Nic Williams at Jan 17, 2013 at 1:00 am
    Chaining/overriding blob stores sounds very nice. I look forward to seeing
    any finish products/repos/gists.
    On Wednesday, January 16, 2013, Mike Heath wrote:

    It uploads all the blobs in my blobs/* directory just fine. I have those
    uploaded to S3 and it works great. I have a script that keeps the
    cf-release blobs synced with our blobs in S3. The other devs on my team can
    checkout our cf-release fork and do a 'bosh sync blobs' and all the blobs
    in the in the blobs directory get downloaded. No problems there.

    The problem we're seeing (that I did not anticipate) is that it appears
    that jobs and packages get cached in the blobstore as well. The index.yml
    files in /.final_builds are full of blobstore_id fields that reference what
    appear to be tarballs in the http://blob.cfblob.com blobstore.

    These job and package blobs are not listed in the config/blobs.yml. Which
    is why I was surprised to find this problem. I like the idea of caching
    these, especially to ensure consistency for future deployments. I don't
    like that I can't do a "bosh add blob ..." to get them in S3. :)

    I've been chatting with Nick Terry about his patch. We'll likely go down
    that route. I like the idea of being able to chain blobstores.

    -Mike


    On Wed, Jan 16, 2013 at 5:07 PM, Dr Nic Williams ({}, 'cvml', 'drnicwilliams@gmail.com');>
    wrote:
    What does "bosh upload blobs" do?


    On Wed, Jan 16, 2013 at 3:24 PM, Mike Heath ({}, 'cvml', 'elcapo@gmail.com');>
    wrote:
    Adding --force eliminates the warnings about my changes not being
    committed in git but doesn't prevent bosh from trying to download packages
    from the blobstore, unfortunately.

    On Wednesday, January 16, 2013 3:57:19 PM UTC-7, Dr Nic Williams wrote:

    bosh create release --force - should skip the check

    or
    bosh upload blobs
    bosh create release

    On Wed, Jan 16, 2013 at 2:19 PM, Mike Heath wrote:

    We're trying to fork cf-release and migrate our blobs to S3. We have
    been able to move all of the blobs in cf-release/blobs over. That part was
    easy. When we try to create a release doing 'bosh create release' we get
    errors saying that it can't find packages in the blobstore.

    Specifically we're getting:

    Building DEV release
    ------------------------------**---

    Building packages
    -----------------
    Building daylimit...
    Final version: NOT FOUND
    Dev version: FOUND LOCAL

    Building mysql55...
    Final version: Downloading `mysql55 (4)'...
    Final version of `mysql55' not found in blobstore



    Is there an easy way to migrate the package blobs to S3?

    -Mike


    --
    Dr Nic Williams
    http://drnicwilliams.com
    cell +1 (415) 860-2185

    --
    Dr Nic Williams
    http://drnicwilliams.com
    cell +1 (415) 860-2185
    --
    Dr Nic Williams
    http://drnicwilliams.com
    cell +1 (415) 860-2185
  • Nterry at Jan 17, 2013 at 11:00 pm
    YAY! We committed the composite blobstore client to be accepted!!! See it
    here:

    http://reviews.cloudfoundry.org/#/c/13445/


    Also whats the process for submitting fixes/additions to the oss-docs
    project (we need to document the format for the private and final yml files
    to use the composite blobstore client)? Do I just create a pull request in
    github?
    On Wednesday, January 16, 2013 6:00:40 PM UTC-7, Dr Nic Williams wrote:

    Chaining/overriding blob stores sounds very nice. I look forward to seeing
    any finish products/repos/gists.
    On Wednesday, January 16, 2013, Mike Heath wrote:

    It uploads all the blobs in my blobs/* directory just fine. I have those
    uploaded to S3 and it works great. I have a script that keeps the
    cf-release blobs synced with our blobs in S3. The other devs on my team can
    checkout our cf-release fork and do a 'bosh sync blobs' and all the blobs
    in the in the blobs directory get downloaded. No problems there.

    The problem we're seeing (that I did not anticipate) is that it appears
    that jobs and packages get cached in the blobstore as well. The index.yml
    files in /.final_builds are full of blobstore_id fields that reference what
    appear to be tarballs in the http://blob.cfblob.com blobstore.

    These job and package blobs are not listed in the config/blobs.yml. Which
    is why I was surprised to find this problem. I like the idea of caching
    these, especially to ensure consistency for future deployments. I don't
    like that I can't do a "bosh add blob ..." to get them in S3. :)

    I've been chatting with Nick Terry about his patch. We'll likely go down
    that route. I like the idea of being able to chain blobstores.

    -Mike


    On Wed, Jan 16, 2013 at 5:07 PM, Dr Nic Williams <drnicwilliams@gmail.com
    wrote:
    What does "bosh upload blobs" do?

    On Wed, Jan 16, 2013 at 3:24 PM, Mike Heath wrote:

    Adding --force eliminates the warnings about my changes not being
    committed in git but doesn't prevent bosh from trying to download packages
    from the blobstore, unfortunately.

    On Wednesday, January 16, 2013 3:57:19 PM UTC-7, Dr Nic Williams wrote:

    bosh create release --force - should skip the check

    or
    bosh upload blobs
    bosh create release

    On Wed, Jan 16, 2013 at 2:19 PM, Mike Heath wrote:

    We're trying to fork cf-release and migrate our blobs to S3. We have
    been able to move all of the blobs in cf-release/blobs over. That part was
    easy. When we try to create a release doing 'bosh create release' we get
    errors saying that it can't find packages in the blobstore.

    Specifically we're getting:

    Building DEV release
    ------------------------------**---

    Building packages
    -----------------
    Building daylimit...
    Final version: NOT FOUND
    Dev version: FOUND LOCAL

    Building mysql55...
    Final version: Downloading `mysql55 (4)'...
    Final version of `mysql55' not found in blobstore



    Is there an easy way to migrate the package blobs to S3?

    -Mike


    --
    Dr Nic Williams
    http://drnicwilliams.com
    cell +1 (415) 860-2185

    --
    Dr Nic Williams
    http://drnicwilliams.com
    cell +1 (415) 860-2185
    --
    Dr Nic Williams
    http://drnicwilliams.com
    cell +1 (415) 860-2185
  • Dr Nic Williams at Jan 17, 2013 at 11:28 pm
    CONGRATS!
    Well done for submitting a (first?) patch!

    Nic

    On Thu, Jan 17, 2013 at 3:00 PM, wrote:

    YAY! We committed the composite blobstore client to be accepted!!! See it
    here:

    http://reviews.cloudfoundry.org/#/c/13445/


    Also whats the process for submitting fixes/additions to the oss-docs
    project (we need to document the format for the private and final yml files
    to use the composite blobstore client)? Do I just create a pull request in
    github?

    On Wednesday, January 16, 2013 6:00:40 PM UTC-7, Dr Nic Williams wrote:

    Chaining/overriding blob stores sounds very nice. I look forward to
    seeing any finish products/repos/gists.
    On Wednesday, January 16, 2013, Mike Heath wrote:

    It uploads all the blobs in my blobs/* directory just fine. I have those
    uploaded to S3 and it works great. I have a script that keeps the
    cf-release blobs synced with our blobs in S3. The other devs on my team can
    checkout our cf-release fork and do a 'bosh sync blobs' and all the blobs
    in the in the blobs directory get downloaded. No problems there.

    The problem we're seeing (that I did not anticipate) is that it appears
    that jobs and packages get cached in the blobstore as well. The index.yml
    files in /.final_builds are full of blobstore_id fields that reference what
    appear to be tarballs in the http://blob.cfblob.com blobstore.

    These job and package blobs are not listed in the config/blobs.yml.
    Which is why I was surprised to find this problem. I like the idea of
    caching these, especially to ensure consistency for future deployments. I
    don't like that I can't do a "bosh add blob ..." to get them in S3. :)

    I've been chatting with Nick Terry about his patch. We'll likely go down
    that route. I like the idea of being able to chain blobstores.

    -Mike


    On Wed, Jan 16, 2013 at 5:07 PM, Dr Nic Williams <
    drnicwilliams@gmail.com> wrote:
    What does "bosh upload blobs" do?

    On Wed, Jan 16, 2013 at 3:24 PM, Mike Heath wrote:

    Adding --force eliminates the warnings about my changes not being
    committed in git but doesn't prevent bosh from trying to download packages
    from the blobstore, unfortunately.

    On Wednesday, January 16, 2013 3:57:19 PM UTC-7, Dr Nic Williams wrote:

    bosh create release --force - should skip the check

    or
    bosh upload blobs
    bosh create release

    On Wed, Jan 16, 2013 at 2:19 PM, Mike Heath wrote:

    We're trying to fork cf-release and migrate our blobs to S3. We have
    been able to move all of the blobs in cf-release/blobs over. That part was
    easy. When we try to create a release doing 'bosh create release' we get
    errors saying that it can't find packages in the blobstore.

    Specifically we're getting:

    Building DEV release
    ------------------------------****---

    Building packages
    -----------------
    Building daylimit...
    Final version: NOT FOUND
    Dev version: FOUND LOCAL

    Building mysql55...
    Final version: Downloading `mysql55 (4)'...
    Final version of `mysql55' not found in blobstore



    Is there an easy way to migrate the package blobs to S3?

    -Mike


    --
    Dr Nic Williams
    http://drnicwilliams.com
    cell +1 (415) 860-2185

    --
    Dr Nic Williams
    http://drnicwilliams.com
    cell +1 (415) 860-2185
    --
    Dr Nic Williams
    http://drnicwilliams.com
    cell +1 (415) 860-2185

    --
    Dr Nic Williams
    http://drnicwilliams.com
    cell +1 (415) 860-2185
  • Dr Nic Williams at Jan 17, 2013 at 11:28 pm
    AFAIK, oss-docs is also on gerrit at the moment.

    On Thu, Jan 17, 2013 at 3:20 PM, Dr Nic Williams wrote:

    CONGRATS!
    Well done for submitting a (first?) patch!

    Nic

    On Thu, Jan 17, 2013 at 3:00 PM, wrote:

    YAY! We committed the composite blobstore client to be accepted!!! See it
    here:

    http://reviews.cloudfoundry.org/#/c/13445/


    Also whats the process for submitting fixes/additions to the oss-docs
    project (we need to document the format for the private and final yml files
    to use the composite blobstore client)? Do I just create a pull request in
    github?

    On Wednesday, January 16, 2013 6:00:40 PM UTC-7, Dr Nic Williams wrote:

    Chaining/overriding blob stores sounds very nice. I look forward to
    seeing any finish products/repos/gists.
    On Wednesday, January 16, 2013, Mike Heath wrote:

    It uploads all the blobs in my blobs/* directory just fine. I have
    those uploaded to S3 and it works great. I have a script that keeps the
    cf-release blobs synced with our blobs in S3. The other devs on my team can
    checkout our cf-release fork and do a 'bosh sync blobs' and all the blobs
    in the in the blobs directory get downloaded. No problems there.

    The problem we're seeing (that I did not anticipate) is that it appears
    that jobs and packages get cached in the blobstore as well. The index.yml
    files in /.final_builds are full of blobstore_id fields that reference what
    appear to be tarballs in the http://blob.cfblob.com blobstore.

    These job and package blobs are not listed in the config/blobs.yml.
    Which is why I was surprised to find this problem. I like the idea of
    caching these, especially to ensure consistency for future deployments. I
    don't like that I can't do a "bosh add blob ..." to get them in S3. :)

    I've been chatting with Nick Terry about his patch. We'll likely go
    down that route. I like the idea of being able to chain blobstores.

    -Mike


    On Wed, Jan 16, 2013 at 5:07 PM, Dr Nic Williams <
    drnicwilliams@gmail.com> wrote:
    What does "bosh upload blobs" do?

    On Wed, Jan 16, 2013 at 3:24 PM, Mike Heath wrote:

    Adding --force eliminates the warnings about my changes not being
    committed in git but doesn't prevent bosh from trying to download packages
    from the blobstore, unfortunately.


    On Wednesday, January 16, 2013 3:57:19 PM UTC-7, Dr Nic Williams
    wrote:
    bosh create release --force - should skip the check

    or
    bosh upload blobs
    bosh create release

    On Wed, Jan 16, 2013 at 2:19 PM, Mike Heath wrote:

    We're trying to fork cf-release and migrate our blobs to S3. We
    have been able to move all of the blobs in cf-release/blobs over. That part
    was easy. When we try to create a release doing 'bosh create release' we
    get errors saying that it can't find packages in the blobstore.

    Specifically we're getting:

    Building DEV release
    ------------------------------****---

    Building packages
    -----------------
    Building daylimit...
    Final version: NOT FOUND
    Dev version: FOUND LOCAL

    Building mysql55...
    Final version: Downloading `mysql55 (4)'...
    Final version of `mysql55' not found in blobstore



    Is there an easy way to migrate the package blobs to S3?

    -Mike


    --
    Dr Nic Williams
    http://drnicwilliams.com
    cell +1 (415) 860-2185

    --
    Dr Nic Williams
    http://drnicwilliams.com
    cell +1 (415) 860-2185
    --
    Dr Nic Williams
    http://drnicwilliams.com
    cell +1 (415) 860-2185

    --
    Dr Nic Williams
    http://drnicwilliams.com
    cell +1 (415) 860-2185


    --
    Dr Nic Williams
    http://drnicwilliams.com
    cell +1 (415) 860-2185
  • James Bayer at Jan 18, 2013 at 1:51 am
    Actually we're in the middle of transitioning some processes for contributing to be a bit simpler, can you try a standard github pull request and see if it gets auto-closed? I think it's not going to do that any longer.

    Thank you,

    James Bayer

    On Jan 17, 2013, at 3:20 PM, Dr Nic Williams wrote:

    AFAIK, oss-docs is also on gerrit at the moment.


    On Thu, Jan 17, 2013 at 3:20 PM, Dr Nic Williams wrote:
    CONGRATS!
    Well done for submitting a (first?) patch!

    Nic


    On Thu, Jan 17, 2013 at 3:00 PM, wrote:
    YAY! We committed the composite blobstore client to be accepted!!! See it here:

    http://reviews.cloudfoundry.org/#/c/13445/


    Also whats the process for submitting fixes/additions to the oss-docs project (we need to document the format for the private and final yml files to use the composite blobstore client)? Do I just create a pull request in github?


    On Wednesday, January 16, 2013 6:00:40 PM UTC-7, Dr Nic Williams wrote:
    Chaining/overriding blob stores sounds very nice. I look forward to seeing any finish products/repos/gists.

    On Wednesday, January 16, 2013, Mike Heath wrote:
    It uploads all the blobs in my blobs/* directory just fine. I have those uploaded to S3 and it works great. I have a script that keeps the cf-release blobs synced with our blobs in S3. The other devs on my team can checkout our cf-release fork and do a 'bosh sync blobs' and all the blobs in the in the blobs directory get downloaded. No problems there.

    The problem we're seeing (that I did not anticipate) is that it appears that jobs and packages get cached in the blobstore as well. The index.yml files in /.final_builds are full of blobstore_id fields that reference what appear to be tarballs in the http://blob.cfblob.com blobstore.

    These job and package blobs are not listed in the config/blobs.yml. Which is why I was surprised to find this problem. I like the idea of caching these, especially to ensure consistency for future deployments. I don't like that I can't do a "bosh add blob ..." to get them in S3. :)

    I've been chatting with Nick Terry about his patch. We'll likely go down that route. I like the idea of being able to chain blobstores.

    -Mike


    On Wed, Jan 16, 2013 at 5:07 PM, Dr Nic Williams wrote:
    What does "bosh upload blobs" do?


    On Wed, Jan 16, 2013 at 3:24 PM, Mike Heath wrote:
    Adding --force eliminates the warnings about my changes not being committed in git but doesn't prevent bosh from trying to download packages from the blobstore, unfortunately.


    On Wednesday, January 16, 2013 3:57:19 PM UTC-7, Dr Nic Williams wrote:
    bosh create release --force - should skip the check

    or
    bosh upload blobs
    bosh create release


    On Wed, Jan 16, 2013 at 2:19 PM, Mike Heath wrote:
    We're trying to fork cf-release and migrate our blobs to S3. We have been able to move all of the blobs in cf-release/blobs over. That part was easy. When we try to create a release doing 'bosh create release' we get errors saying that it can't find packages in the blobstore.

    Specifically we're getting:

    Building DEV release
    ---------------------------------

    Building packages
    -----------------
    Building daylimit...
    Final version: NOT FOUND
    Dev version: FOUND LOCAL

    Building mysql55...
    Final version: Downloading `mysql55 (4)'...
    Final version of `mysql55' not found in blobstore



    Is there an easy way to migrate the package blobs to S3?

    -Mike



    --
    Dr Nic Williams
    http://drnicwilliams.com
    cell +1 (415) 860-2185



    --
    Dr Nic Williams
    http://drnicwilliams.com
    cell +1 (415) 860-2185



    --
    Dr Nic Williams
    http://drnicwilliams.com
    cell +1 (415) 860-2185




    --
    Dr Nic Williams
    http://drnicwilliams.com
    cell +1 (415) 860-2185



    --
    Dr Nic Williams
    http://drnicwilliams.com
    cell +1 (415) 860-2185

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupbosh-users @
postedJan 16, '13 at 10:19p
activeJan 18, '13 at 1:51a
posts12
users6

People

Translate

site design / logo © 2022 Grokbase