FAQ
I'm doing something wrong here. I can't seem to specify correctly the
source of a file that exists on S3. I'm continually getting the message:

Unable to determine upstream hash of source file s3://salt/test_file

In my master config I have these settings:

...
s3.keyid: <snip>
s3.key: <snip>

fileserver_backend:
   - roots
   - s3fs

s3.buckets:
   - test_bucket

The s3 url for the file I want to grab is:

s3://test_bucket/salt/test_file

My impression was that I could create a state.sls file that looks like this:

/tmp/test_file:
   file.managed:
     - source: s3://salt/test_file

Is there something wrong with my configuration? I can verify with the keys
I have set in the config that I can download that file.
--
Stephen Wood

--
You received this message because you are subscribed to the Google Groups "Salt-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to salt-users+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Search Discussions

  • Alan Garfield at Aug 20, 2014 at 11:21 pm
    it's cached on the master so you use salt:// instead of s3://

    -A.

    On Thu, Aug 21, 2014 at 9:00 AM, Stephen Wood wrote:

    I'm doing something wrong here. I can't seem to specify correctly the
    source of a file that exists on S3. I'm continually getting the message:

    Unable to determine upstream hash of source file s3://salt/test_file

    In my master config I have these settings:

    ...
    s3.keyid: <snip>
    s3.key: <snip>

    fileserver_backend:
    - roots
    - s3fs

    s3.buckets:
    - test_bucket

    The s3 url for the file I want to grab is:

    s3://test_bucket/salt/test_file

    My impression was that I could create a state.sls file that looks like
    this:

    /tmp/test_file:
    file.managed:
    - source: s3://salt/test_file

    Is there something wrong with my configuration? I can verify with the keys
    I have set in the config that I can download that file.
    --
    Stephen Wood

    --
    You received this message because you are subscribed to the Google Groups
    "Salt-users" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to salt-users+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
    --
    You received this message because you are subscribed to the Google Groups "Salt-users" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to salt-users+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Stephen Wood at Aug 20, 2014 at 11:36 pm
    Thanks for the response.

    Is there a period of caching that's required before the file is
    "discoverable"? I am getting the message:

    Source file salt://my_folder/my_file not found

    However with tcpdump I can see that there is a significant amount of
    traffic to and from S3.

    --
    You received this message because you are subscribed to the Google Groups "Salt-users" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to salt-users+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Alan Garfield at Aug 20, 2014 at 11:39 pm
    http://docs.saltstack.com/en/latest/ref/file_server/all/salt.fileserver.s3fs.html

    Have you got your bucket paths in the expected format?

    The s3fs module looks for changes in the S3 buckets every 60 seconds, and
    pulls down any files in those buckets to a local cache. Look in the salt
    masters log file and you should see it doing just that.

    -A.

    On Thu, Aug 21, 2014 at 9:35 AM, Stephen Wood wrote:

    Thanks for the response.

    Is there a period of caching that's required before the file is
    "discoverable"? I am getting the message:

    Source file salt://my_folder/my_file not found

    However with tcpdump I can see that there is a significant amount of
    traffic to and from S3.

    --
    You received this message because you are subscribed to the Google Groups
    "Salt-users" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to salt-users+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
    --
    You received this message because you are subscribed to the Google Groups "Salt-users" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to salt-users+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Stephen Wood at Aug 20, 2014 at 11:46 pm
    Does it cache all objects in the bucket, even if they aren't requested?

    For example, if I have a bucket that contains 100 gigs of objects, will all
    of those files be downloaded locally before they are available to minions?

    --
    You received this message because you are subscribed to the Google Groups "Salt-users" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to salt-users+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Alan Garfield at Aug 20, 2014 at 11:49 pm
    Yes. So don't use a bucket with stuff in it you're not expecting to deliver
    to minions.

    -A.

    On Thu, Aug 21, 2014 at 9:45 AM, Stephen Wood wrote:

    Does it cache all objects in the bucket, even if they aren't requested?

    For example, if I have a bucket that contains 100 gigs of objects, will
    all of those files be downloaded locally before they are available to
    minions?

    --
    You received this message because you are subscribed to the Google Groups
    "Salt-users" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to salt-users+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
    --
    You received this message because you are subscribed to the Google Groups "Salt-users" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to salt-users+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Stephen Wood at Aug 21, 2014 at 12:02 am
    Ah, that makes sense.

    My guess is that the master is busy trying to synchronize a bucket that
    contains a few million objects containing many hundreds of TB of data.
    Checking /var/cache/salt/master/s3cache I can see that the master has
    download just a fraction of the bucket. Maybe that's why I'm getting the
    "source file not found" error? The file I'm looking for does not yet exist
    in the cache.

    I'd be very surprised if it can check md5s for all of the objects in this
    bucket in an hour, let alone 60 seconds. Has this module been tested with
    large buckets?

    On Wed, Aug 20, 2014 at 4:49 PM, Alan Garfield wrote:

    Yes. So don't use a bucket with stuff in it you're not expecting to
    deliver to minions.

    -A.

    On Thu, Aug 21, 2014 at 9:45 AM, Stephen Wood wrote:

    Does it cache all objects in the bucket, even if they aren't requested?

    For example, if I have a bucket that contains 100 gigs of objects, will
    all of those files be downloaded locally before they are available to
    minions?

    --
    You received this message because you are subscribed to the Google Groups
    "Salt-users" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to salt-users+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
    --
    You received this message because you are subscribed to the Google Groups
    "Salt-users" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to salt-users+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.


    --
    Stephen Wood
    www.heystephenwood.com

    --
    You received this message because you are subscribed to the Google Groups "Salt-users" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to salt-users+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Alan Garfield at Aug 21, 2014 at 12:08 am
    We use it every day to handle our automatic deploy tarballs, once it's got
    the bucket down initially it just pulls down the deltas there after. So the
    first pull is slow, but the next ones are as fast as the changes made. But
    if the bucket is truly huge and the contents aren't relevant to the minions
    I would move what you want to a separate bucket.

    -A.

    On Thu, Aug 21, 2014 at 10:01 AM, Stephen Wood wrote:

    Ah, that makes sense.

    My guess is that the master is busy trying to synchronize a bucket that
    contains a few million objects containing many hundreds of TB of data.
    Checking /var/cache/salt/master/s3cache I can see that the master has
    download just a fraction of the bucket. Maybe that's why I'm getting the
    "source file not found" error? The file I'm looking for does not yet exist
    in the cache.

    I'd be very surprised if it can check md5s for all of the objects in this
    bucket in an hour, let alone 60 seconds. Has this module been tested with
    large buckets?


    On Wed, Aug 20, 2014 at 4:49 PM, Alan Garfield <
    alan.garfield@learnosity.com> wrote:
    Yes. So don't use a bucket with stuff in it you're not expecting to
    deliver to minions.

    -A.

    On Thu, Aug 21, 2014 at 9:45 AM, Stephen Wood wrote:

    Does it cache all objects in the bucket, even if they aren't requested?

    For example, if I have a bucket that contains 100 gigs of objects, will
    all of those files be downloaded locally before they are available to
    minions?

    --
    You received this message because you are subscribed to the Google
    Groups "Salt-users" group.
    To unsubscribe from this group and stop receiving emails from it, send
    an email to salt-users+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
    --
    You received this message because you are subscribed to the Google Groups
    "Salt-users" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to salt-users+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.


    --
    Stephen Wood
    www.heystephenwood.com

    --
    You received this message because you are subscribed to the Google Groups
    "Salt-users" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to salt-users+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
    --
    You received this message because you are subscribed to the Google Groups "Salt-users" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to salt-users+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupsalt-users @
postedAug 20, '14 at 11:00p
activeAug 21, '14 at 12:08a
posts8
users2

2 users in discussion

Stephen Wood: 4 posts Alan Garfield: 4 posts

People

Translate

site design / logo © 2022 Grokbase