FAQ
I just ran into this issue myself. I'm running an old version of salt
(2014.1.0), but I'm updating this email thread in case others get hit with
it.

After running a saltutil.sync_all, my master node was effectively DDoS'ed.
I couldn't run commands on any minions and the CPU was pegged for each
worker process.

I tried various things (restarting the minions, clearing out the cache on
master and minions), but the only thing that got the server responsive
again was to delete all keys and slowly add them again.

I added the following things to my minion config and so far things are much
more stable. I'm also using --batch-size 25% flags when I run across the
entire fleet.

recon_randomize: True
random_reauth_delay: 120
On Fri, Jun 20, 2014 at 1:38 PM, Colton Myers wrote:

Well, that sounds like something we should look into. Could you please
file an issue on Github?

--
Colton Myers

On Mon, Jun 16, 2014 at 9:54 AM, Matt Hollingsworth wrote:

Nope, it pretty much can't be network related, since it is connecting via
loopback (the minion is on the same machine as the master it's
authenticating to), and iptables are completely open. Yep, it the minion ID
matches, and it works if I go through enough of these restart cycles, it
just doesn't work right off.

-Matt


On Fri, Jun 13, 2014 at 6:28 PM, Colton Myers <colton.myers@gmail.com>
wrote:
Anything unique about your network setup? Is there a lot of latency or
anything? I assume the case of the filename of the minion key on the
master matches the minion ID exactly?

--
Colton Myers


On Wed, Jun 11, 2014 at 8:34 AM, Matt Hollingsworth <mr.hworth@gmail.com
wrote:
Only one, which is both a master and a minion (we add the other minions
later, this is happening during a cluster bootstrapping process).

What I did to get around this problem is:

service salt-master restart
sleep 5
service salt-minion restart
sleep 5
salt-call state.sls.highstate

If salt-call exits non-zero, I do all of that again. I presently have
it set to retry 5 times. It usually works by the third or fourth try, but
that's the only way that it's been 100% reliable.

-Matt



On Tue, Jun 10, 2014 at 5:52 PM, Colton Myers <colton.myers@gmail.com>
wrote:
How many minions are we talking about? Are you bringing up one at a
time or many?

--
Colton Myers


On Fri, Jun 6, 2014 at 8:33 AM, Matt Hollingsworth <
mr.hworth@gmail.com> wrote:
I see the same behavior intermittently, except I am running on AWS. I
have a script that
- Brings up an instance
- Installs salt master and minion
- Preseeds keys per instructions at
http://docs.saltstack.com/en/latest/topics/tutorials/preseed_key.html
- Calls salt-call test.ping

This works about 80% of the time. The other 20%, it fails with

[WARNING ] SaltReqTimeoutError: Waited 60 seconds

Minion failed to authenticate with the master, has the minion key
been accepted?
Any hints as to what could be going on?

-Matt
On Tuesday, April 15, 2014 6:58:37 AM UTC-4, jdelic wrote:

Hey,

I need some help with this. Since salt 2014.1 my vagrant network
setup with pre-seeded keys fails with

"""
Minion failed to authenticate with the master, has the minion key
been accepted?
[WARNING ] SaltReqTimeoutError: Waited 3 seconds
"""

In this case already the master box can't authenticate against
itself. I'm running this on Debian wheezy with the salt distribution from
http://debian.saltstack.com/debian. The interesting part is that
the key has actually been accepted. I.e:

"""
root@saltmaster:/etc/salt# salt-key
Accepted Keys:
saltmaster.test
Unaccepted Keys:
Rejected Keys:
"""

and

"""
root@saltmaster:/etc/salt# openssl rsa -pubin -noout -modulus -in
/etc/salt/pki/master/minions/saltmaster.test | openssl md5
(stdin)= 5a59e647d449606efebc235f498ddaa8

root@saltmaster:/etc/salt# openssl rsa -noout -modulus -in
/etc/salt/pki/minion/minion.pem | openssl md5
(stdin)= 5a59e647d449606efebc235f498ddaa8
"""

when I run salt-call manually, I get:

"""
[DEBUG ] Reading configuration from /etc/salt/minion
[INFO ] Using cached minion ID from /etc/salt/minion_id:
saltmaster.test
[DEBUG ] Configuration file path: /etc/salt/minion
[DEBUG ] Reading configuration from /etc/salt/minion
[DEBUG ] Loaded minion key: /etc/salt/pki/minion/minion.pem
[WARNING ] SaltReqTimeoutError: Waited 3 seconds
Minion failed to authenticate with the master, has the minion key
been accepted?
"""

The /etc/hosts on this box contains:

"""
192.168.56.88 saltmaster.test
"""

and the interface is up and pingable:

"""
root@saltmaster:/etc/salt# ping saltmaster.test
PING saltmaster.test (127.0.1.1) 56(84) bytes of data.
64 bytes from saltmaster.test (127.0.1.1): icmp_req=1 ttl=64
time=0.022 ms
"""

Does anyone have an idea what can be wrong with this setup? or what
I can check next?

Best regards,
jdelic

--
You received this message because you are subscribed to the Google
Groups "Salt-users" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to salt-users+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to a topic in the
Google Groups "Salt-users" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/salt-users/zEA224dbXvo/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
salt-users+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google
Groups "Salt-users" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to salt-users+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to a topic in the
Google Groups "Salt-users" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/salt-users/zEA224dbXvo/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
salt-users+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups
"Salt-users" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to salt-users+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups
"Salt-users" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to salt-users+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


--
Stephen Wood
www.heystephenwood.com

--
You received this message because you are subscribed to the Google Groups "Salt-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to salt-users+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Search Discussions

Related Discussions

Discussion Navigation
viewthread | post
posts ‹ prev | 1 of 1 | next ›
Discussion Overview
groupsalt-users @
postedSep 15, '14 at 11:57p
activeSep 15, '14 at 11:57p
posts1
users1

1 user in discussion

Stephen Wood: 1 post

People

Translate

site design / logo © 2022 Grokbase