FAQ
Hi,

I currently have a MySQL database containing all Puppet storeconfigs. My
intention is to migrate to PuppetDB on a PostgreSQL server, so the first
step is to use the 'storeconfigs' face to export all the DB to a file
PuppetDB can later consume. But the 'puppet storeconfigs export' command
always ends up being killed, I suspect due to some sort of OOM issue. I
have tried different environments, different Puppet / Ruby / whatever
versions, different hardware... No matter what, it always ends up the same.

The process starts well, connects to the database and fetches all the data,
but then it seems to get stuck after this output line:
Info: Connecting to mysql2 database: puppet

Meanwhile, a CPU is working and the amount of memory used by this process
increases constantly. I have tried on several servers, one with up to 28GB
of RAM, and it died the same way.

Here are the facts of my last attempt:

The execution output:

[root@ST-L226-ROGTORG ~]# time puppet storeconfigs --verbose --debug export
Debug: Using settings: adding file resource 'confdir':
'File[/etc/puppet]{:path=>"/etc/puppet", :ensure=>:directory,
:loglevel=>:debug, :links=>:follow, :backup=>false}'
Debug: Using settings: adding file resource 'vardir':
'File[/var/lib/puppet]{:path=>"/var/lib/puppet", :ensure=>:directory,
:loglevel=>:debug, :links=>:follow, :backup=>false}'
Debug: Failed to load library 'ldap' for feature 'ldap'
Debug: Puppet::Type::User::ProviderLdap: feature ldap is missing
Debug: Puppet::Type::User::ProviderDirectoryservice: file /usr/bin/dsimport
does not exist
Debug: Puppet::Type::User::ProviderPw: file pw does not exist
Debug: Puppet::Type::User::ProviderUser_role_add: file roleadd does not
exist
Debug: Failed to load library 'ldap' for feature 'ldap'
Debug: Puppet::Type::Group::ProviderLdap: feature ldap is missing
Debug: Puppet::Type::Group::ProviderDirectoryservice: file /usr/bin/dscl
does not exist
Debug: Puppet::Type::Group::ProviderPw: file pw does not exist
Debug: Using settings: adding file resource 'logdir':
'File[/var/log/puppet]{:path=>"/var/log/puppet", :mode=>"750",
:owner=>"puppet", :group=>"puppet", :ensure=>:directory, :loglevel=>:debug,
:links=>:follow, :backup=>false}'
Debug: Using settings: adding file resource 'statedir':
'File[/var/lib/puppet/state]{:path=>"/var/lib/puppet/state", :mode=>"1755",
:ensure=>:directory, :loglevel=>:debug, :links=>:follow, :backup=>false}'
Debug: Using settings: adding file resource 'rundir':
'File[/var/run/puppet]{:path=>"/var/run/puppet", :mode=>"755",
:owner=>"puppet", :group=>"puppet", :ensure=>:directory, :loglevel=>:debug,
:links=>:follow, :backup=>false}'
Debug: Using settings: adding file resource 'libdir':
'File[/var/lib/puppet/lib]{:path=>"/var/lib/puppet/lib",
:ensure=>:directory, :loglevel=>:debug, :links=>:follow, :backup=>false}'
Debug: Using settings: adding file resource 'certdir':
'File[/var/lib/puppet/ssl/certs]{:path=>"/var/lib/puppet/ssl/certs",
:owner=>"puppet", :ensure=>:directory, :loglevel=>:debug, :links=>:follow,
:backup=>false}'
Debug: Using settings: adding file resource 'ssldir':
'File[/var/lib/puppet/ssl]{:path=>"/var/lib/puppet/ssl", :mode=>"771",
:owner=>"puppet", :ensure=>:directory, :loglevel=>:debug, :links=>:follow,
:backup=>false}'
Debug: Using settings: adding file resource 'publickeydir':
'File[/var/lib/puppet/ssl/public_keys]{:path=>"/var/lib/puppet/ssl/public_keys",
:owner=>"puppet", :ensure=>:directory, :loglevel=>:debug, :links=>:follow,
:backup=>false}'
Debug: Using settings: adding file resource 'requestdir':
'File[/var/lib/puppet/ssl/certificate_requests]{:path=>"/var/lib/puppet/ssl/certificate_requests",
:owner=>"puppet", :ensure=>:directory, :loglevel=>:debug, :links=>:follow,
:backup=>false}'
Debug: Using settings: adding file resource 'privatekeydir':
'File[/var/lib/puppet/ssl/private_keys]{:path=>"/var/lib/puppet/ssl/private_keys",
:mode=>"750", :owner=>"puppet", :ensure=>:directory, :loglevel=>:debug,
:links=>:follow, :backup=>false}'
Debug: Using settings: adding file resource 'privatedir':
'File[/var/lib/puppet/ssl/private]{:path=>"/var/lib/puppet/ssl/private",
:mode=>"750", :owner=>"puppet", :ensure=>:directory, :loglevel=>:debug,
:links=>:follow, :backup=>false}'
Debug: Using settings: adding file resource 'manifestdir':
'File[/etc/puppet/manifests]{:path=>"/etc/puppet/manifests",
:ensure=>:directory, :loglevel=>:debug, :links=>:follow, :backup=>false}'
Debug: Using settings: adding file resource 'masterhttplog':
'File[/var/log/puppet/masterhttp.log]{:path=>"/var/log/puppet/masterhttp.log",
:mode=>"660", :owner=>"puppet", :group=>"puppet", :ensure=>:file,
:loglevel=>:debug, :links=>:follow, :backup=>false}'
Debug: Using settings: adding file resource 'bucketdir':
'File[/var/lib/puppet/bucket]{:path=>"/var/lib/puppet/bucket",
:mode=>"750", :owner=>"puppet", :group=>"puppet", :ensure=>:directory,
:loglevel=>:debug, :links=>:follow, :backup=>false}'
Debug: Using settings: adding file resource 'rest_authconfig':
'File[/etc/puppet/auth.conf]{:path=>"/etc/puppet/auth.conf",
:ensure=>:file, :loglevel=>:debug, :links=>:follow, :backup=>false}'
Debug: Using settings: adding file resource 'yamldir':
'File[/var/lib/puppet/yaml]{:path=>"/var/lib/puppet/yaml", :mode=>"750",
:owner=>"puppet", :group=>"puppet", :ensure=>:directory, :loglevel=>:debug,
:links=>:follow, :backup=>false}'
Debug: Using settings: adding file resource 'server_datadir':
'File[/var/lib/puppet/server_data]{:path=>"/var/lib/puppet/server_data",
:mode=>"750", :owner=>"puppet", :group=>"puppet", :ensure=>:directory,
:loglevel=>:debug, :links=>:follow, :backup=>false}'
Debug: Using settings: adding file resource 'reportdir':
'File[/var/lib/puppet/reports]{:path=>"/var/lib/puppet/reports",
:mode=>"750", :owner=>"puppet", :group=>"puppet", :ensure=>:directory,
:loglevel=>:debug, :links=>:follow, :backup=>false}'
Debug: Using settings: adding file resource 'railslog':
'File[/var/log/puppet/rails.log]{:path=>"/var/log/puppet/rails.log",
:mode=>"600", :owner=>"puppet", :group=>"puppet", :ensure=>:file,
:loglevel=>:debug, :links=>:follow, :backup=>false}'
Debug: /File[/var/lib/puppet/state]: Autorequiring File[/var/lib/puppet]
Debug: /File[/var/lib/puppet/lib]: Autorequiring File[/var/lib/puppet]
Debug: /File[/var/lib/puppet/ssl/certs]: Autorequiring
File[/var/lib/puppet/ssl]
Debug: /File[/var/lib/puppet/ssl]: Autorequiring File[/var/lib/puppet]
Debug: /File[/var/lib/puppet/ssl/public_keys]: Autorequiring
File[/var/lib/puppet/ssl]
Debug: /File[/var/lib/puppet/ssl/certificate_requests]: Autorequiring
File[/var/lib/puppet/ssl]
Debug: /File[/var/lib/puppet/ssl/private_keys]: Autorequiring
File[/var/lib/puppet/ssl]
Debug: /File[/var/lib/puppet/ssl/private]: Autorequiring
File[/var/lib/puppet/ssl]
Debug: /File[/etc/puppet/manifests]: Autorequiring File[/etc/puppet]
Debug: /File[/var/log/puppet/masterhttp.log]: Autorequiring
File[/var/log/puppet]
Debug: /File[/var/lib/puppet/bucket]: Autorequiring File[/var/lib/puppet]
Debug: /File[/etc/puppet/auth.conf]: Autorequiring File[/etc/puppet]
Debug: /File[/var/lib/puppet/yaml]: Autorequiring File[/var/lib/puppet]
Debug: /File[/var/lib/puppet/server_data]: Autorequiring
File[/var/lib/puppet]
Debug: /File[/var/lib/puppet/reports]: Autorequiring File[/var/lib/puppet]
Debug: /File[/var/log/puppet/rails.log]: Autorequiring File[/var/log/puppet]
Debug: Finishing transaction 34502460
Info: Connecting to mysql2 database: puppet
Killed

real 66m35.810s
user 9m42.143s
sys 6m31.294s


My /etc/puppet/puppet.conf file:

[main]
     # The Puppet log directory.
     # The default value is '$vardir/log'.
     logdir = /var/log/puppet

     # Where Puppet PID files are kept.
     # The default value is '$vardir/run'.
     rundir = /var/run/puppet

     # Where SSL certificates are kept.
     # The default value is '$confdir/ssl'.
     ssldir = $vardir/ssl

     facts_terminus = inventory_active_record

     storeconfigs = true
     dbadapter = mysql2
     dbserver = roger-test.ofi.softonic.lan
     dbname = puppet
     dbuser = puppet
     dbpassword =
     dbconnections = 20

[agent]
     # The file in which puppetd stores a list of the classes
     # associated with the retrieved configuratiion. Can be loaded in
     # the separate ``puppet`` executable using the ``--loadclasses``
     # option.
     # The default value is '$confdir/classes.txt'.
     classfile = $vardir/classes.txt

     # Where puppetd caches the local configuration. An
     # extension indicating the cache format is added automatically.
     # The default value is '$confdir/localconfig'.
     localconfig = $vardir/localconfig

Packages:

[root@ST-L226-ROGTORG ~]# rpm -qa | egrep -i "(puppet|ruby)" | sort
libselinux-ruby-2.1.13-15.fc19.x86_64
puppet-3.3.1-1.fc19.noarch
puppetdb-terminus-1.5.0-1.fc19.noarch
puppetlabs-release-19-2.noarch
ruby-2.0.0.247-15.fc19.x86_64
ruby-augeas-0.5.0-1.fc19.x86_64
ruby-devel-2.0.0.247-15.fc19.x86_64
ruby-irb-2.0.0.247-15.fc19.noarch
ruby-libs-2.0.0.247-15.fc19.x86_64
ruby-rgen-0.6.5-1.fc19.noarch
ruby-shadow-1.4.1-19.fc19.x86_64
rubygem-activemodel-3.2.13-1.fc19.noarch
rubygem-activerecord-3.2.13-1.fc19.noarch
rubygem-activesupport-3.2.13-1.fc19.noarch
rubygem-arel-3.0.2-3.fc19.noarch
rubygem-bigdecimal-1.2.0-15.fc19.x86_64
rubygem-builder-3.1.4-3.fc19.noarch
rubygem-i18n-0.6.1-3.fc19.noarch
rubygem-io-console-0.4.2-15.fc19.x86_64
rubygem-json-1.7.7-100.fc19.x86_64
rubygem-multi_json-1.7.1-1.fc19.noarch
rubygem-mysql2-0.3.13-1.fc19.x86_64
rubygem-psych-2.0.0-15.fc19.x86_64
rubygem-rdoc-4.0.1-1.fc19.noarch
rubygems-2.0.10-106.fc19.noarch
rubygem-tzinfo-0.3.35-1.fc19.noarch
rubypick-1.1.0-1.fc19.noarch


I'm positive the MySQL server receives the query and returns the data,
according to MySQL's slow-queries.log:

# Time: 131008 17:06:14
# User@Host: puppet[puppet] @ [172.20.59.127]
# Thread_id: 169 Schema: puppet Last_errno: 0 Killed: 0
# Query_time: 189.509563 Lock_time: 0.000136 Rows_sent: 6168111
Rows_examined: 12712154 Rows_affected: 0 Rows_read: 18446744073709551613
# Bytes_sent: 2426331478 Tmp_tables: 0 Tmp_disk_tables: 0
Tmp_table_sizes: 0
# InnoDB_trx_id: A93
SET timestamp=1381244774;
SELECT `hosts`.`id` AS t0_r0, `hosts`.`name` AS t0_r1, `hosts`.`ip` AS
t0_r2, `hosts`.`environment` AS t0_r3, `hosts`.`last_compile` AS t0_r4,
`hosts`.`last_freshcheck` AS t0_r5, `
hosts`.`last_report` AS t0_r6, `hosts`.`updated_at` AS t0_r7,
`hosts`.`source_file_id` AS t0_r8, `hosts`.`created_at` AS t0_r9,
`resources`.`id` AS t1_r0, `resources`.`title` AS t1
_r1, `resources`.`restype` AS t1_r2, `resources`.`host_id` AS t1_r3,
`resources`.`source_file_id` AS t1_r4, `resources`.`exported` AS t1_r5,
`resources`.`line` AS t1_r6, `resources
`.`updated_at` AS t1_r7, `resources`.`created_at` AS t1_r8,
`param_values`.`id` AS t2_r0, `param_values`.`value` AS t2_r1,
`param_values`.`param_name_id` AS t2_r2, `param_values`.`
line` AS t2_r3, `param_values`.`resource_id` AS t2_r4,
`param_values`.`updated_at` AS t2_r5, `param_values`.`created_at` AS t2_r6,
`puppet_tags`.`id` AS t3_r0, `puppet_tags`.`name`
  AS t3_r1, `puppet_tags`.`updated_at` AS t3_r2, `puppet_tags`.`created_at`
AS t3_r3 FROM `hosts` LEFT OUTER JOIN `resources` ON `resources`.`host_id`
= `hosts`.`id` LEFT OUTER JOIN
  `param_values` ON `param_values`.`resource_id` = `resources`.`id` LEFT
OUTER JOIN `resource_tags` ON `resource_tags`.`resource_id` =
`resources`.`id` LEFT OUTER JOIN `puppet_tags`
  ON `puppet_tags`.`id` = `resource_tags`.`puppet_tag_id` WHERE
`resources`.`exported` = 1;


When the process gets stuck, an strace shows a constant and very quick flow
of:

30666 stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=2593, ...}) = 0

The final moments of the strace showed:

30667 restart_syscall(<... resuming interrupted call ...> <unfinished ...>
30666 mremap(0x7f7aa4d0a000, 2732032, 2752512, MREMAP_MAYMOVE) =
0x7f7aa4d0a000
30666 brk(0) = 0x23aaea000
30666 brk(0x23ab0b000) = 0x23ab0b000
30666 brk(0) = 0x23ab0b000
30666 brk(0x23ab2c000) = 0x23ab2c000
30666 brk(0) = 0x23ab2c000
30666 brk(0x23ab4d000) = 0x23ab4d000
... then lots of...
30666 stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=2593, ...}) = 0
30666 stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=2593, ...}) = 0
30666 stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=2593, ...}) = 0
... and finally ...
30666 stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=2593, ...}) = 0
30666 stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=2593, ...}) = 0
30666 stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=2593, ...}) = 0
30667 +++ killed by SIGKILL +++
30666 +++ killed by SIGKILL +++

Any idea of what is happening? Am I doing something wrong?

Thanks!

Roger Torrentsgenerós

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users.
For more options, visit https://groups.google.com/groups/opt_out.

Search Discussions

  • Roger Torrentsgenerós at Oct 11, 2013 at 12:21 pm
    Just an update. This is what strace shows at the point where the process
    finishes fetching data from the MySQL database:

    17169 read(4, "38:31\373\373\0232013-07-15 17:38:32\373\023201"..., 16384)
    = 384
    17169 read(4, "3:00:07\0074113192\27xavier.ofi.softo"..., 16384) = 16000
    17169 read(4, "305566\17some.user\003194\00222\00741"..., 16384) = 384
    17169 read(4, "192\0232013-04-02 16:11:44\0232013-04-"..., 16384) = 8534
    17169 gettimeofday({1381493430, 91178}, NULL) = 0
    17169 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
    17169 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
    17169 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
    17169 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
    17169 mmap(NULL, 49348608, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS,
    -1, 0) = 0x7f41c7266000
    17169 stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=2593, ...}) = 0
    17169 stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=2593, ...}) = 0
    17169 stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=2593, ...}) = 0

    Beyond this point, tons of exactly the same line, CPU at 100% and memory
    usage increasing until it fills RAM and swap space, and then the SIGKILL
    shown above.

    Roger
    On Tuesday, October 8, 2013 7:22:20 PM UTC+2, Roger Torrentsgenerós wrote:


    Hi,

    I currently have a MySQL database containing all Puppet storeconfigs. My
    intention is to migrate to PuppetDB on a PostgreSQL server, so the first
    step is to use the 'storeconfigs' face to export all the DB to a file
    PuppetDB can later consume. But the 'puppet storeconfigs export' command
    always ends up being killed, I suspect due to some sort of OOM issue. I
    have tried different environments, different Puppet / Ruby / whatever
    versions, different hardware... No matter what, it always ends up the same.

    The process starts well, connects to the database and fetches all the
    data, but then it seems to get stuck after this output line:
    Info: Connecting to mysql2 database: puppet

    Meanwhile, a CPU is working and the amount of memory used by this process
    increases constantly. I have tried on several servers, one with up to 28GB
    of RAM, and it died the same way.

    Here are the facts of my last attempt:

    The execution output:

    [root@ST-L226-ROGTORG ~]# time puppet storeconfigs --verbose --debug export
    Debug: Using settings: adding file resource 'confdir':
    'File[/etc/puppet]{:path=>"/etc/puppet", :ensure=>:directory,
    :loglevel=>:debug, :links=>:follow, :backup=>false}'
    Debug: Using settings: adding file resource 'vardir':
    'File[/var/lib/puppet]{:path=>"/var/lib/puppet", :ensure=>:directory,
    :loglevel=>:debug, :links=>:follow, :backup=>false}'
    Debug: Failed to load library 'ldap' for feature 'ldap'
    Debug: Puppet::Type::User::ProviderLdap: feature ldap is missing
    Debug: Puppet::Type::User::ProviderDirectoryservice: file
    /usr/bin/dsimport does not exist
    Debug: Puppet::Type::User::ProviderPw: file pw does not exist
    Debug: Puppet::Type::User::ProviderUser_role_add: file roleadd does not
    exist
    Debug: Failed to load library 'ldap' for feature 'ldap'
    Debug: Puppet::Type::Group::ProviderLdap: feature ldap is missing
    Debug: Puppet::Type::Group::ProviderDirectoryservice: file /usr/bin/dscl
    does not exist
    Debug: Puppet::Type::Group::ProviderPw: file pw does not exist
    Debug: Using settings: adding file resource 'logdir':
    'File[/var/log/puppet]{:path=>"/var/log/puppet", :mode=>"750",
    :owner=>"puppet", :group=>"puppet", :ensure=>:directory, :loglevel=>:debug,
    :links=>:follow, :backup=>false}'
    Debug: Using settings: adding file resource 'statedir':
    'File[/var/lib/puppet/state]{:path=>"/var/lib/puppet/state", :mode=>"1755",
    :ensure=>:directory, :loglevel=>:debug, :links=>:follow, :backup=>false}'
    Debug: Using settings: adding file resource 'rundir':
    'File[/var/run/puppet]{:path=>"/var/run/puppet", :mode=>"755",
    :owner=>"puppet", :group=>"puppet", :ensure=>:directory, :loglevel=>:debug,
    :links=>:follow, :backup=>false}'
    Debug: Using settings: adding file resource 'libdir':
    'File[/var/lib/puppet/lib]{:path=>"/var/lib/puppet/lib",
    :ensure=>:directory, :loglevel=>:debug, :links=>:follow, :backup=>false}'
    Debug: Using settings: adding file resource 'certdir':
    'File[/var/lib/puppet/ssl/certs]{:path=>"/var/lib/puppet/ssl/certs",
    :owner=>"puppet", :ensure=>:directory, :loglevel=>:debug, :links=>:follow,
    :backup=>false}'
    Debug: Using settings: adding file resource 'ssldir':
    'File[/var/lib/puppet/ssl]{:path=>"/var/lib/puppet/ssl", :mode=>"771",
    :owner=>"puppet", :ensure=>:directory, :loglevel=>:debug, :links=>:follow,
    :backup=>false}'
    Debug: Using settings: adding file resource 'publickeydir':
    'File[/var/lib/puppet/ssl/public_keys]{:path=>"/var/lib/puppet/ssl/public_keys",
    :owner=>"puppet", :ensure=>:directory, :loglevel=>:debug, :links=>:follow,
    :backup=>false}'
    Debug: Using settings: adding file resource 'requestdir':
    'File[/var/lib/puppet/ssl/certificate_requests]{:path=>"/var/lib/puppet/ssl/certificate_requests",
    :owner=>"puppet", :ensure=>:directory, :loglevel=>:debug, :links=>:follow,
    :backup=>false}'
    Debug: Using settings: adding file resource 'privatekeydir':
    'File[/var/lib/puppet/ssl/private_keys]{:path=>"/var/lib/puppet/ssl/private_keys",
    :mode=>"750", :owner=>"puppet", :ensure=>:directory, :loglevel=>:debug,
    :links=>:follow, :backup=>false}'
    Debug: Using settings: adding file resource 'privatedir':
    'File[/var/lib/puppet/ssl/private]{:path=>"/var/lib/puppet/ssl/private",
    :mode=>"750", :owner=>"puppet", :ensure=>:directory, :loglevel=>:debug,
    :links=>:follow, :backup=>false}'
    Debug: Using settings: adding file resource 'manifestdir':
    'File[/etc/puppet/manifests]{:path=>"/etc/puppet/manifests",
    :ensure=>:directory, :loglevel=>:debug, :links=>:follow, :backup=>false}'
    Debug: Using settings: adding file resource 'masterhttplog':
    'File[/var/log/puppet/masterhttp.log]{:path=>"/var/log/puppet/masterhttp.log",
    :mode=>"660", :owner=>"puppet", :group=>"puppet", :ensure=>:file,
    :loglevel=>:debug, :links=>:follow, :backup=>false}'
    Debug: Using settings: adding file resource 'bucketdir':
    'File[/var/lib/puppet/bucket]{:path=>"/var/lib/puppet/bucket",
    :mode=>"750", :owner=>"puppet", :group=>"puppet", :ensure=>:directory,
    :loglevel=>:debug, :links=>:follow, :backup=>false}'
    Debug: Using settings: adding file resource 'rest_authconfig':
    'File[/etc/puppet/auth.conf]{:path=>"/etc/puppet/auth.conf",
    :ensure=>:file, :loglevel=>:debug, :links=>:follow, :backup=>false}'
    Debug: Using settings: adding file resource 'yamldir':
    'File[/var/lib/puppet/yaml]{:path=>"/var/lib/puppet/yaml", :mode=>"750",
    :owner=>"puppet", :group=>"puppet", :ensure=>:directory, :loglevel=>:debug,
    :links=>:follow, :backup=>false}'
    Debug: Using settings: adding file resource 'server_datadir':
    'File[/var/lib/puppet/server_data]{:path=>"/var/lib/puppet/server_data",
    :mode=>"750", :owner=>"puppet", :group=>"puppet", :ensure=>:directory,
    :loglevel=>:debug, :links=>:follow, :backup=>false}'
    Debug: Using settings: adding file resource 'reportdir':
    'File[/var/lib/puppet/reports]{:path=>"/var/lib/puppet/reports",
    :mode=>"750", :owner=>"puppet", :group=>"puppet", :ensure=>:directory,
    :loglevel=>:debug, :links=>:follow, :backup=>false}'
    Debug: Using settings: adding file resource 'railslog':
    'File[/var/log/puppet/rails.log]{:path=>"/var/log/puppet/rails.log",
    :mode=>"600", :owner=>"puppet", :group=>"puppet", :ensure=>:file,
    :loglevel=>:debug, :links=>:follow, :backup=>false}'
    Debug: /File[/var/lib/puppet/state]: Autorequiring File[/var/lib/puppet]
    Debug: /File[/var/lib/puppet/lib]: Autorequiring File[/var/lib/puppet]
    Debug: /File[/var/lib/puppet/ssl/certs]: Autorequiring
    File[/var/lib/puppet/ssl]
    Debug: /File[/var/lib/puppet/ssl]: Autorequiring File[/var/lib/puppet]
    Debug: /File[/var/lib/puppet/ssl/public_keys]: Autorequiring
    File[/var/lib/puppet/ssl]
    Debug: /File[/var/lib/puppet/ssl/certificate_requests]: Autorequiring
    File[/var/lib/puppet/ssl]
    Debug: /File[/var/lib/puppet/ssl/private_keys]: Autorequiring
    File[/var/lib/puppet/ssl]
    Debug: /File[/var/lib/puppet/ssl/private]: Autorequiring
    File[/var/lib/puppet/ssl]
    Debug: /File[/etc/puppet/manifests]: Autorequiring File[/etc/puppet]
    Debug: /File[/var/log/puppet/masterhttp.log]: Autorequiring
    File[/var/log/puppet]
    Debug: /File[/var/lib/puppet/bucket]: Autorequiring File[/var/lib/puppet]
    Debug: /File[/etc/puppet/auth.conf]: Autorequiring File[/etc/puppet]
    Debug: /File[/var/lib/puppet/yaml]: Autorequiring File[/var/lib/puppet]
    Debug: /File[/var/lib/puppet/server_data]: Autorequiring
    File[/var/lib/puppet]
    Debug: /File[/var/lib/puppet/reports]: Autorequiring File[/var/lib/puppet]
    Debug: /File[/var/log/puppet/rails.log]: Autorequiring
    File[/var/log/puppet]
    Debug: Finishing transaction 34502460
    Info: Connecting to mysql2 database: puppet
    Killed

    real 66m35.810s
    user 9m42.143s
    sys 6m31.294s


    My /etc/puppet/puppet.conf file:

    [main]
    # The Puppet log directory.
    # The default value is '$vardir/log'.
    logdir = /var/log/puppet

    # Where Puppet PID files are kept.
    # The default value is '$vardir/run'.
    rundir = /var/run/puppet

    # Where SSL certificates are kept.
    # The default value is '$confdir/ssl'.
    ssldir = $vardir/ssl

    facts_terminus = inventory_active_record

    storeconfigs = true
    dbadapter = mysql2
    dbserver = roger-test.ofi.softonic.lan
    dbname = puppet
    dbuser = puppet
    dbpassword =
    dbconnections = 20

    [agent]
    # The file in which puppetd stores a list of the classes
    # associated with the retrieved configuratiion. Can be loaded in
    # the separate ``puppet`` executable using the ``--loadclasses``
    # option.
    # The default value is '$confdir/classes.txt'.
    classfile = $vardir/classes.txt

    # Where puppetd caches the local configuration. An
    # extension indicating the cache format is added automatically.
    # The default value is '$confdir/localconfig'.
    localconfig = $vardir/localconfig

    Packages:

    [root@ST-L226-ROGTORG ~]# rpm -qa | egrep -i "(puppet|ruby)" | sort
    libselinux-ruby-2.1.13-15.fc19.x86_64
    puppet-3.3.1-1.fc19.noarch
    puppetdb-terminus-1.5.0-1.fc19.noarch
    puppetlabs-release-19-2.noarch
    ruby-2.0.0.247-15.fc19.x86_64
    ruby-augeas-0.5.0-1.fc19.x86_64
    ruby-devel-2.0.0.247-15.fc19.x86_64
    ruby-irb-2.0.0.247-15.fc19.noarch
    ruby-libs-2.0.0.247-15.fc19.x86_64
    ruby-rgen-0.6.5-1.fc19.noarch
    ruby-shadow-1.4.1-19.fc19.x86_64
    rubygem-activemodel-3.2.13-1.fc19.noarch
    rubygem-activerecord-3.2.13-1.fc19.noarch
    rubygem-activesupport-3.2.13-1.fc19.noarch
    rubygem-arel-3.0.2-3.fc19.noarch
    rubygem-bigdecimal-1.2.0-15.fc19.x86_64
    rubygem-builder-3.1.4-3.fc19.noarch
    rubygem-i18n-0.6.1-3.fc19.noarch
    rubygem-io-console-0.4.2-15.fc19.x86_64
    rubygem-json-1.7.7-100.fc19.x86_64
    rubygem-multi_json-1.7.1-1.fc19.noarch
    rubygem-mysql2-0.3.13-1.fc19.x86_64
    rubygem-psych-2.0.0-15.fc19.x86_64
    rubygem-rdoc-4.0.1-1.fc19.noarch
    rubygems-2.0.10-106.fc19.noarch
    rubygem-tzinfo-0.3.35-1.fc19.noarch
    rubypick-1.1.0-1.fc19.noarch


    I'm positive the MySQL server receives the query and returns the data,
    according to MySQL's slow-queries.log:

    # Time: 131008 17:06:14
    # User@Host: puppet[puppet] @ [172.20.59.127]
    # Thread_id: 169 Schema: puppet Last_errno: 0 Killed: 0
    # Query_time: 189.509563 Lock_time: 0.000136 Rows_sent: 6168111
    Rows_examined: 12712154 Rows_affected: 0 Rows_read: 18446744073709551613
    # Bytes_sent: 2426331478 Tmp_tables: 0 Tmp_disk_tables: 0
    Tmp_table_sizes: 0
    # InnoDB_trx_id: A93
    SET timestamp=1381244774;
    SELECT `hosts`.`id` AS t0_r0, `hosts`.`name` AS t0_r1, `hosts`.`ip` AS
    t0_r2, `hosts`.`environment` AS t0_r3, `hosts`.`last_compile` AS t0_r4,
    `hosts`.`last_freshcheck` AS t0_r5, `
    hosts`.`last_report` AS t0_r6, `hosts`.`updated_at` AS t0_r7,
    `hosts`.`source_file_id` AS t0_r8, `hosts`.`created_at` AS t0_r9,
    `resources`.`id` AS t1_r0, `resources`.`title` AS t1
    _r1, `resources`.`restype` AS t1_r2, `resources`.`host_id` AS t1_r3,
    `resources`.`source_file_id` AS t1_r4, `resources`.`exported` AS t1_r5,
    `resources`.`line` AS t1_r6, `resources
    `.`updated_at` AS t1_r7, `resources`.`created_at` AS t1_r8,
    `param_values`.`id` AS t2_r0, `param_values`.`value` AS t2_r1,
    `param_values`.`param_name_id` AS t2_r2, `param_values`.`
    line` AS t2_r3, `param_values`.`resource_id` AS t2_r4,
    `param_values`.`updated_at` AS t2_r5, `param_values`.`created_at` AS t2_r6,
    `puppet_tags`.`id` AS t3_r0, `puppet_tags`.`name`
    AS t3_r1, `puppet_tags`.`updated_at` AS t3_r2, `puppet_tags`.`created_at`
    AS t3_r3 FROM `hosts` LEFT OUTER JOIN `resources` ON `resources`.`host_id`
    = `hosts`.`id` LEFT OUTER JOIN
    `param_values` ON `param_values`.`resource_id` = `resources`.`id` LEFT
    OUTER JOIN `resource_tags` ON `resource_tags`.`resource_id` =
    `resources`.`id` LEFT OUTER JOIN `puppet_tags`
    ON `puppet_tags`.`id` = `resource_tags`.`puppet_tag_id` WHERE
    `resources`.`exported` = 1;


    When the process gets stuck, an strace shows a constant and very quick
    flow of:

    30666 stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=2593, ...}) = 0

    The final moments of the strace showed:

    30667 restart_syscall(<... resuming interrupted call ...> <unfinished ...>
    30666 mremap(0x7f7aa4d0a000, 2732032, 2752512, MREMAP_MAYMOVE) =
    0x7f7aa4d0a000
    30666 brk(0) = 0x23aaea000
    30666 brk(0x23ab0b000) = 0x23ab0b000
    30666 brk(0) = 0x23ab0b000
    30666 brk(0x23ab2c000) = 0x23ab2c000
    30666 brk(0) = 0x23ab2c000
    30666 brk(0x23ab4d000) = 0x23ab4d000
    ... then lots of...
    30666 stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=2593, ...}) = 0
    30666 stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=2593, ...}) = 0
    30666 stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=2593, ...}) = 0
    ... and finally ...
    30666 stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=2593, ...}) = 0
    30666 stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=2593, ...}) = 0
    30666 stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=2593, ...}) = 0
    30667 +++ killed by SIGKILL +++
    30666 +++ killed by SIGKILL +++

    Any idea of what is happening? Am I doing something wrong?

    Thanks!

    Roger Torrentsgenerós
    --
    You received this message because you are subscribed to the Google Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to puppet-users@googlegroups.com.
    Visit this group at http://groups.google.com/group/puppet-users.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Roger Torrentsgenerós at Oct 16, 2013 at 8:23 am
    Well I finally did it.

    First thing to do is to specify a timezone to avoid the massive amount of:

    17169 stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=2593, ...}) = 0

    Issuing '# export TZ=:/etc/localtime' prior to launching the Puppet face is
    enough.

    Then, in my particular case I had to use a VM with 42GB of RAM to deal with
    a 1.4GB MySQL database. After more than 2 hours of processing, a .tgz file
    with the ridiculous size of 1.7MB was created.

    So if your export process gets killed and you see OOM messages in your
    dmesg, you'll need more RAM in your server.

    Roger
    On Friday, October 11, 2013 2:21:12 PM UTC+2, Roger Torrentsgenerós wrote:


    Just an update. This is what strace shows at the point where the process
    finishes fetching data from the MySQL database:

    17169 read(4, "38:31\373\373\0232013-07-15 17:38:32\373\023201"..., 16384)
    = 384
    17169 read(4, "3:00:07\0074113192\27xavier.ofi.softo"..., 16384) = 16000
    17169 read(4, "305566\17some.user\003194\00222\00741"..., 16384) = 384
    17169 read(4, "192\0232013-04-02 16:11:44\0232013-04-"..., 16384) = 8534
    17169 gettimeofday({1381493430, 91178}, NULL) = 0
    17169 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
    17169 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
    17169 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
    17169 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
    17169 mmap(NULL, 49348608, PROT_READ|PROT_WRITE,
    MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f41c7266000
    17169 stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=2593, ...}) = 0
    17169 stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=2593, ...}) = 0
    17169 stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=2593, ...}) = 0

    Beyond this point, tons of exactly the same line, CPU at 100% and memory
    usage increasing until it fills RAM and swap space, and then the SIGKILL
    shown above.

    Roger
    On Tuesday, October 8, 2013 7:22:20 PM UTC+2, Roger Torrentsgenerós wrote:


    Hi,

    I currently have a MySQL database containing all Puppet storeconfigs. My
    intention is to migrate to PuppetDB on a PostgreSQL server, so the first
    step is to use the 'storeconfigs' face to export all the DB to a file
    PuppetDB can later consume. But the 'puppet storeconfigs export' command
    always ends up being killed, I suspect due to some sort of OOM issue. I
    have tried different environments, different Puppet / Ruby / whatever
    versions, different hardware... No matter what, it always ends up the same.

    The process starts well, connects to the database and fetches all the
    data, but then it seems to get stuck after this output line:
    Info: Connecting to mysql2 database: puppet

    Meanwhile, a CPU is working and the amount of memory used by this process
    increases constantly. I have tried on several servers, one with up to 28GB
    of RAM, and it died the same way.

    Here are the facts of my last attempt:

    The execution output:

    [root@ST-L226-ROGTORG ~]# time puppet storeconfigs --verbose --debug
    export
    Debug: Using settings: adding file resource 'confdir':
    'File[/etc/puppet]{:path=>"/etc/puppet", :ensure=>:directory,
    :loglevel=>:debug, :links=>:follow, :backup=>false}'
    Debug: Using settings: adding file resource 'vardir':
    'File[/var/lib/puppet]{:path=>"/var/lib/puppet", :ensure=>:directory,
    :loglevel=>:debug, :links=>:follow, :backup=>false}'
    Debug: Failed to load library 'ldap' for feature 'ldap'
    Debug: Puppet::Type::User::ProviderLdap: feature ldap is missing
    Debug: Puppet::Type::User::ProviderDirectoryservice: file
    /usr/bin/dsimport does not exist
    Debug: Puppet::Type::User::ProviderPw: file pw does not exist
    Debug: Puppet::Type::User::ProviderUser_role_add: file roleadd does not
    exist
    Debug: Failed to load library 'ldap' for feature 'ldap'
    Debug: Puppet::Type::Group::ProviderLdap: feature ldap is missing
    Debug: Puppet::Type::Group::ProviderDirectoryservice: file /usr/bin/dscl
    does not exist
    Debug: Puppet::Type::Group::ProviderPw: file pw does not exist
    Debug: Using settings: adding file resource 'logdir':
    'File[/var/log/puppet]{:path=>"/var/log/puppet", :mode=>"750",
    :owner=>"puppet", :group=>"puppet", :ensure=>:directory, :loglevel=>:debug,
    :links=>:follow, :backup=>false}'
    Debug: Using settings: adding file resource 'statedir':
    'File[/var/lib/puppet/state]{:path=>"/var/lib/puppet/state", :mode=>"1755",
    :ensure=>:directory, :loglevel=>:debug, :links=>:follow, :backup=>false}'
    Debug: Using settings: adding file resource 'rundir':
    'File[/var/run/puppet]{:path=>"/var/run/puppet", :mode=>"755",
    :owner=>"puppet", :group=>"puppet", :ensure=>:directory, :loglevel=>:debug,
    :links=>:follow, :backup=>false}'
    Debug: Using settings: adding file resource 'libdir':
    'File[/var/lib/puppet/lib]{:path=>"/var/lib/puppet/lib",
    :ensure=>:directory, :loglevel=>:debug, :links=>:follow, :backup=>false}'
    Debug: Using settings: adding file resource 'certdir':
    'File[/var/lib/puppet/ssl/certs]{:path=>"/var/lib/puppet/ssl/certs",
    :owner=>"puppet", :ensure=>:directory, :loglevel=>:debug, :links=>:follow,
    :backup=>false}'
    Debug: Using settings: adding file resource 'ssldir':
    'File[/var/lib/puppet/ssl]{:path=>"/var/lib/puppet/ssl", :mode=>"771",
    :owner=>"puppet", :ensure=>:directory, :loglevel=>:debug, :links=>:follow,
    :backup=>false}'
    Debug: Using settings: adding file resource 'publickeydir':
    'File[/var/lib/puppet/ssl/public_keys]{:path=>"/var/lib/puppet/ssl/public_keys",
    :owner=>"puppet", :ensure=>:directory, :loglevel=>:debug, :links=>:follow,
    :backup=>false}'
    Debug: Using settings: adding file resource 'requestdir':
    'File[/var/lib/puppet/ssl/certificate_requests]{:path=>"/var/lib/puppet/ssl/certificate_requests",
    :owner=>"puppet", :ensure=>:directory, :loglevel=>:debug, :links=>:follow,
    :backup=>false}'
    Debug: Using settings: adding file resource 'privatekeydir':
    'File[/var/lib/puppet/ssl/private_keys]{:path=>"/var/lib/puppet/ssl/private_keys",
    :mode=>"750", :owner=>"puppet", :ensure=>:directory, :loglevel=>:debug,
    :links=>:follow, :backup=>false}'
    Debug: Using settings: adding file resource 'privatedir':
    'File[/var/lib/puppet/ssl/private]{:path=>"/var/lib/puppet/ssl/private",
    :mode=>"750", :owner=>"puppet", :ensure=>:directory, :loglevel=>:debug,
    :links=>:follow, :backup=>false}'
    Debug: Using settings: adding file resource 'manifestdir':
    'File[/etc/puppet/manifests]{:path=>"/etc/puppet/manifests",
    :ensure=>:directory, :loglevel=>:debug, :links=>:follow, :backup=>false}'
    Debug: Using settings: adding file resource 'masterhttplog':
    'File[/var/log/puppet/masterhttp.log]{:path=>"/var/log/puppet/masterhttp.log",
    :mode=>"660", :owner=>"puppet", :group=>"puppet", :ensure=>:file,
    :loglevel=>:debug, :links=>:follow, :backup=>false}'
    Debug: Using settings: adding file resource 'bucketdir':
    'File[/var/lib/puppet/bucket]{:path=>"/var/lib/puppet/bucket",
    :mode=>"750", :owner=>"puppet", :group=>"puppet", :ensure=>:directory,
    :loglevel=>:debug, :links=>:follow, :backup=>false}'
    Debug: Using settings: adding file resource 'rest_authconfig':
    'File[/etc/puppet/auth.conf]{:path=>"/etc/puppet/auth.conf",
    :ensure=>:file, :loglevel=>:debug, :links=>:follow, :backup=>false}'
    Debug: Using settings: adding file resource 'yamldir':
    'File[/var/lib/puppet/yaml]{:path=>"/var/lib/puppet/yaml", :mode=>"750",
    :owner=>"puppet", :group=>"puppet", :ensure=>:directory, :loglevel=>:debug,
    :links=>:follow, :backup=>false}'
    Debug: Using settings: adding file resource 'server_datadir':
    'File[/var/lib/puppet/server_data]{:path=>"/var/lib/puppet/server_data",
    :mode=>"750", :owner=>"puppet", :group=>"puppet", :ensure=>:directory,
    :loglevel=>:debug, :links=>:follow, :backup=>false}'
    Debug: Using settings: adding file resource 'reportdir':
    'File[/var/lib/puppet/reports]{:path=>"/var/lib/puppet/reports",
    :mode=>"750", :owner=>"puppet", :group=>"puppet", :ensure=>:directory,
    :loglevel=>:debug, :links=>:follow, :backup=>false}'
    Debug: Using settings: adding file resource 'railslog':
    'File[/var/log/puppet/rails.log]{:path=>"/var/log/puppet/rails.log",
    :mode=>"600", :owner=>"puppet", :group=>"puppet", :ensure=>:file,
    :loglevel=>:debug, :links=>:follow, :backup=>false}'
    Debug: /File[/var/lib/puppet/state]: Autorequiring File[/var/lib/puppet]
    Debug: /File[/var/lib/puppet/lib]: Autorequiring File[/var/lib/puppet]
    Debug: /File[/var/lib/puppet/ssl/certs]: Autorequiring
    File[/var/lib/puppet/ssl]
    Debug: /File[/var/lib/puppet/ssl]: Autorequiring File[/var/lib/puppet]
    Debug: /File[/var/lib/puppet/ssl/public_keys]: Autorequiring
    File[/var/lib/puppet/ssl]
    Debug: /File[/var/lib/puppet/ssl/certificate_requests]: Autorequiring
    File[/var/lib/puppet/ssl]
    Debug: /File[/var/lib/puppet/ssl/private_keys]: Autorequiring
    File[/var/lib/puppet/ssl]
    Debug: /File[/var/lib/puppet/ssl/private]: Autorequiring
    File[/var/lib/puppet/ssl]
    Debug: /File[/etc/puppet/manifests]: Autorequiring File[/etc/puppet]
    Debug: /File[/var/log/puppet/masterhttp.log]: Autorequiring
    File[/var/log/puppet]
    Debug: /File[/var/lib/puppet/bucket]: Autorequiring File[/var/lib/puppet]
    Debug: /File[/etc/puppet/auth.conf]: Autorequiring File[/etc/puppet]
    Debug: /File[/var/lib/puppet/yaml]: Autorequiring File[/var/lib/puppet]
    Debug: /File[/var/lib/puppet/server_data]: Autorequiring
    File[/var/lib/puppet]
    Debug: /File[/var/lib/puppet/reports]: Autorequiring File[/var/lib/puppet]
    Debug: /File[/var/log/puppet/rails.log]: Autorequiring
    File[/var/log/puppet]
    Debug: Finishing transaction 34502460
    Info: Connecting to mysql2 database: puppet
    Killed

    real 66m35.810s
    user 9m42.143s
    sys 6m31.294s


    My /etc/puppet/puppet.conf file:

    [main]
    # The Puppet log directory.
    # The default value is '$vardir/log'.
    logdir = /var/log/puppet

    # Where Puppet PID files are kept.
    # The default value is '$vardir/run'.
    rundir = /var/run/puppet

    # Where SSL certificates are kept.
    # The default value is '$confdir/ssl'.
    ssldir = $vardir/ssl

    facts_terminus = inventory_active_record

    storeconfigs = true
    dbadapter = mysql2
    dbserver = roger-test.ofi.softonic.lan
    dbname = puppet
    dbuser = puppet
    dbpassword =
    dbconnections = 20

    [agent]
    # The file in which puppetd stores a list of the classes
    # associated with the retrieved configuratiion. Can be loaded in
    # the separate ``puppet`` executable using the ``--loadclasses``
    # option.
    # The default value is '$confdir/classes.txt'.
    classfile = $vardir/classes.txt

    # Where puppetd caches the local configuration. An
    # extension indicating the cache format is added automatically.
    # The default value is '$confdir/localconfig'.
    localconfig = $vardir/localconfig

    Packages:

    [root@ST-L226-ROGTORG ~]# rpm -qa | egrep -i "(puppet|ruby)" | sort
    libselinux-ruby-2.1.13-15.fc19.x86_64
    puppet-3.3.1-1.fc19.noarch
    puppetdb-terminus-1.5.0-1.fc19.noarch
    puppetlabs-release-19-2.noarch
    ruby-2.0.0.247-15.fc19.x86_64
    ruby-augeas-0.5.0-1.fc19.x86_64
    ruby-devel-2.0.0.247-15.fc19.x86_64
    ruby-irb-2.0.0.247-15.fc19.noarch
    ruby-libs-2.0.0.247-15.fc19.x86_64
    ruby-rgen-0.6.5-1.fc19.noarch
    ruby-shadow-1.4.1-19.fc19.x86_64
    rubygem-activemodel-3.2.13-1.fc19.noarch
    rubygem-activerecord-3.2.13-1.fc19.noarch
    rubygem-activesupport-3.2.13-1.fc19.noarch
    rubygem-arel-3.0.2-3.fc19.noarch
    rubygem-bigdecimal-1.2.0-15.fc19.x86_64
    rubygem-builder-3.1.4-3.fc19.noarch
    rubygem-i18n-0.6.1-3.fc19.noarch
    rubygem-io-console-0.4.2-15.fc19.x86_64
    rubygem-json-1.7.7-100.fc19.x86_64
    rubygem-multi_json-1.7.1-1.fc19.noarch
    rubygem-mysql2-0.3.13-1.fc19.x86_64
    rubygem-psych-2.0.0-15.fc19.x86_64
    rubygem-rdoc-4.0.1-1.fc19.noarch
    rubygems-2.0.10-106.fc19.noarch
    rubygem-tzinfo-0.3.35-1.fc19.noarch
    rubypick-1.1.0-1.fc19.noarch


    I'm positive the MySQL server receives the query and returns the data,
    according to MySQL's slow-queries.log:

    # Time: 131008 17:06:14
    # User@Host: puppet[puppet] @ [172.20.59.127]
    # Thread_id: 169 Schema: puppet Last_errno: 0 Killed: 0
    # Query_time: 189.509563 Lock_time: 0.000136 Rows_sent: 6168111
    Rows_examined: 12712154 Rows_affected: 0 Rows_read: 18446744073709551613
    # Bytes_sent: 2426331478 Tmp_tables: 0 Tmp_disk_tables: 0
    Tmp_table_sizes: 0
    # InnoDB_trx_id: A93
    SET timestamp=1381244774;
    SELECT `hosts`.`id` AS t0_r0, `hosts`.`name` AS t0_r1, `hosts`.`ip` AS
    t0_r2, `hosts`.`environment` AS t0_r3, `hosts`.`last_compile` AS t0_r4,
    `hosts`.`last_freshcheck` AS t0_r5, `
    hosts`.`last_report` AS t0_r6, `hosts`.`updated_at` AS t0_r7,
    `hosts`.`source_file_id` AS t0_r8, `hosts`.`created_at` AS t0_r9,
    `resources`.`id` AS t1_r0, `resources`.`title` AS t1
    _r1, `resources`.`restype` AS t1_r2, `resources`.`host_id` AS t1_r3,
    `resources`.`source_file_id` AS t1_r4, `resources`.`exported` AS t1_r5,
    `resources`.`line` AS t1_r6, `resources
    `.`updated_at` AS t1_r7, `resources`.`created_at` AS t1_r8,
    `param_values`.`id` AS t2_r0, `param_values`.`value` AS t2_r1,
    `param_values`.`param_name_id` AS t2_r2, `param_values`.`
    line` AS t2_r3, `param_values`.`resource_id` AS t2_r4,
    `param_values`.`updated_at` AS t2_r5, `param_values`.`created_at` AS t2_r6,
    `puppet_tags`.`id` AS t3_r0, `puppet_tags`.`name`
    AS t3_r1, `puppet_tags`.`updated_at` AS t3_r2,
    `puppet_tags`.`created_at` AS t3_r3 FROM `hosts` LEFT OUTER JOIN
    `resources` ON `resources`.`host_id` = `hosts`.`id` LEFT OUTER JOIN
    `param_values` ON `param_values`.`resource_id` = `resources`.`id` LEFT
    OUTER JOIN `resource_tags` ON `resource_tags`.`resource_id` =
    `resources`.`id` LEFT OUTER JOIN `puppet_tags`
    ON `puppet_tags`.`id` = `resource_tags`.`puppet_tag_id` WHERE
    `resources`.`exported` = 1;


    When the process gets stuck, an strace shows a constant and very quick
    flow of:

    30666 stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=2593, ...}) =


    The final moments of the strace showed:

    30667 restart_syscall(<... resuming interrupted call ...> <unfinished ...>
    30666 mremap(0x7f7aa4d0a000, 2732032, 2752512, MREMAP_MAYMOVE) =
    0x7f7aa4d0a000
    30666 brk(0) = 0x23aaea000
    30666 brk(0x23ab0b000) = 0x23ab0b000
    30666 brk(0) = 0x23ab0b000
    30666 brk(0x23ab2c000) = 0x23ab2c000
    30666 brk(0) = 0x23ab2c000
    30666 brk(0x23ab4d000) = 0x23ab4d000
    ... then lots of...
    30666 stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=2593, ...}) =

    30666 stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=2593, ...}) =

    30666 stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=2593, ...}) =

    ... and finally ...
    30666 stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=2593, ...}) =

    30666 stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=2593, ...}) =

    30666 stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=2593, ...}) =

    30667 +++ killed by SIGKILL +++
    30666 +++ killed by SIGKILL +++

    Any idea of what is happening? Am I doing something wrong?

    Thanks!

    Roger Torrentsgenerós
    --
    You received this message because you are subscribed to the Google Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to puppet-users@googlegroups.com.
    Visit this group at http://groups.google.com/group/puppet-users.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Ken Barber at Oct 17, 2013 at 8:40 am
    Roger,

    Can you raise this as a bug in our bug tracker?

    http://projects.puppetlabs.com/projects/puppetdb

    ken.
    On Wed, Oct 16, 2013 at 9:23 AM, Roger Torrentsgenerós wrote:

    Well I finally did it.

    First thing to do is to specify a timezone to avoid the massive amount of:


    17169 stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=2593, ...}) = 0

    Issuing '# export TZ=:/etc/localtime' prior to launching the Puppet face is
    enough.

    Then, in my particular case I had to use a VM with 42GB of RAM to deal with
    a 1.4GB MySQL database. After more than 2 hours of processing, a .tgz file
    with the ridiculous size of 1.7MB was created.

    So if your export process gets killed and you see OOM messages in your
    dmesg, you'll need more RAM in your server.

    Roger

    On Friday, October 11, 2013 2:21:12 PM UTC+2, Roger Torrentsgenerós wrote:


    Just an update. This is what strace shows at the point where the process
    finishes fetching data from the MySQL database:

    17169 read(4, "38:31\373\373\0232013-07-15 17:38:32\373\023201"..., 16384)
    = 384
    17169 read(4, "3:00:07\0074113192\27xavier.ofi.softo"..., 16384) = 16000
    17169 read(4, "305566\17some.user\003194\00222\00741"..., 16384) = 384
    17169 read(4, "192\0232013-04-02 16:11:44\0232013-04-"..., 16384) = 8534
    17169 gettimeofday({1381493430, 91178}, NULL) = 0
    17169 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
    17169 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
    17169 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
    17169 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
    17169 mmap(NULL, 49348608, PROT_READ|PROT_WRITE,
    MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f41c7266000
    17169 stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=2593, ...}) =

    17169 stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=2593, ...}) =

    17169 stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=2593, ...}) =


    Beyond this point, tons of exactly the same line, CPU at 100% and memory
    usage increasing until it fills RAM and swap space, and then the SIGKILL
    shown above.

    Roger
    On Tuesday, October 8, 2013 7:22:20 PM UTC+2, Roger Torrentsgenerós wrote:


    Hi,

    I currently have a MySQL database containing all Puppet storeconfigs. My
    intention is to migrate to PuppetDB on a PostgreSQL server, so the first
    step is to use the 'storeconfigs' face to export all the DB to a file
    PuppetDB can later consume. But the 'puppet storeconfigs export' command
    always ends up being killed, I suspect due to some sort of OOM issue. I have
    tried different environments, different Puppet / Ruby / whatever versions,
    different hardware... No matter what, it always ends up the same.

    The process starts well, connects to the database and fetches all the
    data, but then it seems to get stuck after this output line:
    Info: Connecting to mysql2 database: puppet

    Meanwhile, a CPU is working and the amount of memory used by this process
    increases constantly. I have tried on several servers, one with up to 28GB
    of RAM, and it died the same way.

    Here are the facts of my last attempt:

    The execution output:

    [root@ST-L226-ROGTORG ~]# time puppet storeconfigs --verbose --debug
    export
    Debug: Using settings: adding file resource 'confdir':
    'File[/etc/puppet]{:path=>"/etc/puppet", :ensure=>:directory,
    :loglevel=>:debug, :links=>:follow, :backup=>false}'
    Debug: Using settings: adding file resource 'vardir':
    'File[/var/lib/puppet]{:path=>"/var/lib/puppet", :ensure=>:directory,
    :loglevel=>:debug, :links=>:follow, :backup=>false}'
    Debug: Failed to load library 'ldap' for feature 'ldap'
    Debug: Puppet::Type::User::ProviderLdap: feature ldap is missing
    Debug: Puppet::Type::User::ProviderDirectoryservice: file
    /usr/bin/dsimport does not exist
    Debug: Puppet::Type::User::ProviderPw: file pw does not exist
    Debug: Puppet::Type::User::ProviderUser_role_add: file roleadd does not
    exist
    Debug: Failed to load library 'ldap' for feature 'ldap'
    Debug: Puppet::Type::Group::ProviderLdap: feature ldap is missing
    Debug: Puppet::Type::Group::ProviderDirectoryservice: file /usr/bin/dscl
    does not exist
    Debug: Puppet::Type::Group::ProviderPw: file pw does not exist
    Debug: Using settings: adding file resource 'logdir':
    'File[/var/log/puppet]{:path=>"/var/log/puppet", :mode=>"750",
    :owner=>"puppet", :group=>"puppet", :ensure=>:directory, :loglevel=>:debug,
    :links=>:follow, :backup=>false}'
    Debug: Using settings: adding file resource 'statedir':
    'File[/var/lib/puppet/state]{:path=>"/var/lib/puppet/state", :mode=>"1755",
    :ensure=>:directory, :loglevel=>:debug, :links=>:follow, :backup=>false}'
    Debug: Using settings: adding file resource 'rundir':
    'File[/var/run/puppet]{:path=>"/var/run/puppet", :mode=>"755",
    :owner=>"puppet", :group=>"puppet", :ensure=>:directory, :loglevel=>:debug,
    :links=>:follow, :backup=>false}'
    Debug: Using settings: adding file resource 'libdir':
    'File[/var/lib/puppet/lib]{:path=>"/var/lib/puppet/lib",
    :ensure=>:directory, :loglevel=>:debug, :links=>:follow, :backup=>false}'
    Debug: Using settings: adding file resource 'certdir':
    'File[/var/lib/puppet/ssl/certs]{:path=>"/var/lib/puppet/ssl/certs",
    :owner=>"puppet", :ensure=>:directory, :loglevel=>:debug, :links=>:follow,
    :backup=>false}'
    Debug: Using settings: adding file resource 'ssldir':
    'File[/var/lib/puppet/ssl]{:path=>"/var/lib/puppet/ssl", :mode=>"771",
    :owner=>"puppet", :ensure=>:directory, :loglevel=>:debug, :links=>:follow,
    :backup=>false}'
    Debug: Using settings: adding file resource 'publickeydir':
    'File[/var/lib/puppet/ssl/public_keys]{:path=>"/var/lib/puppet/ssl/public_keys",
    :owner=>"puppet", :ensure=>:directory, :loglevel=>:debug, :links=>:follow,
    :backup=>false}'
    Debug: Using settings: adding file resource 'requestdir':
    'File[/var/lib/puppet/ssl/certificate_requests]{:path=>"/var/lib/puppet/ssl/certificate_requests",
    :owner=>"puppet", :ensure=>:directory, :loglevel=>:debug, :links=>:follow,
    :backup=>false}'
    Debug: Using settings: adding file resource 'privatekeydir':
    'File[/var/lib/puppet/ssl/private_keys]{:path=>"/var/lib/puppet/ssl/private_keys",
    :mode=>"750", :owner=>"puppet", :ensure=>:directory, :loglevel=>:debug,
    :links=>:follow, :backup=>false}'
    Debug: Using settings: adding file resource 'privatedir':
    'File[/var/lib/puppet/ssl/private]{:path=>"/var/lib/puppet/ssl/private",
    :mode=>"750", :owner=>"puppet", :ensure=>:directory, :loglevel=>:debug,
    :links=>:follow, :backup=>false}'
    Debug: Using settings: adding file resource 'manifestdir':
    'File[/etc/puppet/manifests]{:path=>"/etc/puppet/manifests",
    :ensure=>:directory, :loglevel=>:debug, :links=>:follow, :backup=>false}'
    Debug: Using settings: adding file resource 'masterhttplog':
    'File[/var/log/puppet/masterhttp.log]{:path=>"/var/log/puppet/masterhttp.log",
    :mode=>"660", :owner=>"puppet", :group=>"puppet", :ensure=>:file,
    :loglevel=>:debug, :links=>:follow, :backup=>false}'
    Debug: Using settings: adding file resource 'bucketdir':
    'File[/var/lib/puppet/bucket]{:path=>"/var/lib/puppet/bucket", :mode=>"750",
    :owner=>"puppet", :group=>"puppet", :ensure=>:directory, :loglevel=>:debug,
    :links=>:follow, :backup=>false}'
    Debug: Using settings: adding file resource 'rest_authconfig':
    'File[/etc/puppet/auth.conf]{:path=>"/etc/puppet/auth.conf", :ensure=>:file,
    :loglevel=>:debug, :links=>:follow, :backup=>false}'
    Debug: Using settings: adding file resource 'yamldir':
    'File[/var/lib/puppet/yaml]{:path=>"/var/lib/puppet/yaml", :mode=>"750",
    :owner=>"puppet", :group=>"puppet", :ensure=>:directory, :loglevel=>:debug,
    :links=>:follow, :backup=>false}'
    Debug: Using settings: adding file resource 'server_datadir':
    'File[/var/lib/puppet/server_data]{:path=>"/var/lib/puppet/server_data",
    :mode=>"750", :owner=>"puppet", :group=>"puppet", :ensure=>:directory,
    :loglevel=>:debug, :links=>:follow, :backup=>false}'
    Debug: Using settings: adding file resource 'reportdir':
    'File[/var/lib/puppet/reports]{:path=>"/var/lib/puppet/reports",
    :mode=>"750", :owner=>"puppet", :group=>"puppet", :ensure=>:directory,
    :loglevel=>:debug, :links=>:follow, :backup=>false}'
    Debug: Using settings: adding file resource 'railslog':
    'File[/var/log/puppet/rails.log]{:path=>"/var/log/puppet/rails.log",
    :mode=>"600", :owner=>"puppet", :group=>"puppet", :ensure=>:file,
    :loglevel=>:debug, :links=>:follow, :backup=>false}'
    Debug: /File[/var/lib/puppet/state]: Autorequiring File[/var/lib/puppet]
    Debug: /File[/var/lib/puppet/lib]: Autorequiring File[/var/lib/puppet]
    Debug: /File[/var/lib/puppet/ssl/certs]: Autorequiring
    File[/var/lib/puppet/ssl]
    Debug: /File[/var/lib/puppet/ssl]: Autorequiring File[/var/lib/puppet]
    Debug: /File[/var/lib/puppet/ssl/public_keys]: Autorequiring
    File[/var/lib/puppet/ssl]
    Debug: /File[/var/lib/puppet/ssl/certificate_requests]: Autorequiring
    File[/var/lib/puppet/ssl]
    Debug: /File[/var/lib/puppet/ssl/private_keys]: Autorequiring
    File[/var/lib/puppet/ssl]
    Debug: /File[/var/lib/puppet/ssl/private]: Autorequiring
    File[/var/lib/puppet/ssl]
    Debug: /File[/etc/puppet/manifests]: Autorequiring File[/etc/puppet]
    Debug: /File[/var/log/puppet/masterhttp.log]: Autorequiring
    File[/var/log/puppet]
    Debug: /File[/var/lib/puppet/bucket]: Autorequiring File[/var/lib/puppet]
    Debug: /File[/etc/puppet/auth.conf]: Autorequiring File[/etc/puppet]
    Debug: /File[/var/lib/puppet/yaml]: Autorequiring File[/var/lib/puppet]
    Debug: /File[/var/lib/puppet/server_data]: Autorequiring
    File[/var/lib/puppet]
    Debug: /File[/var/lib/puppet/reports]: Autorequiring
    File[/var/lib/puppet]
    Debug: /File[/var/log/puppet/rails.log]: Autorequiring
    File[/var/log/puppet]
    Debug: Finishing transaction 34502460
    Info: Connecting to mysql2 database: puppet
    Killed

    real 66m35.810s
    user 9m42.143s
    sys 6m31.294s


    My /etc/puppet/puppet.conf file:

    [main]
    # The Puppet log directory.
    # The default value is '$vardir/log'.
    logdir = /var/log/puppet

    # Where Puppet PID files are kept.
    # The default value is '$vardir/run'.
    rundir = /var/run/puppet

    # Where SSL certificates are kept.
    # The default value is '$confdir/ssl'.
    ssldir = $vardir/ssl

    facts_terminus = inventory_active_record

    storeconfigs = true
    dbadapter = mysql2
    dbserver = roger-test.ofi.softonic.lan
    dbname = puppet
    dbuser = puppet
    dbpassword =
    dbconnections = 20

    [agent]
    # The file in which puppetd stores a list of the classes
    # associated with the retrieved configuratiion. Can be loaded in
    # the separate ``puppet`` executable using the ``--loadclasses``
    # option.
    # The default value is '$confdir/classes.txt'.
    classfile = $vardir/classes.txt

    # Where puppetd caches the local configuration. An
    # extension indicating the cache format is added automatically.
    # The default value is '$confdir/localconfig'.
    localconfig = $vardir/localconfig

    Packages:

    [root@ST-L226-ROGTORG ~]# rpm -qa | egrep -i "(puppet|ruby)" | sort
    libselinux-ruby-2.1.13-15.fc19.x86_64
    puppet-3.3.1-1.fc19.noarch
    puppetdb-terminus-1.5.0-1.fc19.noarch
    puppetlabs-release-19-2.noarch
    ruby-2.0.0.247-15.fc19.x86_64
    ruby-augeas-0.5.0-1.fc19.x86_64
    ruby-devel-2.0.0.247-15.fc19.x86_64
    ruby-irb-2.0.0.247-15.fc19.noarch
    ruby-libs-2.0.0.247-15.fc19.x86_64
    ruby-rgen-0.6.5-1.fc19.noarch
    ruby-shadow-1.4.1-19.fc19.x86_64
    rubygem-activemodel-3.2.13-1.fc19.noarch
    rubygem-activerecord-3.2.13-1.fc19.noarch
    rubygem-activesupport-3.2.13-1.fc19.noarch
    rubygem-arel-3.0.2-3.fc19.noarch
    rubygem-bigdecimal-1.2.0-15.fc19.x86_64
    rubygem-builder-3.1.4-3.fc19.noarch
    rubygem-i18n-0.6.1-3.fc19.noarch
    rubygem-io-console-0.4.2-15.fc19.x86_64
    rubygem-json-1.7.7-100.fc19.x86_64
    rubygem-multi_json-1.7.1-1.fc19.noarch
    rubygem-mysql2-0.3.13-1.fc19.x86_64
    rubygem-psych-2.0.0-15.fc19.x86_64
    rubygem-rdoc-4.0.1-1.fc19.noarch
    rubygems-2.0.10-106.fc19.noarch
    rubygem-tzinfo-0.3.35-1.fc19.noarch
    rubypick-1.1.0-1.fc19.noarch


    I'm positive the MySQL server receives the query and returns the data,
    according to MySQL's slow-queries.log:

    # Time: 131008 17:06:14
    # User@Host: puppet[puppet] @ [172.20.59.127]
    # Thread_id: 169 Schema: puppet Last_errno: 0 Killed: 0
    # Query_time: 189.509563 Lock_time: 0.000136 Rows_sent: 6168111
    Rows_examined: 12712154 Rows_affected: 0 Rows_read: 18446744073709551613
    # Bytes_sent: 2426331478 Tmp_tables: 0 Tmp_disk_tables: 0
    Tmp_table_sizes: 0
    # InnoDB_trx_id: A93
    SET timestamp=1381244774;
    SELECT `hosts`.`id` AS t0_r0, `hosts`.`name` AS t0_r1, `hosts`.`ip` AS
    t0_r2, `hosts`.`environment` AS t0_r3, `hosts`.`last_compile` AS t0_r4,
    `hosts`.`last_freshcheck` AS t0_r5, `
    hosts`.`last_report` AS t0_r6, `hosts`.`updated_at` AS t0_r7,
    `hosts`.`source_file_id` AS t0_r8, `hosts`.`created_at` AS t0_r9,
    `resources`.`id` AS t1_r0, `resources`.`title` AS t1
    _r1, `resources`.`restype` AS t1_r2, `resources`.`host_id` AS t1_r3,
    `resources`.`source_file_id` AS t1_r4, `resources`.`exported` AS t1_r5,
    `resources`.`line` AS t1_r6, `resources
    `.`updated_at` AS t1_r7, `resources`.`created_at` AS t1_r8,
    `param_values`.`id` AS t2_r0, `param_values`.`value` AS t2_r1,
    `param_values`.`param_name_id` AS t2_r2, `param_values`.`
    line` AS t2_r3, `param_values`.`resource_id` AS t2_r4,
    `param_values`.`updated_at` AS t2_r5, `param_values`.`created_at` AS t2_r6,
    `puppet_tags`.`id` AS t3_r0, `puppet_tags`.`name`
    AS t3_r1, `puppet_tags`.`updated_at` AS t3_r2,
    `puppet_tags`.`created_at` AS t3_r3 FROM `hosts` LEFT OUTER JOIN `resources`
    ON `resources`.`host_id` = `hosts`.`id` LEFT OUTER JOIN
    `param_values` ON `param_values`.`resource_id` = `resources`.`id` LEFT
    OUTER JOIN `resource_tags` ON `resource_tags`.`resource_id` =
    `resources`.`id` LEFT OUTER JOIN `puppet_tags`
    ON `puppet_tags`.`id` = `resource_tags`.`puppet_tag_id` WHERE
    `resources`.`exported` = 1;


    When the process gets stuck, an strace shows a constant and very quick
    flow of:

    30666 stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=2593, ...}) =


    The final moments of the strace showed:

    30667 restart_syscall(<... resuming interrupted call ...> <unfinished
    ...>
    30666 mremap(0x7f7aa4d0a000, 2732032, 2752512, MREMAP_MAYMOVE) =
    0x7f7aa4d0a000
    30666 brk(0) = 0x23aaea000
    30666 brk(0x23ab0b000) = 0x23ab0b000
    30666 brk(0) = 0x23ab0b000
    30666 brk(0x23ab2c000) = 0x23ab2c000
    30666 brk(0) = 0x23ab2c000
    30666 brk(0x23ab4d000) = 0x23ab4d000
    ... then lots of...
    30666 stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=2593, ...}) =

    30666 stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=2593, ...}) =

    30666 stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=2593, ...}) =

    ... and finally ...
    30666 stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=2593, ...}) =

    30666 stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=2593, ...}) =

    30666 stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=2593, ...}) =

    30667 +++ killed by SIGKILL +++
    30666 +++ killed by SIGKILL +++

    Any idea of what is happening? Am I doing something wrong?

    Thanks!

    Roger Torrentsgenerós
    --
    You received this message because you are subscribed to the Google Groups
    "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to puppet-users@googlegroups.com.
    Visit this group at http://groups.google.com/group/puppet-users.
    For more options, visit https://groups.google.com/groups/opt_out.
    --
    You received this message because you are subscribed to the Google Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to puppet-users@googlegroups.com.
    Visit this group at http://groups.google.com/group/puppet-users.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Roger Torrentsgenerós at Oct 18, 2013 at 12:56 pm
    Done

    http://projects.puppetlabs.com/issues/22912
    On Thursday, October 17, 2013 10:40:12 AM UTC+2, Ken Barber wrote:

    Roger,

    Can you raise this as a bug in our bug tracker?

    http://projects.puppetlabs.com/projects/puppetdb

    ken.
    On Wed, Oct 16, 2013 at 9:23 AM, Roger Torrentsgenerós wrote:

    Well I finally did it.

    First thing to do is to specify a timezone to avoid the massive amount of:

    17169 stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=2593, ...}) = 0
    Issuing '# export TZ=:/etc/localtime' prior to launching the Puppet face is
    enough.

    Then, in my particular case I had to use a VM with 42GB of RAM to deal with
    a 1.4GB MySQL database. After more than 2 hours of processing, a .tgz file
    with the ridiculous size of 1.7MB was created.

    So if your export process gets killed and you see OOM messages in your
    dmesg, you'll need more RAM in your server.

    Roger


    On Friday, October 11, 2013 2:21:12 PM UTC+2, Roger Torrentsgenerós
    wrote:

    Just an update. This is what strace shows at the point where the
    process
    finishes fetching data from the MySQL database:

    17169 read(4, "38:31\373\373\0232013-07-15 17:38:32\373\023201"...,
    16384)
    = 384
    17169 read(4, "3:00:07\0074113192\27xavier.ofi.softo"..., 16384) =
    16000
    17169 read(4, "305566\17some.user\003194\00222\00741"..., 16384) = 384
    17169 read(4, "192\0232013-04-02 16:11:44\0232013-04-"..., 16384) =
    8534
    17169 gettimeofday({1381493430, 91178}, NULL) = 0
    17169 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
    17169 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
    17169 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
    17169 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
    17169 mmap(NULL, 49348608, PROT_READ|PROT_WRITE,
    MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f41c7266000
    17169 stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=2593, ...})
    =
    17169 stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=2593, ...})
    =
    17169 stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=2593, ...})
    =

    Beyond this point, tons of exactly the same line, CPU at 100% and
    memory
    usage increasing until it fills RAM and swap space, and then the
    SIGKILL
    shown above.

    Roger

    On Tuesday, October 8, 2013 7:22:20 PM UTC+2, Roger Torrentsgenerós
    wrote:

    Hi,

    I currently have a MySQL database containing all Puppet storeconfigs.
    My
    intention is to migrate to PuppetDB on a PostgreSQL server, so the
    first
    step is to use the 'storeconfigs' face to export all the DB to a file
    PuppetDB can later consume. But the 'puppet storeconfigs export'
    command
    always ends up being killed, I suspect due to some sort of OOM issue.
    I have
    tried different environments, different Puppet / Ruby / whatever
    versions,
    different hardware... No matter what, it always ends up the same.

    The process starts well, connects to the database and fetches all the
    data, but then it seems to get stuck after this output line:
    Info: Connecting to mysql2 database: puppet

    Meanwhile, a CPU is working and the amount of memory used by this
    process
    increases constantly. I have tried on several servers, one with up to
    28GB
    of RAM, and it died the same way.

    Here are the facts of my last attempt:

    The execution output:

    [root@ST-L226-ROGTORG ~]# time puppet storeconfigs --verbose --debug
    export
    Debug: Using settings: adding file resource 'confdir':
    'File[/etc/puppet]{:path=>"/etc/puppet", :ensure=>:directory,
    :loglevel=>:debug, :links=>:follow, :backup=>false}'
    Debug: Using settings: adding file resource 'vardir':
    'File[/var/lib/puppet]{:path=>"/var/lib/puppet", :ensure=>:directory,
    :loglevel=>:debug, :links=>:follow, :backup=>false}'
    Debug: Failed to load library 'ldap' for feature 'ldap'
    Debug: Puppet::Type::User::ProviderLdap: feature ldap is missing
    Debug: Puppet::Type::User::ProviderDirectoryservice: file
    /usr/bin/dsimport does not exist
    Debug: Puppet::Type::User::ProviderPw: file pw does not exist
    Debug: Puppet::Type::User::ProviderUser_role_add: file roleadd does
    not
    exist
    Debug: Failed to load library 'ldap' for feature 'ldap'
    Debug: Puppet::Type::Group::ProviderLdap: feature ldap is missing
    Debug: Puppet::Type::Group::ProviderDirectoryservice: file
    /usr/bin/dscl
    does not exist
    Debug: Puppet::Type::Group::ProviderPw: file pw does not exist
    Debug: Using settings: adding file resource 'logdir':
    'File[/var/log/puppet]{:path=>"/var/log/puppet", :mode=>"750",
    :owner=>"puppet", :group=>"puppet", :ensure=>:directory,
    :loglevel=>:debug,
    :links=>:follow, :backup=>false}'
    Debug: Using settings: adding file resource 'statedir':
    'File[/var/lib/puppet/state]{:path=>"/var/lib/puppet/state",
    :mode=>"1755",
    :ensure=>:directory, :loglevel=>:debug, :links=>:follow,
    :backup=>false}'
    Debug: Using settings: adding file resource 'rundir':
    'File[/var/run/puppet]{:path=>"/var/run/puppet", :mode=>"755",
    :owner=>"puppet", :group=>"puppet", :ensure=>:directory,
    :loglevel=>:debug,
    :links=>:follow, :backup=>false}'
    Debug: Using settings: adding file resource 'libdir':
    'File[/var/lib/puppet/lib]{:path=>"/var/lib/puppet/lib",
    :ensure=>:directory, :loglevel=>:debug, :links=>:follow,
    :backup=>false}'
    Debug: Using settings: adding file resource 'certdir':
    'File[/var/lib/puppet/ssl/certs]{:path=>"/var/lib/puppet/ssl/certs",
    :owner=>"puppet", :ensure=>:directory, :loglevel=>:debug,
    :links=>:follow,
    :backup=>false}'
    Debug: Using settings: adding file resource 'ssldir':
    'File[/var/lib/puppet/ssl]{:path=>"/var/lib/puppet/ssl", :mode=>"771",
    :owner=>"puppet", :ensure=>:directory, :loglevel=>:debug,
    :links=>:follow,
    :backup=>false}'
    Debug: Using settings: adding file resource 'publickeydir':
    'File[/var/lib/puppet/ssl/public_keys]{:path=>"/var/lib/puppet/ssl/public_keys",
    :owner=>"puppet", :ensure=>:directory, :loglevel=>:debug,
    :links=>:follow,
    :backup=>false}'
    Debug: Using settings: adding file resource 'requestdir':
    'File[/var/lib/puppet/ssl/certificate_requests]{:path=>"/var/lib/puppet/ssl/certificate_requests",
    :owner=>"puppet", :ensure=>:directory, :loglevel=>:debug,
    :links=>:follow,
    :backup=>false}'
    Debug: Using settings: adding file resource 'privatekeydir':
    'File[/var/lib/puppet/ssl/private_keys]{:path=>"/var/lib/puppet/ssl/private_keys",
    :mode=>"750", :owner=>"puppet", :ensure=>:directory,
    :loglevel=>:debug,
    :links=>:follow, :backup=>false}'
    Debug: Using settings: adding file resource 'privatedir':
    'File[/var/lib/puppet/ssl/private]{:path=>"/var/lib/puppet/ssl/private",
    :mode=>"750", :owner=>"puppet", :ensure=>:directory,
    :loglevel=>:debug,
    :links=>:follow, :backup=>false}'
    Debug: Using settings: adding file resource 'manifestdir':
    'File[/etc/puppet/manifests]{:path=>"/etc/puppet/manifests",
    :ensure=>:directory, :loglevel=>:debug, :links=>:follow,
    :backup=>false}'
    Debug: Using settings: adding file resource 'masterhttplog':
    'File[/var/log/puppet/masterhttp.log]{:path=>"/var/log/puppet/masterhttp.log",
    :mode=>"660", :owner=>"puppet", :group=>"puppet", :ensure=>:file,
    :loglevel=>:debug, :links=>:follow, :backup=>false}'
    Debug: Using settings: adding file resource 'bucketdir':
    'File[/var/lib/puppet/bucket]{:path=>"/var/lib/puppet/bucket",
    :mode=>"750",
    :owner=>"puppet", :group=>"puppet", :ensure=>:directory,
    :loglevel=>:debug,
    :links=>:follow, :backup=>false}'
    Debug: Using settings: adding file resource 'rest_authconfig':
    'File[/etc/puppet/auth.conf]{:path=>"/etc/puppet/auth.conf",
    :ensure=>:file,
    :loglevel=>:debug, :links=>:follow, :backup=>false}'
    Debug: Using settings: adding file resource 'yamldir':
    'File[/var/lib/puppet/yaml]{:path=>"/var/lib/puppet/yaml",
    :mode=>"750",
    :owner=>"puppet", :group=>"puppet", :ensure=>:directory,
    :loglevel=>:debug,
    :links=>:follow, :backup=>false}'
    Debug: Using settings: adding file resource 'server_datadir':
    'File[/var/lib/puppet/server_data]{:path=>"/var/lib/puppet/server_data",
    :mode=>"750", :owner=>"puppet", :group=>"puppet", :ensure=>:directory,
    :loglevel=>:debug, :links=>:follow, :backup=>false}'
    Debug: Using settings: adding file resource 'reportdir':
    'File[/var/lib/puppet/reports]{:path=>"/var/lib/puppet/reports",
    :mode=>"750", :owner=>"puppet", :group=>"puppet", :ensure=>:directory,
    :loglevel=>:debug, :links=>:follow, :backup=>false}'
    Debug: Using settings: adding file resource 'railslog':
    'File[/var/log/puppet/rails.log]{:path=>"/var/log/puppet/rails.log",
    :mode=>"600", :owner=>"puppet", :group=>"puppet", :ensure=>:file,
    :loglevel=>:debug, :links=>:follow, :backup=>false}'
    Debug: /File[/var/lib/puppet/state]: Autorequiring
    File[/var/lib/puppet]
    Debug: /File[/var/lib/puppet/lib]: Autorequiring File[/var/lib/puppet]
    Debug: /File[/var/lib/puppet/ssl/certs]: Autorequiring
    File[/var/lib/puppet/ssl]
    Debug: /File[/var/lib/puppet/ssl]: Autorequiring File[/var/lib/puppet]
    Debug: /File[/var/lib/puppet/ssl/public_keys]: Autorequiring
    File[/var/lib/puppet/ssl]
    Debug: /File[/var/lib/puppet/ssl/certificate_requests]: Autorequiring
    File[/var/lib/puppet/ssl]
    Debug: /File[/var/lib/puppet/ssl/private_keys]: Autorequiring
    File[/var/lib/puppet/ssl]
    Debug: /File[/var/lib/puppet/ssl/private]: Autorequiring
    File[/var/lib/puppet/ssl]
    Debug: /File[/etc/puppet/manifests]: Autorequiring File[/etc/puppet]
    Debug: /File[/var/log/puppet/masterhttp.log]: Autorequiring
    File[/var/log/puppet]
    Debug: /File[/var/lib/puppet/bucket]: Autorequiring
    File[/var/lib/puppet]
    Debug: /File[/etc/puppet/auth.conf]: Autorequiring File[/etc/puppet]
    Debug: /File[/var/lib/puppet/yaml]: Autorequiring
    File[/var/lib/puppet]
    Debug: /File[/var/lib/puppet/server_data]: Autorequiring
    File[/var/lib/puppet]
    Debug: /File[/var/lib/puppet/reports]: Autorequiring
    File[/var/lib/puppet]
    Debug: /File[/var/log/puppet/rails.log]: Autorequiring
    File[/var/log/puppet]
    Debug: Finishing transaction 34502460
    Info: Connecting to mysql2 database: puppet
    Killed

    real 66m35.810s
    user 9m42.143s
    sys 6m31.294s


    My /etc/puppet/puppet.conf file:

    [main]
    # The Puppet log directory.
    # The default value is '$vardir/log'.
    logdir = /var/log/puppet

    # Where Puppet PID files are kept.
    # The default value is '$vardir/run'.
    rundir = /var/run/puppet

    # Where SSL certificates are kept.
    # The default value is '$confdir/ssl'.
    ssldir = $vardir/ssl

    facts_terminus = inventory_active_record

    storeconfigs = true
    dbadapter = mysql2
    dbserver = roger-test.ofi.softonic.lan
    dbname = puppet
    dbuser = puppet
    dbpassword =
    dbconnections = 20

    [agent]
    # The file in which puppetd stores a list of the classes
    # associated with the retrieved configuratiion. Can be loaded in
    # the separate ``puppet`` executable using the ``--loadclasses``
    # option.
    # The default value is '$confdir/classes.txt'.
    classfile = $vardir/classes.txt

    # Where puppetd caches the local configuration. An
    # extension indicating the cache format is added automatically.
    # The default value is '$confdir/localconfig'.
    localconfig = $vardir/localconfig

    Packages:

    [root@ST-L226-ROGTORG ~]# rpm -qa | egrep -i "(puppet|ruby)" | sort
    libselinux-ruby-2.1.13-15.fc19.x86_64
    puppet-3.3.1-1.fc19.noarch
    puppetdb-terminus-1.5.0-1.fc19.noarch
    puppetlabs-release-19-2.noarch
    ruby-2.0.0.247-15.fc19.x86_64
    ruby-augeas-0.5.0-1.fc19.x86_64
    ruby-devel-2.0.0.247-15.fc19.x86_64
    ruby-irb-2.0.0.247-15.fc19.noarch
    ruby-libs-2.0.0.247-15.fc19.x86_64
    ruby-rgen-0.6.5-1.fc19.noarch
    ruby-shadow-1.4.1-19.fc19.x86_64
    rubygem-activemodel-3.2.13-1.fc19.noarch
    rubygem-activerecord-3.2.13-1.fc19.noarch
    rubygem-activesupport-3.2.13-1.fc19.noarch
    rubygem-arel-3.0.2-3.fc19.noarch
    rubygem-bigdecimal-1.2.0-15.fc19.x86_64
    rubygem-builder-3.1.4-3.fc19.noarch
    rubygem-i18n-0.6.1-3.fc19.noarch
    rubygem-io-console-0.4.2-15.fc19.x86_64
    rubygem-json-1.7.7-100.fc19.x86_64
    rubygem-multi_json-1.7.1-1.fc19.noarch
    rubygem-mysql2-0.3.13-1.fc19.x86_64
    rubygem-psych-2.0.0-15.fc19.x86_64
    rubygem-rdoc-4.0.1-1.fc19.noarch
    rubygems-2.0.10-106.fc19.noarch
    rubygem-tzinfo-0.3.35-1.fc19.noarch
    rubypick-1.1.0-1.fc19.noarch


    I'm positive the MySQL server receives the query and returns the data,
    according to MySQL's slow-queries.log:

    # Time: 131008 17:06:14
    # User@Host: puppet[puppet] @ [172.20.59.127]
    # Thread_id: 169 Schema: puppet Last_errno: 0 Killed: 0
    # Query_time: 189.509563 Lock_time: 0.000136 Rows_sent: 6168111
    Rows_examined: 12712154 Rows_affected: 0 Rows_read:
    18446744073709551613
    # Bytes_sent: 2426331478 Tmp_tables: 0 Tmp_disk_tables: 0
    Tmp_table_sizes: 0
    # InnoDB_trx_id: A93
    SET timestamp=1381244774;
    SELECT `hosts`.`id` AS t0_r0, `hosts`.`name` AS t0_r1, `hosts`.`ip` AS
    t0_r2, `hosts`.`environment` AS t0_r3, `hosts`.`last_compile` AS
    t0_r4,
    `hosts`.`last_freshcheck` AS t0_r5, `
    hosts`.`last_report` AS t0_r6, `hosts`.`updated_at` AS t0_r7,
    `hosts`.`source_file_id` AS t0_r8, `hosts`.`created_at` AS t0_r9,
    `resources`.`id` AS t1_r0, `resources`.`title` AS t1
    _r1, `resources`.`restype` AS t1_r2, `resources`.`host_id` AS t1_r3,
    `resources`.`source_file_id` AS t1_r4, `resources`.`exported` AS
    t1_r5,
    `resources`.`line` AS t1_r6, `resources
    `.`updated_at` AS t1_r7, `resources`.`created_at` AS t1_r8,
    `param_values`.`id` AS t2_r0, `param_values`.`value` AS t2_r1,
    `param_values`.`param_name_id` AS t2_r2, `param_values`.`
    line` AS t2_r3, `param_values`.`resource_id` AS t2_r4,
    `param_values`.`updated_at` AS t2_r5, `param_values`.`created_at` AS
    t2_r6,
    `puppet_tags`.`id` AS t3_r0, `puppet_tags`.`name`
    AS t3_r1, `puppet_tags`.`updated_at` AS t3_r2,
    `puppet_tags`.`created_at` AS t3_r3 FROM `hosts` LEFT OUTER JOIN
    `resources`
    ON `resources`.`host_id` = `hosts`.`id` LEFT OUTER JOIN
    `param_values` ON `param_values`.`resource_id` = `resources`.`id`
    LEFT
    OUTER JOIN `resource_tags` ON `resource_tags`.`resource_id` =
    `resources`.`id` LEFT OUTER JOIN `puppet_tags`
    ON `puppet_tags`.`id` = `resource_tags`.`puppet_tag_id` WHERE
    `resources`.`exported` = 1;


    When the process gets stuck, an strace shows a constant and very quick
    flow of:

    30666 stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=2593,
    ...}) =

    The final moments of the strace showed:

    30667 restart_syscall(<... resuming interrupted call ...> <unfinished
    ...>
    30666 mremap(0x7f7aa4d0a000, 2732032, 2752512, MREMAP_MAYMOVE) =
    0x7f7aa4d0a000
    30666 brk(0) = 0x23aaea000
    30666 brk(0x23ab0b000) = 0x23ab0b000
    30666 brk(0) = 0x23ab0b000
    30666 brk(0x23ab2c000) = 0x23ab2c000
    30666 brk(0) = 0x23ab2c000
    30666 brk(0x23ab4d000) = 0x23ab4d000
    ... then lots of...
    30666 stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=2593,
    ...}) =
    30666 stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=2593,
    ...}) =
    30666 stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=2593,
    ...}) =
    ... and finally ...
    30666 stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=2593,
    ...}) =
    30666 stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=2593,
    ...}) =
    30666 stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=2593,
    ...}) =
    30667 +++ killed by SIGKILL +++
    30666 +++ killed by SIGKILL +++

    Any idea of what is happening? Am I doing something wrong?

    Thanks!

    Roger Torrentsgenerós
    --
    You received this message because you are subscribed to the Google Groups
    "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to puppet-users...@googlegroups.com <javascript:>.
    To post to this group, send email to puppet...@googlegroups.com<javascript:>.
    Visit this group at http://groups.google.com/group/puppet-users.
    For more options, visit https://groups.google.com/groups/opt_out.
    --
    You received this message because you are subscribed to the Google Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to puppet-users@googlegroups.com.
    Visit this group at http://groups.google.com/group/puppet-users.
    For more options, visit https://groups.google.com/groups/opt_out.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
grouppuppet-users @
categoriespuppet
postedOct 8, '13 at 5:27p
activeOct 18, '13 at 12:56p
posts5
users2
websitepuppetlabs.com

People

Translate

site design / logo © 2022 Grokbase