I was looking at the service level logging settings. I have set them all to
WARN level now. I also set it to only retain 48 hours of logs in each of
the monitor services. I noticed another setting to purge job details after
14 days though, so I set this to 7 days.
Under the service level, I don't see anywhere to limit the total logs size
though. All I can see is a few parts (eg. Reports Manager Max Log Size)that limit to 200MB by default. How do I limit the total size of
cloudera-scm-server-db?
Here's a list of the top tables by size from PostgreSQL:
relname | relpages
----------------------------------------+----------
commands | 134
pg_attribute | 68
pg_proc | 54
pg_depend | 48
pg_toast_16982 | 45
pg_depend_reference_index | 40
pg_depend_depender_index | 39
configs | 33
pg_proc_proname_args_nsp_index | 26
pg_statistic | 26
audits | 26
pg_toast_2618 | 25
pg_attribute_relid_attnam_index | 22
pg_description | 21
idx_command_parent | 18
idx_command_schedule | 16
pg_description_o_c_o_index | 16
idx_command_host | 16
processes | 16
idx_command_cluster | 16
pg_attribute_relid_attnum_index | 14
On Tuesday, August 20, 2013 6:43:49 PM UTC+1, Darren Lo wrote:
Hi Jaykay,
I don't know why these configs would have changed, other than a manual
action from CM user (via UI or API). How did you set them in the first
place? Did you set them at service-level, or role-level? You can tell if
you've done a role-level override by looking at the service-level configs
(click on the Mgmt service, then on Configuration -> View and Edit). It
will have a note next to any config that has a role-level override. It's
possible that you set a role-level override before, and now when you
inspected the config you were looking at service-level, which still said
INFO. If you have the Enterprise or Trial edition, you can also look in the
config revision history to see exactly what happened to your configs.
Since you are concerned about the max log size as well as the file growth
rate, you may want to configure the log settings to decrease the size per
file or the maximum number of files kept for each service. This can be
configured in the logs tab for each role.
Thanks,
Darren
On Tue, Aug 20, 2013 at 5:38 AM, Jaykay <[email protected] <javascript:>>wrote:
--
Thanks,
Darren
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].Hi Jaykay,
I don't know why these configs would have changed, other than a manual
action from CM user (via UI or API). How did you set them in the first
place? Did you set them at service-level, or role-level? You can tell if
you've done a role-level override by looking at the service-level configs
(click on the Mgmt service, then on Configuration -> View and Edit). It
will have a note next to any config that has a role-level override. It's
possible that you set a role-level override before, and now when you
inspected the config you were looking at service-level, which still said
INFO. If you have the Enterprise or Trial edition, you can also look in the
config revision history to see exactly what happened to your configs.
Since you are concerned about the max log size as well as the file growth
rate, you may want to configure the log settings to decrease the size per
file or the maximum number of files kept for each service. This can be
configured in the logs tab for each role.
Thanks,
Darren
On Tue, Aug 20, 2013 at 5:38 AM, Jaykay <[email protected] <javascript:>>wrote:
Hi all,
I came back to work today after the weekend to see that the
logs(cloudera-scm-server-db) have gone from 4.5GB to 32GB, which is the
full amount of free disk space. It seems that the log growth had slowed but
is has now gone back to unmanageable. When I looked at the monitoring
service settings, the logging level had reverted back to INFO on all of
them. I had set it to WARN last week. Any idea what would cause it to
revert?
an email to [email protected] <javascript:>.
I came back to work today after the weekend to see that the
logs(cloudera-scm-server-db) have gone from 4.5GB to 32GB, which is the
full amount of free disk space. It seems that the log growth had slowed but
is has now gone back to unmanageable. When I looked at the monitoring
service settings, the logging level had reverted back to INFO on all of
them. I had set it to WARN last week. Any idea what would cause it to
revert?
On Wednesday, August 14, 2013 3:31:30 PM UTC+1, Darren Lo wrote:
If you need a db admin for the embedded postgresql db, that login is in
/var/lib/cloudera-scm-server-**db/data/generated_password.**txt.
--
Thanks,
Darren
To unsubscribe from this group and stop receiving emails from it, sendIf you need a db admin for the embedded postgresql db, that login is in
/var/lib/cloudera-scm-server-**db/data/generated_password.**txt.
On Wed, Aug 14, 2013 at 7:24 AM, Serega Sheypak wrote:
you are welcome :)
2013/8/14 Jaykay <[email protected]>
Thanks again Joe.
you are welcome :)
2013/8/14 Jaykay <[email protected]>
Thanks again Joe.
Thanks to all who helped me out here, Sergey and Gerd too. I have a
lot to learn with this stuff.
lot to learn with this stuff.
On Wednesday, August 14, 2013 2:04:19 PM UTC+1, Joe Travaglini wrote:
it should be located in /etc/cloudera-scm-server/db.**pr**operties.
it should be located in /etc/cloudera-scm-server/db.**pr**operties.
On Wed, Aug 14, 2013 at 8:50 AM, Jaykay wrote:
Yeah default didn't work.
Yeah default didn't work.
On Wednesday, August 14, 2013 1:43:47 PM UTC+1, Serega Sheypak wrote:
did you try admin/admin?
2013/8/14 Jaykay <[email protected]>
Thanks Joe. I ended up rolling back to a snapshot, as it's a VM. If
did you try admin/admin?
2013/8/14 Jaykay <[email protected]>
Thanks Joe. I ended up rolling back to a snapshot, as it's a VM. If
I want to connect to postgres to look at db sizes, any idea what the
default postgres login is with a Cloudera install?
On Wednesday, August 14, 2013 1:00:27 PM UTC+1, Joe Travaglini
wrote:
default postgres login is with a Cloudera install?
On Wednesday, August 14, 2013 1:00:27 PM UTC+1, Joe Travaglini
wrote:
IMO the quickest way to purge space is to find a log file that's
large.
du /var/log | sort -rn | less
echo "" > /path/to/that/file
This will free space immediately even if a process has an open fd
to write to that log file. If you simply rm the file while the process
writing to it is still running, it will not free the disk space.
large.
du /var/log | sort -rn | less
echo "" > /path/to/that/file
This will free space immediately even if a process has an open fd
to write to that log file. If you simply rm the file while the process
writing to it is still running, it will not free the disk space.
On Wed, Aug 14, 2013 at 7:47 AM, Jaykay wrote:
Too late, all my disk space is gone again, so I can't change
anything in the GUI. Is there a way to purge the data from
cloudera-scm-server-db? I would like to connect to the db to see which
tables are taking up space, but I am out of space now, so psql can't run
properly.
On Wednesday, August 14, 2013 7:42:25 AM UTC+1, Gerd Koenig
wrote:
Too late, all my disk space is gone again, so I can't change
anything in the GUI. Is there a way to purge the data from
cloudera-scm-server-db? I would like to connect to the db to see which
tables are taking up space, but I am out of space now, so psql can't run
properly.
On Wednesday, August 14, 2013 7:42:25 AM UTC+1, Gerd Koenig
wrote:
Yes, there is a way to configure the min. log level.
To configure it, e.g. for the Datanodes, navigate to the
configuration property "DataNode Logging Th reshold" via
"Service HDFS" => "Configuration" => "View and Edit" => "Default" =>
"Datanode (Default)" => "Logs"
br....Gerd....
To configure it, e.g. for the Datanodes, navigate to the
configuration property "DataNode Logging Th reshold" via
"Service HDFS" => "Configuration" => "View and Edit" => "Default" =>
"Datanode (Default)" => "Logs"
br....Gerd....
On 13 August 2013 16:15, Jaykay wrote:
Is there a way to set CM to log in less verbose way? I notice
that I can search the logs, and filter by min. log level, which includes
info, debug, warn, error, fatal.
I would like to set it to only record error and fatal.
On Tuesday, August 13, 2013 1:12:04 PM UTC+1, Serega Sheypak
wrote:
Is there a way to set CM to log in less verbose way? I notice
that I can search the logs, and filter by min. log level, which includes
info, debug, warn, error, fatal.
I would like to set it to only record error and fatal.
On Tuesday, August 13, 2013 1:12:04 PM UTC+1, Serega Sheypak
wrote:
We didn't do something special. It should be cleaned in a
while...
2013/8/13 Jaykay <[email protected]>
while...
2013/8/13 Jaykay <[email protected]>
Correction to my earlier post:
same size. Do I need to do some kind of purge to get the size down?
Thanks Serega, it is indeed the
/var/lib/cloudera-scm-server-**d************b that is
taking up all the space.
Host and Service Monitor expiration periods are both set to
7days only. I will change this down to 3days and see how it looks then.
I have changed it down to 3days, but the db is still the/var/lib/cloudera-scm-server-**d************b that is
taking up all the space.
Host and Service Monitor expiration periods are both set to
7days only. I will change this down to 3days and see how it looks then.
same size. Do I need to do some kind of purge to get the size down?
On Tuesday, August 13, 2013 11:31:05 AM UTC+1, Serega
Sheypak wrote:
Sheypak wrote:
If it loks this way:
[[email protected] run]# du -sh /var/lib/* | sort -h
4,0K /var/lib/fprint
4,0K /var/lib/games
4,0K /var/lib/logrotate.status
4,0K /var/lib/misc
4,0K /var/lib/net-snmp
4,0K /var/lib/ntp
4,0K /var/lib/random-seed
4,0K /var/lib/rsyslog
8,0K /var/lib/dbus
8,0K /var/lib/plymouth
8,0K /var/lib/udev
12K /var/lib/prelink
12K /var/lib/samba
12K /var/lib/stateless
16K /var/lib/dhclient
16K /var/lib/pgsql
20K /var/lib/authconfig
28K /var/lib/polkit-1
208K /var/lib/alternatives
868K /var/lib/readahead
3,3M /var/lib/mlocate
6,8M /var/lib/yum
38M /var/lib/rpm
*90G /var/lib/cloudera-scm-server-db*
Then you need to tune management service
Host Monitor Data Expiration Period (hours)
Service Monitor Data Expiration Period (hours)
and so on...
We also had such problem. We are planning to migrate
management service to standalone VM with standalone PostGre SQL server.
2013/8/13 Jaykay <[email protected]>
[[email protected] run]# du -sh /var/lib/* | sort -h
4,0K /var/lib/fprint
4,0K /var/lib/games
4,0K /var/lib/logrotate.status
4,0K /var/lib/misc
4,0K /var/lib/net-snmp
4,0K /var/lib/ntp
4,0K /var/lib/random-seed
4,0K /var/lib/rsyslog
8,0K /var/lib/dbus
8,0K /var/lib/plymouth
8,0K /var/lib/udev
12K /var/lib/prelink
12K /var/lib/samba
12K /var/lib/stateless
16K /var/lib/dhclient
16K /var/lib/pgsql
20K /var/lib/authconfig
28K /var/lib/polkit-1
208K /var/lib/alternatives
868K /var/lib/readahead
3,3M /var/lib/mlocate
6,8M /var/lib/yum
38M /var/lib/rpm
*90G /var/lib/cloudera-scm-server-db*
Then you need to tune management service
Host Monitor Data Expiration Period (hours)
Service Monitor Data Expiration Period (hours)
and so on...
We also had such problem. We are planning to migrate
management service to standalone VM with standalone PostGre SQL server.
2013/8/13 Jaykay <[email protected]>
I upgraded my Cloudera Manager server from 4.2 to 4.6.
Since upgrading, I have lost over 20GB in disk space in about a day. I
assume it is from the logging that comes with the latest version. Any way
to change it so that logging takes up less disk space?
Since upgrading, I have lost over 20GB in disk space in about a day. I
assume it is from the logging that comes with the latest version. Any way
to change it so that logging takes up less disk space?
--
Thanks,
Darren
an email to [email protected] <javascript:>.
--
Thanks,
Darren
