FAQ
In a moment of epic stupidity, having ran out of space on the root
partition of a server due to /var chewing up the space, I added a
separate drive for the purpose of mounting it as /var

To do so, I mounted the new drive as /var2, cp -R (in hindsight should
had rsync to preserve attributes), deleted the original /var to free
up space, edited fstab and rebooted... unsurprisingly to a fubar'd
server.

The thing is it still boots, I can get into single user mode but a
full init get stuck at starting the syslogger. What is the best way to
rescue the server now from my own stupidity and keeping all the
existing configuration and data?

Search Discussions

  • Darod Zyree at Aug 3, 2012 at 6:31 am

    2012/8/3 Emmanuel Noobadmin <centos.admin@gmail.com>:
    In a moment of epic stupidity, having ran out of space on the root
    partition of a server due to /var chewing up the space, I added a
    separate drive for the purpose of mounting it as /var

    To do so, I mounted the new drive as /var2, cp -R (in hindsight should
    had rsync to preserve attributes), deleted the original /var to free
    up space, edited fstab and rebooted... unsurprisingly to a fubar'd
    server.

    The thing is it still boots, I can get into single user mode but a
    full init get stuck at starting the syslogger. What is the best way to
    rescue the server now from my own stupidity and keeping all the
    existing configuration and data?
    _______________________________________________
    CentOS mailing list
    CentOS at centos.org
    http://lists.centos.org/mailman/listinfo/centos
    Did you rewrite the selinux policy on /var or have you tried disabling
    selinux if you haven't do so already?
  • Emmanuel Noobadmin at Aug 3, 2012 at 6:52 am

    On 8/3/12, Darod Zyree wrote:
    Did you rewrite the selinux policy on /var or have you tried disabling
    selinux if you haven't do so already?
    Thank you so much!

    Turning off selinux allowed me get the system running.
    However, after running fixfiles to restore the context for /var, I
    still cannot boot to init 5, with the choke point now NFS statd.

    I'll probably have to slowly hunt down the relevant selinux context
    one by one when nobody's screaming about the server being down.
  • Leonard den Ottolander at Aug 3, 2012 at 7:31 am
    Hello Emmanuel,
    On Fri, 2012-08-03 at 18:52 +0800, Emmanuel Noobadmin wrote:
    Turning off selinux allowed me get the system running.
    However, after running fixfiles to restore the context for /var, I
    still cannot boot to init 5, with the choke point now NFS statd.
    If you copied a live /var over to the new partition you might have
    issues from stale lock files that prevent services from
    starting. /var/lock/subsys/nfsd comes to mind in this particular case.
    If the service ain't running that file shouldn't be there.

    I don't think stale pid files are as troublesome but it doesn't hurt to
    have a look at /var/run as well.

    Regards,
    Leonard.

    --
    mount -t life -o ro /dev/dna /genetic/research
  • Karanbir Singh at Aug 3, 2012 at 7:41 am

    On 08/03/2012 11:52 AM, Emmanuel Noobadmin wrote:
    I'll probably have to slowly hunt down the relevant selinux context
    one by one when nobody's screaming about the server being down.
    Would restorecon not help get this bootrapped ? and then with selinux in
    permissive mode, watch the audit log like a hawk.

    - KB
    --
    Karanbir Singh
    +44-207-0999389 | http://www.karan.org/ | twitter.com/kbsingh
    ICQ: 2522219 | Yahoo IM: z00dax | Gtalk: z00dax
    GnuPG Key : http://www.karan.org/publickey.asc
  • Emmanuel Noobadmin at Aug 5, 2012 at 12:00 am

    On 8/3/12, Karanbir Singh wrote:
    On 08/03/2012 11:52 AM, Emmanuel Noobadmin wrote:
    I'll probably have to slowly hunt down the relevant selinux context
    one by one when nobody's screaming about the server being down.
    Would restorecon not help get this bootrapped ? and then with selinux in
    permissive mode, watch the audit log like a hawk.
    fixfiles/restorecon managed to get init 5 past syslogger but it got
    stuck still at NFS statd which locks up the entire server.

    But with setenforce to permissive, the system appears to work fine and
    yes I would be doing that watch the audit log thing during the next
    scheduled down time.
  • Lamar Owen at Aug 3, 2012 at 11:25 am

    On Friday, August 03, 2012 06:24:46 AM Emmanuel Noobadmin wrote:
    In a moment of epic stupidity, having ran out of space on the root
    partition of a server due to /var chewing up the space, I added a
    separate drive for the purpose of mounting it as /var
    ...

    This sort of things pops up from time to time.... from a thread back in April.....
    On Wednesday, April 11, 2012 05:38:13 PM Jason Pyeron wrote:
    -----Original Message-----
    From: centos-bounces at centos.org
    [mailto:centos-bounces at centos.org] On Behalf Of Alexander Dalloz
    Sent: Wednesday, April 11, 2012 17:12
    To: CentOS mailing list
    Subject: Re: [CentOS] How to fix a chown oops...

    Am 11.04.2012 23:02, schrieb Jason Pyeron:
    chown -R 7.0 /sbin/
    chown -R 98.98 //

    Is there a rpm way fix all the permissions of files managed by rpms?
    http://wiki.centos.org/TipsAndTricks/YumAndRPM#head-20a3ecce3d
    0762b9cdd3307ef2632e0c274a2bfd
    rpm -qa | while read line; do echo $line && rpm --setperms $line; done
    ...

    By extension:


    rpm -qa | while read line; do echo $line && rpm --setugids $line; done

    should handle ownerships. Then, reenable selinux in permissive mode, and set it to relabel on the next boot.
  • Karanbir Singh at Aug 3, 2012 at 12:03 pm
    Hi,
    On 08/03/2012 04:25 PM, Lamar Owen wrote:
    rpm -qa | while read line; do echo $line && rpm --setugids $line; done
    should handle ownerships. Then, reenable selinux in permissive mode, and set it to relabel on the next boot.
    maybe add --setperms as well


    --
    Karanbir Singh
    +44-207-0999389 | http://www.karan.org/ | twitter.com/kbsingh
    ICQ: 2522219 | Yahoo IM: z00dax | Gtalk: z00dax
    GnuPG Key : http://www.karan.org/publickey.asc
  • Lamar Owen at Aug 4, 2012 at 12:58 pm

    On Friday, August 03, 2012 12:03:01 PM Karanbir Singh wrote:
    Hi,
    On 08/03/2012 04:25 PM, Lamar Owen wrote:
    rpm -qa | while read line; do echo $line && rpm --setugids $line; done
    should handle ownerships. Then, reenable selinux in permissive mode, and set it to relabel on the next boot.
    maybe add --setperms as well
    Hmm, I thought by including that in my quoted text that it was implied that one should do both.... My bad. It was a long day....

    I'm not sure if both can be used on a single command line, since both --setperms and --setugids are implemented via popt aliases; I reserve the right to be wrong, of course.

    I also forgot to specify in my reply that these commands would only repair files owned by packages; any non-package-owned files in /var won't be helped by either --setperms or --setugids; but it will give the OP a good start.
  • Emmanuel Noobadmin at Aug 5, 2012 at 12:03 am

    On 8/3/12, Lamar Owen wrote:
    On Friday, August 03, 2012 06:24:46 AM Emmanuel Noobadmin wrote:
    In a moment of epic stupidity, having ran out of space on the root
    partition of a server due to /var chewing up the space, I added a
    separate drive for the purpose of mounting it as /var
    ...

    This sort of things pops up from time to time.... from a thread back in
    April.....
    Lesson learnt, never try to fix things I'm not familiar with when
    feeling pressured by relentless error messages, especially if nobody
    else is complaining yet.
    rpm -qa | while read line; do echo $line && rpm --setperms $line; done
    By extension:

    rpm -qa | while read line; do echo $line && rpm --setugids $line; done

    should handle ownerships. Then, reenable selinux in permissive mode, and
    set it to relabel on the next boot.
    Thanks for this tip, I'll try it and then see if there is anything
    else in audit log that needs attention.
  • John R Pierce at Aug 3, 2012 at 11:46 am

    On 08/03/12 3:24 AM, Emmanuel Noobadmin wrote:
    In a moment of epic stupidity, having ran out of space on the root
    partition of a server due to /var chewing up the space, I added a
    separate drive for the purpose of mounting it as /var

    To do so, I mounted the new drive as /var2, cp -R (in hindsight should
    had rsync to preserve attributes), deleted the original /var to free
    up space, edited fstab and rebooted... unsurprisingly to a fubar'd
    server.

    The thing is it still boots, I can get into single user mode but a
    full init get stuck at starting the syslogger. What is the best way to
    rescue the server now from my own stupidity and keeping all the
    existing configuration and data?
    if you had any database servers like postgresql or mysql, and their data
    files were in the default locations under /var, your databases are
    undoubtably corrupted, unless you stopped the DB server(s) before doing
    this copy.

    --
    john r pierce N 37, W 122
    santa cruz ca mid-left coast
  • Emmanuel Noobadmin at Aug 5, 2012 at 12:07 am

    On 8/3/12, John R Pierce wrote:
    if you had any database servers like postgresql or mysql, and their data
    files were in the default locations under /var, your databases are
    undoubtably corrupted, unless you stopped the DB server(s) before doing
    this copy.
    I think the fortunate thing is that everything else important on the
    server was running in their own VMs with their own LVM partitions. So
    luckily there doesn't seem to be anything important affected by my
    stupidity, the most important I wanted saved were the LVM
    configuration and VM configs which fortunately were in standalone XML
    files.

    I *think* I probably could had quickly reinstall a bare minimum C6 to
    fix it, but after realizing my epic foolishness with replacing /var, I
    didn't want to take any chances of anaconda wiping all the LVM
    partitions.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcentos @
categoriescentos
postedAug 3, '12 at 6:24a
activeAug 5, '12 at 12:07a
posts12
users6
websitecentos.org
irc#centos

People

Translate

site design / logo © 2022 Grokbase