FAQ
Hi,

2 websites, hosted on 2 different CentOS 5.7 servers (one being very
new, about 3 weeks old) keeps loosing data - but it's more like it's
corrupted than being deleted.

For example, a photo would be uploaded last night and today when we
checked it, it doesn't show on the website. So we check if the file is
on the server, and exists but is 0KB in size. Last night it still
worked fine. The photo is 482Kb in size.

The first time this happened we thought it was due to a bug with
CentOS 5.7 + EXT4 + quotas (there's a bug open for this) and since the
server's console kept giving errors about possible data corruption we
thought it would be best if we move everything to a more stable
platform. So we brought a new server, setup CentOS 5.7 + ext3 + quotas
(which has been working fine on all our servers for a long time) and
moved the data across. A few days down the line and I still see this
happening.

I'm out of ideas and hope someone could shed some light on the matter.
I've checked some suggested search results, but couldn't find any
issues with the HDD according to SMART. The servers' both have 4GB RAM
and 8Core CPU's.Neither RAM, nor CPU usage is high. Both are setup
with RAID10 across 4 entrerprise HDD's, one server has software RAID
and the new one hardware RAID. So even when we changed the RAID
subsystem it still happens.



--
Kind Regards
Rudi Ahlers
SoftDux

Website: http://www.SoftDux.com
Technical Blog: http://Blog.SoftDux.com
Office: 087 805 9573
Cell: 082 554 7532

Search Discussions

  • John Doe at Dec 15, 2011 at 6:29 am
    From: Rudi Ahlers <Rudi at SoftDux.com>
    For example, a photo would be uploaded last night and today when we
    checked it, it doesn't show on the website. So we check if the file is
    on the server, and exists but is 0KB in size. Last night it still
    worked fine. The photo is 482Kb in size.
    Only the size changed to 0 or were some other stats changed too?
    Maybe try something like:
    ? inotifywait -m --format "%T %e %f" --timefmt "%D %T" -r /path/to/dir
    At least you would know at what time something happens...

    JD
  • Rudi Ahlers at Dec 15, 2011 at 6:22 pm

    On Thu, Dec 15, 2011 at 1:29 PM, John Doe wrote:
    From: Rudi Ahlers <Rudi at SoftDux.com>
    For example, a photo would be uploaded last night and today when we
    checked it, it doesn't show on the website. So we check if the file is
    on the server, and exists but is 0KB in size. Last night it still
    worked fine. The photo is 482Kb in size.
    Only the size changed to 0 or were some other stats changed too?
    Maybe try something like:
    ? inotifywait -m --format "%T %e %f" --timefmt "%D %T" -r /path/to/dir
    At least you would know at what time something happens...

    JD
    _______________________________________________


    John,

    Where do I get inotifywait ?


    yum what provides "*/inotifywait" didn't return anything


    root at mars:[/]$ yum whatprovides "*/inotifywait"
    Loaded plugins: fastestmirror
    Loading mirror speeds from cached hostfile
    * base: mirror.highspeedweb.net
    * extras: centos.omnispring.com
    * updates: mirrors.igsobe.com
    Excluding Packages in global exclude list
    Finished
    extras/filelists_db
    206 kB 00:00
    updates/filelists_db
    1.7 MB 00:00
    No Matches found


    --
    Kind Regards
    Rudi Ahlers
    SoftDux

    Website: http://www.SoftDux.com
    Technical Blog: http://Blog.SoftDux.com
    Office: 087 805 9573
    Cell: 082 554 7532
  • Ljubomir Ljubojevic at Dec 15, 2011 at 7:16 pm

    Vreme: 12/16/2011 12:22 AM, Rudi Ahlers pi?e:
    John,

    Where do I get inotifywait ?


    yum what provides "*/inotifywait" didn't return anything


    root at mars:[/]$ yum whatprovides "*/inotifywait"
    Loaded plugins: fastestmirror
    Loading mirror speeds from cached hostfile
    * base: mirror.highspeedweb.net
    * extras: centos.omnispring.com
    * updates: mirrors.igsobe.com
    Excluding Packages in global exclude list
    Finished
    extras/filelists_db
    206 kB 00:00
    updates/filelists_db
    1.7 MB 00:00
    No Matches found
    inotify-tools from EPEL?

    --

    Ljubomir Ljubojevic
    (Love is in the Air)
    PL Computers
    Serbia, Europe

    Google is the Mother, Google is the Father, and traceroute is your
    trusty Spiderman...
    StarOS, Mikrotik and CentOS/RHEL/Linux consultant
  • Rudi Ahlers at Dec 18, 2011 at 6:13 am
    Cool, thanx :)
    On Fri, Dec 16, 2011 at 2:16 AM, Ljubomir Ljubojevic wrote:
    Vreme: 12/16/2011 12:22 AM, Rudi Ahlers pi?e:
    John,

    Where do I get inotifywait ?


    yum what provides "*/inotifywait" didn't return anything


    root at mars:[/]$ yum whatprovides "*/inotifywait"
    Loaded plugins: fastestmirror
    Loading mirror speeds from cached hostfile
    ? * base: mirror.highspeedweb.net
    ? * extras: centos.omnispring.com
    ? * updates: mirrors.igsobe.com
    Excluding Packages in global exclude list
    Finished
    extras/filelists_db
    206 kB ? ? 00:00
    updates/filelists_db
    1.7 MB ? ? 00:00
    No Matches found
    inotify-tools from EPEL?

    --

    Ljubomir Ljubojevic
    (Love is in the Air)
    PL Computers
    Serbia, Europe

    Google is the Mother, Google is the Father, and traceroute is your
    trusty Spiderman...
    StarOS, Mikrotik and CentOS/RHEL/Linux consultant
    _______________________________________________
    CentOS mailing list
    CentOS at centos.org
    http://lists.centos.org/mailman/listinfo/centos


    --
    Kind Regards
    Rudi Ahlers
    SoftDux

    Website: http://www.SoftDux.com
    Technical Blog: http://Blog.SoftDux.com
    Office: 087 805 9573
    Cell: 082 554 7532
  • John Doe at Dec 16, 2011 at 4:34 am
    From: Rudi Ahlers <Rudi at SoftDux.com>
    Where do I get inotifywait ?
    yum what provides "*/inotifywait" didn't return anything
    Got inotify-tools from repoforge.

    JD
  • Anax at Dec 15, 2011 at 6:48 am
    Hi Rudi
    we once had a similar problem on a Web:

    This Web had this in particular that its home-page needed to be deleted
    daily and of course reinstalled immediately.
    Then, in a new version of the Web it did not need this delete/reinstall
    cycle any more, so the webadmin just removed the link to the delete
    script. He did not remove the delete-script itself from the Web.
    The effect was, that the home-page was still deleted at random times.

    What we then found out: We used an internal Search-Engine which crawled
    this web. And this Search-Engine had not forgotten the link to the
    delete-script, but invoked it at random times, whenever it crawled the
    particular Web.

    Your case: could it be something similar?

    suomi
    On 2011-12-15 11:28, Rudi Ahlers wrote:
    Hi,

    2 websites, hosted on 2 different CentOS 5.7 servers (one being very
    new, about 3 weeks old) keeps loosing data - but it's more like it's
    corrupted than being deleted.

    For example, a photo would be uploaded last night and today when we
    checked it, it doesn't show on the website. So we check if the file is
    on the server, and exists but is 0KB in size. Last night it still
    worked fine. The photo is 482Kb in size.

    The first time this happened we thought it was due to a bug with
    CentOS 5.7 + EXT4 + quotas (there's a bug open for this) and since the
    server's console kept giving errors about possible data corruption we
    thought it would be best if we move everything to a more stable
    platform. So we brought a new server, setup CentOS 5.7 + ext3 + quotas
    (which has been working fine on all our servers for a long time) and
    moved the data across. A few days down the line and I still see this
    happening.

    I'm out of ideas and hope someone could shed some light on the matter.
    I've checked some suggested search results, but couldn't find any
    issues with the HDD according to SMART. The servers' both have 4GB RAM
    and 8Core CPU's.Neither RAM, nor CPU usage is high. Both are setup
    with RAID10 across 4 entrerprise HDD's, one server has software RAID
    and the new one hardware RAID. So even when we changed the RAID
    subsystem it still happens.

  • Rudi Ahlers at Dec 15, 2011 at 4:52 pm

    On Thu, Dec 15, 2011 at 1:48 PM, anax wrote:
    Hi Rudi
    we once had a similar problem on a Web:

    This Web had this in particular that its home-page needed to be deleted
    daily and of course reinstalled immediately.
    Then, in a new version of the Web it did not need this delete/reinstall
    cycle any more, so the webadmin just removed the link to the delete
    script. He did not remove the delete-script itself from the Web.
    The effect was, that the home-page was still deleted at random times.

    What we then found out: We used an internal Search-Engine which crawled
    this web. And this Search-Engine had not forgotten the link to the
    delete-script, but invoked it at random times, whenever it crawled ?the
    particular Web.

    Your case: could it be something similar?

    suomi

    Thanx for the advice. We found a script which seems to have been
    modified somehow, probably hacker or a rootkit - don't yet know, but
    we're looking to see if we can trace it down.
    I don't want to reveal the code here due to it's uniqueness and I
    don't want to attrack further attension to the sites by the
    attack(er).


    --
    Kind Regards
    Rudi Ahlers
    SoftDux

    Website: http://www.SoftDux.com
    Technical Blog: http://Blog.SoftDux.com
    Office: 087 805 9573
    Cell: 082 554 7532

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcentos @
categoriescentos
postedDec 15, '11 at 5:28a
activeDec 18, '11 at 6:13a
posts8
users4
websitecentos.org
irc#centos

People

Translate

site design / logo © 2022 Grokbase