FAQ
I am heavily modifying (re-writing) some Perl scripts that are bundled
within the dialup_admin web interface for FreeRADIUS.

Eventually I'd like to bring the execution of a few system commands
(executing mysql binary and importing an SQL file into it) into Perl,
however, for now I just need it to work.

Essentially I don't have the time to rewrite that entire portion right
now, so I'd like to know the best way to execute the SQL system commands
and be 100% sure that it ran properly, and if not, I would be notified.

At this point, the command is done via backticks.

What I'd like to know is what would be the best way to quickly hack this
so that if it fails and neglects to insert every row, I could have the
app tell me (via email or whatever) so I don't have to do a manual check
every morning.

AFAICT, I've got the backtick option, system or eval.

Will system or eval be able to see that each row is inserted without
error when the mysql binary is called at system level? Or would it be
best if I just took the time now and did it with DBI immediately instead?

(This is only one instance where I'm dealing with this problem).

Steve

Search Discussions

  • Tom Phoenix at Dec 13, 2007 at 3:54 am

    On 12/12/07, Steve Bertrand wrote:

    I am heavily modifying (re-writing) some Perl scripts that are bundled
    within the dialup_admin web interface for FreeRADIUS.

    Eventually I'd like to bring the execution of a few system commands
    (executing mysql binary and importing an SQL file into it) into Perl,
    however, for now I just need it to work.
    Well, that's pragmatic. :-) But it's generally the best way: Find the
    answer first, improve upon it second.
    Essentially I don't have the time to rewrite that entire portion right
    now, so I'd like to know the best way to execute the SQL system commands
    and be 100% sure that it ran properly, and if not, I would be notified.
    There are few things that are 100% certain. But let's see what we can do.
    At this point, the command is done via backticks.

    What I'd like to know is what would be the best way to quickly hack this
    so that if it fails and neglects to insert every row, I could have the
    app tell me (via email or whatever) so I don't have to do a manual check
    every morning.

    AFAICT, I've got the backtick option, system or eval.
    You don't mean eval. You think you mean exec, but you don't mean that
    either, probably.
    Will system or eval be able to see that each row is inserted without
    error when the mysql binary is called at system level? Or would it be
    best if I just took the time now and did it with DBI immediately instead?
    I think you're looking for the program's exit status. Traditionally on
    Unix and many similar systems, the exit status is an integer, with 0
    meaning "normal exit" and anything else meaning that something went
    wrong. That's the value that's used by programs that run other
    programs (such as the 'make' system utility) and need to know when a
    step has failed. In Perl, you can access the exit status with the $?
    variable after running a command via system or backticks. Beware: If
    your command is run by the shell, you'll get the shell's exit status,
    which may not be the exit status you were looking for.

    You can also consider capturing the STDERR output of the external
    program. Since the exit status doesn't say much, most programs cough
    out some last words when something goes wrong. In fact, if the program
    outputs anything to STDERR, even if it exited with "success", the
    error text may be important news you'd want to treat as a failure. One
    common way to capture STDERR when running an external program is with
    IPC::Open3.

    I must admit to a philosophical problem with the idea of a program
    that sends email when something goes wrong. How, I ask myself, does
    such a program report that it cannot send mail? That 100% is looking
    more and more elusive....

    Hope this helps!

    --Tom Phoenix
    Stonehenge Perl Training
  • Steve Bertrand at Dec 13, 2007 at 5:21 am

    I am heavily modifying (re-writing) some Perl scripts that are bundled
    within the dialup_admin web interface for FreeRADIUS.

    Eventually I'd like to bring the execution of a few system commands
    (executing mysql binary and importing an SQL file into it) into Perl,
    however, for now I just need it to work.
    Well, that's pragmatic. :-)
    LOL, I was hoping to quickly justify my needs to an end :)
    But it's generally the best way: Find the
    answer first, improve upon it second.
    Generally, it's always my approach, but I'm sure as any other person
    here, we've all had to deal with instances where this is but a lucky
    circumstance. IANWNAPP (I am no where near a professional programmer).
    There are few things that are 100% certain.
    I was in a hurry when I wrote that, so just like many of the statistics
    we hear from politicians or some organizations, it came quickly out of
    the wrong spot.
    AFAICT, I've got the backtick option, system or eval.
    You don't mean eval. You think you mean exec, but you don't mean that
    either, probably.
    Being honest, I don't know what I meant. I truthfully didn't have time
    to read through the docs for all the comparisons (although I looked at
    eval in particular: perldoc -f eval which didn't seem proper for my
    specific execution issue (it dealt more with executing Perl code)).
    I think you're looking for the program's exit status. Traditionally on
    Unix and many similar systems, the exit status is an integer, with 0
    meaning "normal exit" and anything else meaning that something went
    wrong. That's the value that's used by programs that run other
    programs (such as the 'make' system utility) and need to know when a
    step has failed. In Perl, you can access the exit status with the $?
    variable after running a command via system or backticks. Beware: If
    your command is run by the shell, you'll get the shell's exit status,
    which may not be the exit status you were looking for.
    Wow, thank you for such a thoughtful and detailed explanation! I don't
    think I've ever run across the $? before (forgive me if it was noted in
    the ref, obj, mod book).

    Personally, I cringe whenever I see `$command` or system "$command"
    inside of anything I have to work on (albeit I have several scripts I've
    done this in myself, and still use). Everyone has their own way of
    coding, and someone's code always looks worse to someone else. After
    all, we are all only learning, no matter what we know...right?
    You can also consider capturing the STDERR output of the external
    program. Since the exit status doesn't say much, most programs cough
    out some last words when something goes wrong. In fact, if the program
    outputs anything to STDERR, even if it exited with "success", the
    error text may be important news you'd want to treat as a failure. One
    common way to capture STDERR when running an external program is with
    IPC::Open3.
    You are speaking of Unix (under shell) STDERR here, correct?

    Your understanding and diligence in providing useful, specific and very
    detailed responses amazes me. You get paid for this, right ;)
    I must admit to a philosophical problem with the idea of a program
    that sends email when something goes wrong. How, I ask myself, does
    such a program report that it cannot send mail? That 100% is looking
    more and more elusive....
    The email thing was an example. I won't get into the drawbacks on
    relying on SMTP as an error messaging protocol, as it's not documented
    in any of the RFC's I've read ;)

    100%, I'd rather not state where I got that figure. Given that I run a
    network and do much of the automation programming, I'd like to restate
    that number to perhaps something that any network op could agree upon
    regarding reliability of perfection...

    Thanks Tom,

    I think I'll check manually the next couple days for the success of my
    system routines, then early next week take the time to get them Perl
    lexical-ized.

    Steve
  • Rob Dixon at Dec 13, 2007 at 5:47 am

    Steve Bertrand wrote:

    Tom Phoenix wrote:
    >>
    [snip]
    I think you're looking for the program's exit status. Traditionally on
    Unix and many similar systems, the exit status is an integer, with 0
    meaning "normal exit" and anything else meaning that something went
    wrong. That's the value that's used by programs that run other
    programs (such as the 'make' system utility) and need to know when a
    step has failed. In Perl, you can access the exit status with the $?
    variable after running a command via system or backticks. Beware: If
    your command is run by the shell, you'll get the shell's exit status,
    which may not be the exit status you were looking for.
    Wow, thank you for such a thoughtful and detailed explanation! I don't
    think I've ever run across the $? before (forgive me if it was noted in
    the ref, obj, mod book).

    Personally, I cringe whenever I see `$command` or system "$command"
    inside of anything I have to work on (albeit I have several scripts I've
    done this in myself, and still use). Everyone has their own way of
    coding, and someone's code always looks worse to someone else. After
    all, we are all only learning, no matter what we know...right?
    You can also consider capturing the STDERR output of the external
    program. Since the exit status doesn't say much, most programs cough
    out some last words when something goes wrong. In fact, if the program
    outputs anything to STDERR, even if it exited with "success", the
    error text may be important news you'd want to treat as a failure. One
    common way to capture STDERR when running an external program is with
    IPC::Open3.
    You are speaking of Unix (under shell) STDERR here, correct?
    [snip]

    Tom was probably thinking of Unix, but same principles apply to other
    platforms.

    It's also worth noting that whatever application you're using to update
    the database may well output something useful to STDOUT and it may be a
    simple strategy to examine this output to determine the status of an
    operation. If you assign the result of a backtick string to a scalar
    variable

    my $status = `command`;

    then the entire output of the command will be stored as a single string.
    If you assign it to an array

    my @status = `command`;

    then the output will be stored as one line per array element.

    HTH,

    Rob
  • Steve Bertrand at Dec 13, 2007 at 5:57 am
    [snipped due to excessive content]
    ...
    [snip]

    Tom was probably thinking of Unix, but same principles apply to other
    platforms.

    It's also worth noting that whatever application you're using to update
    the database may well output something useful to STDOUT and it may be a
    simple strategy to examine this output to determine the status of an
    operation. If you assign the result of a backtick string to a scalar
    variable

    my $status = `command`;

    then the entire output of the command will be stored as a single string.
    If you assign it to an array

    my @status = `command`;

    then the output will be stored as one line per array element.
    The problem here is that I will be dealing with > 500k elements per
    cycle (day).

    I essentially need to know that ALL 500k elements were successful,
    otherwise, I need to know where it broke and where it stopped.

    This is a billing situation so it has to be accurate. I'd rather ensure
    (by doing manual double-entry checking) accuracy then having to go
    through potentially 500,000 lines in a log to see what ones went wrong,
    while trying to eliminate the ones that went 'right'.

    Or perhaps I understood you wrong.

    Tks for the reply,

    Steve
  • Rob Dixon at Dec 13, 2007 at 7:19 am

    Steve Bertrand wrote:
    [snipped due to excessive content]
    ...
    [snip]

    Tom was probably thinking of Unix, but same principles apply to other
    platforms.

    It's also worth noting that whatever application you're using to update
    the database may well output something useful to STDOUT and it may be a
    simple strategy to examine this output to determine the status of an
    operation. If you assign the result of a backtick string to a scalar
    variable

    my $status = `command`;

    then the entire output of the command will be stored as a single string.
    If you assign it to an array

    my @status = `command`;

    then the output will be stored as one line per array element.
    The problem here is that I will be dealing with > 500k elements per
    cycle (day).

    I essentially need to know that ALL 500k elements were successful,
    otherwise, I need to know where it broke and where it stopped.

    This is a billing situation so it has to be accurate. I'd rather ensure
    (by doing manual double-entry checking) accuracy then having to go
    through potentially 500,000 lines in a log to see what ones went wrong,
    while trying to eliminate the ones that went 'right'.

    Or perhaps I understood you wrong.
    I think you did. Unfortunately we are guessing in the dark about what
    this external program of yours does when when it fails (or indeed when
    it succeeds). Tom suggested checking the exit status or the STDERR
    output. I am saying that there's probably also something useful on
    STDOUT, which is easier to examine.

    But each of these is a way to check whether a single run of your
    application succeeded or failed. Look at what you see on the screen when
    you run the command manually. Is there anything in that text that your
    program could check for after each run? I would be surprised if there
    wasn't.

    I hope that's clearer.

    Rob
  • Steve Bertrand at Dec 13, 2007 at 1:51 pm
    [snip]
    I essentially need to know that ALL 500k elements were successful,
    otherwise, I need to know where it broke and where it stopped.

    This is a billing situation so it has to be accurate. I'd rather ensure
    (by doing manual double-entry checking) accuracy then having to go
    through potentially 500,000 lines in a log to see what ones went wrong,
    while trying to eliminate the ones that went 'right'.

    Or perhaps I understood you wrong.
    I think you did. Unfortunately we are guessing in the dark about what
    this external program of yours does when when it fails (or indeed when
    it succeeds). Tom suggested checking the exit status or the STDERR
    output. I am saying that there's probably also something useful on
    STDOUT, which is easier to examine.
    The external program is the MySQL client binary. I know the queries are
    clean (it will run manually with no problems).

    Here is what I'm doing:

    - taking all rows for $today from table1
    - deleting any possible erroneous rows from $today from table2 & table3
    - grouping by fieldX and fieldY and inserting into table2
    - updating (renaming) fieldY values based on it's content in table2.
    fieldY's content will be renamed into one of three names, based on the
    number range it contains (208.70% == 'dialup' 65.70 == 'dsl' etc)
    - taking all rows from table2, grouping them based on fieldY (name), and
    inserting into table3
    But each of these is a way to check whether a single run of your
    application succeeded or failed.
    Indeed. More importantly as stated above, I need to be sure that all
    records (or a random statistical subset) succeeded. mysql will succeed
    even if there were a large chunk of records that had erroneous entries
    in the field I am looking at with a WHERE clause.

    I know if I Perl-ize this, I'll be able to produce myself a report of
    any sort of statistics I want from the run 7 ways from Sunday.

    That is exactly what I plan to do instead of worrying about it.

    Thanks for the replies. All the thought that went into this got me
    thinking on other tracks for other code, but that's for another day.

    Steve

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupbeginners @
categoriesperl
postedDec 13, '07 at 1:45a
activeDec 13, '07 at 1:51p
posts7
users3
websiteperl.org

People

Translate

site design / logo © 2022 Grokbase