Re: [unix, console] : something like wget but with forms + authentification

From: Enrique Perez-Terron (enrio_at_online.no)
Date: 09/24/05

  • Next message: saam: "Re: find files of 0 size and give mail" 
    Date: Sat, 24 Sep 2005 00:32:55 +0200

    On Fri, 23 Sep 2005 19:10:18 +0200, Thomas Baruchel <baruchel@127.0.0.1>
    wrote:

    > Hi,
    >
    > the very single moment of the day when I have to launch X is for having a
    > quick look at a single web page at :
    > https://www.cabourse-2.credit-agricole.fr/g1/ssl/html/index.html
    > with a form at :
    > https://www.cabourse-2.credit-agricole.fr/g1/ssl/jsp/identification/0-identifierClient.id

    To understand what you mean, I clicked the first link above and found a
    "welcome" page with a single link on it, pointing to the second url above.

    The second URL returns a form, that is correct. It is a login form with two
    input fields (name and password) and a submit button. There is also a
    checkbox to have the browser remember the name and password. (Or so says
    the page, I have not checked the code.)

    I don't know if the first page contains code to check a cookie, and bypass
    the login form, or what.

    What happens once you are past the login screen? Is your goal just to
    download the response after submitting your name and password, so you
    can later pick up that response in any browser, or postprocess it to
    extract some data fields on the page?

    Try to be a little more precise about what you want.

    > Do you think I have any chance to manage that from console or shell with
    > either a console browser or a wget-like tool ?

    My first thought is "maybe, or maybe not, but don't waste time
    trying, use Perl or another programming language with similar
    functionality."

    Is this the best forum for this question? I am asking honestly,
    because I don't know offhand a better one. Yet I would think
    that there ought to be. If the OP does not know Perl or Ruby
    or Python, he might not feel it natural to ask in such groups,
    but this will easily become sufficiently complicated that the
    effort to learn the basic syntax of any of these languages
    becomes minor in comparison. (I am supposing that you do have
    some basic programming skills, since you aim at doing some shell
    programming.)

    > I tried with elinks which handles well SSL but not javascript ;
    > I also tried with links, but I couldn't manage it.

    I guess you would have to use Expect to drive links.
    How did you go about it with links?

    There are often some difficulties to overcome when driving an interactive
    program. First, it is too easy to end up with a deadlock, your code
    waiting for the next prompt from the other program (in order to decide
    what to send) and the other program waiting for your input. A typical
    complication here is that the program's prompt does not end in a newline
    so your shell code will not process what it has got, wanting its input
    to consist of complete lines.

    Another problem with driving programs is that the driven program
    often refuses to talk to pipes, wanting a terminal, so it can turn
    off the echoing while you type the password.

    These two problems are the reason for using Expect.

    But sometimes you can succeed by just sending the right "responses"
    with or without a "sleep" here and there, and just dicarding any
    output from the driven program until after the last command has been
    sent.

    In any case, since you are dealing with https, you need all I/O to
    go through some SSL-aware tool.

    > Besides, I would like rather a shell-based tool.
    >
    > Since I don't know anything about HTTP and forms, I didn't even has a
    > look at the source code of the page, but any hint will be welcome.

    How much would you be willing to pay a consultant for a complete solution?

    -Enrique

  • Next message: saam: "Re: find files of 0 size and give mail"