saving threads with wget

news about changes and fixes to the board, and a place to post any technical queries.

Moderator: staff

Post Reply
User avatar
Smeagol
board admin emeritus
board admin emeritus
Posts: 11534
Joined: Wed Nov 12, 2003 4:20 am
Location: UK
Contact:

saving threads with wget

Post by Smeagol » Fri Jul 13, 2007 10:20 pm

deb (or nils, or any other unix geeks)

Is there a reason other than my incompetence why I'm unable to save threads using wget?

I've been trying
wget --post-data="username=MyName&password=MyPasswd" --http-user=MyName --http-passwd=MyPasswd ThreadURL

wget --post-data="username=MyName&password=MyPasswd" --http-user=MyName --http-passwd=MyPasswd ThreadURL
And I've tried using the cookie options. All I can download is the login page though. It saved a cookie file like this but I keep getting redirected to the login page even though the cookie file implies I'm already logged in

I've also tried telling wget to use the cookie file my browser is using.

Am I doing something wrong or is the board set up so that wget actually doesn't work?
Act in such a way as to make yourself feel capable and effective

The change starts now.

If in doubt, don't

Boogie Man
one of us
one of us
Posts: 8203
Joined: Mon Nov 03, 2003 8:51 am
Location: Australia

Post by Boogie Man » Mon Jul 30, 2007 6:51 am

Am I doing something wrong or is the board set up so that wget actually doesn't work?
i'm useless with nix, but i wouldn't be surprised if that's the case. isn't it a fav amongst anon. trolls+bots, who wget all the images on a website at least million times, in the hopes of pulling off a dos attack?
Image

User avatar
mark
one of us
one of us
Posts: 22
Joined: Wed Nov 27, 2002 2:17 am

Post by mark » Thu Aug 16, 2007 4:11 pm

The authentication is cookie based, so you need to post to the login page + store the cookie that is returned by that. You might find something like httrack easier to use.

User avatar
Smeagol
board admin emeritus
board admin emeritus
Posts: 11534
Joined: Wed Nov 12, 2003 4:20 am
Location: UK
Contact:

Post by Smeagol » Sun Aug 19, 2007 10:25 pm

I've now tried with python, and my conclusion is that threads longer than one page get directed to the login page. I was using cookie authentication.

I shall try httrack.
Act in such a way as to make yourself feel capable and effective

The change starts now.

If in doubt, don't

User avatar
Smeagol
board admin emeritus
board admin emeritus
Posts: 11534
Joined: Wed Nov 12, 2003 4:20 am
Location: UK
Contact:

Post by Smeagol » Sun Aug 19, 2007 11:08 pm

Hmph, no luck with httrack. It isn't handling the authentication so I'm getting a page not found error.
Act in such a way as to make yourself feel capable and effective

The change starts now.

If in doubt, don't

Post Reply

Who is online

Users browsing this forum: No registered users and 18 guests