Wget -save-cookies=./session-cookies-$USER \ session-cookies-$USER | grep _sid | cut -d$'\011' -f7`Įcho "Login $USER -> $PHPBB_URL SID=$SID" Wget -save-cookies=./session-cookies-$USER $PHPBB_URL/ucp.php?mode=login -O - 1> /dev/null 2> /dev/null This project looks promising but didn't quite work for me: Here some added info to lots of noise but a start if you need to login.
So this is going to burn a lot of time and bandwidth. In particular memberlist.php and viewtopic.php where p= is specified can create thousands of files!ĭue to this bug in wget it will still download an astounding number of those useless files - esepcially viewtopic.php?p= ones - before simply deleting them. I have also excluded some other pages that lead to a lot of cruft being saved. (Perhaps there's a way to force the recursive wget to start from index.php - I don't know). Except for one squirreled away somewhere - which links to a plain index.php which then continues with no sid= parameter. They seem to get added automatically by the index page, and then get attached to all the links in a virus-like fashion. I wanted to strip out those pesky session id things (sid=blahblahblah). Here's the command I'm using: wget -k -m -E -p -np -R memberlist.php*,faq.php*,viewtopic.php*p=*,posting.php*,search.php*,ucp.php*,viewonline.php*,*sid*,*view=print*,*start=0* -o log.txt I need to download these files for every day of the year for multiple years, so not having a suitable download script is currently slowing/blocking my research.I am doing this right now. I've also tried using a less restrictive encoding 'latin-1', but the result of this is simply 'None'. This implies multiple types of encoding in the result. "UnicodeDecodeError: 'charmap' codec can't decode byte 0x90 in position 5216: character maps to " When using 'Windows-1252' to decode, the result is: checking the encoding using chardet which returns encoding: 'Windows-1252'. ', '-H', 'Authorization: Bearer XXX-TOKEN-XXX']
This is successful, but seems to return data that isn't decodable: UnicodeDecodeError: 'utf-8' codec can't decode byte 0x89 in position 0: invalid start byte > return code('utf-8') if isinstance(result, bytes) else result When using the Python download script from here:
0 kommentar(er)
