Laur IVAN

Mirror a Confluence Site

TL;DR: wget has a mode capable of saving cookies. The other day I thought it’d be interesting to have a local copy of a confluence wiki, to avoid hitting the internet every time I needed a howto. So, I’ve tried curl first and failed (probably my fault really). So, after a few tries, I managed to get a workflow in place. First, we need to log in onto the site:

./wget -qO- --keep-session-cookies \
  --save-cookies cookies.txt \
  --post-data 'os_username=<user>&os_password=<user>' \
  https://site.com/confluence/login.action

./wget -qO- --keep-session-cookies \

--save-cookies cookies.txt \

--post-data 'os_username=<user>&os_password=<user>' \

https://site.com/confluence/login.action

Here, you need to replace the user name and password with something meaningful. Please note that text must be…read more

Jun.10

Mirror a Website with WGet

Posted by laur in Software

TL;DR: if you want to mirror a site where you need to authenticate, you need to do it in two steps. In the process of testing my app, I needed a mirror of the website I’m using. Since the website has authentication (via a form), the mirroring process has 2 parts. First part is to log in and get the cookies:

wget \
  --keep-session-cookies \
  --save-cookies cookies.txt \
  --post-data "login[LOGIN]&amp;password=[PASSWORD]&amp;module=admission&amp;controller=login&amp;action=logindo&amp;auth_act=1" \
  https://europa.eu/epso/application/base/index.cfm

wget \

--keep-session-cookies \

--save-cookies cookies.txt \

--post-data "login[LOGIN]&password=[PASSWORD]&module=admission&controller=login&action=logindo&auth_act=1" \

https://europa.eu/epso/application/base/index.cfm

Now, the cookies are saved in the cookies.txt file. Note that you need to specify –keep-session-cookies. Otherwise, the file will be empty. The second part is to actually…read more