Laur IVAN

Mirror a Confluence Site

TL;DR: wget has a mode capable of saving cookies. The other day I thought it’d be interesting to have a local copy of a confluence wiki, to avoid hitting the internet every time I needed a howto. So, I’ve tried curl first and failed (probably my fault really). So, after a few tries, I managed to get a workflow in place. First, we need to log in onto the site:

./wget -qO- --keep-session-cookies \
  --save-cookies cookies.txt \
  --post-data 'os_username=<user>&os_password=<user>' \
  https://site.com/confluence/login.action

./wget -qO- --keep-session-cookies \

--save-cookies cookies.txt \

--post-data 'os_username=<user>&os_password=<user>' \

https://site.com/confluence/login.action

Here, you need to replace the user name and password with something meaningful. Please note that text must be…read more

Oct.18

Jenkins Pipeline Deployment Variants

Posted by laur in IT, Tutorial

This post outlined a list of steps in a pipeline. When building actual pipelines, I’ve noticed the deployment process is more than just deployment. I’ve identified several variants: orchestration deployment (e.g. helios deployment) file copy like this git update for e.g. assets repositories This might be useful if you’re building a mechanism based on a shared library. HTH, A little experiment: If you find this post and ad below useful, please check the ad out :-)

Oct.12

Rundeck – Speeding Up Even More the Cleaning Process

Posted by laur in IT, Tutorial

As I was running the script, I’ve noticed it ran very slow. I have left it for a few days to run (700K executions), I’ve noticed the speed increased linearly over time. As Rundeck has both DB and files to store its data, it looked to me that it was the file system which was the slow part. Sure enough, when I tried to access the directory where the execution data was stored, it was slow – 1.4M files would take a while to get organised :) To short out…read more

Oct.09

Delete Executions in Rundeck – Wrapping it all Up

Posted by laur in IT, Software, Tutorial

The previous posts have built the components to allow us to get projects, jobs, executions and remove executions from a Rundeck instance via its API. Now, we can just wrap them up :) The list of packages we need is:

import time
import datetime
import sys
import logging

import time

import datetime

import sys

import logging

For convenience, I’ve declared a bunch of constants we can use throughout the project:

EXPIRE_DAYS = 30
SIZE=100
MAXCOUNT=1000
OFFSET=0
BATCH_SIZE=20
EXPIRE_MILISECONDS = EXPIRE_DAYS * 24 * 60 * 60 * 1000

log = logging.getLogger(__name__)

EXPIRE_DAYS = 30

SIZE=100

MAXCOUNT=1000

OFFSET=0

BATCH_SIZE=20

EXPIRE_MILISECONDS = EXPIRE_DAYS * 24 * 60 * 60 * 1000

log = logging.getLogger(__name__)

As we already have all the helper functions defined, we only need to wrap them in a convenient execution command:

def main():
    import argparse

    # The parser
    parser = argparse.ArgumentParser()
    parser.add_argument("host")
    parser.add_argument("port")
    parser.add_argument("api_key")

    args = parser.parse_args()

    projects = getProjectNames(listProjects(args.host, args.port, args.api_key))
    deleted = 0
    for project in projects:
        log.info('project:\t'+project)
        jobids = getJobIDs(listJobsForProject(args.host, args.port, args.api_key, project))
        for jobid in jobids:
            log.info('\tjobid:\t'+jobid)
            executions = getAllExecutionsForAJob(args.host, args.port, 
                args.api_key, jobid, MAXCOUNT)

            # Save the list of timestamps in a file
            with open(jobid + '.csv', 'wt') as f:
                for execution in executions:
                    f.write("%s,%s\n" % (execution,executions[execution]))
            deleted += checkDeletion(args.host, args.port, args.api_key, 
                executions)
        print('I have deleted :%d: jobs for %s' % (deleted,project))

def main():

import argparse

# The parser

parser = argparse.ArgumentParser()

parser.add_argument("host")

parser.add_argument("port")

parser.add_argument("api_key")

args = parser.parse_args()

projects = getProjectNames(listProjects(args.host, args.port, args.api_key))

deleted = 0

for project in projects:

log.info('project:\t'+project)

jobids = getJobIDs(listJobsForProject(args.host, args.port, args.api_key, project))

for jobid in jobids:

log.info('\tjobid:\t'+jobid)

executions = getAllExecutionsForAJob(args.host, args.port,

args.api_key, jobid, MAXCOUNT)

# Save the list of timestamps in a file

with open(jobid + '.csv', 'wt') as f:

for execution in executions:

f.write("%s,%s\n" % (execution,executions[execution]))

deleted += checkDeletion(args.host, args.port, args.api_key,

executions)

print('I have deleted :%d: jobs for %s' % (deleted,project))

The logis is fairly simple: Get the list of projects For each project,…read more

Oct.08

Remove Executions in Rundeck

Posted by laur in IT, Tutorial

TL;DR: You have to get the executions’ list and remove the appropriate ones using hte API. Once we’ve managed to get the list of job executions, we can use the API to remove the relevant executions. The command is rather simple:

def deleteExecutions(server, port, api_key, execution_ids):
    url =  server +':'+port+'/api/12/executions/delete?ids=' + 
        ','.join(execution_ids)
    headers = {'Content-Length': '0', 'X-RunDeck-Auth-Token': api_key }
    requests.post(url, headers=headers, verify=False)

def deleteExecutions(server, port, api_key, execution_ids):

url = server +':'+port+'/api/12/executions/delete?ids=' +

','.join(execution_ids)

headers = {'Content-Length': '0', 'X-RunDeck-Auth-Token': api_key }

requests.post(url, headers=headers, verify=False)

This method requires: server – the Rundeck’s server address port – the port on which we can connect to the Rundeck server api_key – the API key execution_ids – an array of valid execution IDs The function will try to remove all execution IDs in one go, since…read more

Oct.02

Get Job Executions in Rundeck

Posted by laur in IT, Tutorial

Following the previous articles, we are now in position to query job executions. The query is done via API as well. As a job can have an arbitrary number of executions, it’s advisable to use pagination. The code follows the same lines as the one in the previous entries:

# API call to get the list of the executions for a Job.
SIZE = 100
def getExecutionsForAJob(server, port, api_key, job_id, page=0, size=SIZE):
    url =  server +':'+port + \
        '/api/1/job/'+job_id + \
        '/executions?max='+('%d' % size) + \
        '&offset='+('%d' % (page * size))
    log.info(url)
    headers = {'Content-Type': 'application/json','X-RunDeck-Auth-Token': api_key }
    r = requests.get(url, headers=headers, verify=False)
    return r.text.encode('utf-8')

# API call to get the list of the executions for a Job.

SIZE = 100

def getExecutionsForAJob(server, port, api_key, job_id, page=0, size=SIZE):

url = server +':'+port + \

'/api/1/job/'+job_id + \

'/executions?max='+('%d' % size) + \

'&offset='+('%d' % (page * size))

log.info(url)

headers = {'Content-Type': 'application/json','X-RunDeck-Auth-Token': api_key }

r = requests.get(url, headers=headers, verify=False)

return r.text.encode('utf-8')

This function builds the URL for the call and returns an XML string. Its parameters are: server – the rundeck server port – the rundeck server’s port api_key – the api key for rundeck job_id – one of the job…read more

Oct.02

Map Subdomains to Docker Containers with Synology

Posted by laur in IT, Tutorial

The other day I got fed up with having to insert diskstation.local:12345 to point towards an installed service on my NAS. I also thought that my SO has difficulties remembering all the ports (I do too) and cleaning up the browser cache doesn’t help either. So, I thought: Wouldn’t it be nicer if instead of disksation.local:32400 I could just type plex.laurii.lan? Yes! Yes, it would! :) So I started my quest to make things happen. Use case My se case is the following: As a local user, I want to…read more

Sep.06

List Rundeck Jobs for a Project

Posted by laur in IT, Software, Tutorial

Now that we Rundeck have a list of project names, we might want to see which jobs were created for each project. To do this, we need: The Rundeck instance’s address (server, port) The API key (api_key) A project’s name (project_name) First, we need to get the list of jobs for a project:

import requests

def listJobsForProject(server, port, api_key, project_mame):
    url =  server +':'+port+'/api/1/jobs'
    payload = { 'project':  project_mame }
    headers = {'Content-Type': 'application/json',
        'X-RunDeck-Auth-Token': api_key }
    r = requests.get(url, params=payload, headers=headers, verify=False)
    return r.text.encode('utf-8')

import requests

def listJobsForProject(server, port, api_key, project_mame):

url = server +':'+port+'/api/1/jobs'

payload = { 'project': project_mame }

headers = {'Content-Type': 'application/json',

'X-RunDeck-Auth-Token': api_key }

r = requests.get(url, params=payload, headers=headers, verify=False)

return r.text.encode('utf-8')

The result is also an XML, just like the one for the projects. Therefore, we need to parse it:

import xml.etree.ElementTree as ET

def getJobIDs(jobsinfo_xml):
    job_ids = []
    log.info('Get Job IDs')
    root = ET.fromstring(jobsinfo_xml)
    for jobs in root:
        for job in jobs:
            job_ids.append(job.attrib['id'])
    return job_ids

import xml.etree.ElementTree as ET

def getJobIDs(jobsinfo_xml):

job_ids = []

log.info('Get Job IDs')

root = ET.fromstring(jobsinfo_xml)

for jobs in root:

for job in jobs:

job_ids.append(job.attrib['id'])

return job_ids

At the end of this exercise, we’ll have a list of job names (IDs). HTH, A…read more

Sep.06

List Rundeck Projects

Posted by laur in IT, Software, Tutorial

The other day I found out that Rundeck has an API which allows it to be controlled remotely by scripts. I’ve decided to write a script to list all the projects defined in a rundeck instance. Prerequisites Before you start, you need: The instance’s URL (server and port) An api key for the user Make sure the user has enough credentials to list the projects The code I’ve picked up Python as language for the task just because I’m more familiar with it in this context. Therefore, another dependency is…read more

Sep.06

Remove Large Files in Git

Posted by laur in Software, Tutorial

TL;DR: BFG is your friend. java -jar bfg.jar -b 50M myrepo-bfg.git for example The other day I’ve committed a rather large file by mistake (generated movie of the git commit history). This was a bit annoying because all other developers would have suddenly to check out a 1G file :( with no LFS support… As I took the opportunity to clean up the repository also, I’ve noticed the size of the cloned reop was rather huge! So, I was presented with several options Create a new repo with the cleaned…read more