Laur IVAN

Initialise Python with GoCD

Posted by in IT, Tutorial

Following the deployment of a python-aware agent, we can start creating a pipeline. The first stage would be to: Clean up any previous data Create a virtual environment for python Install all dependencies from a requirements.txt file in our project Note: we assume the global pipeline settings and the Materials are properly set up (e.g. out project will be checked out in ./project). Since the above operations must be executed in order, I’ve set up a Setup stage with a job named virtualenv. The job has three tasks: rm virtualenv…read more

Create a Python-friendly GoCD Agent in Docker

Posted by in IT, Tutorial

I’ve got two requirements to make GoCD useful: Run both the server and agent(s) in Docker Make it python-friendly Fortunately, GoCD offers Docker images (and Dockerfiles) for both, so it’s just a matter of extending it:

Unfortunately, pip is not quite adequate, so we re-install it from source. In the Dockerfile we add:

Now we have a fully functional Docker file capable of building python applications and we can actually build the image. In the directory where we have the Dockerfile, execute:

Then, you can run it…read more

Handler for Global Logger in Python

Posted by in Software, Tutorial

While writing a command line tool, I’ve stumbled upon a strange situation: Some logs were written to a file while some others were printed to console. Quite annoying, particularly since every module I use has its own logger via logging.getLogger(__name__). After some investigation I found out you can access the global logger via logging.getLogger() (note the lack of parameters). So, my new setup is:

Simple. HTH, A little experiment: If you find this post and ad below useful, please check the ad out :-)

DRF: What is ‘source’ in Fields?

Posted by in Software, Tutorial

For a couple of days I’ve been trying to de-serialise a JSON to a model. My problem is the model and the JSON have different fields, mainly to keep the model pythonic and independent of a specific JSON source. So… I’ve been trying to use the source parameter for each Field in my serialiser to map to the JSON. E.g. if my Django Model field would be named timestamp and my JSON would be named time, I would have a serialiser Field like this:

THIS IS WRONG! It’s actually…read more

Timestamp to DateTime Serializer Field for DRF

Posted by in Software, Tutorial

I’m currently trying to deserialize JSON data to a Django model. This is quite straight forward for most numeric fields, but I have a DateTime field which is stored as a timestamp in the original JSON. At first, I thought I should read the timestamp as an integer and convert it through post-processing. Then, after reading more of the DRF docs, it became evident I needed a custom field. The code is below:

The idea is that to_representation converts a DateTime field into a timestamp value for serialisation and…read more

Run Celery 4.1.0 on Windows 10

Posted by in Software, Tutorial

TL;DR: Use set FORKED_BY_MULTIPROCESSING=1 on the worker script/shell When running Celery on Windows 10, I got the following error: ValueError: not enough values to unpack (expected 3, got 0) This is quite unfortunate as Windows is supported on “best effort”. A pull request has a change, but it’s not pretty to patch an installation. To get around, you can just define an environment variable:

I only defined it on the worker’s side. HTH, A little experiment: If you find this post and ad below useful, please check the ad…read more

Convert Timestamp to DateTime for Pandas DataFrame

Posted by in Software, Tutorial

To convert a pandas data frame value from unix timestamp to python datetime you need to use:

where: timestamp is the column containing the timestamp value unit=’s’ defines the unit of the timestamp (seconds in this case) You can actually replace the column altogether:

Unfortunately, the conversion is not really aware of the time zone so, if you want to e.g. create a django module, you’ll need to make the datetime object aware:

Moreover, if you want to change to another timezone, you’ll need to:

A…read more

Get all Proposal Names from the Participant Portal

Posted by in Software, Tutorial, Uncategorized

I’ve found out the participant portal for H2020 offers an API. This is quite neat as one can integrate with the data, e.g. by reading and listing the calls. I’ve written a proof of concept to list all calls from this and last years. First, we load the data in a JSON object:

Then, we extract the year:

Lastly, we iterate through all calls and extract the ones we need:

For the calls API, you can have access to the following information (fields): CallIdentifier FileName – e.g….read more

Inject Proxy Settings in Selenium/Chrome

Posted by in Software, Tutorial

I have the following situation: I’m behind a corporate proxy and I need to run a test accessing an external website. To do this, I’ve picked up chrome/chromium as driver of choice. My current solution implies an authentication via plugin. The plugin The chrome plugin is essentially 2 files: a manifest file a JS containing a listener Manifest The manifest file is:

The script Following script performs the proxy authentication:

,where the PROXY_* variables are the components of the proxy URL. Plugin The plugin is created by zipping…read more

0

Strip HTML tags in python

Posted by in Software, Tutorial

Note: This is a post transferred from Laurii for historical and consolidation purposes. A common problem I have to deal with quite often is to remove all HTML tags from a document. While this is easy for XML (well formatted etc.) and you could do it by hand with a regexp, HTML has a bit more problems. There are several solutions to get around this… Using a browser This is a bit overkill, but you could use a browser’s renderer to “display” the string content and then get it as…read more