Laur IVAN

Run Celery 4.1.0 on Windows 10

Posted by in Software, Tutorial

TL;DR: Use set FORKED_BY_MULTIPROCESSING=1 on the worker script/shell When running Celery on Windows 10, I got the following error: ValueError: not enough values to unpack (expected 3, got 0) This is quite unfortunate as Windows is supported on “best effort”. A pull request has a change, but it’s not pretty to patch an installation. To get around, you can just define an environment variable:

I only defined it on the worker’s side. HTH, A little experiment: If you find this post and ad below useful, please check the ad…read more

Convert Timestamp to DateTime for Pandas DataFrame

Posted by in Software, Tutorial

To convert a pandas data frame value from unix timestamp to python datetime you need to use:

where: timestamp is the column containing the timestamp value unit=’s’ defines the unit of the timestamp (seconds in this case) You can actually replace the column altogether:

Unfortunately, the conversion is not really aware of the time zone so, if you want to e.g. create a django module, you’ll need to make the datetime object aware:

Moreover, if you want to change to another timezone, you’ll need to:

A…read more

Get all Proposal Names from the Participant Portal

Posted by in Software, Tutorial, Uncategorized

I’ve found out the participant portal for H2020 offers an API. This is quite neat as one can integrate with the data, e.g. by reading and listing the calls. I’ve written a proof of concept to list all calls from this and last years. First, we load the data in a JSON object:

Then, we extract the year:

Lastly, we iterate through all calls and extract the ones we need:

For the calls API, you can have access to the following information (fields): CallIdentifier FileName – e.g….read more

Inject Proxy Settings in Selenium/Chrome

Posted by in Software, Tutorial

I have the following situation: I’m behind a corporate proxy and I need to run a test accessing an external website. To do this, I’ve picked up chrome/chromium as driver of choice. My current solution implies an authentication via plugin. The plugin The chrome plugin is essentially 2 files: a manifest file a JS containing a listener Manifest The manifest file is:

The script Following script performs the proxy authentication:

,where the PROXY_* variables are the components of the proxy URL. Plugin The plugin is created by zipping…read more

0

Strip HTML tags in python

Posted by in Software, Tutorial

Note: This is a post transferred from Laurii for historical and consolidation purposes. A common problem I have to deal with quite often is to remove all HTML tags from a document. While this is easy for XML (well formatted etc.) and you could do it by hand with a regexp, HTML has a bit more problems. There are several solutions to get around this… Using a browser This is a bit overkill, but you could use a browser’s renderer to “display” the string content and then get it as…read more

Cascade get value from dictionary in Python

Posted by in Software, Tutorial

Several times I’ve been faced with the problem: Get a value from a dictionary. If the key is not present, then get from another key. Normally, the implementation would go like this:

Or, even worse, with try-catch statements. That’s fine if the sequence is fixed, but what if you want to have some flexibility? Well, my solution is like this:

Pretty, isn’t it? Note: This is a post transferred from Laurii for historical and consolidation purposes. A little experiment: If you find this post and ad below useful,…read more

Change a django password manually

Posted by in Software

The other day I’ve been confronted with a strange situation: I forgot the admin password for a django application. I was playing with installing my own pypi repository, I’ve configured and deployed a test application and, after a couple of weeks, I forgot the password. I tried several times bu I knew that I used something silly and… unfortunate. Even worse, the user name I thought I used was not the one I had set up for the test. Oops :) After looking online for options I’ve found the following:…read more

Python and Outlook – An example

Posted by in Software

TL;DR: Install win32 extensions (e.g. using this to put it in your virtual env). Read some code below (there are folders with stuff, messages have CapitalizedPropertyNames). Profit. The other day I needed to perform some statistics on an Outlook mailbox. In particular, use the frequency per day to identify peaks of activity. I could get an excel sheet going with VBA but it’s more fun in python. So, here it goes (mind you, it’s a bit long due to the included code). Preparation First, I’ve defined a class to help…read more

Add XML to PostgreSQL from Python

Posted by in Software

One of the projects I worked on was to import a large number of XML files in a PostgreSQL database (as XML files). I chose python to do it. Here are the steps: Database The data source is the SCOPUS database which has a silly number of entries (approx 20,000,000), most containing 2 XML files (article and citations). The publications with no citations don’t have the citations.xml file. Therefore, initial database has a single table named scpous_data with the following fields: id – string of 20 characters. This is the…read more

Rot Algorithm in Python

Posted by in Software

A while ago I wanted to send a bunch of emails to building companies in order to get some quotes and/or information. Unfortunately, the website in question (pagesdor.be) doesn’t provide emails in clear text to allow me copy/paste. Instead it uses a Rotating cypher algorithm (I guess it must be simple enough to be able to be decrypted by a browser). After a couple of emails, where the process of sending through the website costed me some time I figured that it would be easier to see and implement the…read more