Laur IVAN

Redis on Windows – Save RDB but Not Able to Persist

TL;DR: My windows service wasn’t installed properly. Change the config file to point the DB to an accessible directory and reinstall the service. When trying to build a perioduc Celery task, I got the following error with Redis on Windows (running as a windows service): redis.exceptions.ResponseError: MISCONF Redis is configured to save RDB snapshots, but is currently not able to persist on disk. Commands that may modify the data set are disabled. Please check Redis logs for details about the error. This is quite unfortunate because the error doesn’t show…read more

Sep.22

Run Celery 4.1.0 on Windows 10

Posted by laur in Software, Tutorial

TL;DR: Use set FORKED_BY_MULTIPROCESSING=1 on the worker script/shell When running Celery on Windows 10, I got the following error: ValueError: not enough values to unpack (expected 3, got 0) This is quite unfortunate as Windows is supported on “best effort”. A pull request has a change, but it’s not pretty to patch an installation. To get around, you can just define an environment variable:

set FORKED_BY_MULTIPROCESSING=1

1	set FORKED_BY_MULTIPROCESSING=1

I only defined it on the worker’s side. HTH, A little experiment: If you find this post and ad below useful, please check the ad…read more

Sep.21

Use Docker Behind Proxy on Windows 8

Posted by laur in Software, Tutorial

The other day I had to install Docker at work, aka behind a corporate proxy (on Windows 8). While the docker installer is stand-alone and all is good, dealing with the VM is a different matter… The problem popped up when I tried to create a Django image and pip install failed miserably. To get around, you need to create a docker machine:

docker-machine create -d virtualbox \
    --engine-env HTTP_PROXY=http://example.com:8080 \
    --engine-env HTTPS_PROXY=https://example.com:8080 \
    --engine-env NO_PROXY=example2.com \
    default

docker-machine create -d virtualbox \

--engine-env HTTP_PROXY=http://example.com:8080 \

--engine-env HTTPS_PROXY=https://example.com:8080 \

--engine-env NO_PROXY=example2.com \

default

You also need to pass in the proxy as a build argument to build:

docker build --build-arg https_proxy="http://example.com:8080"

1	docker build --build-arg https_proxy="http://example.com:8080"

You can also pass the build argument to docker-compose but, in…read more

Sep.12

Check Computer Started without Admin Provileges

Posted by laur in IT, Tutorial

TL;DR: open the Event viewer, select the Windows Logs > System and Filter Current Log to the desired interval. Or you can scroll to the earliest entry of the day you need (more irritating because the event window doesn’t refresh when scrolling). The other day I needed to find out when an user logged in on a computer. Unfortunately, this info is only available with admin privileges, which the user did not have. So, knowing the user was the one that started the computer on the day, the next best…read more

Sep.02

Ethereum – Fix Slow Sync

Posted by laur in Tutorial

To fix the slow sync problem in the ethereum miner, you need to: backup your account(s) and application data (from the eth wallet if you’re using it) remove the chaindata content via a. delete the directory (easiest/quickest way) a. use the geth removedb command to do it properly reload the database via:

geth --syncmode "fast" --cache 1024 console

1 2	geth --syncmode "fast" --cache 1024 console

Now, if your database is not in the default path, (e.g. not in %APPDATA%\Ethereum on Windows), you’ll need to tell geth where it is. You do this with the –datadir path/to/data/directory. So, my command looks like:

geth --datadir "D:\Coins\Ethereum"  --syncmode "fast" --cache 1024 console

1	geth --datadir "D:\Coins\Ethereum" --syncmode "fast" --cache 1024 console

Sep.01

Mining Rig Build Log – 1

Posted by laur in Hardware, Tutorial

I’ve decided to join the recent trend of mining cryptocoins with GPUs. First, I got a GPU (an EVGA 1070) to complement my 970 and stuck it in my desktop. This is nowhere near profitable because I’d have to block an I7 machine to mine on 2 cards, and also get annoyed because I can’t use it. also, rather than my previous attempt few years back using a couple of cards stuck in a desktop PC and having all family complaining of the noise, I’ve decided to build a proper…read more

Aug.08

Convert Timestamp to DateTime for Pandas DataFrame

Posted by laur in Software, Tutorial

To convert a pandas data frame value from unix timestamp to python datetime you need to use:

pd.to_datetime(df['timestamp'], unit='s')

1	pd.to_datetime(df['timestamp'], unit='s')

where: timestamp is the column containing the timestamp value unit=’s’ defines the unit of the timestamp (seconds in this case) You can actually replace the column altogether:

df['timestamp'] = pd.to_datetime(df['timestamp'], unit='s')

1	df['timestamp'] = pd.to_datetime(df['timestamp'], unit='s')

Unfortunately, the conversion is not really aware of the time zone so, if you want to e.g. create a django module, you’ll need to make the datetime object aware:

df.timestamp = df.timestamp.dt.tz_localize('UTC')

1	df.timestamp = df.timestamp.dt.tz_localize('UTC')

Moreover, if you want to change to another timezone, you’ll need to:

df.rimestamp.dt.tz_localize('UTC').dt.tz_convert('Europe/Brussels')

1	df.rimestamp.dt.tz_localize('UTC').dt.tz_convert('Europe/Brussels')

A…read more

Aug.07

Save Pandas DataFrame as Django Model

Posted by laur in Software, Tutorial

TL;DR: use bulk_upload coupled with a comprehension to speed up loading. Initially, I started to convert the data frame to a Model object row by row and save it. While easy, it’s also arguably the slowest method to load records, because the save() call performs a commit. Then I looked up how to save pandas data frames and I found out quite a few ways… Manual commit The first step would be to tweak the transaction to manually commit like described here:

from django.db import transaction

@transaction.commit_manually
def save(df):
    ...
    for item in df.to_dict('records'):
        entry = Entry(**item)
        entry.save()
    transaction.commit()

from django.db import transaction

@transaction.commit_manually

def save(df):

...

for item in df.to_dict('records'):

entry = Entry(**item)

entry.save()

transaction.commit()

SQL Other solution would be to use…read more

Jul.08

Styling HTML to PDF with iText 2.1.7

Posted by laur in Software, Tutorial

There are various solutions for producing PDFs from a HTML file, some with excellent results (e.g. ones based on the chrome renderer). Unfortunately, they usually mean adding more dependencies to an existing system. To keep things in check, we’ve chosen to use iText to generate PDFs. Following the previous post, we need to enhance the generated PDF with some styles. Now, the older versions of iText can’t use CSS directly, so we’ll need to produce the styles by hand. iText has a StyleSheet object which can load tag styles via…read more

Jul.07

HTML to PDF with iText 2.1.7

Posted by laur in Software, Tutorial

Sometimes you need to use an older version of a product die to e.g. licensing changes. It’s my case with iText. While the new version looks fantastic, their changes (and removal of e.g. RTF output) are deal breakers for the feature set we need to maintain. Therefore, 2.1.7 is the most recent version we can use. Heh. In maven, this would be:

<dependency>
    <groupId>com.lowagie</groupId>
    <artifactId>itext</artifactId>
    <version>2.1.7</version>
</dependency>

<groupId>com.lowagie</groupId>

<artifactId>itext</artifactId>

</dependency>

The string resulted from the previous article can be printed out fairly easily. First, we create a document and a writer:

// THE PDF
// Doc's begin
final Document document = new Document();
File file = new File("/some/path/out.pdf");
DocWriter docWriter = PdfWriter.getInstance(document, new FileOutputStream(file));

document.open();

// THE PDF

// Doc's begin

final Document document = new Document();

File file = new File("/some/path/out.pdf");

DocWriter docWriter = PdfWriter.getInstance(document, new FileOutputStream(file));

document.open();

We write the document:

HTMLWorker worker = new HTMLWorker(document);
worker.parse(new StringReader(output));

1 2	HTMLWorker worker = new HTMLWorker(document); worker.parse(new StringReader(output));