Laur IVAN

Redis on Windows – Save RDB but Not Able to Persist

TL;DR: My windows service wasn’t installed properly. Change the config file to point the DB to an accessible directory and reinstall the service. When trying to build a perioduc Celery task, I got the following error with Redis on Windows (running as a windows service): redis.exceptions.ResponseError: MISCONF Redis is configured to save RDB snapshots, but is currently not able to persist on disk. Commands that may modify the data set are disabled. Please check Redis logs for details about the error. This is quite unfortunate because the error doesn’t show…read more

Sep.22

Run Celery 4.1.0 on Windows 10

Posted by laur in Software, Tutorial

TL;DR: Use set FORKED_BY_MULTIPROCESSING=1 on the worker script/shell When running Celery on Windows 10, I got the following error: ValueError: not enough values to unpack (expected 3, got 0) This is quite unfortunate as Windows is supported on “best effort”. A pull request has a change, but it’s not pretty to patch an installation. To get around, you can just define an environment variable:

set FORKED_BY_MULTIPROCESSING=1

1	set FORKED_BY_MULTIPROCESSING=1

I only defined it on the worker’s side. HTH, A little experiment: If you find this post and ad below useful, please check the ad…read more

Sep.21

Use Docker Behind Proxy on Windows 8

Posted by laur in Software, Tutorial

The other day I had to install Docker at work, aka behind a corporate proxy (on Windows 8). While the docker installer is stand-alone and all is good, dealing with the VM is a different matter… The problem popped up when I tried to create a Django image and pip install failed miserably. To get around, you need to create a docker machine:

docker-machine create -d virtualbox \
    --engine-env HTTP_PROXY=http://example.com:8080 \
    --engine-env HTTPS_PROXY=https://example.com:8080 \
    --engine-env NO_PROXY=example2.com \
    default

docker-machine create -d virtualbox \

--engine-env HTTP_PROXY=http://example.com:8080 \

--engine-env HTTPS_PROXY=https://example.com:8080 \

--engine-env NO_PROXY=example2.com \

default

You also need to pass in the proxy as a build argument to build:

docker build --build-arg https_proxy="http://example.com:8080"

1	docker build --build-arg https_proxy="http://example.com:8080"

You can also pass the build argument to docker-compose but, in…read more

Sep.18

Vega 56 and ETH – Keep Your Cool

Posted by laur in Hardware

TL;DR: Keep your Vega cool if you want high hash rates! Recently, I’ve got a AMD Vega RX 56, to see for myself how good it is on mining. If you ignore the power consumption, it’s quicker than my RX580 cards. I’m getting 33+MH/s consistently: At the beginning of the session I even got 36MH/s, but, as temperature rose, the hash rate dropped to 33+ stable. Unfortunately, my setup is less than ideal: No decent linux support and I haven’t got the time to tweak a vanilla Ubuntu install with…read more

Sep.12

Check Computer Started without Admin Provileges

Posted by laur in IT, Tutorial

TL;DR: open the Event viewer, select the Windows Logs > System and Filter Current Log to the desired interval. Or you can scroll to the earliest entry of the day you need (more irritating because the event window doesn’t refresh when scrolling). The other day I needed to find out when an user logged in on a computer. Unfortunately, this info is only available with admin privileges, which the user did not have. So, knowing the user was the one that started the computer on the day, the next best…read more

Sep.02

Ethereum – Fix Slow Sync

Posted by laur in Tutorial

To fix the slow sync problem in the ethereum miner, you need to: backup your account(s) and application data (from the eth wallet if you’re using it) remove the chaindata content via a. delete the directory (easiest/quickest way) a. use the geth removedb command to do it properly reload the database via:

geth --syncmode "fast" --cache 1024 console

1 2	geth --syncmode "fast" --cache 1024 console

Now, if your database is not in the default path, (e.g. not in %APPDATA%\Ethereum on Windows), you’ll need to tell geth where it is. You do this with the –datadir path/to/data/directory. So, my command looks like:

geth --datadir "D:\Coins\Ethereum"  --syncmode "fast" --cache 1024 console

1	geth --datadir "D:\Coins\Ethereum" --syncmode "fast" --cache 1024 console

Sep.01

Mining Rig Build Log – 1

Posted by laur in Hardware, Tutorial

I’ve decided to join the recent trend of mining cryptocoins with GPUs. First, I got a GPU (an EVGA 1070) to complement my 970 and stuck it in my desktop. This is nowhere near profitable because I’d have to block an I7 machine to mine on 2 cards, and also get annoyed because I can’t use it. also, rather than my previous attempt few years back using a couple of cards stuck in a desktop PC and having all family complaining of the noise, I’ve decided to build a proper…read more

Aug.08

Convert Timestamp to DateTime for Pandas DataFrame

Posted by laur in Software, Tutorial

To convert a pandas data frame value from unix timestamp to python datetime you need to use:

pd.to_datetime(df['timestamp'], unit='s')

1	pd.to_datetime(df['timestamp'], unit='s')

where: timestamp is the column containing the timestamp value unit=’s’ defines the unit of the timestamp (seconds in this case) You can actually replace the column altogether:

df['timestamp'] = pd.to_datetime(df['timestamp'], unit='s')

1	df['timestamp'] = pd.to_datetime(df['timestamp'], unit='s')

Unfortunately, the conversion is not really aware of the time zone so, if you want to e.g. create a django module, you’ll need to make the datetime object aware:

df.timestamp = df.timestamp.dt.tz_localize('UTC')

1	df.timestamp = df.timestamp.dt.tz_localize('UTC')

Moreover, if you want to change to another timezone, you’ll need to:

df.rimestamp.dt.tz_localize('UTC').dt.tz_convert('Europe/Brussels')

1	df.rimestamp.dt.tz_localize('UTC').dt.tz_convert('Europe/Brussels')

A…read more

Aug.07

Save Pandas DataFrame as Django Model

Posted by laur in Software, Tutorial

TL;DR: use bulk_upload coupled with a comprehension to speed up loading. Initially, I started to convert the data frame to a Model object row by row and save it. While easy, it’s also arguably the slowest method to load records, because the save() call performs a commit. Then I looked up how to save pandas data frames and I found out quite a few ways… Manual commit The first step would be to tweak the transaction to manually commit like described here:

from django.db import transaction

@transaction.commit_manually
def save(df):
    ...
    for item in df.to_dict('records'):
        entry = Entry(**item)
        entry.save()
    transaction.commit()

from django.db import transaction

@transaction.commit_manually

def save(df):

...

for item in df.to_dict('records'):

entry = Entry(**item)

entry.save()

transaction.commit()

SQL Other solution would be to use…read more

Jul.08

Styling HTML to PDF with iText 2.1.7

Posted by laur in Software, Tutorial

There are various solutions for producing PDFs from a HTML file, some with excellent results (e.g. ones based on the chrome renderer). Unfortunately, they usually mean adding more dependencies to an existing system. To keep things in check, we’ve chosen to use iText to generate PDFs. Following the previous post, we need to enhance the generated PDF with some styles. Now, the older versions of iText can’t use CSS directly, so we’ll need to produce the styles by hand. iText has a StyleSheet object which can load tag styles via…read more