Implementing two Graph traversal algorithms in Python: Depth First Search and Breadth First Search

Posted on January 24, 2015 in Learning • Tagged with Programming, Learning, University • 2 min read

Depth First Search and Breadth First Search

I am right in front of a ton of exams and I need to learn about algorithms and data structures. When I read about pseudocode of Graph traversal algorithms, I thought:
Why not actually implement them in a real programming language? So I did so and now you can study my code now here. I guess this problem was solved a thousand times before, but I learnt something and I hope my approach has some uniqueness to it.

Additionlay, you can also generate a topological order after you traversed the whole Graph, which is a nice little extra.

If you want the most recent version of the code, you can visit its own Github repo here.

Well, here's the code. Just download and run it like this: python graph_traversal.py

# -*- coding: utf-8 -*-

__author__ = 'Nikolai Tschacher'
__version__ = '0.1'
__contact__ = 'admin@incolumitas.com'


import time
from collections import deque

"""
This is just a little representation of two basic graph traversal methods.

    - Depth-First-Search
    - Breadth-First-Search

It's by no means meant to be fast or performant. Rather it is for educational
purposes and to understand it better for myself.
"""


class Node(object):
    """Represents a node."""

    def …

Continue reading

Very good program to record audio and desktop on Linux!

Posted on January 18, 2015 in Linux • Tagged with Linux, Software • 2 min read

First post in the new year!

Hey

Happy new year to all of you and let 2015 be a succesful year for us all!

My New Year's resolution is to write at least two blog posts every month and try to get my scraping service on scrapeulous.com up and running!

Good program to record the desktop/audio on linux

But what I really wanted to share today is an awesome way to record your desktop with audio on Linux. I tried my luck several times with VLC, but it's a freaking pain in the ass to use. Furthermore, VLC will probably never be able to capture the desktop with audio (See this stackoverflow thread for more info).

But I just found an wonderful alternative (one could almost assume that I am advertisting, which is not the case, I swear!):

http://wiki.ubuntuusers.de/recordMyDesktop

If you want to visit the home page of the program, click here. Although the home page is very ugly and the program is not longer in active development, it just works like a charm. On Ubuntu you may install it like this:

sudo apt-get install recordmydesktop

Then go to a directory very you want …


Continue reading

Scraping and Extracting Links from any major Search Engine like Google, Yandex, Baidu, Bing and Duckduckgo

Posted on November 12, 2014 in Meta • Tagged with Scraping, Baidu, Extracting, Google, Programming, Python, Searchengine, Bing, Meta • 7 min read

Prelude

It's been quite a while since I worked on my projects. But recently I had some motivation and energy left, which is quite nice considering my full time university week and a programming job besides.

I have a little project on GitHub that I worked on every now and again in the last year or so. Recently it got a little bit bigger (I have 115 github stars now, would've never imagined that I ever achieve this) and I receive up to 2 mails with job offers every week (Sorry if I cannot accept any request :( ).

But unfortunately my progress with this project is not as good as I want it to be (that's probably a quite common feeling under us programmers). It's not a problem of missing ideas and features that I want to implement, the hard part is to extend the project without blowing legacy code up. GoogleScraper has grown evolutionary and I am waisting a lot of time to understand my old code. Mostly it's much better to just erease whole modules and reimplement things completely anew. This is essentially what I made with the parsing module.

Parsing SERP pages with many search engines

So I …


Continue reading

Using the Python cryptography module with custom passwords

Posted on October 19, 2014 in Cryptography • Tagged with Cryptography, Programming, Uncategorized • 1 min read

Hey all

I recently discovered a quite cute crypto module for Python. It is divided in two logical security layers. The first (Fernet) can be used by cryptology unaware programmers in a way that makes it unlikely to introduce any security flaws. The seconds layer (called Hazmat) allows access to all kinds of cryptographical primitives, such as HMACS and asymmetric encryption functions.

The Problem

Normally you don't want to use primitives, because it is tricky to do correct (event for advanced programmers). But unfortunately the secure and simple API functionality Fernet:

>>> from cryptography.fernet import Fernet
>>> key = Fernet.generate_key()
>>> f = Fernet(key)
>>> token = f.encrypt(b"my deep dark secret")
>>> token
'...'
>>> f.decrypt(token)
'my deep dark secret

suffers from the huge inconvenience that you need to store (or imagine:remember!) a 32 byte key in order to decrypt the tokens that Fernet outputs.
It would be much more convenient to just pass a password to Fernet which in turn makes a 32 byte, Base 64 encoded encryption token out of it. Of course your own
password is much less secure then 32 bytes from os.urandom(32), but at least it is somehow usable.

So I came up with …


Continue reading

Beautiful, beautiful python

Posted on July 11, 2014 in Uncategorized • Tagged with Uncategorized • 1 min read

Hey

After a day of programming I went home to program a little bit, trying to find a way to implement some tests for my GoogleScraper project, which lacked focus for a long time. I needed to have some test data, in my case some words to search for with the above mentioned scraper, and once more I realized how powerful Python (or any programming language) is. This silly little code comes in handy, if you need some random words for some testing purposes:

import requests
import re

def random_words(n=50, wordlength=range(10, 15)):
   """Read a random english wiki article and extract some words.

   Arguments:
   n -- The number of words to return. Returns all found ones, if n is more than we were able to found.
   KeywordArguments:
   wordlength -- A range that forces the words to have a specific length.
   """
   valid_words = re.compile(r'[a-zA-Z]{{{},{}}}'.format(wordlength.start, wordlength.stop))
   found = list(set(valid_words.findall(requests.get('http://en.wikipedia.org/wiki/Special:Random').text)))
   try:
       return found[:n]
   except IndexError:
       return found

print(random_words(200, range(5, 6)))
print(random_words(77, range(16, 26))

Lichess.org chess bot!

Posted on April 23, 2014 in Uncategorized • Tagged with Uncategorized, Programming, Chess • 4 min read

22.05.2014: Updated the bot, should work better now

Hi everyone!

I was in a coding mood during Easter and decided to write a small chess bot with selenium and stockfish engine to cheat a bit on lichess.org.

I think the code is pretty self explanatory and I won't discuss it in depth here. You can tweak the config, the comments should explain what the parameters do.

The config is in the beginning of the code, so modify it there. You should maybe modify it to use your username and password. Make sure that you download stockfish and install it. Then supply the correct path in the 'stockfish_binary' parameter.

As always: Have fun!

Some open issues:

  • Sometimes the last move fails because the bot won't to start a new game before it can checkmate
  • Promoting doesn't work yet :/

Here is the code:

__author__ = 'nikolai'
__date__ = 'Easter 2014'

config = {
    'username' : 'probably_a_spider', # the login username
    'password' : 'somepwd', # the login password
    'stockfish_binary' : '/home/nikolai/PycharmProjects/LichessBot/stockfish-dd-src/src/stockfish', # the path to your local stockfish binary
    #Set to true if the bot should play forever
    'pwn_forever' : True, # if the bot should play endlessly
    'min_per_side' : 1, # how long each player may play …

Continue reading