Python threading example, creating Pinger.py

October 24, 2012 - 3 min read
python
DiffEdit
Old and not the way anymore

Lots have happened since I wrote this blog-post. It's now 2021..

Keeping this around for historical reasons, but please lookup another way of doing this now. This is probably the wrong way!

Update 18. Nov 2012: Cleaned up some comments about cores. To make it clear, this will only run on 1 core!

Threading in Python can be confusing in the beginning. Many examples out there are overly complicated so here is another example that I have tried to keep simple.

Here, I want a fast way to ping every host/ip in a list. As fast as we can, threaded, and then at last return a dict with two items. A list of dead nodes, and a list of nodes who answers on ping.

Example:

In [1]: from pinger import Pinger
In [2]: ping = Pinger()
In [3]: ping.thread_count = 8
In [4]: ping.hosts = ['10.0.0.1', '10.0.0.255', '10.0.0.100', 'google.com','nonexisting', '*not able to ping!*', '8.8.8.8']
In [5]: ping.start()
Out[5]: 
{'alive': ['10.0.0.255', '10.0.0.1', 'google.com', '8.8.8.8'],
 'dead': ['*not able to ping!*', 'nonexisting', '10.0.0.100']}

The example above will ping 8 hosts at the time and saving the results to the end. We are using 8 thread_count in this example. Which means that python will have 8 ping command running at the same time.

The whole source of the Pinger class looks like this, read the comments and you will see how it works:

#!/usr/bin/env python

import subprocess
import threading

class Pinger(object):
    status = {'alive': [], 'dead': []} # Populated while we are running
    hosts = [] # List of all hosts/ips in our input queue

    # How many ping process at the time.
    thread_count = 4

    # Lock object to keep track the threads in loops, where it can potentially be race conditions.
    lock = threading.Lock()

    def ping(self, ip):
        # Use the system ping command with count of 1 and wait time of 1.
        ret = subprocess.call(['ping', '-c', '1', '-W', '1', ip],
                                stdout=open('/dev/null', 'w'), stderr=open('/dev/null', 'w'))

        return ret == 0 # Return True if our ping command succeeds

    def pop_queue(self):
        ip = None

        self.lock.acquire() # Grab or wait+grab the lock.

        if self.hosts:
            ip = self.hosts.pop()

        self.lock.release() # Release the lock, so another thread could grab it.

        return ip

    def dequeue(self):
        while True:
            ip = self.pop_queue()

            if not ip:
                return None

            result = 'alive' if self.ping(ip) else 'dead'
            self.status[result].append(ip)

    def start(self):
        threads = []

        for i in range(self.thread_count):
            # Create self.thread_count number of threads that together will
            # cooperate removing every ip in the list. Each thread will do the
            # job as fast as it can.
            t = threading.Thread(target=self.dequeue)
            t.start()
            threads.append(t)

        # Wait until all the threads are done. .join() is blocking.
        [ t.join() for t in threads ]

        return self.status

if __name__ == '__main__':
    ping = Pinger()
    ping.thread_count = 8
    ping.hosts = [
        '10.0.0.1', '10.0.0.2', '10.0.0.3', '10.0.0.4', '10.0.0.0', '10.0.0.255', '10.0.0.100',
        'google.com', 'github.com', 'nonexisting', '127.0.1.2', '*not able to ping!*', '8.8.8.8'
        ]

    print ping.start()
Back to Blog