From Python, I needed to find a process that was performing SSH tunneling on port 8080 and kill it.

The following works in Bash:

ps aux | grep [s]sh.*:8080 | awk '{print $2}' | xargs kill -9

The grep [s]sh trick ensures that the grep command itself won't make it through to awk.

Here's what I came up with in Python using psutil:

def kill_port_forwarding(host_port):
    ssh_args = ["-f", "-N", "-L", "{0}:localhost:{0}".format(host_port)]
    for process in psutil.process_iter():
        try:
            if process.name().endswith('ssh'):
       

I needed to import some plugin code written in Python from a directory whose path isn't known until runtime. Further, I needed a class object that was a subclass of the plugin base class.

from somewhere import PluginBase

class SomePlugin(PluginBase):
    def f1(self): ...
    def f2(self): ...

You can use the imp module to actually load the module from impl_dir. Note that impl_dir needs to be tem­porar­i­ly prepended to sys.path. Then you can find the plugin subclass using dir and issubclass.

import os, imp

def import_class(implementation_filename, base_class):
    impl_dir, impl_filename = os.path.split(implementation_filename)
    module_name, _ = os.path.splitext(impl_filename)

    

[Pre­vi­ous­ly published at the now defunct MetaBrite Dev Blog.]

I needed to sort a Python dictionary by value today, rather than by key. I found it confusing, so I'll share what I learned.

Assume the following dictionary, where each value is a tuple of (ID, score). How do we sort by score; i.e., the second item in the value tuple? (For the purposes of this discussion, ignore the meaning of the dic­tio­nary's key.)

>>> some_dict = dict(a=(123, 0.7), b=(372, 0.2), e=(456, 0.85), d=(148, 0.23), c=(502, 0.1))
>>> some_dict
{'a': (123, 0.7), 'c': (502, 0.1), 'b': (372, 0.2), 'e': (456, 0.85), 'd': (148, 0.23)}

Python dic­tio­nar­ies are inherently unsorted, unless you …continue.

I figured out why I saw the following error every time I ran Nose:

======================================================================
ERROR: Failure: TypeError (type() takes 1 or 3 arguments)
----------------------------------------------------------------------
Traceback (most recent call last):
  File ".../lib/python2.7/site-packages/nose-1.3.7-py2.7.egg/nose/loader.py", line 523, in makeTest
    return self._makeTest(obj, parent)
  File ".../lib/python2.7/site-packages/nose-1.3.7-py2.7.egg/nose/loader.py", line 582, in _makeTest
    return MethodTestCase(obj)
  File ".../lib/python2.7/site-packages/nose-1.3.7-py2.7.egg/nose/case.py", line 345, in __init__
    self.inst = self.cls()
TypeError: type() takes 1 or 3 arguments

It turns out that one module was importing a class called TestApi which had a class­method called run_in­te­gra­tion_tests. The module itself had no tests; it just declared a class called TestO­b­fus­cat­ed­Mix­in, which used some …continue.

Generators decouple iteration from the code that uses the results of the iteration.

—David Beazley, Generators

[Pre­vi­ous­ly published at the now defunct MetaBrite Dev Blog.]

Python generators have a variety of uses. One such is to lazily evaluate sequences. Another is for coroutines. Yet another is to re­cur­sive­ly traverse a tree or a graph, yielding an iterable sequence.

Consider this simple tree of nodes:

node_tree = Node(
    'a', [
        Node('b', [
            Node('e', [
                Node('g')
 

Have you ever attempted a relative import in a Python script?

$ ./foo/bar/script.py some parameters
Traceback (most recent call last):
  File "foo/bar/script.py", line 16, in <module>
    from .quux import find_vcs
ValueError: Attempted relative import in non-package

I prefer to use absolute imports to minimize ambiguity and confusion, and most of my Python modules begin with:

from __future__ import absolute_import, unicode_literals, print_function

(Using uni­code_lit­er­als and print­_­func­tion makes porting to Python 3 easier.)

I recently read the accepted answer to Python relative imports for the billionth time and the solution to the above ValueError occurred to me: Use python -m package instead:

$ python -m foo.bar.script some parameters

(Assuming that …continue.

We deploy on Ubuntu 14.04, the most recent Long Term Support release. It comes with Python 2.7.6, but we need Python 2.7.9+ to get the some important SSL fixes and work with a recent version of the Requests library.

Felix Krull maintains a Personal Package Archive for Python 2.7 Updates, which makes it straight­for­ward to upgrade to Python 2.7.11 on supported versions of Ubuntu.

sudo apt-add-repository ppa:fkrull/deadsnakes-python2.7
sudo apt-get update
sudo apt-get install python2.7 python2.7-dev

Be sure not to use Felix Krull's other Python PPA by mistake. I did that on a col­league's machine yesterday. In our attempts to figure out why we still had Python 2.7.6, we managed to mess up the …continue.

When I declare a derived class that inherits from both a base class and some mixins, I am always tempted to write:

class Derived(Base, Mixin1, Mixin2):
    "Some class"

My reasoning is that Derived is a Base with some Mixin1 and Mixin2 goodness sprinkled on. Generally, that's fine. The exception is when I want one of the mixins to override a method or attribute that's defined in Base. Because the Method Resolution Order is left-to-right, then Base’s im­ple­men­ta­tion will always be found first.

To get the desired behavior of the mixin overriding the base, Base should always appear last in the in­her­i­tance list.

from __future__ import print_function

class Useful(object):
    

[Pre­vi­ous­ly published at the now defunct MetaBrite Dev Blog.]

RFC 1738 allows passwords in URLs, in the form <scheme>://<username>:<password>@<host>:<port>/<url-path>. Although passwords are deprecated by RFC 3986 and other newer RFCs, it's oc­ca­sion­al­ly useful. Several important packages in the Python world allow such URLs, including SQLAlchemy ('post­gresql://scott:tiger@localhost:5432/my­data­base') and Celery ('amqp://guest:guest@localhost:5672//'). It's also useful to be able to log such URLs without exposing the password.

Python 2 has urlparse.urlparse (known as urllib.parse.urlparse in Python 3 and six.moves.url­lib_­parse.urlparse in the Six com­pat­i­bil­i­ty library) to split a URL into six components, scheme, netloc, path, parameters, query, and fragment. The netloc cor­re­sponds to <user>:<password>@<host>:<port>.

Un­for­tu­nate­ly, neither Python 2 nor 3's urlparse properly handle the userinfo (username + optional password in the netloc), as …continue.

New MetaBrite Dev Blog post, Ob­fus­cat­ing Passwords in URLs in Python.