The last time Hackerfall tried to access this page, it returned a not found error. A cached version of the page is below, or clickhereto continue anyway

Postmodern Error Handling in Python 3.6

I’ll be the first person to admit I have no idea what postmodernism actually means, but it sounds cool for an article on error-handling (or rather how to prevent them from happening in the first place), and I would argue that the facilities (post?)modern Python provides us for doing so are pretty damn cool.

Recently, an acquaintance of mine posed this question on a message board we both participate in:

okay nerds what do you call a union-type with three states?

so I was writing a Maybe implementation (in python) to deal with a processing pipeline, but then it turned into an Either, and then it turned into something with three states

basically there’s either 1) the final value, 2) no value but a known error occurred, 3) no value but an unknown error occurred

the final states could be: 1) hey it succeeded, here’s the object’s id 2) no value but your json blob was missing a required attribute (or some other known error), or 3) we encountered an error that the programmer didn’t forsee

To which another strapping young gentleman replied:

if it were Rust you could enum your errors

Well, truth be told, I feel like I probably know about as little about fp as I do about postmodernism, but I thought this was an interesting question and I know that Python’s standard library does, actually, provide us Enums ( -) so I figured I would take a crack at it.

To start, let’s quickly talk about what an Enum (enumeration) is for those of us who have yet to encounter them.

At a very basic level, I would define an Enum as a particular set of choices (or states).

Let’s say, for example, that you lost a bet, and now have a set of options to choose from as punishment.

Those options are

Those are the only 2 possibilities; you don’t get any other choices. We could model this idea using an Enum.

from enum import Enum

class SlapBet(Enum):
    TEN_SLAPS = 1 # get slapped ten times right now
    FIVE_SLAPS = 2 # fear slaps possibly for eternity
    
smart_choice = SlapBet.TEN_SLAPS

stupid_choice = SlapBet.FIVE_SLAPS

smart_choice # notice the pretty repr

smart_choice == SlapBet.TEN_SLAPS

smart_choice == stupid_choice
<SlapBet.TEN_SLAPS: 1>
True
False

So, why would we do this? Why go through the trouble of using an Enum when we could functionally do the same thing with something like integers? Well, the first reason, and the most important one, in my opinion, is readability.

Another reason is that Enum instances can hold values we may want to use.

Let’s say you have a function that has an optional parameter.

def connection(person_1, person_2, relationship='knows'):
    """Return a string that represents the relationship between two people."""
    
    # this sexy f"" string format is part of the new Python 3.6
    # hotness. An article for another day.
    
    return f"{person_1} {relationship} {person_2}"

connection("Ron", "Harry", relationship="is best friends with")
'Ron is best friends with Harry'

Now let’s say we want to constrain the possible types of relationships that people can have.

Maybe we want to do this to prevent errors or to simply prevent weirdness like the following.

connection("Ron", "Harry", relationship="is dating Hermoine but secretly wants")
'Ron is dating Hermoine but secretly wants Harry'

Enums to the rescue!

from enum import Enum

class Relationship(Enum):
    knows = 'knows'
    likes = 'likes'
    loves = 'loves'
    detests = 'detests'
    bff = "is best friends with"

def connection(person_1, person_2, relationship=Relationship.likes):
    """Return a string that represents the relationship between two people."""
    
    # notice we use the .value of the Relationship instance to get the string
    
    return f"{person_1} {relationship.value} {person_2}"

connection("Ron", "Harry", relationship=Relationship.bff)
'Ron is best friends with Harry'

Interlude

Before tackle the original question that started us on this journey, let’s quickly talk about a new incredibly useful and important feature since Python 3.5 that has been vastly improved in Python 3.6 - type annotations.

In our previous example, there would be nothing to prevent someone from doing the following:

evil = lambda: 'power'
evil.value = 'corrupts'

connection("Ron", "Harry", relationship=evil)
'Ron corrupts Harry'

That’s obviously not how we intended our function to be used, but part of what makes Python so powerful, its dynamic nature, is what allows such nefarious behavior. If only there were a way to save ourselves and people who use our code from making such mistakes…

Behold!

from enum import Enum

class Relationship(Enum):
    knows: str = 'knows'
    likes: str = 'likes'
    loves: str = 'loves'
    detests: str = 'detests'
    bff: str = "is best friends with"
    
def connection(person_1: str,
               person_2: str,
               relationship: Relationship=Relationship.knows) -> str:
    """Return a string that represents the relationship between two people."""
    
    # notice we use the .value of the Relationship instance to get the string
    
    return f"{person_1} {relationship.value} {person_2}"

connection("Ron", "Harry", relationship=Relationship.bff)
'Ron is best friends with Harry'

“Huh?”, you say. “It looks to me like things just got more verbosified. I like making up words.”

Cool word, and you would be right, things did get more wordy. Moreover, adding all those annotations won’t prevent someone from doing what we mentioned before,

evil = lambda: 'power'
evil.value = 'corrupts'

connection("Ron", "Harry", relationship=evil)
# this still works _

unless, that is, we run that code using - mypy

Mypy allows you to add type annotations and enforce them prior to running your program, so the only way the above function would run is if the relationship parameter was of type Relationship when called. Amazing!

Now, anyone reading our code would know exactly the types of things that could be passed as parameters to functions, and mypy will help us to enforce the type annotations we set.

That makes our code much more legible and provides us and people using our code some nice sanity checks - knowing certain kinds of human errors will be caught prior to our code being run.

Now, back to our original question about modeling 3 possible conditions.

  1. Task ran without error. Data returned.
  2. A known error occurred during task execution. No data returned.
  3. A catastrophic runtime error occurred. No data returned

So, before we look at the code, it’s cool to note that as of Python 3.6 we now have typed NamedTuples that we can declare using a new syntax. We had typed namedtuples in Python 3.5 but I think the new syntax for declaring them is much nicer.

# old way
Employee = NamedTuple('Employee', [('name', str), ('id', int)])

# new sauce
class Employee(NamedTuple):
    name : str
    id   : int

Note, an Optional describes something that can be of a certain type, or None.

# declare a variable, that can be either an int or None
possible_integer: Optional[int]

You’ll see the use of optional values a lot in functional programming. In fact, the following pattern of combining tuples and optional values is very similar to the way we handle errors in Go. Optionals are also ubiquitous in Swift - not only in the context of error-handling.

Optionals are a really handy concept, and now Python has them as well as a lot of the other type-checking goodness of other languages through mypy and the standard library.

"""
Caveat:

The following is not necessarily the 
most robust way of handling exceptions, IMO.
Python allows you to write custom exceptions 
that one can `raise from` others for good reason.
This is just meant as a way to think about how we 
would model the initial scenario described.
"""
from typing import NamedTuple, Optional
import requests
import logging
import json
import enum


class ApiInteraction(enum.Enum):
    """The 3 possible states we can expect when interacting with the API."""
    SUCCESS = 1
    ERROR = 2
    FAILURE = 3


class ApiResponse(NamedTuple):
    """
    This is sort of a really dumbed-down version of an HTTP response,
    if you think of it in terms of status codes and response bodies.
    """
    status: ApiInteraction
    payload: Optional[dict]


        
def hit_endpoint(url: str) -> ApiResponse:
    """
    1. Send an http request to a url
    2. Parse the json response as a dictionary
    3. Return an ApiResponse object
    """
    
    
    try:
        
        response = requests.get(url) # step 1
        payload = response.json() # step 2
        
    except json.decoder.JSONDecodeError as e:
        
        # something went wrong in step 2; we knew this might happen
        
        # log a simple error message
        
        logging.error(f'could not decode json from {url}')
        
        # log the full traceback at a lower level
        
        logging.info(e, exc_info=True)
        
        # since we anticipated this error, make the
        # ApiResponse.status an ERROR as opposed to a failure
        
        return ApiResponse(ApiInteraction.ERROR, None)

    # 'except Exception' is seen as an anti-pattern by many but
    # this is just a trivial example. Another article for another time.
    except Exception as e:
        
        # something went wrong in step 1 or 2 that
        # we couldn't anticipate
        
        # log the exception with the traceback
        
        logging.error(f"Something bad happened trying to reach {url}")
        logging.info(e, exc_info=True)
        
        # Since something catastrophic happened that
        # we didn't anticipate i.e. (DivideByBananaError)
        # we set the ApiResponse.status to FAILURE
        
        return ApiResponse(ApiInteraction.FAILURE, None)
    
    else:
        
        # Everything worked as planned! No errors!
        
        return ApiResponse(ApiInteraction.SUCCESS, payload)


# Python is awesome. We can either use the function by itself
# or use it as a constructor for our ApiResponse class 
# by doing thefollowing:


ApiResponse.from_url = hit_endpoint
def test_endpoint_response():

    url = 'http://httpbin.org/headers'
    response = ApiResponse.from_url(url)
    assert response.status == ApiInteraction.SUCCESS
    assert response.status == hit_endpoint(url).status # our function and constructor work the same!
    assert response.payload is not None


    url = 'http://twitter.com'
    response = ApiResponse.from_url(url)
    assert response.status == ApiInteraction.ERROR
    assert response.status == hit_endpoint(url).status
    assert response.payload is None


    url = 'foo'
    response = ApiResponse.from_url(url)
    assert response.status == ApiInteraction.FAILURE
    assert response.status == hit_endpoint(url).status
    assert response.payload is None
        
    
test_endpoint_response()
ERROR:root:could not decode json from http://twitter.com
ERROR:root:could not decode json from http://twitter.com
ERROR:root:Something bad happened trying to reach foo
ERROR:root:Something bad happened trying to reach foo

In collusion

I’m really excited about the way Python is evolving as a language, ecosystem, and community. I think that recent developments in typing add a lot to the expressiveness of the language and provide us with some really useful guarantees when leveraged through mypy.

Using Enums and NamedTuples this way not only provides us with some welcome sanity checks, I think it makes the code much more readable and even more testable in certain scenarios. I also like the idea of having relatively simple data structures that we can create arbitrary constructors for outside of their class definitions, similar to other languages, although I can see why some might disagree on that point. I reserve the right to change my mind.

Guido Van Rossum, our benevolent dictator, himself, has said innovations in Python’s type system is what has him most excited about the language moving forward, according to his latest interview with on Michael Kennedy’s TalkPython.fm

The more I read about these features and use them, the more I understand why.

Continue reading on journalpanic.com