Hey guys, today’s post is about how to create a watchdog in Python. But what is a “watchdog”?
A watchdog is a little piece of software that monitors our filesystem looking for any changes (like the creation, change or deletion of a file or of a directory). When a change occurs, the watchdog report it to us raising a specific event that we can handle.
For example, let’s suppose you have developed a program that use a configuration file. Your program could set a watchdog to monitor that file and if the configuration file is modified you could think to reload it and apply the new configuration at runtime, without the need of restarting your program.
That’s cool, uh?
So, today we will code a watchdog in Python.
To code this program we will need an additional module called “watchdog” (wow, who could have guessed it?) written by Yesudeep Mangalapilly, so let’s start by installing it. As always, I raccomand to use virtual environments instead of installing packages system wide. If you want to find out more about virtual environments (that’s probabilly because you haven’t read all my previous post, so shame on you!), just have a look at this article.
Now, create your virtual environment (optional but raccomended… at least by me), activate it and install the package watchdog with the following command:

pip install watchdog

Done, we have almost finished, haven’t we?
I mean, we still need to code the program that will actually use this module, but trust me, that will be really easy!

Ok, let’s start and pretend that we want to create a program that logs all the file that are created or modified in the same directory of our program.

Step 1. Import some stuff

import time
from watchdog.observers import Observer
from watchdog.events import PatternMatchingEventHandler

I won’t explain these imports right now, they will be clear in a moment.

Step 2. Create the event handler

The event handler is the object that will be notified when something happen on the filesystem you are monitoring.

if __name__ == "__main__":
    patterns = ["*"]
    ignore_patterns = None
    ignore_directories = False
    case_sensitive = True
    my_event_handler = PatternMatchingEventHandler(patterns, ignore_patterns, ignore_directories, case_sensitive)

In my example I have used some variables just to made the configuration of the event handler a little bit easier to be undestood.
The “patterns” variable contains the file patterns we want to handle (in my scenario, I will handle all the files), the “ignore_patterns” variable contains the patterns that we don’t want to handle, the “ignore_directories” is just a boolean that we can set to True if we want to be notified just for regular files (not for directories) and the “case_sensitive” variable is just another boolean that, if set to “True”, made the patterns we previously introduced “case sensitive” (that’s normally a good idea, unless you are working with stupid case-insensitive-file-systems… yeah, I’m talking about you Windows! :P ).

Step 3. Handle all the events

Now that we have created the handler we need to write the code we want to run when the events are raised.
So, let’s start creating four different functions that will be used when a file is modified, created, deleted or moved.

def on_created(event):
    print(f"hey, {event.src_path} has been created!")

def on_deleted(event):
    print(f"what the f**k! Someone deleted {event.src_path}!")

def on_modified(event):
    print(f"hey buddy, {event.src_path} has been modified")

def on_moved(event):
    print(f"ok ok ok, someone moved {event.src_path} to {event.dest_path}")

In our example we will just print out something regarding the events that has been captured.
It goes without saying that this is just an example, we could execute any kind of code instead of these useless print functions…

And now, under the creation of our event handler (the code of the previous step) we need to specify to the handler that we want these functions to be called when the corresponding event is raised

my_event_handler.on_created = on_created
    my_event_handler.on_deleted = on_deleted
    my_event_handler.on_modified = on_modified
    my_event_handler.on_moved = on_moved

Step 4. Create an observer

We need another object, known as the observer, that will monitor our filesystem, looking for changes that will be handled by the event handler.

Let’s create it right now.

    path = "."
    go_recursively = True
    my_observer = Observer()
    my_observer.schedule(my_event_handler, path, recursive=go_recursively)

As you can see, I have just created an object of the Observer class and then called the schedule method, passing to it:

• the event handler that will handle the event
• the path to be monitored (in my case is “.”, that’s the current directory)
• a boolean that allow me to catch all the event that occurs even in the sub directories of my current directory.

Step 5. Start the observer

Ok, we almost finished our program, now we need just to start the observer thread.

    my_observer.start()
    try:
        while True:
            time.sleep(1)
    except KeyboardInterrupt:
        my_observer.stop()
        my_observer.join()

I think this snippet doesn’t need any comments, does it? We are just starting the observer thread to get all the events.

Ok, if everything is correct the result should be the one you can see below.

Testing our new shiny watchdog

And now it’s showtime guys, let’s test our watchdog!

Start the program from a terminal and you will see your cursor just hang. That’s a good start.

Now open a different terminal, move under the directory of your program and try to do something. For example, I have started vim (my favourite editor) to create a file:

$ vim dave.txt

by the time vim started, in my running console some statement has been printed:

hey, ./.dave.txt.swp has been created!
hey buddy, . has been modified

Wow, it works! When you try to create a file with Vim it start by creating a .swp file and that’s what our program has detected, printing the first line. But why the second line? Well, if the current directory has a new file we can say that it’s changed as well, can’t we?

Now I will try to save my file and…

hey, ./dave.txt has been created!
hey buddy, . has been modified

Ok, Vim has created my file (dave.txt) and my program has detected it! Well done!

Now I quit vim (for the record and for all the “Windows guys” that are currently reading this… you don’t need to unplug the power cord of your pc to quit vim, it’s enough to press the [esc] key and to type :q )

what the f**k! Someone deleted ./.dave.txt.swp!
hey buddy, . has been modified

Ok, vim has deleted the temporary .swp file and the program has detected it. Mission accomplished! :)

The bottom line

In this article you have seen how easy it could be to set up a watchdog in Python that monitors a filesystem and react when a change occurs.

If you want to find out more about the watchdog module, you can have a look at the official documentation.

Happy Coding! D.

Hi guys, today’s post is just for the ones of you that work with the “OS of the misoriented slashes”: Microsoft Windows. :)

Have you ever had the need of writing a Python script that could run in background as a Windows Service? In this post, you will learn how to do it in less than 10 minutes, no jokes.

I will skip all the introduction about Windows Services, how convenient they could be, how much could be appreciated the fact that they can be run in background even when the user is logged off etc… I mean, if you can code in Python and you use Windows I bet you already know what a Windows Service is, don’t you?

So, first of all, let’s start by installing the Python for Windows extensions:

c:test> pip install pywin32

Once you have done it, let’s write this base class, your Windows service will be a subclass of this base class.

'''
SMWinservice
by Davide Mastromatteo

Base class to create winservice in Python
-----------------------------------------

Instructions:

1. Just create a new class that inherits from this base class
2. Define into the new class the variables
   _svc_name_ = "nameOfWinservice"
   _svc_display_name_ = "name of the Winservice that will be displayed in scm"
   _svc_description_ = "description of the Winservice that will be displayed in scm"
3. Override the three main methods:
    def start(self) : if you need to do something at the service initialization.
                      A good idea is to put here the inizialization of the running condition
    def stop(self)  : if you need to do something just before the service is stopped.
                      A good idea is to put here the invalidation of the running condition
    def main(self)  : your actual run loop. Just create a loop based on your running condition
4. Define the entry point of your module calling the method "parse_command_line" of the new class
5. Enjoy
'''

import socket

import win32serviceutil

import servicemanager
import win32event
import win32service


class SMWinservice(win32serviceutil.ServiceFramework):
    '''Base class to create winservice in Python'''

    _svc_name_ = 'pythonService'
    _svc_display_name_ = 'Python Service'
    _svc_description_ = 'Python Service Description'

    @classmethod
    def parse_command_line(cls):
        '''
        ClassMethod to parse the command line
        '''
        win32serviceutil.HandleCommandLine(cls)

    def __init__(self, args):
        '''
        Constructor of the winservice
        '''
        win32serviceutil.ServiceFramework.__init__(self, args)
        self.hWaitStop = win32event.CreateEvent(None, 0, 0, None)
        socket.setdefaulttimeout(60)

    def SvcStop(self):
        '''
        Called when the service is asked to stop
        '''
        self.stop()
        self.ReportServiceStatus(win32service.SERVICE_STOP_PENDING)
        win32event.SetEvent(self.hWaitStop)

    def SvcDoRun(self):
        '''
        Called when the service is asked to start
        '''
        self.start()
        servicemanager.LogMsg(servicemanager.EVENTLOG_INFORMATION_TYPE,
                              servicemanager.PYS_SERVICE_STARTED,
                              (self._svc_name_, ''))
        self.main()

    def start(self):
        '''
        Override to add logic before the start
        eg. running condition
        '''
        pass

    def stop(self):
        '''
        Override to add logic before the stop
        eg. invalidating running condition
        '''
        pass

    def main(self):
        '''
        Main class to be ovverridden to add logic
        '''
        pass

# entry point of the module: copy and paste into the new module
# ensuring you are calling the "parse_command_line" of the new created class
if __name__ == '__main__':
    SMWinservice.parse_command_line()

Let’s examine the class we have just introduced a little.

def SvcDoRun(self): it’s the method that will be called when the service is requested to start.

def SvcStop(self): it’s the method that will be called when the service is requested to stop.

def start(self): it’s a method that you will be asked to override if you need to do something when the service is starting (before started)

def stop(self): it’s the method that you will be asked to override if you need to do something when the service is stopping (before stopped)

def main(self): it’s the method that will contain the logic of your script, usually in a loop that keeps it alive until the service is stopped.

def parse_command_line(cls): it’s the method that handles the command line interface that you can use to install and update your windows service

Can you see how easy it is with pywin32 to interface with the system to create a Windows Service? The last mention is for the following variables:

svc_name = "PythonCornerExample"
svc_display_name = "Python Corner's Winservice Example"
svc_description = "That's a great winservice! :)"

These are just three variables that contain the name of the service, the “friendly name” that will be used by Windows to display the name on the mmc console and a short description of your service.

As always, enough talk, let’s code something useful!

Let’s pretend that we want to create a Winservice that, when started, creates a random file on our C: drive every 5 seconds.

What? Do you think it is stupid? Well, install it on your boss PC, set the destination folder as its user’s desktop and you will change your mind. :)

However, how can you achieve this result? Super easy.

• Subclass the SMWinservice class we have just met.
• On the new class, override the three variables _svc_name_, _svc_display_name_ and _svc_description_.
• Override the “start” method to set the running condition. Setting a boolean variable will be enough for that.
• Override the “stop” method to invalidate the running condition when the service is requested to be stopped.
• Override the “main” method to add the logic of creating a random file every 5 seconds
• Add the call at the “parse_command_line” function to handle the command line interface.

The result should be something like this:

import time
import random
from pathlib import Path
from SMWinservice import SMWinservice

class PythonCornerExample(SMWinservice):
    _svc_name_ = "PythonCornerExample"
    _svc_display_name_ = "Python Corner's Winservice Example"
    _svc_description_ = "That's a great winservice! :)"

    def start(self):
        self.isrunning = True

    def stop(self):
        self.isrunning = False

    def main(self):
        i = 0
        while self.isrunning:
            random.seed()
            x = random.randint(1, 1000000)
            Path(f'c:{x}.txt').touch()
            time.sleep(5)

if __name__ == '__main__':
    PythonCornerExample.parse_command_line()

That’s it! Now it’s time to install our newly created winservice. Just open a command prompt, navigate to your script directory and install the service with the command:

C:test> python PythonCornerExample.py install
Installing service PythonCornerExample
Service installed

In the future, if you want to change the code of your service, just modify it and reinstall the service with

C:test> python PythonCornerExample.py update
Changing service configuration
Service updated

Now, open the “Services” msc snap in

C:test> mmc Services.msc

locate your new PythonCornerExample winservice, and right click and choose properties. Here you can start your service and configure it at your will.

Now try to start your service and go to see your C: folder contents.

Can you see all these files being created to your C: folder? Yeah, that’s working!

But now it’s time to stop it! :) You can do it from the previous windows or just by using the command line

C:test> net stop PythonCornerExample

Il servizio Python Corner's Winservice Example sta per essere arrestato.. 
Servizio Python Corner's Winservice Example arrestato.

If something goes wrong…

There are a couple of known problems that can happen writing Windows Services in Python. If you have successfully installed the service but starting it you get an error, follow this iter to troubleshoot your service:

• Check if Python is in your PATH variable. It MUST be there. To check this, just open a command prompt and try starting the python interpreter by typing “python”. If it starts, you are ok.
• Be sure to have the file C:\Program Files\Python36\Lib\site-packages\win32\pywintypes36.dll (please note that “36” is the version of your Python installation). If you don’t have this file, take it from C:\Program Files\Python36\Lib\site-packages\pywin32_system32\pywintypes36.dll and copy it into C:\Program Files\Python36\Lib\site-packages\win32

If you still have problems, try executing your Python script in debug mode. To try this with our previous example, open a terminal, navigate to the directory where the script resides and type

c:test> python PythonCornerExample.py debug

Ok, that’s all for today, this was just a brief introduction to developing Windows Services with Python. Try it by yourself and … happy coding!

D.

If the first concern of a developer is to be sure that the code they write works well, the second one is to make sure that it run fast. This is expecially true when you’re dealing with web applications, where the scalability of your application is a crucial topic. For this reason, one of the most important tool we can use to improve the speed of our code is the use of a cache system.

A cache system is a component that stores data so that future requests for data we already served in the past, could be accomplished faster. There are a lot of solutions that can be used to implement a cache system but today I want to point out a specific solution that allows your Python code to use a cache for everyday use, without setting up a complex (and yet more powerful) system like Redis or other: the package cachetools.

As always, let’s make the code talk for us! :)

It goes without saying that to use the package cachetools we need to install it, so before trying the code in this article, please install the required package:

pip install cachetools

As always, my advise is to install everyting into a virtual environment. If you don’t know what a virtual environment is or how to use it, check my previous article about this topic.

Now, let’s pretend that you have an application with a very time consuming function that you call many times in your code, with 100 different parameters, but the result you get, if the input parameter remain the same, doesn’t change very often. To make it more concrete, let’s say that your work is to sell candies, and you have 100 different kind of candies in stock.

In the cash register’s python source code, you have a function that check the price of a single candy by connecting with a web service of a server in the cloud. Unfortunately, this web service is very slow to response (you should fire the guy who wrote it maybe), so this function that check the price is always very slow.

Here is your code:

import time
import datetime

def get_candy_price(candy_id):
    # let's use a sleep to simulate the time your function spends trying to connect to
    # the web service, 5 seconds will be enough.
    time.sleep(5)

    # let's pretend that the price returned by the web service is $1 for candies with a
    # odd candy_id and $1,5 for candies with a even candy_id

    price = 1.5 if candy_id % 2 == 0 else 1

    return (datetime.datetime.now().strftime("%c"), price)


# now, let's simulate 20 customers in your show.
# They are asking for candy with id 2 and candy with id 3...
for i in range(0,20):
    print(get_candy_price(2))
    print(get_candy_price(3))

As you can see, the get_candy_price is very slow (we have put a “sleep” in the function to simulate the slowness due to the bad web service we are relying on).

What can we do?

Well, this seems the perfect function that can benefit from the use of a cache system. The candy price is not going to vary so often and even if it change, we could assume that it’s ok if it’s updated on our system within 5 minutes, with no hurry.

So, how can we implement this behaviour? It’s super easy, we need just three code lines.

import time
import datetime

from cachetools import cached, TTLCache  # 1 - let's import the "cached" decorator and the "TTLCache" object from cachetools
cache = TTLCache(maxsize=100, ttl=300)  # 2 - let's create the cache object.

@cached(cache)  # 3 - it's time to decorate the method to use our cache system!
def get_candy_price(candy_id):
    # let's use a sleep to simulate the time your function spends trying to connect to
    # the web service, 5 seconds will be enough.
    time.sleep(5)

    # let's pretend that the price returned by the web service is $1 for candies with a
    # odd candy_id and $1,5 for candies with a even candy_id

    price = 1.5 if candy_id % 2 == 0 else 1

    return (datetime.datetime.now().strftime("%c"), price)


# now, let's simulate 20 customers in your show.
# They are asking for candy with id 2 and candy with id 3...
for i in range(0,20):
    print(get_candy_price(2))
    print(get_candy_price(3))

Can you spot the three new lines?

The first one is the import statement that we need to use the cachetools package.

The second one is the line to create the cache. We need to specify how many objects we will store (100, because we have just 100 kind of candies) and the time each cached results have to live (300 seconds, 5 minutes for us).

The third line is the decorator, that enable the function to use the cached results.

Now try to execute this program again and you will notice that the first time the function is called (with parameter “2”) it takes 5 seconds to be executed (yes, the cache is empty at that time) and so it does the second time the function is called (with parameter “3”) because in the cache there’s just the result for the candy_id 2 that can’t be used since we are asking for the candy_id 3. But from the third call on, every single call is super fast, since the body of the method is NOT executed at all. The results we get are just taken from the cache system. Isn’t that cool?

Cachetools’ cache type

In the example above we have used a “Time To Live Cache”. This cache associates a time to live value to each item stored in cache. Items that expire because they have exceeded their time-to-live are removed automatically, making room for new values. If no expired items are there to remove, the least recently used items will be discarded first to make space when necessary.

Other kinds of cache that are available in the cachetools package are:

• the LFUCache (Least Frequently Used), that counts how often an item is retrieved, and discards the items used least often to make space when necessary.
• the LRUCache (Least Recently Used), that discards the least recently used items first to make space when necessary.
• the RRCache (Random Replacement), that randomly selects candidate items and discards them to make space when necessary.

And all these cache types can be used in a decorator of a function, like we did it before, or simply by creating a cache object and using it directly, choosing at run time what to add to the cache and when to retrieve the values added.

The drawbacks

One thing a wise person should always keep in consideration is that

not all that glitters is gold

A cache system bring also some disadvantages.

The first and more obvious one is that cached data are  (by definition )  ”old”. That means that when you reuse these old data, you run the risk of presenting data that are no more relevant in the new context. In the example that we’ve seen before, if you are selling candies you can assume that a five minutes delay in retrieving the candy price from the server is fair enough to reuse the cached data. But if you were in a different context, and you were getting the current Bitcoin value, reuse a five minutes old result could be the most stupid thing to do. :)

Another thing that you have to keep in mind when using a cache is that the data you are caching are not stored in “Eega Beeva’s pockets”, they are stored in your RAM. So, if you cache 100 complex objects that are 5 Megabytes each, you are storing 500 Megabytes in the ram of your server.

Developing is often a matter of tradeoffs, think about it not only when you have to decide if to use a cache or not, but always when you start designing a software solution.

Ok, that’s all folks, if you want to dig in this topic, go and check the documentation of the cachetools package.ù

Happy caching! :)

D.

Following a request of a reader, today we’re going to discuss when to use iterators and generators in Python.

First of all, it’s important to know what iterators and generators are, so if you don’t know exactly what they are, I suggest to have a look at my previous article on this topic.

Now that everything is clear, we can start analyzing when to use these features.

Let’s start saying that if you have read my previous article, the use of the iterator protocol should be quite clear: you use iterator protocol when you have a custom object that you want to be “iterable”.

That’s it, so easy.

If you want to use your custom object in a loop with something like

for i in my_object():
    # do something
    pass

you just need to adopt the iterator protocol. But what about generators?

When am I supposed to write a generator instead of a simple function that returns a list of objects?

Well, I have to admit that this can be a little bit tricky for a beginner… so let’s try to answer this question pretending to be there to write a function that returns a list of objects and let’s answer the following questions.

Do I need all the items of the returned list?

This is the first question you should ask yourself when writing a function that returns a list of objects. If the answer is “no”, that probably means that a generator would be a better choice because its main feature is the “lazy evaluation”.

With a generator, you generate a result only when you really need it, so if you’re not going to use all the items in the list, why bother creating them?

You will save time and resources not creating them and your users will be happier!

To make an example, have a look at this program.

import time
import random

def get_winning_numbers():
    random.seed()
    elements = []
    for i in range (0,10):
        time.sleep(1) # let's simulate some kind of delay
        elements.append(random.randint(1,10))

    return elements

random.seed()
my_number = random.randint(1,10)
print ("my number is " + str(my_number))

for winning_number in get_winning_numbers():
    print(winning_number)
    if my_number == winning_number:
        print ("you win!")
        break

The function “get_winning_numbers” is a time-consuming function that generates 10 random “winning numbers” (to simulate the “time-consuming function” we have added a delay of a second for every number generated).

The winning numbers are then checked against “my_number”; if my_number is in these 10 numbers, the player wins and the execution ends.

Made in this way, however, you have always to wait at least 10 seconds because all the winning numbers **are all generated before **the check against the player lucky number. That’s a waste of time because if the first of the winning numbers were the player’s number, we’d had generated 9 other winning numbers (using a time-consuming function) that we don’t need and that we will never use.

Using a generator, we can solve this problem pretty easily:

import time
import random

def get_winning_numbers():
    random.seed()
    for i in range(0,10):
        time.sleep(1)  # let's simulate some kind of delay
        yield random.randint(1,10)

random.seed()
my_number = random.randint(1,10)
print ("my number is " + str(my_number))

for winning_number in get_winning_numbers():
    print(winning_number)
    if my_number == winning_number:
        print ("you win!")
        break

We don’t need to make a big change to our program, we have the same result but the execution is often faster than the old version. In fact, now if the first winning number is equal to the lucky number of the player, we generate just that number, the player wins and the execution ends in just one second.

Do I need to be notified while the results of the list are generated?

If the answer to this question is yes, well, you will probably need a generator.

Think about a function that searches something on your filesystem or any other slow device and returns a list of results. If your function takes 5 seconds to find every single element and there are just 4 elements to be found, you have to wait 20 seconds before getting the results.

import time

def elements():
    elements = []
    for i in range (0,4):
        # simulate a slow search
        time.sleep(5)
        elements.append(i)
    return elements

print("start")
print(elements())
print("end")

In this case, even if you need all the four elements before going on, your app will seem to be frozen for 20 seconds, and this could be annoying for the user.

Wouldn’t it be better to be notified after every result, even just to have the time to update the user interface, maybe showing the partial results found or even a simple progress bar?

import time

def elements():
    elements = []
    for i in range (0,4):
        # simulate a slow search
        time.sleep(5)
        yield(i)

print("start")
for i in elements():
    # show a "console style" progress bar  :)
    print(".", end="", flush=True)
print()
print("end")

Is the memory footprint of the function I’m writing relevant?

If the answer is “yes”, it’s probably a good idea to use a generator.

That’s because with a generator you create a result just when you need it and after the result has been created you can start working on it, removing it from the memory when you have finished and before asking for another item.

Let’s say that your function is supposed to return a huge list of big objects, to return a single list you have to create that list and keep it all in memory.

import time

def get_elements():
    elements = []
    for i in range (0,10000):
        elements.append("x"*10240)
    # return a list of 10.000 items, each of 10KB...
    return elements

characters_count = 0

my_elements=get_elements()
# in this moment, our program has a memory footprint of more than 100MB!!!

for i in my_elements:
    characters_count = characters_count + len(i)

print(characters_count)

As you see, in this case, we are allocating more than 100MB RAM before actually doing anything… But using a generator…

def get_elements():
    for i in range (0,10000):
        yield("x"*10240)

characters_count = 0

my_elements=get_elements()

for i in my_elements:
    characters_count = characters_count + len(i)

print(characters_count)

we get the same result with just 10KB RAM used at a time.

Now I can hear you wondering “well, I have to use generators if the function that creates an element is time-consuming, if the memory footprint of that function is relevant or if I don’t need all the elements of the list, but in every other case it’s ok to create a list, right?”… Well, it could be ok… but… even in this case, why not to use a generator? A generator, for me, is always a better choice and consider that if you have a generator, converting it in a list is a trivial operation that can be done by using the “list” keyword like that.

mylist = list(my_generator())

or by using list comprehension syntax

mylist = [element for element in my_generator()]

To sum up: ask not “why should I use a generator?” but ask “why shouldn’t I?”

:)

Happy Coding!

D.

In my last post, I discussed some ways to avoid nested code in Python and discussing the “filter” and “map” functions I mentioned the lambda functions.

After that article, some reader asked me to write a little more about this topic, so … here I am. :)

Let’s start with a mantra. If you want to know what something is, in Python, just use your REPL.

So, start the Python REPL and define a lambda:

Python 3.6.2 |Anaconda custom (64-bit)| (default, Sep 19 2017, 08:03:39) [MSC v.1900 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.

>>> my_lambda = lambda x: x+1

Now, try to ask Python what is “my_lambda”

>>> print(my_lambda)
<function <lambda> at 0x0000021D14663E18>

It turns out that a “lambda” is… just a function!

Basically, a lambda is just an anonymous function that can be used “inline” whenever your code expects to find a function. In Python, in fact, functions are first-class objects and that basically means that they can be used like any other objects. They can be passed to other functions, they can be assigned to a name, they can be returned from a function and so on.

So, in our first example, we just defined a function that takes an argument (x), sums the value 1 to the input argument, and returns the result of this operation.

What’s the name of the function? It has no name actually, and so I had to assign this anonymous function to the name “my_lambda” to be able to use it in my code.

Now I can hear some of you saying:

Why bother with this stuff? Why not just use a standard named function? Couldn’t I write something like this?

>>> def my_sum_function(x):
... return x+1
...
>>> print (my_sum_function)
<function my_sum_function at 0x0000021D14D83E18>

Yes, you could actually… and I will tell you something more: you can pass this function as well to other functions.

In our example, if we use:

>>> print(list(map(my_lambda, range(10)))) [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]

we get the same results of writing:

>>> print(list(map(my_sum_function, range(10))))
[1, 2, 3, 4, 5, 6, 7, 8, 9, 10]

So, why bother with lambda functions?

Well… it’s just a matter of style and readability, because as we will keep saying: readability counts (“The Zen Of Python”, rule #7).

The case of the RPN Calculator

Let’s pretend that one of your customers asked you to create a program to simulate a “Reverse Polish Notation calculator” that they will install on all their employees’ computers. You accept this work and get the specs for the program:

The program should be able to do all the basic operations (divide, sum, subtract, multiply), to find the square root of a number and to power to 2 a number. Obviously, you should be able to clear all the stack of the calculator or just to drop the last inserted number.

RPN Calculator for newbies If you already know what is a RPN calculator, skip this part. If you do not know what is the Reverse Polish Notation, but you are curious, just check this out. If you do not know anything about the Reverse Polish Notation and you want just a brief explanation to keep reading this article… go ahead. The Reverse Polish Notation was very relevant in 70’s and 80’s as Hewlett-Packard used it in their scientific calculators and I personally love this notation. Basically in a RPN calculator you have a “stack” where you put the operands with a LIFO (last in, first out) logic. Then, when you press the button to compute an operation, the calculator takes out from the stack all the operands that the operation ask for, compute the operation and put the result back to the stack. So, let’s do an example. You want the result of the operation 5 x 6. In a standard calc you would act this way:

• press 5
• press *
• press 6
• press = and on the display you get the result : 30. In a RPN calculator, you act this way:
• press 5
• press ENTER (and the number 5 is put on the stack)
• press 6
• press ENTER (and the number 6 is put on the stack)
• press * Now, before pressing the ‘’ symbol your stack was like this: - 5 6 - after pressing the ‘’ symbol, you find on the stack just the result: 30. That’s because the calculator knows that for a multiplication you need two operands, so it has “popped” the first value (6 — remember, it’s a stack, it’s LIFO), then it has “popped” the second value (5), it has executed the operation and put back the result (30) on the stack. RPN calculators are great since they do not need expressions to be parethesized and there’s more: they’re cool! :)

Now, you accepted the job and you have to start working, but since the deadline of the project is very tight, you decide to ask Max, your IT artist, of writing a GUI for the calculator and Peter, your new intern, of creating the “engine” of this calculator software.

Let’s focus on Peter’s work.

Peter’s work

After a while, Peter comes proudly to you asserting he has finished coding the calculator engine.

And that’s what he has done so far:

"""
Engine class of the RPN Calculator
"""

import math

class rpn_engine:
    def __init__(self):
        """ Constructor """
        self.stack = []

    def push(self, number):
        """ push a value to the internal stack """
        self.stack.append(number)

    def pop(self):
        """ pop a value from the stack """
        try:
            return self.stack.pop()
        except IndexError:
            pass # do not notify any error if the stack is empty...

    def sum_two_numbers(self):
        op2 = self.stack.pop()
        op1 = self.stack.pop()
        self.push(op1 + op2)

    def subtract_two_numbers(self):
        op2 = self.stack.pop()
        op1 = self.stack.pop()
        self.push(op1 - op2)

    def multiply_two_numbers(self):
        op2 = self.stack.pop()
        op1 = self.stack.pop()
        self.push(op1 * op2)

    def divide_two_numbers(self):
        op2 = self.stack.pop()
        op1 = self.stack.pop()
        self.push(op1 / op2)

    def pow2_a_number(self):
        op1 = self.stack.pop()
        self.push(op1 * op1)

    def sqrt_a_number(self):
        op1 = self.stack.pop()
        self.push(math.sqrt(op1))


    def compute(self, operation):
        """ compute an operation """

        if operation == '+':
            self.sum_two_numbers()

        if operation == '-':
            self.subtract_two_numbers()

        if operation == '*':
            self.multiply_two_numbers()

        if operation == '/':
            self.divide_two_numbers()

        if operation == '^2':
            self.pow2_a_number()

        if operation == 'SQRT':
            self.sqrt_a_number()

        if operation == 'C':
            self.stack.pop()

        if operation == 'AC':
            self.stack.clear()

In a moment he understands that you’re looking at him shocked and states: “my code runs fine”.

Now, let’s clarify one thing: Peter’s right. His code runs (so does a burning bus…). Nevertheless, his code sucks. That’s it.

So let’s have a look at how we can improve this “stuff”.

The first problem here is the code duplication. There’s a principle of software engineering that is called “DRY”. It stands for “Don’t Repeat Yourself”.

Peter has duplicated a lot of code because for every single function he has to get the operands, compute the operation and put the result back to the stack. Wouldn’t it be great if we could have a function that does exactly this job, computing the operation we request? How can we achieve this?

Well, it’s really simple actually, because as we said earlier… functions are first-class objects in Python! So, Peter’s code can be simplified a lot.

Let’s have a look at the functions we have to provide.

All the standard operations (divide, sum, add and multiply) needs two operands to be computed. The “sqrt” and the “pow2” functions need just one operand to be computed. The “C” (to drop the last item in the stack) and “AC” (to clear the stack) functions, don’t need any operand to be computed.

So, let’s rewrite Peter’s code this way:

"""
Engine class of the RPN Calculator
"""

import math

class rpn_engine:
    def __init__(self):
        """ Constructor """
        self.stack = []

    def push(self, number):
        """ push a value to the internal stack """
        self.stack.append(number)

    def pop(self):
        """ pop a value from the stack """
        try:
            return self.stack.pop()
        except IndexError:
            pass # do not notify any error if the stack is empty...

    def sum_two_numbers(self, op1, op2):
        return op1 + op2

    def subtract_two_numbers(self, op1, op2):
        return op1 - op2

    def multiply_two_numbers(self, op1, op2):
        return op1 * op2

    def divide_two_numbers(self, op1, op2):
        return op1 / op2

    def pow2_a_number(self, op1):
        return op1 * op1

    def sqrt_a_number(self, op1):
        return math.sqrt(op1)


    def compute(self, operation):
        """ compute an operation """

        if operation == '+':
            self.compute_operation_with_two_operands(self.sum_two_numbers)

        if operation == '-':
            self.compute_operation_with_two_operands(self.subtract_two_numbers)

        if operation == '*':
            self.compute_operation_with_two_operands(self.multiply_two_numbers)

        if operation == '/':
            self.compute_operation_with_two_operands(self.divide_two_numbers)

        if operation == '^2':
            self.compute_operation_with_one_operand(self.pow2_a_number)

        if operation == 'SQRT':
            self.compute_operation_with_one_operand(self.sqrt_a_number)

        if operation == 'C':
            self.stack.pop()

        if operation == 'AC':
            self.stack.clear()

    def compute_operation_with_two_operands(self, operation):
        """ exec operations with two operands """
        try:
            if len(self.stack) < 2:
                raise BaseException("Not enough operands on the stack")

            op2 = self.stack.pop()
            op1 = self.stack.pop()
            result = operation(op1, op2)
            self.push(result)
        except BaseException as error:
            print(error)

    def compute_operation_with_one_operand(self, operation):
        """ exec operations with one operand """
        try:
            op1 = self.stack.pop()
            result = operation(op1)
            self.push(result)
        except BaseException as error:
            print(error)

Isn’t it better? The duplicated code is far less than before and looks at the functions, all they do is to compute the operation and return the results. They are no longer in charge of getting operands and pushing the result to the stack, the readability of the code is definitely improved!

Now, looking at the code, the first thing I really hate is all the “ifs” in the compute function. Perhaps replacing them with a “switch” function… if only the switch function would exist in Python! :)

But we can do something better. Why don’t we create a catalog of the available functions and then we just use this catalog to decide which function to use?

Why don’t we use a dictionary for that?

Let’s try to modify our code again:

"""
Engine class of the RPN Calculator
"""

import math

class rpn_engine:
    def __init__(self):
        """ Constructor """
        self.stack = []
        self.catalog = self.get_functions_catalog()

    def get_functions_catalog(self):
        return {"+": self.sum_two_numbers, 
                "-": self.subtract_two_numbers, 
                "*": self.multiply_two_numbers,
                "/": self.divide_two_numbers,
                "^2": self.pow2_a_number,
                "SQRT": self.sqrt_a_number,
                "C": self.stack.pop,
                "AC": self.stack.clear}

    def push(self, number):
        """ push a value to the internal stack """
        self.stack.append(number)

    def pop(self):
        """ pop a value from the stack """
        try:
            return self.stack.pop()
        except IndexError:
            pass # do not notify any error if the stack is empty...

    def sum_two_numbers(self, op1, op2):
        return op1 + op2

    def subtract_two_numbers(self, op1, op2):
        return op1 - op2

    def multiply_two_numbers(self, op1, op2):
        return op1 * op2

    def divide_two_numbers(self, op1, op2):
        return op1 / op2

    def pow2_a_number(self, op1):
        return op1 * op1

    def sqrt_a_number(self, op1):
        return math.sqrt(op1)


    def compute(self, operation):
        """ compute an operation """

        if operation in ['+', '-', '*', '/']:
            self.compute_operation_with_two_operands(self.catalog[operation])

        if operation in ['^2', 'SQRT']:
            self.compute_operation_with_one_operand(self.catalog[operation])

        if operation in ['C', 'AC']:
            self.compute_operation_with_no_operands(self.catalog[operation])

    def compute_operation_with_two_operands(self, operation):
        """ exec operations with two operands """
        try:
            if len(self.stack) < 2:
                raise BaseException("Not enough operands on the stack")

            op2 = self.stack.pop()
            op1 = self.stack.pop()
            result = operation(op1, op2)
            self.push(result)
        except BaseException as error:
            print(error)

    def compute_operation_with_one_operand(self, operation):
        """ exec operations with one operand """
        try:
            op1 = self.stack.pop()
            result = operation(op1)
            self.push(result)
        except BaseException as error:
            print(error)

    def compute_operation_with_no_operands(self, operation):
        """ exec operations with no operands """
        try:
            operation()
        except BaseException as error:
            print(error)

Wow, almost all our “if(s)” are gone! And now we have a catalog of functions that we can expand as we want. So for example, if we need to implement a factorial function, we will just add the function to the catalog and implement a custom method in the code. That’s really good!

Even if …

It would be great to act only on the catalog, wouldn’t it? But wait… shouldn’t we talk about lambdas in this article?

Here’s where lambdas can be useful! We don’t need a standard defined function for a simple calc, we need just an inline lambda for that!

"""
Engine class of the RPN Calculator
"""

import math

class rpn_engine:
    def __init__(self):
        """ Constructor """
        self.stack = []
        self.catalog = self.get_functions_catalog()

    def get_functions_catalog(self):
        return {"+": lambda x, y: x + y,
                "-": lambda x, y: x - y,
                "*": lambda x, y: x * y,
                "/": lambda x, y: x / y,
                "^2": lambda x: x * x,
                "SQRT": lambda x: math.sqrt(x),
                "C": self.stack.pop,
                "AC": self.stack.clear}

    def push(self, number):
        """ push a value to the internal stack """
        self.stack.append(number)

    def pop(self):
        """ pop a value from the stack """
        try:
            return self.stack.pop()
        except IndexError:
            pass # do not notify any error if the stack is empty...

    def compute(self, operation):
        """ compute an operation """

        if operation in ['+', '-', '*', '/']:
            self.compute_operation_with_two_operands(self.catalog[operation])

        if operation in ['^2', 'SQRT']:
            self.compute_operation_with_one_operand(self.catalog[operation])

        if operation in ['C', 'AC']:
            self.compute_operation_with_no_operands(self.catalog[operation])

    def compute_operation_with_two_operands(self, operation):
        """ exec operations with two operands """
        try:
            if len(self.stack) < 2:
                raise BaseException("Not enough operands on the stack")

            op2 = self.stack.pop()
            op1 = self.stack.pop()
            result = operation(op1, op2)
            self.push(result)
        except BaseException as error:
            print(error)

    def compute_operation_with_one_operand(self, operation):
        """ exec operations with one operand """
        try:
            op1 = self.stack.pop()
            result = operation(op1)
            self.push(result)
        except BaseException as error:
            print(error)

    def compute_operation_with_no_operands(self, operation):
        """ exec operations with no operands """
        try:
            operation()
        except BaseException as error:
            print(error)

Wow, this code rocks now! :)

Even if…

Let’s pretend that we have to add the factorial function, could we just modify the catalog?

Unfortunately no.

There’s another place we have to modify… we have to modify also the compute function because we need to specify that the factorial function is a “one operand function”.

That’s bad, we do know that it is a one operand function, it’s obvious since we need to call the math.factorial(x) function passing just the x argument. If only there were a way to determine how many arguments a function needs at runtime…

There is actually. In the “inspect” module, there’s a “signature” function that can help us inspect the signature of our method at runtime. So, let’s start the REPL and do a quick test:

>>> a = lambda x, y: x + y
>>> from inspect import signature
>>> my_signature = signature(a)
>>> print(my_signature)
(x, y)
>>> print (my_signature.parameters)
OrderedDict([(‘x', <Parameter "x">), (‘y', <Parameter "y">)])
>>> print (len(my_signature.parameters))
2

Yes, amazing. We could determine at runtime how many operands our function needs!

"""
Engine class of the RPN Calculator
"""

import math
from inspect import signature

class rpn_engine:
    def __init__(self):
        """ Constructor """
        self.stack = []
        self.catalog = self.get_functions_catalog()

    def get_functions_catalog(self):
        """ Returns the catalog of all the functions supported by the calculator """
        return {"+": lambda x, y: x + y,
                "-": lambda x, y: x - y,
                "*": lambda x, y: x * y,
                "/": lambda x, y: x / y,
                "^2": lambda x: x * x,
                "SQRT": lambda x: math.sqrt(x),
                "C": lambda: self.stack.pop(),
                "AC": lambda: self.stack.clear()}

    def push(self, number):
        """ push a value to the internal stack """
        self.stack.append(number)

    def pop(self):
        """ pop a value from the stack """
        try:
            return self.stack.pop()
        except IndexError:
            pass # do not notify any error if the stack is empty...

    def compute(self, operation):
        """ compute an operation """

        function_requested = self.catalog[operation]
        number_of_operands = 0
        function_signature = signature(function_requested)
        number_of_operands = len(function_signature.parameters)

        if number_of_operands == 2:
            self.compute_operation_with_two_operands(self.catalog[operation])

        if number_of_operands == 1:
            self.compute_operation_with_one_operand(self.catalog[operation])

        if number_of_operands == 0:
            self.compute_operation_with_no_operands(self.catalog[operation])

    def compute_operation_with_two_operands(self, operation):
        """ exec operations with two operands """
        try:
            if len(self.stack) < 2:
                raise BaseException("Not enough operands on the stack")

            op2 = self.stack.pop()
            op1 = self.stack.pop()
            result = operation(op1, op2)
            self.push(result)
        except BaseException as error:
            print(error)

    def compute_operation_with_one_operand(self, operation):
        """ exec operations with one operand """
        try:
            op1 = self.stack.pop()
            result = operation(op1)
            self.push(result)
        except BaseException as error:
            print(error)

    def compute_operation_with_no_operands(self, operation):
        """ exec operations with no operands """
        try:
            operation()
        except BaseException as error:
            print(error)

As someone said, “That’s one small step for man, one giant leap for mankind.” :)

Note that in this last code we have modified the zero operands functions in the catalog from

"C": self.stack.pop,
"AC": self.stack.clear

to

"C": lambda: self.stack.pop(),
"AC": lambda: self.stack.clear()}

Why?

Well, the problem is that in the *compute *function we are trying to determine the number of parameters from the signature of the method. The problem is that for built-in methods written in C, we can’t do that.

Let’s try it by yourself, start a REPL:

>>> from inpect import signature
>>> a = []
>>> my_sig = signature(a.clear)
Traceback (most recent call last):
 File "<stdin>", line 1, in <module>
 File "C:UsersMASTROMATTEOAppDataLocalContinuumanaconda3libinspect.py", line 3033, in signature
 return Signature.from_callable(obj, follow_wrapped=follow_wrapped)
 File "C:UsersMASTROMATTEOAppDataLocalContinuumanaconda3libinspect.py", line 2783, in from_callable
 follow_wrapper_chains=follow_wrapped)
 File "C:UsersMASTROMATTEOAppDataLocalContinuumanaconda3libinspect.py", line 2262, in _signature_from_callable
 skip_bound_arg=skip_bound_arg)
 File "C:UsersMASTROMATTEOAppDataLocalContinuumanaconda3libinspect.py", line 2087, in _signature_from_builtin
 raise ValueError("no signature found for builtin {!r}".format(func))
ValueError: no signature found for builtin <built-in method clear of list object at 0x000001ED6EB18F88>

as you can see we can’t get the signature of a built-in method.

Our possibilities to solve this problem were:

• Handle this special case in our code, trapping the exception raised when we tried to get the signature for the self.stack.pop() function and the self.stack.clear() function
• Encapsulate the built-in functions in void lambdas, so as to have the signature functions extract the signature from our void lambda function and not from the built-in function contained.

And we have obviously chosen the second possibility since it is the most “Pythonic” we had. :)

That’s all folks. Today’s article has explored some aspect of functions and lambdas in Python and I hope you got the message I wanted to send.

think twice, code once.

Sometimes developers are lazy and don’t think too much at what can mean maintain bad code.

Let’s have a look at the first Peter’s code of the article and try to figure out what could have meant to add the factorial function then. We should have created another function, duplicated more code, and modified the “compute” function, right? With our last code we just need to add a single line to our catalog:

"!": lambda x: math.factorial(x),

Try to think at what could have meant to add another feature to the program for logging all the calculations requested and the given results. We had been supposed to modify a dozen functions of our code to add the feature right? And we would have had to modify as well all the new functions that we will have inserted from now on. Now we can add the feature just in the three methods that really compute the requested calculation depending on the number of the operands requested.

Wait, three methods? Wouldn’t it be possible to have just a method that works regardless of the number of operands that are requested by the function? :)

Happy coding!

D.

Today’s article is about nested code and how to avoid it.

Why we should try to avoid nested code? Well the answer is inside your heart, and in your Python interpreter…

Start your REPL and write:

>>> import this

you will get the “Zen Of Python” by Tim Peters.

The Zen of Python, by Tim Peters

Beautiful is better than ugly.
Explicit is better than implicit.
Simple is better than complex.
Complex is better than complicated.
Flat is better than nested.
Sparse is better than dense.
Readability counts.
Special cases aren't special enough to break the rules.
Although practicality beats purity.
Errors should never pass silently.
Unless explicitly silenced.
In the face of ambiguity, refuse the temptation to guess.
There should be one — and preferably only one — obvious way to do it.
Although that way may not be obvious at first unless you're Dutch.
Now is better than never.
Although never is often better than *right* now.
If the implementation is hard to explain, it's a bad idea.
If the implementation is easy to explain, it may be a good idea.
Namespaces are one honking great idea — let's do more of those!

Now relax, breath, and read carefully all the statements, three times each. Stop at the fifth statement and start meditating about that.

Flat is better than nested Flat is better than nested Flat is better than nested

Do you get it? FLAT IS BETTER THAN NESTED!!

So, let’s start making your program flatter! :)

Tip #1 — List Comprehension

Raise your hand if you have ever written code like this:

my_input_numbers = [1,2,3,4,5,6,7,8]
my_odd_numbers = []

for number in my_input_numbers:
    if number % 2 != 0:
        my_odd_numbers.append(number)

print(my_odd_numbers)

Come on, don’t be shy! Raise your hand! Ok, are you looking at this article with a hand raised?

SHAME ON YOU!

you probably don’t know anything about List Comprehension, do you?

List comprehension is a (great) way of defining a list in Python, just like you’d do in math. For example, by using list comprehension the previous code could be rewritten like this:

my_input_numbers = [1,2,3,4,5,6,7,8]
my_odd_numbers = [x for x in my_input_numbers if x%2 != 0]
print (my_odd_numbers)

Pretty cool uh? And it’s definitely flatter!

But list comprehension makes possible to do something more than just filter out even numbers from a list, for example, we can also manipulate the elements of the list:

my_squared_odd_numbers = [x*x for x in my_input_numbers if x%2 != 0]

Or again, let’s say that you want to create a list of tuples where you will put all the combinations of the (integer) numbers between 1 and 90 taken 2 at a time (quite common if you play the Italian lottery), how can you do this?

ambi = [(x, y) for x in range(1,91) for y in range(x+1,91)]

Yes, that’s so easy! One line of code instead of two-nested-for-loops, that’s it.

Tip #2 — Map, Filter and Reduce

If you have liked list comprehension you will probably like also the three functions we’re going to discuss here.

The map function

map (function, iterable, …)

The “map” function lets you execute a function on all the items of an iterable and returns an iterator with the result of this operation.

For example, let’s write this one:

my_numbers = [1,2,3,4,5,6,7,8,9,10]
my_squared_numbers = map(lambda x:x*x, my_numbers)
print(list(my_squared_numbers))

and we will have as a result:

[1, 4, 9, 16, 25, 36, 49, 64, 81, 100]

Ok, ok, I know that some of you are now thinking “why don’t you use list comprehension to make that?” Yes, it’s true, I could have written something like this:

my_squared_numbers = [x*x for x in my_numbers]
print(my_squared_numbers)

but it’s not the same. Look carefully to the two examples again…

Did you get that?

In the former example, after we used the map *function we had to use the “list” class to print out the results. That’s because “map”* returns an iterator, not a real list.

If you are wondering what are iterators, go and check my previous article about the topic.

The filter function

filter (function, iterable)

The “filter” function lets you filter out a list for certain values that meets a certain condition. Just like the the “map” function, the filter one return an iterator object with the results.

So, for example:

my_numbers = [1,2,3,4,5,6,7,8,9,10]
my_odd_numbers = filter(lambda x: x%2!=0, my_numbers)
print(list(my_odd_numbers))

Like before, if you don’t need an iterable or if you do need a list, you can achieve the same result with list comprehension.

The reduce function

functools.reduce (function, iterable[, initializer])

The “reduce” is the last function we are going to discuss today. It lets you take an iterable and… reduce it literally to a single element by applying a function of two arguments cumulatively to the items of the iterable input. For example:

import functools
my_numbers = [1,2,3,4,5,6,7,8,9,10]
my_sum = functools.reduce(lambda x,y : x+y, my_numbers)
print(my_sum)

And the result will be 55!

That’s great, isn’t it?

As you can see we had to import the functools library to use the reduce function because since the release of Python 3 this function is no more a built-in standard function and it’s now part of the functools library.

Tip #3 — Ternary conditional operator

If you are used to languages like Swift or C# you have probabilly seen the ternary conditional operator before.

For example, in Swift you can write something like this:

let x = 5
// let's get x*x only if x is an odd number...
let squared_if_odd = x%2 == 0 ? x : x*x

this operator is very handy because it avoids you to write code like this:

let x = 5
var squared_if_odd = x

if x%2 != 0 {
    squared_if_odd = x*x
}

This handy ternary conditional operator is available also in Python, even if the syntax it a little different from what we could expect:

x=5
squared_if_odd = x if x%2==0 else x*x

Really convenient and definitely flat!

That’s all folks, go back to your code and start using these techniques; your code will be far more readable from now on.

Enjoy! D.

One of the most underestimated topics that I’ve seen in my working experience is logs management. A lot of people don’t care at all about logging the execution of their programs and I’ve seen a lot of code released in the production environment that doesn’t log anything. To log seems to be a waste of time to them, especially if the code they’re writing is apparently simple. So, why bother logging the execution of a program if the program can run great with no logs?

Actually, logging the execution of your own program avoids you lots of headaches when something goes wrong in production and make your coding experience easier. Besides, logging a lot of debug information can save you time in writing comments, because your code is just well documented by the use of the logging, so to log is always a good idea in production as it is during the development, and that is true for any language you’re using to code, not just in Python.

But Python is not “any language”, Python is a great language that comes with “batteries included”, isn’t it?

In fact, although in other languages you need to write your own logging facility or to download some third-party libraries that solve this problem for you, in Python it just means to use the built-in logging module.

So, what are we waiting for? Let’s start logging in Python!

The first thing that we have to bear in mind is that a log can contain a different kind of messages and each message has its own level of importance. These levels, in Python, are defined in order of verbosity (from the most verbose to the less verbose) as follows:

• DEBUG: Detailed information, typically of interest only when diagnosing problems.- INFO: Confirmation that things are working as expected.
• WARNING: An indication that something unexpected happened, or indicative of some problem in the near future (e.g. ‘disk space low’). The software is still working as expected.
• ERROR: Due to a more serious problem, the software has not been able to perform some function.
• CRITICAL: A serious error, indicating that the program itself may be unable to continue running.

So, each time we want to write something in our log we have to decide what kind of message we are writing to the log, and hence what level it should have.

The second thing that we have to know is how to configure a logger in Python. It’s really easy actually, we just need three components:

• a logger: the object that our code will use to log
• at least one handler: the object that will handle the writing of our log to the target device
• a formatter: an object that defines the format of our log for the handler

Ok, too much talk, let’s start coding.

Let’s start by importing the logging module:

import logging

Then, let’s create the the logger:

logger=logging.getLogger(__name__)

as you can see, to create a logger we just have to call the “getLogger” method of our logging module passing to it a name for the logger. Usually, it’s a good idea to use the name special variable to specify the name of the logger. Doing this you will have different loggers for different modules by design, and you will be able to configure differently every single logger.

Now we have to specify the default log level of our logger. This configuration ensures you that just the log messages with this level or with a less verbose level will be recorded. This enables you to have a super detailed log with all the debug information when you are developing or testing your application and a less verbose log when you deploy your program to the production environment.

# define the default level of the logger. 
# We could specify a greater (LESS VERBOSE) level on the single handler 
logger.setLevel(logging.DEBUG)

Now we have to create our second object: the formatter. As we said before, the formatter is the object that specifies the format we want for our log. In this example, I’m creating a formatter that will output every single message specifying its timestamp and its log level. There are plenty of variables you can use to configure the format of your log, to have a complete list check the official documentation.

# creating a formatter. 
# to see all the attributes you can use
formatter=logging.Formatter('%(asctime)s | %(levelname)s -> %(message)s')

Now we need to create the handler (or the handlers) that will take care of writing our logs to the target device. Usually I create a file handler to log on a file on the filesystem all the information with an INFO level or greater (it’s a must for the production environment) and a stream handler to log every single information also to the console (it’s quite convenient while I’m coding and prevent me to misuse the “print” function). Note that we can specify a different level for each handler unless it is greater or equal to the main level specified for the logger. Even the format can be different (we could need different information for the console and for the log file for example).

# creating a handler to log on the filesystem
file_handler=logging.FileHandler('mylogfile.log')
file_handler.setFormatter(formatter)
file_handler.setLevel(logging.INFO)

# creating a handler to log on the console
stream_handler=logging.StreamHandler()
stream_handler.setFormatter(formatter)
stream_handler.setLevel(logging.INFO)

Ok, we have almost finished now, we just need to put everything together…

# adding handlers to our logger
logger.addHandler(stream_handler)
logger.addHandler(file_handler)

and that’s it! We have configured our logger. It wasn’t a pain, was it? Now, to log a message we will just need to call the “info”, “warning”, “error”, “debug”, or “critical” method passing the message we want to log as a parameter. For example:

logger.info('this is a log message...')

and we’ll get:

2017-08-29 12:41:42,154 | INFO -> this is a log message...

Now, if we were creating a program to divide two given numbers and log the result, we could have done something like this:

a=50
b=10

try:
    c=a/b
    logger.info(f"Operation {a}/{b} gave the result {c}")
except:
    logger.error(f"Error occured during the division of {a} and {b}")

And the output would be:

2017-08-29 12:41:42,204 | INFO -> Operation 50/10 gave the result 5.0

It’s worth to be said that to log the exceptions there’s also an “exception” method of the logger that write a message with the “ERROR” level but add to it also some more information from the exception, like the stacktrace and the exception message. So for example, in this example it was far better to write something like this:

a=50
b=0

try:
    c=a/b
    logger.info(f"Operation {a}/{b} gave the result {c}")
except:
    logger.error(f"Error occured during the division of {a} and {b}")

And the output would be:

2017-08-29 12:41:42,242 | ERROR -> Error occured during the division of 50 and 0
Traceback (most recent call last):
  File "<ipython-input-10-9b4751150eb5>", line 5, in <module>
    c=a/b
ZeroDivisionError: division by zero

As always, for more information and for advanced topics I encourage you to check the official documentation and the original PEP.

That’s all folks, happy coding and… happy logging! :)

D.