PythonPro #30: Hands-On Microservices with Django, High Impact Streamlit, and Python Data Science Handbook
Bite-sized actionable content, practical tutorials, and resources for Python programmers and data scientists
Welcome to a brand new issue of PythonPro!
In today’s Expert Insight we bring you an excerpt from the recently published book, Hands-On Microservices with Django, which teaches you how to implement and manage a Django-based microservice architecture, utilizing RabbitMQ for task distribution.
News Highlights: Researchers uncover critical vulnerabilities in Python AI package, and Codemap offers powerful, privacy-focused code visualizations.
Here are my top 5 picks from our learning resources today:
High Impact Python Streamlit - Beautiful Interactive Maps and Charts🗺️
Safely batch byte-pair merges to speed up tokenization in Python 30X⚡
Dive in, and let me know what you think about this issue in this month’s survey!
Stay awesome!
Divya Anne Selvaraj
Editor-in-Chief
P.S: Our monthly survey is still in progess. If you haven’t yet, we invite you to take the opportunity to tell us what you think about PythonPro so far, request a learning resource for future issues, tell us what you think about a recent development in the Python world, and earn a Packt Credit each month to buy a book of your choice. Take the survey!
🐍 Python in the Tech 💻 Jungle 🌳
🗞️News
Flaws in Python Package for AI Models Uncovered: Researchers have identified critical vulnerabilities in a Python package used for AI models and PDF.js, a library for rendering PDFs in Firefox. Read for more.
Codemap – A Powerful Code Visualizer: Codemap is a tool that automatically visualizes the function call stack of any codebase as an interactive graph, supporting multiple programming languages including Python and OSes while ensuring privacy by running locally. Read to learn more.
💼Case Studies and Experiments🔬
Can LLMs find bugs in large codebases?: A new benchmark called Bug In The Code Stack (BICS) has been developed to evaluate how LLMs detect syntactic bugs in Python codebases. Read to learn about the limitations and strengths of different LLMs in handling complex code analysis tasks.
A Whimsical Journey Through Wait Times: This artilcle explores different types of waiting experiences—like being on hold, microwaving popcorn, and waiting for a lottery win—through a whimsical narrative that integrates Python code and data analysis. Read to learn how different wait times can be analyzed and predicted using Python.
📊Analysis
Why is NumPy faster than python in data processing from a binary machine perspective: This article explains the efficiency of numerical programming libraries like NumPy over OOP in Python, especially for processing large datasets. Read to understand how NumPy works.
Bird by Bird Using Finite Automata: This article discusses using finite state machines (FSM) in Python to analyze and simulate complex behaviors in a system designed for detecting and managing bird intrusions on a lawn. Read to learn the practical applications of FSM for creating robust simulations.
🎓 Tutorials and Guides 🤓
High Impact Python Streamlit - Beautiful Interactive Maps and Charts: This step-by-step guide on creating a Streamlit dashboard uses UNFAO global food insecurity data. Read to learn how to handle and visualize large datasets, and apply these techniques to real-world data analysis scenarios.
📖Open Access Book | Python Data Science Handbook: Covering topics from IPython tools, NumPy, Pandas, and Matplotlib, to advanced ML techniques with Scikit-Learn, this resource integrates detailed instructions with practical examples. Read to learn practical data science skills including data manipulation and visualization.
📖Open Access Book | Introduction to Scientific Programming with Python: This book focuses on the practical aspects of Python for computational and scientific applications. Read to gain skills in data processing, mathematical modeling, and basic computer science principles.
GPT-4o - Learn how to Implement a RAG on the new model, step-by-step!: This article guides you through using OpenAI's new, more affordable GPT-4o model to implement a RAG using Langchain's talkdai/dialog. Read to learn how to set up and use GPT-4o for enhanced content generation.
Aggregating Real-time Sensor Data with Python and Redpanda: This article illustrates a method to downsample high-frequency sensor temperature readings into a lower resolution output every 10 seconds. Read to learn how to implement a basic stream processing pipeline using Python.
🎥Kafka Consumers in Python - A Walkthrough: This video tutorial explains how to extract data from Kafka using Python, following up on a previous session that covered data insertion. Read to learn how to efficiently read data from Kafka using Python.
Building Apps with Streamlit: This article highlights Streamlit's power for building applications that require user input and real-time data processing. Read to learn about Streamlit's capabilities for Python-based UI development.
🔑Best Practices, Advice, and Code Optimization🔏
The Hitchhiker’s Guide to Python!: This practical guide aimed at both novice and expert Python developers, offers best practices for Python installation, configuration, and daily usage. Read to enhance your programming and project management skills.
Safely batch byte-pair merges to speed up tokenization in Python 30X: This notebook presents iterative enhancements to Andrej Karpathy's tokenization tutorial, showing a progression from 5X to over 31X speed improvement. Read to gain insights into practical optimization techniques for tokenization in Python.
Basic Data Types in Python - A Quick Exploration: This article provides an overview of Python's fundamental data types including integers, floating-point numbers, complex numbers, strings, bytes, bytearrays, and Booleans. Read to enhance your understanding of how to efficiently use these types in programming.
Python's Built-in Exceptions - A Walkthrough With Examples: This comprehensive guide to Python's built-in exceptions, details the variety and structure of errors Python can handle, from syntax and import-related errors to file and arithmetic exceptions. Read to gain a solid understanding of Python's exception hierarchy.
The Ultimate Guide to Open-Source Security with Python and R: This guide explores the integration of open-source security in Python and R projects, addressing risks and best practices. Read to discover practical guidelines for managing risks and enhancing system integrity in your Python and R environments.
Take the Survey, Get a Packt credit!
🧠 Expert insight 📚
Here’s an excerpt from “Chapter 2: Introducing the Django Microservices Architecture” in the book, Hands-On Microservices with Django by Tieme Woldman, published in May 2024.
Creating a sample microservice
…(In the following excerpt) we’ll explore a microservice implementation with RabbitMQ to understand how message queueing technically works.…
The Django developer community often uses Celery for task queue management
because it’s simple and effective since Celery abstracts the technical details of the underlying message queuing broker.
Nevertheless, it’s valuable to know the inners of message queueing with RabbitMQ and Redis as well because they serve as a vehicle for Celery, and you might encounter situations where you prefer using RabbitMQ or Redis directly.
…Our sample Django app asks visitors to send a suggestion for improvement. It opens with this form:
Figure 2.3 – The Improvement Suggestion app
After filling out the form and clicking Send, the app offloads a task to send a confirmation email to the user. This way, the user experience remains smooth, even if sending the mail takes 10 seconds…
Offloading a task with a RabbitMQ microservice
RabbitMQ has several methods for sending a message from a producer to a consumer/worker. Chapter 6, Orchestrating Microservices With Celery and RabbitMQ, discusses these methods in more detail. Here, we suffice with the work queue method where RabbitMQ distributes the tasks among multiple workers.
You can find the code for this implementation on GitHub here.
In the upcoming subsections, we’ll address the following:
The directory structure and settings for the Django project.
The producer file, which off loads the tasks for sending mails.
The worker file, which sends the confirmation email.
The form and the view for the Django app.
Directory structure
This is the main directory structure for the Django project with the relevant files:
impl1/
│
├── django_rabbitmq/
│ └── settings.py
│
├── scripts/
│ ├── __init__.py
│ └── worker.py
│
├── suggestion/
│ ├── templates/suggestion
│ │ └── suggestion.html
│ │
│ ├── forms.py
│ ├── producer.py
│ └── views.py
│
└── manage.py
The structure only shows the files which are relevant for the microservice handling. Also, the venv-directory is left out but required, of course.
Django-extensions and the scripts directory
We’ll get to the django_rabbitmq directory in a moment, but first, we‘ll look at the scripts directory. To run the consumer/worker that sends the email within the Django context, we enable it as a Django extension via the django-extensions package. This package assumes a scripts directory, including an empty __init__.py file, where we install extensions like our email worker.
Settings
This also brings us to settings.py, where the following stands out:
INSTALLED_APPS = [
...
"django_extensions",
"suggestion.apps.SuggestionConfig",
]
EMAIL_BACKEND = "django.core.mail.backends.console.EmailBackend"
The django-extensions package must be defined as an installed app, like the suggestion app. We’ll set the console as the email backend to quickly check sent emails without the hustle of setting up a mail server or using external mail servers.
The form and the producer
The suggestion form from Figure 2.3 derives from this set up for the forms.py file:
1 from django import forms
2 from suggestion.producer import send_email_task_message
3
4 class SuggestionForm(forms.Form):
5 name = forms.CharField(label="Your name")
6 email = forms.EmailField(label="Email")
7 suggestion = forms.CharField(
label="Your suggestion",
widget=forms.Textarea(attrs={"rows": 7})
)
8
9 def send_email(self):
10 task_message = {
"name": self.cleaned_data["name"],
"email": self.cleaned_data["email"],
"suggestion": self.cleaned_data["suggestion"]
}
11 send_email_task_message(task_message)
Line 2 imports the producer to enable sending a task message to a consumer/worker.
Lines 9-11 extend the straightforward class definition for the suggestion form with the send_email method.
Line 10 creates the task message in JSON format to send to the message_queue, and the task message contains parameters for the task to process.
In the end, line 11 calls the send_email_task_message function in the producer.py file, which looks like this:
1 import json
2 import pika
3
4 connection =
5 pika.BlockingConnection(pika.ConnectionParameters(
6 host='localhost',
7 credentials=pika.PlainCredentials("myuser",
"mypassword")))
8
9 channel = connection.channel()
10 channel.queue_declare(queue='mail_queue',
durable=True)
11
12 def send_email_task_message(task_message):
13 channel.basic_publish(exchange= "",
routing_key='mail_queue',
body=json.dumps(task_message),
properties=pika.BasicProperties(
delivery_mode=
pika.spec.PERSISTENT_DELIVERY_MODE)
)
The microservices architecture starts here by utilizing the producer, which opens the task message queue. To do so, line 2 imports the pika package, which implements the AMQP protocol communicating with RabbitMQ.
Lines 4-7 open the connection with RabbitMQ. The credentials from line 7 are for demoing only. Next, lines 9-10 create a task message queue named mail_queue.
Line 12 sets the earlier mentioned send_email_taskmessage function.
Line 13 sends a task message to the mail_queue.
Template
The accompanying form template, suggestion.html, is straightforward:
{% extends 'suggestion/base.html' %}
{% block content %}
<h1>Tell us how we can improve</h1>
<form method="post">
{% csrf_token %}
{{ form.as_p }}
<input type="submit" value="Send" />
</form>
{% endblock content %}
This template extends the base.html template and formats the form as HTML paragraphs.
The view
From Django’s perspective, this views.py file steers the form:
from django.views.generic.base import TemplateView
from django.views.generic.edit import FormView
from suggestion.forms import SuggestionForm
class SuggestionFormView(FormView):
template_name = "suggestion/suggestion.html"
form_class = SuggestionForm
success_url = "/success/"
def form_valid(self, form):
form.send_email()
return super().form_valid(form)
class SuccessView(TemplateView):
template_name = "suggestion/success.html"
Here, the form_valid method calls the send_email method, which sets off the process of sending a task message to the task message queue.
The email consumer/worker
Finally, the consumer/worker in worker.py listens to the mail_queue and sends an email when a new task message arrives:
1 import json
2 import pika
3 from django.core.mail import send_mail
4 from time import sleep
5
6 connection =
7 pika.BlockingConnection(pika.ConnectionParameters(
8 host='localhost',
9 credentials=pika.PlainCredentials("myuser",
"mypassword")))
10
11 channel = connection.channel()
12 channel.queue_declare(queue='mail_queue',
durable=True)
13
14 def callback(ch, method, properties, body):
15 task_message = json.loads(body)
16 sleep(10)
17 send_mail(
18 "Your suggestion for improving",
f"We'll include your suggestion –
{task_message['suggestion']} –
into our improvement process.\n\n
Thanks for your contribution!",
"quality@xyz.com",
[task_message['email']],
fail_silently=False,
)
19 ch.basic_ack(delivery_tag=method.delivery_tag)
20
21 def run():
22 channel.basic_qos(prefetch_count=1)
23 channel.basic_consume(queue='mail_queue',
on_message_callback=callback)
24 channel.start_consuming()
Line 3 imports the Django send_mail method.
Lines 6-12 connect the worker to the mail_queue.
Lines 14-19 set the callback function, which runs every time a new task message arrives. This function parses the received task message and emulates a delay in line 16 to emphasize the asynchronous processing of the worker.
Line 21 sets the run function, which enables the worker to act as a Django extension and run inside Django’s context.
Finally, lines 22-24 cause the worker to listen to the mail_queue and call the callback function when a new task message arrives.
RabbitMQ as message queue broker
Besides the producer and consumer/worker, this implementation requires a RabbitMQ instance, which handles the queues and messages. You can run RabbitMQ by downloading and installing it from the RabbitMQ website. Or you can run RabbitMQ from a Docker container... Here, we assume RagbitMQ runs in a container, though it makes no difference for the Django code.
Putting it to work
Okay, we have everything prepared. We activate our virtual environment from a terminal shell and launch the Improvement Suggestion app, including the producer:
$ python3 manage.py runserver
And in a second terminal, we launch the worker as a Django extension:
$ python3 manage.py runscript worker
This sets the RabbitMQ worker to wait for new messages:
Figure 2.4 – Worker waiting prompt
If we now open a browser, navigate to http://127.0.0.1:8000/suggestion, and fill out the form, the app executes these steps:
The form_valid method in views.py fires the send_email method from forms.py.
The app presents the success form.
The send_email method in forms.py composes the task message, calls the send_email_task_message function in producer.py, and passes the message.
The send_email_task_message function in producer.py sends the passed task message to the mail_queue.
The consumer/worker detects a new task message and then launches the callback function to parse the task message and send the confirmation email. After the simulated delay, the worker terminal shows the email:
Figure 2.5 – Email to console (RabbitMQ)
There you have it, the first implementation of a Django microservice.
You can buy Hands-On Microservices with Django by Tieme Woldman, here. Packt library subscribers can continue reading the entire book for free here.
And that’s a wrap.
We have an entire range of newsletters with focused content for tech pros. Subscribe to the ones you find the most useful here. The complete PythonPro archives can be found here.
If you have any suggestions or feedback, or would like us to find you a Python learning resource on a particular subject, take the survey or just respond to this email!