Python Application Performance Monitoring (APM) using Elastic

• Real time monitoring of microservices performance and database queries

• Service dependency mapping and performance bottleneck identification

• Custom dashboards in Kibana for visualizing metrics

• Error logging and alerts for application issues and database errors

Table of Contents

Part 1: Architecture Overview

• Application layers:
• Frontend: A basic web interface
• Backend: Python-based API that serves as the business logic layer
• Database: PostgreSQL for storing application data

• Elastic Stack:
• Elasticsearch: Stores APM data and other logs
• Kibana: Provides visualization for real-time monitoring, service dependencies, and dashboards
• APM Server: Collects and transmits APM metrics from the application

Part 2: Setting up the Python-PostgreSQL Application

2.1 Setup

$ mkdir docker-python-sql
$ cd docker-python-sql
$ mkdir frontend
$ mkdir backend
$ mkdir database

2.2 Database

# Use the official PostgreSQL image as the base image
FROM postgres:15.9

# Set the default password for the PostgreSQL user
ENV POSTGRES_PASSWORD=secret

# Set the default username for PostgreSQL
ENV POSTGRES_USER=username

# Set the default database name for PostgreSQL
ENV POSTGRES_DB=database

# Copy the SQL script to initialize the database with fixtures
COPY create_fixtures.sql /docker-entrypoint-initdb.d/create_fixtures.sql

2.3 Backend

FROM python:3.10

# Set work directory
WORKDIR /app

# Install required dependencies
COPY requirements.txt .
RUN pip install -r requirements.txt

# Copy the application code into the container
COPY . .

# Run the automation service
CMD ["python", "app.py"]

psycopg2-binary==2.9.7
xhtml2pdf==0.2.11
watchdog==2.1.6  # To monitor the input folder for new HTML files

import os
import time
import psycopg2
from xhtml2pdf import pisa
from watchdog.observers import Observer
from watchdog.events import FileSystemEventHandler

# Database configuration
db_config = {
    'dbname': os.getenv('POSTGRES_DB'),
    'user': os.getenv('POSTGRES_USER'),
    'password': os.getenv('POSTGRES_PASSWORD'),
    'host': os.getenv('POSTGRES_HOST')
}

# Define the directories
input_dir = '/input'
output_dir = '/output'

# Function to convert HTML to PDF
def convert_html_to_pdf(html_file):
    pdf_file = os.path.join(output_dir, os.path.basename(html_file).replace('.html', '.pdf'))
    
    with open(html_file, "r") as html:
        with open(pdf_file, "wb") as pdf:
            pisa.CreatePDF(html, dest=pdf)
    
    return pdf_file

# Function to track converted files in the database
def track_converted_file(file_name):
    try:
        conn = psycopg2.connect(**db_config)
        cur = conn.cursor()
        cur.execute("INSERT INTO converted_files (file_name) VALUES (%s);", (file_name,))
        conn.commit()
        cur.close()
        conn.close()
    except Exception as e:
        print(f"Error tracking file: {e}")

# Watchdog event handler
class Watcher(FileSystemEventHandler):
    def on_created(self, event):
        if event.is_directory:
            return
        if event.src_path.endswith(".html"):
            print(f"New HTML file detected: {event.src_path}")
            pdf_file = convert_html_to_pdf(event.src_path)
            track_converted_file(pdf_file)
            print(f"Converted to: {pdf_file}")

# Set up the observer to monitor the input folder
observer = Observer()
observer.schedule(Watcher(), input_dir, recursive=False)
observer.start()

print("Monitoring the input directory for new HTML files...")

try:
    while True:
        time.sleep(1)
except KeyboardInterrupt:
    observer.stop()

observer.join()