HTTP Traffic Monitoring

& Transaction Logging Guide

Framework-Agnostic Network Tapping Techniques

Using Bash, tcpdump, ngrep, mitmproxy & tshark

Prepared by: Dr. Mohd Hanif Mohd Ramli

TXIO Fusion Solutions / Eagle Attech Sdn Bhd

March 2026

Ilahi anta maqsudi wa ridhaka matlubi, a’tini mahabbataka wa ma’rifataka

1. Overview

This guide documents techniques for capturing and logging HTTP transaction data from web applications running on a server. All approaches are framework-agnostic — they operate at the network level, meaning they work regardless of whether the application uses Fastify, Express, Django, PHP, or any other stack.

The core principle is simple: listen on the port or network interface, not the application layer. If HTTP traffic flows through the machine, it can be captured.

1.1 When to Use These Techniques

• Debugging API calls between services without modifying application code
• Logging transaction data (requests, responses, headers, payloads) to .txt files
• Monitoring traffic on servers where the framework or port is unknown
• Continuous background capture with automatic log rotation
• Security auditing and traffic analysis

2. Nginx Access Log Parsing

If your applications already run behind Nginx (e.g. ARBITER, GRADSENSE, SCANIX), this is the simplest and least invasive approach. Configure a custom log format to capture detailed request information.

2.1 Nginx Configuration

Add a detailed log format to your Nginx config:

log_format detailed '$remote_addr - $remote_user [$time_local] '

'"$request" $status $body_bytes_sent '

'"$http_referer" "$http_user_agent" '

'rt=$request_time urt=$upstream_response_time';

access_log /var/log/nginx/transactions.log detailed;

2.2 Bash Listener Script

Use tail -F to continuously follow the log and write to daily .txt files:

#!/bin/bash

LOGFILE="/var/log/nginx/access.log"

OUTDIR="/home/hanif/transactions"

mkdir -p "$OUTDIR"

tail -F "$LOGFILE" | while IFS= read -r line; do

DATEFILE="$OUTDIR/$(date +%Y%m%d).txt"

echo "$line" >> "$DATEFILE"

done

TIP: The -F flag (capital F) follows the file by name, so it survives log rotation.

3. mitmproxy (Full Request/Response Capture)

mitmproxy is the most powerful option for capturing complete HTTP transactions including headers and body content. It sits between your client and application as a reverse or forward proxy.

3.1 Reverse Proxy Mode (Known Port)

Place mitmproxy in front of a specific application:

# Install

pip3 install mitmproxy

# Run as reverse proxy (e.g. app on port 3000)

mitmdump -p 8080 --mode reverse:http://localhost:3000 \

--set flow_detail=2 2>&1 | tee -a transactions.txt

Then point Nginx upstream to port 8080 instead of 3000.

3.2 Forward Proxy Mode (Unknown Endpoints)

Route outbound traffic through a local proxy to capture calls to external APIs:

# Start proxy

mitmproxy --mode regular -p 9090

# Set environment variables

export http_proxy=http://localhost:9090

export https_proxy=http://localhost:9090

# All HTTP calls now go through the proxy

curl https://someapi.com/v1/orders

TIP: For HTTPS traffic, mitmproxy can decrypt content if you install its CA certificate on the client.

4. Network-Level Capture (tcpdump / ngrep)

These tools capture traffic at the packet level. They require no knowledge of the application framework, port, or endpoint. They work on any Linux or macOS server.

4.1 tcpdump — Capture by Port

#!/bin/bash

PORT=${1:-80}

OUTDIR="/home/hanif/captures"

mkdir -p "$OUTDIR"

OUTFILE="$OUTDIR/$(date +%Y%m%d_%H%M%S)_port${PORT}.txt"

echo "Listening on port $PORT..."

sudo tcpdump -A -s 0 -i any "tcp port $PORT" >> "$OUTFILE"

4.2 tcpdump — Capture by Host (Known Endpoint)

When you only know the endpoint URL, resolve its IP and sniff:

#!/bin/bash

ENDPOINT="someapi.example.com"

IP=$(dig +short "$ENDPOINT" | head -1)

echo "Resolved $ENDPOINT -> $IP"

sudo tcpdump -A -s 0 -i any "host $IP" | tee -a capture.txt

4.3 ngrep — Clean HTTP Output

ngrep produces much cleaner, human-readable HTTP output compared to tcpdump:

# Listen on a specific port

sudo ngrep -q -W byline -d any '' port 8080 | tee -a capture.txt

# Listen to ALL HTTP traffic on any port

sudo ngrep -q -W byline -d any 'HTTP'

# Filter for specific endpoint paths

sudo ngrep -q -W byline -d any 'GET /v1/orders|POST /v1/orders'

Sample ngrep output showing a full request/response cycle:

T 192.168.1.5:54321 -> 127.0.0.1:8080 [AP]

POST /api/login HTTP/1.1

Host: localhost:8080

Content-Type: application/json

{"username":"admin","password":"test123"}

T 127.0.0.1:8080 -> 192.168.1.5:54321 [AP]

HTTP/1.1 200 OK

Content-Type: application/json

{"token":"eyJhbGci...","status":"ok"}

5. tshark (Wireshark CLI)

tshark provides the most structured output with parsed HTTP fields, ideal for automated processing:

sudo apt install tshark

sudo tshark -i any -f "tcp port 3000" \

-T fields \

-e frame.time \

-e ip.src \

-e ip.dst \

-e http.request.method \

-e http.request.uri \

-e http.response.code \

-e http.content_type \

-E separator='|' | tee -a transactions.txt

Output is pipe-delimited for easy parsing:

2026-03-10 14:22:01|192.168.1.5|127.0.0.1|POST|/api/order||application/json

2026-03-10 14:22:01|127.0.0.1|192.168.1.5|||200|application/json

6. Port & Service Discovery

When you don’t know which ports are serving HTTP, use this discovery script:

#!/bin/bash

echo "=== Active listening ports ==="

ss -tlnp | grep LISTEN

echo ""

echo "=== HTTP services detected ==="

for port in $(ss -tlnp | grep LISTEN | \

awk '{print $4}' | grep -oP '\d+$'); do

response=$(curl -s -o /dev/null -w "%{http_code}" \

--connect-timeout 1 http://localhost:$port 2>/dev/null)

if [ "$response" != "000" ]; then

echo "Port $port -> HTTP $response"

fi

done

7. Smart Tap — Universal Endpoint Listener

This all-in-one script accepts any URL and automatically resolves the target, extracts the path, and begins capturing:

#!/bin/bash

# smarttap.sh

URL="$1"

if [ -z "$URL" ]; then

echo "Usage: ./smarttap.sh https://someapi.com/api/endpoint"

exit 1

fi

DOMAIN=$(echo "$URL" | awk -F/ '{print $3}')

PATH_FILTER=$(echo "$URL" | sed 's|https\?://[^/]*||')

IP=$(dig +short "$DOMAIN" | head -1)

OUTFILE="tap_$(echo $DOMAIN | tr '.' '_')_$(date +%Y%m%d).txt"

echo "Domain : $DOMAIN"

echo "IP : $IP"

echo "Path : $PATH_FILTER"

echo "Output : $OUTFILE"

echo "---"

sudo ngrep -q -W byline -d any "$PATH_FILTER" "host $IP" | \

tee -a "$OUTFILE"

Usage:

./smarttap.sh https://someapi.com/v1/orders

8. Continuous Listening & Persistence

8.1 nohup (Quick Background)

nohup ./listen_transactions.sh &

8.2 tmux / screen (Interactive Session)

tmux new -s httptap

./listen_transactions.sh

# Ctrl+B then D to detach

# tmux attach -t httptap to reconnect

8.3 systemd Service (Production — Survives Reboots)

Create a service file at /etc/systemd/system/http-tap.service:

[Unit]

Description=HTTP Transaction Logger

After=network.target nginx.service

[Service]

Type=simple

User=hanif

ExecStart=/home/hanif/scripts/listen_transactions.sh

Restart=always

RestartSec=5

StandardOutput=journal

StandardError=journal

[Install]

WantedBy=multi-user.target

Enable and manage the service:

sudo systemctl daemon-reload

sudo systemctl enable http-tap

sudo systemctl start http-tap

sudo systemctl status http-tap

journalctl -u http-tap -f

8.4 Log Rotation

Prevent log files from growing indefinitely. Create /etc/logrotate.d/http-tap:

/home/hanif/transactions/*.txt {

daily

rotate 30

compress

missingok

notifempty

dateext

}

This retains 30 days of logs and compresses older files automatically.

9. Quick Reference

Summary of tools and recommended use cases:

TIP: For your Mac Mini cluster setup, the systemd + ngrep combination is recommended for persistent, framework-agnostic monitoring across ARBITER, GRADSENSE, and SCANIX.

Ilahi anta maqsudi wa ridhaka matlubi, a’tini mahabbataka wa ma’rifataka