Finding Bugs via Patch Diffing

Finding Bugs via Patch Diffing

Overview

Patch diffing is a powerful vulnerability research technique that analyzes the differences between vulnerable and patched versions of software. When vendors release security updates, the patches themselves reveal where the bugs were located. This week, you'll learn to systematically find vulnerabilities by comparing binary versions, understanding what changed, and determining how to exploit the original bug.

This builds on the vulnerability classes you learned in Week 1. While Fuzzing (Week 2) finds bugs by throwing data at targets, Patch Diffing finds bugs by analyzing the vendor's own fixes. Next week (Week 4), we'll learn how to analyze the crashes you find to determine exploitability.

Why Patch Diffing Matters:

  • Single source of truth when CVE details are limited

  • Discover variant vulnerabilities in the same code area

  • Build exploit development skills through focused practice

  • Understand vendor patching patterns and priorities

Real-World Impact:

Day 1: Patch Diffing Theory and Windows Update Extraction

What is Patch Diffing?

Definition: Patch diffing is the technique of comparing a vulnerable version of a binary with a patched version to identify security-related changes. By analyzing what the vendor fixed, we can:

  1. Identify the vulnerability location - Where in the code was the bug?

  2. Understand the root cause - What programming mistake led to the bug?

  3. Develop exploitation techniques - How can the bug be triggered and exploited?

  4. Find variant bugs - Are there similar bugs in related code?

Benefits:

  • Single Source of Truth: Without CVE details or PoC, the patch itself reveals what was broken

  • Vulnerability Discovery: While analyzing one fix, you may find additional bugs nearby

  • Skill Development: Provides focused practice in reverse engineering with known targets

  • Vendor Insight: Learn how different vendors approach security fixes

Challenges:

  • Asymmetry: Small source code changes can drastically affect compiled binaries

  • Finding Security Changes: Patches often bundle security fixes with features and bug fixes

  • Noise Reduction: Must distinguish security-relevant changes from benign updates

  • Tool Limitations: No tool perfectly automates the process; human analysis is essential

  • Patch-Introduced Bugs: Patches can introduce NEW vulnerabilities (see CVE-2025-59287 case study)

  • Feature Flags: Security fixes often come with feature toggles that complicate analysis

Windows Update Structure

Understanding .msu Files:

  • .msu = Microsoft Update Standalone Package

  • Contains one or more .cab (Cabinet) files

  • Nested structure: .msu.cab.cab → actual binaries/manifests

Types of Windows Updates:

Type
Description
Patch Diffing Consideration

Cumulative Update

Contains all previous fixes

Large, many changes to filter

Security Update

Specific security fixes only

Smaller, more focused

Servicing Stack

Update installer itself

Rarely security-relevant

Delta Update

Only changes since last update

Requires base + delta

Express Update

Optimized differential

Complex extraction

Delta vs Full Patches:

Microsoft uses two patching mechanisms:

  1. Full Replacement: Entire binary replaced

    • Easier to diff (compare old vs new directly)

    • Larger download size

  2. Forward Differential (.psf files): Only changed bytes

    • Requires applying patch to base to get full binary

    • Tool: delta.exe from Windows SDK

    • More complex extraction workflow

Extraction Process:

Automated Script

The Extract-Patch.ps1 script handles all Windows Update formats and automatically falls back to WinbIndex for newer WIM+PSF updates:

Script Usage Examples

Handling Delta/Differential Updates:

Some Windows updates (especially recent cumulative updates like KB5070312 and KB5068861) use Forward Differential compression. These updates contain:

Understanding the WIM+PSF Format (Windows 11 24H2+):

This newer update format is fundamentally different from older Windows updates:

Component
Contents
Can Extract Binaries?

.psf file

Binary delta patches (differences only)

No - requires base files

.wim file

Manifests, catalogs, metadata

No - no binaries inside

SSU-*.cab

Servicing Stack Update binaries

SSU files only

*.msix files

UWP app packages

App binaries only (not kernel/system)

*.msix.rif.cab

MSIX metadata

No - just XML/catalogs

Finding the Right Binary Versions

Using WinbIndex:

  1. Search for binary name (e.g., ntdll.dll, tcpip.sys)

  2. View version history with KB numbers

  3. Download specific versions directly

Example - Finding ntdll.dll Versions:

Identifying Changed Files:

  • MSRC security bulletins list affected binaries

  • GitHub security advisories often specify components

  • CVE descriptions may mention specific DLLs/drivers

Symbol Files and PDB

Why Symbols Matter:

  • Function names make analysis infinitely easier

  • Variable names provide context

  • Structure definitions reveal data layouts

  • Simplified debugging and correlation

Downloading Symbols:

Practical Exercise

Task: Extract and prepare two consecutive Windows 11 updates for diffing

  1. Example Analysis:

  1. Identify Target:

    • Choose a recent security update (check MSRC Security Update Guidearrow-up-right)

    • Find the KB number and affected binaries

    • Example targets:

      • CVE-2025-60720: for example you need to download tdx.sys KB5067036 as the vulnerable version and KB5068861 for the patched version

      • CVE-2025-60707: mmcss.sys patched at KB5068861, find the vulnerable version

      • CVE-2025-59255: figure out binary, patched at KB5066835, vulnerable version at KB5065789

      • CVE-2025-59192: find vulnerable binary and patched, vulnerable versions

      • CVE-2025-55224: find vulnerable binary and patched, vulnerable versions

      • CVE-2025-62452: find vulnerable binary and patched, vulnerable versions

  2. Download Updates:

    • find the file that you need to download

    • and vulnerable and patched kb versions

  3. Extract Binaries:

  4. Organize Files:

Success Criteria:

  • Both binary versions extracted

  • File sizes confirmed different

  • Symbols downloaded successfully

  • Directory structure organized for next steps

Key Takeaways

  1. Patch diffing reveals vulnerabilities: The patch is often the only source of truth

  2. Extraction is multi-layered: .msu → .cab → .cab → binaries

  3. Symbols are essential: Makes diffing practical and interpretable

  4. Organization matters: Clean directory structure prevents confusion

  5. Automation saves time: Scripts for extraction and symbol download are invaluable

Discussion Questions

  1. Why might vendors release patches without detailed CVE write-ups?

  2. What are the ethical considerations of patch diffing before a patch is widely deployed?

  3. How do delta patches complicate the extraction process?

  4. What strategies can vendors use to make patch diffing harder for attackers?

Day 2: Binary Diffing Tools (BinDiff and Ghidriff)

Tool Choice: IDA + Diaphora vs Ghidra + Ghidriff

IDA Pro + Diaphora:

  • Pros:

    • Industry standard IDA with excellent decompilation

    • Diaphora is free, open-source, and actively maintained

    • Unique features: microcode diffing, vulnerability detection, pseudo-code patches

    • Rich ecosystem of IDA plugins and scripts

  • Cons:

    • IDA Pro is expensive ($1,000+ for personal license)

    • IDA Free has limitations (no Hex-Rays decompiler)

    • Diaphora uses AGPL license (affects commercial use)

  • Best For: Professional vulnerability researchers with IDA licenses

IDA Pro + BinDiff 8 (Legacy):

  • Status: BinDiff 8 only supports IDA 8.0-8.3, NOT IDA 9.x

  • Alternative: Build from source with newer IDA SDK (requires effort)

Ghidra + Ghidriff:

  • Pros:

    • Completely free and open-source (Ghidra and Ghidriff)

    • Excellent multi-architecture support

    • Built-in version tracking tool in Ghidra

    • Ghidriff automates diffing with markdown/JSON reports

    • Headless analysis perfect for CI/CD pipelines

    • Docker support for reproducible analysis

  • Cons:

    • Decompiler slightly less polished than Hex-Rays

    • Steeper learning curve for GUI

    • Fewer third-party plugins than IDA ecosystem

  • Best For: Budget-conscious researchers, automation needs, open-source preference

This Course: Primary focus on Ghidra + Ghidriff for accessibility. Diaphora workflows covered for those with IDA access.

Installing IDA + BinDiff

Option 1: Use IDA Pro 8.x with BinDiff 8

Option 2: Use Diaphora with IDA Pro 8.x

Diaphoraarrow-up-right is an excellent open-source alternative. Per its README, it supports IDA 7.4+ and requires Python 3.x (tested up to Python 3.11). The README mentions IDA 6.8 to 8.4.

Option 3: Build BinDiff from Source (Advanced)

BinDiff is now open-source and can be built with IDA SDK support:

Option 4: Use Ghidra + Ghidriff (Recommended)

This is the recommended approach for this course - see next section.

Installing Ghidra + Ghidriff

Ghidriff Features:

Ghidriff has evolved significantly with new capabilities for patch analysis:

[!NOTE] you might need to set java, ghidra and python envs like up there if you haven't

  1. String Reference Diffing: Track which functions reference which strings - crucial for spotting new error messages, commands, or embedded data:

  1. JSON Export for Automation: Export structured diff data for CI/CD pipelines:

  1. Docker Support (Recommended for reproducibility):

  1. Debug Logging: Troubleshoot analysis issues:

  1. Symbol Integration: Use PDB symbols for better function names:

Handling Large Binaries:

Large binaries like ntoskrnl.exe or tcpip.sys can overwhelm default settings:

Ghidriff Workflow

Step 1: Basic Diff (Headless)

you'll see something like:

then you can use this pyhton script:

Step 2: Enhanced Diff with Symbols

Step 3: Review Markdown Report

Step 4: Examine Function-Level Diffs

Ghidra Version Tracking (Built-in Alternative)

Step 0: Preparation

Step 1: Create Ghidra Project

Step 2: Create Version Tracking Session

Step 3: Run Correlators

Step 4: Filter and Analyze

Step 5: Cheat Code

BinDiff Workflow (IDA Pro)

Step 1: Create Project and Load Binaries

Step 2: Run BinDiff

Step 3: Analyze Results

Step 4: Visual Diff

Diaphora Workflow (IDA Pro)

Step 1: Export Databases

Step 2: Run Diff

Step 3: Analyze Partial Matches

Step 4: Use Vulnerability Detection

Diaphora 3.0+ includes automatic detection of potentially fixed vulnerabilities:

Step 5: Port Symbols and Comments

Diaphora Command-Line Mode (for automation):

Filtering Noise: Non-Security Changes

Patches often bundle security fixes with other changes. Learn to filter noise:

Common False Positives:

  1. Compiler Version Changes: Different compiler = different code generation

    • Look for: Changed function prologues/epilogues across ALL functions

    • Solution: Focus on functions with logic changes, not just instruction differences

  2. Code Reorganization: Functions moved, not changed

    • Look for: Function at different address but identical code

    • Solution: Ghidriff/BinDiff match by content, not address

  3. String/Resource Updates: Version strings, copyright dates

    • Look for: Changes only in .rdata or .rsrc sections

    • Solution: Focus on .text section changes

  4. Inlined Functions: Compiler inlined what was previously a call

    • Look for: Function "deleted" but code appears in callers

    • Solution: Check if "deleted" function's code exists elsewhere

  5. Debug Symbol Changes: Different PDB compilation

    • Look for: Symbol names changed but code identical

    • Solution: Compare actual instructions, not just names

Tips for Noise Reduction:

Identifying Security-Relevant Changes

Common Patterns to Look For:

  1. Added Bounds Checks:

  2. New Validation Functions:

  3. Changed Size Calculations:

  4. Additional NULL Checks:

  5. Initialization Changes:

  6. Error Handling Changes:

    • New try/catch or __try/__except blocks

    • Changed return value checks

    • Added cleanup paths

  7. Cryptographic Updates:

    • Algorithm changes (MD5 → SHA256)

    • Key length modifications

    • Random number generator updates

  8. Access Control Modifications:

    • New permission checks

    • Changed ACL handling

    • Token validation additions

  9. Memory Management:

    • Changed allocator (malloc → calloc)

    • Added memory zeroing before free

    • Buffer size recalculations

Practical Exercise

Task: Diff a couple of the cases you've chosen from day 1 and identify security changes

  1. Run BinDiff or Ghidriff:

    • Compare vulnerable vs patched binaries

    • Generate similarity report

  2. Identify Top 5 Most-Changed Functions:

    • Sort by similarity score (ascending)

    • List function names and addresses

  3. Analyze Each Function:

    • What code was added?

    • What code was removed?

    • What appears to be the bug?

  4. Categorize the Fix:

    • Bounds check?

    • NULL pointer check?

    • Integer overflow fix?

    • Logic error correction?

  5. Document Findings:

Success Criteria:

  • Successfully completed diff with either tool

  • Identified at least 3 changed functions

  • Recognized common security fix patterns

  • Documented findings in structured format

Key Takeaways

  1. Tools automate comparison: But human analysis finds the meaning

  2. Multiple tools available: Choose based on budget and preferences

  3. Similarity score guides focus: Lower score = more significant changes

  4. Patterns are recognizable: After analyzing several patches, fixes become obvious

  5. Documentation is crucial: Clear notes enable exploitation phase

Discussion Questions

  1. What are the advantages of automated diffing tools over manual comparison?

  2. How can false positives (non-security changes) be filtered out efficiently?

  3. Why might a function show low similarity despite no security fix?

  4. What additional analysis techniques can supplement binary diffing?

Day 3: Case Study - CVE-2022-34718 (EvilESP)

Vulnerability Overview

From Week 1, you briefly classified CVE-2022-34718 (EvilESP) as an out-of-bounds write in tcpip.sys. Here you will dig into the actual patch and see exactly how Microsoft fixed that bug.

CVE-2022-34718:

  • Component: tcpip.sys (Windows TCP/IP stack)

  • Type: Out-of-bounds 1-byte write

  • Impact: Remote Code Execution (RCE)

  • CVSS: 9.8 (Critical)

  • Affected: Windows Server 2022, Windows 11, Windows 10 (with IPsec enabled)

  • Patch Date: September 2022

  • Discoverer: MDSec

Attack Scenario: Unauthenticated attacker sends specially crafted IPv6 packets encapsulated in ESP (IPsec) to trigger out-of-bounds write in kernel memory, leading to RCE with SYSTEM privileges.

Patch Diffing Process

Step 1: Binary Acquisition

Step 2 Option 1: Load in Ghidra and use version tracking

Step 2 Opion 2: Run Binary Diff

Function 1: IppReceiveEsp

Vulnerable Code

Patched Code

Function 2: Ipv6pReassembleDatagram

Vulnerable Code

Patched Code

Root Cause Analysis

Understanding the Vulnerability Context:

This vulnerability exists in the IPv6 fragment reassembly path when processing ESP (Encapsulating Security Payload) packets.

ESP Packet Structure (RFC 4303):

IPv6 Fragment Header:

The Bug (Two-Part Vulnerability):

Part 1 - IppReceiveEsp (Missing Result Validation):

Part 2 - Ipv6pReassembleDatagram (Integer Overflow + OOB Access):

Visual Attack Flow:

Exploitation Primitive

Aspect
Details

Type

Out-of-bounds write (potentially read as well)

Size

Variable (controlled via fragment sizes)

Offset Control

Via nextheader_offset in fragment header

Trigger

Remote, requires IPsec enabled

Prerequisite

IPv6 enabled (default), IPsec service running

Exploitation Approach

Achieving RCE (Theoretical):

  1. Prerequisite: Establish IPsec Security Association with target (requires valid SPI + HMAC key)

  2. Heap Grooming: Send legitimate ESP traffic to create predictable pool state in NonPagedPoolNx

  3. Trigger OOB: Send crafted nested fragment headers inside ESP to corrupt adjacent NET_BUFFER_LIST structures

  4. Control Structure Corruption: Overwrite function pointers or list linkage in adjacent pool allocation

  5. Code Execution: Redirect execution when corrupted structure is processed

Challenges:

  • IPsec SA required - must have matching SPI + HMAC authentication key (critical barrier)

  • ESP crypto validation must pass before reaching vulnerable code path

  • Limited offset control due to extension header field constraints

  • Fragment reassembly timeout (~60 seconds) limits attack window

  • Windows kernel pool is non-deterministic

  • Need to avoid BSOD before achieving stable corruption

Patch Summary

Function
Vulnerability
Fix Added

IppReceiveEsp

Missing result validation after IppReceiveEspNbl

Range check: (iVar3 != 0) && (1 < (uint)(iVar3 - 0x2b))

IppReceiveEsp

Continued execution on error

Added IppDiscardReceivedPackets call with error 0xe0004148

Ipv6pReassembleDatagram

Integer overflow in size (16-bit)

Check: if (uVar14 < 0x10001)

Ipv6pReassembleDatagram

OOB via nextheader_offset

Check: if (*(ushort *)(param_2 + 0xbc) <= uVar13)

Ipv6pReassembleDatagram

Size mismatch

Check: if (uVar14 + 0x28 < *(uint *)(lVar4 + 0x18)) triggers failure

Both

No telemetry

Added ETW events: TCPIP_IP_REASSEMBLY_FAILURE_PKT_LEN

Lessons Learned

  1. Binary diffing is highly effective: Only 2 functions changed - instant focus

  2. Protocol knowledge is essential: Understanding ESP/IPv6 specs was crucial

  3. Simple bugs still exist: Missing bounds check in complex networking code

  4. Limited primitives are still dangerous: 1-byte OOB write still got CVE 9.8

  5. Patches reveal exploitation strategies: Seeing the fix shows how to trigger the bug

Practical Exercise

Task: Apply the EvilESP analysis methodology to a different Windows TCP/IP or network stack vulnerability

Suggested Targets (just a suggestion, choose another yourself if you want):

CVE
Component
Type
Patch Date
Difficulty

CVE-2024-38063

tcpip.sys (IPv6)

RCE

Aug 2024

Medium

CVE-2021-24086

tcpip.sys (IPv6 UDP)

DoS

Feb 2021

Easy

CVE-2021-24074

tcpip.sys (IPv4 source routing)

RCE

Feb 2021

Medium

CVE-2020-16898

tcpip.sys (ICMPv6 Router Advertisement)

RCE

Oct 2020

Medium

CVE-2024-21407

Hyper-V (Guest-to-Host)

RCE

Mar 2024

Hard

Steps:

  1. Research Your CVE:

    • Read the MSRC advisory and any public write-ups

    • Identify the affected binary and KB numbers (vulnerable vs patched)

    • Research the relevant protocol (IPv6, ICMPv6, etc.) using RFCs

  2. Acquire Binaries:

    • Use WinbIndex or Extract-Patch.ps1 to get both versions

    • Download symbols with symchk.exe

    • Verify version numbers match expected builds

  3. Perform Binary Diff:

    • Run ghidriff or use Ghidra Version Tracking

    • Identify changed functions (expect 1-5 for targeted security patches)

    • Filter out noise (compiler changes, unrelated updates)

  4. Analyze Changed Functions:

    • What validation was added? (bounds checks, NULL checks, size limits)

    • What was the root cause? (integer overflow, missing check, race condition)

    • Map the vulnerable code path from input to bug

  5. Research the Protocol:

    • Read the relevant RFC(s) for your vulnerability's protocol

    • Understand packet structures and processing flow

    • Identify what attacker-controlled fields reach the vulnerable code

  6. Write Technical Report:

    • Follow the Day 6 report template

    • Include decompiled code snippets (before/after)

    • Create an attack flow diagram similar to EvilESP

    • Assess exploitability considering modern mitigations

Success Criteria:

  • Successfully acquired both binary versions with symbols

  • Identified the security-relevant changed functions (not just all changes)

  • Located the specific patch additions in decompiled code

  • Explained the root cause vulnerability class (OOB, UAF, integer overflow, etc.)

  • Documented the protocol-level attack vector with RFC references

  • Created attack flow diagram showing: input → processing → vulnerability trigger

  • Assessed real-world exploitability (prerequisites, mitigations, reliability)

Key Takeaways

  1. Binary diffing rapidly focuses analysis: Only 2 functions changed in tcpip.sys—instant prioritization from 10,000+ functions

  2. Protocol knowledge is essential: Understanding ESP (RFC 4303) and IPv6 fragmentation (RFC 8200) was crucial to grasp the attack

  3. Simple bugs in complex code are high-impact: Missing 16-bit overflow check earned CVSS 9.8

  4. Multi-function vulnerabilities are common: IppReceiveEsp's validation failure enabled Ipv6pReassembleDatagram's OOB write

  5. Prerequisites affect real-world risk: IPsec SA requirement limits exploitation despite critical rating

  6. Patches reveal trigger conditions: Seeing the bounds checks shows exactly what inputs cause the bug

Discussion Questions

  1. CVE-2022-34718 requires IPsec SA (valid SPI + HMAC key) yet received CVSS 9.8. How should prerequisites factor into severity ratings?

  2. The bug spanned two functions (IppReceiveEsp and Ipv6pReassembleDatagram). How might static analysis or code review catch such cross-function vulnerabilities?

  3. IPv6 fragment reassembly is a recurring vulnerability source (CVE-2024-38063, CVE-2021-24086, etc.). What makes reassembly logic error-prone?

  4. The patch added ETW telemetry for failed reassembly. How can defenders leverage this, and how might attackers evade detection?

Day 4: Windows 11 Automated Patch Diffing Pipeline

Automation

Why Automate?:

  • Microsoft releases patches monthly (Patch Tuesday - 2nd Tuesday)

  • Analyzing every update manually is time-consuming

  • Early detection of vulnerabilities provides competitive advantage

  • Automation enables continuous monitoring

Pipeline Stages:

  1. Monitor: Detect new security updates

  2. Download: Fetch updated binaries

  3. Extract: Unpack .msu/.cab files

  4. Symbol: Download matching PDB files

  5. Diff: Compare against previous version

  6. Report: Generate HTML/PDF summary

  7. Alert: Notify of high-priority changes

PowerShell Automation Script

  • You can use the Extract-Patch.ps1 script from day 1 of this week

  • Or as an exercise try to write/find a better script for yourself

Python Automation for Ghidriff

Batch Diffing Script:

Scheduled Automation (Windows Task Scheduler)

Create Monthly Task:

Monthly Script (monthly_diff.ps1):

Practical Exercise

Task: Build and test your automated patch diffing pipeline

This exercise walks you through creating a reusable automation workflow that you can run monthly after each Patch Tuesday.

Part 1: Environment Setup

Part 2: Acquire Consecutive Updates

Part 3: Download Symbols

Part 4: Run Batch Diff

Part 5: Analyze Results

Analysis Questions to Answer:

  1. Which binaries had the most changed functions?

  2. Are there functions with similarity < 0.90? (High-priority for review)

  3. Do any function names suggest security fixes? (Look for: Validate, Check, Bounds, Safe, Sanitize)

  4. Are there new functions added? (Could be new security checks)

  5. Cross-reference with MSRC Security Update Guidearrow-up-right - do the changed binaries match announced CVEs?

Part 6: Create Monthly Automation (Optional)

Success Criteria:

  • Workspace directories created and organized

  • Both KB versions downloaded with correct file versions

  • Symbols downloaded for at least 50% of binaries

  • ghidriff_batch.py runs without errors

  • HTML report generated with function statistics

  • At least 3 target binaries successfully diffed

  • Identified changed functions with similarity scores < 0.95

  • Cross-referenced at least one finding with MSRC advisory

Troubleshooting Common Issues:

Issue
Solution

"ghidriff not found"

Ensure Python Scripts folder is in PATH, or use full path to ghidriff.exe

Version mismatch from WinbIndex

Download .msu manually and extract, or use UUP Dump

Out of memory during diff

Reduce --max-section-funcs or diff one binary at a time

No symbols downloaded

Check symchk output for errors; some binaries may not have public symbols

Empty diff report

Verify both versions are actually different (check FileVersion)

LLM-Assisted Patch Summarization

Combining ghidriff output with Large Language Models can accelerate patch analysis:

Workflow:

  1. Generate ghidriff markdown output

  2. Feed the diff to an LLM with security context

  3. Get automated vulnerability summaries

Example Prompt Template:

[!WARNING] LLM Limitations for Patch Analysis:

  • LLMs can hallucinate vulnerability details that don't exist

  • They may miss subtle bugs that require deep domain knowledge

  • Assembly/decompiled code analysis is not their strength

  • Always verify LLM findings by examining the actual code

  • Use LLMs for initial triage and hypothesis generation, not as the final word

When LLMs Help Most:

  • Summarizing large diffs with many changed functions

  • Generating initial hypotheses about vulnerability class

  • Explaining unfamiliar code patterns

  • Drafting report sections (with verification)

When LLMs Struggle:

  • Subtle race conditions or timing issues

  • Complex pointer arithmetic and bounds calculations

  • Understanding Windows kernel internals without context

  • Distinguishing security fixes from optimization changes

Key Takeaways

  1. Automation transforms patch analysis from reactive to proactive: Instead of waiting for public PoCs, you can analyze patches within hours of release and understand vulnerabilities before exploits appear in the wild.

  2. The pipeline has clear stages with different failure modes:

    • Acquisition: WinbIndex collisions, delta-only updates, missing binaries

    • Extraction: PSF format changes, nested CAB structures, corrupted packages

    • Diffing: Memory limits, timeout on large binaries, missing symbols

    • Analysis: Compiler noise, false positives, missing context

  3. Symbols are force multipliers: A diff with symbols shows IppValidatePacketLength changed; without symbols, you see sub_1400A2F40 changed. Invest time in symbol acquisition.

  4. Prioritization is critical for scale: A cumulative update may change 500+ functions across 20 binaries. Use heuristics:

    • Similarity < 0.90 = significant change

    • Security-relevant function names (Validate, Check, Bounds, Parse)

    • Binaries mentioned in MSRC advisories

    • Network-facing components (tcpip.sys, http.sys, afd.sys)

  5. Automation enables pattern recognition over time: After analyzing 6-12 months of patches, you'll recognize Microsoft's fix patterns:

    • Integer overflow → RtlULongAdd, RtlSizeTMult usage

    • Buffer overflow → _s suffix functions, explicit size checks

    • UAF → Reference counting changes, deferred cleanup

    • Logic bugs → Additional if conditions, early returns

  6. Documentation pays dividends: Keep notes on each analysis. Future patches to the same component become easier when you understand the code.

  7. Tools evolve; adapt your pipeline: ghidriff, WinbIndex, and Windows Update formats all change. Budget time for maintenance.

Discussion Questions

  1. Timing: Microsoft releases patches on the second Tuesday of each month. Security researchers often race to analyze patches before attackers can weaponize them.

    • Should automated patch analysis tools be publicly available, or does this help attackers more than defenders?

    • What's the difference between "patch diffing for defense" and "patch diffing for offense"?

  2. Automation vs. Understanding: Automated pipelines can process dozens of binaries overnight, but may miss subtle vulnerabilities that require human insight.

    • How do you balance breadth (analyze everything) vs. depth (understand thoroughly)?

    • What signals should trigger deeper manual analysis?

    • Can automation replace the need to understand Windows internals, or is it just a force multiplier?

  3. False Positives and Noise: Large updates contain many non-security changes mixed with actual fixes.

    • What filtering strategies work best for isolating security-relevant changes?

    • How do you handle compiler optimizations that make identical code look different?

    • Should you track "interesting" non-security changes that might become vulnerabilities later?

  4. Data Sources and Correlation: Patch diffs are one piece of the puzzle.

    • What other data sources could enhance automated analysis? (MSRC advisories, syzkaller reports, GitHub commits, Twitter/X discussions)

    • How would you correlate a binary diff with source-level commits for open-source components in Windows?

    • Could you automate CVE-to-function mapping by combining MSRC descriptions with diff output?

  5. Scaling and Prioritization: Microsoft patches Windows, Office, Edge, Exchange, Azure, and more.

    • How would you prioritize which products/components to analyze first?

    • What metrics indicate a patch is "high priority" for analysis? (CVSS, exploitability, attack surface)

    • Could you build a scoring system to automatically rank patches by security relevance?

  6. LLM Integration: Large Language Models can summarize diffs but have limitations.

    • What tasks are LLMs good at in patch analysis? What should they never do unsupervised?

    • How would you validate LLM-generated vulnerability summaries?

    • Could LLMs help generate initial PoC hypotheses, or is this too risky?

Day 5: Linux Kernel Patch Diffing

Linux Kernel Patch Diffing Workflow

Differences from Windows:

  • Source code often available (but not always for vendor kernels)

  • Binaries are ELF format

  • Distribution-specific modifications complicate diffing

  • DWARF debug symbols instead of PDB

Step 1: Identify Target Kernel Builds

Ubuntu Example:

Step 2: Download Kernel Images and Debug Symbols

Ubuntu/Debian:

Debug Symbols:

Debug symbols provide function names and source mappings. There are multiple ways to obtain them:

Step 3: Extract or Locate vmlinux

The vmlinux file is the uncompressed kernel image needed for binary diffing. There are several ways to obtain it depending on what packages you downloaded:

Option A: From dbgsym package (Best - includes debug symbols)

If you downloaded and extracted the -dbgsym packages in Step 2:

Option B: Extract from vmlinuz (Stripped - no debug symbols)

If you only have the regular kernel package (not dbgsym), you can extract vmlinux from the compressed vmlinuz:

Repeat for the new/patched kernel:

Step 4: Identify Changed Modules

Step 5: Install Ghidra and Ghidriff on Linux

Before running binary diffs, you need to install Ghidra and ghidriff. The Windows installation was covered in Day 2 (Option 4). Here's the Linux setup:

Step 6: Binary Diffing with Ghidra

Module-Specific Diff:

Step 7: Source-Level Diff (When Available)

Using Git:

Step 8: Analyze Specific CVE Fix

Example 1: CVE-2024-1086 (nf_tables UAF) - High-Profile LPE

This vulnerability affected all Linux kernels from 3.15 to 6.8 and had a public exploit achieving reliable root:

Analysis:

  • Bug Type: Use-After-Free via verdict value confusion

  • Location: net/netfilter/nf_tables_api.c:nft_verdict_init()

  • Root Cause: Old code used & NF_VERDICT_MASK to validate verdicts, allowing values like NF_DROP | extra_bits to pass. These "decorated" verdicts caused type confusion in later code paths.

  • Impact: Reliable local privilege escalation (LPE)

  • Exploit: Public exploit by @Notselwyn achieves ~99% success rate

  • Fix: Changed from mask-based validation to exact match validation, rejecting any verdict with extra bits set

Why This Matters for Patch Diffing:

  • The fix restructures ~15 lines, but understanding WHY required deep nf_tables knowledge

  • Demonstrates how mask vs exact-match validation can have critical security implications

  • Public exploit provides validation of patch analysis

Example 2: CVE-2024-26585

Analysis:

  • Bug Type: Race Condition (CWE-362)

  • Location: net/tls/tls_sw.c:tls_encrypt_done()

  • Root Cause: In the original code, the ready flag was set inside a locked section when rec == first_rec, but the actual schedule_delayed_work() call happened after atomic_dec_and_test() completed. This created a race window where ctx could be freed by another thread after the lock was released and encrypt_pending reached zero, but before the work was scheduled.

  • Fix: Moved the schedule_delayed_work() call inside the locked section immediately after checking rec == first_rec, ensuring the scheduling happens atomically with the check before any potential cleanup.

  • Impact: Use-after-free leading to potential privilege escalation or denial of service in TLS socket handling

  • Trigger: Concurrent TLS encryption operations with specific timing

Example 3: CVE-2024-0582

Why io_uring is a Hot Target:

  • Complex async I/O subsystem added in Linux 5.1

  • Frequent syzkaller findings

  • High attack surface (many operations)

  • Often enabled even in containers

Searching for io_uring CVEs:

Step 9: Symbolization and Crash Mapping

Decode Kernel Oops:

Map Address to Source:

Linux-Specific Considerations

Compiler Optimizations:

  • Clang vs GCC produce different code

  • -O2 vs -O3 significantly affects diff

  • Link-Time Optimization (LTO) complicates analysis

  • kcfi (kernel Control-Flow Integrity) adds thunks

[!TIP] Filter by real function body deltas, ignore CFI stubs.

Kernel Livepatch Considerations:

Some distributions use kernel livepatch for security fixes:

Livepatch implications for diffing:

  • Original vmlinux unchanged

  • Fix is in separate livepatch module

  • Must diff the livepatch .ko against nothing (new code)

KCFI/FineIBT Changes:

  • Create many small stub changes

  • Look for __cfi_ prefixed functions

  • Focus on substantial logic changes, not just landing pads

Syzkaller Reports:

  • Consult syzbotarrow-up-right for reproducers

  • Many bugs have C reproducer and syz script

  • Helps validate your diff analysis

[!TIP] Syzkaller routinely bisects kernel bugs to find introduction/fix commits.

Practical Exercise

Task: Analyze recent Ubuntu kernel security update

  1. Download Two Consecutive Kernel Versions

  2. Extract and Prepare

  3. Identify Target Subsystem

  4. Diff Specific Module

  5. Verify Source-Level Changes

  6. Document Findings

    • What subsystem was affected?

    • What functions changed?

    • What was the root cause?

    • How was it fixed?

Success Criteria:

  • Successfully extracted vmlinux from both versions

  • Identified changed modules

  • Completed binary diff with Ghidriff

  • Correlated findings with source-level patch

  • Understood the vulnerability and fix

Key Takeaways

  1. Linux kernel diffing is more accessible: Source often available, open development

  2. Distribution kernels add complexity: Vendor patches, backports complicate analysis

  3. Module-level diffing is practical: Full vmlinux diff is resource-intensive

  4. Source correlation is valuable: Binary diff finds functions, source explains why

  5. Syzkaller is a goldmine: Reproducers, bisection data, and crash reports

Discussion Questions

  1. How do Linux and Windows patch diffing workflows differ in practice?

  2. What advantages does open-source kernel development provide for security research?

  3. How can vendor-specific kernel patches complicate vulnerability analysis?

  4. What role does syzkaller play in modern kernel security?

Day 6: 7-Zip Case Study and Writing Reports

In Week 1 you saw this 7-Zip symlink issue as an example of a logic/path-traversal bug. Here you will patch-diff the actual fix and practice turning that analysis into a professional report.

Background:

  • Software: 7-Zip file archiver

  • Versions Affected: 24.09 and earlier

  • Fixed In: 25.00

  • Vulnerability Type: Path Traversal via Symlink Handling (CWE-22)

  • Impact: Arbitrary File Write (can lead to RCE via DLL hijacking, startup folder abuse)

  • CVSS: Estimated 7.8 (High) - Local attack vector

Discovery Method: Source code review and patch analysis (versions are open-source).

Source-Level Patch Analysis

Target File: CPP/7zip/UI/Common/ArchiveExtractCallback.cpp

Finding the Changes:

Key Changes Identified:

1. IsSafePath Function Signature Changed:

2. CLinkLevelsInfo::Parse Modified:

3. Dangerous Link Check Enhanced:

4. New Normalization Functions Added:

5. Link Type Refactoring:

6. Slash Conversion Macro:

Root Cause Analysis

The Logic Bug:

  1. WSL/Linux Symlinks on Windows: Archives (tar, rar5, etc.) can contain Linux-style symlinks. When extracted on Windows, these symlinks could point to Windows-style absolute paths (e.g., C:\Windows\System32\).

  2. Missing WSL-Aware Path Detection:

    • CLinkLevelsInfo::Parse() always used NName::IsAbsolutePath() which uses Windows logic

    • For WSL symlinks, /etc/passwd was correctly detected as absolute

    • But a WSL symlink containing C:\Users\... was NOT detected as absolute because WSL paths expect / as absolute indicator

  3. Conditional Dangerous Link Check:

    • The SymLinks_AllowDangerous validation had #ifdef _WIN32 and if (_item.IsDir) guards

    • On Windows, only directory symlinks were validated

    • File symlinks bypassed the dangerous link check entirely

  4. No Path Normalization:

    • Absolute paths in archives were passed directly to symlink creation

    • No stripping of \??\, \\?\UNC\, or drive letter prefixes

    • Result: Symlinks could point outside the extraction directory

  5. Late Slash Conversion:

    • Linux separators (/) were converted to Windows separators (\) too late in the process

    • Path validation occurred before normalization in some code paths

Attack Scenario:

How the Patch Fixes It:

Practical Triage Checklist

When Analyzing Path Validation Code:

  1. Search for:

    • IsSafePath, ValidatePath, CheckPath, Normalize functions

    • IsAbsolute, IsRelative, GetRootPrefixSize checks

    • Path concatenation: JoinPath, CombinePath, operator/, + on path strings

    • Symlink handling: CreateSymbolicLink, SetReparseData, readlink, lstat

    • Slash/separator conversion: Replace('/', '\\'), path separator macros

  2. Verify:

    • Absolute path detection works across OS semantics (Linux /, Windows C:\, UNC \\, device paths \??\)

    • WSL/cross-platform symlinks handled with correct path semantics for their origin

    • Path normalization (prefix stripping, redundant separator removal) happens BEFORE validation

    • "Dangerous link" checks run for ALL symlink types (files AND directories)

    • No platform-specific guards (#ifdef _WIN32) that skip security checks

  3. Test Cases:

    • Windows absolute in Linux/WSL symlink: C:\..., \??\C:\...

    • UNC paths in archives: \\server\share, \\?\UNC\server\share

    • Mixed separators: C:\dir/subdir, /home\user

    • Relative paths with .. sequences: ../../../etc/passwd

    • Symlinks to symlinks (chain validation)

    • Device paths: \??\, \\?\, \\.\

    • Path prefix attacks: C:relative (drive-relative), \root (root-relative)

Creating a Report

Report Structure:

Practical Exercise

Task: Write a patch diff report for a source-available vulnerability

  1. Choose a Target (suggestions):

    • 7-Zip: Diff 24.09 vs 25.00 for symlink vulnerabilities (CVE-2025-11001/11002)

    • curl: Check recent CVEs at https://curl.se/docs/security.html

    • OpenSSL: https://www.openssl.org/news/vulnerabilities.html

    • nginx: Security advisories at https://nginx.org/en/security_advisories.html

  2. Perform Analysis:

  3. Write Report: Use template above, focus on:

    • Clear root cause explanation (1-2 sentences)

    • Patch summary table (what changed, why)

    • Actionable recommendations

  4. Self-Review Checklist:

    • CVE/version numbers verified against official sources

    • Root cause explains the programmer's mistake

    • Patch changes mapped to vulnerability fixes

    • Recommendations are specific and actionable

Success Criteria:

  • Accurate CVE/version information (cross-check with vendor advisories)

  • Root cause explains WHY the bug existed, not just WHAT changed

  • Report readable by someone unfamiliar with the codebase

Key Takeaways

  1. Cross-platform code has cross-platform bugs: WSL symlinks exposed path semantics mismatch between Linux and Windows

  2. Conditional compilation hides attack surface: #ifdef _WIN32 guards can create platform-specific vulnerabilities

  3. "Defense in depth" patches are common: 7-Zip added multiple checks (WSL detection, prefix stripping, separator normalization)

  4. Silent fixes get CVEs later: 25.00 fixed the bug (July 2025); ZDI disclosed CVE-2025-11001/11002 in October 2025

  5. Source diffing reveals intent: Seeing Normalize_to_RelativeSafe() added explains the fix strategy better than binary diff alone

Discussion Questions

  1. The 7-Zip fix added 6+ distinct changes. How do you determine which change fixes the core vulnerability vs. adds defense-in-depth?

  2. CVE-2025-11001/11002 were disclosed by ZDI months after 25.00 shipped. What are the pros/cons of "silent" security fixes?

  3. The vulnerability required symlinks AND file extraction through them. How does attack chain complexity affect severity ratings?

  4. 7-Zip 25.01 added -snld20 to bypass the new checks. When are "escape hatches" for security features appropriate?

Day 7: Capstone Project - The Patch Diffing Campaign

  • Goal: Apply the week's techniques to analyze a real-world vulnerability end-to-end, from binary acquisition to professional reporting.

  • Activities:

    • Select a Target: Choose from the suggested CVEs below or find your own recent security patch.

    • Execution: Complete the full pipeline: acquire → extract → diff → analyze → report.

    • Analysis: Identify changed functions, understand root cause, correlate with public documentation.

    • Reporting: Create a comprehensive vulnerability report suitable for disclosure.

Suggested Capstone Targets

Choose ONE of the following based on your interest and available environment:

Option A: Windows - CVE-2024-38063 (tcpip.sys IPv6 RCE)

Option B: Windows - CVE-2024-21338 (appid.sys LPE)

  • Component: appid.sys (AppLocker driver)

  • Type: Local Privilege Escalation

  • CVSS: 7.8 (High)

  • Patch: February 2024

  • Why This One: Kernel driver, smaller than tcpip.sys, good for learning

Option C: Linux - CVE-2024-1086 (nf_tables LPE)

Option D: Application - CVE-2024-4367 (PDF.js Type Confusion)

  • Component: Mozilla PDF.js (Firefox built-in PDF reader)

  • Type: Type confusion leading to code execution

  • CVSS: 8.8 (High)

  • Patch: Firefox 126+

  • Why This One: JavaScript-based, source diffing, affects browsers

Capstone Execution Framework

Phase 1: Target Selection and Research

Document your findings:

Phase 2: Binary Acquisition

For Windows targets:

For Linux targets:

Phase 3: Binary Diffing

Diff Analysis Checklist:

  • How many functions changed? (Targeted patch = few changes)

  • What are the function names? (Security-relevant names?)

  • What was added? (Validation, bounds checks, error handling?)

  • What was removed? (Dead code, vulnerable paths?)

  • Any new functions introduced? (Sanitization helpers?)

Phase 4: Root Cause Analysis

Deep dive into changed functions:

Reconstruct the vulnerability:

Phase 5: Exploitation Assessment

Determine exploitability:

Phase 6: Report Writing

Use the template from Day 6 to create your final report:

Capstone Deliverables Checklist

Required Deliverables:

  • research_notes.md - Initial CVE research and hypotheses

  • vulnerable/ and patched/ directories with binaries

  • analysis/diff_report/ - Ghidriff output

  • report/patch_diff_report.md - Professional vulnerability report

  • report/screenshots/ - Key diff screenshots

Grading Rubric (Self-Assessment):

Criterion
Points
Description

Binary Acquisition

15

Both versions obtained, organized

Diff Execution

15

Ghidriff ran successfully, output reviewed

Root Cause Identification

25

Correctly identified the vulnerability

Patch Understanding

20

Explained what the patch does and why

Exploitation Assessment

15

Realistic assessment of exploitability

Report Quality

10

Clear, professional, actionable

Total

100

Common Capstone Pitfalls

Avoid These Mistakes:

  1. Picking a CVE without public details

    • Some CVEs have no write-ups; you'll struggle without context

    • Prefer CVEs with at least an advisory or blog post

  2. Ignoring compiler noise

    • Many "changed" functions are just recompilation artifacts

    • Focus on functions with < 0.95 similarity AND security-relevant names

  3. Missing the forest for the trees

    • Don't get lost in assembly details

    • Step back and ask: "What was the programmer's mistake?"

  4. Incomplete exploitation assessment

    • "It crashes" is not an assessment

    • Explain: What can an attacker control? What's the primitive?

  5. Report without remediation

    • Always include actionable recommendations

    • "Upgrade to version X" is the minimum

Key Takeaways

  1. Diffing is iterative: You rarely find the bug in the first pass. Filter noise, ignore compiler optimizations, and focus on logic changes.

  2. Context is king: A changed line means nothing without understanding the surrounding function and data flow.

  3. Symbols are essential: Without PDBs or debug symbols, diffing is significantly harder. Always prioritize obtaining them.

  4. Reporting matters: A good finding is useless if you can't communicate the impact and root cause clearly.

  5. Practice makes perfect: Each CVE you analyze builds pattern recognition for the next one.

Discussion Questions

  1. How does the choice of CVE affect the difficulty of patch diffing?

  2. What strategies help when a patch changes hundreds of functions?

  3. Why might a vendor's patch introduce new vulnerabilities?

  4. How would you approach diffing a browser update with thousands of changes?

Last updated