OpenSSL BN_mod_exp_inverse

Hunt Hypothesis

The OpenSSL BN_mod_exp_inverse detection rule identifies potential cryptographic operations that may indicate adversary use of weak or compromised key exchanges. SOC teams should proactively hunt for this behavior in Azure Sentinel to detect early signs of cryptographic manipulation or key compromise in TLS/SSL communications.

YARA Rule

rule OpenSSL_BN_mod_exp_inverse
{	meta:
		author = "Maxx"
		description = "OpenSSL BN_mod_exp_inverse"
	strings:
		$c0 = { B8 18 00 00 00 E8 ?? ?? ?? ?? 53 55 56 57 8B 7C 24 38 33 C0 57 89 44 24 20 89 44 24 24 E8 ?? ?? ?? ?? 57 E8 ?? ?? ?? ?? 57 89 44 24 1C E8 ?? ?? ?? ?? 57 8B F0 E8 ?? ?? ?? ?? 57 89 44 24 28 E8 ?? ?? ?? ?? 57 8B E8 E8 ?? ?? ?? ?? 57 8B D8 E8 ?? ?? ?? ?? 8B F8 8B 44 24 54 50 89 7C 24 38 E8 ?? ?? ?? ?? 83 C4 20 89 44 24 24 85 C0 8B 44 24 2C 0F 84 78 05 00 00 85 C0 75 05 E8 ?? ?? ?? ?? 85 C0 89 44 24 1C 0F 84 63 05 00 00 8B 4C 24 14 6A 01 51 E8 ?? ?? ?? ?? 6A 00 57 E8 }
	condition:
		$c0
}

Deployment Notes

This YARA rule can be deployed in the following contexts:

• Microsoft Defender for Endpoint — Custom indicators / advanced hunting
• Email Gateway — Attachment scanning
• File Share Monitoring — Periodic scanning of shared drives
• YARA CLI — Manual threat hunting on endpoints

This rule contains 1 string patterns in its detection logic.

• Source Rule

False Positive Guidance

• Scenario: System update or patching using openssl command-line tool
  Filter/Exclusion: Check for openssl version or openssl update commands in the process name or command line arguments.
  Example Filter: process.name = "openssl" AND process.args != "version" AND process.args != "update"

• Scenario: Scheduled job running OpenSSL-based encryption/decryption tasks (e.g., using openssl rsautl or openssl cms)
  Filter/Exclusion: Exclude processes associated with known scheduled tasks (e.g., Task Scheduler or cron jobs) by checking the parent process or command line.
  Example Filter: process.parent.name = "schtasks.exe" OR process.parent.name = "cron" OR process.args CONTAINS "scheduled"

• Scenario: Administrative task involving key generation or certificate management using OpenSSL (e.g., openssl genrsa, openssl req)
  Filter/Exclusion: Exclude processes where the command line contains key generation or certificate-related keywords.
  Example Filter: process.args CONTAINS "genrsa" OR process.args CONTAINS "req" OR process.args CONTAINS "cert"

• Scenario: Internal tool or script using OpenSSL for secure data transfer (e.g., openssl s_client, openssl s_server)
  Filter/Exclusion: Exclude processes that are part of internal development or testing environments by checking the source IP or user context.
  Example Filter: source.ip IN ("10.0.0.0/8", "172.16.0.0/12", "192.168.0.0/16")

• Scenario: OpenSSL-based logging or monitoring tool (e.g., openssl log or custom monitoring scripts)
  Filter/Exclusion: Exclude processes that are part of