Bug: agent, vm cleanup - iptables/ipset chain existence not checked before deletion, causing misleading error logs

[rybakovanton-metta]

Title

Cleanup: iptables/ipset chain existence not checked before deletion, causing misleading error logs

ISSUE TYPE

• Bug Report

COMPONENT NAME

scripts/vm/network/security_group.py

CLOUDSTACK VERSION

main branch (4.22)

SUMMARY

When VM cleanup runs, the destroy_network_rules_for_vm() function attempts to flush and delete iptables chains and ipsets without first checking if they exist. The execute() function logs CalledProcessError at ERROR level with full traceback before re-raising, causing confusing duplicate log entries when chains don't exist. Exit code 1 from iptables/ipset is normal when a chain/set doesn't exist (idempotent cleanup), not a real error.

STEPS TO REPRODUCE

1. Create a VM with security groups in CloudStack
2. Delete the VM
3. Observe management server logs for the VM cleanup output

EXPECTED RESULTS

• Cleanup should be idempotent - safe to run multiple times
• "Chain doesn't exist" should be logged at DEBUG level
• Only real errors should be logged at ERROR level
• No duplicate logging of the same issue

ACTUAL RESULTS

2026-03-09 09:46:52,842 ERROR [c.c.v.s.security_group] (Thread-123) Command exited non-zero: iptables -X i-6-526-VM-eg
Traceback (most recent call last):
  File "/usr/share/cloudstack-common/scripts/vm/network/security_group.py", line 53, in execute
    return check_output(cmd, shell=True).decode()
  File "/usr/lib/python3.13/subprocess.py", line 472, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True, ...)
  File "/usr/lib/python3.13/subprocess.py", line 577, in run
    raise CalledProcessError(retcode, process.args, output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command 'iptables -X i-6-526-VM-eg' returned non-zero exit status 1.

2026-03-09 09:46:52,842 DEBUG Ignoring failure to delete chain: i-6-526-VM-eg

The ERROR level log with full traceback is misleading - this is expected behavior when chains don't exist.

PROPOSED FIX

The fix should check if chains exist before attempting deletion. When iptables -S chain returns exit code 1, the chain doesn't exist - this is normal, not a real error. Only real errors (other exit codes) should be logged.

Add helper functions:

def iptables_chain_exists(chain):
    """Check if iptables chain exists"""
    try:
        execute("iptables -S %s 2>/dev/null" % chain)
        return True
    except CalledProcessError as e:
        if e.returncode == 1:
            return False  # Chain not found - not an error
        raise  # Real error (other exit code)

def ipset_exists(setname):
    """Check if ipset exists"""
    try:
        execute("ipset list %s 2>/dev/null" % setname)
        return True
    except CalledProcessError as e:
        if e.returncode == 1:
            return False  # Set not found - not an error
        raise  # Real error (other exit code)

Update destroy_network_rules_for_vm() to check existence before deletion:

for chain in [_f for _f in chains if _f]:
    if iptables_chain_exists(chain):
        try:
            execute("iptables -F " + chain)
            execute("iptables -X " + chain)
        except Exception as e:
            logging.error("Failed to flush/delete chain %s: %s", chain, str(e))
    else:
        logging.debug("Chain %s does not exist, skipping", chain)

for ipset in [vm_ipsetname, vm_ipsetname + '-6']:
    if ipset_exists(ipset):
        try:
            execute('ipset -F ' + ipset)
            execute('ipset -X ' + ipset)
        except Exception as e:
            logging.error("Failed to flush/delete ipset %s: %s", ipset, str(e))
    else:
        logging.debug("Ipset %s does not exist, skipping", ipset)

This approach:

• Makes cleanup idempotent (safe to run multiple times)
• Provides clear logging: "chain doesn't exist" at DEBUG, real errors at ERROR
• Distinguishes between "not found" (exit code 1, expected) and real errors (other exit codes)
• No duplicate logging

[dheeraj12347]

Hi @DaanHoogland @rybakovanton-metta,
I reviewed the issue and it seems the cleanup logic in security_group.py attempts to flush/delete iptables chains and ipsets without verifying whether they exist first. When they don't exist, iptables/ipset return exit code 1, which is expected for idempotent cleanup but currently gets logged as an ERROR with a traceback.

I’d like to work on this by adding checks to verify the chain/ipset existence before deletion and adjusting the logging so that non-existent chains are handled gracefully (e.g., logged at DEBUG instead of ERROR).

Please let me know if it’s okay for me to take this up.

[github-actions]

🏷️ Issue Triaged

Hi @rybakovanton-metta! I've categorized this issue as bug based on the following analysis:

Reasoning: This reports a specific defect where VM cleanup causes misleading ERROR-level logs when iptables/ipset chains don't exist. The issue includes error reproduction, traceback, and a detailed proposed fix, making it clearly a bug report.

View Triage Details

Analysis

• Keywords detected: "Bug Report", "CalledProcessError", "misleading error logs", "cleanup", "iptables", "ipset"
• Issue type indicators: Specific component identified (security_group.py), clear steps to reproduce, expected vs actual behavior described, includes error logs and proposed fix
• Confidence: High

Recommended Next Steps

• Review the proposed fix that checks chain existence before deletion
• Consider implementing idempotent cleanup logic
• Test the solution against the reproduction steps provided

References: Triage run §22957405999

Generated by Issue Triage Agent

To install this workflow, run gh aw add github/gh-aw/.github/workflows/issue-triage-agent.md@94662b1dee8ce96c876ba9f33b3ab8be32de82a4. View source at https://github.com/github/gh-aw/tree/94662b1dee8ce96c876ba9f33b3ab8be32de82a4/.github/workflows/issue-triage-agent.md.