How to Protect VMware VMs from Ransomware with AWS Backup
Posted inaws Cloud

How to Protect VMware VMs from Ransomware with AWS Backup

No Comments

This guide provides complete instructions for implementing an enterprise-grade backup solution that protects against ransomware attacks through three critical security layers:

  1. Immutable Storage – Backups cannot be deleted by anyone, including attackers with administrative access
  2. Logical Air-Gap Vault – Complete isolation of backups from production environment
  3. External Approval Authority – Approval team located in separate AWS account outside your organisation

Table of Contents

Architecture Overview

Three-Tier Security Model

Tier 1: Production Organization

  • Your production AWS accounts and VMware environment
  • Daily automated backups with Changed Block Tracking
  • Standard backup vault for operational restores

Tier 2: External Approval Authority

  • Completely separate AWS account (not in your organization)
  • Contains approval team of 3+ trusted senior executives
  • MFA enforcement for all team members
  • Only function: approve emergency backup access

Tier 3: Clean Recovery Environment

  • Used exclusively during disaster recovery
  • Can request access to protected backups
  • Access granted only after multi-party approval from external team

How This Stops Ransomware Attacks

Traditional Attack Pattern:

  1. Attacker gains access to production systems
  2. Escalates privileges to administrator level
  3. Deletes or encrypts all backups
  4. Encrypts production data
  5. Demands ransom (victim has no recovery option)

How Our Architecture Prevents This:

  • Step 3 Fails: Compliance-mode locks prevent backup deletion by anyone
  • Air-Gapping: Backups logically isolated, not directly accessible
  • External Approval: Attacker cannot compromise approval team (separate organization)
  • Multi-Party Requirement: Need 2+ trusted individuals to approve access
  • Result: Backups remain protected; organization can recover without paying ransom

Prerequisites

Production Environment Requirements

AWS Infrastructure:

  • AWS Organizations with minimum 2 accounts
  • One account designated as backup delegate administrator
  • Site-to-Site VPN between on-premises datacenter and AWS / Internet connectivity from VMware environment to AWS

VMware Environment:

  • VMware vSphere 6.7 or later
  • vCenter Server operational
  • Service account with backup permissions
  • Available resources: 4 vCPUs, 8GB RAM, 80GB disk for gateway VM

Network Requirements:

  • VPN bandwidth: 100 Mbps minimum (1 Gbps recommended)
  • Network latency: Under 50ms recommended
  • Firewall permits: HTTPS (port 443) to AWS endpoints
  • DNS resolution for AWS service endpoints

External Approval Account Requirements

New AWS Account:

  • Must be completely separate from your organization
  • Cannot be member of any AWS Organizations
  • Dedicated email address (example: [email protected])
  • Root account secured with hardware MFA token

Trusted Approvers (3 minimum):

  • Senior executives with authority to approve emergencies
  • Examples: CTO, CISO, CFO, Infrastructure Director
  • Must be available 24/7 for emergency response
  • Have out-of-band communication capability (personal phones, etc)

VMware Backup Gateway Setup

Configure AWS Organisations Delegation

Purpose: Designates your backup account as central backup administrator for entire organization.

Instructions:

  1. Sign in to your AWS Organizations management account
  2. Navigate to AWS Organizations service
  3. Go to Services → AWS service access
  4. Enable access for AWS Backup service
  5. Navigate to Delegated administrators
  6. Register your backup account as delegated administrator for AWS Backup service
  7. Verify delegation appears as “Active”

What This Accomplishes:

  • Backup account can manage backups across all organization accounts
  • Doesn’t require management account access for daily operations
  • Centralizes backup management and policies

Step 1.2: Download and Deploy Backup Gateway

Purpose: Installs virtual appliance that connects AWS Backup to your VMware environment.

In AWS Backup Console (Backup Account):

  1. Select your region (example: EU-WEST-2 for London)
  2. Navigate to AWS Backup → External resources → Gateways
  3. Click Create gateway
  4. Download the OVF template file (approximately 1.2 GB)
  5. Save as: aws-appliance-latest.ova

In VMware vSphere Client:

  1. Connect to vCenter Server
  2. Right-click parent object (datacenter or cluster)
  3. Select Deploy OVF Template
  4. Choose Local file and select downloaded OVA file
  5. Provide gateway name: Backup-Gateway-Production
  6. Select compute resource (cluster or host)
  7. Critical: Select storage disk format: Thick Provision Lazy Zeroed
  8. Select management network (must have internet access)
  9. Complete deployment wizard

Configure VM Settings Before Power-On:

  1. Right-click deployed gateway VM → Edit Settings
  2. Verify configuration:
    • CPU: 4 vCPUs
    • Memory: 8 GB (set memory reservation to 8192 MB)
    • Hard Disk: 80 GB
  3. Go to VM Options → VMware Tools
  4. Enable: Synchronize Time with Host
  5. Enable: Synchronize at startup and resume
  6. Save settings

Configure Gateway Network Settings

Power On Gateway VM:

  1. Right-click gateway VM → Power → Power On
  2. Open VM console

Initial Login:

  • Default username: admin
  • Default password: password
  • You’ll be prompted to change password immediately
  • Create strong password (minimum 12 characters, mixed case, numbers, symbols)

Configure Static IP Address:

  1. At main menu, select Configure Network
  2. Choose Static IP configuration
  3. Enter network details:
    • IP Address: (assign from your management network range)
    • Subnet Mask: (example: 255.255.255.0)
    • Default Gateway: (your network gateway)
    • Primary DNS: (your internal DNS server)
    • Secondary DNS: (backup DNS, can use 8.8.8.8)
  4. Save configuration

Test Network Connectivity:

  1. At main menu, select Test Network Connectivity
  2. Gateway tests:
    • Basic network connectivity
    • DNS resolution
    • Internet access
    • AWS endpoint reachability
    • Time synchronization
  3. All tests should show “OK” or “PASS”
  4. Record the gateway IP address for next step

Firewall Requirements:

If you have firewalls between gateway and internet, allow outbound traffic:

  • Destination: *.backup.[your-region].amazonaws.com (port 443)
  • Destination: *.s3.[your-region].amazonaws.com (port 443)
  • Destination: time.aws.com (port 123 UDP)
  • No inbound rules required (all connections are outbound)

Register Gateway with AWS

In AWS Backup Console:

  1. Navigate to External resources → Gateways
  2. Click Register gateway
  3. Enter gateway details:
    • Gateway IP Address: (IP from previous step)
    • Gateway Name: Production-VMware-Gateway
    • Gateway Timezone: (select your timezone)
  4. Add tags for organization:
    • Environment: Production
    • Purpose: VMware-Backup
    • Location: On-Premises
  5. Click Register gateway

Verify Connection:

  • Wait 2-5 minutes for connection
  • Status should change from “Registering” to “Connected”
  • Green indicator shows healthy connection
  • If connection fails, verify firewall rules and network connectivity

Integrate VMware vCenter

Create vCenter Service Account:

In vCenter Server, create dedicated service account for AWS Backup with these permissions:

Required Permissions:

  • Virtual Machine: All inventory, configuration, state, and provisioning operations
  • Datastore: Browse datastore, allocate space
  • Network: Assign network
  • Apply at: Datacenter or Cluster level
  • Propagate to child objects: Yes

Add Hypervisor in AWS Backup:

  1. Navigate to External resources → Hypervisors
  2. Click Add hypervisor
  3. Select your registered gateway
  4. Choose Hypervisor Type: VMware vCenter
  5. Enter connection details:
    • Host: (vCenter IP address or hostname)
    • Port: 443 (default)
    • Username: (service account created above)
    • Password: (service account password)
  6. Provide hypervisor name: Production-vCenter
  7. Click Test Connection to verify
  8. Click Add hypervisor

Wait for VM Discovery:

  • AWS Backup automatically discovers all VMs (5-10 minutes)
  • Progress shown in console
  • After completion, view discovered VMs under External resources → Virtual machines

Create VMware Tags for Backup Selection

In vSphere Client:

  1. Navigate to Tags & Custom Attributes → Tags
  2. Click New Tag Category

Create Tag Category:

  • Category Name: backup
  • Description: Backup schedule identification
  • Cardinality: Single value per object

Create Tags Under ‘backup’ Category:

  1. Tag: daily
    • For VMs requiring daily backups
    • Example: Production databases, critical applications
  2. Tag: weekly
    • For VMs requiring weekly backups
    • Example: Development servers, secondary systems
  3. Tag: monthly
    • For VMs requiring monthly backups only
    • Example: Archive systems, long-term storage
  4. Tag: none
    • For VMs excluded from backups
    • Example: Temporary VMs, easily recreated systems

Apply Tags to VMs:

  1. Right-click each VM in vSphere inventory
  2. Select Tags & Custom Attributes → Assign Tag
  3. Choose appropriate backup tag
  4. VM will now be automatically included in matching backup plan

Tagging Strategy Example:

  • Mission-critical database servers: backup:daily
  • Application servers: backup:daily
  • File servers: backup:daily
  • Development servers: backup:weekly
  • Test environments: backup:none

Create Backup Plan

In AWS Backup Console:

  1. Navigate to Backup plans
  2. Click Create backup plan
  3. Select Build a new plan

Backup Plan Configuration:

Plan Name: VMware-Production-Daily-Backup

Backup Rule Configuration:

  • Rule Name: DailyBackupRule
  • Backup Vault: Default (temporary; will add air-gapped vault in Phase 2)
  • Schedule:
    • Frequency: Daily
    • Time: 2:00 AM (choose off-peak time for your organization)
    • Timezone: Your local timezone
    • Backup window start: Within 1 hour
    • Completion window: Within 3 hours

Lifecycle Settings:

  • Transition to cold storage: 30 days
  • Expire/Delete: 90 days
  • (Air-gapped vault will have longer retention)

Tags for Recovery Points:

  • BackupType: Daily
  • Environment: Production
  • Automated: True

Create Backup Selection:

After creating plan, immediately create backup selection:

  1. Selection Name: Tagged-VMs-Daily-Production
  2. IAM Role: Select Default role (AWS creates automatically)
  3. Resource Selection: Include specific resource types
  4. Resource Type: Select VM (Virtual Machine)

Define Selection by Tags:

  • Tag Key: backup
  • Tag Value: daily
  • Condition: Equals

Optional Additional Filter:

  • Tag Key: environment
  • Tag Value: production
  • This ensures only production VMs with daily tag are backed up
  1. Click Assign resources

What Happens Now:

  • AWS Backup automatically discovers all VMs with backup:daily tag
  • First backup runs at next scheduled time (2:00 AM)
  • You can trigger manual backup immediately for testing

Execute First Backup

Trigger Manual Backup (Don’t Wait for Schedule):

  1. Navigate to Protected resources
  2. Locate a test VM (non-production, with sample data)
  3. Click the VM name
  4. Click Create on-demand backup
  5. Select backup vault: Default
  6. Use default IAM role
  7. Start backup immediately
  8. Click Create on-demand backup

Monitor Backup Progress:

  1. Navigate to Jobs → Backup jobs
  2. Find your job at top of list
  3. Watch status progression:
    • Created → Job queued
    • Running → Backup in progress (shows percentage)
    • Completed → Backup successful

First Backup Timing:

  • Full backup typically takes 1-3 hours depending on VM size
  • Shows progress percentage throughout
  • Backup size approximately equals VM disk usage

Verify Backup Completed:

  1. Navigate to Backup vaults → Default
  2. Click Recovery points tab
  3. Find your VM’s recovery point
  4. Verify:
    • Status: Completed (green)
    • Backup size: Reasonable for your VM
    • Creation date: Today

Important: Incremental Backups

  • First backup is always full snapshot
  • Second and subsequent backups use Changed Block Tracking (CBT)
  • Incremental backups are 90-95% smaller
  • Complete in minutes instead of hours
  • Automatic – no configuration needed

Test Restore

Initiate Test Restore:

  1. Navigate to Backup vaults → Default
  2. Click Recovery points tab
  3. Select your test VM’s recovery point
  4. Click Restore

Restore Configuration:

VMware Destination Settings:

  • Target Hypervisor: Select your vCenter
  • Resource Pool: Select appropriate pool
  • Datastore: Select storage location
  • VM Folder: Create RestoredVMs folder for test restores
  • Network: Map networks appropriately

VM Settings:

  • VM Name: TestVM-Restored-Validation
  • Power On: Yes (to immediately test functionality)

IAM Role: Select default role

  1. Click Restore

Monitor Restore:

  • Navigate to Jobs → Restore jobs
  • Watch status: Running → Completed
  • Typical restore time: 30 minutes – 2 hours

Validate Restored VM:

In vSphere Client:

  1. Navigate to RestoredVMs folder
  2. Verify VM exists: TestVM-Restored-Validation
  3. Confirm VM is powered on
  4. Open console and verify:
    • Guest operating system boots normally
    • All disks present and accessible
    • Applications start correctly
    • Data integrity is intact (compare sample files)

Document Recovery Metrics:

  • Recovery Point Objective (RPO): Time difference between restore point and actual data
  • Recovery Time Objective (RTO): Time from restore initiation to VM operational
  • These metrics are critical for disaster recovery planning

Delete Test Restore:

  • After validation, delete restored test VM
  • Prevents confusion and saves storage
  • Keep documented results for reference

Your organisation now has:

  • Operational backup gateway connected to AWS
  • VMware vCenter fully integrated
  • Tag-based backup policies configured
  • First full backup completed successfully
  • Incremental backup capability verified
  • Restore process tested and validated

Air-Gapped Vault with Compliance Locks

Compliance-Mode Locks

What is Compliance-Mode Lock:

  • Makes backup vault permanently immutable
  • Backups cannot be deleted before retention period expires
  • Not even root account owner can bypass
  • Not even AWS support can override
  • Once grace period expires, lock is irreversible

During grace period you can:

  • Test backup and restore operations
  • Verify retention policies work correctly
  • Delete vault if you change your mind (last chance)

After grace period expires:

  • Lock becomes permanent
  • No changes possible
  • Vault exists until all backups expire naturally

Warning:

This is a point of no return decision. Before proceeding:

  • Get written approval from senior management
  • Understand financial commitment (vault costs money for entire retention period)
  • Test thoroughly during grace period
  • Document retention requirements clearly

Create Dedicated KMS Encryption Key

In AWS Key Management Service (KMS):

  1. Navigate to KMS service
  2. Select same region as backup vault (example: EU-WEST-2)
  3. Click Customer managed keys → Create key

Key Configuration:

Step 1 – Configure Key:

  • Key Type: Symmetric
  • Key Usage: Encrypt and decrypt
  • Key Material Origin: KMS
  • Regionality: Single-Region key

Step 2 – Add Labels:

  • Alias: backup-airgapped-vault-encryption
  • Description: Encryption key for air-gapped backup vault - Production environment
  • Tags:
    • Purpose: Backup-Encryption
    • Environment: Production
    • VaultType: AirGapped

Step 3 – Define Key Administrators:

  • Select your IAM user or role as key administrator
  • This allows you to manage key policies
  • Key administrators cannot use key to encrypt/decrypt

Step 4 – Define Key Usage Permissions:

  • Select: AWS Backup service (allows AWS Backup to use key)
  • Select: Your backup administrator IAM role
  • This grants permission to encrypt and decrypt backup data

Step 5 – Review and Create:

  • Review all settings carefully
  • Click Finish

Document Key Information:

  • Copy Key ID (format: a1b2c3d4-…)
  • Copy Key ARN (format: arn:aws:kms:region:account:key/…)
  • Store in secure documentation location
  • You’ll need this for vault creation

Create Air-Gapped Backup Vault

In AWS Backup Console:

  1. Navigate to Backup vaults
  2. Click Create backup vault
  3. Important: Select Create logically air-gapped vault option

Vault Configuration:

Basic Information:

  • Vault Name: Production-AirGapped-Vault
  • Vault Type: Logically air-gapped vault
  • Description: Immutable backup vault for ransomware protection

Encryption:

  • Select: Choose a custom encryption key
  • Select the KMS key you created in Step 2.1
  • Key alias: backup-airgapped-vault-encryption

Retention Configuration:

  • Minimum Retention Days: 30
  • Maximum Retention Days: 365
  • Adjust based on your compliance requirements

Tags:

  • Environment: Production
  • Purpose: Ransomware-Protection
  • ComplianceMode: True
  • CreatedDate: (today’s date)
  1. Click Create vault

Document Vault ARN

After creation:

  • Copy the full Vault ARN (format: arn:aws:backup:region:account:backup-vault:Production-AirGapped-Vault)
  • Print this ARN and store in physical safe
  • Store digital copy in password manager
  • You will need this ARN for disaster recovery
  • Without this ARN, you cannot request access during emergency

Step 2.3: Apply Compliance-Mode Vault Lock

FINAL WARNING – READ CAREFULLY:

╔════════════════════════════════════════════════════════╗
║                  POINT OF NO RETURN                     ║
╠════════════════════════════════════════════════════════╣
║                                                        ║
║  You are about to enable COMPLIANCE MODE VAULT LOCK   ║
║                                                        ║
║  After grace period (default 3 days) expires:         ║
║  • Lock becomes PERMANENT and IMMUTABLE                ║
║  • Vault CANNOT be deleted by anyone                   ║
║  • Settings CANNOT be changed or modified              ║
║  • Even AWS support CANNOT bypass this lock            ║
║  • Vault exists until retention period expires         ║
║                                                        ║
║  Financial Commitment:                                 ║
║  • Estimated monthly cost: $325                        ║
║  • Commitment period: Retention period (365 days)      ║
║  • Cannot be canceled or refunded                      ║
║                                                        ║
║  Required Approvals:                                   ║
║  □ Management approval obtained                        ║
║  □ Backup/restore tested successfully                  ║
║  □ Retention requirements verified                     ║
║  □ Budget approval secured                             ║
║  □ Vault ARN documented offline                        ║
║  □ All implications understood                         ║
║                                                        ║
╚════════════════════════════════════════════════════════╝

If you have all approvals and understand implications:

In AWS Backup Console:

  1. Navigate to Backup vaults
  2. Select Production-AirGapped-Vault
  3. Click Actions → Configure vault lock

Vault Lock Configuration:

  • Lock Mode: Compliance mode (recommended for ransomware protection)
  • Minimum Retention Days: 30
  • Maximum Retention Days: 365
  • Grace Period (Changeable For Days): 3 (72 hours to test before permanent)
  1. Review warning dialog carefully
  2. Type confirm to acknowledge
  3. Click Apply vault lock

Grace Period Begins:

  • You now have 3 days to test thoroughly
  • Mark calendar for when lock becomes permanent
  • Use this time to validate restore operations
  • Last chance to delete vault if needed

Update Backup Plan for Air-Gapped Copy

Modify Existing Backup Plan:

  1. Navigate to Backup plans
  2. Select VMware-Production-Daily-Backup
  3. Click Edit
  4. Find DailyBackupRule
  5. Click Edit rule

Add Copy Destination:

Scroll to Copy to destination section:

  • Enable: Yes, copy backups to another vault
  • Destination Vault: Select Production-AirGapped-Vault

Lifecycle for Copied Backups:

  • Transition to cold storage: 90 days
  • Expire: 365 days

Why Different Lifecycle:

  • Primary vault: Short retention (90 days) for quick operational restores
  • Air-gapped vault: Long retention (365 days) for ransomware recovery
  • Cold storage after 90 days saves approximately 90% on storage costs
  1. Click Save changes

How Copy Jobs Work:

  1. Primary backup runs at scheduled time (2:00 AM) to Default vault
  2. After primary backup completes, copy job starts automatically
  3. Backup copied to air-gapped vault (typically 30 minutes – 2 hours)
  4. Both copies exist independently:
    • Primary can be deleted after 90 days (operational use)
    • Air-gapped copy protected for 365 days (ransomware protection)
    • If primary corrupted, air-gapped copy remains safe

Monitor Copy Job Execution

Wait for Next Scheduled Backup or Trigger Manual Backup:

After next backup completes, copy job automatically starts.

Monitor Copy Jobs:

  1. Navigate to Jobs → Copy jobs
  2. Locate most recent copy job
  3. Watch status progression:
    • Created: Job queued
    • Running: Copy in progress (shows percentage)
    • Completed: Copy successful

Typical Timeline:

  • Source backup size: 500 GB
  • Copy duration: 45-90 minutes
  • Network: Internal AWS (no egress charges)

Verify Copy in Air-Gapped Vault:

  1. Navigate to Backup vaults
  2. Select Production-AirGapped-Vault
  3. Click Recovery points tab
  4. Verify:
    • Recovery point from test VM exists
    • Status: Completed
    • Size: Matches primary backup
    • Retention: 365 days
    • Locked: Yes

Check Vault Statistics:

View vault summary:

  • Number of recovery points: Should match expected backup count
  • Total storage: Sum of all backup sizes
  • Locked status: Yes (with grace period countdown or “Locked” if expired)
  • Lock date: When lock becomes or became permanent

Test Restore from Air-Gapped Vault

Critical: Test during grace period while you can still delete vault if problems occur

Initiate Restore from Air-Gapped Vault:

  1. Navigate to Backup vaults
  2. Select Production-AirGapped-Vault
  3. Click Recovery points tab
  4. Select a recovery point
  5. Click Restore

Restore Configuration:

Use same settings as Phase 1 restore test:

  • Target: Your VMware environment
  • VM Name: TestVM-AirGapped-Restore
  • Resource pool, datastore, network: As appropriate
  • Power on after restore: Yes
  1. Click Restore

Monitor and Validate:

  1. Monitor in Restore jobs
  2. Wait for completion (30 minutes – 2 hours)
  3. Verify VM restored successfully
  4. Power on and test all functionality
  5. Confirm data integrity

Validation Proves:

  • Air-gapped vault is functioning correctly
  • You can recover from this vault independently
  • If primary vault compromised, this vault remains accessible
  • Compliance lock doesn’t prevent legitimate restore operations

Grace Period Testing Checklist

Day 1: Restore Validation

  • [ ] Full VM restore from air-gapped vault completed
  • [ ] Restored VM powered on successfully
  • [ ] All applications functional
  • [ ] Data integrity verified (checksums match)
  • [ ] No errors or issues encountered

Day 2: Operational Validation

  • [ ] Second backup copied to air-gapped vault
  • [ ] Copy jobs completing within expected timeframe
  • [ ] Vault storage costs align with projections
  • [ ] No concerns about retention settings
  • [ ] Team trained on vault purpose and access restrictions

Day 3: Final Review and Approval

  • [ ] Management sign-off obtained
  • [ ] Vault ARN documented offline (printed and secured)
  • [ ] Emergency contact information prepared
  • [ ] Recovery procedures documented
  • [ ] Decision: Proceed with permanent lock

If Any Issues Found:

  • Delete vault during grace period (last opportunity)
  • Fix identified issues
  • Create new vault and restart process
  • Grace period provides safety net for testing

If Everything Validates:

  • Allow grace period to expire naturally
  • Lock becomes permanent automatically
  • Vault is now immutable ransomware protection

Post-Lock Verification

After Grace Period Expires:

Verify Permanent Lock Status:

  1. Navigate to Backup vaults
  2. Select Production-AirGapped-Vault
  3. Verify vault details:
    • Locked: Yes (no grace period remaining)
    • Lock Mode: Compliance
    • Lock Date: (date when lock became permanent)
    • Immutable: True

Test Lock Protection (Should Fail – Proves It Works):

Attempt 1: Try to Delete Vault

  1. Select vault
  2. Click Actions → Delete
  3. Expected: Error message “Cannot delete vault – protected by compliance-mode lock”
  4. This proves protection is working correctly

Attempt 2: Try to Modify Lock Settings

  1. Select vault
  2. Click Actions → Configure vault lock
  3. Expected: All options greyed out / disabled
  4. This proves lock is truly immutable

Attempt 3: Try to Delete Individual Backup

  1. Select a recovery point
  2. Try to delete
  3. Expected: Deletion blocked by retention policy
  4. Backup can only be deleted after retention period expires naturally

Document Lock Status:

Record in your documentation:

  • Lock Status: Permanent / Immutable
  • Lock Applied Date: [date]
  • Earliest Possible Deletion: [date + 30 days minimum retention]
  • Verified By: [your name]
  • Next Review: [quarterly review date]

Your organization now has:

  • Dedicated KMS encryption key for air-gapped vault
  • Logically air-gapped vault with compliance-mode lock
  • Automated copy jobs from primary to air-gapped vault
  • Tested restore capability from air-gapped vault
  • Permanent immutable protection active
  • Vault ARN documented in secure offline location

External Approval Team Configuration

Understanding External Approval Teams

Why External Account is Critical:

Vulnerable Setup (What NOT To Do):

Approval Team → Located in your AWS Organization
↓
Attacker compromises organization
↓
Attacker can compromise approval team
↓
Result: Backups accessible to attacker

Secure Setup (What We’re Building):

Approval Team → Separate AWS account (outside organization)
↓
Attacker compromises organization
↓
Approval team remains isolated and secure
↓
Result: Backups protected, attacker cannot approve access

Key Security Principle:

Even if ransomware attackers gain root access to every account in your organization, they cannot access air-gapped vault without approval from external team members who are outside the compromised environment.

Create External Approval Account

Create Completely Separate AWS Account:

Critical Requirements:

  • Must be standalone AWS account
  • Cannot be member of your AWS Organization
  • Cannot be part of any organizational structure
  • Managed by separate administrators
  • Dedicated email address (not shared with production accounts)

Account Creation Process:

  1. Go to https://aws.amazon.com/
  2. Click Create an AWS Account
  3. Use dedicated email address
  4. Account name: Backup-Approval-Authority
  5. Complete registration process
  6. Provide payment method (monthly cost will be ~$0)

Immediate Security Configuration:

1. Secure Root Account:

  • Sign in as root user immediately
  • Navigate to Security credentials
  • Enable MFA using hardware token (strongly recommended) or authenticator app
  • Create strong root password (20+ characters)
  • Store credentials in password manager
  • Record recovery codes securely

2. Set Account Alias:

  • Navigate to IAM → Dashboard
  • Create account alias: backup-approval-authority
  • This creates friendly URL: https://backup-approval-authority.signin.aws.amazon.com

3. Enable CloudTrail Logging:

  • Navigate to CloudTrail service
  • Create trail: approval-team-audit-trail
  • Apply to all regions: Yes
  • Log file validation: Enabled
  • Create new S3 bucket for logs
  • Enable log encryption (optional but recommended)

Why CloudTrail is Critical:

  • Logs every approval action
  • Provides complete audit trail
  • Required for compliance
  • Cannot be disabled (ransomware protection)
  • Helps forensics if incident occurs

Enable IAM Identity Center

What is IAM Identity Center:

  • Centralized user management for AWS
  • Formerly known as “AWS SSO”
  • Built-in MFA enforcement
  • Required for approval team functionality

Important Requirement:

IAM Identity Center must be enabled in US East (N. Virginia) region (us-east-1). This is an AWS requirement and cannot be changed.

Enable Identity Center

Configure Multi-Factor Authentication:

  1. In IAM Identity Center, click Settings
  2. Navigate to Authentication tab
  3. Under Multi-factor authentication section:
    • Enable MFA: Yes
    • Prompt users for MFA: Every time (most secure)
    • Allow these MFA types:
      • Authenticator apps (Google Authenticator, Authy, 1Password)
      • Security keys (YubiKey, other FIDO2 devices)
      • Built-in authenticators
  4. Click Save changes

Configure Password Policy:

Still in Settings → Authentication:

Password requirements:

  • Minimum length: 14 characters
  • Require uppercase letters: Yes
  • Require lowercase letters: Yes
  • Require numbers: Yes
  • Require symbols: Yes
  • Password expiration: 90 days
  • Prevent password reuse: Last 24 passwords
  • Account lockout: 5 failed attempts
  • Lockout duration: 15 minutes

Configure Session Duration:

In Settings:

  • Session duration: 8 hours
  • Idle timeout: 1 hour
  • This balances security with usability

Create Approval Team Members

Identify Trusted Approvers:

Selection Criteria:

  • Senior executive level (C-suite or Director)
  • Technical understanding of disaster recovery
  • Authority to approve emergency actions
  • Available 24/7 for emergency response
  • Trusted with company-critical decisions
  • Ideally not IT administrators (separation of duties)

Example Approval Team Composition:

Approver 1: Chief Technology Officer (CTO)

  • Role: Technical authority and infrastructure oversight
  • Responsibility: Verify technical legitimacy of requests

Approver 2: Chief Information Security Officer (CISO)

  • Role: Security authority and incident response
  • Responsibility: Verify security implications and threats

Approver 3: Chief Financial Officer (CFO) or Infrastructure Director

  • Role: Business continuity and operational authority
  • Responsibility: Authorize business impact decisions

Create Users in IAM Identity Center:

  1. In IAM Identity Center, navigate to Users
  2. Click Add user

For Each Approver, Configure:

User Details:

  • Username: (first.last format, example: john.smith)
  • Email address: (work email, must be valid and monitored)
  • First name: (example: John)
  • Last name: (example: Smith)
  • Display name: (example: John Smith)

Optional but Recommended:

  • Job title: (example: Chief Technology Officer)
  • Department: (example: Executive Leadership)
  • Phone number: (for out-of-band verification)
  1. Click Next
  2. Skip group assignment (for now)
  3. Click Next
  4. Review details
  5. Click Add user

Repeat for all approval team members (minimum 3 recommended)

Users Receive Setup Emails:

Each user receives invitation email:

Subject: Set up your AWS IAM Identity Center account

You've been invited to join the Backup-Approval-Authority 
AWS account.

Click here to complete setup: [Link expires in 7 days]

Setup Requirements:
1. Create password (minimum 14 characters)
2. Configure MFA device (required)
3. Save recovery codes
4. Complete profile

Important: Complete setup within 7 days or invitation expires.

User Setup Process

Each Approval Team Member Must Complete:

Step 1: Create Password

  • Click invitation link received via email
  • Create strong password meeting requirements:
    • Minimum 14 characters
    • Mix of uppercase, lowercase, numbers, symbols
    • Example: MySecure$Backup#Approval2025!
  • Confirm password
  • Click Continue

Step 2: Register MFA Device

Choose MFA device type:

  • Recommended: Authenticator app (Google Authenticator, Authy, Microsoft Authenticator, 1Password)
  • Alternative: Hardware security key (YubiKey or similar)

For Authenticator App:

  1. Open authenticator app on smartphone
  2. Select Add account or scan QR code option
  3. Scan QR code displayed in AWS console
  4. App generates 6-digit codes every 30 seconds
  5. Enter two consecutive codes to verify
  6. Click Assign MFA device

Important MFA Setup Notes:

  • Save backup codes in secure location
  • Test MFA before closing setup
  • If smartphone lost, recovery codes allow access

Step 3: Complete User Profile

  • Verify display name is correct
  • Verify email address
  • Add phone number (used for out-of-band verification)
  • Review profile details
  • Click Complete setup

Step 4: Test Initial Sign-In

  1. Sign out from setup session
  2. Navigate to: https://backup-approval-authority.signin.aws.amazon.com
  3. Enter username
  4. Enter password
  5. Enter current MFA code from authenticator app
  6. Should successfully sign in and see approval portal

Verify Team Member Status

Check All Users Are Active:

  1. In IAM Identity Center, navigate to Users
  2. Verify each user shows:
    • Status: Active (green indicator)
    • MFA: Enabled (green checkmark)
    • Email verified: Yes
    • Last sign-in: Recent date

Expected Status Table:

Username        Status    MFA       Email     Last Sign-In
─────────────────────────────────────────────────────────
john.smith      Active    Enabled   Yes       [Today]
jane.doe        Active    Enabled   Yes       [Today]
bob.johnson     Active    Enabled   Yes       [Today]

If Any User Shows “Pending”:

  • User hasn’t completed setup
  • Send reminder email to user
  • Verify invitation hasn’t expired (7-day limit)
  • Resend invitation if necessary

Team Member Testing:

  • Each member should sign in at least once
  • Verify MFA prompts appear and work correctly
  • Confirm access to approval portal
  • Test portal navigation and visibility

Enable Multi-Party Approval in Backup Account

Switch Context to Production Backup Account:

Now configure your production backup account to accept external approval teams.

In Production Backup Account:

  1. Sign in to backup administrator account
  2. Select your backup region (example: EU-WEST-2)
  3. Navigate to AWS Backup service
  4. Click Settings in left navigation menu

Enable Required Features:

Enable all three cross-account features:

  1. Cross-Account Backup: Enable
  2. Cross-Account Monitoring: Enable
  3. Multi-Party Approval: Enable

What These Settings Enable:

  • Cross-Account Backup: Allows backup sharing between AWS accounts
  • Cross-Account Monitoring: View backup status across multiple accounts
  • Multi-Party Approval: Required for external approval team integration
  1. Click Save changes

Verification:

  • All three settings should show “Enabled” status
  • Green checkmarks next to each feature
  • If any setting fails to enable, check IAM permissions

Create Approval Team

Return to External Approval Account:

In External Approval Account (us-east-1 region):

  1. Navigate to AWS Backup service
  2. Region must be US East (N. Virginia) / us-east-1
  3. Click Approval teams in left menu
  4. Click Create approval team

Approval Team Configuration:

Team Details:

  • Team Name: Production-Recovery-Team
  • Description: External approval team for emergency backup access. Requires minimum 2 of 3 approvals for vault access during ransomware incidents or organizational compromise.

Add Team Members:

For each IAM Identity Center user created earlier:

Member 1:

  • User: Select first user from dropdown
  • Email: (auto-populated from Identity Center)
  • Role: Approver

Member 2:

  • User: Select second user
  • Email: (auto-populated)
  • Role: Approver

Member 3:

  • User: Select third user
  • Email: (auto-populated)
  • Role: Approver

Approval Threshold:

  • Minimum approvals required: 2
  • This means 2 out of 3 members must approve any access request
  • Prevents single point of failure
  • Balances security with availability (if 1 member unavailable, can still proceed)

Tags for Organization:

  • Purpose: Emergency-Recovery
  • Organization: External-Authority
  • CriticalityLevel: High
  • Environment: Production
  1. Click Create approval team

⚠️ Critical: Document Approval Team ARN

After creation:

  • Copy Approval Team ARN (format: arn:aws:backup:us-east-1:ACCOUNT:approval-team/Production-Recovery-Team)
  • Print this ARN and store in physical safe
  • Store in password manager
  • You will need this ARN for disaster recovery
  • Without this ARN, cannot request or grant access during emergency

Team Activation Process

Automatic Invitation Emails Sent:

Immediately after approval team creation, each member receives:

From: AWS Backup Multi-Party Approval
Subject: [ACTION REQUIRED] Join Approval Team: Production-Recovery-Team

You've been invited to join an approval team for emergency 
backup access.

Team: Production-Recovery-Team
Your Role: Approver
Approval Threshold: 2 of 3 members must approve

CRITICAL REQUIREMENTS:
• ALL team members must accept within 24 hours
• If ANY member declines, team becomes inactive
• MFA is REQUIRED (already configured during setup)
• You must be available 24/7 for emergency approvals

To Accept Invitation:
1. Click invitation link: [Link expires in 24 hours]
2. Sign in with IAM Identity Center credentials
3. Complete MFA verification
4. Review team details
5. Accept membership

Approval Portal: https://backup-approvals.aws.amazon.com/

Questions? Contact your security team immediately.

Each Team Member Must:

  1. Click invitation link (within 24 hours)
  2. Sign in to IAM Identity Center:
    • Username: (their username)
    • Password: (their password)
    • MFA Code: (6-digit code from authenticator app)
  3. Review approval team invitation details:
    • Team name: Production-Recovery-Team
    • Team purpose: Emergency backup access approval
    • Minimum approvals: 2 of 3
    • Responsibilities: 24/7 availability, out-of-band verification
  4. Click Accept invitation
  5. Confirm understanding of role and responsibilities

Critical 24-Hour Window:

  • If all 3 members accept within 24 hours → Team becomes ACTIVE
  • If any 1 member doesn’t accept → Team becomes INACTIVE
  • If any 1 member declines → Team becomes INACTIVE
  • Inactive teams must be deleted and recreated with new invitations

Verify Team Activation

Monitor Team Status:

  1. In AWS Backup (external account), navigate to Approval teams
  2. Select Production-Recovery-Team
  3. View Team details

During Invitation Period (First 24 Hours):

Team Status: Pending Activation
Members: 3

Member Status:
- john.smith: Invitation sent, pending acceptance
- jane.doe: Invitation sent, pending acceptance
- bob.johnson: Invitation sent, pending acceptance

Time Remaining: 18 hours 45 minutes
Action Required: All members must accept invitation

After All Members Accept:

Team Status: ACTIVE ✓
Members: 3

Member Details:
- john.smith: Active, MFA: Enabled, Joined: [timestamp]
- jane.doe: Active, MFA: Enabled, Joined: [timestamp]
- bob.johnson: Active, MFA: Enabled, Joined: [timestamp]

Team Information:
Created: [date/time]
Activated: [date/time]
Minimum Approvals Required: 2 of 3
Ready for Assignment: Yes

Verify Each Member:

  • Status: Active (green)
  • MFA: Enabled (green checkmark)
  • Last sign-in: Recent date
  • All members can access approval portal

Test Approval Portal Access

Each Member Should Independently Test:

Access Approval Portal:

  1. Navigate to: https://backup-approvals.aws.amazon.com/
  2. Sign in with IAM Identity Center credentials
  3. Complete MFA verification
  4. Should see: Approval Portal Dashboard

Dashboard Should Display:

  • User’s name and role (Approver)
  • Team membership: Production-Recovery-Team
  • Pending requests: 0 (none yet – this is expected)
  • Approval history: Empty (no approvals yet – this is expected)
  • Team members: List of all 3 team members

Test Portal Navigation:

  • Click Requests tab: Should show “No pending requests”
  • Click Team tab: Should show all team members and their status
  • Click History tab: Should show “No approval history”
  • Verify user can navigate all sections without errors

Troubleshooting Access Issues:

If member cannot access portal:

  • Verify user accepted team invitation (check email)
  • Confirm MFA is configured and working
  • Check user status is “Active” in Identity Center
  • Try signing out completely and signing in again
  • Verify using correct portal URL

Document Portal Access:

Record in documentation:

  • Portal URL: https://backup-approvals.aws.amazon.com/
  • All 3 members tested access successfully: Yes
  • Date tested: [today’s date]
  • Next access test: [quarterly drill date]

Your organization now has:

  • External approval account (completely separate from production organization)
  • IAM Identity Center configured with strong MFA enforcement
  • Three trusted approval team members with active accounts and MFA
  • Approval team created and fully activated
  • All members can access approval portal
  • Approval team ARN documented in secure offline location

Proceed to Phase 4: Cross-Account Integration

Cross-Account Integration

Cross-Account Resource Sharing

What We’re Accomplishing:

Technology: AWS Resource Access Manager (RAM)

  • AWS service for secure resource sharing between accounts
  • Works across AWS Organizations and external accounts
  • Provides audited, secure sharing
  • Required for external approval team integration

The Integration Flow:

External Approval Account (has approval team)
↓
[Share via AWS RAM]
↓
Production Backup Account (has air-gapped vault)
↓
[Associate approval team with vault]
↓
Vault requires approval team authorization for access

Share Approval Team via AWS RAM

In External Approval Account:

  1. Sign in to external approval account
  2. Region: US East (N. Virginia) / us-east-1
  3. Navigate to AWS Resource Access Manager (RAM) service

Create Resource Share:

  1. Click Create resource share

Step 1 – Resource Share Details:

  • Name: Backup-Approval-Team-Share
  • Description: Shares Production-Recovery-Team with production backup account for emergency vault access authorization

Step 2 – Select Resources to Share:

  • Resource type: Select Backup: Approval Team
  • Select resources: Check Production-Recovery-Team
  • This is the approval team created in Phase 3

Step 3 – Grant Access to Principals:

Critical Setting:

  • Principal type: Select AWS account
  • Enter AWS account ID: (your production backup account ID)
  • Allow external principals: ✓ CHECK THIS BOX
    • This is critical for cross-organization sharing
    • Without this, sharing will fail
    • Required because approval account is outside your organization

Step 4 – Add Tags:

  • Purpose: Emergency-Recovery-Authorization
  • TargetAccount: (backup account ID)
  • SecurityLevel: Critical
  • SharedResource: ApprovalTeam
  1. Review all settings
  2. Click Create resource share

Resource Share Status:

  • Status: Pending (waiting for acceptance by backup account)
  • Share ARN: (document this for reference)
  • Shared resources: 1 (approval team)
  • Principals: 1 (backup account)

Accept Resource Share in Backup Account

Switch to Production Backup Account:

  1. Sign in to production backup account
  2. Region: EU-WEST-2 (or your backup region)
  3. Navigate to AWS Resource Access Manager (RAM)

View Pending Invitations:

  1. Click Shared with me in left navigation
  2. Click Resource shares tab
  3. You should see:
    • Resource share name: Backup-Approval-Team-Share
    • Status: Pending
    • Shared from: (external approval account ID)
    • Resources: 1 Backup approval team
    • Received: (timestamp)

Accept the Resource Share:

  1. Select the resource share
  2. Click Accept resource share button
  3. Confirmation dialog appears
  4. Read the confirmation message
  5. Click Accept

Verification After Acceptance:

  • Status changes from Pending to Active
  • Resource share now appears under “Accepted” section
  • Shared resources become available to use

Verify Approval Team Accessibility

In Production Backup Account:

  1. Navigate to AWS Backup service
  2. Region: EU-WEST-2 (your backup region)
  3. Click Approval teams in left navigation menu

You Should Now See:

Approval Teams

Name: Production-Recovery-Team
Shared From: [External Account ID]
Status: Active
Members: 3
Minimum Approvals: 2 of 3
Type: External (Cross-Account)

This Confirms:

  • Cross-account sharing successful
  • Approval team accessible from production account
  • Ready to be assigned to air-gapped vault
  • External approval team will protect your backups

If Approval Team Doesn’t Appear:

  • Verify resource share was accepted (check AWS RAM)
  • Confirm you’re in correct region (EU-WEST-2 or your backup region)
  • Wait 2-3 minutes for propagation
  • Check AWS Service Health Dashboard for any issues

Create and Apply Vault Access Policy

Purpose: Configure vault to require external approval for all restore operations.

In AWS Backup Console:

  1. Navigate to Backup vaults
  2. Select Production-AirGapped-Vault
  3. Click Access policy tab
  4. Click Edit policy

Select Policy Builder Option:

Create policy with three statements:

Statement 1 – Allow Backup Operations:

  • Effect: Allow
  • Principal: Service → backup.amazonaws.com
  • Actions:
    • backup:CopyIntoBackupVault (allows copy jobs)
  • Resources: * (all)
  • Conditions: None

Statement 2 – Require Approval for Restore:

  • Effect: Allow
  • Principal: * (anyone)
  • Actions:
    • backup:StartRestoreJob (restore operations)
    • backup:GetRecoveryPointRestoreMetadata (restore metadata)
  • Resources: * (all)
  • Condition (Critical):
    • Condition key: backup:ApprovalTeamArn
    • Operator: String equals
    • Value: (paste your approval team ARN from Phase 3)

Statement 3 – Deny Dangerous Operations:

  • Effect: Deny
  • Principal: * (anyone, including root)
  • Actions:
    • backup:DeleteRecoveryPoint (prevent backup deletion)
    • backup:DeleteBackupVault (prevent vault deletion)
    • backup:PutBackupVaultAccessPolicy (prevent policy modification)
  • Resources: * (all)
  • Conditions: None
  1. Review all three statements
  2. Click Save policy

What This Policy Enforces:

✓ Allows: AWS Backup service to copy backups into vault (normal operations) ✓ Requires: External approval team approval for any restore operation ✗ Denies: Anyone (including root) from deleting backups or vault ✗ Denies: Anyone from modifying this vault access policy (prevents tampering)

Policy Protection:

Once applied, this policy cannot be modified without:

  • Deleting and recreating vault (impossible due to compliance lock)
  • Or having specific IAM permissions (which should be tightly controlled)

Associate Approval Team with Vault

In Vault Configuration:

  1. While viewing Production-AirGapped-Vault
  2. Click Approval team tab
  3. Click Assign approval team

Association Settings:

  • Select approval team: Production-Recovery-Team (from external account)
  • Review association notice:
    • “This approval team is from an external account”
    • “All vault access will require approval from this team”
    • “Minimum 2 of 3 approvals required”
  1. Confirm you understand implications
  2. Click Assign approval team

Confirmation After Assignment:

You should see:

Approval Team Assignment

Team Name: Production-Recovery-Team
Team ARN: arn:aws:backup:us-east-1:[EXTERNAL-ACCOUNT]:approval-team/...
Status: Assigned
Shared From: [External Account ID]
Members: 3
Minimum Approvals: 2 of 3
Type: External (Cross-Organization)
Assignment Date: [Today's Date]

Comprehensive Integration Verification

Verification Checklist:

1. Vault Configuration:

Navigate to Production-AirGapped-Vault and verify:

  • [ ] Vault Type: Logically air-gapped
  • [ ] Locked: Yes (compliance mode)
  • [ ] Lock Date: (permanent lock date)
  • [ ] Encryption Key: Custom KMS key
  • [ ] Approval Team: Production-Recovery-Team (external)
  • [ ] Minimum Approvals: 2 of 3

2. Access Policy:

Review vault access policy and verify:

  • [ ] Allows: Backup copy operations
  • [ ] Requires: Approval team for restore operations
  • [ ] Denies: Deletion of backups
  • [ ] Denies: Deletion of vault
  • [ ] Denies: Policy modification

3. Recovery Points:

Click Recovery points tab and verify:

  • [ ] Backups present from automatic copy jobs
  • [ ] Status: Completed
  • [ ] Retention: 365 days
  • [ ] Each shows “Requires approval team authorization for access”
  • [ ] Cannot be deleted (test by trying – should fail)

4. Test Policy Enforcement:

Attempt restore without approval (should be blocked):

  1. Select a recovery point
  2. Click Restore
  3. Try to configure restore

Expected Result:

  • Error message: “Access denied – requires multi-party approval from external team”
  • Or: Restore button disabled with explanation
  • This proves policy enforcement is working correctly

5. CloudTrail Audit Verification:

Navigate to CloudTrail → Event history:

  • [ ] Resource share creation logged (external account)
  • [ ] Resource share acceptance logged (backup account)
  • [ ] Approval team assignment logged
  • [ ] Vault access policy changes logged
  • Complete audit trail available for compliance

6. Document Complete Configuration:

Record all details in secure documentation:

Production Air-Gapped Vault - Complete Configuration
═══════════════════════════════════════════════════

Vault Information:
─────────────────
Name: Production-AirGapped-Vault
ARN: arn:aws:backup:eu-west-2:[BACKUP-ACCOUNT]:backup-vault:Production-AirGapped-Vault
Region: EU-WEST-2
Account: [Backup Account ID]

Security Configuration:
──────────────────────
Compliance Lock: Active (Permanent)
Lock Date: [Date]
Minimum Retention: 30 days
Maximum Retention: 365 days
Encryption: Custom KMS Key

Approval Team:
─────────────
Name: Production-Recovery-Team
ARN: arn:aws:backup:us-east-1:[EXTERNAL-ACCOUNT]:approval-team/Production-Recovery-Team
Location: External Account (Outside Organization)
Account: [External Approval Account ID]

Team Members:
────────────
1. [Name] ([Title]) - [Email] - [Phone]
2. [Name] ([Title]) - [Email] - [Phone]
3. [Name] ([Title]) - [Email] - [Phone]

Approval Requirements:
─────────────────────
Minimum Approvals: 2 of 3
MFA Required: Yes (all members)
Out-of-Band Verification: Required before approval

Access Details:
──────────────
Approval Portal: https://backup-approvals.aws.amazon.com/
Emergency Contact Card: [Physical Location]
Vault ARN Document: [Physical Safe Location]

Integration Status:
──────────────────
RAM Resource Share: Active
Approval Team Assignment: Complete
Access Policy: Applied and Enforced
Testing: Completed Successfully

Verification:
────────────
Configuration Date: [Date]
Verified By: [Your Name]
Next Review: [Quarterly Review Date]
Last Tested: [DR Drill Date]

Your organization now has:

  • Approval team shared from external account via AWS RAM
  • Production backup account accepted and integrated shared team
  • Air-gapped vault configured with comprehensive access policy
  • Policy enforces multi-party approval requirement from external team
  • Approval team assigned to vault
  • Integration verified through comprehensive testing
  • Complete configuration documented

🎉 FULL ARCHITECTURE NOW OPERATIONAL!

Your backups are now protected by multiple security layers:

  • Immutable compliance-mode locks (cannot be deleted by anyone)
  • Logical air-gapping (isolated from production environment)
  • External approval authority (outside your organization)
  • Multi-party approval requirement (2+ trusted individuals)
  • MFA enforcement (all approvers)
  • Complete CloudTrail audit trail

Even if ransomware attackers compromise your entire AWS organization with root access, they cannot access or delete your backups without approval from external team members who are completely outside the compromised environment.

Proceed to Testing: Validate Your Complete Recovery Process

Testing Your Recovery Process

Complete Disaster Recovery Simulation

This simulation validates your entire architecture by testing a realistic ransomware attack scenario.

Simulation Scenario

Simulated Ransomware Incident:

Attack Timeline:

09:00 - Ransomware attack begins (phishing email)
09:15 - Attacker gains admin access
09:30 - Lateral movement across organization
09:45 - Attempted backup deletion (BLOCKED by compliance lock ✓)
10:00 - Production systems begin encrypting
10:30 - All systems encrypted
10:45 - Ransom note delivered
11:00 - Incident response team activated

Scope of Compromise:

  • All production AWS accounts compromised
  • All admin credentials stolen
  • Primary backup infrastructure considered untrusted
  • Standard backup vaults accessible but from compromised accounts
  • Air-gapped vault remains protected

Recovery Objective:

  • Access air-gapped backups from clean recovery environment
  • Restore critical systems
  • Resume business operations

Test Backup Solution

Prerequisites:

1. Clean Recovery Account:

  • Separate AWS account (not part of compromised organization)
  • Basic IAM roles configured for restore operations
  • Can be temporary account created just for this drill

2. Approval Team Ready:

  • All 3 members have phones accessible
  • Out-of-band communication method available (personal phones, Signal, WhatsApp)
  • Members briefed that drill is occurring
  • Members understand their role in verification

3. Documentation Available:

  • Vault ARN printed and accessible (simulate no digital access)
  • Approval team contact list printed
  • Recovery procedures printed
  • Emergency contact card accessible

4. Test Environment:

  • Test VM identified (contains sample data, no production data)
  • VM has recent backup in air-gapped vault
  • Target restore environment prepared (VMware resources available)

Execution Steps

Step 1: Declare Simulated Incident

Incident Commander Announces:

“At 10:30 AM, ransomware attack completed encryption of all production systems. All AWS accounts in our organization are compromised. All administrative credentials are untrusted. We are declaring a disaster recovery situation and activating emergency procedures to recover from air-gapped backups using external approval team. This is a drill. Start timer.”

  • Begin stopwatch
  • Begin logging all actions with timestamps
  • Treat as real incident for training value

Step 2: Request Vault Access

In Clean Recovery Account:

  1. Sign in to clean recovery account
  2. Navigate to AWS Backup service
  3. Select region matching your air-gapped vault (example: EU-WEST-2)
  4. Click Backup vaults in left navigation
  5. Click Request vault access button

Complete Access Request Form:

  • Source Vault ARN: (paste from printed documentation) arn:aws:backup:eu-west-2:[BACKUP-ACCOUNT]:backup-vault:Production-AirGapped-Vault
  • Destination Vault Name: Emergency-Recovery-Vault-[Today's-Date]
  • Requester Comment: (be detailed and specific) RANSOMWARE INCIDENT - URGENT Incident Summary: - Attack began: 09:00 AM [Today's Date] - All production systems encrypted: 10:30 AM - Scope: Complete organizational compromise Systems Affected: - All AWS accounts in organization compromised - All administrative credentials stolen and untrusted - Primary backup infrastructure inaccessible - Standard backup vaults accessible only from compromised accounts Requesting emergency access to air-gapped backups for recovery operations. Incident Details: - Incident Ticket: INC-[YYYY]-[001] - Incident Commander: [Name] - Contact Email: [email protected] - Emergency Phone: [Phone Number] Critical Systems to Restore (Priority Order): 1. Domain controllers 2. Database servers 3. Application servers Business Impact: All operations stopped Estimated Loss: $[Amount] per hour This is a legitimate emergency requiring immediate vault access.
  1. Click Submit request

Document Request Details:

  • Request ID: (example: req-abc123xyz)
  • Submission time: 11:15 AM
  • Requested by: [Your Name]

Step 3: Approval Team Notification (11:15 AM)

All Team Members Receive:

Email notification:

Subject: 🚨 URGENT: Vault Access Request Requires Immediate Approval

Request ID: req-abc123xyz
Received: [Today]
Priority: CRITICAL - Emergency Response Required

Source Vault: Production-AirGapped-Vault
Requesting Account: [Clean Recovery Account ID]

Requester Comment:
"RANSOMWARE INCIDENT - URGENT
Attack began: 09:00 AM
All production systems encrypted: 10:30 AM
Complete organizational compromise..."

THIS IS AN EMERGENCY REQUEST

Required Action:
• Access approval portal immediately
• Verify incident via out-of-band communication
• Review request details carefully
• Approve if legitimate emergency

Approval Portal: https://backup-approvals.aws.amazon.com/

Required Approvals: 2 of 3
Time Sensitive: Response needed within 30 minutes

Step 4: Out-of-Band Verification

Before any approval, team must verify via independent channels:

CTO Actions:

  1. Receives notification on personal email/phone
  2. Calls CISO on personal cell phone (not work phone)
  3. Discussion:
    • “Did you receive emergency vault access request?”
    • “Can you confirm this is real incident or drill?”
    • “Have you verified with incident response team?”
  4. Reviews incident details independently
  5. Attempts to verify incident ticket exists

CISO Actions:

  1. Receives notification
  2. Answers CTO verification call
  3. Calls Infrastructure Director for three-way verification
  4. Reviews security operations center reports
  5. Confirms ransomware incident is legitimate (or simulated for drill)
  6. Verifies requesting account ID matches authorized recovery account

Infrastructure Director Actions:

  1. Receives notification
  2. Answers CISO verification call
  3. Contacts incident response team via Signal or other secure channel
  4. Verifies no signs of social engineering attempt
  5. Confirms request details match known incident parameters

Verification Checklist (Team Discussion):

  • [ ] All three members on call together
  • [ ] Incident ticket verified in system
  • [ ] Incident commander identity confirmed
  • [ ] Requesting account ID verified against DR plan
  • [ ] No indicators of social engineering
  • [ ] Timing matches reported attack timeline
  • [ ] Multiple independent confirmations
  • [ ] Decision: Legitimate emergency – proceed with approval

Step 5: First Approval (11:30 AM)

CTO Signs In to Approval Portal:

  1. Navigate to: https://backup-approvals.aws.amazon.com/
  2. Sign in:
    • Username: [CTO username]
    • Password: [password]
    • MFA Code: [6-digit code from authenticator app]
  3. Dashboard shows: 1 Pending Request (highlighted in red/urgent)
  4. Click request ID: req-abc123xyz

Review Request Details:

  • Source vault: Production-AirGapped-Vault
  • Requesting account: [Clean Recovery Account ID]
  • Requester comment: [Full text displayed]
  • Time elapsed: 15 minutes

Add Detailed Approval Comment:

APPROVED - Emergency Vault Access Authorization

Verification Completed:
• Ransomware incident confirmed via phone call with CISO and Infrastructure Director
• Reviewed incident ticket INC-[YYYY]-001 showing complete system encryption 
• Verified primary AWS organization accounts compromised
• Confirmed administrative credentials cannot be trusted
• Verified this is legitimate emergency requiring access to air-gapped backups

Request Validation:
• Requesting account ID [ACCOUNT-ID] verified as authorized recovery account per disaster recovery plan section 4.2
• Request details consistent with reported incident timeline
• No indicators of social engineering or malicious intent
• Out-of-band verification completed successfully with 3 team members

Authorization:
Approving emergency vault access for recovery operations to restore critical business systems.

Authority: Chief Technology Officer - Emergency Powers
DR Plan Reference: Section 4.2 - Catastrophic Failure Recovery
Approved By: [CTO Name]
  1. Click Approve button
  2. Confirm approval

Result:

  • Approval 1 of 2 recorded
  • Status: “Awaiting second approval (1 of 2 received)”
  • CISO and Infrastructure Director see notification: “First approval received”

Step 6: Second Approval

CISO Signs In to Approval Portal:

  1. Navigate to approval portal
  2. Sign in with credentials and MFA
  3. See same request with existing CTO approval

Add CISO Approval Comment:

APPROVED - Security Authority Concurrence

Incident Confirmation:
• Confirmed ransomware incident via Security Operations Center
• Attack timeline verified: initial compromise 09:00 AM, full encryption 10:30 AM
• Reviewed CloudWatch alerts showing automated backup deletion attempts at 09:30 AM (successfully blocked by compliance lock - system working as designed)
• Verified production environment completely compromised

Security Assessment:
• Standard backup infrastructure in compromised organization cannot be trusted for recovery
• Primary accounts have root-level compromise
• Admin credentials stolen and actively being used by attacker
• Recovery must proceed from air-gapped vault via external clean account
• This is only secure path to recovery

Request Validation:
• Verified requesting account [ACCOUNT-ID] is pre-authorized recovery account
• Confirmed out-of-band with CTO and Infrastructure Director
• No signs of social engineering or unauthorized access attempt
• Request legitimacy confirmed through multiple channels

Authorization:
Approving emergency vault access. Recovery operations authorized to proceed immediately.

Authority: Chief Information Security Officer - Incident Response Authority
DR Plan Reference: Section 5.1 - Security Incident Recovery
Approved By: [CISO Name]
Date/Time: [Today] 11:35:00 AM
  1. Click Approve button
  2. Confirm approval

Result:

  • THRESHOLD MET: 2 of 2 Required Approvals Received
  • Status changes to: “APPROVED – Initiating Vault Sharing”
  • Automatic vault sharing process begins
  • All team members receive confirmation notification

Step 7: Automatic Vault Sharing (11:36 AM)

AWS Backup Automatically:

  1. Creates resource share:
    • Shares Production-AirGapped-Vault with clean recovery account
    • Creates read-only access to all recovery points
    • All backups immediately accessible
  2. Creates destination vault:
    • Vault name: Emergency-Recovery-Vault-[Today's-Date]
    • Type: Shared vault
    • Contains: Access to air-gapped vault
    • Status: Active
  3. Logs all actions:
    • Request logged in production account CloudTrail
    • First approval logged in external account CloudTrail
    • Second approval logged in external account CloudTrail
    • Vault sharing logged in production account CloudTrail
    • Access granted logged in recovery account CloudTrail
  4. Sends confirmations:
    • Email to requester (in recovery account)
    • Email to all approvers
    • Portal notifications
    • Audit trail complete

Step 8: Access Backups (11:40 AM)

In Clean Recovery Account:

  1. Navigate to AWS Backup → Backup vaults
  2. You should now see new vault:
Emergency-Recovery-Vault-[Today's-Date]
Type: Shared
Source Vault: arn:aws:backup:eu-west-2:[BACKUP-ACCOUNT]:backup-vault:Production-AirGapped-Vault
Status: Active
Shared From: [Backup Account ID]
Shared At: [Today] 11:36 AM
Recovery Points: 45 (example)
  1. Click vault name
  2. Click Recovery points tab
  3. See all available backups with dates

Step 9: Identify Recovery Point (11:45 AM)

Select Appropriate Backup:

  • Attack started: 09:00 AM
  • Need backup from before attack began
  • Last night’s scheduled backup: 02:00 AM (7 hours before attack)
  • This backup is clean and pre-attack
  • Select this recovery point

Verify Recovery Point:

  • Creation time: 02:00 AM (before attack)
  • Backup size: Reasonable
  • Status: Completed
  • Retention: 365 days

Step 10: Initiate Restore (12:00 PM)

  1. Select clean recovery point from 02:00 AM
  2. Click Restore

Configure Restore:

  • Target: Your VMware environment (or test environment)
  • VM Name: CriticalApp-Restored-[Today's-Date]
  • Resource Pool: Recovery resource pool
  • Datastore: Fast storage for quick restore
  • Network: Map to appropriate networks
  • Power On: Yes (start immediately after restore)
  1. Click Restore button

Monitor Restore Progress:

  • Navigate to Restore jobs
  • Watch status: Created → Running → Completed
  • Progress percentage shown
  • Estimated time: 30 minutes – 2 hours (depends on VM size)

Step 11: Validate Recovery (02:00 PM)

After restore completes:

In vSphere Client:

  1. Navigate to restored VM location
  2. Verify VM present: CriticalApp-Restored-[Today's-Date]
  3. Confirm VM powered on
  4. Open console

System Validation:

  • [ ] Guest OS boots normally
  • [ ] All disks present and accessible
  • [ ] Network connectivity functional
  • [ ] Applications start successfully
  • [ ] Data integrity verified (spot-check critical files)
  • [ ] System responds to user requests
  • [ ] Performance acceptable

Stop Timer – Document Recovery Time

Step 12: Document Drill Results

Test Validation and Metrics

Recovery Metrics Summary
════════════════════════

Detection to Request: 15 minutes ✓
Request to Notification: Immediate (automated) ✓
Notification to Verification: 5 minutes ✓
Verification to Approval: 15 minutes ✓
Approval to Access: 5 minutes ✓
Access to Restore Start: 20 minutes ✓
Restore Duration: 2 hours ✓
───────────────────────────────────────
Total Recovery Time: 3 hours ✓

Recovery Point Objective (RPO): 7 hours (last backup before attack)
Recovery Time Objective (RTO): 4 hours target → 3 hours actual ✓

Result: PASSED - Under RTO target

Validation Checklist:

Architecture Validation:
- [x] Compliance lock prevented attacker from deleting backups
- [x] Primary vaults inaccessible from compromised accounts
- [x] Air-gapped vault remained completely protected
- [x] External approval team isolated from compromise
- [x] Approval process functioned as designed
- [x] Out-of-band verification completed successfully
- [x] Multi-party approval enforced (2 of 3 required)
- [x] MFA required and verified for all approvals
- [x] Vault automatically shared after approval threshold
- [x] Recovery points accessible in clean account
- [x] Restore completed successfully
- [x] Data integrity validated
- [x] System operational and functional

Audit Trail Verification:
- [x] Vault access request logged with full context
- [x] Approval team notifications logged
- [x] First approval logged with approver identity and comment
- [x] Second approval logged with approver identity and comment
- [x] Vault sharing operation logged
- [x] Restore operation logged
- [x] All events captured in CloudTrail
- [x] Complete audit trail available for review

Team Performance:
- [x] Approval team responded within 5 minutes
- [x] Out-of-band verification completed properly
- [x] All approvers understood their roles
- [x] No confusion or procedural delays
- [x] Communication effective throughout
- [x] Procedures followed correctly
- [x] Team worked well under pressure simulation

Security Validation:
- [x] Compromised accounts could not access air-gapped vault
- [x] Compromised accounts could not delete backups
- [x] Compromised accounts could not bypass approval requirement
- [x] External approval team completely isolated from attack
- [x] MFA enforced at every step
- [x] No single point of failure exists in architecture

Issues Identified:
- [List any issues, confusion, or delays encountered]
- [Example: Approval portal login was slow - investigate]
- [Example: Vault ARN lookup took extra time - improve documentation]

Improvements Recommended:
- [List specific improvements to procedures or documentation]
- [Example: Add step-by-step screenshots to approval process]
- [Example: Create printed quick-reference card for approvers]
- [Example: Schedule more frequent training sessions]

Document Lessons Learned:

What Worked Well:

  • External approval team responded quickly (5 minutes)
  • Out-of-band verification prevented potential social engineering
  • Multi-party approval provided oversight without blocking recovery
  • Vault sharing was immediate and automatic after approval
  • Restore process was straightforward from shared vault

What Could Be Improved:

  • [Document specific areas for improvement]
  • [Update procedures based on drill experience]
  • [Identify training needs]

Action Items:

  • [ ] Update runbooks with any procedural clarifications
  • [ ] Schedule follow-up training for approval team
  • [ ] Add automation scripts for recovery account preparation
  • [ ] Create VM priority matrix for restore order
  • [ ] Schedule next quarterly drill (3 months from today)

You have successfully:

  • Simulated complete ransomware compromise of organization
  • Validated external approval team isolation from attack
  • Tested complete multi-party approval workflow
  • Verified out-of-band verification procedures
  • Proven vault access works from clean external account
  • Restored system from air-gapped backup successfully
  • Measured actual RTO/RPO metrics
  • Documented comprehensive lessons learned

Your architecture is proven to work during ransomware attacks!

The complete disaster recovery workflow has been validated end-to-end.

Operational Procedures

Daily Monitoring

  1. Review Backup Jobs:
    • Navigate to AWS Backup → Jobs → Backup jobs
    • Filter: Last 24 hours
    • Check: All jobs show “Completed” (green)
    • Investigate any “Failed” jobs immediately
  2. Review Copy Jobs:
    • Click Copy jobs tab
    • Verify backups copied to air-gapped vault
    • Check completion status
  3. Quick Vault Check:
    • Navigate to Backup vaults → Production-AirGapped-Vault
    • Verify: Locked status = Yes
    • Verify: Recovery point count increasing daily

Weekly Tasks

  1. Approval Team Health:
    • Sign in to external approval account
    • Navigate to IAM Identity Center
    • Verify all users: Active, MFA Enabled
    • Each member should sign in to approval portal once per week
  2. Backup Plan Review:
    • Check which VMs are being backed up
    • Look for new VMs needing tags
    • Remove test/temporary VMs from backup
  3. Security Review:
    • Review CloudTrail for unusual activity
    • Check for failed authentication attempts
    • Verify no unauthorized API calls

Monthly Tasks

  1. Abbreviated DR Drill:
    • Test approval team availability (15 minutes)
    • Request vault access (practice workflow)
    • Don’t do full restore (save time)
    • Document response times
  2. Cost Review:
    • Review AWS Backup costs in Cost Explorer
    • Compare to budget
    • Identify optimization opportunities
  3. Documentation Update:
    • Review runbooks for accuracy
    • Update contact information
    • Verify all ARNs still valid

Quarterly Tasks

  1. Full DR Drill:
    • Complete end-to-end simulation (like testing section)
    • Involve all stakeholders
    • Measure RTO/RPO
    • Document lessons learned
  2. Security Audit:
    • Review all IAM policies
    • Audit CloudTrail logs (full quarter)
    • Test MFA enforcement
    • Verify encryption keys
  3. Capacity Planning:
    • Review storage growth trends
    • Project next quarter needs
    • Evaluate gateway performance

Emergency Procedures

If Backup Job Fails:

  1. Check backup gateway status
  2. Verify VMware connectivity
  3. Review error message details
  4. Retry failed backup
  5. Escalate if not resolved in 30 minutes

If Real Ransomware Attack:

  1. Activate incident response team
  2. Contact approval team via personal phones
  3. Request vault access from clean account
  4. Follow complete DR procedures
  5. Update stakeholders every 30 minutes

Troubleshooting

Gateway Issues

Problem: Gateway Disconnected

Solutions:

  • Check network connectivity
  • Verify firewall rules
  • Test DNS resolution
  • Reboot gateway VM
  • Wait 5 minutes for reconnection

Problem: Backup Jobs Failing

Common causes:

  • VM powered off during backup
  • Insufficient storage space
  • Network timeout
  • VMware credentials expired

Vault Issues

Problem: Cannot Access Vault

Expected behavior:

  • This is correct if trying from production account
  • Vault requires approval team authorization
  • This proves security is working

If legitimate access needed:

  • Request vault access properly
  • Get approval team approval
  • Never try to bypass

Approval Team Issues

Problem: Member Cannot Sign In

Solutions:

  • Reset password in IAM Identity Center
  • Re-register MFA device
  • Wait 15 minutes if account locked
  • Contact admin for assistance

Problem: Team Inactive

Solution:

  • Delete inactive team
  • Create new team
  • All members must accept within 24 hours

Conclusion

You have implemented enterprise-grade ransomware protection.

Your organization is now protected against:

  • Ransomware attacks targeting backups
  • Insider threats attempting backup deletion
  • Administrative account compromise
  • Organizational-level security breaches
  • Accidental deletion of critical backups

Remember:

The best time to implement ransomware protection was yesterday.
The second best time is today.

Don’t wait for an incident. Your backups are now truly safe.

This implementation guide is designed to be used by any organisation implementing AWS Backup for VMware with air-gapped vaults and external approval teams. All examples should be adapted to your specific environment, account IDs, and organisational requirements.

facebookShare on Facebook
TwitterTweet
FollowFollow us
PinterestSave
Tags:

Comments

No comments yet. Why don’t you start the discussion?

Leave a Reply

Your email address will not be published. Required fields are marked *