Skip to content

quantize

AdaptiveQuantizer (DecimalQuantizer)

The quantizer that implements the algorithm 2 of the MDPI paper.

optimize(self, x, bits, weight=None, channel_index=-1, batched=False, **kwargs)

return the updated weight for each step

Source code in qsparse/quantize.py
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
def optimize(self, x, bits, weight=None, channel_index=-1, batched=False, **kwargs):
    batch_size = x.shape[0]
    with torch.no_grad():
        if channel_index >= 0:
            if batched:
                if channel_index != 1:
                    x = x.transpose(1, channel_index)
                shape = tuple(x.shape)
                x = x.view(-1, math.prod(shape[2:]))
            else:
                if channel_index != 0:
                    x = x.transpose(0, channel_index)
                shape = tuple(x.shape)
                x = x.contiguous().view(-1, math.prod(shape[1:]))
        else:
            x = x.view(len(x) if batched else 1, -1)

        lb = x.min(dim=1).values
        ub = x.max(dim=1).values
        _buf = torch.cat([lb.view(-1, 1), ub.view(-1, 1)], dim=1)
        _lines = _buf[:len(lb), :]

        if batched:
            _lines = _lines.view(batch_size, -1, 2)
            sample_avg_lines = torch.cat([_lines[:, :, 0].min(
                dim=0).values.view(-1, 1), _lines[:, :, 1].max(dim=0).values.view(-1, 1)], dim=1).view(-1, 2)
        else:
            sample_avg_lines = _lines

        if weight is None:
            self.t = nn.Parameter(torch.zeros(1).to(
                x.device), requires_grad=False)
            self.t += 1
            return sample_avg_lines
        else:
            assert sample_avg_lines.shape == weight.shape
            self.t += 1
            return (weight * (self.t - 1) + sample_avg_lines) / self.t

BaseQuantizer (Module)

Base class for quantizer, interface for the callback function of quantize.

forward(self, tensor, bits, weight=None, batched=False, channel_index=-1)

return quantized tensor

Source code in qsparse/quantize.py
267
268
269
def forward(self, tensor, bits, weight=None, batched=False, channel_index=-1) -> torch.Tensor:
    """return quantized tensor"""
    raise NotImplementedError

optimize(self, tensor, bits, weight=None, batched=False, channel_index=-1)

return the updated weight for each step

Source code in qsparse/quantize.py
263
264
265
def optimize(self, tensor, bits, weight=None, batched=False, channel_index=-1) -> torch.Tensor:
    """return the updated weight for each step"""
    raise NotImplementedError

DecimalQuantization (Function)

Straight-Through Gradient Estimator (with shift).

Please look for detailed description on arguments in quantize_with_decimal.

backward(ctx, grad_output) staticmethod

gradient computation for quantization operation.

Source code in qsparse/quantize.py
65
66
67
68
69
70
71
72
73
74
75
76
77
@staticmethod
def backward(ctx, grad_output):
    """gradient computation for quantization operation."""
    limit, tof = ctx.saved_tensors
    if ctx.backward_passthrough:
        v = grad_output
    else:
        v = grad_output.clamp_(
            (-limit + ctx.notch) * tof,
            (limit - 1 + ctx.notch) * tof,
        )
        v[v != grad_output] = 0  # reset the clampped values to 0
    return (v,) + (None,) * 6

forward(ctx, input, bits=8, decimal=5, channel_index=1, use_uint=False, backward_passthrough=False, flip_axis=False) staticmethod

quantize the input tensor and prepare for backward computation.

Source code in qsparse/quantize.py
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
@staticmethod
def forward(
    ctx,
    input: torch.Tensor,
    bits: int = 8,
    decimal: TensorOrInt = 5,
    channel_index: int = 1,
    use_uint: bool = False,
    backward_passthrough: bool = False,
    flip_axis: bool = False,
):
    """quantize the input tensor and prepare for backward computation."""
    ctx.backward_passthrough = backward_passthrough
    ctx.notch = 1 if flip_axis else 0
    limit = 2.0 ** (bits - 1)
    tof = 2.0**-decimal
    toi = 2.0**decimal
    shape = [1 for _ in input.shape]
    if isinstance(decimal, torch.Tensor) and decimal.numel() > 1:
        assert (
            len(decimal) == input.shape[channel_index]
        ), "channel of input and decimal must be equal in channel-wise quantization"
        shape[channel_index] = -1
        tof, toi = tof.view(*shape), toi.view(*shape)
    ctx.save_for_backward(ensure_tensor(limit), ensure_tensor(tof))
    q = (input * toi).int()
    if use_uint:
        q.float().clamp_(0, 2 * limit - 1)
    else:
        q.float().clamp_(
            -limit + ctx.notch,
            limit - 1 + ctx.notch,
        )
    return q.float() * tof

DecimalQuantizer (BaseQuantizer)

The quantizer that implements the algorithm 3 of the MDPI paper. The forward function covers the quantization logic and the optimize function covers the parameter update.

It always restricts the scaling factor to be power of 2.

__init__(self, use_uint=False, backward_passthrough=False, flip_axis=False, group_num=-1, group_timeout=512) special

Parameters:

Name Type Description Default
use_uint bool

See quantize_with_decimal. Defaults to False.

False
backward_passthrough bool

See quantize_with_decimal. Defaults to False.

False
flip_axis bool

See quantize_with_decimal. Defaults to False.

False
group_num int

Number of groups used for groupwise quantization. Defaults to -1, which disables groupwise quantization.

-1
group_timeout int

Number of steps when the clustering starts after the activation of the quantization operator. Defaults to 512.

512
Source code in qsparse/quantize.py
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
def __init__(
    self,
    use_uint: bool = False,
    backward_passthrough: bool = False,
    flip_axis: bool = False,
    group_num=-1,
    group_timeout=512
):
    """
    Args:
        use_uint (bool, optional): See [quantize\_with\_decimal][qsparse.quantize.quantize_with_decimal]. Defaults to False.
        backward_passthrough (bool, optional): See [quantize\_with\_decimal][qsparse.quantize.quantize_with_decimal]. Defaults to False.
        flip_axis (bool, optional): See [quantize\_with\_decimal][qsparse.quantize.quantize_with_decimal]. Defaults to False.
        group_num (int, optional): Number of groups used for groupwise quantization. Defaults to -1, which disables groupwise quantization.
        group_timeout (int, optional): Number of steps when the clustering starts after the activation of the quantization operator. Defaults to 512.
    """
    super().__init__()
    self.use_uint = use_uint
    self.backward_passthrough = backward_passthrough
    self.flip_axis = flip_axis
    self.use_float_scaler = False
    self.function = DecimalQuantization.apply
    self.t = 0
    self.group_timeout = group_timeout
    self.groups = None
    self.group_num = group_num

forward(self, tensor, bits, scaler, channel_index=-1, **kwargs)

return quantized tensor

Source code in qsparse/quantize.py
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
def forward(self, tensor, bits, scaler, channel_index=-1, **kwargs):
    if self.t >= self.group_timeout and self.group_num > 0 and scaler.numel() > self.group_num:
        if self.groups is None:
            logging.danger( f"clustering {len(scaler)} channels into {self.group_num} groups")
            clustering = AgglomerativeClustering(
                n_clusters=self.group_num)
            clustering.fit(scaler.detach().cpu().numpy())
            self.groups = nn.Parameter(torch.from_numpy(
                clustering.labels_).to(scaler.device), requires_grad=False)

        group_scaler = torch.clone(scaler)
        for ci in range(self.group_num):
            ind = self.groups == ci
            avg = group_scaler[ind].mean(dim=0)
            group_scaler[ind] = avg
        scaler = group_scaler
    return self.quantize(tensor, bits, scaler, channel_index, **kwargs)

optimize(self, x, bits, weight=None, batched=False, channel_index=-1, **kwargs)

return the updated weight for each step

Source code in qsparse/quantize.py
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
def optimize(self, x, bits, weight=None, batched=False, channel_index=-1, **kwargs):
    with torch.no_grad():
        x = x.abs()
        batch_size = x.shape[0] if batched else -1
        wshape = self.get_weight_shape(x, channel_index)
        if channel_index == -1:
            x = x.view(1, -1)
        elif channel_index != 0:
            num_channel = x.shape[channel_index]
            x = x.transpose(0, channel_index)
            x = x.contiguous().view(num_channel, -1)
        else:
            x = x.view(x.shape[0], -1)
        new_weight = x.max(dim=1).values / (2 ** (bits - 1))
        if batched and channel_index >= 0:
            new_weight = new_weight.view(-1, batch_size).mean(dim=1)
        new_weight = new_weight.view(wshape)
    if self.t == 0:
        weight = new_weight
    else:
        weight.data[:] = (self.t * weight + new_weight) / (self.t + 1)
    self.t += 1
    return weight

LineQuantization (Function)

Straight-Through Gradient Estimator (asymmetric).

Please look for detailed description on arguments in quantize_with_line.

backward(ctx, grad_output) staticmethod

Defines a formula for differentiating the operation with backward mode automatic differentiation (alias to the vjp function).

This function is to be overridden by all subclasses.

It must accept a context :attr:ctx as the first argument, followed by as many outputs as the :func:forward returned (None will be passed in for non tensor outputs of the forward function), and it should return as many tensors, as there were inputs to :func:forward. Each argument is the gradient w.r.t the given output, and each returned value should be the gradient w.r.t. the corresponding input. If an input is not a Tensor or is a Tensor not requiring grads, you can just pass None as a gradient for that input.

The context can be used to retrieve tensors saved during the forward pass. It also has an attribute :attr:ctx.needs_input_grad as a tuple of booleans representing whether each input needs gradient. E.g., :func:backward will have ctx.needs_input_grad[0] = True if the first input to :func:forward needs gradient computated w.r.t. the output.

Source code in qsparse/quantize.py
183
184
185
@staticmethod
def backward(ctx, grad_output):
    return (grad_output,) + (None,) * 5

forward(ctx, x, bits=8, lines=(-0.1, 0.9), channel_index=-1, inplace=False, float_zero_point=True) staticmethod

Performs the operation.

This function is to be overridden by all subclasses.

It must accept a context ctx as the first argument, followed by any number of arguments (tensors or other types).

The context can be used to store arbitrary data that can be then retrieved during the backward pass. Tensors should not be stored directly on ctx (though this is not currently enforced for backward compatibility). Instead, tensors should be saved either with :func:ctx.save_for_backward if they are intended to be used in backward (equivalently, vjp) or :func:ctx.save_for_forward if they are intended to be used for in jvp.

Source code in qsparse/quantize.py
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
@staticmethod
def forward(ctx, 
            x: torch.Tensor, 
            bits: int = 8, 
            lines=(-0.1, 0.9), 
            channel_index=-1, 
            inplace=False, 
            float_zero_point=True):
    with torch.no_grad():
        N = 2**bits
        shape = [1] * len(x.shape)
        if not isinstance(lines, torch.Tensor):
            lines = torch.tensor(lines).view(-1, 2).to(x.device)
        if channel_index >= 0:
            shape[channel_index] = -1
            assert x.shape[channel_index] == lines.shape[0]
        assert lines.shape[1] == 2
        start, end = lines[:, 0].view(shape), lines[:, 1].view(shape)
        x = torch.clamp(x, start, end)
        step = (end - start) / N
        step[step == 0] = 0.0001
        if not float_zero_point:
            qa = (x / step).round()
            qstart = (start / step).round()
            qa = (qa - qstart).clamp(0, N-1)
            qa = (qa + qstart) * step
            return qa
        else:
            if inplace:
                x = x - start
                x /= step
                x = x.round_().clamp_(0, N - 1)
                x = x * step
                x += start
                return x
            else:
                qa = x - start
                qa /= step
                qa = qa.round_().clamp_(0, N - 1)
                qa = qa * step
                qa += start
                return qa

QuantizeLayer (Module)

Applies quantization over input tensor.

Please look for detailed description in quantize

initted: bool property readonly

whether the parameters of the quantize layer are initialized.

forward(self, x)

Quantize input tensor according to given configuration.

Parameters:

Name Type Description Default
x torch.Tensor

tensor to be quantized

required

Returns:

Type Description
torch.Tensor

quantized tensor

Source code in qsparse/quantize.py
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
def forward(self, x):
    """Quantize input tensor according to given configuration.

    Args:
        x (torch.Tensor): tensor to be quantized

    Returns:
        torch.Tensor: quantized tensor
    """
    if not self.initted:
        self.weight = nn.Parameter(
            torch.zeros(
                1 if self.channelwise < 0 else x.shape[self.channelwise],
                self.callback.weight_size,
            ).to(x.device),
            requires_grad=False,
        )
        self._n_updates = nn.Parameter(
            torch.zeros(1, dtype=torch.int).to(x.device),
            requires_grad=False,
        )

    t = self._n_updates.item()
    if self.timeout > 0:
        if t >= self.timeout:
            if self.training:
                if t == self.timeout:
                    logging.warn(f"quantizing {self.name} with {self.bits} bits")
                new_weight = self.callback.optimize(x, self.bits, self.weight, 
                                                    batched=self.batch_dimension == 0, channel_index=self.channelwise)
                if new_weight is not None:
                    self.weight.data[:] = new_weight
                self._quantized = True
            if self._quantized:
                out = self.callback(
                    x, self.bits, self.weight, channel_index=self.channelwise, inplace=self.batch_dimension == 0)
            else:
                out = x
        else:
            out = x

        if self.training:
            self._n_updates += 1
    else:
        out = x
    return out

ScalerQuantization (Function)

Straight-Through Gradient Estimator (with scaler).

Please look for detailed description on arguments in quantize_with_scaler.

backward(ctx, grad_output) staticmethod

gradient computation for quantization operation.

Source code in qsparse/quantize.py
119
120
121
122
123
124
125
126
127
128
129
130
131
@staticmethod
def backward(ctx, grad_output):
    """gradient computation for quantization operation."""
    limit, scaler = ctx.saved_tensors
    if ctx.backward_passthrough:
        v = grad_output
    else:
        v = grad_output.clamp_(
            (-limit + ctx.notch) * scaler,
            (limit - 1 + ctx.notch) * scaler,
        )
        v[v != grad_output] = 0  # reset the clampped values to 0
    return (v,) + (None,) * 6

forward(ctx, input, bits=8, scaler=0.1, channel_index=1, use_uint=False, backward_passthrough=False, flip_axis=False) staticmethod

quantize the input tensor and prepare for backward computation.

Source code in qsparse/quantize.py
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
@staticmethod
def forward(
    ctx,
    input: torch.Tensor,
    bits: int = 8,
    scaler: TensorOrFloat = 0.1,
    channel_index: int = 1,
    use_uint: bool = False,
    backward_passthrough: bool = False,
    flip_axis: bool = False,
):
    """quantize the input tensor and prepare for backward computation."""
    ctx.backward_passthrough = backward_passthrough
    ctx.notch = 1 if flip_axis else 0
    limit = 2.0 ** (bits - 1)
    shape = [1 for _ in input.shape]
    if isinstance(scaler, torch.Tensor) and math.prod(scaler.shape) > 1:
        assert (
            len(scaler) == input.shape[channel_index]
        ), "channel of input and decimal must be equal in channel-wise quantization"
        shape[channel_index] = -1
        scaler = scaler.view(*shape)
    ctx.save_for_backward(ensure_tensor(limit), ensure_tensor(scaler))
    q = (input / scaler).round().int()
    if use_uint:
        q.float().clamp_(0, 2 * limit - 1)
    else:
        q.float().clamp_(
            -limit + ctx.notch,
            limit - 1 + ctx.notch,
        )
    return q.float() * scaler

ScalerQuantizer (DecimalQuantizer)

The quantizer that implements the algorithm 3 of the MDPI paper, without the power of 2 restriction.

quantize(inp=None, bits=8, channelwise=1, timeout=1000, callback=None, bias_bits=-1, name='')

Creates a QuantizeLayer which is usually used for feature quantization if no input module is provided, or creates a weight-quantized version of the input module.

Parameters:

Name Type Description Default
inp nn.Module

input module whose weight is to be quantized. Defaults to None.

None
bits int

bitwidth for weight. Defaults to 8.

8
channelwise int

dimension index for channel. Defaults to 1. When channelwise >= 0, channel-wise quantization is enabled. When set to -1, channel-wise quantization is disabled.

1
timeout int

the steps to compute the best decimal bits. Defaults to 1000.

1000
callback BaseQuantizer

callback module for actual operation of quantizing tensor and finding quantization parameters. Defaults to ScalerQuantizer.

None
bias_bits int

bitwidth for bias. Defaults to -1, means not quantizing bias.

-1
name str

name of the quantize layer created, used for better logging. Defaults to "".

''

Returns:

Type Description
nn.Module

input module with its weight quantized or a instance of QuantizeLayer for feature quantization

Source code in qsparse/quantize.py
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
def quantize(
    inp: nn.Module = None,
    bits: int = 8,
    channelwise: int = 1,
    timeout: int = 1000,
    callback: BaseQuantizer = None,
    # for bias quantization, default to -1 is to not quantize bias
    bias_bits: int = -1,
    # for debug purpose
    name: str = "",
) -> nn.Module:
    """Creates a [QuantizeLayer][qsparse.quantize.QuantizeLayer] which is
    usually used for feature quantization if no input module is provided, or
    creates a weight-quantized version of the input module.

    Args:
        inp (nn.Module, optional): input module whose weight is to be quantized. Defaults to None.
        bits (int, optional): bitwidth for weight. Defaults to 8.
        channelwise (int, optional): dimension index for channel. Defaults to 1. When channelwise >= 0, channel-wise quantization is enabled. When set to -1, channel-wise quantization is disabled.
        timeout (int, optional): the steps to compute the best decimal bits. Defaults to 1000.
        callback (BaseQuantizer, optional):  callback module for actual operation of quantizing tensor and finding quantization parameters. Defaults to [ScalerQuantizer][qsparse.quantize.ScalerQuantizer].
        bias_bits (int, optional): bitwidth for bias. Defaults to -1, means not quantizing bias.
        name (str, optional): name of the quantize layer created, used for better logging. Defaults to "".

    Returns:
        nn.Module: input module with its weight quantized or a instance of [QuantizeLayer][qsparse.quantize.QuantizeLayer] for feature quantization
    """
    callback = callback or ScalerQuantizer()

    kwargs = dict(
        bits=bits,
        channelwise=channelwise,
        timeout=timeout,
        callback=callback,
        bias_bits=bias_bits,
        name=name
    )

    def get_quantize_layer(batch_dimension=0, is_bias=False):
        if bias_bits == -1 and is_bias:
            return lambda a: a
        else:
            return QuantizeLayer(
                bits=bias_bits if is_bias else bits,
                channelwise=(0 if channelwise >= 0 else
                             - 1) if is_bias else channelwise,
                timeout=int(timeout),
                callback=callback,
                name=name,
                batch_dimension=batch_dimension
            )

    if inp is None:
        layer = get_quantize_layer()
        setattr(layer, "_kwargs", kwargs)
        return layer
    elif isinstance(inp, nn.Module):
        return imitate(
            inp,
            "quantize",
            get_quantize_layer(-1),
            get_quantize_layer(-1, is_bias=True),
        )
    else:
        raise ValueError(f"{inp} is not a valid argument for quantize")

quantize_with_decimal(input, bits=8, decimal=5, channel_index=-1, use_uint=False, backward_passthrough=False, flip_axis=False)

Applying power-of-2 uniform quantization over input tensor

Parameters:

Name Type Description Default
input torch.Tensor

tensor to be quantized

required
bits int

Bitwidth. Defaults to 8.

8
decimal TensorOrInt

Number of bits used to represent fractional number (shift). Defaults to 5.

5
channel_index int

Channel axis, for channelwise quantization. Defaults to -1, which means tensorwise.

-1
use_uint bool

Whether use uint to quantize input. If so, it will ignores the negative number, which could be used for ReLU output. Defaults to False.

False
backward_passthrough bool

Whether to skip the saturation operation of STE on gradients during the backward pass. Defaults to False.

False
flip_axis bool

Whether use flip the axis to represent numbers (the largest positive number increases from 2^{d}-1 to 2^{d}, while the smallest negative number reduces its absolute value). Defaults to False.

False

Returns:

Type Description
torch.Tensor

quantized tensor

Source code in qsparse/quantize.py
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
def quantize_with_decimal(
        input: torch.Tensor,
        bits: int = 8,
        decimal: TensorOrInt = 5,
        channel_index: int = -1,
        use_uint: bool = False,
        backward_passthrough: bool = False,
        flip_axis: bool = False) -> torch.Tensor:
    """Applying power-of-2 uniform quantization over input tensor

    Args:
        input (torch.Tensor): tensor to be quantized
        bits (int, optional): Bitwidth. Defaults to 8.
        decimal (TensorOrInt, optional): Number of bits used to represent fractional number (shift). Defaults to 5.
        channel_index (int, optional): Channel axis, for channelwise quantization. Defaults to -1, which means tensorwise.
        use_uint (bool, optional): Whether use uint to quantize input. If so, it will ignores the negative number, which could be used for ReLU output. Defaults to False.
        backward_passthrough (bool, optional): Whether to skip the saturation operation of STE on gradients during the backward pass. Defaults to False.
        flip_axis (bool, optional): Whether use flip the axis to represent numbers (the largest positive number increases from `2^{d}-1` to `2^{d}`, while the smallest negative number reduces its absolute value). Defaults to False.

    Returns:
        torch.Tensor: quantized tensor
    """
    return DecimalQuantization.apply(input, bits, decimal, channel_index, use_uint, backward_passthrough, flip_axis)

quantize_with_line(x, bits=8, lines=(-0.1, 0.9), channel_index=-1, inplace=False, float_zero_point=True)

Applying asymmetric uniform quantization over input tensor

Parameters:

Name Type Description Default
x torch.Tensor

tensor to be quantized

required
bits int

Bitwidth. Defaults to 8.

8
lines Union[Tuple[float, float], List[Tuple[float, float]]]

The estimated lower and upper bound of input data. Defaults to (-0.1, 0.9).

(-0.1, 0.9)
channel_index int

Channel axis, for channelwise quantization. Defaults to -1, which means tensorwise.

-1
inplace bool

Whether the operation is inplace. Defaults to False.

False
float_zero_point bool

Whether use floating-point value to store zero-point. Defaults to True, recommend to turn on for training and off for evaluation.

True

Returns:

Type Description
torch.Tensor

quantized tensor

Source code in qsparse/quantize.py
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
def quantize_with_line(x: torch.Tensor, 
                bits: int = 8, 
                lines: Union[Tuple[float, float], List[Tuple[float, float]]]=(-0.1, 0.9), 
                channel_index: int=-1, 
                inplace: bool=False, 
                float_zero_point: bool=True) -> torch.Tensor:
    """Applying asymmetric uniform quantization over input tensor

    Args:
        x (torch.Tensor): tensor to be quantized
        bits (int, optional): Bitwidth. Defaults to 8.
        lines (Union[Tuple[float, float], List[Tuple[float, float]]], optional): The estimated lower and upper bound of input data. Defaults to (-0.1, 0.9).
        channel_index (int, optional): Channel axis, for channelwise quantization. Defaults to -1, which means tensorwise.
        inplace (bool, optional): Whether the operation is inplace. Defaults to False.
        float_zero_point (bool, optional): Whether use floating-point value to store zero-point. Defaults to True, recommend to turn on for training and off for evaluation. 

    Returns:
        torch.Tensor: quantized tensor
    """
    return LineQuantization.apply(x, bits, lines, channel_index, inplace, float_zero_point)

quantize_with_scaler(input, bits=8, scaler=0.1, channel_index=-1, use_uint=False, backward_passthrough=False, flip_axis=False)

Applying scaling-factor based uniform quantization over input tensor

Parameters:

Name Type Description Default
input torch.Tensor

tensor to be quantized

required
bits int

Bitwidth. Defaults to 8.

8
scaler TensorOrFloat

Scaling factor. Defaults to 0.1.

0.1
channel_index int

Channel axis, for channelwise quantization. Defaults to -1, which means tensorwise.

-1
use_uint bool

Whether use uint to quantize input. If so, it will ignores the negative number, which could be used for ReLU output. Defaults to False.

False
backward_passthrough bool

Whether to skip the saturation operation of STE on gradients during the backward pass. Defaults to False.

False
flip_axis bool

Whether use flip the axis to represent numbers (the largest positive number increases from 2^{d}-1 to 2^{d}, while the smallest negative number reduces its absolute value). Defaults to False.

False

Returns:

Type Description
torch.Tensor

quantized tensor

Source code in qsparse/quantize.py
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
def quantize_with_scaler(
        input: torch.Tensor,
        bits: int = 8,
        scaler: TensorOrFloat = 0.1,
        channel_index: int = -1,
        use_uint: bool = False,
        backward_passthrough: bool = False,
        flip_axis: bool = False) -> torch.Tensor:
    """Applying scaling-factor based uniform quantization over input tensor

    Args:
        input (torch.Tensor): tensor to be quantized
        bits (int, optional): Bitwidth. Defaults to 8.
        scaler (TensorOrFloat, optional): Scaling factor. Defaults to 0.1.
        channel_index (int, optional): Channel axis, for channelwise quantization. Defaults to -1, which means tensorwise.
        use_uint (bool, optional): Whether use uint to quantize input. If so, it will ignores the negative number, which could be used for ReLU output. Defaults to False.
        backward_passthrough (bool, optional): Whether to skip the saturation operation of STE on gradients during the backward pass. Defaults to False.
        flip_axis (bool, optional): Whether use flip the axis to represent numbers (the largest positive number increases from `2^{d}-1` to `2^{d}`, while the smallest negative number reduces its absolute value). Defaults to False.

    Returns:
        torch.Tensor: quantized tensor
    """
    return ScalerQuantization.apply(input, bits, scaler, channel_index, use_uint, backward_passthrough, flip_axis)