EntropyAssessment

Cryptographic random bit generators (RBGs), also known as random number generators (RNGs), require a noise source that produces digital outputs with some level of unpredictability, expressed as min-entropy. SP 800-90B provides a standardized means of estimating the quality of a source of entropy.

Installing

Prerequisites

You will need to have the bz2 and ‘divsufsort’ libraries installed, as well as a g++-compatible compiler that suports c++11. On debian/ubuntu like systems the following command sould be sufficient:

sudo apt install libbz2-dev libdivsufsort-dev build-essential

You should be using a virtual environment with at least python version 3.

Using pip

You can simply run:

pip install sp800-90b

Using setuptools

Don’t forget to check out the nist_impl submodule:

git submodule init
git submodule update

You will also need the setuptools and the pybind11 python package:

pip install setuptools pybind11

Now you can install the package using:

python setup.py build
python setup.py install

Manual compilation

With the same prerequisites as in the setuptools installation you can also run

make -f Manual.make

to manually compile the package.

Note: You will only be able to use it in the directory where the generated .so library is present.

Installing

Use the python command help(sp800_90b) for documentation.

Manual test

The following commands can be used for a crude test of the package:

(venv) hnj@prokrastinator:~/sp800_90b$ python
Python 3.6.9 (default, Oct  8 2020, 12:12:24)
Type 'copyright', 'credits' or 'license' for more information
IPython 7.16.1 -- An enhanced Interactive Python. Type '?' for help.

In [1]: import random

In [2]: r = bytes(random.getrandbits(8) for _ in range(1000000))

In [3]: n = bytes(random.getrandbits(8) for _ in range(500000))

In [4]: n = n + bytes(random.getrandbits(7) for _ in range(500000))

In [5]: import sp800_90b

In [6]: help(sp800_90b)


In [7]: rd = sp800_90b.Data(r)

In [8]: rd.iid_tests()
Beginning initial tests...
Beginning permutation tests... these may take some time
Out[8]: True

In [9]: rd.h_initial()
Out[9]: 7.189185202883665

In [10]: nd = sp800_90b.Data(n)

In [11]: nd.iid_tests()
igamc: UNDERFLOW
Out[11]: False

In [12]: nd.h_initial()
Out[12]: 5.194695581531671

Disclaimer

This software is a repackaged and slightly modified version of a NIST-developed project (that can be found here: https://github.com/usnistgov/SP800-90B_EntropyAssessment).

Original NIST Disclaimer

NIST-developed software is provided by NIST as a public service. You may use, copy and distribute copies of the software in any medium, provided that you keep intact this entire notice. You may improve, modify and create derivative works of the software or any portion of the software, and you may distribute such modifications or works. Modified works should carry a notice stating that you changed the software and should note the date and nature of any such change. Please explicitly acknowledge the National Institute of Standards and Technology as the source of the software.

Modifications

The modifications that were made were are:

  • Adding a header file and some helper functions to wrap the provided nist code inside an object file that can be linked to other components (``src/nist.hpp` <src/nist.hpp>`_ ans ``src/nist.cpp` <src/nist.cpp>`_).

  • Writing a class that provides the functionality of the nist code as a class interface (``src/data.hpp` <src/data.hpp>`_ ans ``src/data.cpp` <src/data.cpp>`_).

  • Adding pybind11 bindings and boilderplate code to create a python package (``src/bindings.cpp` <src/bindings.cpp>`_, ``setup.py` <setup.py>`_ and ``MANIFEST.in` <MANIFEST.in>`_).

More Information

For more information on the nist code that is running, look into the original repository.

For more information on the estimation methods, see SP 800-90B.

View the code here: https://github.com/hnj2/sp800_90b

API

SP 800-90B entropy assesment, using the nist-provided implementation.

class sp800_90b.Data

Samples that can be assessed for entropy.

chi_square_tests(self: sp800_90b.Data) → bool

Tests if the data is independently and identically distributed using Pearson’s chi-squared test.

h_bitstring_collision(self: sp800_90b.Data) → float

Estimates the entropy of the data as a bistring with a collision test (Section 6.3.2).

h_bitstring_compression(self: sp800_90b.Data) → float

Estimates the entropy of the data as a bistring with a compression test (Section 6.3.4).

h_bitstring_lag_prediction(self: sp800_90b.Data) → float

Estimates the entrop of the data as a bistringy with a lag prediction test (Section 6.3.8).

h_bitstring_lz78y(self: sp800_90b.Data) → float

Estimates the entropy of the data as a bistring with a LZ78Y test (Section 6.3.10).

h_bitstring_markov(self: sp800_90b.Data) → float

Estimates the entropy of the data as a bistring with a markov test (Section 6.3.3).

h_bitstring_max(self: sp800_90b.Data) → float

The maximal entropy for the bitstring data (is 1.0).

h_bitstring_min_all(self: sp800_90b.Data) → float

Computes the minimum of the entropy estimates for the data in bitstring form.

h_bitstring_most_common(self: sp800_90b.Data) → float

Estimates the entropy of the data as a bistring with the most common value (Section 3.6.1).

h_bitstring_multi_markov(self: sp800_90b.Data) → float

Estimates the entropy of the data as a bistring with a multi markov model with counting test (Section 6.3.9).

h_bitstring_multi_most_common(self: sp800_90b.Data) → float

Estimates the entropy of the data as a bistring with a multi most common in window test (Section 6.3.7).

h_bitstring_t_tuple_and_lrs(self: sp800_90b.Data) → Tuple[float, float]

Estimates the entropy of the data as a bistring with a t-tuple and a lrs test (Sections 6.3.5 and 6.3.6).

h_both(self: sp800_90b.Data) → Tuple[float, float]

Computes both entropy estimates.

h_collision(self: sp800_90b.Data) → float

Estimates the entropy with a collision test (Section 6.3.2).

h_compression(self: sp800_90b.Data) → float

Estimates the entropy with a compression test (Section 6.3.4).

h_conditioned(self: sp800_90b.Data) → float

Computes the entropy estimate foa a conditioned seqauential dataset (Section 3.1.5.2).

h_initial(self: sp800_90b.Data) → float

Computes the initial entropy estimate (Section 3.1.3).

h_lag_prediction(self: sp800_90b.Data) → float

Estimates the entropy with a lag prediction test (Section 6.3.8).

h_lz78y(self: sp800_90b.Data) → float

Estimates the entropy with a LZ78Y test (Section 6.3.10).

h_markov(self: sp800_90b.Data) → float

Estimates the entropy with a markov test (Section 6.3.3).

h_max(self: sp800_90b.Data) → float

The maximal entropy for the dataset (word_size).

h_min_all(self: sp800_90b.Data) → float

Computes the minimum of the entropy estimates for the original data.

h_most_common(self: sp800_90b.Data) → float

Estimates the entropy with the most common value (Section 3.6.1).

h_multi_markov(self: sp800_90b.Data) → float

Estimates the entropy with a multi markov model with counting test (Section 6.3.9).

h_multi_most_common(self: sp800_90b.Data) → float

Estimates the entropy with a multi most common in window test (Section 6.3.7).

h_t_tuple_and_lrs(self: sp800_90b.Data) → Tuple[float, float]

Estimates the entropy with a t-tuple and a lrs test (Sections 6.3.5 and 6.3.6).

iid_tests(self: sp800_90b.Data) → bool

Tests if the data is independently and identically distributed using chi_square_test, lrs_tests and permutation_tests.

property is_binary

Whether the data only consisty of two different symbols.

lrs_tests(self: sp800_90b.Data) → bool

Tests if the data is independently and identically distributed using an LSR test.

property median

Statistical median value of the data.

permutation_tests(self: sp800_90b.Data) → bool

Tests if the data is independently and identically distributed using permutations.

property rawmean

Statistical mean value of the data.