ML on Encrypted Data

Running machine learning models on encrypted inputs requires rethinking how we implement inference. Linear operations are straightforward, but non-linear activations require polynomial approximations or alternative approaches.

Encrypted Linear Operations

Linear operations are natural in HE because they only require additions and multiplications:

import tenseal as ts
import numpy as np

# Setup CKKS context
context = ts.context(
    ts.SCHEME_TYPE.CKKS,
    poly_modulus_degree=8192,
    coeff_mod_bit_sizes=[60, 40, 40, 60]
)
context.generate_galois_keys()
context.global_scale = 2**40

# Client encrypts input
plain_input = [1.0, 2.0, 3.0, 4.0]
encrypted_input = ts.ckks_vector(context, plain_input)

# Server applies linear layer (weights are plaintext)
weights = np.array([[0.5, -0.3],
                     [0.1, 0.8],
                     [-0.2, 0.4],
                     [0.6, -0.1]])
bias = [0.1, -0.2]

# Matrix-vector multiply on encrypted data
encrypted_output = encrypted_input.mm(weights) + bias

# Client decrypts result
result = encrypted_output.decrypt()
print(f"Decrypted output: {result}")

The Activation Function Challenge

Standard activation functions (ReLU, sigmoid, tanh) cannot be computed directly in HE because they are non-polynomial. Solutions include:

Polynomial Approximations

Replace non-polynomial activations with polynomial approximations:

• Square activation: f(x) = x² — simple, one multiplication, but changes model behavior.
• Polynomial ReLU: Approximate ReLU with a low-degree polynomial over the expected input range.
• Minimax polynomials: Optimal polynomial approximations that minimize the maximum error over a given interval.

TFHE Approach

The TFHE scheme supports programmable bootstrapping: arbitrary lookup tables can be evaluated during bootstrapping. This allows exact non-linear functions at the cost of a bootstrapping operation per activation.

Encrypted Logistic Regression

Logistic regression is one of the simplest models to run under HE because it only requires one matrix multiplication followed by a sigmoid approximation:

from concrete.ml.sklearn import LogisticRegression
from sklearn.datasets import make_classification

# Train on plaintext data
X, y = make_classification(n_samples=1000, n_features=10)
model = LogisticRegression(n_bits=8)
model.fit(X, y)

# Compile the model for FHE
fhe_circuit = model.compile(X)

# Generate encryption keys
fhe_circuit.client.keygen()

# Encrypt input, run inference, decrypt output
encrypted_input = fhe_circuit.client.encrypt(X[0:1])
encrypted_result = fhe_circuit.server.run(encrypted_input)
prediction = fhe_circuit.client.decrypt(encrypted_result)
print(f"Encrypted prediction: {prediction}")

Encrypted Neural Networks

Running neural networks under HE requires adapting the architecture:

• Replace ReLU: Use polynomial activations (x², degree-3 polynomials) or the square function.
• Limit depth: Fewer layers means less multiplicative depth. Wide, shallow networks often work better under HE.
• Batch normalization: Fold batch norm parameters into the preceding linear layer at inference time.
• Avoid max pooling: Use average pooling (linear operation) instead of max pooling (non-linear).