Researchers Offer Falcon, a Privacy-Preserving Communication Protocol for AI Training and Inference

In an academic article published this week on the Arxiv.org preprint server, a team of researchers from Princeton, Microsoft, the nonprofit Algorand Foundation and Technion are proposing Falcon, an end-to-end framework for the secure computation of digital models. ‘AI on distributed systems. . They claim it is the first secure C ++ framework to support high-capacity AI models and batch normalization, a technique that can improve both model speed and stability. Additionally, they say Falcon automatically gives up when it detects the presence of malicious attackers and can outperform existing solutions by up to a factor of 200.

Claims are high, but if there is any truth to it, Falcon could be a step towards a suitable pipeline for areas where privacy and security are table stakes, like healthcare. Despite the emergence of techniques such as federated learning and homomorphic encryption, running machine learning models in such a way as to preserve confidentiality without compute compromise remains an unresolved challenge.

In essence, Falcon assumes that there are two types of users in a distributed AI use scenario: data owners, who own the training datasets, and query users, who query the system. after learning. A machine learning model of interest is trained on the data of the data holders and then queried by the query users, so that the data holders share their data securely between the servers (which use the shared data). and train the model safely). Query users can submit queries to the system and receive responses based on the newly trained models, and in this way, inputs from data holders are confidential to computer servers and queries are kept secret.

Falcon exploits new protocols for the computation of nonlinear functions, such as rectified linear units (ReLU), a type of activation function. AI models contain neurons (mathematical functions) arranged in layers that transmit the signals of the input data and adjust the strength (weight) of each connection. This is how they extract characteristics and learn to make predictions; the activation function of a node defines the output of the node according to the inputs, taking into account weights and sources of error.

Falcon also uses semi-honest protocols, where parties must follow exactly pre-defined rules and cannot change their inputs or outputs, and malicious protocols, where corrupt parties can break the rules by changing inputs and outputs or ignoring said rules. In addition, it incorporates existing techniques to operate on smaller data types, reducing the complexity of communication up to 2 times.

To assess Falcon’s performance, the team ran it on six different models, ranging from 3-layer networks with 118,000 parameters (configuration variables internal to the models needed to make predictions) to 16-layer networks with 138,000 parameters. million parameters, all have been trained on the MINST, CIFAR-10 and Tiny ImageNet corpus as needed depending on the size of the networks. They tested a WAN with servers in different geographies and a LAN, and in both cases they relied on Amazon Elastic Cloud Compute instances for the compute (with 16-core Intel processors and 64 GB of RAM. ).

According to the co-authors, Falcon was orders of magnitude faster in terms of inference, reaching speeds 32 times, 16 times, and 8 times faster compared to the Gazelle, XONN, and SecureNN baselines, respectively. In private training, it was 4.4 times faster than ABY and 6 times faster than SecureNN.

The researchers further claim that Falcon – which contains approximately 12,300 lines of code – does not incur much of a performance penalty compared to running the model in plain, insecure text. In a test using a single epoch (i.e. a complete pass through training data) for the AlexNet Image Classification Model, Falcon took 2300 seconds on a processor versus 570 seconds for the plain text, i.e. a difference factor 6.

The team attributes a large portion of Falcon’s performance gains to its support for batch normalization, which they say speeds up model training by enabling higher learning rates; prevents extreme values ​​of activations; and reduces overfitting (a phenomenon that occurs when a model over-learns a data set) by providing a regularization effect that improves training stability.

“[T]a sensitive nature of [certain] data requires deep learning frameworks that allow training on aggregated data from multiple entities while ensuring strong guarantees of confidentiality and confidentiality, ”the researchers conclude. “A synergistic combination of secure computation primitives with deep learning algorithms would allow sensitive applications to benefit from the high predictive accuracy of neural networks. “