PrivAID: Supercharge Your Browser with Private AI On Device

Introduction

In an era where artificial intelligence is revolutionizing our digital experiences, AI-powered browsers have emerged as powerful tools for enhancing web navigation and productivity. These browsers offer features like instant summaries, contextual answers, and intelligent content generation, promising a more efficient and intuitive browsing experience. However, this convenience often comes at a significant cost: your privacy.

Most AI-enhanced browsing solutions rely heavily on cloud-based processing, which means your personal browsing data—including your history, preferences, and online behavior—is transmitted to and processed on remote servers. This practice raises serious concerns about data security and potential misuse. It fundamentally contradicts the concept of "private browsing" that many users have come to expect and rely upon.

For individuals, this means their personal digital footprint is potentially exposed to third parties. For businesses, the stakes are even higher. Sensitive corporate information, confidential research, and proprietary data could all be at risk when processed on external servers.

The dilemma is clear: How can we harness the power of AI to enhance our browsing experience without compromising the privacy and security of our data?

This is where PrivAID steps in, offering a revolutionary solution that brings together the best of both worlds—cutting-edge AI capabilities and unwavering commitment to user privacy. Our breakthrough is made possible by recent advancements in hardware technology, specifically the Neural Processing Units (NPUs) developed by AMD for their Ryzen processors. These NPUs provide the computational power necessary to run sophisticated AI models directly on your device, eliminating the need for cloud-based processing.

By leveraging AMD's innovative NPU technology, PrivAID is able to offer a unique solution: an AI-powered browsing experience that keeps your data entirely under your control. This marriage of AMD's hardware capabilities and PrivAID's privacy-focused software design opens up new possibilities for secure, intelligent browsing in an increasingly data-sensitive world.

The Technology Behind On-Device AI

To fully appreciate the innovation behind PrivAID, it's crucial to understand the key technologies that make on-device AI possible. This section explores the core components and concepts that power our privacy-focused browsing solution.

Neural Processing Units (NPUs)

At the heart of PrivAID's on-device AI capabilities are AMD's Neural Processing Units (NPUs) integrated into Ryzen processors. NPUs are specialized hardware designed to accelerate AI and machine learning workloads. Unlike general-purpose CPUs, NPUs are optimized for the specific computational patterns found in neural networks, allowing for faster and more energy-efficient AI processing.

AMD's Ryzen AI technology, featured in their latest processors, brings NPU capabilities to consumer devices. This enables complex AI tasks to be performed locally, without the need for cloud processing or powerful discrete GPUs.

For more details, please refer to https://ryzen-ai.com/en/,

Large Language Models (LLMs)

Large Language Models are at the core of PrivAID's AI capabilities. These models are based on the transformer architecture, a neural network design that has revolutionized natural language processing tasks.

In our project, we utilize Phi-3, a model developed by Microsoft. Phi-3 is a remarkable achievement in AI efficiency, having just 3.82 billion parameters but capabilities of models much bigger than that. Microsoft has marketed Phi-3 as a "Small Language Model" (SLM) due to its relatively compact size compared to other state-of-the-art models that often have hundreds of billions of parameters.

The use of Phi-3 in PrivAID offers several advantages:

• Efficiency: With its smaller parameter count, Phi-3 requires less computational resources, making it ideal for on-device processing.
• Performance: Despite its compact size, Phi-3 demonstrates impressive performance across a wide range of language tasks, including those critical for PrivAID's features.
• Privacy: By using a model that can run entirely on-device, we ensure that user data never leaves their system for processing.
• Responsiveness: The smaller size of Phi-3 allows for faster inference times, providing users with quick, real-time AI assistance.

Phi-3's architecture and training methodology allow it to achieve near-state-of-the-art performance on many benchmarks while maintaining a significantly smaller footprint. This makes it an excellent choice for applications like PrivAID, where we need to balance high-quality AI capabilities with the constraints of running on consumer hardware.

By leveraging Phi-3 in combination with AMD's NPU technology, PrivAID is able to offer sophisticated AI features directly in the browser, all while maintaining strict standards of data privacy and system performance.

Refer here for Phi-3 https://huggingface.co/microsoft/Phi-3-mini-4k-instruct, https://azure.microsoft.com/en-us/products/phi-3

Quantization and INT8 Precision

To run complex LLMs on device-level NPUs, we employ a technique called quantization. Quantization is the process of reducing the precision of the model's parameters and computations, typically from 32-bit floating-point (FP32) to 8-bit integer (INT8) representation.

INT8 quantization significantly reduces the model's memory footprint and computational requirements, with minimal impact on accuracy. This is crucial for running AI models on devices with limited resources. AMD's Ryzen AI is optimized for INT8 operations, making it an ideal platform for running quantized LLMs.

Ryzen AI Software Stack

A crucial component of PrivAID's ability to leverage on-device AI is the Ryzen AI software stack provided by AMD. This comprehensive suite of tools and technologies is specifically designed to optimize and run AI models on Ryzen processors equipped with Neural Processing Units (NPUs).

At the heart of this stack is the Vitis AI Execution Provider, a powerful tool that provides all the necessary components to quantize and run models on the NPU efficiently. Here's a breakdown of the key elements:

• Vitis AI Execution Provider: This enables the execution of ONNX (Open Neural Network Exchange) and Pytorch models on AMD's NPUs. It serves as a bridge between the high-level model representation and the low-level hardware instructions.
• Quantization Tools: Vitis AI includes advanced quantization tools that convert floating-point models to fixed-point (typically INT8) representations. This process is crucial for running complex models like Phi-3 on resource-constrained devices.
• Model Optimizer: The stack includes optimizers that can restructure and simplify neural networks to run more efficiently on NPU hardware, without sacrificing accuracy.
• Runtime Libraries: These libraries handle the low-level operations of deploying and executing the quantized models on the NPU, ensuring optimal performance.
• Profiling and Debugging Tools: The stack provides tools to analyze model performance, identify bottlenecks, and fine-tune execution for maximum efficiency.
• AMD NPU Kernel Driver: This driver enables direct communication between the software stack and the NPU hardware, allowing for efficient utilization of the AI acceleration capabilities.

The Ryzen AI software stack, with its Vitis AI Execution Provider, is instrumental in enabling PrivAID to run sophisticated AI models directly on users' devices. It handles the complex tasks of model quantization, optimization, and execution, allowing us to focus on developing the high-level features and user experience of our privacy-focused AI browser extension.

Refer to the Ryzen AI documentation for more info - https://ryzenai.docs.amd.com/en/latest/

Deployment Guide

Follow these steps to deploy PrivAID on your system:

Prerequisites

• Ensure you have Windows 11 installed on your system.
• Install Anaconda: Download and install from the official Anaconda website.
• Install Git: Download and install from the official Git website.

Step 1: Install Ryzen AI

Clone the Ryzen AI Software repository:

git clone https://github.com/amd/RyzenAI-SW.git

• Navigate to the cloned directory and follow the installation instructions in the README.md file.

Step 2: Prepare the AI Model

• Navigate to example/transformers/models/llm/ in the Ryzen AI installation directory.
• Follow the instructions in README.md file.
• Use the "microsoft/Phi-3-mini-4k-instruct" model for quantization

python run_awq.py --model_name microsoft/Phi-3-mini-4k-instruct --task quantize

• This will generate the quantized model file: quantized_models\quantized_Phi-3-mini-4k-instruct_w4_g128_pergrp.pth

Step 3: Set Up PrivAID

Clone the PrivAID repository:

git clone https://github.com/dutta-d1/PrivAID.git

• Create the registry entry for the native bridge

HKEY_CURRENT_USER\SOFTWARE\Google\Chrome\NativeMessagingHosts\online.privaid.chromeextension.json

• Set the default value to the path of the privaid json. For example,

C:\Users\dipan\Developer\privaid-chrome-extension\online.privaid.chromeextension.json

• Copy the llm_eval file from ryzen-ai/example/transformers/tools/llm_eval.py in the PrivAID repo and paste them into the example/transformers/models/llm directory of your Ryzen AI installation. This is needed because we added support for streaming decoding of the queries.
• Copy the chrome_native_bridge.py and ryzen_message_processor.py in the PrivAID repo from ryzen-ai/example/transformers/models/llm/ and paste into the respective Ryzen AI llm directory.
• Modify the paths in run_native_bridge.bat to match your install location for the Ryzen AI software.

Step 4: Install the Chrome Extension

• Open Google Chrome browser.
• Navigate to chrome://extensions/.
• Enable "Developer mode" in the top right corner.
• Click "Load unpacked" and select the PrivAID directory.

Step 5: Launch PrivAID

• Click on the PrivAID extension icon in your Chrome browser to launch the sidebar.

Usage and Features

PrivAID integrates seamlessly into your Chrome browser as a sidebar extension, offering powerful AI-driven features that respect your privacy. Here's how to use PrivAID and its key features:

Getting Started

• Click on the PrivAID extension icon in your Chrome browser to open the sidebar.
• The sidebar contains two main tabs: Chat and Compose.

Chat Tab Features

Ask Questions:

• Type your query in the input area at the bottom of the Chat tab.
• Click "Send" or press Enter to submit your question.
• The AI processes your query in the context of the current webpage, providing relevant answers.
• Your messages appear as user messages (right-aligned).
• AI responses appear as bot messages (left-aligned) in the scrollable chatbox.

Summarize Page:

• Click the "Summarize Page" button to get an AI-generated summary of the current webpage's content.
• This feature is particularly useful for quickly grasping the main points of long articles or complex web pages.

Compose Tab Features

Generate Content:

• Enter a compose instruction (e.g., "Write an email about..." or "Draft a blog post on...") in the input area.
• Click "Send" or press Enter to submit your instruction.
• The AI will generate content based on your prompt, displayed in the text area above.

Edit Content:

• The generated content appears in an editable text area.
• You can freely modify, refine, or expand upon the AI-generated text.

Copy Content:

• Click the "Copy" button to copy the entire content of the text area to your clipboard.
• This allows for easy pasting into other applications or documents.

Insert Content:

• Click the "Insert" button to automatically insert the content into any text input field on the current webpage.
• This feature is particularly useful for quickly filling in forms, comment sections, or email compose windows.

Privacy and Performance

• All AI processing occurs locally on your device, leveraging your AMD Ryzen processor's NPU capabilities.
• Your data never leaves your device for AI processing, ensuring maximum privacy.
• The extension works offline, not requiring an internet connection for its core functionalities.
• Responses are generated in real-time, allowing for fluid conversations and quick content creation.

Troubleshooting

• If you are facing issues with the extension not able to connect to the Ryzen AI and giving errors, go to chrome://extensions/ and reload the extension. Then refresh the page where you would like to load the extension. Then open the extension. It will reconnect to Ryzen AI.

Conclusion and Future Plans

Conclusion

PrivAID represents a significant leap forward in bringing powerful, privacy-focused AI capabilities directly into the web browsing experience. By leveraging the advanced NPU capabilities of AMD Ryzen processors and the efficiency of the Phi-3 language model, PrivAID offers users a unique combination of features:

• On-device AI processing, ensuring complete privacy of user data
• Seamless integration with Chrome through a user-friendly sidebar interface
• Contextual understanding of web content for more relevant AI interactions
• Versatile functionality including chat-based queries, page summarization, and content composition

PrivAID demonstrates that it's possible to harness the power of advanced AI technologies while maintaining strict standards of user privacy and data security. This project sets a new standard for what users can expect from their browsing assistants, proving that there's no need to compromise between functionality and privacy.

Future Plans

While PrivAID already offers a robust set of features, we have ambitious plans for future improvements and expansions:

Image Interpretation:

• Implement a "Screenshot" feature allowing users to select an area of the current webpage.
• Enable the AI to interpret and analyze the selected image, providing context-aware insights.

Multi-Modal Composition:

• Expand the Compose tab to allow users to generate not just text, but also sounds and images.
• This will enable more diverse content creation directly within the browser.

User Interest Profiling:

• Leverage PrivAID's offline capabilities to build a secure, on-device profile of the user's interests.
• Use this profile to provide personalized recommendations for news articles, websites, and other content.

Enhanced Contextual Understanding:

• Improve the AI's ability to understand and utilize the context of the entire browsing session, not just the current page.

Expanded Language Support:

• Introduce multi-language capabilities to make PrivAID accessible to a global user base.

Performance Optimization:

• Continuously refine the AI model and processing techniques to improve response times and efficiency.

These future improvements will further cement PrivAID's position as a cutting-edge, privacy-first AI assistant, pushing the boundaries of what's possible with on-device AI processing. As we move forward, we remain committed to our core principle of empowering users with advanced AI capabilities while ensuring their data remains completely under their control.

By continuing to innovate at the intersection of AI, privacy, and user experience, PrivAID aims to redefine the future of intelligent web browsing.