Narrative Canvas: Image-Inspired Storytelling

Narrative Canvas: Image-Inspired Storytelling

Narrative Canvas, also known as "Language Within the Paintings," is the very essence of this project. Here, each canvas is not merely a combination of colors and lines but a collection of untold stories waiting to be discovered. Artists unleash their imaginations onto the canvas, and every stroke and every brushstroke carries profound emotions and a unique perspective. These artworks, akin to poems without words, quietly narrate their own tales.

This project has successfully implemented image inference tasks, text generation tasks, and image generation tasks on the Jetson development board. It utilizes TensorRT for accelerated inference and Flask to run the UI page. This project was awarded first place in the Nvidia 9th Sky Hackathon competition.

The entire project workflow can be divided into the following steps:

Image Inference

• Image Inference
  

Story Generation

• Story Generation
  

Image Generation

• Image Generation
  

Prerequisites

Prepare Model && Calibration Data

ONNX

Our project models are based on the mmpretrain pre-trained models from the mmlab algorithm library. We have carefully selected 25 classic backbone networks for the image classification task in this project. We also provide scripts for converting PyTorch (pt) models to ONNX models, including the recent work on EfficientVit. Additionally, we offer conversion scripts to export ONNX models in Dynamic Shape mode.

We provide both preprocessed ONNX models using Polygraphy and the original exported ONNX model files, You can choose to download it from Google Drive or Hugging Face.

Please place the downloaded ONNX file into the models/onnx directory.

Calibdata

Our calibration dataset consists of 510 images selected from the ImageNet 1K validation dataset. We also provide a download link for the calibration dataset.

Please place the downloaded calibdata file into the models/calibdata directory.

Prepare API

Before running this project, you need to prepare the Nvidia NGC llama2-70b-steerlm API and the Nvidia NGC Stable Diffusion XL API and fill in their details in the config.json file. You can also fill in your Azure OpenAI API key in the config.json if you have one, but this is not mandatory.

"sdxl": {
     "invoke_url": "" ,
     "fetch_url_format": "",
     "headers": {
         "Authorization": "",
         "Accept": "application/json"
     }
 },
 "llama2": {
     "invoke_url": "",
     "fetch_url_format": "",
     "headers": {
         "Authorization": "",
         "Accept": ""
     }
 },
 "azure_openai":{
     "api_key": "",
     "api_base": "",
     "deployment_name": "",
     "api_version": ""
 }

Setup Runtime Environment

We provide two methods for building the runtime environment for different hardware environments. One is deploying the environment using Nvidia Container on Windows or Linux, and the other is configuring the environment using pip on a Jetson Orin development board.

Nvidia Container

Pytorch 23.10 (Ubuntu 22.04 + TensorRT 8.6.1.6 + CUDA 12.2.1)

• Pytorch 23.10 (Ubuntu 22.04 + TensorRT 8.6.1.6 + CUDA 12.2.1)
  
• Nvidia Container
  
  Pytorch 23.10 (Ubuntu 22.04 + TensorRT 8.6.1.6 + CUDA 12.2.1)
  

Nvidia Jetson Orin

Jetpack 5.1.2 (Jetson Linux 35.4.1 + TensorRT 8.5.2 + DLA 3.12.1 + cuDNN 8.6.0 + CUDA 11.4.19)

• Jetpack 5.1.2 (Jetson Linux 35.4.1 + TensorRT 8.5.2 + DLA 3.12.1 + cuDNN 8.6.0 + CUDA 11.4.19)
  
• Nvidia Jetson Orin
  
  Jetpack 5.1.2 (Jetson Linux 35.4.1 + TensorRT 8.5.2 + DLA 3.12.1 + cuDNN 8.6.0 + CUDA 11.4.19)
  

Note: If you are using the Jetson Xavier NX hardware platform, please refer to this project: https://github.com/1438802682/NarrativeCanvas-JetsonXavierNX

Note: If you are using the Jetson Xavier NX hardware platform, please refer to this project: https://github.com/1438802682/NarrativeCanvas-JetsonXavierNX

Windows && Linux

We provide a Dockerfile to ease environment setup. Please execute the following command to build the docker image after nvidia-docker installation:

docker build -t sky docker

Jetson Orin

Before building the runtime environment on the Jetson Orin platform, please upgrade the Jetson Orin's JetPack environment version to 5.1.2, and then execute the following command:

pip3 install requirements.txt

Run

We can then run the docker with the following command:

Windows && Linux

Windows:

• Windows:
  

docker run --gpus all --rm -it -p 3008:3008 -v %cd%:/sky sky

Linux:

• Linux:
  

docker run --gpus all --rm -it -p 3008:3008 -v $PWD:/sky sky

After you have completed the steps above, please visit http://127.0.0.1:3008/ to embark on your creative journey!

Jetson Orin

gunicorn -b 127.0.0.1:3008 app:app

After you have completed the steps above, please visit http://127.0.0.1:3008/ to embark on your creative journey!

Note

Demonstration Sample

UI Prototype

Project Architecture Diagram

Flowchart

If you encounter any issues or would like to obtain more technical details, please feel free to contact me at 1438802682@qq.com

• If you encounter any issues or would like to obtain more technical details, please feel free to contact me at 1438802682@qq.com
  

References

Narrative Canvas on Jetson Xavier NX

• Narrative Canvas on Jetson Xavier NX
  

EfficientViT: Multi-Scale Linear Attention for High-Resolution Dense Prediction

• EfficientViT: Multi-Scale Linear Attention for High-Resolution Dense Prediction
  

MMPreTrain is an open source pre-training toolbox based on PyTorch

• MMPreTrain is an open source pre-training toolbox based on PyTorch