三、將best.onnx轉(zhuǎn)為RKNN格式

? ? ? ? 這一步就需要我們進(jìn)入到Ubuntu20.04系統(tǒng)中了，我的Ubuntu系統(tǒng)中已經(jīng)下載好了anaconda，使用anaconda的好處就是可以方便的安裝一些庫(kù)，而且還可以利用conda來(lái)配置虛擬環(huán)境，做到環(huán)境與環(huán)境之間相互獨(dú)立。

??????? 對(duì)于我來(lái)說(shuō)，使用了以下命令創(chuàng)建了一個(gè)名為rknn_ceshi的虛擬環(huán)境

conda create -n rknn_ceshi python=3.8

??????? 之后，點(diǎn)擊GitHub - rockchip-linux/rknn-toolkit2把整個(gè)項(xiàng)目給下載下來(lái)，解壓后，我們進(jìn)入剛剛創(chuàng)立虛擬環(huán)境下安裝rknn-toolkit2。進(jìn)入doc目錄后，輸入命令

pip install -r requirements_cp38-1.4.0.txt -i https://mirror.baidu.com/pypi/simple

??????? 這里一定要帶上百度的鏡像源，要不然會(huì)報(bào)錯(cuò)，這個(gè)坑折磨了我整整3小時(shí)。

??????? 安裝完成后，出現(xiàn)下面的界面就說(shuō)明已經(jīng)安裝好了

????????接下來(lái)，我們進(jìn)入packages文件夾，輸入一下命令

pip install rknn_toolkit2-1.4.0_22dcfef4-cp38-cp38-linux_x86_64.whl

??????? 出現(xiàn)以下界面，但是這里一定要注意，有一個(gè)超級(jí)大坑，超級(jí)大坑，超級(jí)大坑，后續(xù)我也是詢問(wèn)了大佬才解決的這個(gè)問(wèn)題，這個(gè)地方安裝的是rknn-toolkit2-1.4.0-22dcfef4！！！

??????? 出現(xiàn)上面那個(gè)界面后，我們?cè)诮K端輸入python，再輸入以下命令，如果沒有報(bào)錯(cuò)，則證明我們的環(huán)境已經(jīng)搭載好了

from rknn.api import RKNN

??????? 接下來(lái)，我們要做的就是修改test.py里面的一些內(nèi)容

????????我第一次做的時(shí)候，報(bào)了錯(cuò)誤，當(dāng)時(shí)提示的是沒有這個(gè)2-1.4.0-22dcfef4版本，我就跑到conda的環(huán)境包下，把所有的2-1.4.0-22dcfef4版本改成了2-1.4.0，才解決這個(gè)問(wèn)題。

??????? 可是當(dāng)我這一次在執(zhí)行這個(gè)文件的時(shí)候，就沒報(bào)這個(gè)錯(cuò)誤，直接就跑起來(lái)了。

??????? 之后在我們的文件夾下出現(xiàn)了best.rknn這樣就可以到香橙派5上部署了！！！

四、香橙派5部署rknn實(shí)現(xiàn)NPU加速YOLOV5視頻推理

? ? ? ? 這里給大家強(qiáng)調(diào)一下，我使用的是RKNN的python版本來(lái)實(shí)現(xiàn)NPU加速的，這里我們需要到Github上下載RKNN官方教程，下載完成后進(jìn)入該文件夾，輸入指令

cd /examples/onnx/yolov5

? ? ? ? 進(jìn)入文件夾后，創(chuàng)建一個(gè)名為demo.py的文件，將以下代碼復(fù)制即可，我已經(jīng)實(shí)現(xiàn)了實(shí)時(shí)視頻為了保護(hù)隱私，這里我依舊采用官方的yolov5s.rknn模型，而沒有用我自己的

import os
import urllib
import traceback
import time
import datetime as dt
import sys
import numpy as np
import cv2
from rknnlite.api import RKNNLiteRKNN_MODEL = 'yolov5s.rknn'
DATASET = './dataset.txt'QUANTIZE_ON = TrueOBJ_THRESH = 0.25
NMS_THRESH = 0.45
IMG_SIZE = 640CLASSES = ("person", "bicycle", "car", "motorbike ", "aeroplane ", "bus ", "train", "truck ", "boat", "traffic light","fire hydrant", "stop sign ", "parking meter", "bench", "bird", "cat", "dog ", "horse ", "sheep", "cow", "elephant","bear", "zebra ", "giraffe", "backpack", "umbrella", "handbag", "tie", "suitcase", "frisbee", "skis", "snowboard", "sports ball", "kite","baseball bat", "baseball glove", "skateboard", "surfboard", "tennis racket", "bottle", "wine glass", "cup", "fork", "knife ","spoon", "bowl", "banana", "apple", "sandwich", "orange", "broccoli", "carrot", "hot dog", "pizza ", "donut", "cake", "chair", "sofa","pottedplant", "bed", "diningtable", "toilet ", "tvmonitor", "laptop	", "mouse	", "remote ", "keyboard ", "cell phone", "microwave ","oven ", "toaster", "sink", "refrigerator ", "book", "clock", "vase", "scissors ", "teddy bear ", "hair drier", "toothbrush ")def sigmoid(x):return 1 / (1 + np.exp(-x))def xywh2xyxy(x):# Convert [x, y, w, h] to [x1, y1, x2, y2]y = np.copy(x)y[:, 0] = x[:, 0] - x[:, 2] / 2  # top left xy[:, 1] = x[:, 1] - x[:, 3] / 2  # top left yy[:, 2] = x[:, 0] + x[:, 2] / 2  # bottom right xy[:, 3] = x[:, 1] + x[:, 3] / 2  # bottom right yreturn ydef process(input, mask, anchors):anchors = [anchors[i] for i in mask]grid_h, grid_w = map(int, input.shape[0:2])box_confidence = sigmoid(input[..., 4])box_confidence = np.expand_dims(box_confidence, axis=-1)box_class_probs = sigmoid(input[..., 5:])box_xy = sigmoid(input[..., :2])*2 - 0.5col = np.tile(np.arange(0, grid_w), grid_w).reshape(-1, grid_w)row = np.tile(np.arange(0, grid_h).reshape(-1, 1), grid_h)col = col.reshape(grid_h, grid_w, 1, 1).repeat(3, axis=-2)row = row.reshape(grid_h, grid_w, 1, 1).repeat(3, axis=-2)grid = np.concatenate((col, row), axis=-1)box_xy += gridbox_xy *= int(IMG_SIZE/grid_h)box_wh = pow(sigmoid(input[..., 2:4])*2, 2)box_wh = box_wh * anchorsbox = np.concatenate((box_xy, box_wh), axis=-1)return box, box_confidence, box_class_probsdef filter_boxes(boxes, box_confidences, box_class_probs):"""Filter boxes with box threshold. It's a bit different with origin yolov5 post process!# Argumentsboxes: ndarray, boxes of objects.box_confidences: ndarray, confidences of objects.box_class_probs: ndarray, class_probs of objects.# Returnsboxes: ndarray, filtered boxes.classes: ndarray, classes for boxes.scores: ndarray, scores for boxes."""boxes = boxes.reshape(-1, 4)box_confidences = box_confidences.reshape(-1)box_class_probs = box_class_probs.reshape(-1, box_class_probs.shape[-1])_box_pos = np.where(box_confidences >= OBJ_THRESH)boxes = boxes[_box_pos]box_confidences = box_confidences[_box_pos]box_class_probs = box_class_probs[_box_pos]class_max_score = np.max(box_class_probs, axis=-1)classes = np.argmax(box_class_probs, axis=-1)_class_pos = np.where(class_max_score >= OBJ_THRESH)boxes = boxes[_class_pos]classes = classes[_class_pos]scores = (class_max_score* box_confidences)[_class_pos]return boxes, classes, scoresdef nms_boxes(boxes, scores):"""Suppress non-maximal boxes.# Argumentsboxes: ndarray, boxes of objects.scores: ndarray, scores of objects.# Returnskeep: ndarray, index of effective boxes."""x = boxes[:, 0]y = boxes[:, 1]w = boxes[:, 2] - boxes[:, 0]h = boxes[:, 3] - boxes[:, 1]areas = w * horder = scores.argsort()[::-1]keep = []while order.size > 0:i = order[0]keep.append(i)xx1 = np.maximum(x[i], x[order[1:]])yy1 = np.maximum(y[i], y[order[1:]])xx2 = np.minimum(x[i] + w[i], x[order[1:]] + w[order[1:]])yy2 = np.minimum(y[i] + h[i], y[order[1:]] + h[order[1:]])w1 = np.maximum(0.0, xx2 - xx1 + 0.00001)h1 = np.maximum(0.0, yy2 - yy1 + 0.00001)inter = w1 * h1ovr = inter / (areas[i] + areas[order[1:]] - inter)inds = np.where(ovr <= NMS_THRESH)[0]order = order[inds + 1]keep = np.array(keep)return keepdef yolov5_post_process(input_data):masks = [[0, 1, 2], [3, 4, 5], [6, 7, 8]]anchors = [[10, 13], [16, 30], [33, 23], [30, 61], [62, 45],[59, 119], [116, 90], [156, 198], [373, 326]]boxes, classes, scores = [], [], []for input, mask in zip(input_data, masks):b, c, s = process(input, mask, anchors)b, c, s = filter_boxes(b, c, s)boxes.append(b)classes.append(c)scores.append(s)boxes = np.concatenate(boxes)boxes = xywh2xyxy(boxes)classes = np.concatenate(classes)scores = np.concatenate(scores)nboxes, nclasses, nscores = [], [], []for c in set(classes):inds = np.where(classes == c)b = boxes[inds]c = classes[inds]s = scores[inds]keep = nms_boxes(b, s)nboxes.append(b[keep])nclasses.append(c[keep])nscores.append(s[keep])if not nclasses and not nscores:return None, None, Noneboxes = np.concatenate(nboxes)classes = np.concatenate(nclasses)scores = np.concatenate(nscores)return boxes, classes, scoresdef draw(image, boxes, scores, classes, fps):"""Draw the boxes on the image.# Argument:image: original image.boxes: ndarray, boxes of objects.classes: ndarray, classes of objects.scores: ndarray, scores of objects.fps: int.all_classes: all classes name."""for box, score, cl in zip(boxes, scores, classes):top, left, right, bottom = boxprint('class: {}, score: {}'.format(CLASSES[cl], score))print('box coordinate left,top,right,down: [{}, {}, {}, {}]'.format(top, left, right, bottom))top = int(top)left = int(left)right = int(right)bottom = int(bottom)cv2.rectangle(image, (top, left), (right, bottom), (255, 0, 0), 2)cv2.putText(image, '{0} {1:.2f}'.format(CLASSES[cl], score),(top, left - 6),cv2.FONT_HERSHEY_SIMPLEX,0.6, (0, 0, 255), 2)def letterbox(im, new_shape=(640, 640), color=(0, 0, 0)):# Resize and pad image while meeting stride-multiple constraintsshape = im.shape[:2]  # current shape [height, width]if isinstance(new_shape, int):new_shape = (new_shape, new_shape)# Scale ratio (new / old)r = min(new_shape[0] / shape[0], new_shape[1] / shape[1])# Compute paddingratio = r, r  # width, height ratiosnew_unpad = int(round(shape[1] * r)), int(round(shape[0] * r))dw, dh = new_shape[1] - new_unpad[0], new_shape[0] - new_unpad[1]  # wh paddingdw /= 2  # divide padding into 2 sidesdh /= 2if shape[::-1] != new_unpad:  # resizeim = cv2.resize(im, new_unpad, interpolation=cv2.INTER_LINEAR)top, bottom = int(round(dh - 0.1)), int(round(dh + 0.1))left, right = int(round(dw - 0.1)), int(round(dw + 0.1))im = cv2.copyMakeBorder(im, top, bottom, left, right, cv2.BORDER_CONSTANT, value=color)  # add borderreturn im, ratio, (dw, dh)# ==================================
# 如下為改動(dòng)部分，主要就是去掉了官方 demo 中的模型轉(zhuǎn)換代碼，直接加載 rknn 模型，并將 RKNN 類換成了 rknn_toolkit2_lite 中的 RKNNLite 類
# ==================================rknn = RKNNLite()# load RKNN model
print('--> Load RKNN model')
ret = rknn.load_rknn(RKNN_MODEL)# Init runtime environment
print('--> Init runtime environment')
# use NPU core 0 1 2
ret = rknn.init_runtime(core_mask=RKNNLite.NPU_CORE_0_1_2)
if ret != 0:print('Init runtime environment failed!')exit(ret)
print('done')# Create a VideoCapture object and read from input file
# If the input is the camera, pass 0 instead of the video file name
cap = cv2.VideoCapture(0)# Check if camera opened successfully
if (cap.isOpened()== False): print("Error opening video stream or file")# Read until video is completed
while(cap.isOpened()):start = dt.datetime.utcnow()# Capture frame-by-frameret, img = cap.read()if not ret:breakimg = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)img = cv2.resize(img, (IMG_SIZE, IMG_SIZE))# Inferenceprint('--> Running model')outputs = rknn.inference(inputs=[img])print('done')# post processinput0_data = outputs[0]input1_data = outputs[1]input2_data = outputs[2]input0_data = input0_data.reshape([3, -1]+list(input0_data.shape[-2:]))input1_data = input1_data.reshape([3, -1]+list(input1_data.shape[-2:]))input2_data = input2_data.reshape([3, -1]+list(input2_data.shape[-2:]))input_data = list()input_data.append(np.transpose(input0_data, (2, 3, 0, 1)))input_data.append(np.transpose(input1_data, (2, 3, 0, 1)))input_data.append(np.transpose(input2_data, (2, 3, 0, 1)))boxes, classes, scores = yolov5_post_process(input_data)duration = dt.datetime.utcnow() - startfps = round(10000000 / duration.microseconds)# draw process result and fpsimg_1 = cv2.cvtColor(img, cv2.COLOR_RGB2BGR)cv2.putText(img_1, f'fps: {fps}',(20, 20),cv2.FONT_HERSHEY_SIMPLEX,0.6, (0, 125, 125), 2)if boxes is not None:draw(img_1, boxes, scores, classes, fps)# show outputcv2.imshow("post process result", img_1)# Press Q on keyboard to  exitif cv2.waitKey(25) & 0xFF == ord('q'):break# When everything done, release the video capture object
cap.release()# Closes all the frames
cv2.destroyAllWindows()

? ? ? ? 這里我開啟了CPU定頻，可是并沒有提高多少NPU的使用率，干脆我也不再把代碼放上來(lái)了，等到以后有時(shí)間我再研究一下。

? ? ? ? 之后在終端中，運(yùn)行命令

python demo.py

? ? ? ? 效果我放到了B站，感興趣的小伙伴可以點(diǎn)進(jìn)去看一下B站視頻