设为首页收藏本站
首页 > 论坛 > 行空板

[M10项目]行空板M10 小智AI 聊天机器人 精华

云天 | 初级技神 | 创造力： | 帖子： | 发消息 | 串个门 | 加好友 | 打招呼
2025-2-26 10:50:47 [显示全部楼层]
419浏览
0回复
2收藏
查看: 419|回复: 0

[M10项目] 行空板M10 小智AI 聊天机器人

[复制链接]
本帖最后由 云天 于 2025-2-26 11:04 编辑

【项目背景】
       近期，网络上涌现出众多基于 ESP32-S3 设备的“小智 AI聊天机器人”复刻项目。然而，这些项目大多在组装完成后直接下载固件，用户无法对程序进行进一步修改。（也可以在Windows上搭建 ESP IDF 5.3开发环境编译小智）
        最近，有人开源了一种使用 Python 实现“小智 AI”的方案，该方案可在 Windows 电脑和树莓派上运行。鉴于行空板 M10 搭载了 Python 编程环境，我将相关代码移植至该平台，并进行了适配优化，使其能够更好地运行。

【小智AI ESP32S3】

        小智AI聊天机器人是一款基于人工智能技术的交互式语音助手，具备语音识别、自然语言处理、语音合成等功能，能够通过语音与用户进行交流，通常基于ESP32-S3开发板等硬件平台实现，支持Wi-Fi或4G联网，并配备麦克风、扬声器、显示屏等模块。        开源平台：https://xiaozhi.me/
行空板M10 小智AI 聊天机器人图1


【小智AI Python】
       开源网址：https://github.com/zhh827/py-xiaozhi，用python实现的小智客户端,用于代码学习和在没有硬件条件下体验AI小智的语音功能。大家可以选使用Mind+的python环境下试运行。
        注意事项：1.通过Mind+的库管理，安装requirements.txt文件中的相关库。2.将opus.dll拷贝到至C:\Windows\System32目录中。3.需要手动修改py-xiaozhi.py脚本中的全局变量MAC_ADDR（字母小写），运行py-xiaozhi.py后，根据窗口中打印出的注册码，到https://xiaozhi.me/的控制台中进行注册。
【小智AI 行空板】
1.使用行空板上A、B键，实现开始录音、停止录音，与“小智”进行对话。

  1. 

  2. #!/usr/bin/python

  3. # -*- coding: UTF-8 -*-

  4. import json

  5. import time

  6. import requests

  7. import paho.mqtt.client as mqtt

  8. import threading

  9. import pyaudio

  10. import opuslib  # windwos平台需要将opus.dll 拷贝到C:\Windows\System32

  11. import socket

  12. from cryptography.hazmat.primitives.ciphers import Cipher, algorithms, modes

  13. from cryptography.hazmat.backends import default_backend

  14. from os import urandom

  15. import logging

  16. from pynput import keyboard as pynput_keyboard

  17. from unihiker import GUI

  18. 

  19. import webrtcvad

  20. from pinpong.board import Pin

  21. from pinpong.board import Board

  22. from pinpong.board import NeoPixel

  23. 

  24. # 初始化行空板硬件接口

  25. Board().begin()

  26. gui=GUI()

  27. pin1 = Pin(Pin.D23)

  28. np1 = NeoPixel(pin1,24)

  29. np1.brightness(128)

  30. 

  31. np1.clear()

  32. 

  33. gui.clear()

  34. 

  35. fontSize=20

  36. max_lines = 16

  37. max_chars=8

  38. # 图形界面元素

  39. status_label = gui.draw_text(x=80, y=10, text='初始化中...', color='red')

  40. log_text = gui.draw_text(x=10, y=100, text='', font_size=fontSize,color='blue')

  41. emotion = gui.draw_text(x=110, y=50, text='', font_size=fontSize,color='green')

  42. OTA_VERSION_URL = 'https://api.tenclass.net/xiaozhi/ota/'

  43. MAC_ADDR = '7a:da:6b:5c:76:50'

  44. # {"mqtt":{"endpoint":"post-cn-apg3xckag01.mqtt.aliyuncs.com","client_id":"GID_test@@@cc_ba_97_20_b4_bc",

  45. # "username":"Signature|LTAI5tF8J3CrdWmRiuTjxHbF|post-cn-apg3xckag01","password":"0mrkMFELXKyelhuYy2FpGDeCigU=",

  46. # "publish_topic":"device-server","subscribe_topic":"devices"},"firmware":{"version":"0.9.9","url":""}}

  47. mqtt_info = {}

  48. aes_opus_info = {"type": "hello", "version": 3, "transport": "udp",

  49.                  "udp": {"server": "120.24.160.13", "port": 8884, "encryption": "aes-128-ctr",

  50.                          "key": "263094c3aa28cb42f3965a1020cb21a7", "nonce": "01000000ccba9720b4bc268100000000"},

  51.                  "audio_params": {"format": "opus", "sample_rate": 24000, "channels": 1, "frame_duration": 60},

  52.                  "session_id": "b23ebfe9"}

  53. 

  54. iot_msg = {"session_id": "635aa42d", "type": "iot",

  55.            "descriptors": [{"name": "Speaker", "description": "当前 AI 机器人的扬声器",

  56.                             "properties": {"volume": {"description": "当前音量值", "type": "number"}},

  57.                             "methods": {"SetVolume": {"description": "设置音量",

  58.                                                       "parameters": {

  59.                                                           "volume": {"description": "0到100之间的整数", "type": "number"}

  60.                                                       }

  61.                                                       }

  62.                                         }

  63.                             },

  64.                            {"name": "Lamp", "description": "一个测试用的灯",

  65.                             "properties": {"power": {"description": "灯是否打开", "type": "boolean"}},

  66.                             "methods": {"TurnOn": {"description": "打开灯", "parameters": {}},

  67.                                         "TurnOff": {"description": "关闭灯", "parameters": {}}

  68.                                         }

  69.                             }

  70.                            ]

  71.            }

  72. iot_status_msg = {"session_id": "635aa42d", "type": "iot", "states": [

  73.     {"name": "Speaker", "state": {"volume": 50}}, {"name": "Lamp", "state": {"power": False}}]}

  74. goodbye_msg = {"session_id": "b23ebfe9", "type": "goodbye"}

  75. local_sequence = 0

  76. listen_state = None

  77. tts_state = None

  78. key_state = None

  79. audio = None

  80. udp_socket = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)

  81. # udp_socket.setblocking(False)

  82. conn_state = False

  83. recv_audio_thread = threading.Thread()

  84. send_audio_thread = threading.Thread()

  85. mqttc = None

  86. 

  87. 

  88. def get_ota_version():

  89.     global mqtt_info

  90.     header = {

  91.         'Device-Id': MAC_ADDR,

  92.         'Content-Type': 'application/json'

  93.     }

  94.     post_data = {"flash_size": 16777216, "minimum_free_heap_size": 8318916, "mac_address": f"{MAC_ADDR}",

  95.                  "chip_model_name": "esp32s3", "chip_info": {"model": 9, "cores": 2, "revision": 2, "features": 18},

  96.                  "application": {"name": "xiaozhi", "version": "0.9.9", "compile_time": "Jan 22 2025T20:40:23Z",

  97.                                  "idf_version": "v5.3.2-dirty",

  98.                                  "elf_sha256": "22986216df095587c42f8aeb06b239781c68ad8df80321e260556da7fcf5f522"},

  99.                  "partition_table": [{"label": "nvs", "type": 1, "subtype": 2, "address": 36864, "size": 16384},

  100.                                      {"label": "otadata", "type": 1, "subtype": 0, "address": 53248, "size": 8192},

  101.                                      {"label": "phy_init", "type": 1, "subtype": 1, "address": 61440, "size": 4096},

  102.                                      {"label": "model", "type": 1, "subtype": 130, "address": 65536, "size": 983040},

  103.                                      {"label": "storage", "type": 1, "subtype": 130, "address": 1048576,

  104.                                       "size": 1048576},

  105.                                      {"label": "factory", "type": 0, "subtype": 0, "address": 2097152, "size": 4194304},

  106.                                      {"label": "ota_0", "type": 0, "subtype": 16, "address": 6291456, "size": 4194304},

  107.                                      {"label": "ota_1", "type": 0, "subtype": 17, "address": 10485760,

  108.                                       "size": 4194304}],

  109.                  "ota": {"label": "factory"},

  110.                  "board": {"type": "bread-compact-wifi", "ssid": "mzy", "rssi": -58, "channel": 6,

  111.                            "ip": "192.168.124.38", "mac": "cc:ba:97:20:b4:bc"}}

  112. 

  113.     response = requests.post(OTA_VERSION_URL, headers=header, data=json.dumps(post_data))

  114.     print('=========================')

  115.     print(response.text)

  116.     logging.info(f"get version: {response}")

  117.     mqtt_info = response.json()['mqtt']

  118. 

  119. 

  120. def aes_ctr_encrypt(key, nonce, plaintext):

  121.     cipher = Cipher(algorithms.AES(key), modes.CTR(nonce), backend=default_backend())

  122.     encryptor = cipher.encryptor()

  123.     return encryptor.update(plaintext) + encryptor.finalize()

  124. 

  125. 

  126. def aes_ctr_decrypt(key, nonce, ciphertext):

  127.     cipher = Cipher(algorithms.AES(key), modes.CTR(nonce), backend=default_backend())

  128.     decryptor = cipher.decryptor()

  129.     plaintext = decryptor.update(ciphertext) + decryptor.finalize()

  130.     return plaintext

  131. 

  132. 

  133. def send_audio():

  134.     global aes_opus_info, udp_socket, local_sequence, listen_state, audio

  135.     key = aes_opus_info['udp']['key']

  136.     nonce = aes_opus_info['udp']['nonce']

  137.     server_ip = aes_opus_info['udp']['server']

  138.     server_port = aes_opus_info['udp']['port']

  139.     # 初始化Opus编码器

  140.     encoder = opuslib.Encoder(16000, 1, opuslib.APPLICATION_AUDIO)

  141.     # 打开麦克风流, 帧大小，应该与Opus帧大小匹配

  142.     mic = audio.open(format=pyaudio.paInt16, channels=1, rate=16000, input=True, frames_per_buffer=960)

  143.     try:

  144.         while True:

  145.             if listen_state == "stop":

  146.                 continue

  147.                 time.sleep(0.1)

  148.             # 读取音频数据

  149.             data = mic.read(960)

  150.             # 编码音频数据

  151.             encoded_data = encoder.encode(data, 960)

  152.             # 打印音频数据

  153.             # print(f"Encoded data: {len(encoded_data)}")

  154.             # nonce插入data.size local_sequence_

  155.             local_sequence += 1

  156.             new_nonce = nonce[0:4] + format(len(encoded_data), '04x') + nonce[8:24] + format(local_sequence, '08x')

  157.             # 加密数据，添加nonce

  158.             encrypt_encoded_data = aes_ctr_encrypt(bytes.fromhex(key), bytes.fromhex(new_nonce), bytes(encoded_data))

  159.             data = bytes.fromhex(new_nonce) + encrypt_encoded_data

  160.             sent = udp_socket.sendto(data, (server_ip, server_port))

  161.     except Exception as e:

  162.         print(f"send audio err: {e}")

  163.     finally:

  164.         print("send audio exit()")

  165.         local_sequence = 0

  166.         udp_socket = None

  167.         # 关闭流和PyAudio

  168.         mic.stop_stream()

  169.         mic.close()

  170. 

  171. 

  172. def recv_audio():

  173.     global aes_opus_info, udp_socket, audio

  174.     key = aes_opus_info['udp']['key']

  175.     nonce = aes_opus_info['udp']['nonce']

  176.     sample_rate = aes_opus_info['audio_params']['sample_rate']

  177.     frame_duration = aes_opus_info['audio_params']['frame_duration']

  178.     frame_num = int(frame_duration / (1000 / sample_rate))

  179.     print(f"recv audio: sample_rate -> {sample_rate}, frame_duration -> {frame_duration}, frame_num -> {frame_num}")

  180.     # 初始化Opus编码器

  181.     decoder = opuslib.Decoder(sample_rate, 1)

  182.     spk = audio.open(format=pyaudio.paInt16, channels=1, rate=sample_rate, output=True, frames_per_buffer=frame_num)

  183.     try:

  184.         while True:

  185.             data, server = udp_socket.recvfrom(4096)

  186.             # print(f"Received from server {server}: {len(data)}")

  187.             encrypt_encoded_data = data

  188.             # 解密数据,分离nonce

  189.             split_encrypt_encoded_data_nonce = encrypt_encoded_data[:16]

  190.             # 十六进制格式打印nonce

  191.             # print(f"split_encrypt_encoded_data_nonce: {split_encrypt_encoded_data_nonce.hex()}")

  192.             split_encrypt_encoded_data = encrypt_encoded_data[16:]

  193.             decrypt_data = aes_ctr_decrypt(bytes.fromhex(key),

  194.                                            split_encrypt_encoded_data_nonce,

  195.                                            split_encrypt_encoded_data)

  196.             # 解码播放音频数据

  197.             spk.write(decoder.decode(decrypt_data, frame_num))

  198.     # except BlockingIOError:

  199.     #     # 无数据时短暂休眠以减少CPU占用

  200.     #     time.sleep(0.1)

  201.     except Exception as e:

  202.         print(f"recv audio err: {e}")

  203.     finally:

  204.         udp_socket = None

  205.         spk.stop_stream()

  206.         spk.close()

  207. 

  208. def wrap_hanzi(text, first_line_width=5, other_line_width=16):

  209.     """将字符串格式化为第一行指定宽度，后续行指定宽度"""

  210.     lines = []

  211.     

  212.     # 处理第一行

  213.     if len(text) > first_line_width:

  214.         lines.append(text[:first_line_width])

  215.         remaining_text = text[first_line_width:]

  216.     else:

  217.         lines.append(text)

  218.         remaining_text = ""

  219.     

  220.     # 处理后续行

  221.     for i in range(0, len(remaining_text), other_line_width):

  222.         lines.append(remaining_text[i:i + other_line_width])

  223.     

  224.     return "\n".join(lines)

  225. def get_ascii_emotion(emotion):

  226.     """根据情绪类型返回对应的 ASCII 表情符号"""

  227.     if emotion == "happy":

  228.         return ":)"

  229.     elif emotion == "sad":

  230.         return ":("

  231.     elif emotion == "winking":

  232.         return ";)"

  233.     elif emotion == "surprised":

  234.         return ":O"

  235.     elif emotion == "angry":

  236.         return ">:(("

  237.     elif emotion == "laughing":

  238.         return ":D"

  239.     elif emotion == "cool":

  240.         return "B-)"

  241.     elif emotion == "crying":

  242.         return ":'("

  243.     elif emotion == "shy":

  244.         return "^_^"

  245.     elif emotion == "thinking":

  246.         return ":|"

  247.     elif emotion == "love":

  248.         return "<3"

  249.     elif emotion == "sleepy":

  250.         return "-.-"

  251.     elif emotion == "neutral":

  252.         return ":|"

  253.     elif emotion == "excited":

  254.         return ":D"

  255.     elif emotion == "confused":

  256.         return ":S"

  257.     else:

  258.         return ":("  # 默认表情

  259. 

  260. def on_message(client, userdata, message):

  261.     global aes_opus_info, udp_socket, tts_state, recv_audio_thread, send_audio_thread,max_chars

  262.     msg = json.loads(message.payload)

  263.     print(f"recv msg: {msg}")

  264.     if msg['type'] == 'hello':

  265.        

  266.         aes_opus_info = msg

  267.         udp_socket.connect((msg['udp']['server'], msg['udp']['port']))

  268.          # 检查recv_audio_thread线程是否启动

  269.         if not recv_audio_thread.is_alive():

  270.             # 启动一个线程，用于接收音频数据

  271.             recv_audio_thread = threading.Thread(target=recv_audio)

  272.             recv_audio_thread.start()

  273.         else:

  274.             print("recv_audio_thread is alive")

  275.         # 检查send_audio_thread线程是否启动

  276.         if not send_audio_thread.is_alive():

  277.             # 启动一个线程，用于发送音频数据

  278.             send_audio_thread = threading.Thread(target=send_audio)

  279.             send_audio_thread.start()

  280.         else:

  281.             print("send_audio_thread is alive")

  282.     if msg['type'] == 'llm':

  283.         ascii_emotion = get_ascii_emotion(msg['emotion'])

  284.         emotion.config(text=ascii_emotion)

  285.     if msg['type'] == 'tts' and msg['state']=='start':

  286.             status_label.config(text="讲话中……")

  287.     if msg['type'] == 'tts' and msg['state']=='stop':

  288.             status_label.config(text="就绪")

  289.     if msg['type'] == 'tts' and msg['state']=='sentence_start':

  290.         tts_state = msg['state']

  291.         text=msg['text']

  292.         text=wrap_hanzi(text, 5,max_chars)

  293.         log_text.config(text="小智: " + text)

  294.         if msg['text'] == '开灯':

  295.            np1.range_color(0,23,0x0000FF)

  296.         if msg['text'] == '关灯':

  297.            np1.clear()

  298.     if msg['type'] == 'stt':

  299.         

  300.         text=msg['text']

  301.         text=wrap_hanzi(text, 5,max_chars)

  302.         log_text.config(text="我: " + text)

  303.     if msg['type'] == 'goodbye' and udp_socket and msg['session_id'] == aes_opus_info['session_id']:

  304.         print(f"recv good bye msg")

  305.         aes_opus_info['session_id'] = None

  306. 

  307. 

  308. def on_connect(client, userdata, flags, rs, pr):

  309.     subscribe_topic = mqtt_info['subscribe_topic'].split("/")[0] + '/p2p/GID_test@@@' + MAC_ADDR.replace(':', '_')

  310.     print(f"subscribe topic: {subscribe_topic}")

  311.     # 订阅主题

  312.     client.subscribe(subscribe_topic)

  313. 

  314. 

  315. def push_mqtt_msg(message):

  316.     global mqtt_info, mqttc

  317.     mqttc.publish(mqtt_info['publish_topic'], json.dumps(message))

  318. 

  319. 

  320. def listen_start():

  321.     global key_state, udp_socket, aes_opus_info, listen_state, conn_state

  322.     if key_state == "press":

  323.         return

  324.     key_state = "press"

  325.     # 判断是否需要发送hello消息

  326.     if conn_state is False or aes_opus_info['session_id'] is None:

  327.         conn_state = True

  328.         # 发送hello消息,建立udp连接

  329.         hello_msg = {"type": "hello", "version": 3, "transport": "udp",

  330.                      "audio_params": {"format": "opus", "sample_rate": 16000, "channels": 1, "frame_duration": 60}}

  331.         push_mqtt_msg(hello_msg)

  332.         print(f"send hello message: {hello_msg}")

  333.     if tts_state == "start" or tts_state == "entence_start":

  334.         # 在播放状态下发送abort消息

  335.         push_mqtt_msg({"type": "abort"})

  336.         print(f"send abort message")

  337.     if aes_opus_info['session_id'] is not None:

  338.         # 发送start listen消息

  339.         msg = {"session_id": aes_opus_info['session_id'], "type": "listen", "state": "start", "mode": "manual"}

  340.         print(f"send start listen message: {msg}")

  341.         status_label.config(text="聆听中……")

  342.         push_mqtt_msg(msg)

  343. def listen_stop():

  344.     global aes_opus_info, key_state

  345.     key_state = "release"

  346.     # 发送stop listen消息

  347.     if aes_opus_info['session_id'] is not None:

  348.         msg = {"session_id": aes_opus_info['session_id'], "type": "listen", "state": "stop"}

  349.         print(f"send stop listen message: {msg}")

  350.         push_mqtt_msg(msg)

  351. def run():

  352.     global mqtt_info, mqttc

  353.     # 获取mqtt与版本信息

  354.     get_ota_version()

  355. 

  356.     # 创建客户端实例

  357.     mqttc = mqtt.Client(callback_api_version=mqtt.CallbackAPIVersion.VERSION2, client_id=mqtt_info['client_id'])

  358.     mqttc.username_pw_set(username=mqtt_info['username'], password=mqtt_info['password'])

  359.     mqttc.tls_set(ca_certs=None, certfile=None, keyfile=None, cert_reqs=mqtt.ssl.CERT_REQUIRED,

  360.                   tls_version=mqtt.ssl.PROTOCOL_TLS, ciphers=None)

  361.     mqttc.on_connect = on_connect

  362.     mqttc.on_message = on_message

  363.     mqttc.connect(host=mqtt_info['endpoint'], port=8883)

  364.     gui.on_a_click(listen_start)

  365.     gui.on_b_click(listen_stop)

  366.     mqttc.loop_forever()

  367. 

  368. 

  369. if __name__ == "__main__":

  370.     audio = pyaudio.PyAudio()

  371. 

  372.     run()

  373. 

复制代码

（演示视频中，让小智根据对话情景开关灯）

2.通过行空板引脚24外接按钮实现与“小智”对话
行空板M10 小智AI 聊天机器人图2


  1. 

  2. #增加了24引脚按键，使用舵机模拟双手

  3. #!/usr/bin/python

  4. # -*- coding: UTF-8 -*-

  5. import json

  6. import time

  7. import requests

  8. import paho.mqtt.client as mqtt

  9. import threading

  10. import pyaudio

  11. import opuslib  # windwos平台需要将opus.dll 拷贝到C:\Windows\System32

  12. import socket

  13. from cryptography.hazmat.primitives.ciphers import Cipher, algorithms, modes

  14. from cryptography.hazmat.backends import default_backend

  15. from os import urandom

  16. import logging

  17. from pynput import keyboard as pynput_keyboard

  18. from unihiker import GUI

  19. 

  20. 

  21. from pinpong.board import Pin

  22. from pinpong.board import Board

  23. from pinpong.board import Servo

  24. 

  25. # 初始化行空板硬件接口

  26. Board().begin()

  27. gui=GUI()

  28. gui.clear()

  29. #舵机引脚及按键引脚设置

  30. pin2 = Pin(Pin.D21)

  31. pin3 = Pin(Pin.D22)

  32. servo1 = Servo(pin2)

  33. servo2 = Servo(pin3)

  34. p_p24_in=Pin(Pin.P24, Pin.IN)

  35. #显示内容设置

  36. fontSize=20

  37. max_lines = 16

  38. max_chars=8

  39. # 图形界面元素

  40. status_label = gui.draw_text(x=80, y=10, text='初始化中...', color='red')

  41. log_text = gui.draw_text(x=10, y=100, text='', font_size=fontSize,color='blue')

  42. emotion = gui.draw_text(x=110, y=50, text='', font_size=fontSize,color='green')

  43. OTA_VERSION_URL = 'https://api.tenclass.net/xiaozhi/ota/'

  44. MAC_ADDR = '7a:da:6b:5c:76:50'

  45. # {"mqtt":{"endpoint":"post-cn-apg3xckag01.mqtt.aliyuncs.com","client_id":"GID_test@@@cc_ba_97_20_b4_bc",

  46. # "username":"Signature|LTAI5tF8J3CrdWmRiuTjxHbF|post-cn-apg3xckag01","password":"0mrkMFELXKyelhuYy2FpGDeCigU=",

  47. # "publish_topic":"device-server","subscribe_topic":"devices"},"firmware":{"version":"0.9.9","url":""}}

  48. mqtt_info = {}

  49. aes_opus_info = {"type": "hello", "version": 3, "transport": "udp",

  50.                  "udp": {"server": "120.24.160.13", "port": 8884, "encryption": "aes-128-ctr",

  51.                          "key": "263094c3aa28cb42f3965a1020cb21a7", "nonce": "01000000ccba9720b4bc268100000000"},

  52.                  "audio_params": {"format": "opus", "sample_rate": 24000, "channels": 1, "frame_duration": 60},

  53.                  "session_id": None}

  54. 

  55. iot_msg = {"session_id": "635aa42d", "type": "iot",

  56.            "descriptors": [{"name": "Speaker", "description": "当前 AI 机器人的扬声器",

  57.                             "properties": {"volume": {"description": "当前音量值", "type": "number"}},

  58.                             "methods": {"SetVolume": {"description": "设置音量",

  59.                                                       "parameters": {

  60.                                                           "volume": {"description": "0到100之间的整数", "type": "number"}

  61.                                                       }

  62.                                                       }

  63.                                         }

  64.                             },

  65.                            {"name": "Lamp", "description": "一个测试用的灯",

  66.                             "properties": {"power": {"description": "灯是否打开", "type": "boolean"}},

  67.                             "methods": {"TurnOn": {"description": "打开灯", "parameters": {}},

  68.                                         "TurnOff": {"description": "关闭灯", "parameters": {}}

  69.                                         }

  70.                             }

  71.                            ]

  72.            }

  73. iot_status_msg = {"session_id": "635aa42d", "type": "iot", "states": [

  74.     {"name": "Speaker", "state": {"volume": 50}}, {"name": "Lamp", "state": {"power": False}}]}

  75. goodbye_msg = {"session_id": "b23ebfe9", "type": "goodbye"}

  76. local_sequence = 0

  77. listen_state = None

  78. tts_state = None

  79. key_state = None

  80. audio = None

  81. udp_socket = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)

  82. # udp_socket.setblocking(False)

  83. conn_state = False

  84. recv_audio_thread = threading.Thread()

  85. send_audio_thread = threading.Thread()

  86. anjian_thread = threading.Thread()

  87. mqttc = None

  88. # 抬手 放手函数

  89. def handup():

  90.     servo1.write_angle(170)

  91.     servo2.write_angle(10)

  92. def handdown():

  93.     servo1.write_angle(10)

  94.     servo2.write_angle(170)

  95. 

  96. def get_ota_version():

  97.     global mqtt_info

  98.     header = {

  99.         'Device-Id': MAC_ADDR,

  100.         'Content-Type': 'application/json'

  101.     }

  102.     post_data = {"flash_size": 16777216, "minimum_free_heap_size": 8318916, "mac_address": f"{MAC_ADDR}",

  103.                  "chip_model_name": "esp32s3", "chip_info": {"model": 9, "cores": 2, "revision": 2, "features": 18},

  104.                  "application": {"name": "xiaozhi", "version": "0.9.9", "compile_time": "Jan 22 2025T20:40:23Z",

  105.                                  "idf_version": "v5.3.2-dirty",

  106.                                  "elf_sha256": "22986216df095587c42f8aeb06b239781c68ad8df80321e260556da7fcf5f522"},

  107.                  "partition_table": [{"label": "nvs", "type": 1, "subtype": 2, "address": 36864, "size": 16384},

  108.                                      {"label": "otadata", "type": 1, "subtype": 0, "address": 53248, "size": 8192},

  109.                                      {"label": "phy_init", "type": 1, "subtype": 1, "address": 61440, "size": 4096},

  110.                                      {"label": "model", "type": 1, "subtype": 130, "address": 65536, "size": 983040},

  111.                                      {"label": "storage", "type": 1, "subtype": 130, "address": 1048576,

  112.                                       "size": 1048576},

  113.                                      {"label": "factory", "type": 0, "subtype": 0, "address": 2097152, "size": 4194304},

  114.                                      {"label": "ota_0", "type": 0, "subtype": 16, "address": 6291456, "size": 4194304},

  115.                                      {"label": "ota_1", "type": 0, "subtype": 17, "address": 10485760,

  116.                                       "size": 4194304}],

  117.                  "ota": {"label": "factory"},

  118.                  "board": {"type": "bread-compact-wifi", "ssid": "mzy", "rssi": -58, "channel": 6,

  119.                            "ip": "192.168.124.38", "mac": "cc:ba:97:20:b4:bc"}}

  120. 

  121.     response = requests.post(OTA_VERSION_URL, headers=header, data=json.dumps(post_data))

  122.     print('=========================')

  123.     print(response.text)

  124.     logging.info(f"get version: {response}")

  125.     mqtt_info = response.json()['mqtt']

  126. 

  127. 

  128. def aes_ctr_encrypt(key, nonce, plaintext):

  129.     cipher = Cipher(algorithms.AES(key), modes.CTR(nonce), backend=default_backend())

  130.     encryptor = cipher.encryptor()

  131.     return encryptor.update(plaintext) + encryptor.finalize()

  132. 

  133. 

  134. def aes_ctr_decrypt(key, nonce, ciphertext):

  135.     cipher = Cipher(algorithms.AES(key), modes.CTR(nonce), backend=default_backend())

  136.     decryptor = cipher.decryptor()

  137.     plaintext = decryptor.update(ciphertext) + decryptor.finalize()

  138.     return plaintext

  139. 

  140. 

  141. def send_audio():

  142.     global aes_opus_info, udp_socket, local_sequence, listen_state, audio

  143.     key = aes_opus_info['udp']['key']

  144.     nonce = aes_opus_info['udp']['nonce']

  145.     server_ip = aes_opus_info['udp']['server']

  146.     server_port = aes_opus_info['udp']['port']

  147.     # 初始化Opus编码器

  148.     encoder = opuslib.Encoder(16000, 1, opuslib.APPLICATION_AUDIO)

  149.     # 打开麦克风流, 帧大小，应该与Opus帧大小匹配

  150.     mic = audio.open(format=pyaudio.paInt16, channels=1, rate=16000, input=True, frames_per_buffer=960)

  151.     try:

  152.         while True:

  153.             if listen_state == "stop":

  154.                 continue

  155.                 time.sleep(0.1)

  156.             # 读取音频数据

  157.             data = mic.read(960)

  158.             # 编码音频数据

  159.             encoded_data = encoder.encode(data, 960)

  160.             # 打印音频数据

  161.             # print(f"Encoded data: {len(encoded_data)}")

  162.             # nonce插入data.size local_sequence_

  163.             local_sequence += 1

  164.             new_nonce = nonce[0:4] + format(len(encoded_data), '04x') + nonce[8:24] + format(local_sequence, '08x')

  165.             # 加密数据，添加nonce

  166.             encrypt_encoded_data = aes_ctr_encrypt(bytes.fromhex(key), bytes.fromhex(new_nonce), bytes(encoded_data))

  167.             data = bytes.fromhex(new_nonce) + encrypt_encoded_data

  168.             sent = udp_socket.sendto(data, (server_ip, server_port))

  169.     except Exception as e:

  170.         print(f"send audio err: {e}")

  171.     finally:

  172.         print("send audio exit()")

  173.         local_sequence = 0

  174.         udp_socket = None

  175.         # 关闭流和PyAudio

  176.         mic.stop_stream()

  177.         mic.close()

  178. 

  179. 

  180. def recv_audio():

  181.     global aes_opus_info, udp_socket, audio

  182.     key = aes_opus_info['udp']['key']

  183.     nonce = aes_opus_info['udp']['nonce']

  184.     sample_rate = aes_opus_info['audio_params']['sample_rate']

  185.     frame_duration = aes_opus_info['audio_params']['frame_duration']

  186.     frame_num = int(frame_duration / (1000 / sample_rate))

  187.     print(f"recv audio: sample_rate -> {sample_rate}, frame_duration -> {frame_duration}, frame_num -> {frame_num}")

  188.     # 初始化Opus编码器

  189.     decoder = opuslib.Decoder(sample_rate, 1)

  190.     spk = audio.open(format=pyaudio.paInt16, channels=1, rate=sample_rate, output=True, frames_per_buffer=frame_num)

  191.     try:

  192.         while True:

  193.             data, server = udp_socket.recvfrom(4096)

  194.             # print(f"Received from server {server}: {len(data)}")

  195.             encrypt_encoded_data = data

  196.             # 解密数据,分离nonce

  197.             split_encrypt_encoded_data_nonce = encrypt_encoded_data[:16]

  198.             # 十六进制格式打印nonce

  199.             # print(f"split_encrypt_encoded_data_nonce: {split_encrypt_encoded_data_nonce.hex()}")

  200.             split_encrypt_encoded_data = encrypt_encoded_data[16:]

  201.             decrypt_data = aes_ctr_decrypt(bytes.fromhex(key),

  202.                                            split_encrypt_encoded_data_nonce,

  203.                                            split_encrypt_encoded_data)

  204.             # 解码播放音频数据

  205.             spk.write(decoder.decode(decrypt_data, frame_num))

  206.     # except BlockingIOError:

  207.     #     # 无数据时短暂休眠以减少CPU占用

  208.     #     time.sleep(0.1)

  209.     except Exception as e:

  210.         print(f"recv audio err: {e}")

  211.     finally:

  212.         udp_socket = None

  213.         spk.stop_stream()

  214.         spk.close()

  215. 

  216. def wrap_hanzi(text, first_line_width=5, other_line_width=16):

  217.     """将字符串格式化为第一行指定宽度，后续行指定宽度"""

  218.     lines = []

  219.     

  220.     # 处理第一行

  221.     if len(text) > first_line_width:

  222.         lines.append(text[:first_line_width])

  223.         remaining_text = text[first_line_width:]

  224.     else:

  225.         lines.append(text)

  226.         remaining_text = ""

  227.     

  228.     # 处理后续行

  229.     for i in range(0, len(remaining_text), other_line_width):

  230.         lines.append(remaining_text[i:i + other_line_width])

  231.     

  232.     return "\n".join(lines)

  233. def get_ascii_emotion(emotion):

  234.     """根据情绪类型返回对应的 ASCII 表情符号"""

  235.     if emotion == "happy":

  236.         return ":)"

  237.     elif emotion == "sad":

  238.         return ":("

  239.     elif emotion == "winking":

  240.         return ";)"

  241.     elif emotion == "surprised":

  242.         return ":O"

  243.     elif emotion == "angry":

  244.         return ">:(("

  245.     elif emotion == "laughing":

  246.         return ":D"

  247.     elif emotion == "cool":

  248.         return "B-)"

  249.     elif emotion == "crying":

  250.         return ":'("

  251.     elif emotion == "shy":

  252.         return "^_^"

  253.     elif emotion == "thinking":

  254.         return ":|"

  255.     elif emotion == "love":

  256.         return "<3"

  257.     elif emotion == "sleepy":

  258.         return "-.-"

  259.     elif emotion == "neutral":

  260.         return ":|"

  261.     elif emotion == "excited":

  262.         return ":D"

  263.     elif emotion == "confused":

  264.         return ":S"

  265.     else:

  266.         return ":("  # 默认表情

  267. 

  268. def on_message(client, userdata, message):

  269.     global aes_opus_info, udp_socket, tts_state, recv_audio_thread, send_audio_thread,max_chars,listen_state

  270.     msg = json.loads(message.payload)

  271.     print(f"recv msg: {msg}")

  272.     if msg['type'] == 'hello':

  273.        

  274.         aes_opus_info = msg

  275.         udp_socket.connect((msg['udp']['server'], msg['udp']['port']))

  276.          # 检查recv_audio_thread线程是否启动

  277.         if not recv_audio_thread.is_alive():

  278.             # 启动一个线程，用于接收音频数据

  279.             recv_audio_thread = threading.Thread(target=recv_audio)

  280.             recv_audio_thread.start()

  281.         else:

  282.             print("recv_audio_thread is alive")

  283.         # 检查send_audio_thread线程是否启动

  284.         if not send_audio_thread.is_alive():

  285.             # 启动一个线程，用于发送音频数据

  286.             send_audio_thread = threading.Thread(target=send_audio)

  287.             send_audio_thread.start()

  288.         else:

  289.             print("send_audio_thread is alive")

  290.     if listen_state is  None:

  291.         listen_state="hello"

  292.         # 发送start listen消息

  293.         msg = {"session_id": aes_opus_info['session_id'], "type": "listen", "state": "start", "mode": "manual"}

  294.         print(f"send start listen message: {msg}")

  295.         status_label.config(text="聆听中……")

  296.         push_mqtt_msg(msg)

  297.     if msg['type'] == 'tts':

  298.         tts_state = msg['state']

  299.     if msg['type'] == 'llm':

  300.         ascii_emotion = get_ascii_emotion(msg['emotion'])

  301.         emotion.config(text=ascii_emotion)

  302.     if msg['type'] == 'tts' and msg['state']=='start':

  303.             status_label.config(text="讲话中……")

  304.     if msg['type'] == 'tts' and msg['state']=='stop':

  305.             status_label.config(text="就绪")

  306.     if msg['type'] == 'tts' and msg['state']=='sentence_start':

  307.         

  308.         text=msg['text']

  309.         text=wrap_hanzi(text, 5,max_chars)

  310.         log_text.config(text="小智: " + text)

  311.         if msg['text'] == '举手':

  312.           handup()

  313.         if msg['text'] == '放下':

  314.           handdown()

  315.     if msg['type'] == 'stt':

  316.         

  317.         text=msg['text']

  318.         text=wrap_hanzi(text, 5,max_chars)

  319.         log_text.config(text="我: " + text)

  320.     if msg['type'] == 'goodbye' and udp_socket and msg['session_id'] == aes_opus_info['session_id']:

  321.         print(f"recv good bye msg")

  322.         aes_opus_info['session_id'] = None

  323.         log_text.config(text="")

  324.         status_label.config(text="休息中")

  325. def on_connect(client, userdata, flags, rs, pr):

  326.     subscribe_topic = mqtt_info['subscribe_topic'].split("/")[0] + '/p2p/GID_test@@@' + MAC_ADDR.replace(':', '_')

  327.     print(f"subscribe topic: {subscribe_topic}")

  328.     # 订阅主题

  329.     client.subscribe(subscribe_topic)

  330.     status_label.config(text="就绪")

  331. 

  332. def push_mqtt_msg(message):

  333.     global mqtt_info, mqttc

  334.     mqttc.publish(mqtt_info['publish_topic'], json.dumps(message))

  335. 

  336. 

  337. def listen_start():

  338.     global key_state, udp_socket, aes_opus_info, listen_state, conn_state

  339.     if key_state == "press":

  340.         return

  341.     key_state = "press"

  342.     # 判断是否需要发送hello消息

  343.     if conn_state is False or aes_opus_info['session_id'] is None:

  344.         conn_state = True

  345.         # 发送hello消息,建立udp连接

  346.         hello_msg = {"type": "hello", "version": 3, "transport": "udp",

  347.                      "audio_params": {"format": "opus", "sample_rate": 16000, "channels": 1, "frame_duration": 60}}

  348.         push_mqtt_msg(hello_msg)

  349.         print(f"send hello message: {hello_msg}")

  350.     if tts_state == "start" or tts_state == "entence_start":

  351.         # 在播放状态下发送abort消息

  352.         push_mqtt_msg({"type": "abort"})

  353.         print(f"send abort message")

  354.     if aes_opus_info['session_id'] is not None:

  355.         # 发送start listen消息

  356.         msg = {"session_id": aes_opus_info['session_id'], "type": "listen", "state": "start", "mode": "manual"}

  357.         print(f"send start listen message: {msg}")

  358.         status_label.config(text="聆听中……")

  359.         push_mqtt_msg(msg)

  360. def listen_stop():

  361.     global aes_opus_info, key_state

  362.     key_state = "release"

  363.     # 发送stop listen消息

  364.     if aes_opus_info['session_id'] is not None:

  365.         msg = {"session_id": aes_opus_info['session_id'], "type": "listen", "state": "stop"}

  366.         print(f"send stop listen message: {msg}")

  367.         push_mqtt_msg(msg)

  368. def run():

  369.     global mqtt_info, mqttc

  370.     # 获取mqtt与版本信息

  371.     get_ota_version()

  372. 

  373.     # 创建客户端实例

  374.     mqttc = mqtt.Client(callback_api_version=mqtt.CallbackAPIVersion.VERSION2, client_id=mqtt_info['client_id'])

  375.     mqttc.username_pw_set(username=mqtt_info['username'], password=mqtt_info['password'])

  376.     mqttc.tls_set(ca_certs=None, certfile=None, keyfile=None, cert_reqs=mqtt.ssl.CERT_REQUIRED,

  377.                   tls_version=mqtt.ssl.PROTOCOL_TLS, ciphers=None)

  378.     mqttc.on_connect = on_connect

  379.     mqttc.on_message = on_message

  380.     mqttc.connect(host=mqtt_info['endpoint'], port=8883)

  381. 

  382.     anjian_thread = threading.Thread(target=anjian)

  383.     anjian_thread.start()

  384.     mqttc.loop_forever()

  385. def anjian():

  386.     bs=1

  387.     while True:

  388.      if p_p24_in.read_digital()==True:

  389.          if bs==1:

  390.              listen_start()

  391.          else:

  392.              listen_stop()

  393.          bs=1-bs

  394.          time.sleep(1)

  395. 

  396. if __name__ == "__main__":

  397.     audio = pyaudio.PyAudio()

  398. 

  399.     run()

  400. 

复制代码

（演示视频中让小智根据对话情景实现“举手”、“放下”）

3.使用webrtcvad库进行人声检测，当检测到人声时，开始录音，当“静音”时，停止录音，实现不用手动操作，完成与小智连续对话。

  1. 

  2. ## pip install paho-mqtt pyaudio opuslib cryptography webrtcvad

  3. #!/usr/bin/python

  4. # -*- coding: UTF-8 -*-

  5. import json

  6. import time

  7. import requests

  8. import paho.mqtt.client as mqtt

  9. import threading

  10. import pyaudio

  11. import opuslib  # windwos平台需要将opus.dll 拷贝到C:\Windows\System32

  12. import socket

  13. from cryptography.hazmat.primitives.ciphers import Cipher, algorithms, modes

  14. from cryptography.hazmat.backends import default_backend

  15. from os import urandom

  16. import logging

  17. from pynput import keyboard as pynput_keyboard

  18. from unihiker import GUI

  19. 

  20. 

  21. from pinpong.board import Pin

  22. from pinpong.board import Board

  23. from pinpong.board import Servo

  24. import webrtcvad

  25. vad = webrtcvad.Vad(3)

  26. # 初始化行空板硬件接口

  27. Board().begin()

  28. gui=GUI()

  29. gui.clear()

  30. #舵机引脚及按键引脚设置

  31. pin2 = Pin(Pin.D21)

  32. pin3 = Pin(Pin.D22)

  33. servo1 = Servo(pin2)

  34. servo2 = Servo(pin3)

  35. p_p24_in=Pin(Pin.P24, Pin.IN)

  36. #显示内容设置

  37. fontSize=20

  38. max_lines = 16

  39. max_chars=8

  40. # 图形界面元素

  41. status_label = gui.draw_text(x=80, y=10, text='初始化中...', color='red')

  42. log_text = gui.draw_text(x=10, y=100, text='', font_size=fontSize,color='blue')

  43. emotion = gui.draw_text(x=110, y=50, text='', font_size=fontSize,color='green')

  44. OTA_VERSION_URL = 'https://api.tenclass.net/xiaozhi/ota/'

  45. MAC_ADDR = '7a:da:6b:5c:76:50'

  46. # {"mqtt":{"endpoint":"post-cn-apg3xckag01.mqtt.aliyuncs.com","client_id":"GID_test@@@cc_ba_97_20_b4_bc",

  47. # "username":"Signature|LTAI5tF8J3CrdWmRiuTjxHbF|post-cn-apg3xckag01","password":"0mrkMFELXKyelhuYy2FpGDeCigU=",

  48. # "publish_topic":"device-server","subscribe_topic":"devices"},"firmware":{"version":"0.9.9","url":""}}

  49. mqtt_info = {}

  50. aes_opus_info = {"type": "hello", "version": 3, "transport": "udp",

  51.                  "udp": {"server": "120.24.160.13", "port": 8884, "encryption": "aes-128-ctr",

  52.                          "key": "263094c3aa28cb42f3965a1020cb21a7", "nonce": "01000000ccba9720b4bc268100000000"},

  53.                  "audio_params": {"format": "opus", "sample_rate": 24000, "channels": 1, "frame_duration": 60},

  54.                  "session_id": None}

  55. 

  56. iot_msg = {"session_id": "635aa42d", "type": "iot",

  57.            "descriptors": [{"name": "Speaker", "description": "当前 AI 机器人的扬声器",

  58.                             "properties": {"volume": {"description": "当前音量值", "type": "number"}},

  59.                             "methods": {"SetVolume": {"description": "设置音量",

  60.                                                       "parameters": {

  61.                                                           "volume": {"description": "0到100之间的整数", "type": "number"}

  62.                                                       }

  63.                                                       }

  64.                                         }

  65.                             },

  66.                            {"name": "Lamp", "description": "一个测试用的灯",

  67.                             "properties": {"power": {"description": "灯是否打开", "type": "boolean"}},

  68.                             "methods": {"TurnOn": {"description": "打开灯", "parameters": {}},

  69.                                         "TurnOff": {"description": "关闭灯", "parameters": {}}

  70.                                         }

  71.                             }

  72.                            ]

  73.            }

  74. iot_status_msg = {"session_id": "635aa42d", "type": "iot", "states": [

  75.     {"name": "Speaker", "state": {"volume": 50}}, {"name": "Lamp", "state": {"power": False}}]}

  76. goodbye_msg = {"session_id": "b23ebfe9", "type": "goodbye"}

  77. local_sequence = 0

  78. listen_state = None

  79. tts_state = None

  80. key_state = None

  81. audio = None

  82. vaddata=None

  83. udp_socket = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)

  84. # udp_socket.setblocking(False)

  85. conn_state = False

  86. recv_audio_thread = threading.Thread()

  87. send_audio_thread = threading.Thread()

  88. anjian_thread = threading.Thread()

  89. mqttc = None

  90. # 抬手 放手函数

  91. def handup():

  92.     servo1.write_angle(170)

  93.     servo2.write_angle(10)

  94. def handdown():

  95.     servo1.write_angle(10)

  96.     servo2.write_angle(170)

  97. 

  98. def get_ota_version():

  99.     global mqtt_info

  100.     header = {

  101.         'Device-Id': MAC_ADDR,

  102.         'Content-Type': 'application/json'

  103.     }

  104.     post_data = {"flash_size": 16777216, "minimum_free_heap_size": 8318916, "mac_address": f"{MAC_ADDR}",

  105.                  "chip_model_name": "esp32s3", "chip_info": {"model": 9, "cores": 2, "revision": 2, "features": 18},

  106.                  "application": {"name": "xiaozhi", "version": "0.9.9", "compile_time": "Jan 22 2025T20:40:23Z",

  107.                                  "idf_version": "v5.3.2-dirty",

  108.                                  "elf_sha256": "22986216df095587c42f8aeb06b239781c68ad8df80321e260556da7fcf5f522"},

  109.                  "partition_table": [{"label": "nvs", "type": 1, "subtype": 2, "address": 36864, "size": 16384},

  110.                                      {"label": "otadata", "type": 1, "subtype": 0, "address": 53248, "size": 8192},

  111.                                      {"label": "phy_init", "type": 1, "subtype": 1, "address": 61440, "size": 4096},

  112.                                      {"label": "model", "type": 1, "subtype": 130, "address": 65536, "size": 983040},

  113.                                      {"label": "storage", "type": 1, "subtype": 130, "address": 1048576,

  114.                                       "size": 1048576},

  115.                                      {"label": "factory", "type": 0, "subtype": 0, "address": 2097152, "size": 4194304},

  116.                                      {"label": "ota_0", "type": 0, "subtype": 16, "address": 6291456, "size": 4194304},

  117.                                      {"label": "ota_1", "type": 0, "subtype": 17, "address": 10485760,

  118.                                       "size": 4194304}],

  119.                  "ota": {"label": "factory"},

  120.                  "board": {"type": "bread-compact-wifi", "ssid": "mzy", "rssi": -58, "channel": 6,

  121.                            "ip": "192.168.124.38", "mac": "cc:ba:97:20:b4:bc"}}

  122. 

  123.     response = requests.post(OTA_VERSION_URL, headers=header, data=json.dumps(post_data))

  124.     print('=========================')

  125.     print(response.text)

  126.     logging.info(f"get version: {response}")

  127.     mqtt_info = response.json()['mqtt']

  128. 

  129. 

  130. def aes_ctr_encrypt(key, nonce, plaintext):

  131.     cipher = Cipher(algorithms.AES(key), modes.CTR(nonce), backend=default_backend())

  132.     encryptor = cipher.encryptor()

  133.     return encryptor.update(plaintext) + encryptor.finalize()

  134. 

  135. 

  136. def aes_ctr_decrypt(key, nonce, ciphertext):

  137.     cipher = Cipher(algorithms.AES(key), modes.CTR(nonce), backend=default_backend())

  138.     decryptor = cipher.decryptor()

  139.     plaintext = decryptor.update(ciphertext) + decryptor.finalize()

  140.     return plaintext

  141. 

  142. 

  143. def send_audio():

  144.     global aes_opus_info, udp_socket, local_sequence, listen_state, audio,vaddata

  145.     key = aes_opus_info['udp']['key']

  146.     nonce = aes_opus_info['udp']['nonce']

  147.     server_ip = aes_opus_info['udp']['server']

  148.     server_port = aes_opus_info['udp']['port']

  149.     # 初始化Opus编码器

  150.     encoder = opuslib.Encoder(16000, 1, opuslib.APPLICATION_AUDIO)

  151.     # 打开麦克风流, 帧大小，应该与Opus帧大小匹配

  152.     mic = audio.open(format=pyaudio.paInt16, channels=1, rate=16000, input=True, frames_per_buffer=480)

  153.     try:

  154.         while True:

  155.             if listen_state == "stop":

  156.                 continue

  157.                 time.sleep(0.1)

  158.             # 读取音频数据

  159.             vaddata = mic.read(480)

  160.             data=vaddata+mic.read(480)

  161.             # 编码音频数据

  162.             encoded_data = encoder.encode(data, 960)

  163.             # 打印音频数据

  164.             #print(f"Encoded data: {len(encoded_data)}")

  165.             # nonce插入data.size local_sequence_

  166.             local_sequence += 1

  167.             new_nonce = nonce[0:4] + format(len(encoded_data), '04x') + nonce[8:24] + format(local_sequence, '08x')

  168.             # 加密数据，添加nonce

  169.             encrypt_encoded_data = aes_ctr_encrypt(bytes.fromhex(key), bytes.fromhex(new_nonce), bytes(encoded_data))

  170.             data = bytes.fromhex(new_nonce) + encrypt_encoded_data

  171.             sent = udp_socket.sendto(data, (server_ip, server_port))

  172.     except Exception as e:

  173.         print(f"send audio err: {e}")

  174.     finally:

  175.         print("send audio exit()")

  176.         local_sequence = 0

  177.         udp_socket = None

  178.         # 关闭流和PyAudio

  179.         mic.stop_stream()

  180.         mic.close()

  181. 

  182. 

  183. def recv_audio():

  184.     global aes_opus_info, udp_socket, audio,speekstoptime

  185.     key = aes_opus_info['udp']['key']

  186.     nonce = aes_opus_info['udp']['nonce']

  187.     sample_rate = aes_opus_info['audio_params']['sample_rate']

  188.     frame_duration = aes_opus_info['audio_params']['frame_duration']

  189.     frame_num = int(frame_duration / (1000 / sample_rate))

  190.     print(f"recv audio: sample_rate -> {sample_rate}, frame_duration -> {frame_duration}, frame_num -> {frame_num}")

  191.     # 初始化Opus编码器

  192.     decoder = opuslib.Decoder(sample_rate, 1)

  193.     spk = audio.open(format=pyaudio.paInt16, channels=1, rate=sample_rate, output=True, frames_per_buffer=frame_num)

  194.     try:

  195.         while True:

  196.             

  197.             data, server = udp_socket.recvfrom(4096)

  198.             print(f"Received from server {server}: {len(data)}")

  199.             encrypt_encoded_data = data

  200.             # 解密数据,分离nonce

  201.             split_encrypt_encoded_data_nonce = encrypt_encoded_data[:16]

  202.             # 十六进制格式打印nonce

  203.             # print(f"split_encrypt_encoded_data_nonce: {split_encrypt_encoded_data_nonce.hex()}")

  204.             split_encrypt_encoded_data = encrypt_encoded_data[16:]

  205.             decrypt_data = aes_ctr_decrypt(bytes.fromhex(key),

  206.                                            split_encrypt_encoded_data_nonce,

  207.                                            split_encrypt_encoded_data)

  208.             # 解码播放音频数据

  209.             spk.write(decoder.decode(decrypt_data, frame_num))

  210.           

  211.             speekstoptime=time.time()

  212.             

  213.     # except BlockingIOError:

  214.     #     # 无数据时短暂休眠以减少CPU占用

  215.     #     time.sleep(0.1)

  216.     except Exception as e:

  217.         print(f"recv audio err: {e}")

  218.     finally:

  219.         udp_socket = None

  220.         spk.stop_stream()

  221.         spk.close()

  222. 

  223. def wrap_hanzi(text, first_line_width=5, other_line_width=16):

  224.     """将字符串格式化为第一行指定宽度，后续行指定宽度"""

  225.     lines = []

  226.     

  227.     # 处理第一行

  228.     if len(text) > first_line_width:

  229.         lines.append(text[:first_line_width])

  230.         remaining_text = text[first_line_width:]

  231.     else:

  232.         lines.append(text)

  233.         remaining_text = ""

  234.     

  235.     # 处理后续行

  236.     for i in range(0, len(remaining_text), other_line_width):

  237.         lines.append(remaining_text[i:i + other_line_width])

  238.     

  239.     return "\n".join(lines)

  240. def get_ascii_emotion(emotion):

  241.     """根据情绪类型返回对应的 ASCII 表情符号"""

  242.     if emotion == "happy":

  243.         return ":)"

  244.     elif emotion == "sad":

  245.         return ":("

  246.     elif emotion == "winking":

  247.         return ";)"

  248.     elif emotion == "surprised":

  249.         return ":O"

  250.     elif emotion == "angry":

  251.         return ">:(("

  252.     elif emotion == "laughing":

  253.         return ":D"

  254.     elif emotion == "cool":

  255.         return "B-)"

  256.     elif emotion == "crying":

  257.         return ":'("

  258.     elif emotion == "shy":

  259.         return "^_^"

  260.     elif emotion == "thinking":

  261.         return ":|"

  262.     elif emotion == "love":

  263.         return "<3"

  264.     elif emotion == "sleepy":

  265.         return "-.-"

  266.     elif emotion == "neutral":

  267.         return ":|"

  268.     elif emotion == "excited":

  269.         return ":D"

  270.     elif emotion == "confused":

  271.         return ":S"

  272.     else:

  273.         return ":("  # 默认表情

  274. 

  275. def on_message(client, userdata, message):

  276.     global aes_opus_info, udp_socket, tts_state, recv_audio_thread, send_audio_thread,max_chars,listen_state,speekstate,speekstoptime

  277.     msg = json.loads(message.payload)

  278.     print(f"recv msg: {msg}")

  279.     if msg['type'] == 'hello':

  280.        

  281.         aes_opus_info = msg

  282.         udp_socket.connect((msg['udp']['server'], msg['udp']['port']))

  283.          # 检查recv_audio_thread线程是否启动

  284.         if not recv_audio_thread.is_alive():

  285.             # 启动一个线程，用于接收音频数据

  286.             recv_audio_thread = threading.Thread(target=recv_audio)

  287.             recv_audio_thread.start()

  288.         else:

  289.             print("recv_audio_thread is alive")

  290.         # 检查send_audio_thread线程是否启动

  291.         if not send_audio_thread.is_alive():

  292.             # 启动一个线程，用于发送音频数据

  293.             send_audio_thread = threading.Thread(target=send_audio)

  294.             send_audio_thread.start()

  295.         else:

  296.             print("send_audio_thread is alive")

  297.     if listen_state is  None:

  298.         listen_state="hello"

  299.         # 发送start listen消息

  300.         msg = {"session_id": aes_opus_info['session_id'], "type": "listen", "state": "start", "mode": "manual"}

  301.         print(f"send start listen message: {msg}")

  302.         status_label.config(text="聆听中……")

  303.         push_mqtt_msg(msg)

  304.     if msg['type'] == 'tts':

  305.         tts_state = msg['state']

  306.     if msg['type'] == 'llm':

  307.         ascii_emotion = get_ascii_emotion(msg['emotion'])

  308.         emotion.config(text=ascii_emotion)

  309.     if msg['type'] == 'tts' and msg['state']=='start':

  310.             status_label.config(text="讲话中……")

  311.             speekstate=0

  312.     if msg['type'] == 'tts' and msg['state']=='stop':

  313.             status_label.config(text="就绪")

  314.             speekstate=1

  315.     if msg['type'] == 'tts' and msg['state']=='sentence_start':

  316.         

  317.         text=msg['text']

  318.         text=wrap_hanzi(text, 5,max_chars)

  319.         log_text.config(text="小智: " + text)

  320.         if msg['text'] == '举手':

  321.           handup()

  322.         if msg['text'] == '放下':

  323.           handdown()

  324.     if msg['type'] == 'stt':

  325.         

  326.         text=msg['text']

  327.         text=wrap_hanzi(text, 5,max_chars)

  328.         log_text.config(text="我: " + text)

  329.     if msg['type'] == 'goodbye' and udp_socket and msg['session_id'] == aes_opus_info['session_id']:

  330.         print(f"recv good bye msg")

  331.         aes_opus_info['session_id'] = None

  332.         listen_state= None

  333.         log_text.config(text="")

  334.         status_label.config(text="休息中")

  335.         # 关闭 UDP 连接

  336.         if udp_socket:

  337.                 udp_socket.close()

  338.                 udp_socket = None

  339.         for thread in (recv_audio_thread, send_audio_thread):

  340.             if thread and thread.is_alive():

  341.                 thread.join(timeout=1)

  342. 

  343. def on_connect(client, userdata, flags, rs, pr):

  344.     subscribe_topic = mqtt_info['subscribe_topic'].split("/")[0] + '/p2p/GID_test@@@' + MAC_ADDR.replace(':', '_')

  345.     print(f"subscribe topic: {subscribe_topic}")

  346.     # 订阅主题

  347.     client.subscribe(subscribe_topic)

  348.     status_label.config(text="就绪")

  349. 

  350. def push_mqtt_msg(message):

  351.     global mqtt_info, mqttc

  352.     mqttc.publish(mqtt_info['publish_topic'], json.dumps(message))

  353. 

  354. 

  355. def listen_start():

  356.     global key_state, udp_socket, aes_opus_info, listen_state, conn_state

  357.     if key_state == "press":

  358.         return

  359.     key_state = "press"

  360.     

  361.     # 判断是否需要发送hello消息

  362.     if conn_state is False or aes_opus_info['session_id'] is None:

  363.         # 清理旧连接

  364.         if udp_socket:

  365.                 udp_socket.close()

  366.                 udp_socket = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)

  367. 

  368.         conn_state = True

  369.         # 发送hello消息,建立udp连接

  370.         hello_msg = {"type": "hello", "version": 3, "transport": "udp",

  371.                      "audio_params": {"format": "opus", "sample_rate": 16000, "channels": 1, "frame_duration": 60}}

  372.         push_mqtt_msg(hello_msg)

  373.         print(f"send hello message: {hello_msg}")

  374.     if tts_state == "start" or tts_state == "entence_start":

  375.         # 在播放状态下发送abort消息

  376.         push_mqtt_msg({"type": "abort"})

  377.         print(f"send abort message")

  378.     if aes_opus_info['session_id'] is not None:

  379.         # 发送start listen消息

  380.         msg = {"session_id": aes_opus_info['session_id'], "type": "listen", "state": "start", "mode": "manual"}

  381.         print(f"send start listen message: {msg}")

  382.         

  383.         push_mqtt_msg(msg)

  384. def listen_stop():

  385.     global aes_opus_info, key_state

  386.     key_state = "release"

  387.     # 发送stop listen消息

  388.     if aes_opus_info['session_id'] is not None:

  389.         msg = {"session_id": aes_opus_info['session_id'], "type": "listen", "state": "stop"}

  390.         print(f"send stop listen message: {msg}")

  391.         push_mqtt_msg(msg)

  392. def run():

  393.     global mqtt_info, mqttc

  394.     # 获取mqtt与版本信息

  395.     get_ota_version()

  396. 

  397.     # 创建客户端实例

  398.     mqttc = mqtt.Client(callback_api_version=mqtt.CallbackAPIVersion.VERSION2, client_id=mqtt_info['client_id'])

  399.     mqttc.username_pw_set(username=mqtt_info['username'], password=mqtt_info['password'])

  400.     mqttc.tls_set(ca_certs=None, certfile=None, keyfile=None, cert_reqs=mqtt.ssl.CERT_REQUIRED,

  401.                   tls_version=mqtt.ssl.PROTOCOL_TLS, ciphers=None)

  402.     mqttc.on_connect = on_connect

  403.     mqttc.on_message = on_message

  404.     mqttc.connect(host=mqtt_info['endpoint'], port=8883)

  405. 

  406.     anjian_thread = threading.Thread(target=anjian)

  407.     anjian_thread.start()

  408.     mqttc.loop_forever()

  409. speekstate=0

  410. speekstoptime=0

  411. def anjian():

  412.     global vaddata,speekstate,speekstoptime

  413.     bs=0

  414.     last_voice_time=0

  415.    

  416.     while True:

  417.          if vaddata is not None:

  418.             

  419.             if vad.is_speech(vaddata, 16000) and speekstate==1 and time.time()-speekstoptime>1.5:

  420.                 status_label.config(text="聆听中……")

  421.                 print(".",end="")

  422.                 bs=1

  423.                 listen_start()

  424.                 last_voice_time = time.time()

  425.             elif time.time() - last_voice_time >1.5 and bs==1:

  426.                 

  427.                 bs=0

  428.                 listen_stop()

  429.          else:

  430.                 time.sleep(2)

  431.                 bs=1

  432.                 listen_start()

  433.                 last_voice_time = time.time()

  434. if __name__ == "__main__":

  435.     audio = pyaudio.PyAudio()

  436. 

  437.     run()

  438. 

复制代码

（webrtcvad库检测效果一般，我使用的是模式 3：非常激进模式，灵敏度最高。）


4.借助行空板上的触摸屏，点击“按钮”，与小智对话
行空板M10 小智AI 聊天机器人图3


行空板M10 小智AI 聊天机器人图4

开源网址：https://github.com/Huang-junsen/py-xiaozhi.git，主要修改了“mqtt_client.py”、“gui.py”。


修改的代码文件：下载附件gui及mqtt_client.zip

回复

使用道具 举报

高级模式
B Color Image Link Quote Code Smilies
您需要登录后才可以回帖 登录 | 立即注册

本版积分规则

为本项目制作心愿单
购买心愿单
心愿单 编辑
[[wsData.name]]

硬件清单

  • [[d.name]]
btnicon
我也要做!
点击进入购买页面
撰写新帖
关于楼主
勋章墙
活跃会员 宣传大使 创客造 编辑选择奖 志“童”道合 编辑选择奖 编辑选择奖

楼主的其它帖子

background 创造力兑换 background 心愿单 background 创客导航
上海智位机器人股份有限公司 沪ICP备09038501号-4 备案 沪公网安备31011502402448

© 2013-2025 Comsenz Inc. Powered by Discuz! X3.4 Licensed

mail