Digital Human / TTS

The Digital Human module of NiteMoon AI Platform provides multi-engine TTS (Text-to-Speech) support, enabling high-quality voice synthesis and digital human interaction through a WebSocket bidirectional communication protocol.

Key Features

• 3 TTS engines: Volcengine, GPT-SoVITS, MiMo
• WebSocket bidirectional protocol: Real-time voice interaction
• Custom binary protocol: Supports compression and serialization
• Voice file generation: Supports multiple audio formats
• Digital human configuration management: Flexible digital human character configuration

TTS Engines

Volcengine (ByteDance)

Enterprise-grade TTS service with excellent audio quality and multiple voice options.

tts:
  providers:
    volcengine:
      app-id: ${VOLCENGINE_APP_ID}
      access-token: ${VOLCENGINE_ACCESS_TOKEN}
      cluster: volcengine_streaming

Features:

• High-quality voice synthesis
• Multiple voice options
• Real-time streaming synthesis
• Chinese-optimized

GPT-SoVITS

Open-source voice cloning solution supporting few-shot voice cloning.

tts:
  providers:
    gpt-sovits:
      base-url: http://localhost:9880
      model-path: /path/to/model

Features:

• Voice cloning
• Emotional expression
• Open source and free
• Local deployment

MiMo

Lightweight TTS engine, suitable for embedded scenarios.

tts:
  providers:
    mimo:
      base-url: http://localhost:8080

Features:

• Lightweight
• Low latency
• Minimal resource usage

WebSocket Protocol

Protocol Format

public class TTSMessage {
    private MessageType type;      // Message type
    private byte[] data;           // Audio data
    private Map<String, Object> metadata;  // Metadata
}

public enum MessageType {
    START,      // Start synthesis
    AUDIO,      // Audio data
    END,        // End synthesis
    ERROR,      // Error
    CONTROL     // Control message
}

Binary Protocol

+--------+--------+--------+--------+
| Type   | Flags  | Length (16bit)   |
+--------+--------+--------+--------+
| Payload (variable length)          |
+------------------------------------+

• Type: Message type (1 byte)
• Flags: Flag bits (1 byte)
• Length: Payload length (2 bytes)
• Payload: Actual data

Use Cases

Intelligent Customer Service

// Create digital human
DigitalHuman human = new DigitalHuman();
human.setName("Assistant");
human.setVoice("zh-CN-Neural");
human.setTtsProvider("volcengine");
digitalHumanService.save(human);

// Voice interaction
WebSocketSession session = ttsService.connect(human.getId());
session.sendText("Hello, how can I help you?");
byte[] audio = session.receiveAudio();

Voice Assistant

// Configure voice assistant
VoiceAssistant assistant = VoiceAssistant.builder()
    .name("Xiaoyue")
    .voice("zh-CN-Xiaoyan")
    .speed(1.0)
    .pitch(1.0)
    .build();

// Voice synthesis
byte[] audio = ttsService.synthesize(
    assistant.getVoice(),
    "The weather is really nice today"
);

Content Narration

// Batch voice synthesis
List<byte[]> audioSegments = ttsService.synthesizeBatch(
    voice,
    paragraphs
);

// Merge audio
byte[] fullAudio = audioService.merge(audioSegments);

Digital Human Configuration

Digital Human Information

@Entity
public class DigitalHuman {
    @Id
    private Long id;
    private String name;           // Name
    private String avatar;         // Avatar
    private String voice;          // Voice
    private String ttsProvider;    // TTS provider
    private Double speed;          // Speed
    private Double pitch;          // Pitch
    private Map<String, Object> config;  // Additional configuration
}

Digital Human Configuration

@Entity
public class HumanConfig {
    @Id
    private Long id;
    private Long humanId;          // Digital human ID
    private String prompt;         // System prompt
    private String personality;    // Personality description
    private List<String> skills;   // Skill list
    private Map<String, Object> settings;  // Settings
}

Code Examples

TTS Service Interface

public interface TtsService {
    
    // Connect to TTS service
    WebSocketSession connect(String provider);
    
    // Synthesize speech
    byte[] synthesize(String voice, String text);
    
    // Streaming synthesis
    Flux<byte[]> synthesizeStream(String voice, String text);
    
    // Get available voices
    List<Voice> getVoices(String provider);
}

Volcengine Implementation

@Service
public class VolcengineTtsService implements TtsService {
    
    @Override
    public byte[] synthesize(String voice, String text) {
        // Build request
        VolcengineTtsRequest request = VolcengineTtsRequest.builder()
            .appId(appId)
            .voice(voice)
            .text(text)
            .build();
        
        // Send request
        VolcengineTtsResponse response = client.synthesize(request);
        
        return response.getAudioData();
    }
    
    @Override
    public Flux<byte[]> synthesizeStream(String voice, String text) {
        return Flux.create(sink -> {
            WebSocketSession session = connect("volcengine");
            
            session.sendText(buildStartMessage(voice));
            session.sendText(buildTextMessage(text));
            
            session.receive()
                .filter(msg -> msg.getType() == MessageType.AUDIO)
                .subscribe(
                    msg -> sink.next(msg.getData()),
                    error -> sink.error(error),
                    () -> sink.complete()
                );
        });
    }
}

Digital Human Management

@Service
public class DigitalHumanService {
    
    public DigitalHuman create(String name, String voice, String provider) {
        DigitalHuman human = new DigitalHuman();
        human.setName(name);
        human.setVoice(voice);
        human.setTtsProvider(provider);
        return humanRepository.save(human);
    }
    
    public byte[] speak(Long humanId, String text) {
        DigitalHuman human = humanRepository.findById(humanId);
        TtsService ttsService = ttsProviderFactory.getProvider(
            human.getTtsProvider()
        );
        
        return ttsService.synthesize(human.getVoice(), text);
    }
}

Best Practices

1. Choose the right engine: Select the TTS engine based on scenario requirements
2. Optimize audio quality: Adjust speed, pitch, and other parameters
3. Cache audio: Cache frequently used voice outputs
4. Monitor latency: Monitor TTS latency in real time
5. Error handling: Handle network interruptions, synthesis failures, and other exceptions

Related Links

• Multi-Model LLM Integration
• Visual Workflow Engine
• Quick Start