multi-speaker multi-track audio

[Tharun1718]

Hi,

I am trying to transcribe live stereo audio to mono audio and transcribe them, is there any recommended methods to implement this, I have tried converting stereo to mono and my result is very inaccurate.

Thanks in advance for the help

[Gldkslfmsd]

what is in your stereo?
yes, converting to mono sounds best.
Or one whisper streaming per track?

[Tharun1718]

hey thanks, actually i am trying to stream conversation between two people(agent and a customer), Can you suggest me some guides where I can study more on this, any suggestion would be great.

[Gldkslfmsd]

if you have the voices in separate tracks, it's good, you don't need diarization (good topic to know about).

Then you probably need voice activity controller that sends the speaking part track into WhisperStreaming. You can modify this class:

whisper_streaming/whisper_online.py

Line 518 in 0a00254

class VACOnlineASRProcessor(OnlineASRProcessor):

You can use multiple Silero VAD Iterator objects: https://github.com/ufal/whisper_streaming/blob/main/silero_vad.py , one for each track, to control sending the voice to the OnlineASRProcessor . When it returns the output, wrap it with info who spoke that.

In any way, you should make sure that the context of the previous turns is not cleared. Use finalize() but do not clear the HypothesisBuffer.

[Tharun1718]

Thank you for you input, will work on it