EXPERTS ONLY ! Voice chat with multiple clients

[CVMichael]

Honestly, I don't expect anyone would be able to solve my problem (because of it's difficulty), but I'm gonna try anyways...

I made a voice chat program that works well. It's using ACM to encode to MP3, and the reverse on receiving side. So server and client encodes and decodes simultaniously since sound has to go both ways.
The program is using TCP for general messages, and UDP for sound transfer.
The program sends sound data only when the sound goes over a threshold of 5%, so in other words, the program does not send anything until you make a noise (you talk).

This works fine for a 1 to 1 connection.

What I want is to make the voice chat program connect directly to each client, so for example if there are 3 people talking to each other, I want each one to be connected to the other 2, and so on for more clients.

The problem is handling the sound for multiple users.

Decoding the individual sounds from diferent clients is no problem, the problem is once it becomes wave data, I don't know how to mix the sounds.

I know this sounds ironic since I made a sound tutorial, but this is different, the sound is a stream, and is not always there....

What I mean is, sometimes you have one client talking sometimes to have 2 at the same time (or more), where the sound might or might not overlap at diferent point in time.

I don't know how to keep track of buffer positions, and also how to do the mixing of the sounds.

From what I know the formula for mixing sounds is:
mixed_sound = (sound_1 + sound_2) / 2
in other words, making a average of the 2 sounds, and if there are 3 sounds, then an average of the 3, and so on...

Lets say client A starts talking and he talks to 10 seconds, client B starts 3 seconds after client A started and he talks for 2 seconds.

This means that from secod 3 to 5, it will have to mix the sounds A and B, this also means that client A will sound at lower volume while B is talking (the rule of mixing)

I was thinking not to divide the sounds by number of sounds, just add them together, ie. mixed_sound = sound_1 + sound_2 + .... etc

I know that regular chat programs don't allow you to talk at all when one person is talking, so only one person can voice chat at one point in time, but I want to break that barrier.

Anyways, this is becomming a long post, what do you guys think so far ?

[DigiRev]

I've always used WAV streaming in the past, never MP3, so I don't know about that.

All I know is by using WaveStream you can send the data straight to the sound device and it will handle playback.

Also, most chat programs have a voice server. Clients don't really all connect directly to each other unless it's 1 on 1 (in a private message for example).

Voice servers are always under a high load, but I imagine it would be a lot better with MP3 encoding.

Sorry I can't really be of any help.

[skankerer]

Yes, I think a server would be the easiest way to go sir.

[CVMichael]

I don't have a server, so each client program is a server also...
I'm more interested on how to do it ? (what I want to do)

As I was saying, I don't know how to handle so many buffers for each clients, and another buffer for the wave data.

I have to mix all the buffers somehow, I just don't know what is the best way.

I have to keep track of the position in each buffer, and mix the sounds, but it is so dificult, I can't make a "full picture" in my mind on how to do it.

When I do programming I usually imagine how the program will be, and how to structure the code even before I write the first line of code.

But this time, it is so difficult that I just can't put all in my mind at the same time...

And out of the many choices (ways) to write the code, i don't know wich is the best.

I already had an attempt: I made an instance of a DirectX Sound for each client, but DirectX has a bug in it... the events that are supposed to fire in one instance, fire in ALL the instances, and the other way around... this makes such a big mess.

Imagine that even an external application that is using DirectX Sound (like WinAmp), when playing sound in WinAmp, it will fire events in my program even when the sound is not playing in my application !!!

When making multiple instances it basically fires events ALL the time regardless wheather it is supposed to fire events or not...

Anyways... what I just mentioned was one of the ways to do this, but it does not work... the only way I can think of is to mix the sounds by myself, and not rely on DirectX Sound. I will have only one instance of DirectX Sound, to play the final (mixed) sound.

Anyways... any help is appreciated
Thanks for trying DigiRev...

[DigiRev]

Well, when two or more sounds are played together, they're averaged together (or "summed") like you said.

Sound1 + Sound2 + Sound3 + Sound4 / NumberOfSounds

How are you encoding the WAV data to MP3?

You mentioned DirectSound...I would imagine DirectSound would handle the summing itself.

But you said you're having problems with DirectSound, so it looks like you will need another option.

I honestly have no idea where to even begin with audio encoding/buffering. Only thing I understand about this is Winsock.

Thanks for trying DigiRev...

You're welcome. Wish I knew more about audio coding.

Good luck.

[Andrew G]

Not sure if this is much help....
http://forums.devx.com/archive/index.php/t-144806.html

[CVMichael]

Sound1 + Sound2 + Sound3 + Sound4 / NumberOfSounds

Yea, that's what I want to do, the only problem is that the number of sounds change. What I mean is that no always everyone talks at the same time... well mostly it should be one at one time, and sometimes 2, and rarelly 3 at the same time. So then the formula should change by the number of people talking.
But let's say that user A starts talking, so then there is no mixing... then user B starts talking, then it should mix A + B / 2.... and if user C starts talking also, then it should be A + B + C / 3, and so on...

Here's an example:


Each color representing a user, as you can see, it could be any combination... not all of then start and stop at the same time...

But the problem is that every time I mix sounds, the sounds are actually played at lower volume since I'm doing the division by 2 or 3... but then again... if I don't divide, the sounds will become too loud all together...

How are you encoding the WAV data to MP3?

I'm using ACM (Audio Compression Manager), Lame ACM encoder

You mentioned DirectSound...I would imagine DirectSound would handle the summing itself.

DirectSound can mix with no problems with simple static buffers, but not for streamming.
I'm using a DirectSound like this: http://www.vbforums.com/showthread.php?t=232243 , except I don't get the data from a file, I get it from winsock, then decode into a buffer, and then it goes in the DirectSound buffer.
As you notice the DirectSound buffer has events, when I make more instances of it, the events overlap, instead of being independent for each instance.
The events that are supposed to fire ONLY in instance B, fire in instance A, and vice versa... this makes such a big mess

So, the option is that I have to use only ONE DirectSound instance, and mix the sounds on my own

[Arne Elster]

DirectSound can mix with no problems with simple static buffers, but not for streamming.
I'm using a DirectSound like this: http://www.vbforums.com/showthread.php?t=232243 , except I don't get the data from a file, I get it from winsock, then decode into a buffer, and then it goes in the DirectSound buffer.
As you notice the DirectSound buffer has events, when I make more instances of it, the events overlap, instead of being independent for each instance.
The events that are supposed to fire ONLY in instance B, fire in instance A, and vice versa... this makes such a big mess

So, the option is that I have to use only ONE DirectSound instance, and mix the sounds on my own

You shouldn't ever use the DirectSound events.
Use a timer instead, and poll the buffer's play cursor.
When it passed a specific position, fill the buffer.

[smUX]

CV, I'm no expert but I thought I'd put in my tuppence worth...you speak above about the mixing problem with people stopping talking meaning there's less streams to merge...how about this?

Everyone who is connected is an active stream WHETHER they talk or not. If threshold for any person goes below 5% a *null* (or quiet) audio value is used BUT their stream is still a part of the data stream. This way there's no need to worry about fluctuating inputs for each frame of audio data. It may result in higher data, I'm not sure.

And a server is a *MUST* with this...perhaps with one user acting as a server and possibly with other users acting as backup relays who get their data from the main server, but you need to have as few sources as possible for the data otherwise there'll possibly be lag issues. If it's only a chat with a max of 10 people, it shouldn't be much of a problem :-)

[DigiRev]

CV, I'm no expert but I thought I'd put in my tuppence worth...you speak above about the mixing problem with people stopping talking meaning there's less streams to merge...how about this?

Everyone who is connected is an active stream WHETHER they talk or not. If threshold for any person goes below 5% a *null* (or quiet) audio value is used BUT their stream is still a part of the data stream. This way there's no need to worry about fluctuating inputs for each frame of audio data. It may result in higher data, I'm not sure.

And a server is a *MUST* with this...perhaps with one user acting as a server and possibly with other users acting as backup relays who get their data from the main server, but you need to have as few sources as possible for the data otherwise there'll possibly be lag issues. If it's only a chat with a max of 10 people, it shouldn't be much of a problem :-)

That's a good idea, but silenced audio is just as large as audio with sounds and would be a waste of bandwidth.

That's how it is with WAV data anyway, with MP3 maybe it's different.

But the problem is that every time I mix sounds, the sounds are actually played at lower volume since I'm doing the division by 2 or 3

Is the overall volume lower or just each person's volume?

The overall (average) volume should stay the same.

Maybe there is a little more to it than just Sound1 + Sound2 + Sound3 / NumberOfSounds?

[smUX]

That's a good idea, but silenced audio is just as large as audio with sounds and would be a waste of bandwidth.

That's how it is with WAV data anyway, with MP3 maybe it's different.

I'd assume with MP3 it would be vastly different (because of the fact that MP3 cuts out imperceptible audio and the whole thing is imperceptible :-)), but I am also thinking that there would be little or no bandwidth issues...I thought of this as I was writing it but didn't say so. Let's say 3 people were talking and sending in 40k blocks. Person 3 doesn't say anything and instead of a 40k block a 1 byte block is sent saying "N" for null (or something) and the person receiving this block loads a default "quiet" block and uses that in the merge. Also, for the data going back *out* they'd all be merged into one stream so that isn't an added issue either :-)

[smUX]

Michael, a little project for you: Get two tape recorders or *something* that can record and play back audio (not your PC, you'll be using it)...now record a different sound on each one...record them individually...something like a male and a female saying different words.

Then record them individually into your computer using the microphone...and now you see why I say two of them...play them TOGETHER this time and record that and then crunch the data looking specifically for information on how the two audio sources merge into one (what effect the second audio has on the stream)

There might be better ways of doing this...maybe playing a DVD in the background at loud volume and recording it then repeating but talking over it...but you get the idea. Doing this, you SHOULD be able to work out exactly how to merge two audio sources, and all without asking other people for advice on the methods behind it.

Alternatively, email someone who's written a professional program (like Audacity, perhaps?) that deals with audio and ask how they manage it or if they have any good URLs for you to look into :-)

[DigiRev]

Alternatively, email someone who's written a professional program (like Audacity, perhaps?) that deals with audio and ask how they manage it or if they have any good URLs for you to look into :-)

I'll second that. Audacity is a good freeware recording program.

Reaper is another good one and the one that I use on my recordings. The developer frequents the forum and I'm sure he wouldn't mind giving you advice.

When merging 2 or more audio sources in mono, the volume will increase each time. This is what happens in my recordings anyway. So maybe only the frequencies are divided and not the amplitude.

[smUX]

There must be some sort of filtering you have to do to normalise the audio.

And I've just had a pretty damning thought...if you're writing a multi-user voice chat then there's NO point in merging all audios because you'd merge your own audio in there with everyone else's and you want to play everyone's *except* your own (it'd create a freaky loopback or delayed return effect if your own audio is in there...wouldn't sound natural)...this could be a bit trickier than I thought it would have been :-)

[Arne Elster]

I'd do it like this:

1) Write a playback class which has an audio buffer queue.
It should support multiple instances, therefore an instance for every user in the chat.
The playback class should depend on DirectSound and do streaming.
When the audio buffer queue is empty, silence shall be written to the DirectSound buffer (so the class acts like WaveOut).
2) When there's an incoming packet, add it to the audio buffer queue of the playback class instance associated with the user who sent the packet.

Step 1 includes that when a user says nothing, no data shall be sent.
That saves bandwidth.
If you don't mind quality loss, Windows comes with an GSM codec,
which is also used by ISDN and maybe even mobile phones (not sure on that one).
I believe it has a better compression.

For the class I'd use an API timer and map it somehow to the instance which created it,
for keeping track of the DirectSound buffer playback cursor, as the DS events are unreliable.

Further you have your own write cursor which always points to the end of the last written byte in the DirectSound buffer.
When the difference between your write cursor and the play cursor of the DirectSound buffer is only about 60-100 ms, write the next chunk of bytes to the DirectSound buffer by reading some bytes out of the audio buffer queue.
When the audio buffer queue is empty, write silence to the buffer, but have flag that you currently play silence.
Now when there is a buffer appended to the queue, immediately pause the
DirectSound buffer, write some data to the current playback position
and set the DirectSound buffer's state to playback again, also update your own write cursor.

[DigiRev]

Not sure if you're checking back on this thread or not, but I think it may be a phase issue that is making the sound seem lower in volume.

Although phase is usually rare when it's people talking, because everyone's voice is so different. On a musical recording, like a guitar, for example, it happens more often because the frequencies are usually the same.

Here is a link if you want to read it, but you may know this stuff already.

http://www.physicsclassroom.com/Class/sound/u11l3a.html

If it's not a phase issue then there's something wrong with the way you're summing the audio.

I would ask the developer of Audacity or Reaper to see how they did it. Audacity is open-source so I'm sure they wouldn't mind helping you out.

Good luck.

[CVMichael]

I am checking, but i'm not sanying anything because I did not had the time to test anything that you guys are saying. I should have more time in the weekend.

Arne Elster gave me a good idea on how to fix the events problem.
First of all, if I use a timer, it won't be acurate enough, and timers don't interrupt the code execution like a DirectSound event does.

So what I will do in each function that receives the event, I will check if the event belongs to current instance by checking the cursor position, and the time it took from the last call. If the time diference is too short it means that this event belongs to another instance, and if the cursor position if too far from the beginning, or middle, then again it means that it belongs to another instance (since events should be called only at the beginning of the buffer and middle of the buffer), so checking the cursor position is a good idea. That should solve the mess in the events.

But even that, it does not seem very "clean" to me...

I think in the end I will do what I originally thought, and that is to make only one instance of DirectXSound, that way I don't have interferance, or maybe apply some simple checking just in case I get interferance from other applications like WinAmp, and mix the sounds on my own (with functions made by me). I already started to make the mixing functions yesterday.

First I will try without division, like this
Output_sound = Sound1 + Sound2 + Sound3 ... etc...

[Arne Elster]

Arne Elster gave me a good idea on how to fix the events problem.
First of all, if I use a timer, it won't be acurate enough, and timers don't interrupt the code execution like a DirectSound event does.

There are better timers.
http://www.vbaccelerator.com/codelib...r/hirestmr.htm
Also a DS event can't interrupt the code flow.
DoEvents or just doing nothing (so the VB window proc will process incoming messages) does the trick.
Same with timers.

[CVMichael]

I finally DID IT !

It was so difficult... I had so many problems, and a few basic mistakes that took me a looooggg time to see them, like forgetting to put ByVal in CopyMemory for memory pointers, and crashing the app, it took me a few hours just to see that mistake

If anyone is curious how I did it, here's how:
First of all, sending the sound is easy, no problems there, record, encode, and send to each client the same thing...
The difficult part is mixing the sound from all clients...
I had to make so many buffers...
In short, 2 buffers for each client, one for incomming MP3, and another for decoded wave. The decoded wave had to be the same format, regardless of the format of the MP3...
For example, from Client1 the format is 22050 Hz, stereo, from Client2 the format is 16000Hz, mono, and from Client3 format 22050 Hz, mono... etc, So I had to make it convert to a common format, like say 44100 Hz, stereo...
So, regardless of what format I receive, it had to be converted to 44100, stereo.
For that I had to make another buffer for the final wave, where all waves from all clients get mixed into one...
Then finally, I could play the wave using DirectSound (one instance only, so no conflicts)

I'm an still working on small things to make it complete (usable), but the biggest problem is done...

When I finish the program, is anyone interested to try my program (I don't want to release the code, I want someone to try it by installing it and chat with me using voice) ??

I also want oppinions, like what to change/add/improve ?

[CVMichael]

Everyone seems to be very excited about it...

[Andrew G]

Everyone seems to be very excited about it...

We are excited

Just wondering, would you also be adding video support, so that multiple people can see each other and also chat?

[CVMichael]

As much I wish to do so, I'm afraid not...

I don't know much about streaming video, it took me a few weeks to make the voice chat program, it will probaly take me months to make it for video.
If I would have have a job, I would be starting on it right now...

[rory]

i can help you with the video side, at least if they were to each use the same capture card, as alot of them come with ActiveX controls just for that purpose. Webcams probably have ActiveX controls also for developers, but depends on the brand; ill have to get my hands on one some day soon.

So are we going to see the code for the chat?

[Al42]

I'd be interested in testing the audio part with you. I'm usually on line from around 8PM to midnight EST. If you get the video part worked out I could test that too - my new laptop has a camera built in just above the screen and I've never had a use for it.

[CVMichael]

So are we going to see the code for the chat?

Sorry, I don't want to post the code because I've been working on it for too long, and I don't want to see someone using my code with their name on it... I'm sure that you, and many other people on this forum won't do that, but I'm pretty sure that other people that are not even subscribed to vbforums will take advantage with no remorse whatsoever.

If we would have a section where I could post, and only a select # of people would have access to it, then I won't mind sharing it with people worthy of it.

I'd be interested in testing the audio part with you. I'm usually on line from around 8PM to midnight EST. If you get the video part worked out I could test that too - my new laptop has a camera built in just above the screen and I've never had a use for it.

I don't know how to translate 8PM EST to my time....
Lets first just see what is the time difference between me and you.
Right now the time here is: 1:32 PM, what time do you see this post was made for you ?

[CVMichael]

So are we going to see the code for the chat?

Continuing to my previous post, I just got an idea....

I can put the code on my web-server (I did not install it yet, but I can do it in the weekend), so rory and Al42 PM me if you still want the code.

And Al42, can we test the program in the weekend ? I still don't know what is the time difference between us...

[Al42]

According to the clock on the server the forum is running on, it's about 1 minute.

We're in the same time zone - east coast of North America down here, one zone west of the Maritimes for you. TO is in the eastern time zone - EST is Eastern Standard Time (as opposed to Eastern Daylight Time - EDT). What do you call it up there?

The weekend sounds good - I'll probably be out shopping a few hours on Saturday and I'm taking my wife out to dinner for her birthday on Sunday, but I'm free the rest of the weekend.

PM coming up.

[rory]

thanks CVMichael, ill send you a pm tonight ..

Rory

[KittyFied]

is the source to this available?

[CVMichael]

is the source to this available?

No it's not... I've worked way too much and too hard to just make it free for everyone... and it's not complete on top of that (yes... after all this time, I still did not finish it)

[KittyFied]

no problem. just attempting my own version and wanted to compare ideas.

thank you for replying any way, it's much appreciated.

[TieBender]

Hi CVMichael.

Did you ever get round to finishing this chat program?
I hope you did, and I hope you would let me take a shot at deploying it usefully: I'm making an online video game for some blind kids in Denmark, and I need some way, other than skype/teamspeak etc. to have them communicate.(Hard for blind ppl to navigate menus, especially kids.) I hope you will let me tinker with your source so I can build it into the game. The work I do with this game is charity, and have thereby no way to pay you for it, but I will offcourse add your name in the credits.
The game will likely be attached to a study in the Danish Institute of Technology in a study of "life improvement technologies for the visually impaired" to be initiated May 2011, ending Sep. 2015.

I hope you will agree to this.
Kind Regards
Roland B.