TCP - images, streams and 3AM :(

[bensonsearch]

Hi All,

I have decided to try and learn about the programming side of TCP connections (well more try and get it working).

Im trying to send an image over the connection using network stream, it doesn't seem to be working as the receiver just freezes and end of story.

the image is made from a videosourceplayer which is a aForge.Net control that lets you hook a webcam to it, so basically the image is a webcam snap shot (if there is a way to send the output of the cam straight down the connection ill gladly take that). I have noticed there is a Image.FromStream which seems handy but is there maybe something like Image.ToStream???

My code so far (ps i have got text to send but i wish to have image).

Imports System.Net.Sockets
Imports System.IO
Imports AForge.Video.DirectShow

Public Class Form1
    Dim l As New TcpListener(3000)
    Dim c As TcpClient
    Dim Messages As String = ""
    Dim VideoCaptureSource As VideoCaptureDevice
    Dim VideoDevices As New FilterInfoCollection(FilterCategory.VideoInputDevice)


    Private Sub Form1_FormClosing(sender As Object, e As System.Windows.Forms.FormClosingEventArgs) Handles Me.FormClosing
'stop webcam,timer and listener
        VideoSourcePlayer1.SignalToStop()
        VideoSourcePlayer1.WaitForStop()
        Timer1.Stop()
        l.Stop()
    End Sub
    Private Sub Form1_Load(sender As System.Object, e As System.EventArgs) Handles MyBase.Load
'add ips, start listener, start timer
        ComboBox1.Items.Add("192.168.x.x")
        ComboBox1.Items.Add("192.168.x.x")
        l.Start()
        Timer1.Start()
'see if we have a webcam
        If VideoDevices.Count = 0 Then
        Else
'setup webcam and send video to videosourceplayer1
            Dim VideoCaptureSource2 As New VideoCaptureDevice
            VideoCaptureSource2 = New VideoCaptureDevice(VideoDevices(1).MonikerString)
            VideoSourcePlayer1.VideoSource = VideoCaptureSource2
            VideoSourcePlayer1.Start()
        End If

    End Sub

    Private Sub Button1_Click(sender As System.Object, e As System.EventArgs) Handles Button1.Click
        Try
'lets send something to ip in combobox
            c = New TcpClient(ComboBox1.Text, 3000)
            Dim ms = New MemoryStream()
            Dim uu As Image
            uu = VideoSourcePlayer1.GetCurrentVideoFrame
'save image into memorystream
            uu.Save(ms, System.Drawing.Imaging.ImageFormat.Jpeg)
'put memorystream into byte array
            Dim bytes = ms.ToArray()
'make network stream
            Dim ii As NetworkStream = c.GetStream
'write the byte array
            ii.Write(bytes, 0, bytes.Length)
            ii.Flush()

        Catch ex As Exception
            MsgBox(ex.Message)
        End Try

    End Sub

    Private Sub Timer1_Tick(sender As System.Object, e As System.EventArgs) Handles Timer1.Tick
        Try
'setup listening
            If l.Pending = True Then
                c = l.AcceptTcpClient()
'set network stream
                Dim nn As NetworkStream = c.GetStream()
'make new image
                Dim h As Image
'create image from stream
                h = Image.FromStream(nn)
'set picture=stream
                PictureBox1.Image = h

            End If
        Catch ex As Exception
            MsgBox(ex.Message)
        End Try
    End Sub
End Class

[bensonsearch]

anyone?

[jmcilhinney]

I would strongly suggest that you don't test this in a single application. Follow the CodeBank link in my signature and check out my Asynchronous TCP thread. I suggest that you do something similar.

[bensonsearch]

Hi jmchilhinney, long time no see

I have just read the 7 pages of that thread...... that's too much reading for me lol.

I test this on two physically separate machines on a LAN so the sender isn't also the receiver, however looking at your thread MDI may be more convenient. Also the Asynchronous part was to come after the logic but now I have your code I don't need to worry on that part.

My questions still remains though, I can send text fine but I can not for the life of me figure out how to send an image, as far as I know my code should be converting the image into a byte array and the receiver then should translate it back into an image. The problem is this isn't happening, just the receiver freezes. This leads me to believe the sender is fine and its the receiver that has an issue.

This in itself is a mistery to me as from what I have read the

 h = Image.FromStream(nn)

should not care what kind of stream is passed so long as its an image...... I am just so far out of my comfort and there is apparently no real sources to read on (that I can find, i'm prob not searching right).

Also as a side note, i'm thinking of extending your code to send anything using the object type. seems this will allow me to send it as its meant to convert anything to byte, but I can not think of a good way to receive it without testing against each and every type. (this was converted online from C#)

Private Function ObjectToByteArray(obj As Object) As Byte()
	If obj Is Nothing Then
		Return Nothing
	End If
	Dim bf As New BinaryFormatter()
	Dim ms As New MemoryStream()
	bf.Serialize(ms, obj)
	Return ms.ToArray()
End Function

Thanx for your help, I know your a busy man.

ps. seeing as your in Sydney maybe we should catch a coffee

[jmcilhinney]

With regards to sending different types of data, I would suggest a header that includes a code that indicates the type of data to follow. Your receiver can then read the header and know what to expect and therefore how to treat it.

[passel]

If you're using Image.FromStream, then the documentation says the stream has to remain open for the duration of the lifetime of the image.
I think that probably means that Image.FromStream is probably more suited to a memory stream, rather than a network stream.
In anycase, I wrote something sometime ago that could transfer an image over a network, and as jmc says, I prefaced the data with something that could be used to know how to handle the coming data.
In this case, I didn't go with a full header of various information, just a single 32-bit integer that indicated the length of the data to follow, so I new how many bytes to read from the stream.
So the transmitting side looked something like this. In my case, the user on the other end wanted the Length to be in BigEndian order, so I has to swap the bytes of the length (the first four bytes to be transmitted), but that wouldn't be necessary if you had control of both ends, you could just transmit the length in its "natural" (on Intel boxes) LittleEndian order.
Notice that I do two writes to the TCP stream, the first the Length, the second the image data.
(some code cut out to shorten the example)

        Using mstream As System.IO.MemoryStream = New System.IO.MemoryStream
          bmpCapture.Save(mstream, System.Drawing.Imaging.ImageFormat.Png) 'save the picturebox image to memory in png format

          Dim b() As Byte = mstream.ToArray()     'Copy the png image from memory into a byte array
          If myTcpClient IsNot Nothing Then
            If myTcpClient.Connected Then
              Dim il As Int32 = mstream.Length                'Copy length from long into an int32
              Dim bl() As Byte = BitConverter.GetBytes(il)    'Convert the int32 into a byte array
              Dim bo(3) As Byte
              bo(0) = bl(3) : bo(1) = bl(2) : bo(2) = bl(1) : bo(3) = bl(0) 'swap the bytes into big endian
              TcpOut.Write(bo, 0, 4)                          'output the  length
              TcpOut.Write(b, 0, mstream.Length)              'followed by the data
            End If
          End If
        End Using
      End Using

The receiver I was sending to used Java, but I wrote a VB.Net version of a receiver to test that I could receive and display the image.
The main part of the receiver.
I used a BinaryReader on the network stream, to read the Length first, and then to read the data (based on the length expected) into a byte array.
So, I'll pick the critical lines from the receiver for an example. The TCPClient already accepted, and NetworkStream (nStream) gotten.
Also, this was running in a background thread so wrote the received data to a memory stream, and then invoked a sub on the GUI thread that read the memory steam into an image for display.

      Dim i As Int32
      Dim br As BinaryReader = New BinaryReader(nStream)
      Try
        Do Until exiting
          i = nStream.Read(bLen, 0, 4) 'read the 4 bytes of the length from the network stream
          iLen = IPAddress.NetworkToHostOrder(BitConverter.ToInt32(bLen, 0)) 'convert BigEdian to Little Endian and save Length

          bData = br.ReadBytes(iLen)  'read the number of bytes indicated from the stream
            If bData.Length = iLen Then 'If we've have all the data
              If mstream IsNot Nothing Then mstream.Dispose() 'probably redundant since should be disposed below
              mstream = New System.IO.MemoryStream
              mstream.Write(bData, 0, bData.Length)

              Me.Invoke(updateDelegate)  'Update the picturebox on the GUI
              mstream.Dispose()
              mstream = Nothing
            End If
          End If
        Loop

The invoked procedure (some shutdown related code removed)

  Private Sub UpdateDisplay() 'invoked as updateDelegate
    With PictureBox1
      If .Image IsNot Nothing Then .Image.Dispose()
      .Image = New Bitmap(mstream)
    End With
  End Sub

[bensonsearch]

Hi Passel,
By the looks of it this will suit me perfectly, thank you.

I will test it out later today. It may be that its early here but I have a question

iLen = IPAddress.NetworkToHostOrder(BitConverter.ToInt32(bLen, 0))

Im assuming iLen is an integer as you compare it to the length of the byte array bData, but what is bLen? where did its value come from? should this be the value i from reading?

[passel]

Guess I snipped too much.
Dim bLen(3) As Byte

bLen is an array of 4 bytes, so I'm reading four bytes from the stream to get the size information stored, and in my case have to flip them back to little endian.

I'll post the whole receiving side application, as it is short. It was just used to display the image in a picture box. It is the server side of the TCP connection.
In my case I could be transmitting on demand, or periodically at a user specified rate, so the TCP connection was expected to be connected, and then remain connected for any and all images transmitted after that. If the other side shut down, an exception would be raised, and the code would loop back up and wait for another connection.

The comparison of ilen to 10 million was just a crude sanity check. The images coming over would just be a portion of the screen from one of the glass cockpit displays in a simulator, and be either jpg or png format, so would never be anywhere near 10MB in size, so If I got a relatively large number for ilen, I'm assuming I got bad data somehow so don't try to process it.
Whether the code would be able to recover from that situation, I don't know. It might cause an exception at some point and the code would treat it as a failed connection, and reset to start listening for a new connection. The sending side should hopefully sense the loss of connection, and try to reconnect. In any case, I wasn't responsible for the receiving side, so this wasn't used in front of the customer, it was just for my testing.

Imports System.Net.Sockets
Imports System.Net
Imports System.IO

Public Class Form1
  Dim server As TcpListener
  Delegate Sub noParam()
  Dim updateDelegate As New noParam(AddressOf UpdateDisplay)
  Dim ListenThread As Threading.Thread = New Threading.Thread(AddressOf Listener)
  Dim bData() As Byte
  Dim mstream As System.IO.MemoryStream
  Dim exiting As Boolean

  Private Sub Form1_FormClosing(ByVal sender As Object, ByVal e As System.Windows.Forms.FormClosingEventArgs) Handles Me.FormClosing
    If Not exiting Then
      exiting = True
      e.Cancel = True
    End If
  End Sub

  Private Sub Form1_Load(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles MyBase.Load

    ListenThread.IsBackground = True
    ListenThread.Start()

  End Sub

  Private Sub Listener()
    Dim lPort As Int32 = 3030
    Dim lAddr As IPAddress = IPAddress.Parse("127.0.0.1")
    Dim bLen(3) As Byte
    Dim iLen As Int32

    Do Until exiting   'Outer loop to restart listening if we loose the connection

      server = New TcpListener(lAddr, lPort)
      server.Start()

      Dim lClient As TcpClient = server.AcceptTcpClient()
      Dim nStream As NetworkStream = lClient.GetStream
      Dim i As Int32
      Dim br As BinaryReader = New BinaryReader(nStream)

      Try
        Do Until exiting  'Inner loop where we read the stream and display images
          i = nStream.Read(bLen, 0, 4)
          iLen = IPAddress.NetworkToHostOrder(BitConverter.ToInt32(bLen, 0))
          If iLen < 10000000 Then
            bData = br.ReadBytes(iLen)
            If Not exiting Then
              If bData.Length = iLen Then
                If mstream IsNot Nothing Then mstream.Dispose()
                mstream = New System.IO.MemoryStream
                mstream.Write(bData, 0, bData.Length)

                Me.Invoke(updateDelegate)
                mstream.Dispose()
                mstream = Nothing
              End If
            End If
          End If
        Loop

      Catch ex As Exception
      Finally
        br.Dispose()
        mstream.Dispose()
        nStream.Dispose()
        lClient.Close()
      End Try

      server.Stop()
      server = Nothing
    Loop

  End Sub

  Private Sub UpdateDisplay()
    If Not exiting Then
      With PictureBox1
        If .Image IsNot Nothing Then .Image.Dispose()
        .Image = New Bitmap(mstream)
      End With
    Else
      Me.Close()
    End If
  End Sub
End Class