Results 1 to 15 of 15

Thread: [RESOLVED] Vb.net - Split large text file by number of lines

  1. #1

    Thread Starter
    Addicted Member
    Join Date
    May 2017
    Location
    Italy
    Posts
    170

    Resolved [RESOLVED] Vb.net - Split large text file by number of lines

    How can I split a single text file with 1000 lines into multiple smaller files of, for example, 300 lines ? I found two C# code, they feel good, but I'm not able to convert them to vb.net
    Code:
    using (System.IO.StreamReader sr = new System.IO.StreamReader("path"))
    {
        int fileNumber = 0;
    
        while (!sr.EndOfStream)
        {
            int count = 0;
    
            using (System.IO.StreamWriter sw = new System.IO.StreamWriter("other path" + ++fileNumber))
            {
                sw.AutoFlush = true;
    
                while (!sr.EndOfStream && ++count < 20000)
                {
                    sw.WriteLine(sr.ReadLine());
                }
            }
        }
    }
    Code:
    var reader = File.OpenText(infile);
            string outFileName = "file{0}.txt";
            int outFileNumber = 1;
            const int MAX_LINES = 300;
            while (!reader.EndOfStream)
            {
                var writer = File.CreateText(string.Format(outFileName, outFileNumber++));
                for (int idx = 0; idx < MAX_LINES; idx++)
                {
                    writer.WriteLine(reader.ReadLine());
                    if (reader.EndOfStream) break;
                }
                writer.Close();
            }
            reader.Close();

  2. #2
    .NUT jmcilhinney's Avatar
    Join Date
    May 2005
    Location
    Sydney, Australia
    Posts
    107,470

    Re: Vb.net - Split large text file by number of lines

    So what specific problems did you have while converting? Did you make any effort at all? That first code snippet has two using statements and two while loops, two declarations, a property setting and a method call. What exactly is preventing you doing the same thing in VB? Apart from the fact that you should be able to make a pretty good fist of a manual conversion for yourself if you bother to try, there are plenty of code converters available only that are of varying quality but can pretty much all handle simple code like that. There's also the downloadable Instant VB that can handle extremely complex code so will have no issue with that basic stuff. Basically, if you had made a reasonable effort then you could have done this yourself and you could have still done some of it yourself even with an unreasonable effort.

  3. #3

    Thread Starter
    Addicted Member
    Join Date
    May 2017
    Location
    Italy
    Posts
    170

    Re: Vb.net - Split large text file by number of lines

    I don't know C#, I tried with some code converters without success, they can not convert them.
    I have already solved the problem with a much longer code, I would like to understand how to reduce the code
    Code:
    Dim OriginalFilePath = "F:\Documenti\Visual Studio 2017\Projects\TextFileSplit.txt"
            Dim lines = IO.File.ReadAllLines(OriginalFilePath)
            Dim NumOfLines As Integer = lines.Length, LinesPerFile As Integer = 300
            Dim NumOfFiles As Integer = NumOfLines \ LinesPerFile + 1
            Dim StartIndex As Integer
            Dim EndIndex As Integer
            Dim sb As New System.Text.StringBuilder
            For i = 0 To NumOfFiles - 1
                EndIndex = StartIndex + LinesPerFile - 1
                If EndIndex >= NumOfLines - 1 Then
                    EndIndex = NumOfLines - 1
                End If
                For index = StartIndex To EndIndex
                    sb.AppendLine(lines(index))
                Next
                Dim NewFilePath As String = "F:\Download\File" + i.ToString + ".txt"
                IO.File.WriteAllText(NewFilePath, sb.ToString)
                StartIndex = EndIndex + 1
                sb.Clear()
            Next
    Last edited by patel45; Jul 25th, 2021 at 10:55 AM.

  4. #4
    Karen Payne MVP kareninstructor's Avatar
    Join Date
    Jun 2008
    Location
    Oregon
    Posts
    6,600

    Re: Vb.net - Split large text file by number of lines

    If you would like to see a proven method to chunk files (and there is a great deal more code as it's ported from a C# enterprise utility I wrote) see the following GitHub repository.

  5. #5

    Thread Starter
    Addicted Member
    Join Date
    May 2017
    Location
    Italy
    Posts
    170

    Re: Vb.net - Split large text file by number of lines

    Thanks Karen

  6. #6
    Fanatic Member 2kaud's Avatar
    Join Date
    May 2014
    Location
    England
    Posts
    722

    Re: Vb.net - Split large text file by number of lines

    split a single text file with 1000 lines into multiple smaller files of, for example, 300 lines
    Why? I would consider a 1000 line file as 'small' ? What problems are you getting with a file of this size that requires it to be split on number of lines?
    All advice is offered in good faith only. You are ultimately responsible for the effects of your programs and the integrity of the machines they run on. Anything I post, code snippets, advice, etc is licensed as Public Domain https://creativecommons.org/publicdomain/zero/1.0/

    C++20 Compiler: Microsoft VS2019 (16.11.3)

  7. #7
    Angel of Code Niya's Avatar
    Join Date
    Nov 2011
    Posts
    6,379

    Re: Vb.net - Split large text file by number of lines

    This web site converted the code in the OP pretty comfortably.

    It produced this:-
    vbnet Code:
    1. Using sr As StreamReader = New StreamReader("path")
    2.     Dim fileNumber As Integer = 0
    3.  
    4.     While Not sr.EndOfStream
    5.         Dim count As Integer = 0
    6.  
    7.         Using sw As StreamWriter = New StreamWriter("other path" & Threading.Interlocked.Increment(fileNumber))
    8.             sw.AutoFlush = True
    9.  
    10.             While Not sr.EndOfStream AndAlso Threading.Interlocked.Increment(count) < 20000
    11.                 sw.WriteLine(sr.ReadLine())
    12.             End While
    13.         End Using
    14.     End While
    15. End Using

    Though I'd probably refactor all that Interlocked.Increment code.
    Treeview with NodeAdded/NodesRemoved events | BlinkLabel control | Calculate Permutations | Object Enums | ComboBox with centered items | .Net Internals article(not mine) | Wizard Control | Understanding Multi-Threading | Simple file compression | Demon Arena

    Copy/move files using Windows Shell

    C++ programmers will dismiss you as a cretinous simpleton for your inability to keep track of pointers chained 6 levels deep and Java programmers will pillory you for buying into the evils of Microsoft. Meanwhile C# programmers will get paid just a little bit more than you for writing exactly the same code and VB6 programmers will continue to whitter on about "footprints". - FunkyDexter

    There's just no reason to use garbage like InputBox. - jmcilhinney

    The threads I start are Niya and Olaf free zones. No arguing about the benefits of VB6 over .NET here please. Happiness must reign. - yereverluvinuncleber

  8. #8

    Thread Starter
    Addicted Member
    Join Date
    May 2017
    Location
    Italy
    Posts
    170

    Re: Vb.net - Split large text file by number of lines

    Thanks Niya, it runs well

  9. #9
    eXtreme Programmer .paul.'s Avatar
    Join Date
    May 2007
    Location
    Chelmsford UK
    Posts
    24,183

    Re: Vb.net - Split large text file by number of lines

    Code:
    Dim fileLines() as String = IO.File.ReadAllLines("path")
    Dim AllLines(CInt(Math.Floor(fileLines.Length / 300)))() as String
    
    For x as integer = 0 to AllLines.GetUpperBound(0)
        AllLines(x) = fileLines.Take(300).ToArray
    Next
    Edit: Actually there's an error...

    Code:
    Dim fileLines() as String = IO.File.ReadAllLines("path")
    Dim AllLines(CInt(Math.Floor(fileLines.Length / 300)))() as String
    
    For x as integer = 0 to AllLines.GetUpperBound(0)
        AllLines(x) = fileLines.Skip(x * 300).Take(300).ToArray
    Next
    Last edited by .paul.; Jul 31st, 2021 at 09:09 PM.

  10. #10

    Thread Starter
    Addicted Member
    Join Date
    May 2017
    Location
    Italy
    Posts
    170

    Re: Vb.net - Split large text file by number of lines

    Thanks Paul, very compact code, I modified it so
    Code:
            Dim OriginalFilePath = "F:\Documenti\Visual Studio 2017\Projects\TextFileSplit.txt"
            Dim LinesPerFile As Integer = 20
            Dim fileLines() As String = IO.File.ReadAllLines(OriginalFilePath)
            Dim AllLines(CInt(Math.Floor(fileLines.Length / LinesPerFile)))() As String
            For i = 0 To AllLines.Length - 1
                AllLines(i) = fileLines.Skip(i * LinesPerFile).Take(LinesPerFile).ToArray
                Dim NewFilePath As String = "F:\Download\File" + i.ToString + ".txt"
                IO.File.WriteAllLines(NewFilePath, AllLines(i))
            Next

  11. #11
    .NUT jmcilhinney's Avatar
    Join Date
    May 2005
    Location
    Sydney, Australia
    Posts
    107,470

    Re: Vb.net - Split large text file by number of lines

    Replace this:
    vb.net Code:
    1. CInt(Math.Floor(fileLines.Length / LinesPerFile))
    with this:
    vb.net Code:
    1. fileLines.Length \ LinesPerFile

  12. #12
    eXtreme Programmer .paul.'s Avatar
    Join Date
    May 2007
    Location
    Chelmsford UK
    Posts
    24,183

    Re: Vb.net - Split large text file by number of lines

    As you’re just writing it directly to new files…

    Code:
    Dim OriginalFilePath = "F:\Documenti\Visual Studio 2017\Projects\TextFileSplit.txt"
    Dim LinesPerFile As Integer = 20
    Dim fileLines() As String = IO.File.ReadAllLines(OriginalFilePath)
    
    For i = 0 To AllLines.Length - 1
        Dim subsetLines() As String = fileLines.Skip(i * LinesPerFile).Take(LinesPerFile).ToArray
        Dim NewFilePath As String = "F:\Download\File" + i.ToString + ".txt"
        IO.File.WriteAllLines(NewFilePath, subsetLines)
    Next

  13. #13
    .NUT jmcilhinney's Avatar
    Join Date
    May 2005
    Location
    Sydney, Australia
    Posts
    107,470

    Re: Vb.net - Split large text file by number of lines

    Given that we are supposedly talking about large files here, I'd be tempted to avoid calling ReadAllLines.
    vb.net Code:
    1. Dim inputFilePath = "file path here"
    2. Dim outputFolderPath = "folder path here"
    3. Dim linesPerFile = 20
    4. Dim fileNumber = 1
    5. Dim lines As New List(Of String)(linesPerFile)
    6.  
    7. For Each line In File.ReadLines(inputFilePath)
    8.     lines.Add(line)
    9.  
    10.     If lines.Count = linesPerFile Then
    11.         Dim outputFilePath = Path.Combine(outputFolderPath, $"File{fileNumber}.txt")
    12.  
    13.         File.WriteAllLines(outputFilePath, lines)
    14.         fileNumber += 1
    15.         lines.Clear()
    16.     End If
    17. Next
    18.  
    19. If lines.Count > 0 Then
    20.     Dim outputFilePath = Path.Combine(outputFolderPath, $"File{fileNumber}.txt")
    21.  
    22.     File.WriteAllLines(outputFilePath, lines)
    23. End If
    It's also worth noting that this:
    vb.net Code:
    1. Dim subsetLines() As String = fileLines.Skip(i * LinesPerFile).Take(LinesPerFile).ToArray
    is going to enumerate the same array over and over again. It will only enumerate part of the array each time but that part will get larger and larger. It would be more performant to create the target array ahead of time and call Array.CopyTo and specify the section of the source array to copy. Not a big deal for small data sets but a potential issue for large ones.

  14. #14
    eXtreme Programmer .paul.'s Avatar
    Join Date
    May 2007
    Location
    Chelmsford UK
    Posts
    24,183

    Re: Vb.net - Split large text file by number of lines

    Quote Originally Posted by jmcilhinney View Post
    Given that we are supposedly talking about large files here...
    1000 lines is a large file???

  15. #15
    .NUT jmcilhinney's Avatar
    Join Date
    May 2005
    Location
    Sydney, Australia
    Posts
    107,470

    Re: Vb.net - Split large text file by number of lines

    Quote Originally Posted by .paul. View Post
    1000 lines is a large file???
    Not really, although the OP did say "large text file". I guess it's just large in relation to the very small files they want to end up with. Still, if nothing else, it's good to know the different ways that you can approach something like this and what the pros and cons of each option are.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  



Click Here to Expand Forum to Full Width