Merge Two Text Files With No Duplication...

I have to merge two application config/settings files like the following...

File one is a configuration file already deployed on a users system and looks something like this, for example...

;TestingBS
;company.testing=BS

They commented out the company.testing value to suit their needs.

File two looks something like this. This is the file that will be included in an upgrade. Anything that matches in any way to File one should be left alone...

;TestingBS
company.testing=BS

;New Value
value.new=1


You'll notice that the first value in this file is not commented out, but that is not to be translated to the output file as the users current configuration is to be maintained. Also, any new values should be pumped to the new file. the resulting file should look like this...

;TestingBS
;company.testing=BS

;New Value
value.new=1

I've been trying a few things, but I can't get it quite right. If I do get the new value in the output, the company.testing appears both commented and uncommented.

Can anyone think of a quick way to do this?

Try this:

Code:

---

Dim file1() As String = IO.File.ReadAllLines("file1.txt")
Dim file2() As String = IO.File.ReadAllLines("file2.txt")
Dim combinedFiles As New List(Of String)

combinedFiles.AddRange(file1)
combinedFiles.AddRange((From line As String In file2 Where Not combinedFiles.Contains(line) AndAlso Not combinedFiles.Contains(";" & line)).ToArray())

---

I feel like I should explain my code in a little bit of detail rather than just throwing you a couple of lines of code.

• The first two lines get an array of String which represents the lines in your first and second file respectively
• The third line declares a new list to hold the merged content of both of your files
• The fourth line adds everything from the first file(since this will remain static)
• The fifth line add everything from the second file that is not included in the first file and not included in the first file as a comment

Quote:

---

Originally Posted by dday9

Try this:

Code:

---

Dim file1() As String = IO.File.ReadAllLines("file1.txt")
Dim file2() As String = IO.File.ReadAllLines("file2.txt")
Dim combinedFiles As New List(Of String)

combinedFiles.AddRange(file1)
combinedFiles.AddRange((From line As String In file2 Where Not combinedFiles.Contains(line) AndAlso Not combinedFiles.Contains(";" & line)).ToArray())

---

I feel like I should explain my code in a little bit of detail rather than just throwing you a couple of lines of code.

• The first two lines get an array of String which represents the lines in your first and second file respectively
• The third line declares a new list to hold the merged content of both of your files
• The fourth line adds everything from the first file(since this will remain static)
• The fifth line add everything from the second file that is not included in the first file and not included in the first file as a comment

---

I'll give that a shot. I guess I could then write out combinedFiles to a final file merge.properties, for example. ??

That's got me close with so much less code. Here is one issue I found so far...

If file one looked like this...

;TestingBS
;adept.testing=BS

;New Value
value.new=1

And the new file coming in the install has an updated value.new, which is 2. The new value is not to update any values in File one and it didn't, sort of. It added the new value.new=2.

Here's file two in this scenario...

;TestingBS
adept.testing=BS

;New Value
value.new=2

And here is the resultant merged file...

;TestingBS
;adept.testing=BS

;New Value
value.new=1
value.new=2

The commented adept.testing value still remains, which is good, but the additional/duplicate value was added. Maybe that is where all the extra code is needed. ??

Again, thanks so much for the code you've provided so far. I hope I can use that without much more to get the job done because it will certainly clean things up.

Or, would it be easier to have everything dumped to combinedFiles, then weed out duplication as described above after the files have been combined. I've been trying to filter out the duplicates within the confines of the code you provided, but can't get it to work.

What you're wanting to do can be difficult depending on how strict your rules need to be. See, at first your rules were:

1. Keep everything in file1
2. Remove anything in file2 that either is exactly like a line in file1 or almost exactly like a line in file1 only commented
3. A comment was denoted by starting with a semi-colon

Now your rules have grown to include a fourth criteria where if a value is being assigned to a different value in file2 then it should be removed as well. Only you'll have to define what is a new value.

Before I give you another example, what you should do is clearly define the rules that you need to follow and consider all possibilities. This is so we don't have a constant back-and-forth where code I provide may solve one rule but not a new rule that wasn't previously thought of or it may solve the new problem but throw off the previous code that solved the older rules and thus the flow of the code would have to be completely rewritten.

Quote:

---

Originally Posted by dday9

What you're wanting to do can be difficult depending on how strict your rules need to be. See, at first your rules were:

1. Keep everything in file1
2. Remove anything in file2 that either is exactly like a line in file1 or almost exactly like a line in file1 only commented
3. A comment was denoted by starting with a semi-colon

Now your rules have grown to include a fourth criteria where if a value is being assigned to a different value in file2 then it should be removed as well. Only you'll have to define what is a new value.

Before I give you another example, what you should do is clearly define the rules that you need to follow and consider all possibilities. This is so we don't have a constant back-and-forth where code I provide may solve one rule but not a new rule that wasn't previously thought of or it may solve the new problem but throw off the previous code that solved the older rules and thus the flow of the code would have to be completely rewritten.

---

I thought when I said no duplication meant that any value either commented or uncommented in file 1 should not be altered in any way by file 2 and that any new item, actual comments headed by a semicolon or values, commented or not, in file 2 should make it to the combined file. Sorry about the confusion.

Maybe I'm adding to confusion by using the term value and should maybe use key instead as in key=value. If a key already exists in File 1, it should not be added again to the merge if it contains a different value - key=new value, nor should the value from file 1 be updated.

I guess one would have to actually search for key substrings before the equal sign from file 2 to see if that same substring already exists in the combined holding from file 1.

I think the following example covers the rules...

File 1 (end user modified file):
;TestingBS - commented values should remain commented
;adept.testing=BS

;Needed Value - any values uncommented should remain uncommented
value.needed=xxx

;Changed Value - any changed values should hold their value and not revert to seeded default
changed.value=new

File 2 (seed file in upgrade installation):
;TestingBS
adept.testing=BS

;Needed Value
;value.needed=xxx

;Changed Value
changed.value=super_new

;New Value
value.new=1

Merged File (should basically reflect File 1 with any additional, new comments and keys from File 2):
;TestingBS - commented values should remain commented
;adept.testing=BS

;Needed Value - any values uncommented should remain uncommented
value.needed=xxx

;Changed Value - any changed values should hold their value and not revert to seeded default
changed.value=new

;New Value
value.new=1

Again, sorry for the confusion. I guess it can be summed up by saying that the Merged or final file should reflect everything in file 1 plus anything entirely new from File 2 only. Let me know if that is clearer or if that muddied the waters further and thanks for your efforts to help. I greatly appreciate it!

Use a brute force method:

Read all lines from both files;
Have nested loops loop through first file and for each line in first file loop through second file;
Apply rules in the nested loop.

Note: for 'applying rules', just call a routine with line1 and line2. If line 1 needs writing, return true, if line2 needs writing, return false (or something like that). Outside the inner loop, determine if you need to write line1 or replace it with line2.

While it may be 'nice' to have one liners, it can be no quicker than using a brute force method. Also, if it doesn't quite work, it does you no good.

Quote:

---

Originally Posted by SJWhiteley

Use a brute force method:

Read all lines from both files;
Have nested loops loop through first file and for each line in first file loop through second file;
Apply rules in the nested loop.

Note: for 'applying rules', just call a routine with line1 and line2. If line 1 needs writing, return true, if line2 needs writing, return false (or something like that). Outside the inner loop, determine if you need to write line1 or replace it with line2.

While it may be 'nice' to have one liners, it can be no quicker than using a brute force method. Also, if it doesn't quite work, it does you no good.

---

That's basically what I had in place and that too didn't quite work. I think I've ironed out the kinks and it seems to be 'working' as desired now, but it's not the easiest thing to troubleshoot.

What I do is basically dump the first file to a mResult string list. Then, when reading through the second use mResult.contains to see if it should be written to mResult for future writing to the final file. The rules seem currently handled in this routine.

I did run up against an issue if a user uncommented and changed a value in their end user file. What was happening was that the commented line for that value was being written to the file from the install seed file when it should not have been.

It's a tangled web of checking Left(item, 1) = ";", use of substring, etc.

Like I said, it appears to be what we need for right now. I was hoping for more concise code and the initial example offering seemed really close and would have rid me of a bunch of code. Oh well. Onward and upward! :thumb:

this is a brute force method using classes but might be a little easier to understand.
I am sure SJ, dday or Shaggy could find a way to pare it down some.

Untested...

Code:

---

Imports System.IO

Public Class Form1
  Private Sub Form1_Load(sender As Object, e As EventArgs) Handles MyBase.Load
    Dim MasterFile() As String = File.ReadAllLines("file1.txt")
    Dim AddOnFile() As String = File.ReadAllLines("file2.txt")

    Dim OutputList As New List(Of String)

    Dim MasterList As New List(Of clsItem)
    Dim AddOnList As New List(Of clsItem)

    For Each sline As String In MasterFile
      MasterList.Add(New clsItem(sline))
    Next

    For Each sline As String In AddOnFile
      AddOnList.Add(New clsItem(sline))
    Next

    For Each Item As clsItem In AddOnList
      If Not MasterList.Exists(Function(x) x.ID = Item.ID) Then
        MasterList.Add(Item)
      End If
    Next

    Dim sOut As String = ""
    For Each Item As clsItem In MasterList
      With Item
        sOut &= .Comment & .ID & .Value & vbNewLine
      End With
    Next

    File.WriteAllText("File3.txt", sOut)
  End Sub
End Class

Public Class clsItem
  Public Property ID As String = ""
  Public Property Comment As String = ""
  Public Property Value As String = ""

  Public Sub New(sItem As String)
    sItem = sItem.Trim
    If sItem.StartsWith(";") Then
      Comment = ";"
      sItem = sItem.Substring(1)
    End If
    If sItem.Contains("=") Then
      Dim segs() As String = sItem.Split("="c)
      sItem = segs(0).Trim
      Value = "=" & segs(1).Trim
    End If
    ID = sItem
  End Sub
End Class

---