normalize weighting

i'm trying to figure out a fair weighting system for words in a file. this is what i mean. lets say i have a file that has 200 unique words and the word 'foo' is there 5 times. then i have another file that has 150 unique words with 'foo' also listed 5 times. it works out to 5/200=.025 and 5/150=.033. just because the second file has less words doesn't make it a better match for foo. they should be equal. is there anyway i can normalize the weights. and i can't just use the number of occurances because there are other metrics involved. i hope this makes sense. any help would be appreciated.

Welcome to the Forums :wave:

Quote:

---

just because the second file has less words doesn't make it a better match for foo

---

Yes it does. A higher proportion of the words in that file are "foo", and hence it has more relevance. Unless by "better match" you mean that a 200,000 word file with 2 instances of "foo" would be a better match than a 1 word file consisting solely of "foo", in which case you have to go by the absolute number.
If you want to normalise something, then you have to know what you're normalising it to. If you want to make it more relevant if the first word is "foo" than the second, then there are ways of doing such things, but you need to decide how you're determining what makes one file a "better match" than another before you can proceed.

zaza