Huffman Algoritme, I think It can be enhanced

Alright guys, here I am again between two classes just checking the answers for brief minutes.

You are right, Jian... We need a name. But I also agree with Pony, it would be nice to have the algorithms posted in the thread so we can know what to AIM, see their advantages and disadvatages so we can work on them. I would like to suggest also that someone would program those basic algorithms so we can also try them out and get an idea of how to code our own algorithms. VbBud, you could do this with the posted algorithms.

Jacksoft, as Jian said before a good compresion algorithm is based in the combination of a couple of them, so it would be a good idea to post the main points of our algorithms and seek some other ones to be able to create a final product.

For example, lets think Jian comes up with a good compresion algorithm that will use ASCII or something you might even be able to compress with the algorithm you developed, if both are used we can get a better ratio.

Dmitri K... This is not only to get a 0.1% better than other algorithms, we are trying to do something that might go beyond, ideas like this have changed the world. You have no right to come here and tell us to criticize if you don't even have an idea of what we are talking here. We would like help and constructive criticism, not someone to come and say "Duhh... That doesn't work, why don't we all go to check the arcadia videogames around the corner?" Did you think that Montante's Algortihm was developed that way?

No, it was not... So I would appreciate if you either contribute or remain quiet. Thanks!

P.S. We can try ASM too as he said, but that would be after getting the general idea of the algorithm... Don't you all agree?

Well, If my algorith works there will never be an other compression ore use some together cause mine will be able to compress itself upto 65536 levels deep. So I could make all data into a standard length. But I really can't say to much right now, cause I'm checking it out a little, the only problem is that vb can't Mod and Div with to big numbers (like 1*10^21) so I have to figure something out for that and then I will give details if the basic works.

If you say so... Just let us know when you get it working then.

Dmtri K...
we're just developing the pseudocodes, not VB codes. Besides,
I did say we need C programmers. I always know VB's no good at
that.

Jacsoft...
I always believe in wut Shannon said, that data compression has
a limit and that we can't break the barrier because the fact that it
is a fact...
And however, keep up the good work, dont get stopped by the
compiler's limit. If u cant DIV with a big number, never mind. Just
go on use that method. We can always figure out a way later.
Software floating-point simulation might be a good way but surely
we dont do it on VB.

You seem very professional!:)

BTW,maybe we can do some something other than discussing on the the trivial.

One detail plan is needed first,as you mentioned above, the problem of compiler, and so on. I mean we should act as an organization although we are not.

Also, you can schedule some work to others, for me, I'm good at math:) So maybe I can do some work on the algorithm research. for jacsoft, maybe compiler is his advantage,just guess. Anyway, a large project needs a clear plan,, step by step,right?

You seem very professional!:)

BTW,maybe we can do some something other than discussing on the the trivial.

One detail plan is needed first,as you mentioned above, the problem of compiler, and so on. I mean we should act as an organization although we are not.

Also, you can schedule some work to others, for me, I'm good at math:) So maybe I can do some work on the algorithm research. for jacsoft, maybe compiler is his advantage,just guess. Anyway, I mean a large project needs a clear plan,, step by step,right?

Greetings people..

Concerning the leaders of this project..
I definitly think that Jian should be the developmentleader.

Jian, you said that we should communicate using a forum on "our" site.
I know, from expirience, that we need some sort of real-time chatenviroment to get this thing going. A forum is a great place to exchange ideas, but we need to sit down and discuss issues in real-time.
If you wont use irc then we'll haveto find some other app. Jacsofts AMP sounds as something we should try.

Jacsoft, are there a site where we can download AMP making it possible for us to look at it?

Dmitri, I can honestly say that no one in this thread has had their mind set on writing the compression algo. in VB. If anything at all is done in VB then that thing would be the GUI, having activeX dlls, or standard DLLs in the background doing all the hard work.

Even if improve the speed/compression by only 1 or 2 % then we have at least proven that it can be done. No matter if we succeed or not, we have at least gained a whole lot of experience.

So what do you say people? Should we give jacsofts app a chance and try doing a real-time chat a.s.a.p?

hmmm...sorry for above...I edited my previous post. Was so much in a hurry and my head was buried so much in a Lunh algo program that I mistakenly wrote Lunh instead of Huffman...what the hell was I thinking of.

Anyway I have quite a clear idea of your thoughts jacsoft but are you up to something yet? I've been trying to figure out your algo and it seems to me that it might be quite resource hungry...even my brain cells feel like burning...:D

Just wandering, wont it end in using the same amount of space when its finished. Besides will you be keeping a binary table along with the compressed file? Have you had a look at Adaptive Huffman Compression/Decompression? I think its worth a try buts its still with the classic 8bits byte files.

Well,

Quote:

---

So what do you say people? Should we give jacsofts app a chance and try doing a real-time chat a.s.a.p?

---

About this. I think it could be handy, I am making some chances for this projekt so we could use it good.

But first we need kind of a team I think. Why start brainstorming with 2 or 3 men\female when more people want to join.

If you want to take a look (I have made much improvements with the one on the net) but you can go to AMP

You must register only a email adres (the rest is optional) and use it. There will be bugs in it I know, but most things work fine.

About my idea, I have tried a lot right now and haven't got a good result yet, but I am enhancing each time and now I'm at the good track. If I have one good result then I will explain the general idea cause I know I won''t be the best and there are much improvements to make, but I'll keep you updated

I'm back with some goodies.

Quote:

---

Originally posted by The_Fog:
No matter if we succeed or not, we have at least gained a whole
lot of experience.

---

Good nice sayin', dude!
And by "our site", guess we do that later. We need plans.

Pony:
U r rite. So we need a planner, rite? Volunteers, anyone?
Or otherwise I guess we need to wait for Jacsoft's AMP first?

Well I'm still working on my compression method, but I thought of an other compression method we could give a try.

This best compression is a compression method wich takes x bytes of compressed data and make it < x. So If we take x bytes and represent them as bit string (instead of a number). Then we start leaf off the first couple of 0's (cause we can get it back cause we now that all data have a standard length and if we decompress we just count how much bit we have and if it's < x*8 then we just fill it up to the left with 0's) We start looking for a pattern (will not have to be completely good) wich we will save in the first bit. This can be done by a kind of Operational Code (Opcode) like a computer uses itself. So we can say for example that 0001 = 10101010101 and 0010 = 100100100100.

so if we have a code like 101101010101. We could save it as pattern 0001. Then we only have to save were the pattern differs.

This might look stupid cause then we loose much data. But now we have kind of a structure. Now we start looking for another pattern and do kinda the same. If we say we take 4 bits for a OPcode then it we would take 8 bytes as the data to compress and say we takes 7 bytes to compress it into (were pattern 0000 stands for no more) then we could save 2*7 = 14 patterns.
I think this could also work, but just a quick second thought.

And dafhi, that is what most people may have thought of and also I have done much of research on that topic but I have never found a way to compress all strings into 'primes' cause if you for example take the list of all prime numbers then how would you like to construct the number 4,8 or 256 cause 4 is 2*2 and that could not be saved in a prime list and 8 = 2*2*2 and so on, so believe me I think there is no such option, or else try it for yourself.

About my AMP projekt. It would really help a lot if one of the members here has good an always online internet connection with an internet ip wich will kinda 'host', this will save not only we some time, but also us self, cause else the first person only must each time invite one if one other person comes online, but it can be done otherwisem but it will only take me a lot of time.

hmmm...seems that i have a permanent connection but how does your AMP thing work really...will it keep the firewall busy or wat... please give me details. And since we are keeping high value info here I suppose it would be wise if you give me the source code of this AMP too before it can be activated...

Quote:

---

Originally posted by vbud
hmmm...seems that i have a permanent connection but how does your AMP thing work really...will it keep the firewall busy or wat... please give me details. And since we are keeping high value info here I suppose it would be wise if you give me the source code of this AMP too before it can be activated...

---

Well I can't give you the source code out of security reasons, cause the program supports plug-ins and I use all kind of security for it and if I give the source code (very large btw) then I am worried that my security will be hacked, so I made a deal with my fellow programmers that we won't give the source code away.

And if you have a firewall then yes everytime one of your buddies (in this case fellow programmers of the algoritm) comes online your firewall will pop-up I think, cause you have to get a connection with each user (Peer 2 Peer), cause that's the only way not to use a server. The thing is my program is just a Peer 2 Peer chat program with file resuming chatting drawing etc. But when I have a standard Ip from one of the members (or at least an always online internet ip) then I can let everyone connect there and let them auto join the multichat. Otherwise each time someone comes online one person has to invite him/her and when the person who invites him/her leaves then all the persons he/she has invited will also disappear. So then it's kinda complicated to create a multichat. But if no one does then I have to make a protocol with the webmaster of amp to let this all be done by the server.


But I can see your concern. We found out that AMP works for 95 % the same like KAZAA only without banner/spyware and less users, But with plug-in features, chatting right MSN messenger sending complete directories etc. So

Fear not.
I'm developing another IM program. It employs my homegrown
technology, which doesn't require any host/permanent connection
yet we can still *see* each other on the network.

Simply say, it sort of make use of other's server to sort of *host*
our conversation. Look, I said make use, but I'm sure it's a legal
thing. Gonna apply patent 4 it hehe...but still under development,

i'm still developing it's core engine, but I decided to hold it and
create an IM which pretty much employs the same technique but
easier, so I think I can finish it in a couple days.

Jacsoft:
could u elaborate more on dat algo? I almost got ur point. Did u
mean this:
If we have 00100101, we remove the first two 0's, making it like
100101? So how r we gonna determine if this bit string lost two
0's?
Search the bit stream for a pattern such as 10101010101 which
may be denoted by 0001? After 0001 we put a 0000 to tell the
decoder that no more such denotion, later bits are not required
to be decoded?
Correct me if I'm wrong.

How 'bout those numbers' that's powered? eg 4,9,16,25,...

Believe me dafhi I have done research in all those kind of compressions but I found out (and heard from a fellow student) that this cannot be done with just mathematical functions (Philips even proved this). So believe me your can try as much as you want but you cannot compress text with just math functions and get the good data back.


Jihan:
[qoute]could u elaborate more on dat algo? I almost got ur point. Did u
mean this:
If we have 00100101, we remove the first two 0's, making it like
100101? So how r we gonna determine if this bit string lost two
0's?
Search the bit stream for a pattern such as 10101010101 which
may be denoted by 0001? After 0001 we put a 0000 to tell the
decoder that no more such denotion, later bits are not required
to be decoded?
Correct me if I'm wrong.[/qoute]

Yes you are right Johan, cause of the fact we would just store the decimal length in bits and if we compress fixed length strings then we now how long the decompression string must be. So lets say we compress allways 2 bytes (won't work by te way) and we have the number 0000001010101101, then we chop of the 0's so we have 1010101101. We save that (somehow (in ascii)) in a file and when decompresseing when we have a char and it represnets the number 1010101101 then we now we have not yet 16 bits so we now that on the left hand we must add enough 0's to make it correct again.

About the IM, mine is also kinda using a server in case some people are behind a gateway within a LAN it works great, but the problem is when you connect through the internet and one is behind a gateway (so has an internet ip) then you can never connect with it, so we also use a server (a user with an internet ip) just like Kazaa, but the problem is when that user goes offline we have a problem, I fixed it, but I think you haven't thought about that, that's a big problem.

The problem with your IM is that hosting a connection some where else that can see all data that is to be sended and that's not an option I think.

It's possible that they 'proved this' much in the same way that mathematicians using the Roman Numeral system 'prove' that more symbols are necessary to distinguish higher numbers.

That is why it is important to develop a new numbering system. Then it will work.

No, it also works like P2P. Only that it uses a very special method
to let u find your buddies even if you don't know their IPs on a
network, in this case, the Internet.
I don't use any clients as the server, so if anyone disconnects,
it doesn't affect others. It's just that my special method will sort
of *host* all the clients. It's not hosting but looks like one.
Actually, it doesn't use any server to host.
Dont care 'bout that, u'll c later :D

OK, jacsoft I C ur point. So this decimal length denotes the
quantity of zeros being chopped off or wut? Or perhaps we only
chopped a fixed-length of zeros? And so how'bout those that
chars doesn't require chopping?
I've no idea on how u encode them into the file. You need special
"ID-bit" to tell the decoder which bit strings' zeros've been cut
off or not.

Sorry, I'm kinda busy with studying lately (got exams) so I won't be around here much.

Well on the internet you cannot connect to someone without his/her ip, so you need a server somewhere even to check if password / username is correct

No with a fixed length of compressing one can recover the data cause when you have decoded it you just count the number of bits and if that's not enough then add with 0's.

But I think there isn't I good algoritm with can decompress a random string of let's say 8 bytes to 7 (what I'm trying). So now I'm working with opcodes and a multiple of compression methods, like pattern matching longest bit string, math functions etc, I think working with opcodes is the best way for a good compression algo.

Oh...2 bad :(
Wut's this opcode?
Isn't it CPU operation codes?

Yeah, It's kinda the CPU operational code.

SO If we have for example only 4 bits. Then you can save for example the 4 2 bits to store wich action is to be performed and the last to bits for the data. So you can do 4 different actions (2^2 ) with 2 bits (4 data capacities).

Yea I've this very same idea a year back but got stucked
coz' later I found out not all data can be encoded this way,
only a certain entropy data...So I thought perhaps I need
additional bits to indicate different encoding method so each
encoding method will suit different random datas perfectly.
For example, if I found out that currently processed input bit stream
is best/suitable compressed with certain method then I'll use that
method and also tell the decoder that for these bit streams I'm
deploying what kind of compression method.

Yep that's the only right idea. Now I'm trying to find a way to compress each random 8 byte data string into a 7 byte. If that would work......
But the problem is how to find a couple of compression methods witch can make a 8 byte string into a 7 byte string.

Like I've said, not all data can be compressed into 7/8
Otherwise maybe 4gigs of data may be compressed till 1 bit
haha, nevertheless, combining multiple methods we'll get it to
7/8 :) Heard of Zip Zipper? That's wut we're gonna do :D

So I'm thinking, does the algorithms that winzip had deployed
had already reach the compression limits?

No never heard of Zip Zipper, what is it? Just a zip program voor zip files or something.

Of'coz u never heard of it. I made it up :D
Wut I mean is, we're gonna develop this kind of program, which
is gonna compress further of wut winzip had already produced/
compressed. :D

Well then we must first find out how WinZip is compressing :)

not necessary.
instead of analyzing winzip's output. See what kind of pattern
occurs in it or whatever we can do about it.