Compression in VB6: modern solutions

[Tanner_H]

Overview:

This project provides convenient VB6 wrappers for multiple compression APIs, including zLib, zLib-ng, zstd, lz4, and the Microsoft Compression APIs (Win 8+). A small sample project allows you to compare compression time, decompression time, and compression ratio across all libraries. Just drag+drop a file onto the text box to test it.

This project is also available on GitHub. The GitHub version may have additional improvements and features not available here (because the project is too large to fit within vbforums attachment size limits).

Long version:

Compression has never been an easy task for VB developers. Until Windows 8, Microsoft provided very little support for it, and while 3rd-party solutions have long been available, they tend to be either very expensive or very complicated.

Because of this, VB6 developers typically use the free zLib compression library (http://zlib.net/). zLib is an open-source compression library with a very permissive license, and it is "good enough" for most tasks: decent compression ratios, but relatively slow performance.

Unfortunately, in recent years, zLib has lost some of its luster. The stdcall variant of zLib hasn't been updated in over a decade, meaning it contains serious security vulnerabilities. You can always compile your own version of zLib from the latest source code, but the core library definitions are bugged, so this requires C experience and a lot of patience. (Also, zLib's source code is infrequently updated, and there are a large number of bug fixes that have yet to be incorporated.)

And even if you do manage to survive all this and successfully build an up-to-date version of zLib, you're still left with compression technology that is now 20+ years old. Compression technology has advanced greatly since 1995 (when zLib first released), and we now have open-source libraries that outperform zLib in every way.

This project aims to make those libraries available to VB6 developers. It provides a "Compression" module that wraps five different open-source compression libraries: zLib, zLib-ng, zstd, lz4, and lz4_hc. If you're on Windows 8 or newer, it also wraps the four compression engines provided by the Windows Compression API. All compression/decompression functions are unified so you can simply call the "Compress" function, and pass an enum that specifies which compression engine you want to use. The Compression module takes care of the rest.

To simplify this demo, precompiled DLLs are provided for each 3rd-party library. Because the 3rd-party libraries are all open-source projects (links to code below), I believe these still meet the vbforums requirements for precompiled binaries. You are of course free to compile these yourself, from the latest source code, but you will need a modern copy of Visual Studio, some knowledge of compiling C code, and you must manually modify the project files to build stdcall variants. (Like most libraries, they all default to cdecl.)

The 3rd-party DLLs are all bare C libraries, so they do not need to be registered on target PCs. Simply ship them in a subfolder of your project - for example, this demo project uses a "\Plugins\" subfolder, and the DLLs are loaded at run-time via LoadLibrary.

As for the Windows Compression APIs - they are not redistributable, so you can only use them in programs running on Windows 8, 8.1, or 10. Sorry. This project will automatically enable them on compatible systems, so if you don't see them, load the project again on a Windows 8+ PC.

Here is a brief overview of the supported compression libraries, and aside from the Microsoft ones, all are 100% open-source and free to use in personal or commercial projects (with attribution - see the included license files for details).

- zLib is the granddaddy of open-source compression libraries. The version supplied here is v1.2.8, which is slightly out-of-date relative to zLib master, but zLib master has some concerning bug reports that make me leery to recommend it. Despite its age, zLib remains a solid general-purpose compression library, with good compression ratios across a wide variety of data, but slow compression speeds compared to the competition. Generally speaking, there is no longer much reason to use it, unless you specifically need the DEFLATE algorithm it provides (e.g. to work with .gz files).

- zLib-ng is an attempt to create a "next-generation" version of zLib, with an emphasis on improved performance and much-needed code cleanup. zLib-ng has yet to release a "stable" build, so I've simply included the most recent developer build. Unfortunately, most of the zLib-ng improvements have thus far focused on 64-bit builds, so I can't guarantee that you'll see any huge improvements. Feedback welcome if you find workloads where it does especially well.

- zstd (or "zstandard") is a modern replacement for zLib. It was originally developed by Yann Collet, and its ongoing development is now sponsored by Facebook. It is 100% open-source and BSD licensed. zstd is significantly faster than zLib at both compression and decompression, and it also achieves better compression ratios. It provides a "compression level" parameter just like zLib, but with a much wider range, including extremely slow speeds but extremely good compression ratios if you need that sort of thing. For most users, zstd could replace zLib in their existing projects, and they'd immediately get a "free" performance boost from it.

- lz4 is a real-time compression engine that emphasizes performance above all else. It was also developed by Yann Collet, and it is also 100% open-source and BSD licensed. lz4 is so fast that it is now used for OS-level compression (Linux), file system compression (OpenZFS, SquashFS), database compression (MySQL), RAM caching (Emscripten, ZRam), and a whole bunch of video games (Battlefield 4, Black Ops 3, etc). LZ4's speed comes at a trade-off, however - it does not compress as well as zLib or zstd on most data. It also provides an adjustable "compression level" parameter, but instead of providing "slower but better" compression as you increase this value, lz4 provides "faster but worse" compression. It is the best solution when speed is paramount. (For example, lz4 is one of the few algorithms fast enough to provide a performance benefit vs raw uncompressed data when reading/writing to a hard drive.)

- lz4_hc comes "for free" with lz4. It is a "high-compression" variant of lz4, with much better compression ratios but much slower compression speeds. Decompression speed remains the same. It is a great solution for "compress once, decompress many" scenarios. (This is the version that various video game engines use, for example.)

- The Windows Compression API available in Windows 8+ provides four different compression algorithms: MSZIP, XPRESS, XPRESS_HUFF, and LZMS. The pros and cons of each algorithm are described on MSDN.

The included demo project allows you to compare compression speed, decompression speed, and compression ratio across all libraries. A baseline comparison of "no compression" is also provided, which measures timing against bare RtlMoveMemory calls. I've included a UTF-8 XML file for comparison (because it's small enough to fit inside vbforum size limits), but for best results, you should test some of your own files. Just drag-and-drop a file onto the project window to run an automated test across all selected libraries.

Checkboxes allow you to toggle various test settings. Large files may take awhile to test. Results can be sorted by various criteria.

At present, the Compression module operates entirely on byte arrays and/or bare pointers (passed using VarPtr()). This makes it trivial to compress source data of any size or type. Specialized functions for Strings or other data types could always be added, but for now, those are left as an exercise to the reader.

Bug reports and feedback welcome, of course. Thank you to everyone who has contributed feedback so far.

Updates:

06 September 2017: Added zlib-ng to the list of compressors
                    Added ability to sort results by various criteria (time, ratio, etc)
                    Updated lz4 and zstd builds
                    Minor bug-fixes and code clean-up
12 December 2016: Fixed text box scroll behavior in test program.
                    (Thank you to Steve Grant for reporting.)
                  Added functions to report each library's default, minimum, and maximum compression settings.  
                  Also updated the test framework to let you test these settings.  
                    (Thank you to Arnoutdv for the suggestion.)
                  Added support for the Windows Compression API, available on Win 8 or newer.  
                    (Thank you to dilettante for the idea.)
11 December 2016: Recompile liblz4.dll to fix issues on old OS versions.  
                    (Thank you to Steve Grant for reporting.)
10 December 2016: Initial release

Download here: