Printable View
In order to decode a file I must work with nibbles, so I have to do some byte shifting and-ing and or-ing. Because the files are large, around 10 Mb, speed really counts. I thought it would be faster if I could directly work on nibbles, but no such data type exists in VB. Could there be some way around? Btw I do the bit shifting by multiplying or dividing by powers of 2 as there's no shift operator as far as I know.
Does your data allow you to use a "brute-force" method? That is, if you only have a limited number of values in your data set you can hard-code the processing for each value.
Does this make sense?
The number of values is limited, of course, but it can be different according to the file I'm working with. But I'm afraid I don't know what you mean by that hard-coding, can you set an example?Quote:
Originally Posted by trisuglow
I was just thinking that if there was some kind of relationship between the lower and upper nybbles then you could know the upper nybble just by looking at the lower nybble. It sounds like this isn't going to work for you though. Since its not going to be applicable I can't give an example. Sorry.
Nibbles are very easy to work with...
00001111 = 15
11110000 = 240
Those are the masks used to pull just the lower order or high order nibble out of the byte.
lngLowOrder = lngValue and 15 ' lngLowOrder now has only the low order nibble
lngHighOrder = (lngValue and 240) / 16 ' lngHighOrder now has the high order nibble and it's bit shifted down
Yes, that's exactly how I'm doing it, but the aim of this post was rather to make sure there wasn't some trick to achieve the same result but faster. The thing is I'm trying to optimize in terms of speed this file decoding I've mentioned in the first post, and one of the possibilities was better manipulation of nibbles.Quote:
Originally Posted by szlamany
That is the best you are going to get out of a language like VB.
It the nibble work you need to do can be done in ASM (or maybe C) then a routine in that language might be better.
But the last thing you want to do is start calling a function (expensive to call) just to work a byte - it would have to do all the work before coming back to VB (or at least a large part of it).
A number of years ago I used Microsoft C 6.0 in a IDE running on DOS. But I don't know how I'd call a sub written in C from VB. Would I have to make a DLL or something?Quote:
Originally Posted by szlamany
Yes, make a DLL with C. It is likely to be 10-100 times faster to do string parsing in C than in VB.
Well, it's not exactly string parsing what I've got to do but the nibble operations mentioned above.Quote:
Originally Posted by Dave Sell
Now I come to think of it, I know how to make a dll in the IDE of VB6 but how would I go about it in c? Do I have to install the VC part of Visual Studio or is there a more straightforward way in the old DOS-C itself?
Anything resembling traversing a ginormous array of bytes is considered string-parsing (an old C nomenclature).Quote:
Originally Posted by krtxmrtz
I am not an expert making C or C++ DLLs but I have done it a few times in the past. I searched around and found plenty of tutorials and followed them. They always had me using VS C++ 6. I recommend you go that route also. C++ is very fast - I consider it the same speed as C when all you are doing is string-parsing.
OK, I'll have a go at it. Thanks.Quote:
Originally Posted by Dave Sell
I'm sure you know this, krtxmrtz, but szlamany makes an excellent point. Do not make a function that accepts data byte per byte; rather, give the DLL your entire buffer, or at least a KiloByte at a time...Quote:
Originally Posted by szlamany
Yes I was aware of this but it's one of those things you so easily tend to overlook so, thanks for reminding me of it.Quote:
Originally Posted by Dave Sell
Hi,Quote:
Originally Posted by Dave Sell
Just curious as to WHY "C" is faster than VB. I know it's true, I've just never learned the why. Any insight or any links to info on this would be appreciated.
I'm curious too, I don't understand why VB compilers couldn't be designed to be as efficient as c compilers. What's he difference?Quote:
Originally Posted by SideOut
I don't use C, but I imagine it's how the language can look at a spot in memory.Quote:
Originally Posted by SideOut
Back in our mainframe BASIC days we wrote little PEEK and POKE functions - pass the function the "location" of a string and the offset into that string you wanted to touch and it would pull a byte, word or longword from it.
In VB (I assume) the only way to do that is to LEFT/MID/RIGHT the string, which makes a new string (probably an 8 byte memory structure just to hold the byte, word or longword) and then some other FUNCTION call (ASC?) to convert it to a value.
That type of requirement to get at a memory location to play with bytes is expensive.
I would choose to develop the routines needed in this thread in ASM myself - but from what I hear C can imbed ASM and ASM-like call directly.
Could you possibly post some of the code you're using? It's impossible for us to come up with any optimization without seeing the code you're using.
Ahhhhh... So it's this kind of overhead that does it !Quote:
Originally Posted by szlamany
Thankyou - Very well put.
Yes - what seems so routine in higher level languages - a simple call to a VB function, or a SUB/Function in a module you write yourself causes all kinds of memory framing action so that "new variable" space can be created - arguments passed (by ref less expensive - by value more expensive).Quote:
Originally Posted by SideOut
It's so common on this forum for suggestions to use SPLIT to find things in a string. SPLIT creates an array of new strings - that's expensive.
It's also common in the VB world to use REDIM with PRESERVE - that's extremely expensive as well. Most likely moving the entire array to a new large or smaller location.
This is the core of it.Quote:
Originally Posted by Joacim Andersson
What this does is, it reads a file where a matrix of values z=f(x,y) is
stored. Now, these values have a 12-bit precision and in order to save
some space, they are packed in 3 bytes -call them b0, b1 and b2- in
this fashion:
The first 12-bit value has its 8 low order bits in b0 and its 4 high order
bits in the high order nibble of b2. The second 12-bit value has its 8 low
order bits in b1 and its 4 high order bits in the low order nibble of b2.
I hope this is clear at least from the attached drawing (sorry about this large size, I didn't mean to make it like that)
VB Code:
By the way...
If I am to write one or two DLLs in C I'd like someone to recommend a tutorial, but not just a C tutorial as I more or less know the syntax and structure of the language. What I'd like is some push to get blitz started in the VC environment of Visual Studio.
I did some changes to your code but since I can't test it it's hard if these will make any noticable differences (even though it should be faster).The largest different here is that I took away the NormdData array. I don't know if you need it later on in which case you can simply remove the comment. Some other improvments you might want to think about is to remove the Array bound check, the Integer overflow check, and the Floating point error check. Of course those optimisations will not be noticable until you compile the program.VB Code:
Well, I'll stick this into my own code and see what comes out from the tests. In the meantime I've been playing around and I'm wondering if anything is to be gained by using a 1D rather than 2D DataByte array, but that'll take me some time to implement. I'll keep you updated...Quote:
Originally Posted by Joacim Andersson
I think it's correct because if you callQuote:
Originally Posted by Joacim Andersson
N = nSize - HeaderSize
then, this N is the total number of data bytes. For every 3 of these, 2 Val12bit values are built so, first you divide by 3 to get the number of triads,
N\3 triads
from each triad you get 2 Val12bit values so, the number of those is
2*N\3
and finally you've got to redim from 0 to 2*N\3 -1
This is one reason why I find this feature confusing, starting arrays at 0 rather than at 1, yet I've been using 0-based arrays recently, I suppose it's more convenient for me to adjust to what most people do.
You might notice that it's running faster in the IDE but I doubt very much that it would make any difference when you've compiled it to native code since the machine code doesn't know the difference between a 1D and a 2D arrayQuote:
Originally Posted by krtxmrtz
That means that the compiler will probably create the same native code regardless of how many dimensions your array have.
Yeah you're right. I wasn't thinking correctly since the array is zero based. Change this line (if you're using my code)Quote:
Originally Posted by krtxmrtz
VB Code:
To answer the question why C is faster than VB, C has no overhead of Classes, COM, goofy data structures that represent a String.
In VB 6, a string is a pointer to a structure with a size variable and then another pointer to the actual array of bytes. Also, you cannot just talk about the byte at position 6. You have to call functions that call other functions and make new data structures, and the list goes on and on.
In C, you start with an array of bytes. You can talk about each byte in the array as an integer offset to the array. It is the fastest way possible to traverse an array byte by byte. No goofy VB overhead.
For pure string-parsing, nothing beats C, not even assembler.
I don't know if this helps you or not, but here is one way you can create your C++ dll from VC++
let's assume dll is called MyDll.dll
Create a new Win32 dll
Add the following code to the cpp file
Notice that the rows and columns are reversed in C++Code:#include <stdio.h>
#include <math.h>
#include <stdlib.h>
/******************************************************************************
declare Sub ProcessData Lib "MyDll.dll" (DataByte as Byte, Val12bit as Long, NormdData as Single, GreyLevel as Byte, ByVal BufferSize as Long)
******************************************************************************/
__declspec( dllexport ) void _stdcall ProcessData (char DataByte[][3], short Val12bit[], float NormdData[], char GreyLevel[], long BufferSize)
{
//OK do your magic
return;
}
Then from VB declare your new sub
And call it like soVB Code:
Code:EXPORTS
ProcessData
If you need help translating the magic I can help too.
I find this quite surprising not to say hard to believe. If c is so efficient at string parsing you'd only have to take the compiled c code and convert it to assembler, on the basis that each machine instruction corresponds to an actual assembler instruction... But my assembler days belong to the past and to the another computer architecture so I may overlook some of the facts.Quote:
Originally Posted by Dave Sell
If you take a string parsing routine written in C, say a for loop using integer offsets to the base address of the string (array of bytes), and then decompile it to assembler, you will get about a 1-1 ratio of C code to assembler code. In other words, you may not get 1-line to 1-line correspondence, but you would not be able to reduce the assembler any further. C can be extremely close to assembler in terms of efficiency.Quote:
Originally Posted by krtxmrtz
Well, I used to write C code for DOS a few years ago so I hope I can still work out the magic as you say, but finding my way in the VC++ IDE without any background and coming from VB is not so straighforward so your post came in like the answer to my prayers... :thumb: Thanks a lot.Quote:
Originally Posted by moeur
Oh I see what you mean, C is as efficient as assembler. I thought you meant assembler was less efficient than C, silly me !!! :blush:Quote:
Originally Posted by Dave Sell
In many cases a C compiler will produce faster code than you will produce writing in assembly language yourself. This is beacuse the compiler knows a lot of tricks that you don't. I've actually tested it.
This is what I've tried so far (I have made a few changes)Quote:
Originally Posted by moeur
But when I try to add a reference to the dll an error message shows up, probably I didn't add this MyDll.def file in the correct way: I just added a text file with the text you posted.Code:#include <stdio.h>
#include <math.h>
#include <stdlib.h>
/************************************************************************************************************************
declare Sub ProcessData Lib "MyDll.dll" (DataByte as Byte, Val12bit as Long, Max12bit as Long, ByVal BufferSize as Long)
*************************************************************************************************************************/
__declspec( dllexport ) void _stdcall ProcessData (char DataByte[][3], short Val12bit[], long Max12bit,long BufferSize)
{
int i;
int j;
Max12bit=0;
for (i=0;i<=BufferSize;i++)
j=2*i;
Val12bit[j] = (DataByte[i][2] & 240) * 16 | DataByte[i][0];
Val12bit[j + 1] = (DataByte[i][2] & 15) * 256 | DataByte[i][1];
if (Max12bit < Val12bit[j])
Max12bit = Val12bit[j];
if (Max12bit < Val12bit[j + 1])
Max12bit = Val12bit[j + 1];
return;
}
You get an error from VB?
What is the error?
After creating the def file, add it to your project
Project\Add to Project\Files
Files of Type\Definition Files (.def)
This should add the following to
Project\Settings\Link\Project Options
/def:".\MyDll.def"
It's been a while since I did this but if you want to expose a function in a DLL don't you need to include windows.h? As I see it there is no reason at all to include stdio.h or math.h since you don't use any functions in those libraries.
NoQuote:
if you want to expose a function in a DLL don't you need to include windows.h
This is true, but it doesn't hurt to include them since he may be adding code later that needs them. Or not...Quote:
there is no reason at all to include stdio.h or math.h since you don't use any functions in those libraries.
Well, I doubt very much that he will need any functions from the stdio.h library since that's an old C library used for IO in the console world (which BTW is subsided by the iostrem.h in C++).
Anyway thanks for the info about not needing the windows.h library to expose a Windows DLL function (I wanted to give you a rating for that, but I wasn't allowed to since I rated you earlier and I haven't rated enough people to give you a new one :)), but I do still believe that windows.h is more valuable then stdio.h in the Windows world :)
I have to make sure what exactly the error was but at the moment I'm sitting in front of a computer where VC++ is not installed. I think it was something like "It was not possible to add a reference to this library" when I tried to add the reference in the VB project/references menu.Quote:
Originally Posted by moeur
As I said, I had to put the
EXPORTS
ProcessData
lines in a text file. But when I tried to add it to the project there were no def files among the different types available (to be sure, I'd had a couple of beers before that) so I added it as a cpp code file.