I'm a little confused. You're using two decks with 20 cards each with only Ace, K, Q, J, 10 for a total of 40 cards and the hands are 12 cards?

I'm a little rusty but assuming the above I would do:

Number of Hands that have 3 of a kind of 4 different cards
= (8,3) (5,1) (8,3) (4,1) (8,3) (3,1) (8,3) (2,1)

Where (n,m) = number of ways of choosing m items from n choices (nCr)

This equals 56 x 5 x 56 x 4 x 56 x 3 x 56 x 2 = 56^4(5!) = 1180139520

Now you have to subtract how many of those hands contain the three 10's which is 5.

So we are left with 1180139515

Total Hands = (40,12) = 5586853480

Probability = 1180139520 / 5586853480 = 0.211

There must be something wrong here either in the assumptions about the deck/hands or in my calculations since this is way higher than you observed. Can you clarify how many cards are in the deck and how many cards are in a hand?