Hi Tronic,
Yes, the manual can be used to get the Formats for my Digitizer, I could not find my manual (found and bought another CalComp Digitizer on eBay with manual - will have it in a few days). So I used this PDF you also found, myself. I am very familiar with these Digitizers, as we used them to digitize artwork for laser shows from the early 80's to 2003, where I did R&D Engineering.
I believe the tablet is set up for "Format 23: 9500 Format #5", but as you can see a few are very similar and only vary in low / high byte pattern and resolution based on the size of the Digitizer (12" x 12" to more than 22" x 34"), and thus the high order bits may not be used, and on my Digitizer bits 16, 15, 14 for x and y are not used (I am 98% sure, the manual will tell me more). This Digitizer has a menu strip at the top, and while I can read what has been set, the "Format Mode" is a number between 1 and 9, and not the number of the Format - so that is why I need the manual.
So from the PDF you found, you can see that byte 3 is not used (proximity is not working, as it is a setup mode which I need the manual for, but merely tells the user if the cursor is not getting data from the Digitizer, i.e. out of range, etc.) And on byte 0, only the cursor info is used, and not the high order x bits. Note that C4 indicates a button has been pressed, and C3, C2, C1, & C0 indicate which of the 16 buttons (0 through F) have been pushed.
The bit mask decoding I am using seems to be working just fine, except for the the fact that when the incoming numbers are converted somewhere, sometimes they evaluate to false which causes the error when you try to perform math on a "false" and not an actual number and probably why Ruby throws and that particular error - it gets a number and everything is fine, but when a "false" enters the math messes up.
The final X and Y values are about 0 to 15239, for example the x position is on byte 1 (xhi 6 bits) and 2 (xlo 6 bits), so xlo + ( xhi * 128) is equal to the x position value, and the same for y on bytes 4 and 5. The number typically does not go up to 16384 due to mechanical limits of the internal PCB, but does meet the value of 0 to 15000 which is 1250 samples per inch over 12" (12" x 1250 = 15000) in high resolution mode.
I hope that helps you, just let me know and I will get more info to you.
Note: I tried to create the simulation array the same way the data enters the Ruby module from the Com Module... but could not capture the array of values properly. I believe that at various times the data may contain empty values, since the data streaming speed from the Digitizer can be changed, but of course the baud rate is always 9600, so data may get extra null or empty values, but I have not 100% verified this...
I have included the latest version code as well...
Thanks so much for helping me on this !
Aron