Your Problem and a note from my friend Carol caught my attention, so i started to look up calibration and basis conversion. I would suggest reading in the Raster Graphics Handbook published several years ago by the Conrac Corp. (ISBN 0-9604972-0-X) It focuses on line scan monitor displays, and has some good starting points applicable to conversions. First listed are the coordinates for the D50 (5000degK) illuminant. (x=.345; y=.352) or (X=.980; Y=1; Z=.861) This is your white point, and you must align your scanner to it. To scale your conversions the RGB values at the white point should be measured. The basis of the conversions is to create an equalateral triangle with the white point in the middle and RBG at the verteces. The next items needed are the coordinates of RBG. If you use the standard CIE recommended points they are R 700nm, G 546nm, B 400nm, and the chromaticity coefficients for these points are listed R (x=.735; y=.265) G (x=.274; y=.717) B (x=.167; y=.009) If we assign to each of these points a coordinate value of 1 and RGB coordinates of 1/3, 1/3, 1/3 to the white point a set of 6 equations can be generated and solveed for a two row three column matrix converting x,y to RGB. My solution was:
K(11: 1.576) K(12: -.749) K(13: 6.268)
K(21: -.598) K(22: 1.681) K(23: -5.196)
If you have a different set of RGB primaries you must determine their x,y coordinates and solve for the matrix values. An alternate procedure would be to align your Illuminant white and measure x,y and RBG values at six well seperated points and throw the measurements into a math computer utility (Derive, Matlab) to solve for the conversion coefficients, or measure several points in the field (about 20(well distributed but near the extremes)) and run a regression analysis. If you can not find the book another reference noted in it is Van Duesen Computer Videographics: Color, Composition, Typography (1981)
Good Luck. I hope you are successful. Bob