Summary: I am multiplying 2 matrices in SciDB. Previously I recorded the calculation ran in 5 hours, but now I am observing / estimating it to run in 33 hours. This post is a description of my investigation and my attempt to speed up the calculation by reconfiguring SciDB to use more processors, and ultimately running the calculation piecemeal so I could monitor its progress.
Update: Using a system with 4 worker instances instead of 1 decreased the time by approximately a factor of 3.
Further Update: adding the specifier 'dense' to the multiply command increased the speed further by a factor of 1.5
BackgroundI have 2 matrices in SciDB that I want to multiply:
particleStem_3 is 873637 x 42315
eigVect_3 is 42315 x 100