The general Japanese term is "hijikake", which seems to just mean "armrest". The long cushion is called "chanchin" and the cube is "tanchin", but Google Does Not Like Me.
sigh I can't find anybody selling them in North America, apparently; I can't find the specific names for the armrest cushions (which is what I actually want); nor how they're constructed. At least I got closer.
I have been googling and I can find these pillows online when I search for Traditional Korean Block Pillows but I have not yet figured out what they are called. There is a possibility that those are traditional style Buckwheat pillows. Usually they used wooden "pillows" called Mok'chim but these are not wood unless they have a fabric covering to look nicer.