You have to control the volume on your AV receiver if you use the optical-out, keyboard or system volume only works with analog output.
5.1 signal comes from your source file (music/video) or it should be "converted" by your AV, have you tried to switch modes on it?