It's really hard to tell from your description. A good description of the noise and when it happens would be a good start.
You already said that the noise happens when you play a note. What happens if the volume is turned down? Either using the control on the electronic piano or by using the pot in your circuit. If the noise goes away then you can assume that the problem is either with the signal level on the input, or that you're asking the amp to put out more than the rated 3.5 watts.
If I were a betting man, I would guess this is the case. You're using a 3.5 watt amp to power a 15 watt speaker. Normally for a 15 watt speaker you would be using a 15 to 30 watt amp. So your amp is undersized.
Adding a cap like what @Manmanguruman suggests isn't a bad idea, but I would hope that there is already some caps inside the amp module. Still, you would use the largest cap that is somewhat practical. Start with about 470 uF and higher.
The pot should have a log taper, not a linear taper. Changing the pot will make the volume control more useful-- but will do nothing for your noise problem.
It is also possible that you have a signal loading problem with the piano. Basically, the piano might not be able to handle having an amp plugged into it. I would hope the piano was designed better, but I have seen some products do some stupid things. Turning the volume down with your linear/log pot should help this issue.
Other than that, the only thing I can think of would be a component failure. Something like the speaker, amp, battery, or piano actually being broken in some way.
Not only can HF noise influence your signal, it will influence your signal.
Only if a signal is phase locked to another will they have no dynamic influence on each other. Since your SMPS will create either one fixed frequency, or at the very least one completely unrelated to the audio you are making (Fun idea to try and lock them maybe, but utterly infeasible), there will always be interference from that noise into the samples generated.
It is also why good, medium-high to high resolution mixed signal designs include completely separated digital and analogue power systems.
If you have a HF transient on the analogue power right between (sensitive) samples, such as made by an SMPS at 1MHz, or by a digital parallel (or serial) bus, the difference in the samples will jump up or down a bit. By how much would greatly depend on your DAC. Having any of those transients on the analogue reference voltage will most certainly have a sample-to-sample influence!
This HF noise may not propagate fully or "as-is" through your analogue system, but it will most certainly have an effect on the audio output if it has any remaining significance to it.
1/1000th on the power rail, or 1/10000th on the analogue reference should be your absolute maximum, in my opinion, but in all fairness that's just a touchy-feely judgement call from someone not specifically an audio engineer. So that may be off by a decade if you talk to an actual audio pro, just don't get caught into Audiophoolery trying to make noise μV per kV or some such.
As for filtering, that can be easy enough:
simulate this circuit – Schematic created using CircuitLab
Here I chose to use a coupled inductor, because the total inductance for a noise signal through the coupled conductor will be much higher for smaller/cheaper inductors. You could add a chip-bead in series with it to eliminate very high frequency noise to a higher degree. You can also include a chip-bead with an extra 100nF to 1μF at each chip/device/stage that uses the power to also isolate any digital noise picked up in DACs and such, but in an audio system that's often going well and truly over-board.
The values in this schematic are a quick back-of-head estimate based on a simplicifation of the real part of the "resistance" each element adds to the system. You can go into transfer functions, but I used a 50μH total coupled inductance for the coil and came to:
Z(C) = 1 / (jwC), means for a quick estimation the apparent resistance for a capacitor will come close to: "R(C)" = 1 / (2*pi* C). Deeper theories apply, but that would make this answer take several hours I don't have right now.
Z(L) = jwL, comes to: "R(L)" = 2*pi* L.
Which gives:
C1 will present as nearly equivalent to a 0.72 Ohm resistance at 10kHz.
C2 will present as nearly equivalent to a 0.16 Ohm resistance at 10kHz.
A coupled (or single) inductance of total effective 50uH will present as nearly equivalent to a 3.1 Ohm resistance at 10kHz.
This means that a 10kHz signal will be attenuated by about a factor of 20 by the inductance + C2 stage alone. Basically C1 is just there to give the high frequency components close to the filter somewhere to go, but as you can see with a 0.5Ohm resistance in cables and such, it already helps to filter noise. A factor 20 in voltage would, if I am not mistaken, be equivalent to -26dB. At higher frequencies this will quickly snow-ball into -70dB or better.
For well chosen parts, up to several MHz, the resistance in the capacitors will be divided by each multiplication in frequency. So 0.072 Ohm for 100kHz for C1, etc. Whereas the "apparent resistance" of the inductance will multiply, so 31 Ohm for 100kHz, etc.
It is important to realise though, that at these low apparent resistances the ESR/DC-resistance of the components has a real influence. If the capacitors have 5 Ohm ESR, then the filtering will be severely limited, so you may want to make the 100uF capacitance a small group of smaller caps. Possibly including a couple of 1uF/100nF ceramics to really get the sting out of MHz level noise. Also take note that long winding wires or traces to the capactiors from the power will increase resistance and wire inductance, to again limit the filtering capability.
Of course, the 3dB point and the total suppression at frequency X or Y will be much better represented when using proper filter maths, but it's my work day and writing that all up would take too much time, where this comes close enough. Look up LC filtering tutorials or Q&A's for deeper details into the proper maths.
If you have a separate digital system that can be powered independantly from the analogue supply, it is best to "tap" that supply from the raw and noisy supply and filter it independantly. One reason is that the digital system will need much less filtering. Another reason is that then the digital transients on power will by separated from the analogue power by 2 independent L-C-filters.
Although you can see that having a simple 50Hz/60Hz transformer that already acts as a huge filtering coil to AC-transients makes filtering a very simple matter of 0.2 Ohm wire resistance and 10mF of capacitance to basically get no-noise. One of the reasons SMPS's have not been used in audio systems for so long. "Tried and true" argument.
Best Answer
I found BC859/BC860 with a search on the NXP web site. They seem similar to the BC108 which used to be the low-noise device most people used to use.