Without a schematic I cannot determine if your grounding is correct, i.e. common ground, a common reason for some of the symptoms you describe.
Another point to consider is the liberal (correct) use of electrolytic capacitors and bypass capacitors.
Using a battery-only supply does not guarantee clean power supply rails. Are we sure the battery has adequate reserve available during circuit power up?
Given those conditions are properly met, you might consider the use of a "Supervisory" circuit for your processor.
MCP120/130
• Holds microcontroller in reset until supply voltage reaches stable operating level
• Resets microcontroller during power loss
• Precision monitoring of 3V, 3.3V and 5V systems
• 7 voltage trip points available
• Active low RESET pin
• Open drain output
• Internal pull-up resistor (5 kΩ) for MCP130
• Holds RESET for 350 ms (typical)
• RESET to VCC = 1.0V
• Accuracy of ±125 mV for 5V system
The Microchip Technology Inc. MCP120/130 is a voltage supervisory device designed to keep a microcontroller in reset until the system voltage has reached the proper level and stabilized. It also operates as protection from brown-out conditions when the supply voltage drops below a safe operating level. Both devices are available with a choice of seven different trip voltages and both have open drain outputs. The MCP130 has an internal 5 kΩ pullup resistor. Both devices have active low RESET pins. The MCP120/130 will assert the RESET signal whenever the voltage on the VDD pin is below the trip-point voltage.
They are available in TO-92, SOT-23-2 and 150mil SOIC.
Datasheet: http://www.mouser.com/ds/2/268/11184d-68220.pdf
Very handy little device for $0.60USD, or less!
Best Answer
The big factor that you didn't consider is that "free space" has capacitance and therefore, an open circuit of (say) 1 G ohms shunted with (say) 1 pF (for example) has a bandwidth of: -
$$\dfrac{1}{2\pi RC}$$
Numerically this is 0.159 Hz and the equivalent noise bandwidth is that multiplied by \$\pi/2\$. This restricts noise voltage very significantly.
No it doesn't because of the effect of capacitance.