(Arduino Uno) Wrong Input on Pin 2 even though it’s not really connected

I'm new to Arduino and I have following problem:

I tried to control a servo with two buttons (left, right) and it worked perfectly. But today I tried to do the same but with some additional LED feedback. (if going left, left LED. If right, right LED)

I connected everything how it should be, but now every time I power the board the left LED is going on and off very quickly and the servo is going to the left. (I'm not pressing anything)

I disconnected the wire, going from the pin(Arduino) to the pinboard. But it's still the same thing happening. (a simple wire without anything on it connected to the Arduino and its telling me that I push the button ?)

I disconnected the LEDs, the servo and the right button and wrote an program to tell me if the button is pushed(1) or not(0). So I can see whats going on.

int pushButton = 2;

void setup() {
  Serial.begin(9600);
  pinMode(pushButton, INPUT);
}
void loop() {
  int buttonState = digitalRead(pushButton);
  Serial.println(buttonState);
  delay(5);
}

It's showing me exactly the same that the LED showed me. Like i would press the Button very quickly a several times per second.

Left Button = Pin2, Right Button = Pin3, Left LED = Pin4, Right LED = Pin5, Servo = Pin13. (all digital of course)

I used 330 Ohms for the LED and 10K Ohm for the Buttons. Everything is connected to the 5V.

The Arduino UNO is connected via USB to my laptop and to an Additional 9V for the Servo.

^{simulate this circuit – Schematic created using CircuitLab}