If they both use I2C then you can attach them both to the same two pins (SDA and SCL)
You select which one you wish to talk to by using it's address for the first byte of each command (which should be given in the documentation along with example communications) For example on page 16 of the datasheet for the BMP085 this is given as 0xEE (write) and 0xEF (read)
It looks like the first sensor has pullup resistors included which means you don't have to add them yourself.
I take it the Arduino Wire library is a software implementation of I2C (as opposed to a dedicated on board peripheral) If this is the case I imagine you can select which pins you want to use in the defines section. So in your case you would select A4 and A5 as SCL and SDA (which ever way round you want them)
The first link I looked at had various helpful links including quickstart guide, datasheet, wiring example, and an example Arduino sketch, so I'd probably start by following those. To keep it simple, I would try and get that sensor talking to the Arduino then add the other one.
I notice that the BMP085 runs from 3.3V, so you will need to use a 3.3V supply and disable the weak pullups to 5V in the Arduino wire library (see comments on product page)
You can probably run the other sensor from 3.3V too but you would need to confirm this - it should work fine from 5V with logic high to 3.3V anyway if necessary.
As this is only your second time working with electronics, I'll try to keep my terminology simple.
It's hard to see exactly how you have wired up the sensor and LED, but I can take a guess. (If I'm wrong, then everything below probably makes no sense).
The sensor is connected between + power and the Arduino input, while the LED is connected between - power and the Arduino input.
When you press the sensor, electrical current can flow from the + side of the power the Arduino sensor pin, charging it up and giving it a high voltage. Current also flows through the LED, causing it to light up.
Now, what happens When you release the sensor? The electrical charge inside the Arduino sensor pin which was giving it a high voltage, will now flow as current through the LED to - power, bringing the voltage down, so that the Arduino sees you've let go.
But what happens if you don't have an LED in there? The electrical charge in the Arduino sensor pin has nowhere to go, and so it just stays there, and the voltage doesn't change.
The reason the Arduino's sensor pin behaves like this is because it behaves like a tiny capacitor. It can store a small amount of electrical charge, and thus 'remember' the voltage that was placed on them by the sensor.
So, how can you fix it? You'll need to have somewhere for this charge to flow. If not an LED, then a resistor should do. Any value between 1k and 1000k will probably work fine.
Best Answer
Stop thinking Arduino and challenge yourself learning the bare metal microcontroller. 4k flash is not a lot for an Arduino sketch and none of the Arduino libraries are guaranteed to work, although some do. It is not supported though and there is no definitive list of libraries or library calls that work on different microcontrollers.
Check the ATtiny4313 datasheet and search for TWI (two wire interface). The datasheets is like the bible for your microcontroller, everything you ever wanted to know about it is written in there and in my opinion the Atmel datasheets are pretty good to read. From experience I know I2C is a bit tricky to get working, but you can check the Arduino library for how they solved it. The ATtiny and ATmega families are closely related and the code will be reasonably easy to port.
Acutally what you can do is work from your Arduino and port the Arduino specific code line by line until all Arduino specific commands are ported. Easy to test, and you can do it step by step. Then porting from ATmega to ATtiny is pretty straightforward.