Magento – Magento page loading taking too much time

I have magento website. There are no any users (max 2-3 at time).

Our server is: CPU: 2000MHz RAM: 2048Mb HDD: 50000Mb.

I installed ZendServerCE (apc + memcached + Zend Optimizer + Zend Data Cache).
I turned memcached off, because website loaded much worst. I set flat type structure, reindexed and cached data in admin console.

So I have apc + Zend Optimizer + Zend Data Cache.

1. The first problem is I checked runtime how does dispatch work. start_session() calling takes about 500-700ms. Seems its not good result. Why so long, I don't know.

2. I've read this one: http://dev.mysql.com/doc/refman/5.0/en/server-system-variables.html#sysvar_key_buffer_size and figured out optimal options for my server.

Per hour:

Key_read_requests = 8887
Key_reads         = 252
Key_write_request = 187
Key_writes        = 146 

You see that 252/8887 > 0.01, but not too much. It's optimal value I have ever get. Other results started from > 6.

Here is my.cnf:

key_buffer              = 48M
myisam_sort_buffer      = 2M
sort_buffer             = 2M
read_buffer_size        = 2M
join_buffer             = 2M
read_rnd_buffer         = 2M
max_allowed_packet      = 128M
thread_stack            = 192K
thread_cache_size       = 16
query_cache_type        = 1
myisam-recover         = BACKUP
max_connections        = 50
table_cache            = 256
#thread_concurrency     = 10
query_cache_limit       = 8M
query_cache_size        = 98M

3. Memcached for some reason wasn't good. I turned it off. But zend data cache and zend optimizer still work.

4 . APC seems correct. To load controller action takes 3-4seconds for the first time (I set die() there to check it) and for the first time it take 1 – 1.3 seconds.

5 . After several minutes I've restarted mysql I got good result. Pages were loading from 1.5 to 2.5 seconds. But now (after several hours) it takes 6-10 seconds. I cannot find the reason.

So do you see some incorrect configuration here? May be my server doesn't suitable for magento?

UPDATE 1:
about 600 categories and 1000 products today
and about 20000 categories (for different webstores) and 1500-3000 products in the future.

There are not many attributes.

UPDATE 2
I've nocied that ssh console works too slow. I rebooted server and now it works fast. it means I have a problem with RAM. There are not enough space.

It's initial status whiout apache:

             total       used       free     shared    buffers     cached
Mem:          2048        600       1447

UPDATE 3
I got it. Now it is loaded for 0.5-1.5 sec

Here is configuration:
mysql

[mysqld]
key_buffer_size         = 256M
tmp_table_size      = 32M
max_heap_table_size     = 32M
myisam_sort_buffer      = 4M
sort_buffer             = 4M
read_buffer_size        = 4M
join_buffer     = 4M
read_rnd_buffer     = 4M
max_allowed_packet  = 64M
thread_stack        = 192K
thread_cache_size       = 16
query_cache_type        = 1
myisam-recover          = BACKUP
max_connections         = 20
table_cache             = 1024
innodb_buffer_pool_size = 128M
query_cache_limit   = 24M
query_cache_size        = 256M

php

[apc]
apc.stat=1
apc.enabled=1
apc.optimization=0
apc.cache_by_default=1
apc.shm_segments=10
apc.shm_size=256M
apc.ttl=0
apc.user_ttl=0
apc.num_files_hint=10000
;apc.mmap_file_mask="/tmp/apc"
apc.max_file_size=5M
apc.enable_cli=1
apc.mmap_file_mask="/tmp/apc.XXXXXX"
apc.slam_defense=0
apc.user_entries_hint=10000

All works perfect, but one question remains.
APC shows me this statisic:

Why hits so small? Any ideas?