The pcap filter syntax used for tcpdump should work exactly the same way on wireshark capture filter.
With tcpdump I would use a filter like this.
tcpdump "tcp[tcpflags] & (tcp-syn|tcp-ack) != 0"
Check out the tcpdump man page, and pay close attention to the tcpflags.
Be sure to also check out the sections in the Wireshark Wiki about capture and display filters. Unfortunately the two types of filters use a completely different syntax, and different names for the same thing.
If you wanted a display filter instead of capture filter you would probably need to build an expression combining tcp.flags.ack, and tcp.flags.syn. I am far more familiar with capture filters though, so you'll have to work that out on your own.
My best bet would be to use something like:
tcpdump -ieth0 -s96 -w traffic.dump 'ip or icmp or tcp or udp'
Where the "tricky" part will be to chose a correct value for the "-s" (snaplen) parameter (snaplen is the maximum length of the packet tcpdump will capture).
From the tcpdump man pages:
Snarf snaplen bytes of data from each
packet rather than the default of 68
(with NIT, the minimum is actually
96). 68 bytes is adequate for IP,
ICMP, TCP and UDP but may truncate
protocol information from name server
and NFS packets (see below). Packets
truncated because of a limited
snapshot are indicated in the output
with ``[|proto]'', where proto is the
name of the protocol level at which
the truncation has occurred. Note that
taking larger snapshots both increases
the amount of time it takes to process
packets and, effectively, decreases
the amount of packet buffering. This
may cause packets to be lost. You
should limit snaplen to the smallest
number that will capture the protocol
information you're interested in.
In this example i'm using 96 to be "almost" sure that I would capture 100% of ethernet+ip+(icmp || udp || tcp) header values.
In case your traffic have IP or TCP options (i.e. timestamps) and you want to also capture this info, then you will have to play with the snaplen parameter (i.e. increase/decrease it).
In case the length of the headers of your packet is less than snaplen, you may also capture part of the payload.
Finally, to read the traffic captured, I would use something like:
tcpdump -e -nn -vv -r traffic.dump
Where the important part is to use the "-e" option so you can get the ethernet headers printed.
This page gives you an idea about the size of the ethernet/tcp/udp headers under different circumstances and may help you to arrive to a "correct" value for the snaplen parameter.
Best Answer
Your command should work, maybe there's a bug.
Use tshark (wireshark package) instead:
tshark -i eth0 -b duration:3600 -b filesize:102400 -s 65535 -w trace.pcap