You can't tell by visual inspection, that's for sure because some of them are lacquered/painted and even those that aren't all tend to look dark-grey. What you are asking is really tricky to fathom because there are so many characteristics that look the same between two ferrites at one frequency but are vastly different at another. If you are still interested I'll try and say what I'd do (what I'd really do is throw all my unboxed/unmarked ferrites in the trash and buy some more).
I'd consider winding (say) 5 equally spaced turns and putting the coil in a circuit to see what its inductance was - maybe a colpitts oscillator with a few caps that can be switched in and out. Maybe even make a band-pass filter from it and see where it resonates if you have a signal generator.
First type of result this will tell you is the inductance of the wound core. Then using the squared relationship between turns and inductance you can deduce its "effective permeability". This should enable you to narrow down the type of core to a range of possibilities.
You need to be be avoiding "test frequencies" significantly above 100kHz and preferably more like 10kHz - this is to reduce parasitic capacitance giving you errors.
OK so far, you might have determined the approximate "effective permeability" of the core BUT there are plenty of suppliers toting vastly different materials that you'd have to read through to try and identify the part so I'd next consider seeing how the indctance varied with temperature.
You don't need to test over a vast range, maybe just 25ºC to 50ºC would give you a decent shot at trying to uncover the ferrite. Use the oscillator/filter idea mentioned earlier and a controlled temperature - almost certainly the inductance will rise with temperature although there are a small percentage that will stay stable or fall but this will give you another tell-tale characteristic of the ferrite.
So now you have effective permeability and some idea what its temperature characteristic looks like. Scanning through various supplier's websites might narrow down the ferrite to maybe five or ten types.
It's going to be a long process this way and you may never uncover what it is that is sitting in your junk box. I suppose if your effective permeability is low it's likely to be either very temperature stable (i.e. good for filters up to (say) 1MHz) or it could have very low losses up to over 50MHz. The temperature test that indicated hardly any change in inductance across 25ºC might tell you its a material like Ferroxcube's 3D3: -
Also shown is 3C90 for comparison. 3D3 has a flat curve of inductance/permeability against temperature; probably changing something like 5% in a 25ºC change around ambient. 3C90 probably changes about 20%. It also has a much higher permeabilty. I'd recognize these two ferrites from their characteristics!
I think I've definitely convinced myself to throw all unrecognizable ferrites in the bin.
Bottom line - if you have a target circuit try it.
EDIT Also, here's is a question/answer on EE stack exchange that might also be useful or provoke some other ideas.
Best Answer
There is a color standard for painted toroids, and yellow means it has hysteresis to prevent saturation and is meant for filter inductors. But a side effect is that it has very low permeability. Black ferrite is usually a good choice for transformers. Blue is an expensive Permalloy that is more efficient than ferrite. Green is low frequency filters made with silicon steel tape wrapped to form a toroid.
This chart is generic as it is not including fine details such as permeability, and does not state if iron, steel, ferrite or permalloy, which is a nickel-iron alloy.
PC power supplies can put out over 1,000 watts and they use E cores as they are easy to wind by machine, and can have a cross section large enough to handle as much as 10 ampere/turns, and a tiny 10 mil air gap helps a lot. Large toroids need expensive winding machine heads so toroids are better used at low voltages were the number of windings is low, such as car stereo power supplies.
NOTE: Sometimes practical reasons determine what material and shape of transformer are used, which is not always the best choice. Cost and size compete with efficiency. The opinions of engineering and marketing and sales are not the same, and who wins determines what is used. "Just good enough" wins most of the time.
Hysteresis is a gap in which a iron or ferrite core needs a more intense magnetic field to magnetize, retaining a bit of the field after the current has been removed. It takes a stronger current of reverse polarity to reverse the cores magnetic field. In general a LCR meter which works with little drive currents will show a core with hysteresis built into its material has a much lower inductance than a core of the same turns of wire and the same cross section, but is made of ferrite or Permalloy.
To cover all the variations of cores made by many manufactures you would need a book full of charts specific to each core material. For any given core of any shape you need the manufactures datasheet or chart for that core to get an idea of permeability and any hysteresis factors and peak current values vs. pulse width. To quote Ali..chenski's comment:
Link to Magnetic Hysteresis