C# language version history:

These are the versions of C# known about at the time of this writing:

• C# 1.0 released with .NET 1.0 and VS2002 (January 2002)
• C# 1.2 (bizarrely enough); released with .NET 1.1 and VS2003 (April 2003). First version to call Dispose on IEnumerators which implemented IDisposable. A few other small features.
• C# 2.0 released with .NET 2.0 and VS2005 (November 2005). Major new features: generics, anonymous methods, nullable types, and iterator blocks
• C# 3.0 released with .NET 3.5 and VS2008 (November 2007). Major new features: lambda expressions, extension methods, expression trees, anonymous types, implicit typing (var), and query expressions
• C# 4.0 released with .NET 4 and VS2010 (April 2010). Major new features: late binding (dynamic), delegate and interface generic variance, more COM support, named arguments, tuple data type and optional parameters
• C# 5.0 released with .NET 4.5 and VS2012 (August 2012). Major features: async programming, and caller info attributes. Breaking change: loop variable closure.
• C# 6.0 released with .NET 4.6 and VS2015 (July 2015). Implemented by Roslyn. Features: initializers for automatically implemented properties, using directives to import static members, exception filters, element initializers, await in catch and finally, extension Add methods in collection initializers.
• C# 7.0 released with .NET 4.7 and VS2017 (March 2017). Major new features: tuples, ref locals and ref return, pattern matching (including pattern-based switch statements), inline out parameter declarations, local functions, binary literals, digit separators, and arbitrary async returns.
• C# 7.1 released with VS2017 v15.3 (August 2017). New features: async main, tuple member name inference, default expression, and pattern matching with generics.
• C# 7.2 released with VS2017 v15.5 (November 2017). New features: private protected access modifier, Span<T>, aka interior pointer, aka stackonly struct, and everything else.
• C# 7.3 released with VS2017 v15.7 (May 2018). New features: enum, delegate and unmanaged generic type constraints. ref reassignment. Unsafe improvements: stackalloc initialization, unpinned indexed fixed buffers, custom fixed statements. Improved overloading resolution. Expression variables in initializers and queries. == and != defined for tuples. Auto-properties' backing fields can now be targeted by attributes.
• C# 8.0 released with .NET Core 3.0 and VS2019 v16.3 (September 2019). Major new features: nullable reference-types, asynchronous streams, indices and ranges, readonly members, using declarations, default interface methods, static local functions, and enhancement of interpolated verbatim strings.
• C# 9.0 released with .NET 5.0 and VS2019 v16.8 (November 2020). Major new features: init-only properties, records, with-expressions, data classes, positional records, top-level programs, improved pattern matching (simple type patterns, relational patterns, logical patterns), improved target typing (target-type new expressions, target typed ?? and ?), and covariant returns. Minor features: relax ordering of ref and partial modifiers, parameter null checking, lambda discard parameters, native ints, attributes on local functions, function pointers, static lambdas, extension GetEnumerator, module initializers, and extending partial.

In response to the OP's question:

What are the correct version numbers for C#? What came out when? Why can't I find any answers about C# 3.5?

There is no such thing as C# 3.5 - the cause of confusion here is that the C# 3.0 is present in .NET 3.5. The language and framework are versioned independently, however - as is the CLR, which is at version 2.0 for .NET 2.0 through 3.5, .NET 4 introducing CLR 4.0, service packs notwithstanding. The CLR in .NET 4.5 has various improvements, but the versioning is unclear: in some places it may be referred to as CLR 4.5 (this MSDN page used to refer to it that way, for example), but the Environment.Version property still reports 4.0.xxx.

As of May 3, 2017, the C# Language Team created a history of C# versions and features on their GitHub repository: Features Added in C# Language Versions. There is also a page that tracks upcoming and recently implemented language features.

Manually edit .sln file

This method is entirely aimed at renaming the directory for the project, as viewed in Windows Explorer.

This method does not suffer from the problems in the Remove/add project file method below (references disappearing), but it can result in problems if your project is under source control (see notes below). This is why step 2 (backup) is so important.

1. Close Visual Studio.
2. Create a backup of your .sln file (you can always roll back).
3. Imagine you want to rename directory Project1 to Project2.
4. If not using source control, rename the folder from Project1 to Project2 using Windows Explorer.
5. If using source control, rename the folder from Project1 to Project2 using the functions supplied by source control. This preserves the history of the file. For example, with TortoiseSVN, right click on the file, select TortoiseSVN .. Rename.
6. In the .sln file, edit all instances of Project1 to be Project2, using a text editor like NotePad.
7. Restart Visual Studio, and everything will work as before, but with the project in a different directory.

You can also see renaming solution manually or post which describes this manual process.

Advantages

• You can make the directory within Windows Explorer match the project name within the solution.
• This method does not remove any references from other projects to this file (an advantage over the Remove/add project file method, see my other answer below).

Warnings

• It's important to back everything up into a .zip file before renaming anything, as this method can create issues with source control.
• If your project is under source control, it may create issues if you rename files or directories outside of source control (using Windows Explorer). Its preferable to rename the file using the source control framework itself, if you can, to preserve the history of that file (check out the context menu on a right click - it may have a function to rename the file).

Update 2014-11-02

ReSharper has added an automated method for achieving the same result as the manual method above. If the namespace is underlined with a squiggly blue line, click on the action pyramid icon to either:

• Rename the namespace to match the directory name in Windows Explorer, or;
• Rename the directory in Windows Explorer to match the namespace.

In the second case, the final word defines the new directory name in Windows Explorer, e.g. if we changed the namespace to ViewModel2, it would offer to move the file to folder ViewModel2.

However, this will not necessarily update files in source control, so you may still have to use the manual method.

Update 2018-01-31

Tested with Visual Studio 2008, 2010, 2012, 2013, 2015, 2017 Update 1, 2, 3, 4, 5.

Update 2020-05-02

Tested with Visual Studio 2019.