Firstly right to the jargon ,is there any actual difference between
the two or do they mean the same?
Fluid in its historical context meant it was NOT fixed i.e it expanded if a wider screen was available. Precursor to responsive. Fluid does not mean it adapts to smalleer/tablet/smartphone screens.
Responsive means, it adapts to device/screensize.
Is it safe to classify the current development mainly a html5/css3
based one ?
This feature is based on Media Queries which I think is a feature of CSS3, so CSS3 is a prerequisite.
What Popular frameworks are available to easily implement this?
Bootstrap is one.
What are the most common compatibility issues in terms of different browser types?
Volumes could be written on incompatibilities. But you just should be aware that there are differences. Proper resets have been developed and improved. Keep an eye on those and try to test as much as you can. Newer features have varying level of compliance in different browsers, so be aware of that.
I understand there are methods like this http://css-tricks.com/resolution-specific-stylesheets/ which does this come under?.
This is an example of media query. The feature which makes responsive design work.
Are there any external browser detection methods besides the api calls specific to the >browser that are employed in this regard?
I am not sure, but generally, browsers have an identifier which is part of the HTTP request. It is not strictly speaking "reliable" as you can easily craft an HTTP request and claim to be from Chrome, but practically, it works.
Is it safe to assume that just accounting for proper scaling from width 980px onwards to the maximum size would be sufficient to accommodate this? given we aren't presenting any new information for the new size.
Just because the industry trend is to lock at 980px does not mean you cannot design for a bigger screen. If your usecase mandates that you present a more richer content to bigger screen, perhaps you can tinker with it. But generally the practice is that 980 is the max contemporary designers consider desktop top be, considering there is no new content. (but this assumption is not set in stone - you can decide to ditch it if it makes sense)
Does it make sense to have additional information ( which conflicts with purpose of responsive web design) to utilize the top size and beyond?
Actually this debate is still ongoing, but I personally think, responsive design means you have the opportunity to present content differently, and even different content. So what is best for a smart phone is not going to be sufficient for a desktop. In the end it would be your personal judgement based on your requirements.
Stop worrying so much about specific devices.
The ability to detect whether a user is viewing your website via a smartphone, tablet, or desktop, seems to be increasingly difficult if all you're going by is screen resolution
It is getting increasingly difficult to detect via screen resolution, and it will only get harder as more devices enter the market, but the purpose of media queries is not to detect device types.What media queries are good at is making tweaks to your design when it is no longer pleasant to use at the current dimensions. If the site starts to fall apart at 550px wide, then it doesn't matter if there's a device with those dimensions or not: you set the breakpoint there.
Its the same deal with feature detection. If the device supports touch, then what does it matter if it's a mobile device or some future wall-sized monitor? Either way the touch enhancements will probably be useful.
User-agent sniffing - as you've stated - is completely unreliable. I could change my User-agent string to Shakespeare quotes if I really wanted too. What would your site decide about my browser then?
User agents also require a lot of work to handle all the possibilities. Do you have one for Firefox on android? Chrome on IOS? Dolphin on Froyo? The WiiU Browser? The extremely limited browser on my old LG feature phone? Lynx? IE 13? Links? IceWeasel? The Blackberry playbook's browser? How are you telling the difference between Opera running on a tablet and opera running on a phone?
This list can only grow as time goes on.
Best Answer
I would follow your first impulse and go responsive - I feel that when implemented properly it makes for a better user experience. Use a responsive CSS library and some javascript MVC frameworks/templating libraries to do your heavy lifting on the client side.
Addressing your concerns:
Low Quality Code from Handling Disparate Prototypes
I think using battle tested JS and CSS libraries and best practices should help you here. Think along the lines of backbone, knockout or angular to provide your clientside code with structure.
Sending Unneeded Code Files to Mobile Devices
I don't think this will that big an issue. Build out mobile functionality first and go from there. This will ensure you get the core functionality created and allow you to build out to the bigger devices from there. Use only the bare necessities to keep things light. If you use templating engines you can reduce the amount of HTML you're sending over the wire. Combine and minify JS and CSS files on the server side to minimize what you're sending to the mobile devices.
Wasting CSS when Hiding HTML Elements
Use a templating engine. These can either be built into the clientside MVC frameworks or separate (Mustache, Handlebars). This will prevent you from sending over too much extra HTML. Additionally, you can lazy load certain sections if that floats your boat.
Conclusion
A well thought out Responsive Web Design should be capable of either avoiding or minimizing each of the issues. A lot of work has been done in the open source community to help negate those problems. I would highly suggest searching out those solutions and seeing if they are helpful with your project.
Its your team's decision in the end, but make sure you consider the vast number of libraries and elect for the best user experience your team can support within the non-functional requirements (timeline, cost, staff knowledge) of your project.
Popular Libraries
JS MVC: backbone + templating library (see below), Knockout, Angular, Ember
Templating: Handlerbars, Mustache
CSS: Twitter Bootstrap, Foundations, Skeleton
Minification/Combination Tools: Yahoo YUI, UglifyJS
Miscellaneous JS: underscore, require
These are the examples I hear the most about. I've personally used Knockout and Twitter Bootstrap. I'm looking to play around with backbone and Foundations as well though. There is a lot of research to do in this area and not really enough space in a single answer. I think if you start researching these libraries and tools you should find plenty of information to get you to the right endpoint.