There are (at least) three different ways a team could work. Choose the one that works for your team.
1. Detail vs. High-Level Overview
Use the issue tracker to keep track of individual tasks. Use the card board to maintain a big picture of the major features, as a summary of the tasks in the issue tracker.
2. Bugs vs. Features
Use the issue tracker to keep track of individual bugs and customer service requests. Use the card board to keep track of new features under development.
3. Milestone software delivery vs. continuous software delivery
Use an issue tracker if you deliver large blocks of new functionality irregularly (for example, if you are the Windows team and there's a new version every few years). It is ideal for a development process in which large, completed projects are delivered to customers all at once (that includes games, embedded software, and traditional software based on annual or semi-annual releases).
Use a card board if you are delivering new features continuously to customers as they are developed, for example, on a web team that has continuous or very frequent delivery of value to customers. In this scenario your software development process is almost like an assembly line and less like a project with a beginning and an end.
You're attempting to repurpose an agile project planning tool into a waterfall one.
I'm no expert, but this is the very problem that Agile intends to address i.e: "Responding to change over following a plan". "Going Agile" involves something of a tradeoff - exhanging concrete costs and deadlines for flexibility and responsiveness of development. In a nutshell, the customer has to be (delicately) asked the following question:
Which of these is more important to your business, the features or the
delivery/cost?
NB: This is an abbreviation of a commonly accepted rule of thumb, see Thomas Owens comment below re: project management triangle. I chose to present a slightly more cynical slant on it...
Inevitably, clients want both, but armies of burnt out developers/agencies shows that up front estimation rarely matches the reality of execution and generally this leads to hideous overtime or "late" delivery and angry clients.
The whole point of Agile is that it is an attempt to bring an element of empiricism to delivery dates, measuring performance in terms of velocity relative to "task complexity". IMO the problem is that you've equated story size with hourly estimates. The meaning of story size is entirely dependent on how you break them up, which is why 1-5 is not already presented as a unit of time (unlike typical estimation tools).
If they absolutely need a concrete estimate, then they are not ready for an Agile relationship, so use the time honoured method of padding out of your schedule, and hope that the competition is putting in more padding in than you.
As I said, I'm no expert, but I've been through a experience to the one you're describing, and I asked a similar question then too: Funding Agile Projects
EDIT: JFTR - I heartily endorse Pivotal Tracker, we used it extensively on the last project I worked on, but it was something of a paradigm shift for the management/sales team to get their head round...as illustrated by the thread above, Agile is not such an easy sell outside of the dev/enterprise community
Best Answer
It is a feature.
It is defined and scheduled and tracked like any other features.
If implementing this feature is not sufficiently valuable (to the client or to you) for it to ever be scheduled, that's a different problem.