VoIP over the public Internet can be a problem, but it usually works good enough, most of the time, although there can be times where it sucks. Most ISPs have extra cost features where they will honor some of your QoS markings and policies.
(I know Verizon Business, among others, has some specific packages for QoS, and you may need to adjust your policies and markings sent to it to match one of its packages. We have a problem where we use multiple ISPs, and the QoS packages don't match between them, so we need to fine tune for the particular ISP, or pick a close enough package.)
The problem arises when the traffic must pass through other ISPs. You have no control over what happens in that case. The larger ISPs will have better (possibly direct) connections to many VoIP providers, as the VoIP providers will try to directly connect to the large ISPs or a Tier 1 ISP that directly connects to the large ISPs.
There may also be a possibility of having a VoIP provider, especially a telco like AT&T or Verizon, connect SIP trunks directly into your data center(s).
The big monkey wrench in the works today is the requirement for E911. You, and the VoIP provider, will need to maintain a database of where each phone is connected so that emergency services will get not only the address of a 911 call, but the floor and section of the floor. (We have spent large sums of money to meet the E911 requirements.) While this requirement is not necessarily in every state today, it is being phased in for all the states, and it is not something you can ignore if you don't yet have the requirement (it is far easier to put it in place as you roll out VoIP than it is to try to retrofit it at a later time).
Best Answer
Most ISPs have to buy service from other ISPs to be able to reach every part of the internet. That is called "transit". If you start a small ISP you usually buy transit from one or more bigger ISPs. They will have connectivity to the whole internet, and they will sell that connectivity to you.
You can also set up direct connections to other networks. This is called "peering". Usually those are the networks you exchange a lot of traffic with. Paying a transit provider for the packets you exchange can be a waste of money if you can easily connect directly to the other network and cut out the middle man. Sometimes other networks want you to pay them to set up a direct connection, but in many cases the peering if settlement-free.
The problem with setting up direct connections is the number of network interfaces you need, pulling cables between routers etc. Because of that there are "Internet Exchanges" (IX). An IX is basically a large switch (large IXes are more complex, but from the outside they function as a large switch) that many internet providers and enterprises connect to. That way they can all talk to each other, if they want to. One connection to an IX can let you set up direct peering with hundreds of networks.
As you can see there is a reason you pay your ISP. They invest in routers, connections, and very likely pay larger ISPs to transport part of their traffic. You could set all of that up for yourself, but be prepared to invest tens of thousands of dollars/euros/etc for equipment and connections. And then you'll probably pay more than you pay your current ISP for the transit costs, as you won't be able to negotiate lower prices for buying in bulk.