I have just completed an installation of Openstack through puppet, and after jumping through a few hoops have gotten to a point where I can attempt to log into the dashboard. When I do this I'm greeted with the something went wrong page (An unexpected error has occurred. Try refreshing the page. If that doesn't help, contact your local administrator.)
The errors I encountered when installing through Puppet include the horizon user not being created, the horizons.log not being created, allowed_hosts being too restrictive and local_settings.py in /usr/share/openstack_dashboard/openstack_dashboard/local. All of which I've fixed for now.
Nothing is being logged to the apache log files, however, this is what's being logged to horizon.log:
ConnectionError: [Errno 111] Connection refused
2014-04-17 23:10:37,795 1958 DEBUG openstack_dashboard.api.nova novaclient connection created using token "0c27d608ca94bab56bf6614ddb714286" and url "http://x.x.x.x:8774/v2/64d62e9095bc4e0f81ef9a49e68d2aaa"
2014-04-17 23:10:37,799 1958 ERROR django.request Internal Server Error: /horizon/admin/
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/django/core/handlers/base.py", line 111, in get_response
response = callback(request, *callback_args, **callback_kwargs)
File "/usr/lib/python2.7/dist-packages/horizon/decorators.py", line 38, in dec
return view_func(request, *args, **kwargs)
File "/usr/lib/python2.7/dist-packages/horizon/decorators.py", line 86, in dec
return view_func(request, *args, **kwargs)
File "/usr/lib/python2.7/dist-packages/horizon/decorators.py", line 54, in dec
return view_func(request, *args, **kwargs)
File "/usr/lib/python2.7/dist-packages/horizon/decorators.py", line 38, in dec
return view_func(request, *args, **kwargs)
File "/usr/lib/python2.7/dist-packages/horizon/decorators.py", line 86, in dec
return view_func(request, *args, **kwargs)
File "/usr/lib/python2.7/dist-packages/django/views/generic/base.py", line 48, in view
return self.dispatch(request, *args, **kwargs)
File "/usr/lib/python2.7/dist-packages/django/views/generic/base.py", line 69, in dispatch
return handler(request, *args, **kwargs)
File "/usr/lib/python2.7/dist-packages/horizon/tables/views.py", line 155, in get
handled = self.construct_tables()
File "/usr/lib/python2.7/dist-packages/horizon/tables/views.py", line 146, in construct_tables
handled = self.handle_table(table)
File "/usr/lib/python2.7/dist-packages/horizon/tables/views.py", line 118, in handle_table
data = self._get_data_dict()
File "/usr/lib/python2.7/dist-packages/horizon/tables/views.py", line 182, in _get_data_dict
self._data = {self.table_class._meta.name: self.get_data()}
File "/usr/share/openstack- dashboard/openstack_dashboard/wsgi/../../openstack_dashboard/dashboards/admin/overview/views.py", line 41, in get_data
data = super(GlobalOverview, self).get_data()
File "/usr/share/openstack-dashboard/openstack_dashboard/wsgi/../../openstack_dashboard/usage/views.py", line 33, in get_data
self.usage.summarize(*self.usage.get_date_range())
File "/usr/share/openstack-dashboard/openstack_dashboard/wsgi/../../openstack_dashboard/usage/base.py", line 98, in summarize
_('Unable to retrieve usage information.'))
File "/usr/share/openstack-dashboard/openstack_dashboard/wsgi/../../openstack_dashboard/usage/base.py", line 95, in summarize
self.usage_list = self.get_usage_list(start, end)
File "/usr/share/openstack-dashboard/openstack_dashboard/wsgi/../../openstack_dashboard/usage/base.py", line 130, in get_usage_list
return api.nova.usage_list(self.request, start, end)
File "/usr/share/openstack-dashboard/openstack_dashboard/wsgi/../../openstack_dashboard/api/nova.py", line 474, in usage_list
novaclient(request).usage.list(start, end, True)]
File "/usr/lib/python2.7/dist-packages/novaclient/v1_1/usage.py", line 35, in list
"tenant_usages")
File "/usr/lib/python2.7/dist-packages/novaclient/base.py", line 62, in _list
_resp, body = self.api.client.get(url)
File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 230, in get
return self._cs_request(url, 'GET', **kwargs)
File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 217, in _cs_request
**kwargs)
File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 199, in _time_request
resp, body = self.request(url, method, **kwargs)
File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 170, in request
**kwargs)
File "/usr/lib/python2.7/dist-packages/requests/api.py", line 44, in request
return session.request(method=method, url=url, **kwargs)
File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 279, in request
resp = self.send(prep, stream=stream, timeout=timeout, verify=verify, cert=cert, proxies=proxies)
File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 374, in send
r = adapter.send(request, **kwargs)
File "/usr/lib/python2.7/dist-packages/requests/adapters.py", line 206, in send
raise ConnectionError(sockerr)
ConnectionError: [Errno 111] Connection refused
I'm trying to connect directly to keystone, not sure if I'm doing this right but this is the error I get and i'm 99.99% sure the login/password combination is right.
ubuntu@pawn2:~$ keystone --os-username=admin --os-auth-url=http://localhost:35357/v2.0 user-list
OS Password:
Unable to authorize user
I also found this page, and I'm getting results with the commands that don't require any authentication which leads me to believe the service is actually running. For example, running curl http://x.x.x.x:35357 gives me something useful in return. using curl on port 8774 also returns some info.
http://docs.openstack.org/developer/keystone/api_curl_examples.html
If I can assume that the service is running, but I can't get authorized that my password is wrong or something else is preventing me from logging in. But if the password was wrong wouldn't I get a wrong login/password combo? Furthermore the password was set via puppet and I can still look at the original file and my typing skills aren't that bad. I also feel I've exhausted my google skills, and haven't found anything all that useful (i think).
I'm not really sure where to go from here, any help would be greatly appreciated!
(Although primarily openstack in my opinion, I've tagged this post with puppet in case it has somehow contributed to my problem)
Edit1:
I've managed to narrow down the problem, it looks like the glance tables were not created during the puppet installation. I didn't see anything regarding this in the reports generated but anyway, here's the message
ERROR [glance.registry.db.api] Could not ensure database connection and consistency. Ensure database configuration and permissions are correct and database has been migrated since last upgrade by running 'glance-manage db_sync'
ERROR [glance.registry.db.api] (ProgrammingError) (1146, "Table 'glance.images' doesn't exist") 'SELECT images.created_at AS images_created_at, images.updated_at AS images_updated_at, images.deleted_at AS images_deleted_at, images.deleted AS images_deleted, images.id AS images_id, images.name AS images_name, images.disk_format AS images_disk_format, images.container_format AS images_container_format, images.size AS images_size, images.status AS images_status, images.is_public AS images_is_public, images.location AS images_location, images.checksum AS images_checksum, images.min_disk AS images_min_disk, images.min_ram AS images_min_ram, images.owner AS images_owner, images.protected AS images_protected \nFROM images \n LIMIT %s' (1,)
I used glance-manage db_sync to fix this but I'm still getting this error message and nothing is showing up in apache or horizon logs but this is in the glance/registry.log
2014-04-18 14:45:10 727 ERROR [glance.registry.db.api] Could not ensure database connection and consistency. Ensure database configuration and permissions are correct and database has been migrated since last upgrade by running 'glance-manage db_sync'
So I went in and (re)set the proper databasea permissions, still the same error shows up when I try to access the dashboard except now I'm not even being given a chance to login. Nothing in the apache/horizon/glance/keystone logs.
I just noticed my system time is way off, I'm going to fix this issue and then continue on my journey.
Edit2:
Fixed the time.
nova-api is spitting this, continuously, into its nova.-api.log file.
ERROR nova.service [-] Unhandled exception
TRACE nova.service Traceback (most recent call last):
TRACE nova.service File "/usr/lib/python2.7/dist-packages/nova/service.py", line 307, in _start_child
TRACE nova.service self._child_process(wrap.server)
TRACE nova.service File "/usr/lib/python2.7/dist-packages/nova/service.py", line 284, in _child_process
4321 TRACE nova.service launcher.run_server(server)
4321 TRACE nova.service File "/usr/lib/python2.7/dist-packages/nova/service.py", line 148, in run_server
4321 TRACE nova.service server.wait()
4321 TRACE nova.service File "/usr/lib/python2.7/dist-packages/nova/service.py", line 666, in wait
4321 TRACE nova.service self.server.wait()
4321 TRACE nova.service File "/usr/lib/python2.7/dist-packages/nova/wsgi.py", line 23, in wait
TRACE nova.service self._server.wait()
TRACE nova.service File "/usr/lib/python2.7/dist-packages/eventlet/greenthread.py", line 166, in wait
TRACE nova.service return self._exit_event.wait()
TRACE nova.service File "/usr/lib/python2.7/dist-packages/eventlet/event.py", line 116, in wait
TRACE nova.service return hubs.get_hub().switch()
TRACE nova.service File "/usr/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 177, in switch
4321 TRACE nova.service return self.greenlet.switch()
TRACE nova.service File "/usr/lib/python2.7/dist-packages/eventlet/greenthread.py", line 192, in main
TRACE nova.service result = function(*args, **kwargs)
TRACE nova.service TypeError: server() got an unexpected keyword argument 'url_length_limit'
TRACE nova.service
Edit3:
Turns out there was nothing in the nova database, recreated that and will continue searching along those lines.
Edit4:
The tables in the nova database aren't being created when I run nova-manage db sync, and no error messages either. Trying to figure this out…
Final Edit:
I've veered way off course with this question so I'm going to stop myself here.
For the last problem I encountered I didn't solve it, but I tried copying the tables from a (nearly) identical install and importing it into the empty nova database and nothing changed. I'll stop myself here, thanks for reading!
Best Answer
Final Edit
Not sure what I did to get it to work last time but the true cause of my misfortune was omitting this step,
http://docs.openstack.org/havana/install-guide/install/apt/content/basics-packages.html
which guides you through the process of installing the havana ubuntu cloud archive.
Thanks for reading!
old post starts here
I managed to fix the issue.
In the keystone database, the endpoints table more specifically there was a mix of localhoist/and ip address based entries in the "url" field, they look like this x.x.x.x:9292, x.x.x.x:8776/v1/%(tenant_id)s and so on. I replaced the ip address with localhost in all cases making it uniform and I'm now able to log into the dashboard from the machine itself. I can't login other than from localhost but I'm assuming this is a separate issue because I tried replacing localhost with other entries (the public ip address for example and I can still login via dashboard from localhost).
To summarize, although I could be misinterpreting what happened, I think the fact that the endpoints url fields weren't uniform was causing the initial connection refused issue. I believe the specific endpoints that were written out as localhost caused the problem, unfortunately I didn't take note of which were which at the time.
Edit: I tried accessing the dashboard from my browser in a incognito window using chrome and I could log in no problem, I tried internet explorer and it works great. But even after clearing out all my history/cache/etc from chrome I still get the error. Not sure why but at least I've narrowed things down a bit, and it works, kinda!