R – ggplot2: how to add text to multiple vertical lines (geom_vlines) on a time x-axis

Please have a look at the following example

library(dplyr)
library(lubridate)
library(ggplot2)
data <- data_frame(time = c(ymd(20160201),
                            ymd(20160202),
                            ymd(20160203),
                            ymd(20160201),
                            ymd(20160202)),
                            value = c(1,1,1,2,2), 
                            group = c('A','A','B','B','B'))

events <- data_frame(time = c(ymd(20160201), 
                              ymd(20160202)),
                     text = c('who let the dogs out?',
                              'who? who? who?'))

ggplot(data, aes(x = time, y = value, group = group, color = group)) + 
  geom_line(size = 2 ) +
  geom_vline(data = events, aes(xintercept = as.numeric(time)))  

> data
# A tibble: 5 × 3
        time value group
      <date> <dbl> <chr>
1 2016-02-01     1     A
2 2016-02-02     1     A
3 2016-02-03     1     B
4 2016-02-01     2     B
5 2016-02-02     2     B

> events
# A tibble: 2 × 2
        time                  text
      <date>                 <chr>
1 2016-02-01 who let the dogs out?
2 2016-02-02        who? who? who?

I would like to get a line chart for the variable value for each group (A and B) and plot vertical lines every time there is an event in the events dataframe.

Using the ideas ggplot vertical line with date axis, How to get a vertical geom_vline to an x-axis of class date? and How to add legend for vertical lines in ggplot? I can easily do that:

ggplot(data, aes(x = time, y = value, group = group, color = group)) + 
  geom_line(size = 2 ) +
  geom_vline(data = events, aes(xintercept = as.numeric(time)))

The problem is that I would like to label each vertical line (each event) with the corresponding text, as in R ggplot2: Labelling a horizontal line on the y axis with a numeric value.

Unfortunately, doing the following does not work

ggplot(data, aes(x = time, y = value, group = group, color = group)) + 
  geom_line(size = 2 ) +
  geom_vline(data = events, aes(xintercept = as.numeric(time)))  +
  geom_text(data = events, aes(x = as.numeric(time), y = 0, label = text))

What is wrong here? any ideas?
Thanks!