TL;DR We need to ship D8. ;)
I was sent this question today from a co-worker:
"We always talk anecdotally about how Drupal adoption slows before a new release and then picks back up. Do we have any data to support that for Drupal 7 or Drupal 6? I’d love to know the impact of Drupal 8 as well – but not sure that’s possible. Any thoughts?"
This is a great question, but since email is where information goes to die ;), I figured I would copy my response into a blog post as well.
Show me the data!
Since D8 has been in development so long, we don't have enough data showing on https://www.drupal.org/project/usage/drupal anymore since it prunes it after 3 years. :(
Here's a graph I made from trawling through historical data on https://archive.org/web/ though (source):
This only goes back to June 2008 which is after D6 came out, so it's not ideal, but we can still glean some useful data out of it.
Drupal 6
Here is a screenshot of the data from just prior to Drupal 7's release in January 2011:
- In December 2008 there were 77K installs of D6 (compared to 0 in January since it wasn't out yet :)) (77K% increase). This is when D7 was in active development.
- At the end of 2009 there were 203K installs of D6 (163% increase). This was when D7 was in feature freeze.
- At the end of 2010 there were 323K installs of D6 (59% increase). This was when D7 was just about to ship.
- At the end of 2011 there were 292K installs of D6 (9% decrease). This is when D7 had been out for about a year and several key contributed modules were ported.
- D6 usage has been declining ever since, and is currently at about 135K installs.
Drupal 7
Here is the data from 2011 to today:
- At the end of 2010 there were 6.5K installs of D7. This is when D7 was just about to be released.
- At the end of 2011 there were 230K installs of D7 (3438% increase). This is when D7 had been out for about a year and several key contributed modules were ported, and D8 was just beginning development (was mostly D7 bug fixes at this point). Of note, D7 usage eclipsed D6 usage just a few months later (Feb 2012).
- At the end of 2012 there were 522K D7 installs (127% increase). This is when D8 was nearly done with feature development.
- At the end of 2013 there were 728K D7 installs (39% increase). This is after D8 was in code freeze.
- At the end of 2014 there were 869K (19% increase). This is when D8 was in beta.
- As of last week (mid-2015) there were 984K installs (13% increase). D8 is currently still in beta, with ~25 critical issues remaining before release candidates.
Patterns
There are a few patterns we can discern from this data:
- There is an enormous uptick in Drupal usage every new major release (though it's delayed until it reaches a "stable" state, i.e. after enough contributed modules are ported).
- After that initial year or two of exponential growth, it slows down a lot.
- The closer the next version is to being released, the slower the growth is of the current version. Generally, this is because people will postpone projects and/or use non-Drupal solutions to avoid incurring a major version upgrade.
- Usage of the older stable version starts to decline after the newer major version reaches the "stable" state.
Why Drupal 8 will make this more betterer
There are a few enormous shifts coming with D8 that should change these patterns significantly:
- Drupal 8 is much more fully-featured out of the box than any of its predecessors, so for many sites there is no need to wait on any contributed modules to begin building. Therefore we reach "stable" state (for sites that can do what they need to with just core) at Day 0, not 6-12 months later.
- A number of key contributed modules that delayed porting of other key contributed modules in D6/D7 (Views, Entity Reference, Date, etc.) were moved into core in D8. So they're available right now—even before release—to build on. And indeed we're seeing other big ecosystem modules (Commerce, Rules, etc.) porting now, while D8 is still in development.
- D8 will end the 3-4 year "big bang" release cycle. Instead, we'll be doing "small bang" releases every 6 months with non-backwards compatibility breaking feature/API improvements. That means we should hopefully stave off adoption decline much longer, and possibly even sustain the "hyper adoption" rate for much longer.
- We will still eventually have a D9 "big bang" release (3-4 years from now) with backwards compatibility breaks, but only after it's amassed enough awesome functionality that couldn't be otherwise backported to D8. This will provide us with another "epochal" marketing event that D8 is giving us today (well, soon) in order to drive adoption even further.
Sorry, that was probably Way Too Much Information™ but hey, the more you know. ;)
Comments
What's up with the weird dips?
Thanks Angie - This is an awesome analysis! It does look like there's some weird dips in the reported number of installs around the end of the last few years (plus May 19, 2013). Do we know what's causing those? Is it maintenance on Drupal.org, or something else?
Great question! No idea. :)
That's a great question. No idea. :) Almost all of them are right at the end of the year. So my best guess is some kind of "off-by-one" error in counting weeks in some older core versions. Either that or maybe a lot of servers take the last week of the year off as a holiday. ;)
May 2013 I also have no explanation for other than that was when Lily was born, so it's possible that event was so EPIC it caused a massive electricity outages everywhere. ;)
Filed issue here: https://www.drupal.org/node/2501385
holiday vacation
People take vacations right before the end of the year - that means fewer developer laptops are running, it means people aren't triggering poormanscron on dev/staging sites, it means that testbots who spin up environments for each commit are...spinning up less often. I consider that drop to be the developer/test installation % but I don't have conclusive evidence for it.
Exactly my experience
Exactly my experience tracking the metrics for Drupal Commerce for years. Every holiday season there's a dip because these thousands of development sites aren't being accessed (and therefore triggering the poor man's cron that pings d.o for updates). I personally believe the number of dev sites to be a multiple of that dip, as there are undoubtedly sites that have development and staging servers that ping d.o via cron jobs or the poor man's cron triggered on other scheduled events.
Discussion of that exact
Discussion of that exact phenomenon here: http://bleen.net/blog/do-websites-go-vacation
Comments are interesting...
Thanks for the post Angie,
Thanks for the post Angie, there's really a lot of data in here that would be hard for anyone to put together. But at the same time it's a bit frightening. In 2012 we launched a startup based on D7 (took us 3 weeks in D7 versus competitors that took months, or in some cases years, with other platforms). However, in the three years since then the entire industry has flipped on it's head. Mostly thanks to mobile usage / growth. To quote you:
|
In the same year we launched our company in D7, D8 was nearly done with feature development. The industry moves so quickly that we're already starting to split off components of our startup into microservices (non-drupal, non-php) as the entire methodology of web development has shifted (app engine / docker / heroku / etc.). The 4+ year development cycle of Drupal will undoubtably kill the project if it hasn't already.
About 80% of my day-to-day income comes from Drupal, so I'm a huge supporter, but it's starting to feel foolish not to jump ship. Not because D8 has been in development for so long, but because it's starting to feel like an aged methodology before it's even launched.
To be honest, I haven't played with D8 as much as I'd have liked to and am quite invested in D7 so I'm probably unfairly biased. Would love to hear what you think about my take on this.
Mono
An excellent recent article by Martin Fowler makes the suggestion that what you've done (Monolithic system first, then splitting out into microservices later) isn't actually an indication that you should start with microservices for everything, but instead a natural progression that every site should take. Start monolithic, microservice later. It's a good read and would indicate that Drupal in the future won't be less used in new site roll-outs, because the methodology isn't outdated, it's just monolithic isn't the entire solution.
The link to the article by
The link to the article by Martin Fowler is much appreciated. And thanks Angie, for taking time to pull this post together.
That was presented at Drupagora last year
Funnnily enough, this was the closing session JulienD and myself (fgm) did at Drupagora last year http://fr.slideshare.net/Juliendubreuil/le-futur-de-drupal-et-des-applic... and it caused some turmoil in the local community, where some did not want to hear that monolithic was not the perfect be-all-end-all http://drupalfr.org/forum/general/discussion-generale/42568-resolu-drupa...
Thanks for the question!
In addition to what others have said, I'd say D8 is well-poised to take advantage of the general move to microservices architecture because of a few things:
1) D8 ships with the tools to do both "push" (via the Web Services modules) and "pull" (via Guzzle) service requests. That gives you the flexibility to either use Drupal 8 as a unified admin backend to numerous other services, or as the central content store which you call into from other microservice APIs.
2) We've removed the hard-coded assumption baked into D7 and earlier that every request is a themed HTML page and if you don't want that you need to explicitly bail out with drupal_exit() or similar. In removing that assumption, we're making use of Symfony components, specifically the HttpKernel component, which converts Requests into Responses. These can be HTML, JSON, XML, whatever.
3) Even the most ardent Drupal fan will tell you that Drupal isn't perfect for every need, and sometimes you want to write highly customized code for a specific use case. Because Drupal 8 is powered by Symfony, you can easily move "down the stack" and use Symfony components for this as the need arises. This way you don't need to branch into other languages/frameworks to strip off parts of your "monolith-first" design and replace them with microservices.
So if anything, my hope is that Drupal 8 will provide ample opportunity for those who've been forced to move away from Drupal due to changing best practices to come back, due to the better architecture and improved flexibility.
Impressive Data
Angie, thanks for putting this together.
It looks promising, and I'd love to be pushing out projects in D8 right now, but some key modules I use still are awaiting updates. If this weren't the case, most of my projects would be built on D8, especially because of CMI.
I help organize sprints in my area, and will be keen on pushing more improvements on D8. Hopefully my PHP chops will improve enough for me to make more meaningful improvements to the code base.
Maybe work on Migrate in core?
If you organize sprints and know PHP, but the crop of critical issues feels a bit too challenging, a great place to jump in would be Migrate in core https://www.drupal.org/node/2456261, specifically the D7 => D8 migrations: https://www.drupal.org/node/2456259
There's a lot here that's relatively low-hanging fruit, and it all helps folks on D7 adopt D8 sooner!
Doesn't it actually show that core releases have a small effect?
Interesting post and data!
However, I don't agree with your conclusion. When I look at the graph, I don't see an enormous uptick in Drupal usage after a new major release at all (and definitely nothing like exponential growth). There is probably a small inflection point in 2011 where the growth rate decreased a bit before Drupal 7 was ready for production use and rebounded to its previous level, or a slightly higher level, afterwards (i.e. what your colleague's email asked about), but it's a pretty small effect overall. Frankly the growth of Drupal from 2008 to today looks almost linear throughout.
Now I'm not going to argue that if Drupal 6 and 7 were never released and we were all still using Drupal 5 in the year 2015, that Drupal would have the same install base it has today :) But I think the data is closer to suggesting that new Drupal core major versions do not affect adoption than it is to suggesting that they do.
growth curves
The description "After that initial year or two of exponential growth, it slows down a lot." sounds rather like: https://en.wikipedia.org/wiki/Logistic_function where we'd think the release of a new Drupal version creates a new and larger population of adopters whose adoption of that version eventually saturates.
Yeah, the charts are fairly linear
Yeah, the charts are fairly linear, but I think that's because the growth is off-set by the abandonment of the older version (a lot of those sites in the declining D6 line are crossing over to the D7 line). What makes D8 exciting is the features keep coming on a predictable (and much shorter) cycle, but you don't have to endure a major version upgrade to get them. So those users from 8.0.x, 8.1.x, 8.2.x, etc. will compound, rather than replacing each other.
Pattern analysis is always fun
Like others in this thread, I derive most of my income directly from Drupal. So, I very much don't want to be a nay-sayer, but . . .
Based upon the charts, comparing mid-year to mid-year regions to eliminate end of year dips, it seems like Drupal installs are flat lining across the board.
From the charts, I also don't reach this conclusion, "There is an enormous uptick in Drupal usage every new major release." Both the D6 and D7 lines seem to be basically linear. The linearity is easier to see on the Drupal.org Weekly project usage you linked to.
### ###
Other observations related to Drupal installs / Drupal major releases:
###
I will speculate that all these numbers don't include many production sites, as about everyone I know only turns on the update module on a pure production site when they are actually going to update the site.
###
A new major release tends to kill off potential customers who are unfamiliar with Drupal's lack of usability until key contributed modules are ported.
Specifically, this statement, "At the end of 2011 there were 230K installs of D7 (3438% increase). This is when D7 had been out for about a year and several key contributed modules were ported ..." is somewhat circumspect.
Personal experience, sample size of one: I lost a 20-hour per week client in April 2012, due to D7 not being production ready by April 2012. Simple things like Ubercart not having enough functionality ported to D7 yet to replace their home brew pure PHP solution. Arguably it's the owner's fault for not listening to my advise to use D6 when the project started November 2011, but he spent 10s of thousands of development dollars on a failed conversion and will never use Drupal for anything, ever.
To fulfill the axiom of just don't complain, suggest a constructive solution: It would be nice if the release announcements for D8 had realistic statements on both the current usability and the expected usability of D8. To whit:
"D8 production is released today. It can current power simple brochure sites and extremely simple eCommerce sites. In six months we expect it to handle reasonable complex brochure sites. In twelve months we expect it to handle moderately complex eCommerce sites. In two years we expect it to handle very complex eCommerce sites."
And give real world examples: "Site xyz.com spent X$s and Y man hours converting from D7 to D8, maintaining the same functionality."
###
In total usage there are 2x, 3x, 4x, ??x non-produciton sites to production sites? Considering that it's probable that almost all of these totals are from non-produciton sites, then there aren't that many production sites? Less than ~300K?
[ Although that gets into a long and energetic discussion on just what constitutes a production site. :) ]
###
It would be interesting to see major core security release points added to the D7 chart. If I'm eyeballing it correctly, last summer's core security release flat lined total installs for a couple months?
### ###
Thanks for the charts!
Best,
Michael
Wow, lots here. :)
- See above comment about linearity. I'm cautiously optimistic D8 can turn that around. :)
- Agreed we should manage expectations carefully once D8 is released in that 6-12 month window before a variety of contrib modules are available.
- Great idea about interleaving security announcement dates. I unfortunately am not fancy enough in my spreadsheet prowess to do so, but hopefully someone else can take the source data and do that analysis.
- I don't think that's true that most production sites turn off Update Manager. In a quick glance, all of the Drupal sites I have admin access to have it on, at any rate. And I would guess a huge portion of users don't even know that it's an option to turn off security updates; it's automatically turned on during install without further explanation.
- Remember too that Update Manager can only track public-facing sites that have the module turned on, and are able to ping outside the firewall. This means any intranet site, sites powered by large-scale Drupal hosting solutions who manage their own updates (e.g. Drupal Gardens, a lot of University installations, etc.) are not reflected in these numbers.
So basically, take these numbers/graphs with an enormous grain of salt, all around. :) But they do at least offer relatively apples-to-apples comparisons over ~7 years.
I agree with clear, honest announcements
Just like I wish the Drupageddon announcement would have been more forthcoming, widespread and noticeable, I think Drupal 8 should have a section in it's information where a listing of ported sites for D8 vs. D7 would be displayed (yes, D7 would be long), that would at least give users a fair feeling of where the new D8 is at coming out of the gate. I waited 2 years before switching over to D7 after it launched as I knew it would take at least that amount of time to get many common top 100 modules ported. Most are still stuck in dev releases only (a.k.a. "stable recommended" are buggy and dev is the only version currently usable). I think planting an info section like this in the main D8 info would help while not deterring adoption as much as some headline "Not ready for 2 more years" kind of thing. It would also be very helpful if those listings were broken down into categories so that switchers that have no idea what these module names mean can at least have an idea of their usage.
Correction: "where a listing
Correction: "where a listing of ported modules for D8 vs. D7 would be displayed"
Ported modules, not sites, sorry.
Huh? Your third bullet point
Huh? Your third bullet point says that "D8 will end the 3-4 year 'big bang' release cycle," and the next bullet point says that "we will still eventually have a D9 'big bang' release (3-4 years from now) ..." Whaaaaa...?