webchick.net

very helpful lioness

Webchick's "plain Drupal English" Guide to the Remaining Drupal 8 Critical Issues: DrupalCon Bogotá Edition

Sat, 02/14/2015 - 01:09 -- webchick

(Apologies for the atrocious state of the HTML that follows; this content is originally from this Google Doc.)

Updated Feb 15, 2015 with "needs triage" marker, per xjm.

Webchick's "plain Drupal English" Guide to the Remaining Drupal 8 Critical Issues: DrupalCon Bogotá Edition

DrupalCon Bogotá just finished up, and critical issue-wise we've managed to stay in the 50s for a few days (down from a high of 150 in the summer of 2013!), so now seems like as good a time as any to write down what's left to ship Drupal 8!

This post will attempt to document all of the remaining 55 criticals (as of this writing), and attempt to offer a somewhat "plain English" (or at least "Drupal English" ;)) description of each, loosely categorized into larger areas in which we could really use extra help. There are over 2,600 contributors to Drupal 8 at this time, please join us!

(Note: These descriptions might not be 100% accurate; this is my best approximation based on the issue summary and last few comments of each issue. If I got the description of your pet issue wrong, please update your issue summary. ;))

Table of contents

Quick vocabulary lesson

Current state of critical issues

Security

Security Parity with Drupal 7

Session and User Authentication API

REST

New security improvements

Performance

Profiling

Fix regressions relative to Drupal 7

Entity Field API

Views

Configuration system

"Fix it, or else"

General house-keeping

Other

Thrilling conclusion! (also known as "TL;DR")

Quick vocabulary lesson

Within this list, there are numerous "markers" used to signify that some of the issues in this list are more important to fix ASAP. These are:

  • D8 upgrade path: An issue tagged D8 upgrade path (currently, 13) means it blocks a beta-to-beta upgrade path for Drupal 8, generally because they materially impact the data schema or they impact security. Once we resolve all of these blockers, early adopters will no longer need to reinstall Drupal between beta releases, but can just run the update.php script as normal. This is currently our biggest priority.
  • Blocker: An issue tagged blocker (currently, 5) means it blocks other issues from being worked on. This is currently our second-biggest priority (or 0th priority in the case an issue blocks a D8 upgrade path issue :D). I've noted these as "sub-bullets" of the issues that are blocking them.
  • Postponed: Issues that are marked postponed (currently, 9) are either currently blocked by one of the "Blocker" issues, or we've deliberately chosen to leave off until later.
  • >30 days: These patches have a patch more than 30 days old, and/or were last meaningfully commented on >30 days ago. If you're looking for a place to start, re-rolling these is always helpful!
  • Needs triage: Anything with a Triaged D8 critical tag (33 as of today) means that at least a couple of Drupal core's branch maintainers have agreed that this issue should indeed block Drupal 8's release. Conversely, those tagged Needs D8 critical triage (11 as of today), and also those issues without either tag, are those that may or may not actually be critical and still need looking over. As a result, before dumping a bunch of time into solving these, they should be evaluated against the priority levels of issues document to see if they can be downgraded.
  • No patch: This issue doesn't have a patch yet. Oh the humanity! Want to give it a shot?

Other weird core issue nomenclature:

  • "meta" means a discussion/planning issue, with the actual patch action happening in related/child issues.
  • "PP-3" means "this issue is postponed on 3 other issues" (PP-1 means 1 other issue; you get the drift).

Current state of critical issues

Sections roughly organized from "scariest" to "least scary" in terms of how likely they are to make Drupal 8 take a longer time to come out.

Security

Because Drupal 8 hasn't shipped yet, it's not following Drupal's standard Security Advisory policy, so there are still outstanding, public security issues (13 as of this writing). We need to resolve most of these prior to providing a Drupal 8 beta-to-beta upgrade path, as this is the time when we signal to early adopters that it's an OK time to start cautiously building real sites on Drupal 8.

Skills needed: Various

Security Parity with Drupal 7

This class of security issue is to ensure that when Drupal 8 ships, it won't have any regressions security-wise relative to Drupal 7.

  • Port SA-CONTRIB-2013-096 to D8 (D8 upgrade path) Here's one such issue for Entity Reference module. SA-CONTRIB-2013-096 addressed a relatively esoteric remote access bypass bug, and the patch needs to be forward-ported to Drupal 8.
  • Port SA-CONTRIB-2015-039 to D8 (D8 upgrade path)  SA-CONTRIB-2015-039 addressed two issues in Views module, a redirect and default permissions for disabled views. The first was fixed in D8, but access checks are still missing from a few views for the second.

Session and User Authentication API

Because of various intricate dependencies, the authentication part of Drupal 8 isn't yet converted to object-oriented code, and prevents us from further optimizing bootstrap. This set of issues fixes various problems with this part of the code, and ensures these important security APIs are complete and ready to ship.

REST

  • REST user updates bypass tightened user account change validation (D8 upgrade path) Since Drupal 7, when you edit your user account, you have to provide the existing password when you want to change the password or e-mail. This security feature is currently by-passed by REST user updates as you can change the password or e-mail without providing the password.
  • External caches mix up response formats on URLs where content negotiation is in use (>30 days) Drupal 8's request processing system is currently based on content negotiation (which allows you to serve multiple versions of a document at the same URI based on what headers are sent e.g. Accept: text/html or Accept: application/json). This is generally considered the "right way" to do REST. However, various external caches and CDNs have trouble with this mechanism, and can mix them up and can send random formats back. The issue proposes changing from content negotiation to separate, distinct paths such as /node/1.json.

New security improvements

These issues affect new security improvements we want to make over and above what Drupal 7 does.

  • [meta] Document or remove every SafeMarkup::set() call One of the big security improvements in Drupal 8 is the introduction of Twig's autoescape feature, which ensures that all output to the browser is escaped by default. However, this is quite a big change that requires all of the code that was previously escaping content to stop doing that, else it gets double-escaped (so you start seeing < and " and whatnot in the UI). We originally introduced the ability to manually mark markup safe with SafeMarkup::set(), but the recommended approach is actually to use Twig everywhere, so this issue is to ensure that all remaining instances of the manual way are fixed, or at least documented to explain why they're using the non-recommended method.
  • Passing in #markup to drupal_render is problematic (>30 days, Needs triage) Another issue in the Twig autoescape space, we need to ensure that markup set by the "#markup" in e.g. form definitions is properly escaped.
  • Limit PDO MySQL to executing single statements if PHP supports it (Needs triage) Remember SA-CORE-2014-005? Yeah, so do we. ;) This issue is to make sure that if another SQL injection vulnerability is ever found again, the damage it can do is more limited by eliminating the ability for MySQL to execute multiple queries per PDO statement.

Performance

Tied with security, 13 of the remaining issues are tagged Performance. While it may seem odd/scary to have this be a big chunk of the work left, it's a common practice to avoid premature optimization, and instead focus on optimization once all of the foundations are in place.

Skills needed: Profiling, caching, optimization, render API

Profiling

Here are a sub-set of issues where we need performance profiling to determine what gives us the biggest bang for our effort.

Fix regressions relative to Drupal 7

  • [meta] Resolve known performance regressions in Drupal 8 This is the main tracking issue in this space. During the 8.x cycle we've introduced several known performance regressions compared to Drupal 7 (sometimes to make progress on features/functionality, other times because we introduced changes that we hoped would buy us better scalability down the line), which we need to resolve before release so that Drupal 8 isn't slower than Drupal 7. The performance team meets weekly and tracks their progress in a detailed spreadsheet.

Entity Field API

Tracked under the Entity Field API tag (currently 6 issues).

Skills needed: Entity/Field API, Form API, Schema API

  • Schema for newly defined entity types is never created (D8 upgrade path) When you first install a module that defines an entity type (for example, Comment), its database tables are correctly generated. However, if an entity definition is later added by a developer to an already-installed module, the related database schema won't get created, nor will it be detected in update.php as an out-of-date update to run.
  • FileFormatterBase should extend EntityReferenceFormatterBase (D8 upgrade path) Entity Reference fields define a EntityReferenceFormatterBase class, which contains logic about which entities to display in the lookup, including non-existing entities and autocreated entities. File field's FileFormatterBase class currently duplicates that logic, except it misses some parts, including access checking, which makes this a security issue. The issue proposes to simply make File field's base class a sub-class of Entity Reference's, removing the need of "sort of but not quite the same" code around key infrastructure.
  • FieldTypePluginManager cannot instantiate FieldType plugins, good thing TypedDataManager can instantiate just about anything Currently, you get a fatal error if you attempt to use Drupal 8's Plugin API to create a new instance of a field type. The current code in core is avoiding this problem by going roundabout via the Typed Data API instead. This issue's critical because these are two of the most central APIs in Drupal 8, and they should work as expected.
  • [META] Untie content entity validation from form validation Despite all the work to modernize Drupal 8 into a first-class REST server, there still remain places where validation is within form validation functions, rather as part of the proper entity validation API, which means REST requests (or other types of workflows that bypass form submissions) are missing validation routines. This meta issue tracks progress of moving the logic to its proper place.
  • Entity forms skip validation of fields that are edited without widgets (>30 days) If a field can be edited with a form element that is not a Field API widget, we do not validate its value at the field-level (i.e., check it against the field's constraints). Fixing this issue requires ensuring that all entity forms only use widgets for editing field values.
  • Entity forms skip validation of fields that are not in the EntityFormDisplay (No patch, >30 days) Drupal 8 has a new feature called "form modes" (basically analogous to "view modes" in Drupal 7, except allowing you to set up multiple forms for a given entity instead). Currently, we're only validating fields that are displayed on a given form mode, even though those fields might have validation constraints on other fields that are not displayed. Critical because it could present a security issue.

Views

Views issues are generally tracked with the VDC tag. There are currently 6 criticals at this point which touch on Views (some already covered in earlier sections).

Configuration system

The configuration system is remarkably close to being shippable! Only 4 critical issues left. We're now working on finalizing the niggly bits around edge cases that involve configuration that depends on other configuration.

Skills needed: Configuration system, Entity Field API, Views

"Fix it, or else"

This subset of issues are things that are part of core currently, and we would really like to keep, but are willing to make some hard choices in the event they are among the last remaining criticals blocking release. The "postponed" among this list means "postponed until we're down to only a handful of criticals left." If these issues end up remaining in the list, we will move their functionality to contrib, and hope to add it back to core in a later point release if it gets fixed up.

Skills required: Various, but mainly low-level infrastructure and non-MySQL database skills.

  • [meta] Drupal.org (websites/infra) blockers to a Drupal 8 release (Blocker) This issue contains a "grab bag" of Drupal.org blockers that prevent an optimal Drupal 8 release, including things like semantic versioning support, testing support for multiple PHP/database versions, and support for Composer-based installations. If this issue is one of the last remaining criticals, we might choose to ship Drupal 8 anyway, and jettison one or more features in the process, such as…
  • [Meta] Make Drupal 8 work with PostgreSQL (Needs triage) The meta/planning issue for fixing PostgreSQL (both in terms of functionality and in terms of failing tests). bzrudi71 is predominantly leading the charge here and making steady progress, but more hands would be greatly appreciated.
  • [meta] Database tests fail on SQLite (>30 days) Same deal as PostgreSQL but for SQLite. Unlike PostgreSQL though, this one doesn't have anyone leading the charge at this time, and it's also a lot harder to punt this to contrib, since we use it for various things such as testbot. Help wanted!

General house-keeping

These are all basic things we need to keep on top of between now and release, to ensure that when we're down to only a handful of criticals, we're ready to ship a release candidate. The good news is, these are also all generally really easy patches to make, and often also to test.

Skills needed: Basic patch rolling / reviewing / testing skills. (good for newbies!)

  • [meta] Ship minified versions of external JavaScript libraries (Postponed) Basically, in the Gilded Mobile Age™ we want to ensure that we're sending as little over the wire as possible, so scrunching various JS libraries down to the smallest possible file size needs to be the default. Separate issue from above because it needs to happen for both updated and existing JS libraries. Postponed because there'll be less work to do once all of the out-of-date JS libraries are updated and minified at the same time.

Other

I couldn't figure out a nice heading for these, so here's the rest.

  • Remove _system_path from $request->attributes (Needs triage) Symfony provides a $request object, which has an "attributes" property for the purpose of storing various contextual bits. But the problem with $request->attributes->get('_MAGIC_KEY') is that the values are undocumented, there's no IDE autocompletion, and it's not clear which are internal vs. public properties, so we have an issue at [meta] Stop using $request->attributes->get(MAGIC_KEY) as a public API. to try and stop doing that.

    However, _system_path in particular is used a ton, since it's very common to want to know the path of the current request. The patch exposes a "CurrentPath" service instead, which eliminates all of those issues.
  • Potential data loss: concurrent node edits leak through preview (Needs triage) Because the temp store that Drupal 8's new node preview system employs uses an entity's ID as the key, rather than something uniquely identifiable to a user, if two users are editing the same node and hit preview at the same time, one of them is going to lose data due to a race condition.
  • Ajax file uploads fail on IE 9 Pretty much exactly what it says on the tin. :P

Thrilling conclusion! (also known as "TL;DR")

Well, not so thrilling, but at least a conclusion. :)

  • Anywhere you see a blocker issue, attack it with fire. Those are holding other criticals up.
  • The biggest area of focus right now is D8 upgrade path blockers. Many of them are security issues.
  • Another big area is Performance, both fixing existing regressions, and profiling to determine where our biggest wins are.
  • Views and Entity Field API are tied in third place for number of remaining criticals. Let's have a race, shall we? ;)
  • The configuration system is looking pretty good, but still has a handful of sticky issues left.
  • There are a series of important features we'll lose if they're not fixed up in time.
  • If you're looking for something somewhat easy/mundane, help yourself to one of the general house-keeping issues.
  • Don't forget about the other miscellaneous issues I was too tired to categorize.
  • You're safer fixing a Triaged D8 critical issue, since those are all guaranteed to block release, while others may be downgraded in the future.

Sorry this post was so long (and probably has its share of inaccuracies) but I hope it will be helpful to some. It's basically what I needed to get back up to speed after taking a few months off of Drupal 8, so figured I'd document my way to understanding.

Now, let's get 'er done! :D

Comments