Search API in Drupal 8 is about to be stable!

Wed, 2017-04-05 13:09 -- drunken monkey

TL; DR: After more than three years of development (and several blog posts), the Drupal 8 port of the Search API module is finally getting a stable release! The current Beta 5 release will be the last Beta, and there should be a release candidate and the final, stable release within the next two weeks. So, test it now, if you haven't already, to help eliminate any remaining (major) bugs!

The current status in detail

Last Saturday, April 1, I committed the fix for issue #2682369: Fix problems with overridden config entities, the last (and incredibly complicated – dedicated blog post upcoming) release blocker[1] for the stable release. The day after, I created the final Beta 5 release to give everyone still a few more days for testing.

Unless testing of this new Beta release discovers a serious enough problem, the first release candidate (RC) will be released on the coming weekend, April 8 or 9. Finally, if that doesn't uncover any major problems, the first stable release, Search API 8.x-1.0, will follow the weekend after that, on April 15 or 16. And then: rejoicing, parties and a little shield icon, to make everyone feel better.

To be clear, though, the Search API has been pretty usable for a long time (at least a year, I guess – possibly more, depending on your requirements), and we didn't have any major bugs or problems (unless you used config overrides) for some time now. The release blockers were mostly due to desired framework changes which would have been more difficult to do after a stable release, especially with other modules with stable releases already building on it. There were also several usability problems classified as release blockers, which was probably a mistake in hindsight. Having a clearer UI is nice, and helps avoid floods of bug reports due to confused users – but it shouldn't delay a stable release, and possibly dissuade site builders from using a basically working module. It also shouldn't prevent us from getting Security Team coverage, of course.

One plea regarding testing this last Beta: In case you are using config overrides for search indexes or servers in your site's setup or development workflow, it would be very helpful if you could verify that this last release fixes all associated problems you might have had – indexing/deleting using the wrong server (settings), overridden values being written back to the database, etc. This was a recent and pretty complex fix, so it would be great to feel more confident about that.

Also perhaps note-worthy: I've used the last few days of Dev Days Seville in March to finally update the module's documentation to Drupal 8. While it's still not quite finished, especially the documentation for developers, it should already provide a pretty good guide to the module's functionality and should help to answer most questions for first-time (or even advanced) users.

What's new

Compared to Drupal 7, the Search API in Drupal 8 is a massive step forwards – we didn't just spend those three years twiddling our thumbs, after all! Here's a run-down of the most important improvements we made:

Indexes can contain more than one type of item/entity

This might have been the single largest restriction in the Drupal 7 Search API: indexes could only be created for a single entity type (or custom item type). If you wanted a site search across entity types, you had to either use the Multi-Index Searches modules or, since August 2015, use the "Multiple types" item type, essentially a workaround-type backport of the D8 functionality.

In Drupal 8, however, creating an index for multiple item types (now called "datasources") is baked right into the framework, all parts of the Search API will support this natively. Furthermore, changing or re-configuring (think: changing the content types of nodes indexed) datasources is also supported right by the framework, and all such changes will be handled without any need to manually correct tracking data, or clear the index, etc.

Overhaul of the "Fields" tab and functionality

Also a continuing source of frustration, even if mostly just a usability annoyance, the "Fields" tab has seen a major revamp. Instead of seeing all available properties right on one page (killing the web server in some cases, for particularly large sites) and enabling the ones you want, we've transitioned to a UI inspired by Views: we only list the fields that are currently indexed and use a second page/dialog for discovery of available properties and adding the ones desired. This is also a lot more convenient than the old "Add related fields" box for indexing fields of related entities or data structures.

Furthermore, it's now possible to add the same property more than once, eliminating the need to use the "Aggregated fields" work-around of creating duplicates of a field. You can also freely change the human-readable name and the machine name of a field, and some processor-generated fields (like "Rendered HTML output") even allow per-field configuration (so you could have two "Rendered HTML output" fields, with each using different view modes, for different purposes). Also, like Views, changes made to the fields (but only to them) won't be effective immediately, but need to be explicitly saved afterwards. This avoids triggering a reindex numerous times when changing fields.

Improved language/translation support

Language support has also been vastly improved. While multilingual Drupal 7 sites needed the Search API Entity Translations contrib modules (at least when using the "entity translation" system, not "content translation") for getting proper translation support, which then didn't work natively with some other contrib modules, support for Drupal 8's new translation system is included right in the Search API itself. Automatically, all entities will be indexed once for every one of their translations, and each entity-based search item is identified not only by entity ID, but also by its translation language. Furthermore, additional internal improvements also make it easier for other modules (e.g., Search API Solr Multilingual) to add additional language-specific functionality.

Processors are more powerful than ever

This item is something that's probably only apparent as a developer, but processors have become a lot more powerful, being able to influence a lot of different areas of the search system. For non-developers, the most notable change is probably that the previous distinction between "processors" and "data alterations" has been removed – but not only have processors gained all the functionality previously held by data alterations, several other functions have been added, too.

For site builders, though, the main benefit here (in addition to more useful processors potentially becoming available) is a cleaner, less confusing user interface.

Objects everywhere!
Also not really anything that non-developers will notice, but internally the Search API has fully embraced Drupal 8's move to object-oriented programming (OOP). While its Drupal 7 version was already among the most OOP-using modules available (I'm pretty sure), this gained even more momentum in this new version. Not only did we, of course, make use of all the new OOP-based APIs in Core, like entities, plugins, services, etc. – we also converted a lot of the internal data structures (which, like Core, often used "magic" array structures in Drupal 7) to properly classed objects (e.g., indexed items, search results, fields) and moved most helper functions to methods on service classes.
Great test coverage
The last item, which no-one ever gets really excited about, but is actually also a huge step forward, is that we have improved test coverage throughout the module tremendously – aided a lot, of course, by the corresponding improvements in Drupal Core and on drupal.org. This (hopefully) means a lot less bugs that go undiscovered, a lot less regressions when introducing new features, simpler, more confident refactoring of old code and consequently less technical debt.

Other projects

Search API has always relied on additional contrib modules to provide powerful search solutions – with just the Search API module, you wouldn't get far. So, what is the current status of related projects in Drupal 8?

  • First off, the Database Search module (which provides the "Database" backend for servers) has been moved back into the Search API project itself, so this is already completely stable and very thoroughly tested. So, if you have a smaller site and don't require any advanced functionality (as offered by Solr or Elasticsearch), by just downloading the Search API project you can already set up a pretty good database-based search.
  • The Facets module (replacement for the Drupal 7 Facet API module) is still in Alpha as there are still some API changes and breaks planned. However, its basic functionality is already there and quite extensive, and it's already used on quite a few production sites. So, if you want to add facetting to your search, you should at least give it a try.
  • The Search API Solr module is in a similar position: It's already used on a lot of production sites, but there's still some rough edges (especially regarding support of optional features) and the module is currently considered to be in Beta state. However, the test coverage is quite good, so there shouldn't be any larger bugs lurking under the surface and, if you want to use Solr for your site, you shouldn't hesitate to give it a try. (Just beware that, like all modules with no stable releases, it's currently not covered by the Drupal Security Team, so security issues are a larger risk than usual.)
  • If you want to use Elasticsearch, things look a bit worse. There are (confusingly) several modules in Drupal 7, but only Elasticsearch Connector has a Drupal 8 version yet. And since even the Drupal 7 version is not yet beyond Alpha (and the Drupal 8 version has nothing more than a dev snapshot release), it may take some time until it becomes stable. I haven't evaluated the code or functionality, though, so it might actually already work quite well – if you have information, please leave a comment below!
  • The Pages module has been ported to Drupal 8 (and is currently maintained) by swentel. It, too, is already used on some production sites and has most of its functionality working fine (though some adaptions might be needed for the lasted Search API changes). It's currently in Alpha state.
  • Preliminary ports also exist for both the Autocomplete and the Location module, with at least some functionality already working. The former of those will probably be my next focus, since it was one of the most popular extension modules in Drupal 7 (apart from the mostly-ported ones mentioned above). And the latter is set to be the focus of an upcoming Google Summer of Code project by dbjpanda, so we'll probably have a full-fledged Drupal 8 version of this by the end of August.
  • The Saved Searches module has not been ported yet. I plan to start porting it once the Autocomplete port has been finished and I get the time.

Credits

Once again, I have to state that this release wouldn't have been possible, and the development would have taken even longer, if it weren't for a lot of helping hands who did their part in moving this module port along. Chief among them are Acquia, who sponsored two months of my time a year ago to focus on this port, and Joris Vercammen (borisson_) who was an invaluable help just by discussing ideas and reviewing patches (and being probably the second-best informed regarding the Drupal 8 Search API module). Thanks to both of them, but also to everyone else who contributed – much too many to list them all!

[1] To be accurate, there are still two open but postponed ones, but those are mere clean-up issues that will be committed right before RC creation.

Image credit: bayasaa

Comments

Submitted by dcam (not verified) on

I'm really enjoying working with the D8 version of the Search API. Thanks for all your hard work on it. Following your advice to @sylus, I've been working on a datasource module for displaying external Solr content, a successor to D7's Sarnia. At midnight yesterday I finally got it to the point where Views was displaying some of my index's fields! Hopefully I'll be able to add my module to the list of related projects sometime soon.

Submitted by drunken monkey on

That's great to hear, thanks for sharing!
However, before making this a related project, it might actually be something that could just be included in the Search API Solr project – either as a separate sub-module, or even directly as part of the Solr module (depending on the amount of code required, I guess, and how much it would mess up the original module). So, once you have something worth showing (and it seems you're already pretty far), maybe post it in the issue and let us take a look at it?

Submitted by dcam (not verified) on

I hadn't seen that issue before now, but I'll post there when I'm ready. Hopefully that will be late tonight (I'm developing this on my personal time). I created TypedData implementations for the Solr docs and fields. Until it finally worked, I was never certain that what I was doing was correct. As a result, I have several uncommitted files right now that I'm doing QA checks on. Once I'm finished then I'll create a sandbox, push it, and let you know it's there. Thanks!

Submitted by Nick (not verified) on

Thanks Thomas for your continued perseverance and dedication to get the eco-system to this point. This was surely a complete team-effort that required a lot of thinking and fiddling. To me, one of the most important changes the team introduced here is, as you say, the testing. Not only is there an insane amount of tests added but there are also integration tests that run in travisci so that we know all these modules work together with Apache Solr! My hope is that we can get these tests up to drupal.org or at least show that they exists in the respective projects on Drupal.org :)

Great job all.

Submitted by Steven (not verified) on

This all sounds very exciting. Search API is an amazing module and it seems to become ways better with D8. Thanks for your continious commitment!

Submitted by Kevin Baringer (not verified) on

Hi Thomas,
I've been digging in deep to the latest Search API for the past couple of weeks and I'm very impressed by the flexibility and extensibility!

However, my current stumbling block (in release 8.x-1.0-beta5) is with indexing a content type that has an Entity Reference Revisions field pointing to a Paragraph. We cannot seem to configure the index to include the fulltext inside the Paragraph entities.

I know that there are some issues around this but can't seem to find a clear answer as to whether this works right now or not. Can you let me know if this scenario is currently supported in the latest version of search_api, or if it's currently in progress?

Thanks so much for all your hard work! ~ Kevin

Submitted by drunken monkey on

I haven't yet tried that module myself, so I can't be certain, sorry! I have received several issues (I expect you've already found them) about this, though, so it's entirely possible that this is currently not supported, for whatever reason.
It's not a current focus of mine, but definitely something we should look at. (Seems the Paragraphs module is pretty popular already.) Could you maybe bump one of the related issues for me, as a reminder?

Submitted by Craig MC (not verified) on

I'm building out my searches with this as a backend right now but the current dev branch needs two patches to be able to work with the latest release of SearchAPI
https://www.drupal.org/node/2826804#comment-12034621
https://www.drupal.org/node/2868277#comment-12032418

If you're using the Drupal Project (which I highly, highly recommend) for Drupal 8, it's super easy to make sure those patches stay in-place
https://github.com/drupal-composer/drupal-project

I just added the following to the "extra" part of composer.json

"patches": {
  "drupal/elasticsearch_connector": {
    "Fix breaking change from search_api RC2": "https://www.drupal.org/files/issues/elasticsearch_connector-replace-remo...    "Fix breaking change to allow using pass-through servers from SearchAPI": "https://www.drupal.org/files/issues/elasticsearch_connector-fix-search-a...  }
}

As for Paragraphs, I'm going to be working on a processor to be able to add related Paragraph content to a node.

Overall feedback is that this all works great, but the documentation is very terse.
I'll try to write up an example processor to add new field(s) to an object.
the hook_search_api_alter_index doesn't really show the kinds of things I'm used to being able to do, and/or should have some pointer text in it for "If you want to add fields to be searchable, here's a specific link to example entity field, or here's an example processor class with boilerplate text.

Submitted by drunken monkey on

I think there should be enough documentation already to make clear that you can use a processor to add fields – you just have to read it.
But, of course, helpful hands expanding the documentation are always welcome. There is already a documentation page for adding a new property with a processor, it's just outdated. So, if you want to provide an up-to-date example, it would be great if you could just edit that page.

Submitted by Gregory (not verified) on

Thanks for all your work!! It's definitely impressive :).
I plan to use Search API to do proximity searches but unfortunately the location module is not working yet (there is patch to be able to do the location index in search_api_solr) but nothing working to display filters and search correctly.
You said in your post that it will be your next focus so I’m glad to hear that! There is no other way to do a proper proximity filtering in D8 today.

Submitted by drunken monkey on

Sorry, but it seems you misread: the Autocomplete module will be my next focus, not Location. But I know others are working on Location, so hopefully it will still improve over the next months.

Submitted by Gregory (not verified) on

Ouuups, I have to read more carefully :)
Thanks!

Submitted by jürgen (not verified) on

I really love Search API, great work! It would be great if you can integrate solr spellchecking for drupal 8

Submitted by omid (not verified) on

Thanks Thomas

Add new comment

To prevent spam, submitting full URLs in comments is not allowed. Please omit the "http[s]://" portion of the URL and I will restore the complete URL on review.

Filtered HTML

  • Lines and paragraphs break automatically.
  • Web page addresses and e-mail addresses turn into links automatically.
  • Allowed HTML tags: <a> <em> <strong> <q> <cite> <blockquote> <code> <ul> <ol> <li> <dl> <dt> <dd> <sup> <sub> <p> <br>
  • To post pieces of code, surround them with <code>...</code> tags. For PHP code, you can use <?php ... ?>, which will also colour it based on syntax.

Plain text

  • No HTML tags allowed.
  • Web page addresses and e-mail addresses turn into links automatically.
  • Lines and paragraphs break automatically.