SocialistWorker.org - Drupal newspaper site case study

SocialistWorker.org front page from 2008-06-06

Derek Wright (dww), Josh On, and the International Socialist Organization are pleased to announce that on May Day 2008, we launched the all-new, Drupal-powered SocialistWorker.org newspaper website. For the first time in its history, Socialist Worker is now a daily publication, providing more timely news and analysis of struggles not usually covered, and voices not usually heard, in the mainstream media.

The new SocialistWorker.org is a testament to the power of Drupal for building newspaper sites. By taking advantage of contributed modules such as Panels, Views, CCK, NodeQueue and the Drupal Markup Engine, the site provides a fundamentally improved set of functionality for both editors and readers of the paper. Read on for technical details about how the site was built using these and other modules, and how this project has helped improve Drupal itself.

Note: in this article, screenshot images of administrative or editorial pages on the site link to high-resolution versions of themselves for more detail. Screenshots from pages that are publicly visible link back to the site so you can experience the page directly.

Site organization

// TODO: edit this section down? (leave most of it).

Originally, the goal of the redesign was to retain the weekly rhythm of the print edition of the paper, structure the site around the weekly issue numbers, and start to lay the ground-work for web-only functionality and more frequent content. As the new site began to take shape, we decided to redesign the site from the ground up based on daily content, while still providing access to the stories through the weekly issue numbers.

As a result, the site's primary structure is organized around the date each story is published. However, stories that ran in the print edition are marked with an issue number (a freetagging taxonomy vocabulary) so that per-issue story lists are also available. Every story that belongs to a specific issue links to that issue's story list, for example, issue 666.

Each story is classified into a single department that it belongs to, such as International, Labor, Opinion and so on. Regular columnists such as John Pilger, Sharon Smith or Dave Zirin each have their own sub-departments. Browsing stories by department is encouraged by means of a left navigation bar on the front page, and by links back to the department listing for each story's own department on every story page and most story listings (issue lists, recent news, etc.).

Story nodes

The central content type of the site is a story. Due to the complicated requirements for front page layout, story lists, and topic archives, each story node has a large number of CCK fields and Taxonomy vocabularies. All aspects of site organization (department, publication date, optional issue number, topic, etc) are classified via fields on the story nodes. We wrote some custom form handling and JavaScript code to help try to keep the UI for adding a story reasonably sane (for example, certain fields are only relevant depending on the values of other fields, so we use jQuery to hide the dependent fields unless certain values of the parent field are selected).

Authors

One of the challenges we faced was how to represent the authors of the stories. Many of the writers do not have accounts on the Drupal site -- the only people directly inputing content are the editorial staff. Furthermore, some stories have more than one author. Therefore, it was impossible to use the core Authored by field for this. At first, we used a free tagging taxonomy vocabulary to create an Authors field, and some custom theme code to display it as a byline. This was nice since it provided auto-completion for free, and let us avoid creating Drupal accounts for every potential author. However, we ran into trouble with the ordering of multiple authors. By storing this information in a taxonomy vocabulary, the order was lost, so it became impossible to control if someone should have been credited as the main author and listed first.

To solve this, we introduced a new CCK node type called person, and changed the story nodes to use a CCK node reference field (with multiple values) for the authors. This way, the order of the node references is preserved. This also retains the auto-completion interface when adding new stories, although it's no longer possible to enter a new author simultaneously while creating the story. Automatically creating a new person node if the title entered doesn't correspond to an existing node would be a relatively easy usability enhancement, though it'd be nice to do it in a generic way that was reusable on other sites, not just more custom code here. There are a few modules that use JavaScript to bring up a node add form in cases like this, but I haven't had a chance to evaluate those.

We kept the free tagging taxonomy for a Contributors field on the story nodes. This is an optional list of names that appears at the bottom of the story (ordered alphabetically via the theme) such as "Jane Doe, John Smith and Jane Smithson contributed to this article.".

Issue numbers, dates and paths

// TODO
- issue number and auto-publication date, custom validation for all that that matches the URL alias

URL redirects and preventing "link rot"

// TODO

Insert boxes

Many stories have related text that we wanted to appear inline with the story, not just in a sidebar. For example: suggestions for further reading on a given topic, information about actions you can take to support a struggle described in a story, information about a columnist, a description of a book or movie being reviewed, and so on. Each of these boxes of inline text are represented by an insert box node (another CCK-defined node type). This has numerous advantages:

Easy to reuse insert boxes in multiple stories, which is frequently necessary (many stories about the same topic, columnist bios, etc).
Complete flexibility in the number of insert boxes to use on any given story.
Easy to use a taxonomy vocabulary to define different types of insert boxes, and then style them differently depending on the type.
Revision history of changes to each insert box.

For maximum flexibility to place these boxes inline at the right paragraph in the text, we use the Drupal Markup Engine to define our own markup tags. DME works as another filter you can enable for any input formats. By implementing a hook, we can specify the tag identifier (in this case, "box"), and whatever attributes we need the tag to support. The hook also lets you define what to replace the tag with when filtering the node body for output. For insert boxes, all we need to define is the node ID of the insert box to use, and whether we want it to aligned right or left in the text. An example DME tag for an insert box would look something like: <dme:box nid=24 align="right" /> although the alignment defaults to the right, so the align="right" attribute could be left off.

Warning: DME version 5.x-1.0 contains a critical bug that prevents it from working in many cases. I've provided a patch that fixes it, which is what we use on SocialistWorker.org. If you want to try out DME yourself, I highly recommend applying that patch, first.

Image handling

Aside from logos and icons handled directly by the theme layer, there are three ways that images are used on the site:

Images inserted inline with the body of a story.
Big images on the front page to draw attention to a lead story.
Thumbnail images associated with certain stories to draw some attention to them on the front page, which we call the story's teaser thumbnail.

All images are represented by image nodes, which are constructed using CCK and Image field. Different derivative sizes of the images are rendered on demand by means of Image cache, with specific scaling presets for the different widths necessary for various spots on the front page.

Image placement

Just like the insert boxes, inline images inserted into the body of stories are handled by a custom DME tag. Image tags have more attributes than insert boxes -- beyond the node ID and alignment, images can also specify their size (image cache width preset), and an optional caption specific to this particular use of the image. If the caption is not specified at all, nothing is printed. If the caption is specified as "default" then the caption field on the image CCK node (if any) will be used. So, an example DME image tag on the site would be: <dme:img nid=812 size=242 caption="default" />.

The way images are placed on the front page, both large images and teaser thumbnails, will be described in detail below.

Image captions

//TODO (a few sentences)

Code consolidation

//TODO (a few sentences)
//TODO: Explain consolidation of code for rendering images

Why not Image and Image Assist?

//TODO: Discussion of this vs. image + img_assist? (1 paragraph max)

The front page

//TODO
- manual DnD layout by editors on the /draft panel page

Custom panel layouts

-- custom panel layouts (w/ full-res screenshot)

Custom pane types

In panels, content for each region of the layout is populated by panes. Panels provides a rich API for defining custom pane types, which we make heavy use of. There are three pane types that we added for this site:

Story panes

// TODO: screenshot of the story add form, with rest of DnD in background

The front page is primarily composed of stories (headlines and related title fields, teasers, and headlines of related articles). These are all controlled via a pane which takes the node ID (nid) or headline of a story node, selectors to control if the teaser should be displayed and how large the headline should be, a set of fields for the nids of related stories.

Each story also has a set of fields to define its teaser thumbnail: an integer for the nid of the image node, a size selector, and an alignment (to control if the thumbnail should float left or right of the teaser). These fields are therefore properties of the pane, not the story itself. As stories move down the front page and fall out of view, the specific size or alignment that works best might change. It's easier for the editors to modify these settings all in the same display editor where they lay out the front page, instead of having to switch to the story editing form for this. This also preserves the exact layout for the story for each day of the front page archive (described below).

Images panes

// TODO

Header panes

// TODO

Making the Draft Front Page Live

// TODO - big red button (just explanation)

Front Page Archive

// TODO

Node queues

//TODO
- Ad queues
- Custom story lists (near future)
- *not* most of the front page (why not)

Content finders

Given all the places that the editors need to know the node ID of something on the site (an image, insert box, or stories when laying out the front page), we needed to have a quick way for the editors to find what they're looking for. In the administration area of the site, we created a set of pages for finding content of various types. All of these pages are created with table views. Views made it trivial to have whatever fields we wanted to see in the table, to add exposed filters to let the editors drill down and refine their search, and to sort the table by certain columns. The imagecache views support made it easy to build an image finder that includes the thumbnail of each image.

The only customization here beyond pure views is that on the insert box and image finders, I wanted a column in the table with a sample DME tag that could be cut and pasted to place that particular piece of content inline in a story. Obviously, there's no database column containing these example tags, so this couldn't be done directly via the views administrative UI. However, it was easy to add a theme template for these two views and to add my own column to the tables. Since the only unique thing about each of these DME tags is the node ID, and since the view already contains the node ID, it was trivial to construct the appropriate DME tag for each row in the table. Here's the theme function for the "find_insert" view:

<?php
function phptemplate_views_view_table_find_insert($view, $nodes) {
  // All of this is the default code from views...
  $fields = _views_get_fields();
  foreach ($nodes as $node) {
    $row = array();
    foreach ($view->field as $field) {
      if ($fields[$field['id']]['visible'] !== FALSE) {
        $cell['data'] = views_theme_field('views_handle_field', $field['queryname'], $fields, $field, $node, $view);
        $cell['class'] = "view-field ". views_css_safe('view-field-'. $field['queryname']);
        $row[] = $cell;
      }
    }
    // Here's the only custom part -- add another cell to the row:
    $cell = array(
      'data' => check_plain(sw_example_insert_tag('box', $node->nid)),
      'class' => 'view-field view-field-dme-box-tag',
    );
    $row[] = $cell;
    $rows[] = $row;
  }
  $header = $view->table_header;
  // We also need to include another table header for our new column:
  $header[] = array(
    'data' => 'Tag',
    'class' => 'view-cell-header view-field-dme-box-tag',
  );
  return theme('table', $header, $rows);
}
?>

The only other minor theme customization is to add a Clear search link in each view's exposed filter bar, which clears out all of the selected filters. I believe merlinofchaos already fixed this in views2, so I'm not planning to generalize this and contribute it back to views1.

Theme

In general, our design was motivated by trying to keep things as clean and clear as possible, keeping all the bling and "wow" to a bare minimum. We wanted to fit more content onto the screen and give readers a chance to quickly see lots of headlines, instead of big splashes of superfluous color, borders, gradients, and so on. I'd like to think that Edward Tufte would approve, someone whose ideas about the visual display of information I respect a great deal. I wish more designers working with Drupal adopted his perspective.

Believe it or not, this site started with core's bluemarine theme. Needless to say, it went through massive editing to get to the current look. The theme is fixed-width, since the layout of the panels on the front page is very specific and we wanted to ensure consistency. We also wanted a fixed width on the story pages themselves, since we wanted to keep the story from getting too wide to be readable, without resorting to extra sidebars of stuff that clutters the page and draws attention away from the story itself. We also use a slightly customized fluid zen theme for the administrative theme.

However, lots of the theme code has been very specific to this site, and I don't think it would be useful to contribute the code back to drupal.org. I'll be the first to admit I'm not the best theme developer, and some of what we've done on this site would probably be considered a hack by people who were really familiar and comfortable with the theme system. So, no, don't expect to see a "RedMarine" theme in the contributions repository anytime soon. ;)

That said, if there are specific questions about certain aspects of the theme that people are interested in learning more about, I'd be willing to answer questions in the comments below.

Print-friendly

// TODO new screenshot once http://tasks.socialistworker.org/node/311 lands
// TODO: different story to feature?

//TODO
- Custom code to move DME insert boxes to the bottom
- Custom code to add footnotes for URLs
- custom menu stuff to make the URLs nid-less

E-mail this

// TODO: use http://socialistworker.org/email/2008/06/13/why-did-clinton-lose for the example and screenie?
Lots of the traffic to the site is driven by e-mailing stories to interested people and lists, so we wanted to make sure the e-mail functionality was well done. The e-mail page is built with send, mimemail, html2txt backport, a custom menu handler to have a nice URL, and a bunch of patches waiting to be committed. ;)

I almost always prefer text e-mail over HTML, and luckily, the whole editorial staff agrees. As with the print-friendly version, inline insert boxes are moved to the bottom of the message. The content is then converted into plain text using html2txt which does nice things like converting italics into /italics/, converts all <a href...> tags into footnoted URLs at the bottom, and so on.

To show people what they'll be sending, the page renders this plain-text version of the story under the form where they enter their contact information and the addresses they wish to send to. This is handled by a theme function for the send form.

Giving back to the community

This site wouldn't have been possible without Drupal itself, and especially the contributions outlined above. Having benefited greatly from the Open Source community, we have tried to give back as much as possible. Nearly every time I start using a new module, I find myself making fixes and improvements, which I always attempt to get folded back into the "upstream" version. This project has resulted in numerous patches:

Panels

Views

NodeQueue

Send/mimemail

Core

ImageCache

Other

#168877: Path-redirect as an option on the node form (add or edit) (Path redirect)
#246313: hack to rely on nodeapi() causes tags to only be prepared but not processed in many cases (DME)
#257274: Technorati search link is wrong (Service Links)
#258291: Share produces invalid XHTML (Share)

Future work

The newly launched SocialistWorker.org is just the beginning of what we hope to do with the site over the coming months. Our future plans involve harnessing Drupal's community management and location functionality to encourage users to login to the site, and to provide them with location-specific stories, information about events in their area, and so on. We'd like to connect the site's online readership with actual struggles taking place on the ground in communities across the United States, and around the world. We'd also like to make it easier for readers of the site to become contributors by uploading stories and images directly.

In the short term, one of our biggest challenges is importing all the content from the old static site into Drupal. Due to the changing structure and format of the static site over the years, automating this task hasn't been easy. If anyone has expertise in this area that they'd like to share, I'd be most grateful for any pointers or help. ;)

How you can help

// TODO

Special thanks

I'd like to thank the following people for their invaluable help during this project:

Josh On for his design skills and vision.
Alan Maass, the editor of Socialist Worker, for his political clarity, enthusiasm, patience, and attention to detail.
Earl Miles (merlinofchaos), for his amazingly powerful and extensible contributed modules, and his willingness to answer all my questions and problems trying to use and improve them.
Allie Micka (vauxia) and pajunas interactive for hosting the new site, for all her patience and effort resolving any problems, and for her send and mimemail modules.
Sam Boyer (sdboyer) for his help with Panels2 and general enthusiasm for seeing this project succeed.

// TODO: verify that these people want to be thanked, etc.
// merlin -- done
// josh -- todo
// alan -- todo
// vauxia -- done
// sdboyer -- done

Thanks, Drupal!

-Derek Wright