Build Gatsby MDX from non-fs remote content

We're pleased to share a major milestone achieved in enabling DFRNT data products to become easier to share. One of our top priorities is to provide subscribers an easy process to use data products to provide business value. In this post we share the key findings we made to create the gatsby-mdx-remote plugin.

We wanted to combine React and Markdown

To achieve this goal, we leveraged the power of React, one of the most popular ecosystems for developing component based websites, together with Markdown, a simple markup language for web content with React components. 

The combination, MDX, of these two technologies, was previously not available for headless CMS use in GatsbyJS. Today we unveil two open source plugins to make GatsbyJS and DFRNT together a perfect platform to build websites from data products.

Remote MDX was the solution, but was not supported in GatsbyJS

Previously, we developed an initial template that allowed our clients to build static websites from TerminusDB data products with Gatsby. However, the use of MDX was not supported in this template, which was a major issue for our subscribers who wanted to leverage the benefits of this cutting-edge technology that combines the simplicity of Markdown with the flexibility of React components.

We discovered that many in the Gatsby community had been grappling with this issue, but no feasible solution had been found. One of the major challenges was that the MDX provider was rebuilt for Gatsby 5 and support for remote headless CMS content was not yet available. Moreover, in the new version, the authors had reworked how to build MDX content bringing an end to the provision that allowed non-local sources like before.

Building support for programmatic MDX creation

After extensive research and testing, we have found a promising solution that entails manually sourcing nodes. We are excited to announce that we are in the process of creating two Gatsby plugins from our findings. One to source content from DFRNT via GraphQL, and one to build MDX programmatically from any source, including DFRNT data products. We will be providing more details about this plugin on our blog in the coming days.

Long story short, to build MDX pages, the MDX-formatted content needs to be on disk. In this quick overview, we will show how we ensured such on-disk Gatsby File nodes could be created programmatically from remote content.

The solution: Build cached files from buffers

One way to accomplish the goal of having MDX data on disk, is to write them there using the `createFileNodeFromBuffer`. You essentially create File nodes from a buffer with the right extension, including both frontmatter (yamlified json) and the MDX content in a string, and write it to disk.

Then Gatsby can build MDX pages from it, and you can use the regular MDX operations from that point

Include File node creation in your onCreateNode section

For keeping this post brief, I assume you have sourced nodes already that have a frontmatter field (JSON), and an mdx field (string).

We use the `onCreateNode` API to create the MDX File nodes for us, using the `createFileNodeFromBuffer`. You can then create File nodes programmatically from them, that then get processed by the MDX plugin. They work like any other MDX File node:

This is the magic sauce to make the MDX work, which we then built the [`gatsby-mdx-remote`](https://www.npmjs.com/package/gatsby-mdx-remote) plugin from 

```javascript
import { createRemoteFileNode } from "gatsby-source-filesystem";
  // ... onCreateNode in gatsby-node.js
  // We are assuming you have access to node.frontmatter and node.mdx in your onCreateNode section
  const mdxContent = node.frontmatter && node.mdx && `---\n${YAML.stringify(node.frontmatter)}\n---\n\n${node.mdx}`;
  const mdxFile =
    mdxContent &&
    (await createFileNodeFromBuffer({
      buffer: Buffer.from(mdxContent),
      ext: ".mdx",
      getCache: getCache,
      createNode: createNode,
      createNodeId: createNodeId,
    }));
  
  // ... rest of gatsby-node.js
```

Wrapping up our findings

Non-filesystem remote MDX content can be built into static websites in Gatsby using programmatically created File nodes with the `.mdx` extension. Such nodes can be created programmatically using the `createFileNodeFromBuffer` method by constructing a string. Other transformations on MDX nodes can then be performed as usual.

We are following up with another blog post on how to source nodes for consumption with the `onCreateNode` and later how to use the new plugins (`gatsby-mdx-remote` and `@dfrnt/gatsby-source-graphql-nodes`), as part of this series on GatsbyJS.

In conclusion, we are proud to say that our team at DFRNT continues to work hard to make our data products easily accessible to our customers and users, and this new development is a big step towards achieving this goal. We look forward to sharing more updates with you as we continue to innovate and improve our products.

Subscribe below to receive our upcoming blogposts as we unfold the Gatsby plugins to build static websites from TerminusDB data products.