Working with data

Overview

LocusZoom.js aims to provide reusable and highly customizable visualizations. Towards this goal, a separation of concerns is enforced between data adapters (data) and data layers (presentation).

Your first plot: defining how to retrieve data

All data retrieval is performed by adapters: special objects whose job is to fetch the information required to render a plot. A major strength of LocusZoom.js is that it can connect several kinds of annotation from different places into a single view: the act of organizing data requests together is managed by an object called LocusZoom.DataSources.

Below is an example that defines how to retrieve the data for a “classic” LocusZoom plot, in which GWAS, LD, and recombination rate are overlaid on a scatter plot, with genes and gnomAD constraint information on another track below. In total, five REST API endpoints are used to create this plot: four standard datasets, and one user-provided summary statistics file.

const apiBase = 'https://portaldev.sph.umich.edu/api/v1/';
const data_sources = new LocusZoom.DataSources()
    .add('assoc', ['AssociationLZ', {url: apiBase + 'statistic/single/', source: 45, id_field: 'variant' }])
    .add('ld', ['LDServer', { url: 'https://portaldev.sph.umich.edu/ld/', source: '1000G', population: 'ALL', build: 'GRCh37' }])
    .add('recomb', ['RecombLZ', { url: apiBase + 'annotation/recomb/results/', build: 'GRCh37' }])
    .add('gene', ['GeneLZ', { url: apiBase + 'annotation/genes/', build: 'GRCh37' }])
    .add('constraint', ['GeneConstraintLZ', { url: 'https://gnomad.broadinstitute.org/api/', build: 'GRCh37' }]);

Of course, defining datasets is only half of the process; see the Getting Started Guide for how to define rendering instructions (layout) and combine these pieces together to create the LocusZoom plot.

Understanding the example

In the example above, a new data source is added via a line of code such as the following:

data_sources.add('assoc', ['AssociationLZ', {url: apiBase + 'statistic/single/', source: 45, id_field: 'variant' }]);

A lot is going on in this line!

The importance of genome build

You may notice that in the example above, many of the datasets specify build: 'GRCh37. For “standard” datasets that are widely used (LD, genes, recombination, and GWAS catalog), the UMich APIs will automatically try to fetch the most up-to-date list of genes and GWAS catalog entries for the specified genome build. We currently support build GRCh37 and GRCh38. Be sure to use the genome build that matches your dataset.

We periodically update our API server. If you think new information is missing, please let us know.

What should the data look like?

In theory, LocusZoom.js can display whatever data it is given: layouts allow any individual layer to specify what fields should be used for the x and y axes.

In practice, it is much more convenient to use pre-existing layouts that solve a common problem well out of the box: the set of options needed to control point size, shape, color, and labels is rather verbose, and highly custom behaviors entail a degree of complexity that is not always beginner friendly. For basic LocusZoom.js visualizations, our default layouts assume that you use the field names and format conventions defined in the UM PortalDev API docs. This is the quickest way to get started. Most layouts in the provided LocusZoom.Layouts registry will tell you what fields are expected (via the _auto_fields property), and most plots will log a console error message if the response is missing expected information.

Most users will only need to implement their own way of retrieving GWAS summary statistics; the other annotations are standard datasets and can be freely used from our public API. For complex plots (like annotations of new data), see our example gallery.

How data gets to the plot

If you are building a custom tool for exploring data, it is common to show the same data in several ways (eg, a LocusZoom plot next to a table of results). The user will have a better experience if the two widgets are synchronized to always show the same data, which raises a question: which widget is responsible for making the API request?

In LocusZoom.js, the user is allowed to change the information shown via mouse interaction (drag or zoom to change region, change LD calculations by clicking a button… etc). This means that LocusZoom must always be able to ask for the data it needs, and initiate a new request to the repository if the required data is not available locally: a pull approach. This contrasts with static plotting libraries like matplotlib or excel that render whatever data they are given initially (a push approach).

The act of contacting an external data repository, and fetching the information needed, is coordinated by Adapters. It is possible to share data with other widgets on the page via event callbacks, so that those widgets retrieve the newest data whenever the plot is updated (see subscribeToData in the guide to interactivity for details).

Not every web page requires an API

LocusZoom.js is designed to work well with REST APIs, but you do not need to create an entire web server just to render a single interactive plot. As long as the inputs can be transformed into a recognized format, they should work with the plot.

Some examples of other data retrieval mechanisms used in the wild are:

Example: Loading data from static JSON files

One way to make a LocusZoom plot quickly is to load the data for your region in a static file, formatted as JSON objects to look like the payload from our standard REST API. The key concept below is that instead of a server, the URL points to the static file. This demonstration is subject to the limits described above, but it can be a way to get started.

data_sources = new LocusZoom.DataSources()
    .add("assoc", ["AssociationLZ", {url: "assoc_10_114550452-115067678.json", params: {source: null}}])
    .add("ld", ["LDLZ", { url: "ld_10_114550452-115067678.json" }])
    .add("gene", ["GeneLZ", { url: "genes_10_114550452-115067678.json" }])
    .add("recomb", ["RecombLZ", { url: "recomb_10_114550452-115067678.json" }])
    .add("constraint", ["GeneConstraintLZ", {  url: "constraint_10_114550452-115067678.json" }]);

Mix and match

Each data adapter in the chain is largely independent, and it is entirely normal to mix data from several sources: for example, GWAS data from a tabix file alongside genes data from the UMich API server.

If a single data layer needs to combine two kinds of data (eg association and LD), you will achieve the best results if the sources have some common assumptions about data format. Adapters are highly modular, but because they do not enforce a specific contract of field names or payload structure, you are responsible for ensuring that the resulting data works with the assumptions of your layout.

Every layer has its own, private, view of the data

Each data layer in LocusZoom is intended to be independent of others. Thus, each layer must individually specify its own way to connect data together.

Likewise, each layer defines a “local” way of identifying where to find the data it needs. For example, consider a layer layout as follows:

{
  namespace: {'assoc': 'mystudy'},
  id_field: 'assoc:variant'
}

The namespace is specified as a key-value pair of local_name: global_data_source_name. This instruction says: “whenever this layer asks for something from assoc, make a request to an item named mystudy”. Every field in the layout refers to the local_name. This layer of indirection allows a layout to be used many times to plot different datasets, and only the namespace needs to be changed.

Any changes made to the data within one layer should not affect copies of the same data in other places. This property makes it easier to display the same data in two different ways, without having to make a duplicate network request.

What if my data doesn’t fit the expected format?

The built-in adapters are designed to work with a specific set of known REST APIs and fetch data over the web, but we provide mechanisms to customize every aspect of the data retrieval process, including how to construct the query sent to the server and how to modify the fields returned. See the guidance on “custom adapters” below.

In general, the more similar that your field names are to those used in premade layouts, the easier it will be to get started with common tasks. Certain features require additional assumptions about field format, and these sorts of differences may cause behavioral (instead of cosmetic) issues. For example:

If the only difference is field names, you can customize the layout to tell it where to find the required information. (see: guide to layouts and rendering for details) Transformation functions (like neglog10) can then be used to ensure that custom data is formatted in a way suitable for rendering and plotting.

Combining two kinds of data in one place

Sometimes, a single data layer will depend on information from two different sources (like an association plot colored by LD information).

Data operations (layout directive)

The act of combining two pieces of data is performed by data operations. In order to let the same piece of data be rendered in different ways, these instructions are placed within each data layer layout. The decision on how to combine two pieces of information is specified at the point where the data is used.

A sample layout directive for data operations would be as follows:

{
    namespace: { 'assoc': 'mystudy', 'ld': 'some_ld' },
    data_operations: [
        {
            type: 'fetch',
            from: ['assoc', 'ld(assoc)'],
        },
        {
            type: 'left_match',
            name: 'assoc_plus_ld',
            requires: ['assoc', 'ld'],
            // Tell the join function which field names to be used for the join. Each join function has its own possible parameters.
            params: ['assoc:position', 'ld:position2'], 
        },
    ]
}

Dependencies

The first instruction in the example above specifies how to retrieve the data: {type: 'fetch', from: ['assoc', 'ld(assoc)'}

A single “fetch” operation is used to specify all of the data retrieval. This specifies what information a data layer should retrieve. It refers to the “local” namespace name (like “assoc”), and tells locuszoom to make a request to the global datasource named “mystudy”.

Some adapters can not begin to define their request until they see the data from another adapter: eg LD information is retrieved relative to the most significant association variant in the region. This is a dependency, and one (or more) dependencies can be specified using the syntax some_request(first_dependency, nth_dependency). LocusZoom will automatically reorder and parallelize requests based on the dependencies given. If several requests each have no dependencies, they will be performed in parallel.

Each individual adapter will receive data it depends on as a function argument to getData. The string syntax reflects how the data is connected internally! In the example above, “ld” is only retrieved after a request to “assoc” is complete, and the LD adapter received the assoc data to help define the next request.

NOTE: Sometimes it is useful to extend an existing layout to fetch additional kinds of data. It can be annoying to extend a “list” field like fetch.from, so LocusZoom has some magic to help out: if an item is specified as a source of data for this layer (eg namespace: {'assoc': 'assoc'}), it will automatically be added to the fetch.from instruction. This means that everything listed in data_layer.namespace will trigger a data retrieval request. You only need to modify the contents of fetch.from if you want to specify that the new item depends on something else, eg “don’t fetch LD until assoc data is ready: ld(assoc)”. All of the built in LZ layouts are verbose and explicit, in an effort to reduce the amount of “magic” and make the examples easier to understand.

Joins (“Data functions”)

The second instruction in the example above is a join. It specifies how to compare multiple sets of data into one list of records (“one record per data point on the plot”).

Any function defined in LocusZoom.DataFunctions can be referenced. Several builtin joins are provided (including left_match, inner_match, and full_outer_match). Any arbitrary join function can be added to the LocusZoom.DataFunctions registry, with the call signature ({plot_state, data_layer}}, [recordsetA, recordsetB...], ...params) => combined_results.

Data operations are synchronous; they are used for data formatting and cleanup. Use adapters for asynchronous operations like network requests.

The plot will receive the last item in the dependency graph (taking into account both adapters and joins). If something looks strange, make sure that you have specified how to join all your data sources together.

TIP: Because data functions are given access to plot.state and the data layer instance that initiated the request, they are able to do surprisingly powerful things, like filtering the returned data in response to a global plot_state parameter or auto-defining a layout (data_layer.layout) in response to dynamic data (axis labels, color scheme, etc). This is a very advanced usage and has the potential for side effects; use sparingly! See the LocusZoom unit tests for a few examples of what is possible.

Creating your own custom adapter

Can I reuse existing code?

The built-in LocusZoom adapters can be used as-is in many cases, so long as the returned payload matches the expected format. You may need to write your own custom code (adapter subclass) in the following scenarios:

  1. If an expression must be evaluated in order to retrieve data (eg constructing URLs based on query parameters, or LD requests where the reference variant is auto-selected from the most significant hit in a GWAS)
  2. If the actual headers, body, or request method must be customized in order to carry out the request. This happens when data is retrieved from a particular technology (REST vs GraphQL vs Tabix), or if the request must incorporate some form of authentication credentials.
  3. If some of the fields in your custom API format need to be transformed or renamed in order to match expectations in LZ code. For example, the LD adapter may try to suggest an LD reference variant by looking for the GWAS variant with the largest log_pvalue. (over time, built-in LZ adapters trend towards being more strict about field names; it is easier to write reliable code when not having to deal with unpredictable data!)

Customizing data retrieval via subclasses

Most custom sites will only need to change very small things to work with their data. For example, if your REST API uses the same payload format as the UM PortalDev API, but a different way of constructing queries, you can change just one function and define a new data adapter:

const AssociationLZ = LocusZoom.Adapters.get('AssociationLZ');
class CustomAssociation extends AssociationLZ {
    _getURL(request_options) {
        // Every adapter receives the info from plot.state, plus any additional request options calculated/added in the function `_buildRequestOptions`
        // The inputs to the function can be used to influence what query is constructed. Eg, since the current view region is stored in `plot.state`:
        const {chr, start, end} = request_options;
        // Fetch the region of interest from a hypothetical REST API that uses query parameters to define the region query, for a given study URL such as `data.example/gwas/<id>/?chr=_&start=_&end=_`
        return `${this._url}/${this.source}/?chr=${encodeURIComponent(chr)}&start=${encodeURIComponent(start)}&end${encodeURIComponent(end)}`
  }
  
  _annotateRecords(records, options) {
      // Imagine that your API returns a payload that mostly works, but a few fields are a little different than what is expected.
      // Usually LocusZoom is pretty tolerant of changing field names- but because Association plots connect to so many extra annotations, certain magic features require a little extra attention to detail
      records.forEach((item) => {
          // LocusZoom connects assoc summstats to other datasets, by finding the most significant variant. To find that variant and ask for LD, it expects two special fields with very specific names, and sort of specific formats. If you already have these fields, great- if not, they can be manually added by custom code, even if no one will let you change the API server directly
          item.log_pvalue = -Math.log10(item.pValue);  // It's better if you send log_pvalue from the server directly, not calculate it on the front end (JS is subject to numerical underflow, and sending pValue to the browser may cause numerical underflow problems)
          item.variant = `${item.chrom}:${item.pos}_${item.ref}/${item.alt}`;
      });
      return records;
  }
}
// A custom adapter should be added to the registry before using it
LocusZoom.Adapters.add('CustomAssociation', CustomAssociation);

// From there, it can be used anywhere throughout LocusZoom, in the same way as any built-in adapter
data_sources.add('mystudy', ['CustomAssociation', {url: 'https://data.example/gwas', source: 42 }]);

In the above example, an HTTP GET request will be sent to the server every time that new data is requested. If further control is required (like sending a POST request with custom body), you may need to override additional methods such as fetchRequest. See below for more information, then consult the detailed developer documentation for details.

Common types of data retrieval that are most often customized:

What happens during a data request?

The adapter performs many functions related to data retrieval: constructing the query, caching to avoid unnecessary network traffic, and parsing the data into a transformed representation suitable for use in rendering.

Methods are provided to override all or part of the process, called in roughly the order below:

getData(plot_state, ...dependent_data)
    _buildRequestOptions(plot_state, ...dependent_data)
    _getCacheKey(request_options)
    _performRequest(request_options)
        _getURL(request_options)
    // Parse JSON, convert columnar data to rows, etc
    _normalizeResponse(raw_response, options)
        cache.add(records)
    // User-specified cleanup of the parsed response fields. Can be used for things like field renaming, record filtering, or adding new calculated fields on the fly
    _annotateRecords(records, options)
    // Usually not overridden: this method can be used to apply prefixes (assoc:id, catalog:id, etc) and to simplify down verbose responses to just a few `limit_fields` of interest
    _postProcessResponse(records, options)

The parameters passed to getData are as follows:

This function will return row-format data representing the response for just that request. It will use a cached response if feasible. By default, LocusZoom will apply certain transformation to the returned response so that it is easier to use with a data layer (see adapter documentation for details). All fields in the original response will be returned as given.

Step 1: Fetching data from a remote server and converting to records

The first step of the process is to retrieve the data from an external location. _buildRequestOptions can be used to store any parameters that customize the request, including information from plot.state. If the response is cached, the cache value will be returned; otherwise, a request will be initiated. Once the request is complete, the result will be parsed and cached for future use.

At the conclusion of this step, we typically have an array of records, one object per row of data. The field names and contents are very close to what was returned by the API.

Most custom data sources will focus on customizing two things:

Step 2: Transforming data into a form usable by LocusZoom

In a prior step, the records were normalized, but kept in a form that is usually close to what the API returned. In this step, records are modified for use in the plot:

See also