Commit b66c79a3 authored by Josh Adam's avatar Josh Adam

Merge branch 'tabular-analysis-output-to-browser' into 'development'

Render tabular analysis output files in the browser

Closes #607

See merge request !1257
parents dcdfdace f173a3b9
Pipeline #6725 passed with stage
in 48 minutes and 33 seconds
......@@ -10,6 +10,7 @@ Changes
* [Developer]: Removed old gulp dependencies from the `package.json` file.
* [Developer]: Update to stable releases of `node` and `yarn`.
* [Administration]: Disabled automated SISTR results from saving to sample metadata. Also disabled retrospective results from being added during the database update. Installations that have already performed the 0.20.0 update will have their retrospective automated SISTR results automatically added to sample metadata. Installations that jump directly to 0.20.1 and above will not have this data added to sample metadata. (0.20.1)
* [UI/Workflow]: Preview analysis output files in a tabular or plain-text view in the analysis details page under the Preview tab.
0.19.0 to 0.20.0
----------------
......
......@@ -47,10 +47,10 @@ ul, ol, dl, figure,
* Images
*/
img {
max-width: 80%;
max-width: 90%;
margin-left: 30px;
vertical-align: middle;
box-shadow: 5px 5px 5px #888888;
box-shadow: 2px 2px 2px #888888;
}
img.logo {
......@@ -74,10 +74,22 @@ figure > img {
}
figcaption {
font-size: $small-font-size;
font-size: $smaller-font-size;
color: #888;
margin-left: 30px;
margin-bottom:20px;
margin-top: -10px;
max-width: 90%;
}
figcaption:before {
display: inline-block;
line-height: 1.2;
margin-right: 5px;
font-family: "FontAwesome";
content: "\f0d8";
color: #888;
}
/**
* Lists
......
......@@ -9,6 +9,8 @@
$base-font-family: 'Open Sans', Arial, sans-serif;
$base-font-size: 16px;
$small-font-size: $base-font-size * 0.875;
$smaller-font-size: $base-font-size * 0.75;
$tiny-font-size: $base-font-size * 0.5;
$base-line-height: 1.5;
$spacing-unit: 30px;
......
......@@ -163,11 +163,69 @@ You can either click on the <img src="images/download-icon.png" class="inline" a
The analysis details page shows you more detailed information about your pipeline submission, including the names of the files that were produced by the analysis (on the left-hand side of the page), a preview of the outputs (if available), and some tabs to view more details about how the pipeline was submitted:
![Analysis details page.](images/analysis-details-page.png)
<figcaption>Example SNVPhyl pipeline phylogenetic tree preview</figcaption>
![view-results-preview-refseq-masher]
<figcaption>Example `refseq_masher` tabular results preview</figcaption>
To download output files, you can use the "Output Files" section from this page. To download an individual file, click on the file name. To download *all* the outputs produced by the pipeline, you can click on the "Download Files" button.
![Analysis download.](images/analysis-details-download.png)
### Previewing analysis output files
All analysis pipelines produce analysis output files. You can preview these output files under the **Preview** tab on the **Analysis Details** page:
![view-results-preview-refseq-masher]
<figcaption>Example `refseq_masher` tabular results preview</figcaption>
For each analysis output file, you will see a panel and in each panel you will see:
- a panel heading with the Galaxy tool name (e.g. "RefSeq Masher Matches") and version (e.g. "(0.1.1)"), internal IRIDA output name (e.g. "refseq-masher-matches") and output file name (e.g. "refseq-masher-matches.tsv")
- a file download link
- a preview of the file contents displayed as plain text or in a table
#### Previewing tabular analysis output
Some of the output files will be rendered in a table:
![view-results-tabular-snvphyl-snv-table]
<figcaption>SNVPhyl SNV results shown in an interactive table.</figcaption>
![view-results-tabular-refseq_masher-contains-default]
<figcaption>`refseq_masher` results shown in an interactive table.</figcaption>
When you scroll to the bottom row in the table, more lines will be fetched as they are needed. You can also resize the table by clicking and dragging the "resize icon" in the corner of the panel:
![view-results-resize]
<figcaption>
Click and drag the "resize icon" to resize the table.
</figcaption>
![view-results-refseq_masher-contains-resized]
<figcaption>Resized `refseq_masher` results table</figcaption>
#### Default plain text preview of analysis output
Analysis output files with file extensions like `.log`, `.txt` or `.fasta` can be previewed in the browser as plain text:
![view-results-plain-text-snvphyl-mapping]
If an analysis output file is small enough like this log file, it will be loaded in its entirety:
![view-results-plain-text-shovill-log]
<figcaption>Notice that the `4.0 kB / 4.0 kB (100%)` indicates that 100% of the file has been loaded.</figcaption>
If an analysis output file is fairly large like this FASTA file, it will be loaded in chunks as needed for viewing:
![view-results-plain-text-shovill-fasta]
<figcaption>Notice that the `24.0 kB / 693.9 kB (3.5%)` indicates that only 3.5% of the file has been loaded into this view. Scrolling to the end will trigger loading of the next chunk of this file into the view!</figcaption>
### Viewing the sequencing data submitted for analysis
To view the data that was submitted to the pipeline for analysis, start from the [analysis details page](#viewing-pipeline-results) and click on the "Input Files" tab:
......@@ -254,4 +312,14 @@ If you follow the link to the analysis page, you can view all the job error info
This information may be helpful for troubleshooting and communicating what went wrong in a particular analysis pipeline.
<a href="../samples/">Previous: Managing Samples</a>
\ No newline at end of file
<a href="../samples/">Previous: Managing Samples</a>
[view-results-plain-text-shovill-fasta]: images/view-results-plain-text-shovill-fasta.png
[view-results-plain-text-shovill-log]: images/view-results-plain-text-shovill-log.png
[view-results-plain-text-snvphyl-mapping]: images/view-results-plain-text-snvphyl-mapping.png
[view-results-preview-refseq-masher]: images/view-results-preview-refseq-masher.png
[view-results-refseq_masher-contains-resized]: images/view-results-refseq_masher-contains-resized.png
[view-results-tabular-refseq_masher-contains-default]: images/view-results-tabular-refseq_masher-contains-default.png
[view-results-tabular-snvphyl-snv-table]: images/view-results-tabular-snvphyl-snv-table.png
[view-results-resize]: images/view-results-resize.png
......@@ -177,6 +177,17 @@ public class Analysis extends IridaResourceSupport implements IridaThing {
return false;
}
@Override
public String toString() {
return "Analysis{" + "id=" + id +
", createdDate=" + createdDate +
", description='" + description +
'\'' + ", executionManagerAnalysisId='" + executionManagerAnalysisId + '\'' +
", additionalProperties=" + additionalProperties +
", analysisOutputFilesMap=" + analysisOutputFilesMap +
", analysisType=" + analysisType + '}';
}
/**
* Get all output files produced by this {@link Analysis}.
*
......@@ -187,6 +198,10 @@ public class Analysis extends IridaResourceSupport implements IridaThing {
return ImmutableSet.copyOf(analysisOutputFilesMap.values());
}
public Map<String, AnalysisOutputFile> getAnalysisOutputFilesMap() {
return ImmutableMap.copyOf(this.analysisOutputFilesMap);
}
public String getDescription() {
return description;
}
......
package ca.corefacility.bioinformatics.irida.ria.utilities;
import java.io.BufferedReader;
import java.io.FileNotFoundException;
import java.io.IOException;
import java.io.RandomAccessFile;
import java.nio.charset.Charset;
import java.nio.file.Files;
import java.util.ArrayList;
import java.util.List;
import java.util.Set;
import java.util.regex.Matcher;
import java.util.regex.Pattern;
import java.util.stream.Collectors;
import java.util.zip.ZipEntry;
import java.util.zip.ZipOutputStream;
......@@ -29,6 +37,7 @@ public class FileUtilities {
public static final String CONTENT_TYPE_APPLICATION_ZIP = "application/zip";
public static final String CONTENT_TYPE_TEXT = "text/plain";
public static final String EXTENSION_ZIP = ".zip";
private static final Pattern regexExt = Pattern.compile("^.*\\.(\\w+)$");
/**
* Utility method for download a zip file containing all output files from
......@@ -143,4 +152,83 @@ public class FileUtilities {
private static String formatName(String name) {
return name.replace(" ", "_");
}
/**
* Get file extension from filename.
* <p>
* Uses simple regex to parse file extension {@code ^.*\.(\w+)$}.
*
* @param filename Filename
* @return File extension if found; otherwise empty string
*/
public static String getFileExt(String filename) {
Matcher matcher = regexExt.matcher(filename);
String ext = "";
if (matcher.matches()) {
ext = matcher.group(1);
}
return ext.toLowerCase();
}
/**
* Read bytes of length {@code chunk} of a file starting at byte {@code seek}.
*
* @param raf File reader
* @param seek FilePointer position to start reading at
* @param chunk Number of bytes to read from file
* @return Chunk of file as String
* @throws IOException if error enountered while reading file
*/
public static String readChunk(RandomAccessFile raf, Long seek, Long chunk) throws IOException {
raf.seek(seek);
byte[] bytes = new byte[Math.toIntExact(chunk)];
final int bytesRead = raf.read(bytes);
if (bytesRead == -1) {
return "";
}
return new String(bytes, 0, bytesRead, Charset.defaultCharset());
}
/**
* Read a specified number of lines from a file.
*
* @param reader File reader
* @param limit Limit to the number of lines to read
* @param start Optional line number to start reading at
* @param end Optional line number to read up to
* @return Lines read from file
*/
public static List<String> readLinesLimit(BufferedReader reader, Long limit, Long start, Long end) {
Long linesLimit = (limit != null) ? limit : 100L;
start = (start == null) ? 0 : start;
if (end != null && end > start) {
linesLimit = end - start + 1;
}
return reader.lines()
.skip(start == 0 ? 1L : start)
.limit(linesLimit)
.collect(Collectors.toList());
}
/**
* Read lines from file using a {@link RandomAccessFile}.
* <p>
* Use this method if preserving the {@link RandomAccessFile#getFilePointer()} for continuing reading is important.
* For most use cases, {@link FileUtilities#readLinesLimit(BufferedReader, Long, Long, Long)} will perform better
* due to bufffered reading.
*
* @param randomAccessFile File reader
* @param limit Limit to the number of lines to read
* @return Lines read from file
* @throws IOException if error enountered while reading file
*/
public static List<String> readLinesFromFilePointer(RandomAccessFile randomAccessFile, Long limit)
throws IOException {
ArrayList<String> lines = new ArrayList<>();
String line;
while (lines.size() < limit && (line = randomAccessFile.readLine()) != null) {
lines.add(line);
}
return lines;
}
}
package ca.corefacility.bioinformatics.irida.ria.web.analysis;
import java.io.BufferedReader;
import java.io.FileNotFoundException;
import java.io.FileReader;
import java.io.IOException;
import java.io.*;
import java.nio.file.Files;
import java.nio.file.Path;
import java.security.Principal;
......@@ -27,7 +24,6 @@ import org.springframework.web.bind.annotation.*;
import ca.corefacility.bioinformatics.irida.exceptions.EntityNotFoundException;
import ca.corefacility.bioinformatics.irida.exceptions.ExecutionManagerException;
import ca.corefacility.bioinformatics.irida.exceptions.IridaWorkflowNotFoundException;
import ca.corefacility.bioinformatics.irida.exceptions.NoPercentageCompleteException;
import ca.corefacility.bioinformatics.irida.model.enums.AnalysisState;
import ca.corefacility.bioinformatics.irida.model.enums.AnalysisType;
import ca.corefacility.bioinformatics.irida.model.joins.impl.ProjectMetadataTemplateJoin;
......@@ -42,9 +38,11 @@ import ca.corefacility.bioinformatics.irida.model.workflow.IridaWorkflow;
import ca.corefacility.bioinformatics.irida.model.workflow.analysis.Analysis;
import ca.corefacility.bioinformatics.irida.model.workflow.analysis.AnalysisOutputFile;
import ca.corefacility.bioinformatics.irida.model.workflow.analysis.JobError;
import ca.corefacility.bioinformatics.irida.model.workflow.analysis.ToolExecution;
import ca.corefacility.bioinformatics.irida.model.workflow.submission.AnalysisSubmission;
import ca.corefacility.bioinformatics.irida.model.workflow.submission.ProjectAnalysisSubmissionJoin;
import ca.corefacility.bioinformatics.irida.ria.utilities.FileUtilities;
import ca.corefacility.bioinformatics.irida.ria.web.analysis.dto.AnalysisOutputFileInfo;
import ca.corefacility.bioinformatics.irida.ria.web.components.datatables.DataTablesParams;
import ca.corefacility.bioinformatics.irida.ria.web.components.datatables.DataTablesResponse;
import ca.corefacility.bioinformatics.irida.ria.web.components.datatables.config.DataTablesRequest;
......@@ -62,8 +60,8 @@ import com.fasterxml.jackson.core.JsonParseException;
import com.fasterxml.jackson.core.type.TypeReference;
import com.fasterxml.jackson.databind.JsonMappingException;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.google.common.collect.ImmutableList;
import com.google.common.collect.ImmutableMap;
import com.google.common.collect.ImmutableSet;
/**
* Controller for Analysis.
......@@ -262,6 +260,177 @@ public class AnalysisController {
return "redirect:/analysis/" + submissionId;
}
/**
* For an {@link AnalysisSubmission}, get info about each {@link AnalysisOutputFile}
*
* @param id {@link AnalysisSubmission} id
* @return map of info about each {@link AnalysisOutputFile}
*/
@RequestMapping(value = "/ajax/{id}/outputs", method = RequestMethod.GET)
@ResponseBody
public List<AnalysisOutputFileInfo> getOutputFilesInfo(@PathVariable Long id) {
AnalysisSubmission submission = analysisSubmissionService.read(id);
Analysis analysis = submission.getAnalysis();
Set<String> outputNames = analysis.getAnalysisOutputFileNames();
return outputNames.stream()
.map((outputName) -> getAnalysisOutputFileInfo(submission, analysis, outputName))
.collect(Collectors.toList());
}
/**
* Get {@link AnalysisOutputFileInfo}.
*
* @param submission {@link AnalysisSubmission} of {@code analysis}
* @param analysis {@link Analysis} to get {@link AnalysisOutputFile}s from
* @param outputName Workflow output name
* @return {@link AnalysisOutputFile} info
*/
private AnalysisOutputFileInfo getAnalysisOutputFileInfo(AnalysisSubmission submission, Analysis analysis,
String outputName) {
// set of file extensions for indicating whether the first line of the file should be read
final ImmutableSet<String> FILE_EXT_READ_FIRST_LINE = ImmutableSet.of("tsv", "txt", "tabular", "csv", "tab");
final AnalysisOutputFile aof = analysis.getAnalysisOutputFile(outputName);
final Long aofId = aof.getId();
final String aofFilename = aof.getFile()
.getFileName()
.toString();
final ToolExecution tool = aof.getCreatedByTool();
final String toolName = tool.getToolName();
final String toolVersion = tool.getToolVersion();
final AnalysisOutputFileInfo info = new AnalysisOutputFileInfo();
info.setId(aofId);
info.setAnalysisSubmissionId(submission.getId());
info.setAnalysisId(analysis.getId());
info.setOutputName(outputName);
info.setFilename(aofFilename);
info.setFileSizeBytes(aof.getFile()
.toFile()
.length());
info.setToolName(toolName);
info.setToolVersion(toolVersion);
final String fileExt = FileUtilities.getFileExt(aofFilename);
info.setFileExt(fileExt);
if (FILE_EXT_READ_FIRST_LINE.contains(fileExt)) {
addFirstLine(info, aof);
}
return info;
}
/**
* Add the {@code firstLine} and {@code filePointer} file byte position after reading the first line of an {@link AnalysisOutputFile} to a {@link AnalysisOutputFileInfo} object.
*
* @param info Object to add {@code firstLine} and {@code filePointer} info to
* @param aof {@link AnalysisOutputFile} to read from
*/
private void addFirstLine(AnalysisOutputFileInfo info, AnalysisOutputFile aof) {
RandomAccessFile reader = null;
final Path aofFile = aof.getFile();
try {
reader = new RandomAccessFile(aofFile.toFile(), "r");
info.setFirstLine(reader.readLine());
info.setFilePointer(reader.getFilePointer());
} catch (FileNotFoundException e) {
logger.error("Could not find file '" + aofFile + "' " + e);
} catch (IOException e) {
logger.error("Could not read file '" + aofFile + "' " + e);
} finally {
try {
if (reader != null) {
reader.close();
}
} catch (IOException e) {
logger.error("Could not close file handle for '" + aofFile + "' " + e);
}
}
}
/**
* Read some lines or text from an {@link AnalysisOutputFile}.
*
* @param id {@link AnalysisSubmission} id
* @param fileId {@link AnalysisOutputFile} id
* @param limit Optional limit to number of lines to read from file
* @param start Optional line to start reading from
* @param end Optional line to stop reading at
* @param seek Optional file byte position to seek to and begin reading
* @param chunk Optional number of bytes to read from file
* @param response HTTP response object
* @return JSON with file text or lines as well as information about the file.
*/
@RequestMapping(value = "/ajax/{id}/outputs/{fileId}", method = RequestMethod.GET)
@ResponseBody
public AnalysisOutputFileInfo getOutputFile(@PathVariable Long id, @PathVariable Long fileId,
@RequestParam(defaultValue = "100", required = false) Long limit,
@RequestParam(required = false) Long start, @RequestParam(required = false) Long end,
@RequestParam(defaultValue = "0", required = false) Long seek, @RequestParam(required = false) Long chunk,
HttpServletResponse response) {
AnalysisSubmission submission = analysisSubmissionService.read(id);
Analysis analysis = submission.getAnalysis();
final Optional<AnalysisOutputFile> analysisOutputFile = analysis.getAnalysisOutputFiles()
.stream()
.filter(x -> Objects.equals(x.getId(), fileId))
.findFirst();
if (analysisOutputFile.isPresent()) {
final AnalysisOutputFile aof = analysisOutputFile.get();
final Path aofFile = aof.getFile();
final ToolExecution tool = aof.getCreatedByTool();
final AnalysisOutputFileInfo contents = new AnalysisOutputFileInfo();
contents.setId(aof.getId());
contents.setAnalysisSubmissionId(submission.getId());
contents.setAnalysisId(analysis.getId());
contents.setFilename(aofFile.getFileName()
.toString());
contents.setFileExt(FileUtilities.getFileExt(aofFile.getFileName()
.toString()));
contents.setFileSizeBytes(aof.getFile()
.toFile()
.length());
contents.setToolName(tool.getToolName());
contents.setToolVersion(tool.getToolVersion());
try {
final File file = aofFile.toFile();
final RandomAccessFile randomAccessFile = new RandomAccessFile(file, "r");
randomAccessFile.seek(seek);
if (seek == 0) {
if (chunk != null && chunk > 0) {
contents.setText(FileUtilities.readChunk(randomAccessFile, seek, chunk));
contents.setChunk(chunk);
contents.setStartSeek(seek);
} else {
final BufferedReader reader = new BufferedReader(new FileReader(randomAccessFile.getFD()));
final List<String> lines = FileUtilities.readLinesLimit(reader, limit, start, end);
contents.setLines(lines);
contents.setLimit((long) lines.size());
contents.setStart(start);
contents.setEnd(start + lines.size());
}
} else {
if (chunk != null && chunk > 0) {
contents.setText(FileUtilities.readChunk(randomAccessFile, seek, chunk));
contents.setChunk(chunk);
contents.setStartSeek(seek);
} else {
final List<String> lines = FileUtilities.readLinesFromFilePointer(randomAccessFile, limit);
contents.setLines(lines);
contents.setStartSeek(seek);
contents.setStart(start);
contents.setLimit((long) lines.size());
}
}
contents.setFilePointer(randomAccessFile.getFilePointer());
} catch (IOException e) {
logger.error("Could not read output file '" + aof.getId() + "' " + e);
response.setStatus(HttpServletResponse.SC_INTERNAL_SERVER_ERROR);
contents.setError("Could not read output file");
}
return contents;
} else {
response.setStatus(HttpServletResponse.SC_NOT_FOUND);
return null;
}
}
/**
* Get a map with list of {@link JobError} for an {@link AnalysisSubmission} under key `jobErrors`
* @param submissionId {@link AnalysisSubmission} id
......
package ca.corefacility.bioinformatics.irida.ria.web.analysis.dto;
import java.util.List;
/**
* DTO for {@link ca.corefacility.bioinformatics.irida.model.workflow.analysis.AnalysisOutputFile} text contents
*/
public class AnalysisOutputFileInfo {
/**
* {@link ca.corefacility.bioinformatics.irida.model.workflow.analysis.AnalysisOutputFile#id}
*/
private Long id;
/**
* {@link ca.corefacility.bioinformatics.irida.model.workflow.submission.AnalysisSubmission#id}
*/
private Long analysisSubmissionId;
/**
* {@link ca.corefacility.bioinformatics.irida.model.workflow.analysis.Analysis#id}
*/
private Long analysisId;
/**
* {@link ca.corefacility.bioinformatics.irida.model.workflow.analysis.AnalysisOutputFile} filename
*/
private String filename;
/**
* {@link ca.corefacility.bioinformatics.irida.model.workflow.analysis.AnalysisOutputFile} file extension
*/
private String fileExt;
/**
* {@link ca.corefacility.bioinformatics.irida.model.workflow.analysis.AnalysisOutputFile} file size in bytes
*/
private Long fileSizeBytes;
/**
* {@link ca.corefacility.bioinformatics.irida.model.workflow.analysis.ToolExecution#toolName}
*/
private String toolName;
/**
* {@link ca.corefacility.bioinformatics.irida.model.workflow.analysis.ToolExecution#toolVersion}
*/
private String toolVersion;
/**
* {@link ca.corefacility.bioinformatics.irida.model.workflow.analysis.AnalysisOutputFile} output name
*/
private String outputName;
/**
* Text read from file
*/
private String text;
/**
* Number of bytes to read at once
*/
private Long chunk;
/**
* User-specified byte position to seek to and start reading lines or bytes
*/
private Long startSeek;
/**
* {@link java.io.RandomAccessFile} file pointer; file byte position to which file was read
*/
private Long filePointer;
/**
* Lines read from the {@link ca.corefacility.bioinformatics.irida.model.workflow.analysis.AnalysisOutputFile}
*/
private List<String> lines;
/**
* First line from a tabular file which is expected to contain field headers
*/
private String firstLine;
/**
* Number of lines to read from tabular file
*/
private Long limit;
/**
* Line to start reading from in tabular file
*/
private Long start;
/**
* Line to read until in tabular file
*/
private Long end;
/**
* Error message if any
*/
private String error;
public AnalysisOutputFileInfo() {
this.id = null;
this.analysisSubmissionId = null;
this.analysisId = null;
this.filename = null;
this.fileExt = null;
this.fileSizeBytes = null;
this.toolName = null;
this.toolVersion = null;
this.outputName = null;
this.text = null;
this.chunk = null;
this.startSeek = null;
this.filePointer = null;
this.lines = null;
this.firstLine = null;
this.limit = null;
this.start = null;
this.end = null;
this.error = null;
}
@Override
public String toString() {
return "AnalysisOutputFileInfo{" + "id=" + id + ", analysisSubmissionId=" + analysisSubmissionId
+ ", analysisId=" + analysisId + ", filename='" + filename + '\'' + ", fileExt='" + fileExt + '\''
+ ", fileSizeBytes=" + fileSizeBytes + ", toolName='" + toolName + '\'' + ", toolVersion='"
+ toolVersion + '\'' + ", outputName='" + outputName + '\'' + ", text='" + text + '\'' + ", chunk="
+ chunk + ", startSeek=" + startSeek + ", filePointer=" + filePointer + ", lines=" + lines
+ ", firstLine='" + firstLine + '\'' + ", limit=" + limit + ", start=" + start + ", end=" + end
+ ", error='" + error + '\'' + '}';
}
public Long getId() {
return id;
}
public void setId(Long id) {
this.id = id;
}
public Long getAnalysisSubmissionId() {
return analysisSubmissionId;
}
public void setAnalysisSubmissionId(Long analysisSubmissionId) {
this.analysisSubmissionId = analysisSubmissionId;
}
public Long getAnalysisId() {
return analysisId;
}
public void setAnalysisId(Long analysisId) {
this.analysisId = analysisId;
}
public String getFilename() {
return filename;
}
public void setFilename(String filename) {
this.filename = filename;
}
public String getFileExt() {
return fileExt;
}
public void setFileExt(String fileExt) {
this.fileExt = fileExt;