...
 
Commits (14)
1.7.0 to 2.0.0
==============
* The UI is completely rewritten to better show what's happened with samples as they're uploaded, and to facilitate quality control implementation later.
* Add an experimental auto-upload feature to automatically upload new runs as the sequencer finishes a run (monitoring run directories for `CompletedJobInfo.xml`).
* Use the sample name and ID columns correctly in `SampleSheet.csv`: the sample name column is used for naming `.fastq.gz` files, the sample ID column is used for sending data to IRIDA.
* Discard the cache of projects whenever sending data to the server.
1.5.0 to 1.6.0
==============
* Add an about dialog to show the version number in the UI.
* Changed the way that update notifications are shown: don't have an open in browser button, just create a link.
* Changed the installer to uninstall the previous version before installing the new version (user settings are preserved).
* Use the `pubsub` built into wxPython rather than pulling in an external version.
1.4.0 to 1.5.0
==============
* Changed the `Makefile` so that the uploader can be hacked on in other Linux distros, specifically Arch (thanks to @eric.enns)
* Fixed a UI issue where upload labels were overlapping in some cases (thanks to @eric.enns)
* Fixed an issue where the uploader found files that had `SampleSheet.csv` *anywhere* in the filename (i.e., `old_SampleSheet.csv`), now it matches **exactly** `SampleSheet.csv`.
* Changed the label `Upload speed` to `Average upload speed` to more accurately reflect the value we're reporting.
* Added support for resuming uploads on failure. If the uploader fails when uploading a run, it will now skip any files that were already uploaded.
* Added better error reporting when the uploader can't find the files for the sample by including the directory to help find issues.
1.4.0 to 1.4.1
==============
* Fix a performance regression when scanning directories for fastq files (thanks to @eric.enns)
1.3.0 to 1.4.0
==============
* Use [pytest](https://www.pytest.org) for testing instead of the custom testing framework build arount `unittest`.
* Add a feature to check the github repository for new releases.
* Add a menu File > Exit menu for familiarity.
* Use a wx widget for directory selection instead of building our own.
* Sample sheet parsing happens with a state machine instead of guessing sections based on number of columns.
* Extensive refactoring to move application functionality out of the GUI layer.
1.2.1 to 1.3.0
==============
* Added caching for some responses from the server so that we don't have to do a round-trip for every sample.
* Added default directory setting to settings panel.
* Limiting depth of directory scanning to 2 levels.
1.2.0 to 1.2.1
==============
* Fixed a bug where empty fields in the header metadata section were being incorrectly parsed as read lengths.
1.1.0 to 1.2.0
==============
* Simplify searching for `SampleSheet.csv` using `os.walk`. Fixes a bug where Windows 7 junction points were being followed, and the OS was reporting permission denied errors on the junction points. This came from auto-scanning the default directory, and the default directory defaulting to the user's home directory.
* Add a license!
* Fix an issue where samples that have a common prefix would upload the wrong files for each sample (i.e., `Sample1`, `Sample11`, `Sample111` would all upload files for `Sample111`).
1.0.1 to 1.1.0
==============
* Fixed some bugs with trailing Windows newlines and trailing commas that get added to header information by Excel.
* Fixed a bug with the settings dialog where when focus was lost on a field the display would get messed up.
* Added a feature to automatically scan the default directory when the app starts up.
* Upgraded to wxWidgets 3 series, and fixed a bug where selecting directories included the drive label.
1.0.0 to 1.0.1
==============
* Changed the installer to use `pynsist` instead of by-hand construction in Windows with NSIS.
......@@ -2,17 +2,17 @@ IRIDA Uploader
==============
Installation
Download / Installation
--------------------
Installation instructions can be found in our documentation.
[ReadTheDocs](todo:link to read the docs needs to go here)
Creating the Windows installer
Creating the Windows installer from source code
------------------------------
First see the instructions for installing python 3 and pip.
A new windows installer can be built on linux, so first see the installation instructions for installing on linux in our documentation.
You will also need `nsis` installed to create the windows installer.
......
......@@ -735,8 +735,7 @@ class ApiCalls(object):
def get_seq_runs(self):
"""
Get list of pair files SequencingRuns
/api/sequencingRuns returns all SequencingRuns so this method
Get list of all SequencingRun objects
return list of SequencingRuns
"""
......@@ -840,7 +839,7 @@ class ApiCalls(object):
def project_exists(self, project_id):
"""
Check if a sample exists on a project
Check if a project exists
:param project_id: project that we are checking for existence
:return: True or False
......
......@@ -54,10 +54,6 @@ Example:
C:\Users\username> iridauploader --config \path\to\config.conf \path\to\my\samples\
```
### Linux:
Use the the `irida-uploader.sh` script included with the source code to upload.
`
# The `api` Module
The `api` module is essentially a python wrapper for the REST api.
The `api` module is essentially a python wrapper for the [IRIDA REST API](https://irida.corefacility.ca/documentation/developer/rest/).
It is used in the IRIDA uploader handle interaction between the rest of the code and IRIDA, but the module can be used on it's own to interact with IRIDA.
It is used in the IRIDA uploader to handle interaction between the uploader logic and IRIDA, but the module can be used on it's own to interact with IRIDA.
## Setup
The module can be used as follows:
```
```python
# import the module
import api
......@@ -19,43 +19,167 @@ api_instance = api.ApiCalls(client_id, client_secret, base_url, username, passwo
For more information on the arguments passed to `ApiCalls`, please see the [configuration documentation](../configuration.md)
## Use
The api is currently in development to improve usability for non-uploader applications. What follows is the current public method list.
Once the more robust api is finished, complete docs will be included here.
### Getting Data from IRIDA
#### get_projects(self)
API call to api/projects to get list of projects
**returns:**
List containing projects. each project is Project object.
#### get_samples(self, project_id)
API call to api/projects/project_id/samples
**arguments:**
project_id -- project identifier from irida
**returns:**
list of samples for the given project.
Each sample is a Sample object.
#### get_sequence_files(self, project_id, sample_name)
API call to api/projects/project_id/sample_id/sequenceFiles
We fetch the sample file through the project id on this route
**arguments:**
sample_name -- the sample id to get from irida, relative to a project
project_id -- the id of the project the sample is on
**returns:**
list of sequencefile dictionary for given sample_id
### Sending Data to IRIDA
#### send_project(self, project, clear_cache=True)
post request to send a project to IRIDA via API
the project being sent requires a name that is at least
5 characters long
**arguments:**
project -- a Project object to be sent.
**returns:**
A dictionary containing the result of post request.
when post is successful the dictionary it returns will contain the same
name and projectDescription that was originally sent as well as
additional keys like createdDate and identifier.
when post fails then an error will be raised so return statement is
not even reached.
#### send_sample(self, sample, project_id)
Post request to send a sample to a project
**arguments:**
sample -- Sample object to send
project_id -- id of project to send sample too
**returns:**
Unmodified json response from server
#### send_sequence_files(self, sequence_file, sample_name, project_id, upload_id)
Post request to send sequence files found in given sample argument
raises error if either project ID or sample ID found in Sample object
doesn't exist in irida
**arguments:**
sample -- Sample object
upload_id -- the run to upload the files to
**returns:**
unmodified json response from server.
### Getting / Creating / Modifying Sequencing Runs
#### get_seq_runs(self)
Get list of all SequencingRun objects
**returns:**
list of SequencingRuns
#### create_seq_run(self, metadata)
Create a sequencing run.
uploadStatus "UPLOADING"
There are some parsed metadata keys from the SampleSheet.csv that are
currently not accepted/used by the API so they are discarded.
Everything not in the acceptable_properties list below is discarded.
**arguments:**
metadata -- SequencingRun's metadata
**returns:**
the sequencing run identifier for the sequencing run that was created
#### set_seq_run_complete(self, identifier)
Update a sequencing run's upload status to "COMPLETE"
**arguments:**
identifier -- the id of the sequencing run to be updated
**returns:**
unmodified json response from server
#### set_seq_run_uploading(self, identifier)
Update a sequencing run's upload status to "UPLOADING"
**arguments:**
identifier -- the id of the sequencing run to be updated
**returns:**
unmodified json response from server
#### set_seq_run_error(self, identifier)
Update a sequencing run's upload status to "ERROR"
**arguments:**
identifier -- the id of the sequencing run to be updated
**returns:**
unmodified json response from server
### Querying IRIDA
#### project_exists(self, project_id)
Check if a sample exists on a project
**arguments:**
project_id -- project that we are checking for existence
**returns:**
True or False
#### sample_exists(self, sample_name, project_id)
Check if a sample exists on a project
**arguments:**
sample_name -- sample to confirm existence of
project_id -- project that we think the sample is on
**returns:**
#### sample_exists(self, sample_name, project_id)
\ No newline at end of file
True or False
......@@ -27,10 +27,11 @@ When creating a new project on IRIDA using the API, a `name` at least 5 characte
### Sample `model/sample.py`
The `Sample` object includes:
* `name` : How the sample is identified on IRIDA
* `description` : the description of the sample on IRIDA
* `sequence_file` : A `SequenceFile` object that holds the files to upload.
* `sample_dict` : a meta data dictionary
* `name` : How the sample is identified on IRIDA
* `description` : the description of the sample on IRIDA
* `sequence_file` : A `SequenceFile` object that holds the files to upload.
* `sample_dict` : a meta data dictionary
When using the API to get samples from IRIDA, the `get_irida_id` method can be used to get the samples numerical identification number.
......
#Common Errors
Here are some common error messages you might see when trying to upload.
##Parsing
No runs could be found in a directory: Double check that that is the directory with your sample sheet, and that you are using the right parser.
Alternatively, you may be using the wrong parser, check your configuration parser matches the type of run/directory you are trying to upload.
ERROR ERROR! An error occurred with directory 'tests/parsers/miseq/no_dirs/', with message: The directory tests/parsers/miseq/no_dirs/ has no sample sheet file in the MiSeq format with the name SampleSheet.csv
<br>
Invalid sample sheet. Problems were found when trying to parse your sample sheet.
Pay close attention to the error list
ERROR Errors occurred while getting sample sheet
ERROR ERROR! Errors occurred during validation with message: Errors occurred while getting sample sheet
ERROR Error list: [SampleSheetError('[Header] section not found in SampleSheet', 'tests/parsers/miseq/invalid_sample_sheet/SampleSheet.csv'),
SampleSheetError('[Data] section not found in SampleSheet', 'tests/parsers/miseq/invalid_sample_sheet/SampleSheet.csv'),
SampleSheetError('[Reads] section not found in SampleSheet', 'tests/parsers/miseq/invalid_sample_sheet/SampleSheet.csv'),
SampleSheetError('Missing required data header(s): Sample_ID, Sample_Name, Sample_Project, Description', 'tests/parsers/miseq/invalid_sample_sheet/SampleSheet.csv')]
<br>
Files matching the sample name could not be found, check your sample sheet and file names
ERROR Errors occurred while building sequence run from sample sheet
ERROR ERROR! Errors occurred during validation with message: Errors occurred while building sequence run from sample sheet
ERROR Error list: [SequenceFileError('The uploader was unable to find an files with a file name that ends with .fastq.gz for the sample in your sample sheet with name 011111 in the directory tests/parsers/miseq/ngs_not_pf_list/Data/Intensities/BaseCalls. This usually happens when the Illumina MiSeq Reporter tool does not generate any FastQ data.',)]
<br>
To many, or not enough files found with names matching the sample name.
ERROR Errors occurred while building sequence run from sample sheet
ERROR ERROR! Errors occurred during validation with message: Errors occurred while building sequence run from sample sheet
ERROR Error list: [SequenceFileError("The following file list ['01-1111_S1_L001_R1_001.fastq.gz', '01-1111_S1_L001_R3_001.fastq.gz', '01-1111_S1_L001_R2_001.fastq.gz'] found in the directory tests/parsers/miseq/ngs_not_valid_pf_list/Data/Intensities/BaseCalls is invalid. Please verify the folder containing the sequence files matches the SampleSheet file",)]
<br>
Invalid haracter in sample name (such as a space). This is not allowed on IRIDA
ERROR Did not create sample on server. Response code is '400' and error message is '{"sampleName":["The name you have supplied contains a space character. Names must NOT include the space character and the following: ? ( ) [ ] / = + < > : ; \" ' , * ^ | & ."]}'
##Uploading
<br>
Invalid credentials in configuration file, double check that your configuration file is correct
ERROR Can not get access token from IRIDA
ERROR ERROR! Could not initialize irida api.
ERROR Errors: ('Could not get access token from IRIDA. Credentials may be incorrect. IRIDA '
"returned with error message: ('Decoder failed to handle access_token with "
'data as returned by provider. A different decoder may be needed. Provider '
'returned: b\\\'{"error":"invalid_grant","error_description":"Bad '
'credentials"}\\\'\',)',)
<br>
Cannot connect to IRIDA, it is likely the base url in the configuration file is incorrect.
Also check that the IRIDA instance you are trying to connect to is running.
Alternatively, the client_id / client_secret may be incorrect
ERROR Can not connect to IRIDA
ERROR ERROR! Could not initialize irida api.
ERROR Errors: ('Could not connect to the IRIDA server. URL may be incorrect. IRIDA returned '
'with error message: (MaxRetryError("HTTPConnectionPool(host=\'lcalhost\', '
'port=8080): Max retries exceeded with url: /irida-latest/api/oauth/token '
"(Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at "
'0x7f2d3b6759b0>: Failed to establish a new connection: [Errno -2] Name or '
'service not known\',))",),)',)
<br>
An invalid parser name was given, Double check your config file has a valid parser in the parser field
AssertionError: Bad parser creation, invalid parser_type given: miseqa
......@@ -8,6 +8,7 @@
## Upcoming Features
* Automated uploads
* File upload checksum validation
* Post-processing tasks
* GUI
* Pause and resume uploads
......@@ -30,7 +31,7 @@ TODO: You can download pre-built packages for Windows from our [GitHub releases
Run an installer (links above) and follow along with the install wizard.
You will need to configure your uploader before running.
You will need to configure your uploader before running. See [Configuration](configuration.md) for details
#### Linux
......@@ -105,6 +106,15 @@ Use the the `irida-uploader.sh` script included with the source code to upload.
`./irida-uploader.sh /path/to/the/sequencing/run/`
# Problems?
### Problems uploading?
Check the [Errors Page](errors.md) for help with common errors.
### Found a bug or have an idea for a feature?
Create an issue on our [GitHub](todo: link to github)
# Developers
Want to create a parser for a sequencer that we don't yet support or have an idea for an IRIDA related project?
......
......@@ -8,6 +8,7 @@ use_directory_urls: false
nav:
- Overview: index.md
- configuration.md
- errors.md
- Parsers:
- parsers/directory.md
- parsers/miseq.md
......