storage Package

This is just a shortcut, in order to make all DAO functions accessible from a single point, in an uniform manner, without supplementary name space. We want DAO functions to be separated in multiple files, because they are too many to easily follow.

class tvb.core.entities.storage.DAO

Bases: tvb.core.entities.storage.datatype_dao.DatatypeDAO, tvb.core.entities.storage.operation_dao.OperationDAO, tvb.core.entities.storage.project_dao.CaseDAO, tvb.core.entities.storage.burst_dao.BurstDAO, tvb.core.entities.storage.workflow_dao.WorkflowDAO

Empty class, build only for inheriting from all DAO classes.

session = <tvb.core.entities.storage.session_maker.SessionMaker object at 0x7f71af27b750>

burst_dao

class tvb.core.entities.storage.burst_dao.BurstDAO

Bases: tvb.core.entities.storage.root_dao.RootDAO

DAO layer for Burst entities.

count_bursts_with_name(burst_name, project_id)

Return the number of burst already named ‘custom_b%’ and NOT ‘custom_b%_%’ in current project.

get_burst_by_id(burst_id)

Get the BurstConfiguration entity with the given id

get_burst_for_operation_id(operation_id)

get_bursts_for_project(project_id, page_start=0, page_size=None, count=False)

Get latest 50 BurstConfiguration entities for the current project

get_max_burst_id()

Return the maximum of the currently stored burst IDs to be used as the new burst name. This is not a thread-safe value, but we use it just for a label.

session = <tvb.core.entities.storage.session_maker.SessionMaker object at 0x7f71af27b750>

datatype_dao

DAO operations related to generic DataTypes are defined here.

class tvb.core.entities.storage.datatype_dao.DatatypeDAO

Bases: tvb.core.entities.storage.root_dao.RootDAO

DATATYPE and DATA_TYPES_GROUPS RELATED METHODS

compute_bursts_disk_size(burst_ids)

SUM the disk_size of all data types generated by each requested burst Do not count DataType Groups as those already include the size of the entities inside the group. :returns a map from burst id to disk size

count_all_datatypes()

Gives you the count of all the datatypes currently stored by TVB. Is used by the file storage update manager to upgrade from version to the next.

count_datatypes(project_id, datatype_class)

count_datatypes_generated_from(datatype_gid)

Returns a count of all the datatypes that were generated by an operation having as input the datatype ginen by ‘datatype_gid’

count_datatypes_in_burst(burst_id)

Returns the number of DataTypes from the specified DataTypeGroup ID.

count_datatypes_in_group(datatype_group_id)

Returns the number of DataTypes from the specified DataTypeGroup ID.

get_all_datatypes(page_start=0, page_size=20)

Return a list with all of the datatypes currently available in TVB. Is used by the file storage update manager to upgrade from version to the next.

Parameters:

• page_start – the index from which to start adding datatypes to the result list
• page_size – maximum number of entities to retrieve

get_data_in_project(project_id, visibility_filter=None, filter_value=None)

Get all the DataTypes for a given project, including Linked Entities and DataType Groups.

Parameters:

• visibility_filter – when not None, will filter by DataTye fields
• filter_value – when not None, will filter with ilike multiple DataType string attributes

get_datatype_by_gid(gid, load_lazy=True)

Retrieve a DataType DB reference by a global identifier.

get_datatype_details(datatype_gid)

Returns the details for the dataType with the given GID.

get_datatype_group_by_gid(datatype_group_gid)

Returns the DataTypeGroup with the specified gid.

get_datatype_group_disk_size(dt_group_id)

Return the size of all the DataTypes from this datatype group.

get_datatype_in_group(datatype_group_id=None, operation_group_id=None)

Return a list of id-s of the DataTypes in the given dt group.

get_datatypegroup_by_op_group_id(operation_group_id)

Returns the DataTypeGroup corresponding to a certain OperationGroup.

get_datatypes_for_range(op_group_id, range_json)

Retrieve from DB, DataTypes resulted after executing a specific range operation.

get_datatypes_from_datatype_group(datatype_group_id)

Retrieve all datatype which are part from the given datatype group.

get_datatypes_in_project(project_id, only_visible=False)

Get all the DataTypes for a given project with no other filter apart from the projectId

get_disk_size_for_operation(operation_id)

Return the disk size for the operation by summing over the disk space of the resulting DataTypes.

get_last_data_with_uid(uid, datatype_class=<class 'tvb.core.entities.model.model_datatype.DataType'>)

Retrieve the last dataType ID witch has UDI field as the passed parameter, or None if nothing found.

get_linked_datatypes_in_project(project_id)

Return a list of datatypes linked into this project :param project_id: the id of the project

get_links_for_datatype(data_id)

Get the links to a specific datatype

get_selections_for_project(project_id, datatype_gid, filter_ui_name=None)

Get available selections for a given project and data type.

get_stored_pse_filters(datatype_group_gid, filter_ui_name=None)

Returns:Stored PSE filters for a given DatTypeGroup or None

get_summary_for_group(datatype_group_id)

:return (disk_size SUM, subject)

get_values_of_datatype(project_id, datatype_class, filters=None, page_size=50)

Retrieve a list of dataTypes matching a filter inside a project. :returns: (results, total_count) maximum page_end rows are returned, to avoid endless time when loading a page

is_datatype_group(datatype_gid)

Used to check if the DataType with the specified GID is a DataTypeGroup.

session = <tvb.core.entities.storage.session_maker.SessionMaker object at 0x7f71af27b750>

set_datatype_visibility(datatype_gid, is_visible)

Sets the dataType visibility. If the given dataType is a dataTypeGroup or it is part of a dataType group than this method will set the visibility for each dataType from this group.

try_load_last_entity_of_type(project_id, datatype_class)

try_load_last_surface_of_type(project_id, surface_type)

exceptions

Created on Jan 15, 2013

exception tvb.core.entities.storage.exceptions.BaseStorageException(message)

Bases: builtins.Exception

Base class for all TVB storage exceptions.

exception tvb.core.entities.storage.exceptions.InvalidTransactionAccess(message)

Bases: tvb.core.entities.storage.exceptions.BaseStorageException

Exception raised in case you have any faulty access to a transaction.

exception tvb.core.entities.storage.exceptions.NestedTransactionUnsupported(message)

Bases: tvb.core.entities.storage.exceptions.BaseStorageException

Nested transactions are not supported unless in testing.

operation_dao

DAO operations related to Algorithms and User Operations are defined here.

class tvb.core.entities.storage.operation_dao.OperationDAO

Bases: tvb.core.entities.storage.root_dao.RootDAO

OPERATION RELATED METHODS

compute_disk_size_for_started_ops(user_id)

Get all the disk space that should be reserved for the started operations of this user.

count_resulted_datatypes(operation_id)

Returns the number of resulted datatypes from the specified operation.

filter_category(displayname, rawinput, display, launchable, order_nr)

Retrieve category with given id

get_adapters_from_categories(categories)

Retrieve a list of stored adapters in the given categories.

get_algorithm_by_id(algorithm_id)

get_algorithm_by_module(module_name, class_name)

get_algorithm_categories()

Retrieve all existent categories of Algorithms.

get_all_operations_for_uploaders(project_id)

Returns all finished upload operations.

get_applicable_adapters(compatible_class_names, launch_categ)

Retrieve a list of algorithms in a given list of categories with a given dataType classes as required input.

get_category_by_id(categ_id)

Retrieve category with given id

get_figure_count(project_id, user_id)

Used to generate sequential image names.

get_figures_for_operation(operation_id)

Retrieve Figure entities, resulted after executing an operation.

get_filtered_operations(project_id, filter_chain, page_start=0, page_size=20, is_count=False)

Parameters:

• project_id – current project ID
• filter_chain – instance of FilterChain
• is_count – when True, return a number, otherwise the list of operation entities

:return a list of filtered operation in current project, page by page, or the total count for them.

get_launchable_categories(elimin_viewers=False)

Retrieve algorithm categories which can be launched on right-click (optionally filter visualizers)

get_operation_by_gid(operation_gid)

Retrieve OPERATION entity for a given gid.

get_operation_by_id(operation_id)

Retrieve OPERATION entity for a given Identifier.

get_operation_numbers(proj_id)

Count total number of operations started for current project.

get_operation_process_for_operation(operation_id)

Get the OperationProcessIdentifier for this operation id.

get_operationgroup_by_gid(gid)

Retrieve by GID

get_operationgroup_by_id(op_group_id)

Retrieve by ID

get_operations_for_datatype(datatype_gid, only_relevant=True, only_in_groups=False)

Returns all the operations which uses as an input parameter the dataType with the specified GID. If the flag only_relevant is True than only the relevant operations will be returned.

If only_in_groups is True than this method will return only the operations that are part from an operation group, otherwise it will return only the operations that are NOT part of an operation group.

get_operations_for_datatype_group(datatype_group_id, only_relevant=True, only_in_groups=False)

Returns all the operations which uses as an input parameter a datatype from the given DataTypeGroup. If the flag only_relevant is True than only the relevant operations will be returned.

If only_in_groups is True than this method will return only the operations that are part from an operation group, otherwise it will return only the operations that are NOT part of an operation group.

get_operations_in_burst(burst_id, is_count=False)

Return a list with all the operations generated by a given burst. These need to be removed when the burst is deleted. :param: is_count When True, a counter of the filtered operations is returned.

get_operations_in_group(operation_group_id, is_count=False, only_first_operation=False, only_gids=False)

Retrieve OPERATION entities for a given group.

get_previews(project_id, user_id=None, selected_session_name='all_sessions')

This method returns a tuple of 2 elements. The first element represents a dictionary of form {‘$session_name’: [list_of_figures]}. This dictionary contains data only for the selected self.session. If the selected session is ‘all_sessions’ than it will contain data for all the sessions. The second element of the returned tuple is a dictionary of form {‘$session_name’: $no_of_figures_in_this_session, ...}. This dictionary contains information about all the sessions.

selected_session_name - represents the name of the session for which you

want to obtain the stored figures.

get_raw_categories()

Retrieve categories with raw_input = true

get_results_for_operation(operation_id, filters=None)

Retrieve DataTypes entities, resulted after executing an operation.

get_uploader_categories()

Retrieve categories with raw_input = true

get_visualisers_categories()

Retrieve categories with display = true

is_upload_operation(operation_gid)

Returns True only if the operation with the given gid is an upload operation.

load_figure(figure_id)

Load a figure with all it’s lazy load fields to have all required info available.

session = <tvb.core.entities.storage.session_maker.SessionMaker object at 0x7f71af27b750>

set_operation_and_group_visibility(entity_gid, is_visible, is_operation_group=False)

Sets the operation visibility.

If ‘is_operation_group’ is True than this method will change the visibility for all the operation from the OperationGroup with the GID field equal to ‘entity_gid’.

try_get_operation_by_id(operation_id)

Try to call self.get_operation_by_id, but when operation was not found, instead of failing, return None. This could be called from situations like: stopping & removing op. A check for None is compulsory after this call!

project_dao

DAO operation related to Users and Projects are defined here.

class tvb.core.entities.storage.project_dao.CaseDAO

Bases: tvb.core.entities.storage.root_dao.RootDAO

USER and PROJECT RELATED OPERATIONS

add_members_to_project(proj_id, selected_user_ids)

Add link between Users and Project.

compute_user_generated_disk_size(user_id)

Do a SUM on DATA_TYPES table column DISK_SIZE, for the current user. :returns 0 when no DT are found, or SUM from DB.

count_projects_for_name(name, different_id)

Retrieve the number of projects with a given name currently in DB.

count_users_for_name(name)

Retrieve the number of users in DB for a given name.

delete_members_for_project(project_id, members)

Remove all linked user to current project.

delete_project(project_id)

Remove PROJECT entity by ID.

get_administrators()

Retrieve all users with Admin role

get_all_projects(page_start=0, page_size=20, is_count=False)

Retrieve all Project entities currently in the system. WARNING: use this wisely, as it might easily overflow the system.

get_all_users(different_name=' ', page_start=0, page_size=20, is_count=False)

Retrieve all USERS in DB, except current user and system user.

get_link(dt_id, project_id)

:return link between a given DT and a given project id

get_linkable_projects_for_user(user_id, data_id)

Return all projects a given user can link some data given by a data_id to.

get_links_for_project(project_id)

:return all links referring to a given project_id

get_members_of_project(proj_id)

Retrieve USER entities with rights on current project.

get_project_by_gid(project_gid)

Retrieve PROJECT entity for a given identifier. THROW SqlException when not found.

get_project_by_id(project_id)

Retrieve PROJECT entity for a given identifier. THROW SqlException when not found.

get_project_disk_size(project_id)

Do a SUM on DATA_TYPES table column DISK_SIZE, for the current project. :returns 0 when no DT are found, or SUM from DB.

get_project_for_operation(operation_id)

Find parent project for current operation. THROW SqlException when not found.

get_projects_for_user(user_id, page_start=0, page_size=20, is_count=False)

Return all projects a given user can access (administrator or not).

get_system_user()

Retrieve System user from DB.

get_user_by_email(email, name_hint='')

Find a user by email address and name.

Parameters:

• email – Valid email address string, to search for its exact match in DB
• name_hint – string for a user’s name; to search with like in DB

Returns:

None if none or more than one users matches the criteria.

get_user_by_id(user_id)

Retrieve USER entity by ID.

get_user_by_name(name)

Retrieve USER entity by name.

get_user_for_datatype(dt_id)

Get the user who created a DT

session = <tvb.core.entities.storage.session_maker.SessionMaker object at 0x7f71af27b750>

root_dao

Base DAO behavior.

class tvb.core.entities.storage.root_dao.RootDAO

Bases: builtins.object

GLOBAL METHODS

EXCEPTION_DATATYPE_GROUP = 'DataTypeGroup'

EXCEPTION_DATATYPE_SIMULATION = 'SimulationState'

get_datatype_by_id(data_id)

Retrieve DataType entity by ID.

get_generic_entity(entity_type, filter_value, select_field='id')

Retrieve an entity of entity_type, filtered by select_field = filter_value.

get_time_series_by_gid(data_gid)

logger = <Logger tvb.core.entities.storage.root_dao (INFO)>

remove_datatype(gid)

When removing dataType, load fully so that sql-alchemy removes from all tables referenced.

remove_entity(entity_class, entity_id)

Find entity by Id and Type, end then remove it. Return True, when entity was removed successfully, of False when exception.

session = <tvb.core.entities.storage.session_maker.SessionMaker object at 0x7f71af27b750>

store_entities(entities_list)

Store in DB a list of generic entities.

store_entity(entity, merge=False)

Store in DB one generic entity.

session_maker

tvb.core.entities.storage.session_maker.MetaClassFactory(decorator_functions, new_attributes)

A meta-class factory that creates a meta-class which makes sure a list of decorators are applied to all it’s classes and also adds a dictionary of attributes.

Parameters:

• decorator_functions – a list of functions. These will be applied as decorators to all methods from the class that uses the returned meta-class.
• new_attributes – a dictionary of attribute_name & attribute_value pairs that will be added to the class that uses the returned meta-class

tvb.core.entities.storage.session_maker.SESSION_META_CLASS

alias of MetaClass

tvb.core.entities.storage.session_maker.SessionMaker(*args, **kwargs)

Called when a new instance is about to be created.

class tvb.core.entities.storage.session_maker.SessionsStack

Bases: builtins.object

Helper class that holds a stack of SqlAlchemy’s session object and a counter that keeps track of how many transactions are opened.

close_session()

Method called by all ‘add_session decorated methods. First check if there are any changes that needed to be committed but weren’t. Then either close the session if it’s not part of a transaction, or just expunge all objects otherwise.

close_transaction()

Close a transaction. Make sure to commit beforehand so all changes are written to database. Then depending on if we are top level or not either close or expunge session.

current_session

Property just for ease of access. Current session will always be top of stack.

open_session()

Create a new session. If we are part of a transaction we bind it to the parent session, otherwise just create a new session.

rollback_transaction()

RollBack a transaction. If we are part of nested transaction - rollback everything up to top parent transaction.

start_transaction()

Start a new transaction. If this is top level transaction just created new session. Otherwise depending if we support nested or not, either raise exception or create a session bound to parent one.

tvb.core.entities.storage.session_maker.add_session(func)

Decorator that handles session related precautions before/after method call. Before each new method a session is created that will later on be closed/rolled back as necessary. This is intended to be used on all DAO methods

tvb.core.entities.storage.session_maker.singleton(cls)

Class decorator that makes sure only one instance of that class is ever returned.

tvb.core.entities.storage.session_maker.transactional(func)

Decorator that makes sure that all DAO calls that will result from the decorated method will be encapsulated in a transaction that will be rolled back if any unexpected exceptions appear. This is intended to be used on service layer methods.

workflow_dao

DAO layer for WorkFlow and Burst entities.

class tvb.core.entities.storage.workflow_dao.WorkflowDAO

Bases: tvb.core.entities.storage.root_dao.RootDAO

DAO layer for WorkFlow and Burst entities.

get_all_datatypes_in_burst(burst_id)

Get all dataTypes in burst, order by their creation, desc.

:param burst_id BurstConfiguration Identifier. :returns: list dataType GIDs or empty list.

get_available_portlets()

Get all the stored portlets form the db.

get_configured_portlets_for_id(portlet_id)

Get the workflow steps that were generated from the portlet given by portlet_id.

get_dynamic(dyn_id)

get_dynamic_by_name(name)

get_dynamics_for_user(user_id)

get_non_validated_entities(reference_time)

Get a list of all categories, portlets and algorithm groups that were not found valid since the reference_time. Used in initializer on each start to filter out any entities that for some reason became invalid. :return tuple (list of entities to get invalidated) (list of entities to be removed)

get_portlet_by_id(portlet_id)

Given an portlet id retieve the portlet entity.

get_portlet_by_identifier(portlet_identifier)

Given an identifer retieve the portlet that corresponds to it.

get_visualization_steps(workflow_id)

Retrieve all the visualization steps for a workflow.

get_workflow_by_id(workflow_id)

“Returns the workflow instance with the given id

get_workflow_for_operation_id(operation_id)

Get the workflow from which operation_id was generated.

get_workflow_step_by_step_index(workflow_id, step_index)

Returns:WorkflowStep entity or None.

get_workflow_step_for_operation(operation_id)

Returns the executed workflow step from which resulted the operation with the given id ‘operation_id’. Returns None if there is no such executed workflow step.

get_workflow_steps(workflow_id)

Retrieve all the simulation/analyzers steps for a workflow.

get_workflow_steps_for_position(workflow_id, tab_index, index_in_tab)

Retrieve a list of analyzers corresponding to current cell in the portlets grid. Will be used for deciding the interface.

get_workflows_for_burst(burst_id, is_count=False)

Returns all the workflows that were launched for this burst id

session = <tvb.core.entities.storage.session_maker.SessionMaker object at 0x7f71af27b750>