storage Package
This is just a shortcut, in order to make all DAO functions
accessible from a single point, in an uniform manner,
without supplementary name space.
We want DAO functions to be separated in multiple files,
because they are too many to easily follow.
-
class tvb.core.entities.storage.DAO[source]
Bases: tvb.core.entities.storage.datatype_dao.DatatypeDAO, tvb.core.entities.storage.operation_dao.OperationDAO, tvb.core.entities.storage.project_dao.CaseDAO, tvb.core.entities.storage.burst_dao.BurstDAO, tvb.core.entities.storage.workflow_dao.WorkflowDAO
Empty class, build only for inheriting from all DAO classes.
-
session = <tvb.core.entities.storage.session_maker.SessionMaker object at 0x7f78f97ae4d0>
burst_dao
-
class tvb.core.entities.storage.burst_dao.BurstDAO[source]
Bases: tvb.core.entities.storage.root_dao.RootDAO
DAO layer for Burst entities.
-
count_bursts_with_name(burst_name, project_id)[source]
Return the number of burst already named ‘custom_b%’ and NOT ‘custom_b%_%’ in current project.
-
get_burst_by_id(burst_id)[source]
Get the BurstConfiguration entity with the given id
-
get_burst_for_direct_operation_id(operation_id, is_group=False)[source]
-
get_burst_for_operation_id(operation_id, is_group=False)[source]
-
get_bursts_for_project(project_id, page_start=0, page_size=200, count=False)[source]
Get latest 50 BurstConfiguration entities for the current project
-
get_max_burst_id()[source]
Return the maximum of the currently stored burst IDs to be used as the new burst name.
This is not a thread-safe value, but we use it just for a label.
-
session = <tvb.core.entities.storage.session_maker.SessionMaker object at 0x7f78f97ae4d0>
datatype_dao
DAO operations related to generic DataTypes are defined here.
-
class tvb.core.entities.storage.datatype_dao.DatatypeDAO[source]
Bases: tvb.core.entities.storage.root_dao.RootDAO
DATATYPE and DATA_TYPES_GROUPS RELATED METHODS
-
compute_bursts_disk_size(burst_gids)[source]
SUM the disk_size of all data types generated by each requested burst
Do not count DataType Groups as those already include the size of the entities inside the group.
:returns a map from burst id to disk size
-
count_all_datatypes()[source]
Gives you the count of all the datatypes currently stored by TVB. Is used by
the file storage update manager to upgrade from version to the next.
-
count_datatypes(project_id, datatype_class)[source]
-
count_datatypes_in_burst(burst_gid)[source]
Returns the number of DataTypes from the specified BurstConfiguration ID.
-
count_datatypes_in_group(datatype_group_id)[source]
Returns the number of DataTypes from the specified DataTypeGroup ID.
-
get_all_datatypes(page_start=0, page_size=200)[source]
Return a list with all of the datatypes currently available in TVB. Is used by
the file storage update manager to upgrade from version to the next.
Parameters: |
- page_start – the index from which to start adding datatypes to the result list
- page_size – maximum number of entities to retrieve
|
-
get_data_in_project(project_id, visibility_filter=None, filter_value=None)[source]
Get all the DataTypes for a given project, including Linked Entities and DataType Groups.
Parameters: |
- visibility_filter – when not None, will filter by DataTye fields
- filter_value – when not None, will filter with ilike multiple DataType string attributes
|
-
get_datatype_by_gid(gid, load_lazy=True, load_lazy_extra_info=False)[source]
Retrieve a DataType DB reference by a global identifier.
-
get_datatype_details(datatype_gid)[source]
Returns the details for the dataType with the given GID.
-
get_datatype_group_by_gid(datatype_group_gid)[source]
Returns the DataTypeGroup with the specified gid.
-
get_datatype_group_disk_size(dt_group_id)[source]
Return the size of all the DataTypes from this datatype group.
-
get_datatype_in_group(datatype_group_id=None, operation_group_id=None)[source]
Return a list of id-s of the DataTypes in the given dt group.
-
get_datatypegroup_by_op_group_id(operation_group_id)[source]
Returns the DataTypeGroup corresponding to a certain OperationGroup.
-
get_datatypegroup_for_project(project_id)[source]
Returns all DataTypeGroup entities from a project.
-
get_datatypes_for_range(op_group_id, range_json)[source]
Retrieve from DB, DataTypes resulted after executing a specific range operation.
-
get_datatypes_from_datatype_group(datatype_group_id)[source]
Retrieve all datatype which are part from the given datatype group.
-
get_datatypes_in_project(project_id, only_visible=False)[source]
Get all the DataTypes for a given project with no other filter apart from the projectId
-
get_disk_size_for_operation(operation_id)[source]
Return the disk size for the operation by summing over the disk space of the resulting DataTypes.
-
get_last_data_with_uid(uid, datatype_class=<class 'tvb.core.entities.model.model_datatype.DataType'>)[source]
Retrieve the last dataType ID witch has UDI field as
the passed parameter, or None if nothing found.
-
get_linked_datatypes_in_project(project_id)[source]
Return a list of datatypes linked into this project
:param project_id: the id of the project
-
get_links_for_datatype(data_id)[source]
Get the links to a specific datatype
-
get_number_of_bursts(project_id)[source]
-
get_selections_for_project(project_id, datatype_gid, filter_ui_name=None)[source]
Get available selections for a given project and data type.
-
get_stored_pse_filters(datatype_group_gid, filter_ui_name=None)[source]
Returns: | Stored PSE filters for a given DatTypeGroup or None |
-
get_summary_for_group(datatype_group_id)[source]
:return (disk_size SUM, subject)
-
get_values_of_datatype(project_id, datatype_class, filters=None, page_size=50)[source]
Retrieve a list of dataTypes matching a filter inside a project.
:returns: (results, total_count) maximum page_end rows are returned, to avoid endless time when loading a page
-
is_datatype_group(datatype_gid)[source]
Used to check if the DataType with the specified GID is a DataTypeGroup.
-
session = <tvb.core.entities.storage.session_maker.SessionMaker object at 0x7f78f97ae4d0>
-
set_datatype_visibility(datatype_gid, is_visible)[source]
Sets the dataType visibility. If the given dataType is a dataTypeGroup or it is part of a
dataType group than this method will set the visibility for each dataType from this group.
-
try_load_last_entity_of_type(project_id, datatype_class)[source]
operation_dao
DAO operations related to Algorithms and User Operations are defined here.
-
class tvb.core.entities.storage.operation_dao.OperationDAO[source]
Bases: tvb.core.entities.storage.root_dao.RootDAO
OPERATION RELATED METHODS
-
compute_disk_size_for_started_ops(user_id)[source]
Get all the disk space that should be reserved for the started operations of this user.
-
count_resulted_datatypes(operation_id)[source]
Returns the number of resulted datatypes from the specified operation.
-
filter_category(displayname, rawinput, display, launchable, order_nr)[source]
Retrieve category with given id
-
get_adapters_from_categories(categories)[source]
Retrieve a list of stored adapters in the given categories.
-
get_algorithm_by_id(algorithm_id)[source]
-
get_algorithm_by_module(module_name, class_name)[source]
-
get_algorithm_categories()[source]
Retrieve all existent categories of Algorithms.
-
get_all_algorithms()[source]
-
get_all_operations_for_uploaders(project_id)[source]
Returns all finished upload operations.
-
get_applicable_adapters(compatible_class_names, launch_categ)[source]
Retrieve a list of algorithms in a given list of categories with a given dataType classes as required input.
-
get_category_by_id(categ_id)[source]
Retrieve category with given id
-
get_figure_count(project_id, user_id)[source]
Used to generate sequential image names.
-
get_filtered_operations(project_id, filter_chain, page_start=0, page_size=200, is_count=False)[source]
Parameters: |
- project_id – current project ID
- filter_chain – instance of FilterChain
- is_count – when True, return a number, otherwise the list of operation entities
|
:return a list of filtered operation in current project, page by page, or the total count for them.
-
get_launchable_categories(elimin_viewers=False)[source]
Retrieve algorithm categories which can be launched on right-click (optionally filter visualizers)
-
get_operation_by_gid(operation_gid)[source]
Retrieve OPERATION entity for a given gid.
-
get_operation_by_id(operation_id)[source]
Retrieve OPERATION entity for a given Identifier.
-
get_operation_lazy_by_gid(operation_gid)[source]
Retrieve OPERATION entity for a given gid.
-
get_operation_numbers(proj_id)[source]
Count total number of operations started for current project.
-
get_operation_process_for_operation(operation_id)[source]
Get the OperationProcessIdentifier for this operation id.
-
get_operationgroup_by_gid(gid)[source]
Retrieve by GID
-
get_operationgroup_by_id(op_group_id)[source]
Retrieve by ID
-
get_operations(status=['4-PENDING', '3-STARTED'], algorithm_classname='SimulatorAdapter')[source]
-
get_operations_in_group(operation_group_id, is_count=False, only_first_operation=False, only_gids=False)[source]
Retrieve OPERATION entities for a given group.
-
get_operations_with_error_in_project(project_id)[source]
Retrieve OPERATION with errors entities for a given project.
-
get_previews(project_id, user_id=None, selected_session_name='all_sessions')[source]
This method returns a tuple of 2 elements. The first element represents a dictionary
of form {‘$session_name’: [list_of_figures]}. This dictionary contains data only for the selected self.session.
If the selected session is ‘all_sessions’ than it will contain data for all the sessions.
The second element of the returned tuple is a dictionary of form
{‘$session_name’: $no_of_figures_in_this_session, ...}.
This dictionary contains information about all the sessions.
- selected_session_name - represents the name of the session for which you
- want to obtain the stored figures.
-
get_raw_categories()[source]
Retrieve categories with raw_input = true
-
get_results_for_operation(operation_id, filters=None)[source]
Retrieve DataTypes entities, resulted after executing an operation.
-
get_uploader_categories()[source]
Retrieve categories with raw_input = true
-
get_visualisers_categories()[source]
Retrieve categories with display = true
-
is_upload_operation(operation_gid)[source]
Returns True only if the operation with the given gid is an upload operation.
-
load_figure(figure_id)[source]
Load a figure with all it’s lazy load fields to have all required
info available.
-
session = <tvb.core.entities.storage.session_maker.SessionMaker object at 0x7f78f97ae4d0>
-
set_operation_and_group_visibility(entity_gid, is_visible, is_operation_group=False)[source]
Sets the operation visibility.
If ‘is_operation_group’ is True than this method will change the visibility for all
the operation from the OperationGroup with the GID field equal to ‘entity_gid’.
-
try_get_operation_by_id(operation_id)[source]
Try to call self.get_operation_by_id, but when operation was not found, instead of failing, return None.
This could be called from situations like: stopping & removing op.
A check for None is compulsory after this call!
project_dao
DAO operation related to Users and Projects are defined here.
-
class tvb.core.entities.storage.project_dao.CaseDAO[source]
Bases: tvb.core.entities.storage.root_dao.RootDAO
USER and PROJECT RELATED OPERATIONS
-
add_members_to_project(proj_id, selected_user_ids)[source]
Add link between Users and Project.
-
compute_user_generated_disk_size(user_id)[source]
Do a SUM on DATA_TYPES table column DISK_SIZE, for the current user.
:returns 0 when no DT are found, or SUM from DB.
-
count_projects_for_name(name, different_id)[source]
Retrieve the number of projects with a given name currently in DB.
-
count_users_for_name(name)[source]
Retrieve the number of users in DB for a given name.
-
delete_members_for_project(project_id, members)[source]
Remove all linked user to current project.
-
delete_project(project_id)[source]
Remove PROJECT entity by ID.
-
get_administrators()[source]
Retrieve all users with Admin role
-
get_all_projects(page_start=0, page_size=200, is_count=False)[source]
Retrieve all Project entities currently in the system.
WARNING: use this wisely, as it might easily overflow the system.
-
get_all_users(different_names=None, page_start=0, page_size=200, is_count=False)[source]
Retrieve all USERS in DB, except given users and system user.
-
get_link(dt_id, project_id)[source]
:return link between a given DT and a given project id
-
get_linkable_projects_for_user(user_id, data_id)[source]
Return all projects a given user can link some data given by a data_id to.
-
get_links_for_project(project_id)[source]
:return all links referring to a given project_id
-
get_members_of_project(proj_id)[source]
Retrieve USER entities with rights on current project.
-
get_project_by_gid(project_gid)[source]
Retrieve PROJECT entity for a given identifier.
THROW SqlException when not found.
-
get_project_by_id(project_id)[source]
Retrieve PROJECT entity for a given identifier.
THROW SqlException when not found.
-
get_project_by_name(project_name)[source]
Retrieve PROJECT entity for a given name.
THROW SQLException when not found.
-
get_project_disk_size(project_id)[source]
Do a SUM on DATA_TYPES table column DISK_SIZE, for the current project.
:returns 0 when no DT are found, or SUM from DB.
-
get_project_for_operation(operation_id)[source]
Find parent project for current operation.
THROW SqlException when not found.
-
get_project_lazy_by_gid(project_gid)[source]
Retrieve PROJECT entity for a given identifier.
THROW SqlException when not found.
-
get_projects_for_user(user_id, page_start=0, page_size=200, is_count=False)[source]
Return all projects a given user can access (administrator or not).
-
get_system_user()[source]
Retrieve System user from DB.
-
get_user_by_email(email, name_hint='')[source]
Find a user by email address and name.
Parameters: |
- email – Valid email address string, to search for its exact match in DB
- name_hint – string for a user’s name; to search with like in DB
|
Returns: | None if none or more than one users matches the criteria.
|
-
get_user_by_gid(gid)[source]
Retrieve USER entity by gid.
-
get_user_by_id(user_id)[source]
Retrieve USER entity by ID.
-
get_user_by_name(name)[source]
Retrieve USER entity by name.
-
get_user_for_datatype(dt_id)[source]
Get the user who created a DT
-
session = <tvb.core.entities.storage.session_maker.SessionMaker object at 0x7f78f97ae4d0>
root_dao
Base DAO behavior.
-
class tvb.core.entities.storage.root_dao.RootDAO[source]
Bases: builtins.object
GLOBAL METHODS
-
EXCEPTION_DATATYPE_GROUP = 'DataTypeGroup'
-
EXCEPTION_DATATYPE_SIMULATION = 'SimulationState'
-
get_datatype_by_id(data_id)[source]
Retrieve DataType entity by ID.
-
get_generic_entity(entity_type, filter_value, select_field='id')[source]
Retrieve an entity of entity_type, filtered by select_field = filter_value.
-
get_time_series_by_gid(data_gid)[source]
-
logger = <Logger tvb.core.entities.storage.root_dao (INFO)>
-
remove_datatype(gid)[source]
When removing dataType, load fully so that sql-alchemy removes from all tables referenced.
-
remove_entity(entity_class, entity_id)[source]
Find entity by Id and Type, end then remove it.
Return True, when entity was removed successfully, of False when exception.
-
session = <tvb.core.entities.storage.session_maker.SessionMaker object at 0x7f78f97ae4d0>
-
store_entities(entities_list)[source]
Store in DB a list of generic entities.
-
store_entity(entity, merge=False)[source]
Store in DB one generic entity.
session_maker
-
tvb.core.entities.storage.session_maker.MetaClassFactory(decorator_functions, new_attributes)[source]
A meta-class factory that creates a meta-class which makes sure a list of decorators
are applied to all it’s classes and also adds a dictionary of attributes.
Parameters: |
- decorator_functions – a list of functions. These will be applied as decorators to
all methods from the class that uses the returned meta-class.
- new_attributes – a dictionary of attribute_name & attribute_value pairs that will
be added to the class that uses the returned meta-class
|
-
tvb.core.entities.storage.session_maker.SESSION_META_CLASS
alias of MetaClass
-
tvb.core.entities.storage.session_maker.SessionMaker(*args, **kwargs)[source]
Called when a new instance is about to be created.
-
class tvb.core.entities.storage.session_maker.SessionsStack[source]
Bases: builtins.object
Helper class that holds a stack of SqlAlchemy’s session object and a counter that
keeps track of how many transactions are opened.
-
close_session()[source]
Method called by all ‘add_session decorated methods. First check if there
are any changes that needed to be committed but weren’t. Then either close the
session if it’s not part of a transaction, or just expunge all objects otherwise.
-
close_transaction()[source]
Close a transaction. Make sure to commit beforehand so all changes are written to database. Then
depending on if we are top level or not either close or expunge session.
-
current_session[source]
Property just for ease of access. Current session will always be top of stack.
-
open_session()[source]
Create a new session. If we are part of a transaction we bind it to the parent
session, otherwise just create a new session.
-
rollback_transaction()[source]
RollBack a transaction.
If we are part of nested transaction - rollback everything up to top parent transaction.
-
start_transaction()[source]
Start a new transaction. If this is top level transaction just created new session.
Otherwise depending if we support nested or not, either raise exception or create a session
bound to parent one.
-
tvb.core.entities.storage.session_maker.add_session(func)[source]
Decorator that handles session related precautions before/after method call.
Before each new method a session is created that will later on be closed/rolled back as necessary.
This is intended to be used on all DAO methods
-
tvb.core.entities.storage.session_maker.build_db_engine()[source]
-
tvb.core.entities.storage.session_maker.singleton(cls)[source]
Class decorator that makes sure only one instance of that class is ever returned.
-
tvb.core.entities.storage.session_maker.transactional(func)[source]
Decorator that makes sure that all DAO calls that will result from the decorated
method will be encapsulated in a transaction that will be rolled back if any
unexpected exceptions appear.
This is intended to be used on service layer methods.
workflow_dao
DAO layer for WorkFlow and Burst entities.
-
class tvb.core.entities.storage.workflow_dao.WorkflowDAO[source]
Bases: tvb.core.entities.storage.root_dao.RootDAO
DAO layer for WorkFlow and Burst entities.
-
get_available_portlets()[source]
Get all the stored portlets form the db.
-
get_dynamic(dyn_id)[source]
-
get_dynamic_by_name(name)[source]
-
get_dynamics_for_user(user_id)[source]
-
get_non_validated_entities(reference_time)[source]
Get a list of all categories, portlets and algorithm groups that were not found valid since the reference_time.
Used in initializer on each start to filter out any entities that for some reason became invalid.
:return tuple (list of entities to get invalidated) (list of entities to be removed)
-
get_portlet_by_id(portlet_id)[source]
Given an portlet id retieve the portlet entity.
-
get_portlet_by_identifier(portlet_identifier)[source]
Given an identifer retieve the portlet that corresponds to it.
-
session = <tvb.core.entities.storage.session_maker.SessionMaker object at 0x7f78f97ae4d0>