storage Package

This is just a shortcut, in order to make all DAO functions accessible from a single point, in an uniform manner, without supplementary name space. We want DAO functions to be separated in multiple files, because they are too many to easily follow.

class tvb.core.entities.storage.DAO

Bases: DatatypeDAO, OperationDAO, CaseDAO, BurstDAO, WorkflowDAO

Empty class, build only for inheriting from all DAO classes.

session = <tvb.core.entities.storage.session_maker.SessionMaker object>

burst_dao

class tvb.core.entities.storage.burst_dao.BurstDAO

Bases: RootDAO

DAO layer for Burst entities.

count_bursts_with_name(burst_name, project_id)

Return the number of burst already named ‘custom_b%’ and NOT ‘custom_b%_%’ in current project.

get_burst_by_id(burst_id)

Get the BurstConfiguration entity with the given id

get_burst_for_direct_operation_id(operation_id, is_group=False)

get_burst_for_operation_id(operation_id, is_group=False)

get_bursts_for_project(project_id, page_start=0, page_size=200, count=False)

Get latest 50 BurstConfiguration entities for the current project

get_max_burst_id()

Return the maximum of the currently stored burst IDs to be used as the new burst name. This is not a thread-safe value, but we use it just for a label.

session = <tvb.core.entities.storage.session_maker.SessionMaker object>

datatype_dao

DAO operations related to generic DataTypes are defined here.

class tvb.core.entities.storage.datatype_dao.DatatypeDAO

Bases: RootDAO

DATATYPE and DATA_TYPES_GROUPS RELATED METHODS

compute_bursts_disk_size(burst_gids)

SUM the disk_size of all data types generated by each requested burst Do not count DataType Groups as those already include the size of the entities inside the group. :returns a map from burst id to disk size

count_all_datatypes()

Gives you the count of all the datatypes currently stored by TVB. Is used by the file storage update manager to upgrade from version to the next.

count_datatypes(project_id, datatype_class)

count_datatypes_in_burst(burst_gid)

Returns the number of DataTypes from the specified BurstConfiguration ID.

count_datatypes_in_group(datatype_group_id)

Returns the number of DataTypes from the specified DataTypeGroup ID.

get_all_datatypes(page_start=0, page_size=200)

Return a list with all of the datatypes currently available in TVB. Is used by the file storage update manager to upgrade from version to the next.

Parameters:

• page_start – the index from which to start adding datatypes to the result list

• page_size – maximum number of entities to retrieve

get_data_in_project(project_id, visibility_filter=None, filter_value=None)

Get all the DataTypes for a given project, including Linked Entities and DataType Groups.

Parameters:

• project_id – ID for the current project to filter for

• visibility_filter – when not None, will filter by DataTye fields

• filter_value – when not None, will filter with ilike multiple DataType string attributes

get_datatype_by_gid(gid, load_lazy=True, load_lazy_extra_info=False)

Retrieve a DataType DB reference by a global identifier.

get_datatype_details(datatype_gid)

Returns the details for the dataType with the given GID.

get_datatype_extra_info(datatype_gid)

get_datatype_group_by_gid(datatype_group_gid)

Returns the DataTypeGroup with the specified gid.

get_datatype_group_disk_size(dt_group_id)

Return the size of all the DataTypes from this datatype group.

get_datatype_in_group(datatype_group_id=None, operation_group_id=None)

Return a list of id-s of the DataTypes in the given dt group.

get_datatype_measure_group_from_ts_from_pse(ts_gid, datatype_measure_class)

Having a Time Series that was part of a pse, this method should return the datatype group of a datatype measure that was part of the same pse

get_datatypegroup_by_op_group_id(operation_group_id)

Returns the DataTypeGroup corresponding to a certain OperationGroup.

get_datatypegroup_for_project(project_id)

Returns all DataTypeGroup entities from a project.

get_datatypes_for_range(op_group_id, range_json)

Retrieve from DB, DataTypes resulted after executing a specific range operation.

get_datatypes_from_datatype_group(datatype_group_id)

Retrieve all datatype which are part from the given datatype group.

get_datatypes_in_project(project_id, only_visible=False)

Get all the DataTypes for a given project with no other filter apart from the projectId

get_disk_size_for_operation(operation_id)

Return the disk size for the operation by summing over the disk space of the resulting DataTypes.

get_last_data_with_uid(uid, datatype_class=<class 'tvb.core.entities.model.model_datatype.DataType'>)

Retrieve the last dataType ID witch has UDI field as the passed parameter, or None if nothing found.

get_linked_datatypes_in_project(project_id)

Return a list of datatypes linked into this project :param project_id: the id of the project

get_links_for_datatype(data_id)

Get the links to a specific datatype

get_number_of_bursts(project_id)

get_selections_for_project(project_id, datatype_gid, filter_ui_name=None)

Get available selections for a given project and data type.

get_stored_pse_filters(datatype_group_gid, filter_ui_name=None)

Returns:

Stored PSE filters for a given DatTypeGroup or None

get_summary_for_group(datatype_group_id)

:return (disk_size SUM, subject)

get_values_of_datatype(project_id, datatype_class, filters=None, page_size=50)

Retrieve a list of dataTypes matching a filter inside a project. :returns: (results, total_count) maximum page_end rows are returned, to avoid endless time when loading a page

is_datatype_group(datatype_gid)

Used to check if the DataType with the specified GID is a DataTypeGroup.

session = <tvb.core.entities.storage.session_maker.SessionMaker object>

set_datatype_visibility(datatype_gid, is_visible)

Sets the dataType visibility. If the given dataType is a dataTypeGroup or it is part of a dataType group than this method will set the visibility for each dataType from this group.

try_load_last_entity_of_type(project_id, datatype_class)

exceptions

Created on Jan 15, 2013

exception tvb.core.entities.storage.exceptions.BaseStorageException(message)

Bases: Exception

Base class for all TVB storage exceptions.

exception tvb.core.entities.storage.exceptions.InvalidTransactionAccess(message)

Bases: BaseStorageException

Exception raised in case you have any faulty access to a transaction.

exception tvb.core.entities.storage.exceptions.NestedTransactionUnsupported(message)

Bases: BaseStorageException

Nested transactions are not supported unless in testing.

operation_dao

DAO operations related to Algorithms and User Operations are defined here.

class tvb.core.entities.storage.operation_dao.OperationDAO

Bases: RootDAO

OPERATION RELATED METHODS

compute_disk_size_for_started_ops(user_id)

Get all the disk space that should be reserved for the started operations of this user.

count_resulted_datatypes(operation_id)

Returns the number of resulted datatypes from the specified operation.

filter_category(displayname, rawinput, display, launchable, order_nr)

Retrieve category with given id

get_adapters_from_categories(categories)

Retrieve a list of stored adapters in the given categories.

get_algorithm_by_id(algorithm_id)

get_algorithm_by_module(module_name, class_name)

get_algorithm_categories()

Retrieve all existent categories of Algorithms.

get_all_algorithms()

get_all_operations_for_uploaders(project_id)

Returns all finished upload operations.

get_applicable_adapters(compatible_class_names, launch_categ)

Retrieve a list of algorithms in a given list of categories with a given dataType classes as required input.

get_category_by_id(categ_id)

Retrieve category with given id

get_figure_count(project_id, user_id)

Used to generate sequential image names.

get_filtered_operations(project_id, filter_chain, page_start=0, page_size=200, is_count=False)

Parameters:

• project_id – current project ID

• filter_chain – instance of FilterChain

• is_count – when True, return a number, otherwise the list of operation entities

:return a list of filtered operation in current project, page by page, or the total count for them.

get_launchable_categories(elimin_viewers=False)

Retrieve algorithm categories which can be launched on right-click (optionally filter visualizers)

get_operation_by_gid(operation_gid)

Retrieve OPERATION entity for a given gid.

get_operation_by_id(operation_id)

Retrieve OPERATION entity for a given Identifier.

get_operation_lazy_by_gid(operation_gid)

Retrieve OPERATION entity for a given gid.

get_operation_numbers(proj_id)

Count total number of operations started for current project.

get_operation_process_for_operation(operation_id)

Get the OperationProcessIdentifier for this operation id.

get_operationgroup_by_gid(gid)

Retrieve by GID

get_operationgroup_by_id(op_group_id)

Retrieve by ID

get_operations(status=None, algorithm_classname='SimulatorAdapter')

get_operations_for_hpc_job()

get_operations_in_group(operation_group_id, is_count=False, only_first_operation=False, only_gids=False)

Retrieve OPERATION entities for a given group.

get_operations_with_error_in_project(project_id)

Retrieve OPERATION with errors entities for a given project.

get_previews(project_id, user_id=None, selected_session_name='all_sessions')

This method returns a tuple of 2 elements. The first element represents a dictionary of form {‘$session_name’: [list_of_figures]}. This dictionary contains data only for the selected self.session. If the selected session is ‘all_sessions’ than it will contain data for all the sessions. The second element of the returned tuple is a dictionary of form {‘$session_name’: $no_of_figures_in_this_session, …}. This dictionary contains information about all the sessions.

selected_session_name - represents the name of the session for which you

want to obtain the stored figures.

get_raw_categories()

Retrieve categories with raw_input = true

get_results_for_operation(operation_id)

Retrieve DataTypes entities, resulted after executing an operation.

get_uploader_categories()

Retrieve categories with raw_input = true

get_visualisers_categories()

Retrieve categories with display = true

is_upload_operation(operation_gid)

Returns True only if the operation with the given gid is an upload operation.

load_figure(figure_id)

Load a figure with all it’s lazy load fields to have all required info available.

session = <tvb.core.entities.storage.session_maker.SessionMaker object>

set_operation_and_group_visibility(entity_gid, is_visible, is_operation_group=False)

Sets the operation visibility.

If ‘is_operation_group’ is True than this method will change the visibility for all the operation from the OperationGroup with the GID field equal to ‘entity_gid’.

try_get_operation_by_id(operation_id)

Try to call self.get_operation_by_id, but when operation was not found, instead of failing, return None. This could be called from situations like: stopping & removing op. A check for None is compulsory after this call!

project_dao

DAO operation related to Users and Projects are defined here.

class tvb.core.entities.storage.project_dao.CaseDAO

Bases: RootDAO

USER and PROJECT RELATED OPERATIONS

add_members_to_project(proj_id, selected_user_ids)

Add link between Users and Project.

compute_user_generated_disk_size(user_id)

Do a SUM on DATA_TYPES table column DISK_SIZE, for the current user. :returns 0 when no DT are found, or SUM from DB.

count_projects_for_name(name, different_id)

Retrieve the number of projects with a given name currently in DB.

count_users_for_name(name)

Retrieve the number of users in DB for a given name.

delete_members_for_project(project_id, members)

Remove all linked user to current project.

delete_project(project_id)

Remove PROJECT entity by ID.

get_administrators()

Retrieve all users with Admin role

get_all_projects(page_start=0, page_size=200, is_count=False)

Retrieve all Project entities currently in the system. WARNING: use this wisely, as it might easily overflow the system.

get_all_users(different_names=None, page_start=0, page_size=200, is_count=False)

Retrieve all USERS in DB, except given users and system user.

get_link(dt_id, project_id)

:return link between a given DT and a given project id

get_linkable_projects_for_user(user_id, data_id)

Return all projects a given user can link some data given by a data_id to.

get_links_to_project(project_id)

:return all links referring to a given project_id

get_members_of_project(proj_id)

Retrieve USER entities with rights on current project.

get_project_by_gid(project_gid)

Retrieve PROJECT entity for a given identifier. THROW SqlException when not found.

get_project_by_id(project_id)

Retrieve PROJECT entity for a given identifier. THROW SqlException when not found.

get_project_by_name(project_name)

Retrieve PROJECT entity for a given name. THROW SQLException when not found.

get_project_disk_size(project_id)

Do a SUM on DATA_TYPES table column DISK_SIZE, for the current project. :returns 0 when no DT are found, or SUM from DB.

get_project_for_operation(operation_id)

Find parent project for current operation. THROW SqlException when not found.

get_project_lazy_by_gid(project_gid)

Retrieve PROJECT entity for a given identifier. THROW SqlException when not found.

get_projects_for_user(user_id, page_start=0, page_size=200, is_count=False)

Return all projects a given user can access (administrator or not).

get_system_user()

Retrieve System user from DB.

get_user_by_email(email, name_hint='')

Find a user by email address and name.

Parameters:

• email – Valid email address string, to search for its exact match in DB

• name_hint – string for a user’s name; to search with like in DB

Returns:

None if none or more than one users matches the criteria.

get_user_by_gid(gid)

Retrieve USER entity by gid.

get_user_by_id(user_id)

Retrieve USER entity by ID.

get_user_by_name(name)

Retrieve USER entity by name.

get_user_for_datatype(dt_id)

Get the user who created a DT

session = <tvb.core.entities.storage.session_maker.SessionMaker object>

root_dao

Base DAO behavior.

class tvb.core.entities.storage.root_dao.RootDAO

Bases: object

GLOBAL METHODS

EXCEPTION_DATATYPE_GROUP = 'DataTypeGroup'

EXCEPTION_DATATYPE_SIMULATION = 'SimulationState'

get_datatype_by_id(data_id)

Retrieve DataType entity by ID.

get_generic_entity(entity_type, filter_value, select_field='id')

Retrieve an entity of entity_type, filtered by select_field = filter_value.

get_time_series_by_gid(data_gid)

logger = <Logger tvb.core.entities.storage.root_dao (INFO)>

remove_datatype(gid)

When removing dataType, load fully so that sql-alchemy removes from all tables referenced.

remove_entity(entity_class, entity_id)

Find entity by Id and Type, end then remove it. Return True, when entity was removed successfully, of False when exception.

session = <tvb.core.entities.storage.session_maker.SessionMaker object>

store_entities(entities_list)

Store in DB a list of generic entities.

store_entity(entity, merge=False)

Store in DB one generic entity.

session_maker

tvb.core.entities.storage.session_maker.MetaClassFactory(decorator_functions, new_attributes)

A meta-class factory that creates a meta-class which makes sure a list of decorators are applied to all it’s classes and also adds a dictionary of attributes.

Parameters:

• decorator_functions – a list of functions. These will be applied as decorators to all methods from the class that uses the returned meta-class.

• new_attributes – a dictionary of attribute_name & attribute_value pairs that will be added to the class that uses the returned meta-class

tvb.core.entities.storage.session_maker.SESSION_META_CLASS

alias of MetaClass

tvb.core.entities.storage.session_maker.SessionMaker(*args, **kwargs)

Called when a new instance is about to be created.

class tvb.core.entities.storage.session_maker.SessionsStack

Bases: object

Helper class that holds a stack of SqlAlchemy’s session object and a counter that keeps track of how many transactions are opened.

close_session()

Method called by all ‘add_session decorated methods. First check if there are any changes that needed to be committed but weren’t. Then either close the session if it’s not part of a transaction, or just expunge all objects otherwise.

close_transaction()

Close a transaction. Make sure to commit beforehand so all changes are written to database. Then depending on if we are top level or not either close or expunge session.

property current_session

Property just for ease of access. Current session will always be top of stack.

open_session()

Create a new session. If we are part of a transaction we bind it to the parent session, otherwise just create a new session.

rollback_transaction()

RollBack a transaction. If we are part of nested transaction - rollback everything up to top parent transaction.

start_transaction()

Start a new transaction. If this is top level transaction just created new session. Otherwise depending if we support nested or not, either raise exception or create a session bound to parent one.

tvb.core.entities.storage.session_maker.add_session(func)

Decorator that handles session related precautions before/after method call. Before each new method a session is created that will later on be closed/rolled back as necessary. This is intended to be used on all DAO methods

tvb.core.entities.storage.session_maker.build_db_engine()

tvb.core.entities.storage.session_maker.singleton(cls)

Class decorator that makes sure only one instance of that class is ever returned.

tvb.core.entities.storage.session_maker.transactional(func)

Decorator that makes sure that all DAO calls that will result from the decorated method will be encapsulated in a transaction that will be rolled back if any unexpected exceptions appear. This is intended to be used on service layer methods.

workflow_dao

DAO layer for WorkFlow and Burst entities.

class tvb.core.entities.storage.workflow_dao.WorkflowDAO

Bases: RootDAO

DAO layer for WorkFlow and Burst entities.

get_dynamic(dyn_id)

get_dynamic_by_name(name)

get_dynamics_for_user(user_id)

get_non_validated_entities(reference_time)

Get a list of all categories and algorithm groups that were not found valid since the reference_time. Used in initializer on each start to filter out any entities that for some reason became invalid. :return tuple (list of entities to get invalidated) (list of entities to be removed)

session = <tvb.core.entities.storage.session_maker.SessionMaker object>