MOMA#

class momaapi.moma.MOMA(dir_moma: str, paradigm: typing_extensions.Literal[standard, few - shot] = 'standard', reset_cache: bool = False)#

Class to interface with the MOMA-LRG dataset. Initialization requires passing in a directory containing the MOMA-LRG dataset.

The MOMA object can be used for few-shot experiments, which reduces the number of classes and examples, or used in the standard paradigm.

The following conventions are used throughout the documentation as shorthand:

  • act: activity

  • sact: sub-activity

  • hoi: higher-order interaction

  • entity: entity

  • ia: intransitive action

  • ta: transitive action

  • att: attribute

  • rel: relationship

  • ann: annotation

  • id: instance ID

  • cname: class name

  • cid: class ID

Parameters

  • dir_moma (str) – directory containing the MOMA dataset

  • paradigm (Literal['standard', 'few-shot']) – the experiment configuration, which is either 'standard' or 'few-shot'

  • reset_cache (bool) – flag that indicates whether to reset cached data

  • taxonomy (Taxonomy) – a Taxonomy object containing information about the dataset taxonomy

  • lookup (Lookup) – a Lookup object containing information about class IDs and class names

  • statistics – a Statistics object that can generate dataset-level statics

  • num_classes (int) – the number of classes contained in the MOMA object

get_anns_act(ids_act: list) → list#

Given activity instance IDs, return their annotations

Parameters

ids_act – activity instance IDs

Returns

annotations for the given activity instance IDs

Return type

list

get_anns_hoi(ids_hoi: list) → list#

Given higher-order interaction instance IDs, return their annotations

Parameters

ids_hoi – higher-order interaction instance IDs

Returns

annotations for the given higher-order interaction instance IDs

Return type

list

get_anns_sact(ids_sact: list) → list#

Given sub-activity instance IDs, return their annotations

Parameters

ids_sact – sub-activity instance IDs

Returns

annotations for the given sub-activity instance IDs

Return type

list

get_cids(kind: typing_extensions.Literal[act, sact, actor, object, ia, ta, att, rel], threshold: int, split: typing_extensions.Literal[train, val, test, either, all, combined]) → list#
Parameters

  • kind (Literal['act', 'sact', 'actor', 'object', 'ia', 'ta', 'att', 'rel']) – the kind of annotations needed to be retrieved

  • threshold (int) – exclude classes with fewer than this number of total instances

  • split (Literal['train', 'val', 'test', 'either', 'all', 'combined']) – the split to be used for the retrieval. Here, train refers to the training set, val refers to the validation set, and test refers to the test set. either will exclude a class if the smallest number of instances in across splits is less than the threshold, all will exclude a class if the largest number of instances in across splits is less than the threshold, and combined will exclude a class if the smallest number of instances in across splits is less than the threshold

Returns

a list of class IDs

Return type

List[int]

get_clips(ids_hoi: list) → list#

Given higher-order interaction instance IDs, return their clips

Parameters

ids_hoi – higher-order interaction instance IDs

Returns

clips for the given higher-order interaction instance IDs

Return type

list

get_cnames(cids_act: Optional[list] = None, cids_sact: Optional[list] = None, cids_actor: Optional[list] = None, cids_object: Optional[list] = None, cids_ia: Optional[list] = None, cids_ta: Optional[list] = None, cids_att: Optional[list] = None, cids_rel: Optional[list] = None) → list#

Returns the associated class names given the class IDs.

Parameters

  • cids_act (Optional[List[int]]) – a list of class IDs of activities

  • cids_sact (Optional[List[int]]) – a list of class IDs of sub-activities

  • cids_actor (Optional[List[int]]) – a list of class IDs of actors

  • cids_object (Optional[List[int]]) – a list of class IDs of objects

  • cids_ia (Optional[List[int]]) – a list of class IDs of intransitive actions

  • cids_ta (Optional[List[int]]) – a list of class IDs of transitive actions

  • cids_att (Optional[List[int]]) – a list of class IDs of attributes

  • cids_rel (Optional[List[int]]) – a list of class IDs of relationships

Returns

a list of class names

Return type

List[str]

get_ids_act(split: Optional[str] = None, cnames_act: Optional[list] = None, ids_sact: Optional[list] = None, ids_hoi: Optional[list] = None) → list#

Get the unique activity instance IDs that satisfy certain conditions

Parameters

  • split (Union['train', 'val', 'test', 'either', 'all', 'combined']) – get activity IDs that belong to the given dataset split

  • cnames_act (list) – get activity IDs that belong to the given activity classes

  • ids_sact (list) – get activity IDs for given sub-activity IDs

  • ids_hoi (list) – get activity IDs for given higher-order interaction IDs [ids_hoi]

Returns

a list of activity IDs

Return type

list

get_ids_hoi(split: Optional[str] = None, ids_act: Optional[list] = None, ids_sact: Optional[list] = None, cnames_actor: Optional[list] = None, cnames_object: Optional[list] = None, cnames_ia: Optional[list] = None, cnames_ta: Optional[list] = None, cnames_att: Optional[list] = None, cnames_rel: Optional[list] = None) → list#

Get the unique higher-order interaction instance IDs that satisfy certain conditions dataset split

Parameters

  • split (Union['train', 'val', 'test', 'either', 'all', 'combined']) – get higher-order interaction IDs [ids_hoi] that belong to the given dataset split

  • ids_act (list) – get higher-order interaction IDs [ids_hoi] for given activity IDs [ids_act]

  • ids_sact (list) – get higher-order interaction IDs [ids_hoi] for given sub-activity IDs [ids_sact]

  • cnames_actor (list) – get higher-order interaction IDs [ids_hoi] for given actor class names [cnames_actor]

  • cnames_object (list) – get higher-order interaction IDs [ids_hoi] for given object class names [cnames_object]

  • cnames_ia (list) – get higher-order interaction IDs [ids_hoi] for given intransitive action class names [cnames_ia]

  • cnames_ta (list) – get higher-order interaction IDs [ids_hoi] for given transitive action class names [cnames_ta]

  • cnames_att (list) – get higher-order interaction IDs [ids_hoi] for given attribute class names [cnames_att]

  • cnames_rel (list) – get higher-order interaction IDs [ids_hoi] for given relationship class names [cnames_rel]

get_ids_sact(split: Optional[str] = None, cnames_sact: Optional[list] = None, ids_act: Optional[list] = None, ids_hoi: Optional[list] = None, cnames_actor: Optional[list] = None, cnames_object: Optional[list] = None, cnames_ia: Optional[list] = None, cnames_ta: Optional[list] = None, cnames_att: Optional[list] = None, cnames_rel: Optional[list] = None) → list#

Get the unique sub-activity instance IDs that satisfy certain conditions dataset split

Parameters

  • split (Union['train', 'val', 'test', 'either', 'all', 'combined']) – get sub-activity IDs [ids_sact] that belong to the given dataset split

  • cnames_sact (list) – get sub-activity IDs [ids_sact] for given sub-activity class names [cnames_sact]

  • ids_act (list) – get sub-activity IDs [ids_sact] for given activity IDs [ids_act]

  • ids_hoi (list) – get sub-activity IDs [ids_sact] for given higher-order interaction IDs [ids_hoi]

  • cnames_actor (list) – get sub-activity IDs [ids_sact] for given actor class names [cnames_actor]

  • cnames_object (list) – get sub-activity IDs [ids_sact] for given object class names [cnames_object]

  • cnames_ia (list) – get sub-activity IDs [ids_sact] for given intransitive action class names [cnames_ia]

  • cnames_ta (list) – get sub-activity IDs [ids_sact] for given transitive action class names [cnames_ta]

  • cnames_att (list) – get sub-activity IDs [ids_sact] for given attribute class names [cnames_att]

  • cnames_rel (list) – get sub-activity IDs [ids_sact] for given relationship class names [cnames_rel]

Returns

a list of sub-activity IDs

Return type

list

get_metadata(ids_act: list) → list#

Get the metadata for the given activity IDs. The metadata returned is that associated with the raw videos that contain instances of the activity IDs.

Parameters

ids_act – get metadata for the given activity IDs

Returns

video metadata for the given activity ID

Return type

list

get_paths(ids_act: Optional[list] = None, ids_sact: Optional[list] = None, ids_hoi: Optional[list] = None, id_hoi_clip: Optional[str] = None, full_res: bool = False, sanity_check: bool = True) → list#

Given activity, sub-activity, higher-order interaction, or clip IDs, return the paths to the videos.

Parameters

  • ids_act (list) – activity instance IDs

  • ids_sact (list) – sub-activity instance IDs

  • ids_hoi (list) – higher-order interaction instance IDs

  • id_hoi_clip (str) – clip ID

  • full_res (bool) – return full-resolution videos

  • sanity_check (bool) – check that the video exists

Returns

paths to the videos

Return type

list

is_sact(id_act: int, time: int, absolute: bool = False) → bool#

Checks whether a certain time in an activity has a sub-activity.

Parameters

  • id_act (int) – activity ID

  • time (int) – time in the activity

  • absolute (bool) – relative to the full video if True or relative to the activity video if False

map_cids(split: typing_extensions.Literal[train, val, test, either, all, combined], cids_act_contiguous: Optional[list] = None, cids_act: Optional[list] = None, cids_sact_contiguous: Optional[list] = None, cids_sact: Optional[list] = None) → list#

Map class IDs between standard class IDs and split-specific contiguous class IDs. For the few-shot paradigm only.

Parameters

  • split (Literal['train', 'val', 'test', 'either', 'all', 'combined']) – the dataset split to use

  • cids_act_contiguous (Optional[List[int]]) – a list of contiguous class IDs in the activity set

  • cids_act (Optional[List[int]]) – a list of class IDs in the activity set

  • cids_sact_contiguous (Optional[List[int]]) – a list of contiguous class IDs in the sub-activity set

  • cids_sact (Optional[List[int]]) – a list of class IDs in the sub-activity set

Returns

mapping between standard class IDs and split-specific contiguous IDs

sort(ids_sact: Optional[list] = None, ids_hoi: Optional[list] = None, sanity_check: bool = True)#

Given a list of sub-activity or higher-order interaction instance IDs, return them in sorted order by when they occured in the video.

Parameters

  • ids_sact (list) – sub-activity instance IDs

  • ids_hoi (list) – higher-order interaction instance IDs

  • sanity_check (bool) – check that the video exists

Returns

sorted IDs

Return type

list